XR Stories researcher in residence develops first UK robotic system for virtual production

Steven Xiaotian Dai is an Assistant Professor in the Department of Computer Science at the University of York and the Principal Investigator of the project RAVEN (Real-time Adaptive Virtual-Twin Environment for Next-Generation Robotics in Virtual Production).

In 2025, Steven was awarded a researcher residency with XR Stories to support his ongoing project developing the UK’s first robotic system for virtual production driven by humanoid robots and real-time digital twins. 

As part of his residency, Steven is utilising the XR Stories immersive technology lab as a virtual production stage to test the new system.

On an LED-volume stage, the virtual world is already fully digital — directors can iterate on it in seconds inside the game engine. The physical layer around the camera (operator moves, lighting rigs, practical effects) has no such twin: every change means manual rigging, fresh rehearsals, and added safety overhead.

The RAVEN system embeds a live digital twin of the virtual production stage in Unreal Engine 5 (UE5), Epic’s flagship real-time game engine. It then connects the twin to the Robot Operating System 2 (ROS 2), so that a humanoid robot can perform camera operation — and later lighting/FX — with frame-accurate timing, low latency, and predictive safety.

The work supports directors and camera operators by allowing them to author, preview and play robotic camera moves entirely inside UE5, with visible safety zones and timing aligned to the LED scanout and camera shutter.

The system aims to deliver ultra-low latency control and tightly coupled digital-physical synchronisation. It is not intended to replace human roles but instead will ensure safe and seamless collaboration between humans and robots to enhance precision, efficiency and creative flexibility.

RAVEN is funded by the University’s Research Priming Capital Fund, and partially by the Google Cloud Research Credits Program.

Published on 13 May 2026