As part of our work to understand the future of the screen industries, XR Stories and SIGN researchers are exploring how virtual production is being adopted in the film and television industries and the implications this has for storytelling. Nina Willment and Jon Swords introduce the idea of virtual production and highlight productions which have adopted the approach.
What is virtual production?
Neatly defining virtual production is not straightforward. The technologies used, the methods applied and the uses of virtual production approaches are rapidly evolving.
In broad terms, virtual production is a way of making film and television which harnesses computer generated content allowing real-time visualisation and control of the digital environment in which you are shooting. Importantly, virtual environments are captured ‘in camera’, rather than added in post-production.
Virtual production has seen large investments of time, money and R&D from major production companies and studios. For example, virtual production was used by the BBC for their coverage of the Tokyo Olympics and is regularly used in the production of advertisements and music videos (Kadnar, 2019). It is predicted that from 2022 to 2030, the virtual production industry will have an annual growth rate of 17.8 % globally. In 2021 it was valued as being worth $1.6 billion (Grand View Research, 2022).
BBC for their coverage of the Tokyo Olympics
What are the key technologies used in virtual production?
Blue or green screen
Many film and TV productions utilise green or blue screen backdrops that allow CGI environments to be added in post-production. For decades, green or blue screen compositing was the go-to solution for creating fantastical environments. The reflection of hues from green screen or blue screen on costumes or props, however, is a costly and time-consuming process to correct in post-production. In virtual production, blue or green screens are used to allow real-time rendering of virtual worlds as the live action occurs in a studio, rather than being added in post-production.
Although not true virtual production, The Jungle Book (2016) was a massive step forward in testing the technologies. Neel Sethi who played Mowgli was filmed on a blue screen stage and the film was directed as if it were traditional live action, but the jungle was created in computer graphics (CG). The virtual world was combined with this live action, physical sets, motion capture and virtual effects (VFX) to create what we see on screen. Prior to shooting with actors, block animation for Mowgli and other characters was created along with a virtual environment. This allowed the filmmakers to prepare for shooting in the pre-visualised environments. Characters could then be played back in real-time allowing cast and crew to see the combination of virtual worlds and human performers on monitors (Giardina, 2017).
The Jungle Book: Creating the Virtual World and Creatures Featurette
Virtual Reality (VR)
VR is typically experienced using a headset (such as Oculus Rift or Google Cardboard) with headphones, and involves experiencing some degree of virtual realm or ‘reality’. VR technologies enable filmmakers to be able to scout a digital version of a real place or construct a virtual location to visualise a scene to plan the filming process. Doing this in VR also allows real-time collaboration across the world.
VR platforms have also been used to run virtual production systems, with trackers on cameras, props and real spaces used to provide the tracking needed to accurately combine movements and filming angles in the physical world with virtual environments.
‘The Lion King’ (2019) was shot in a 100 mile-wide virtual world which could be freely explored using VR. The world was created using traditional animation and modelling tools, such as Maya and translated into a video game engine, Unity, to create an environment explorable using headsets. This pre-visualisation allowed the director, cinematographer and camera team to understand where to shoot from. For final shooting, the crew used virtual cameras linked to tracker-enabled physical camera rigs to move it around the virtual set. The 3D path of the camera was tracked and reflected inside the virtual world (Summers, 2019).
Jon Favreau on how VR made The Lion King possible.
Major advancements in LED panel technology, developed in the live events field and combined with technologies from the aviation industry and military flight simulation systems, have enabled virtual production using large walls of LED screens called volumes. Virtual environments are projected on the screens in camera-ready quality. Their size allows the use of physical props and sets with virtual environments, and the volumes are big enough for cast and crew to operate in.
LED volumes harness volumetric capture technologies which allow tracking and synchronisation of cameras, lights and the virtual environments displayed on the LED panels. Video games engines (typically Unity or Unreal Engine) are used to control virtual sets and environments, and can be combined with sections of greenscreen for compositing other elements if VFX work is required in post-production. The majority of the VFX work with LED screens is captured in final-pixel in-camera, however, thus reducing the amount of work which needs to be completed in post-production (Unreal Engine, 2019).
‘The Mandalorian’ was shot in an LED volume and has become the poster-child for virtual production.
Jon and Nina have been exploring the geographies of virtual production and interviewing leading figures using and developing the technologies involved. A common view is that virtual production will help reduce carbon emissions in the filmmaking process, but this claim is yet unverified. Jon and Nina will lead a panel about the environmental credentials of virtual production at the BEYOND conference on 19th October.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.