XR Spotlight – Using Virtual Reality to tell stories

In the lead up to this year’s Aesthetica Short Film Festival and our role running the VR Lab, we revisit a roundtable discussion between some of the creatives whose Virtual Reality (VR) experiences featured at last year’s festival.

Moderated by Eric Fanghanel, lecturer in Immersive Media and Mixed Reality at University of the Arts London, the discussion explored different perspectives on using VR to tell stories. It encompassed topics including thoughts and hopes on where the VR industry is going and advice to artists looking to work in VR.

Participants: Lee Harris, Locker Room; Bambou Kenneth, Kindred; Alex Rühl, Rock, Paper, Scissors; Austin Wolfe, The Journey.

Eric Fanghanel: Can you start by introducing yourselves and your projects?

Lee Harris: I’m the co-founder of Electric Skies, an immersive content production company. I was a producer on Glimpse, which won the award at Aesthetica in 2022, and executive producer on Kindred, which Bambou directed. Locker Room, which is a VR experience where you are Muhammad Ali in the lead up to ‘The Rumble in the Jungle’ is my first project as a director and writer.

Bambou Kenneth: My introduction to VR was as an art director on Glimpse, produced by Mr. Kite. I come from an art background, then made my way through the VR world and fell in love.

Kindred, which I directed, is a short animation, based on the true story of the first transgender non-binary parent in the UK to adopt a child. It was made in communication with the real people, but uses fantasy elements to communicate its themes.

Alex Rühl: I have a background in filmmaking, specifically television. My first VR piece, Keyed Alike, was a 360° rom-com. I worked with visual artist Ben Fredericks on a piece I exec-produced called Playing God, a thought experiment around the refugee crisis.

My latest piece, Rock, Paper, Scissors, which I wrote and directed, is the first BFI Network funded immersive piece. It’s a coming-of-age story between a mother and daughter, wrestling with the notion of what is fair. It uses hand-tracking mechanics to play rock, paper, scissors to drive the narrative.

Austin Wolfe: The Journey was my first VR project. I made it jumping off of my PhD on creating narratively engaging stories in VR. It’s a story about a whale, driven purely by music, emotion, and the physical motion of what’s happening around you.

What specifically inspired you to work in virtual reality?

AR: The first time I experienced VR was a lightning bolt moment. I remember watching the Mr. Robot VR experience. Bringing this TV show into VR, and the way that you tell that story totally changing, really blew my mind.

LH: My inspiration goes back a long way, and at the time it wasn’t an inspiration to become involved in VR specifically. In about 2010, I went to visit the ruins at Pompeii. One building was called the House of the Faun — the audio commentary explained that a faun statue was found here, which could now be seen in the Archaeological Museum in Naples. I was disappointed by that, because I wanted to see the house how it really was in those times. Once VR emerged, that thought and this new technology fitted together somewhere in my brain.

BK: I wasn’t initially interested in VR, but I lived with Michael and Ben of Mr. Kite, the directors and writers of Glimpse. I was seeing first-hand how you craft VR. I worked a bit in film productions and had curated art exhibitions, but seeing the creative freedom they had and how they invented their own pipelines was so inspiring. It opened other ways of telling stories. As a creator you get to invent new ways every time you do something.

At this point, Austin dropped out of the conversation due to technical difficulties. A one-on-one interview with Austin has since been published on Medium.

What does it take for a piece to be well-executed in VR? What’s your process of deciding whether something is a good fit for these new technologies?

AR: There are two defining things that make an experience work in this format. Being in a place that you would never otherwise get to experience, or stories where you’re inside the heads of those characters.

LH: That intimacy is what makes VR special. It’s something you can achieve in other mediums, but not to the same extent.

BK: There’s the term ‘narrative transportation’ — when the individual is so completely immersed within a story that they forget about real life. VR has the power to do that, probably more than other mediums. When I’m thinking about projects in VR, I always think about who doesn’t have a voice in our daily life that should have. VR is the place for them to get a voice

LH: If you can make it as a documentary or a short or a feature film or flat animation – do it that way if that’s the best way to get the story across. But if you can add another dimension or get another viewpoint across that you wouldn’t be able to in most traditional methods, then that’s the case for making something in VR.

Are you married to VR or XR now, for your next projects?

BK: There’s something communal about creating VR. You go to festivals and you meet the same people. It’s really small, but it is a bit of a family. So it is maybe closest to my heart when I think about future projects.

LH: With this medium, we’re all getting to make the rules and figure out how things are done and I find that really exciting. The film industry has been around for over a hundred years, and of course technology moves things on, but a lot of how you make a movie is pretty set. With VR, there’s an opportunity to set the benchmark. Not necessarily set standards, because I think one of the great things about working in this medium is that there aren’t any. There’s that opportunity to keep reinventing with each project that you work on. There’s new conundrums to figure out every time and that excites me.

AR: I agree with what you’ve both said. It is about the art form, but it also is about the industry, it’s about the people. You’re surrounded by people that you’ve grown up in this industry with and talking about what’s going well, what’s not. It’s beautiful to be part of that. 

In Apple’s keynote for the Vision Pro, they highlighted that they believe this is going to be a whole new kind of storytelling medium – which we’ve known for years! Thanks Apple, for telling the world what we already knew, but that is really validating. Because we’re going to get new people coming in and saying “okay, I’ll try and tell a story in that way”. Then you’re going to see even more experimental and innovative stuff, which will shape and morph the medium. It’s a really exciting time to be a part of it. I highly recommend to anyone that’s not in it right now to give it a go.

LH: I really love that community aspect. Everyone is so supportive. I tried for many years to get a job in film and it’s hard. It’s so well-established and it’s who you know or who are friends of your parents or whatever. The XR industry is so welcoming to new storytellers and practitioners and creatives and developers across all sectors of the industry. That’s great from a people perspective but also in terms of moving the industry forward. People who perhaps didn’t have a chance in more traditional mediums to tell the stories they wanted to tell — they now can.

Are you thinking about the possibilities that new headsets – the Vision Pro, or the Meta Quest 3 with its Passthrough – allow for mixed reality experiences?

AR: What’s exciting is that if you’re creating a piece, you don’t have to lock it down to a specific ecosystem. 

VR stories are different to Mixed Reality (MR) or Augmented Reality (AR), they have a totally different impact. You can do shared VR, but really VR is about being fully immersed in the story. Some of the popular stuff in MR feels more fast-paced. It’s about collaboration, and a bit more interactive. So I don’t think it impacts how I would tell a story because if I was gonna tell a VR story, it would still be a VR story. What it does mean is that if I create something, I can create it for the same device that MR creators are creating for now. Now that you can build really good MR experiences for the Quest 3, and I can build really good VR experiences for the Quest 3, an audience member has the choice. We don’t need lots of devices to experience these different stories.

LH: I was a VR purist, but I’ve changed my outlook. I produced one of the release title games for the Quest 3, which opened my eyes a little bit. 

One of the issues with VR is getting audiences into it. MR is actually a really good introduction. You put the headset on and you’re completely closed off to the outside world immediately. Whereas with MR, you put the headset on and you’re still in the space that you were in a second before. You can then gradually introduce the VR elements before fully immersing them.

That’s something I’m really looking at: how do we create that transition? Still creating a full VR experience, but having almost an onboarding segment in the headset itself. 

BK: VR is not very accessible — there’s a small userbase, and it’s a bit privileged. The idea of having MR as another step on the way is a beautiful way of looking at it. MR is amazing in itself as well, but if we can reach more people through that, then that’s a great opportunity. 

One thing that struck me about all of your pieces was the diversity of styles. There’s such a wide vocabulary of stylistic decisions.

BK: Kindred is a very human story. It’s about empathy, it’s about family, it’s very intimate. At the same time it has this layer of politics, identity politics and social constructs and things that are heavier and harder for people to deal with. In order to strip down those layers it needed a bit of distance from reality. I thought stripping down our conceptions and ideas about reality, about gender, about whatever, is maybe the best way of getting into people’s hearts, before you get into their sense of what’s right and what’s wrong. 

The language, the colours and the shapes are very fantastic. It was important for me to have the characters unidentifiable as a certain race, age, gender. It’s something beyond that and that’s part of the fun with animation. You can create something that is a bit more blurry.

LH: Every project is unique and there are multiple things that go into those creative choices. Unfortunately, with my producer hat on, lots of budgetary and schedule decisions go into those things as well. One of our first projects used some motion capture onto models. If we’d had the time and money I probably would have done that as a pure animation, because we’d have achieved better-looking results from a creative point of view.

On Locker Room, one of the decisions stylistically was based around the context in which it was being exhibited. This is part of a larger live immersive experience called ‘Rumble in the Jungle’. It’s essentially secret cinema for a sporting event. There’s live actors walking around, there’s sets built to show street scenes in Kinshasa, the ring, or the stadium. They made a physical twin of the locker room from the VR experience, where people could watch the VR experience within the larger show. That meant we had to create a more realistic style within the VR piece too, so that it all fit together. We used archive material and 3D scans of real-life artefacts — a pair of Ali’s actual gloves were scanned, and you wear them in the experience.

AR: It’s artistic, what’s right for the story, but you have to be realistic. Our industry doesn’t have as much funding as others, and it costs a lot more to do everything. Any stylistic choice has repercussions. 

Rock, Paper, Scissors was the first piece that I properly made in animation, and I use the word animation very lightly because it’s actually a lot of still artwork. Part of that was for budget reasons, but also I was fascinated by this idea of — like Bambou was saying — wanting distance from the kind of memories that you’re going through in this piece. I don’t want to give you too much visually, and that’s why we use a lot of colour to evoke certain emotions or give textures to the underlying theme of the scene but without making it too realistic. It’s meant to evoke that sense of ‘oh, I’ve been in this situation before’ and you conjure up your own memories and project them onto it — like you would if you were reading a book. You’re creating your own visuals.

Audiences don’t really think about the style. They talk about the story and the themes and “Me and my dad used to do this when I grew up”. Creators on the other hand are like “I loved it, but why did you do it in Quill?!” For me the choice to do it in artistic animation is that you can walk around the scene, you can have more freedom with the interactivity. Whereas if you want six degrees of freedom and interactivity in a photorealistic piece, your budget goes up by ten, because volumetric capture is really expensive. Where we are right now is a bit of a trade-off. Do you want the photorealism, but reduced interactivity or ability to roam, or do you want the flexibility to do that but go more computer-generated?

Sometimes you think, all these stylistic choices that I’ve made led the piece to where it is now, so it’s only a good thing. But it is funny that there’s other versions that live in your head. I’ve always struggled with that as a director. I’m indecisive in my life so to have a team look to you and say “what’s your creative vision?”, and the truth is you’re making it up with what you’ve got available.

Are there any Artificial Intelligence (AI) tools or other tools that you’re excited about? Is anyone using AI in any way?

LH: The portrait of President Mobutu in Locker Room is a Midjourney creation. Mainly because we couldn’t licence any images of him. I couldn’t draw it, so I got Midjourney to do it for me.

AR: I’m obsessed with using it as art therapy. Trying different styles and prompts. It’s useful for storyboarding. I was talking to someone this morning about the fact that there’s this really cool tool, Skybox, where you can very quickly prompt 360° images, put it in a headset and see what it feels like. I cannot wait until generative AI is good enough to create 3D models. That’s gonna be a game changer for our industry.

LH: My opinion is that it’s a great basis for a lot of the work, whether that’s concept art or roughly modelling stuff out or creating portraits of President Mobutu. As someone who’s not technical — I can’t create 3D models or spaces, I can’t make things work in a certain way in Unreal or Unity — being able to use those tools to translate what’s in your head to the people who can do all of that amazing stuff is really useful.

I don’t think it’s gonna take over completely. I think all the skills and talents of the people that work in this industry and others will still be needed. But it will cut down on that back-and-forth of “not like this, more like this”. So we’ll speed up processes and make things more cost-efficient. You still need all the storytellers, all the artists, all the technical people to make it work in the right way. But for improving workflows and making things more efficient, I think it’s great

BK: As someone who comes from art I have a kind of love/fear relationship with AI. But as a communication tool — like you were describing, Lee — it’s pretty cool. I still have those dystopian futures in my mind, but I’m just limiting it to what it’s good at at this moment, and that is getting ideas across.

Do you have anything to say to someone that’s thinking about either submitting to a festival or starting to work on VR?

BK: VR can be overwhelming. They tell architects that the real job is working with limitation, and it’s similar in VR. The financial limitations, technological limitations, creative limitations. Make your limitations your tools in order to create a better piece.

LH: Making something is the best way to learn. It’s a bit of a flippant comment because you need money or the tools to do it, but there’s ways around those things. As we’ve said, this is a very open and welcoming community and industry. If you’ve got a great idea that you think is going to make a really interesting VR piece then try to speak to people like us. Even if they can’t help directly, they might be able to point you on the right path.

AR: I would also say if you’re thinking about getting into this, you’d best have experienced a lot of it for yourself. Know what you like and don’t like, what styles work and don’t work, what resonates with you and what you’re trying to achieve. There is an accessibility thing there — not everyone is going to have a headset in their house. But there’s plenty of local independent arts venues that have this kit. There’s probably someone in the XR creator network that lives in your area that would be happy to meet up and let you borrow the headset or show you some of their work. Going to festivals is super important.

Locker Room, Kindred, The Journey, and Rock, Paper, Scissors were all available to experience as part of the 2023 Aesthetica Short Film Festival.

Additional information

Visit the Aesthetica Short Film Festival website to find out more about this year’s festival.