How to create the most realistic virtual sets using a 360 camera array
Virtual Sets & Virtual Studios – Revolutionizing Film Production
If you’ve seen the Mandalorian and you’ve wondered why it looked SO REAL, then you have come to the right spot. Learn how the Mandalorian utilized a breakout technology – an in-camera virtual set – to bring the most life-like experience to viewers around the globe.
The Evolution of Filming and Use of Screens
To fully appreciate how far we’ve come since the invention of film and the film industry (just over a century old), let’s have a look at the evolution of film, in particular – how directors created backgrounds when they were not able to film the scene live.
Single screen behind an actor
- Remember those horribly fake looking backgrounds in EVERY driving scene in movies? The actors always moved the steering wheel in such a jerky fashion while the background stayed in the same viewpoint? Who drives like that anyway?!
- Problems with single screen: If the director wanted to move the camera angle during the scene the projected footage couldn’t move with the camera, so directors were forced to work with one, and only one, angle when filming a chase scene.
Green screens or Chroma Keying
- A literal green screen is positioned behind the actor(s). Due to the nature of the color green and the way it is captured on film, it is possible to film a blank background and later digitally remove or “key out” that color, dropping that scene onto the background of choice in post-production.
- Problem with Green Screens: actors have to imagine their environment. Just think about how difficult it must have been to film a whole fight scene from the Avengers movies in which the alien invaders were falling from the sky. The actors had to visual all of that in their heads and react accordingly. (I know action films often get a lot of heat for the less than stellar acting – but perhaps they’re better than we think if they can make us believe they are fighting armies of invaders when in fact they are surrounded by bright neon walls.)
LED screens/wall technology
- Digitally created or real-live footage is displayed on a large collection of LED screens, usually in a semi-circle formation, creating a wall and ceiling
- Benefits of LED screens: actors can see their environment on the screens in the appropriate time and react accordingly
No, that is not your eyes deceiving you – that is a virtual set. The walls and ceiling are dozens upon dozens of LED screens fitted together and displaying an incredibly clear, true-to-life environment for the actors to ‘play’ in front of.
With the touch of a few buttons, the director can change that gorgeous twilight sky into a fiercely bright midday blaze.
What is the difference between a set and a virtual set (or virtual studio)?
A virtual set is a collection of several extremely novel technologies which project a life-like environment onto LED screens located behind actors while filming a scene for a movie or TV show (these are the two biggest uses now, but theoretically any sort of film production – including commercials, news, etc will most likely benefit from this technology in the not to distant future).
A virtual studio allows for real objects and real people (actors) to seamlessly interact in real-time with computer-generated environments and objects.
What is virtual production?
The key element of a virtual set comes in the way that the real-life camera that is on set can move in 3D space, while at the exact same time, the image coming from the virtual camera is being rendered from the same perspective. The different camera settings – pan, angle, zoom, traveling – can adapt at any time to be in tune and in line with the action of the actors.
Unlike the other technologies used until now, virtual production is done in real-time and does not require post-production work.
How Virtual Sets Will Dominate Previous Filming Solution
Virtual sets are already becoming the norm in many high-end film productions due to the many benefits and very few drawbacks.
The benefits of using a virtual set
- Motion-tracking creates hyper-realistic scenes
- Using Unreal Engine (a 3D video game “engine”), artists can create a photorealistic 3D background that moves strictly with the camera’s field of view (aka: frustum), so if the camera swings around and changes angles, the background shifts in precisely the same way. This allows motion-tracked cameras to execute traditional cinematography techniques within the virtual set, achieving cinematic movements like parallax effect, (object in the foreground moves at a different speed than the background). This amplifies the illusion of filming at an actual location.
- Solves the problem of lighting
- Light coming from LEDs provide realistic colors and reflections on the actors and props (not possible with green screen)
- Removes green spill
- This refers to the green light that is reflected onto the actors and props because of all of the green screens, giving everything a greenish glow.
- Big impact on time and budgets
- Since most of the virtual set can be confined to a particular area, less space is needed overall, thereby limiting the amount of stage space necessary in filming/shooting
- Developing and building the LED screens is definitely expensive the first time, but the initial investment will soon make up for it by the savings on traveling to locations, building new sets, and costly post-production
- Customization of the features
- Artists and directors can make changes and control the world on the day of shooting
- Exposure, color, animation playback, and fill lighting are available
- Can move a mountain with the press of a finger
- Still able to use green screens within an LED wall
- It is still possible to use a specific portion of the LED and transform into a green screen.
- Ex: Can remove people and add additional action behind them
- This is due in part to the fact that some special effects are still best kept at traditional methods (prevents damage on the LED screens)
- It is still possible to use a specific portion of the LED and transform into a green screen.
- Speed of setup
- Rather than traveling to different locations or setting up in-studio sets with real objects, setting up a virtual set is much quicker. Also, when working within a normal set, changing the lighting can add significant time to each shot as cranes and lights have to be moved around and adjusted often.
How to create virtual sets
Let’s look at the technologies as well as the steps and the people needed to create a virtual set.
Technologies necessary for creating a virtual set
There exist many technical solutions for creating virtual sets and studios, but most of them include the following components:
Optical or mechanical measurements are used in camera tracking to create a live stream of data describing the exact perspective of the camera.
Real time rendering software uses the camera tracking data and generates a synthetic image of a television studio.
A video mixer combines the video from the camera with the video from the real time rendering software to produce a final video output. Replacing a chroma key background is one of the most common ways to mix the video.
Steps to creating and using virtual sets
While the actual steps and processes involved are much more complicated then what can be covered here, we will focus on why this affects Mosaic. For a more thorough understanding of the ins and outs of creating a virtual set, check out this incredible article here. We will stick to what we know – data capture – and why we’re so excited for the applications of our Mosaic cameras and mobile mapping services for something as exciting and innovative as in-camera VFX on virtual sets.
Hopefully at this point we all understand that a virtual set is basically the way in which actors can film in front of a flat backdrop but feel as if they are in an actual, real scene. The imagery which is displayed on those LED screens has to come from somewhere though, right?
Who is behind creating virtual sets?
There are several methods of achieving the images to be displayed on virtual set LED screens.
- Option A: 3D graphic artists and 3D computer graphics software
- Option B: capture real-life images (and generate complete 3D models of large scale scenes) with the highest resolution camera array on the market today.
This is where we get involved and why we’re so amped up! The images portrayed on these screens have for now been created through the hard work of 3D graphic artists and 3D computer graphics software who create virtual backgrounds and all of the bells and whistles that go along with making a scene look real.
The Mandalorian in particular used images and environments sourced from Unreal Engine – which is basically a library for super advanced, super real-time 3D creations. Up until now, this technology has been used mostly in the gaming industry for those almost-too realistic first player shooter games.
But Mosaic cameras, with the highest resolution camera array in the world, are capable of collecting 3D environments at a much quicker and efficient manner than 3D graphic artists can create them.
With image capture abilities from a camera array which is expertly synchronized to collect a full 360-degree view of the surrounding area, film production has a brand new tool at their disposal. They can effortlessly gather real-life data in almost no time at all, thereby reducing time and resources significantly.
Check out some of these videos captured by the Mosaic Viking – the world’s highest resolution camera.
The future of film production
Certainly, 3D artists, VFX specialists and complementary software will be necessary when creating the content for films or shows which are set in unreal places. For situations such as those, the virtual environments will need to be carefully crafted to meet the demands of the directors, producers and writers.
However, for scenes which are set in real-life places – downtown Hong Kong, the middle of Santiago Bernabéu Stadium (Real Madrid), or the depths of the Amazon forest – there will be a much faster, easier and cost-efficient manner in which the image data is collected. If you are interested in more information about the Mosaic line of cameras or our data collection services, feel free to drop us a line below.
