Direct from the desk of the CEO: Epic Games acquires RealityCapture – This is HUGE!
HUGE NEWS in the 3D world! Capturing Reality, the Bratislava-based company behind RealityCapture, a leading photogrammetry tool, has been acquired by Epic Games, one of the world’s largest video game companies, and makers of Unreal Engine, one of the leading 3D engines used in video games today.
What does this mean, and what’s going to happen? That’s always the question. Who are these companies, and what is this software all about, anyway? Let’s dive in a little bit.
What is Photogrammetry Software?
Before we step into the worlds of RealityCapture and Epic Games, let’s take one step back and look at the software involved to get a better understanding of what we’re in for.
There are a few competing solutions in the market for photogrammetry software. But before we get ahead of ourselves, what is photogrammetry software, anyway? Basically, it is software that allows you to take a bunch of photos (and/or laser scans) of an object or place, and process those photos into an actual 3D shape: either a dense point cloud, or a solid mesh made of triangles, colored (textured) by the original photos.
As you might guess, this is a pretty computationally-intensive process, and in terms of input data, the sky’s the limit: You might use thousands or tens of thousands of source photos to create, say, an aerial 3d model of a city. Over the years, photogrammetry has gone from a niche affair requiring a huge workstation to make even a humble model from thousands of photos, to a fairly fast process, allowing a regular person with a gaming-grade GPU (more on that later) to process a thousand photos into an extremely high-poly (lots of polygons), very detailed, 3D file of an object or environment.
There are quite a few options for using photogrammetry software, if you want to get started yourself:
Meshroom / Alicevision
Meshroom is an open source, free project that is fully functional and very powerful. This was the first 3d software that I used, and without knowing what I was doing, I dropped 120 photos into it, clicked “start” and out came a nicely textured 3d mesh of a sandstone sculpture.
Meshroom is built and maintained principally by a very talented team in France at MIKROS, including Benoit Manujean, Fabien Castan, among others. The main complaint about Meshroom is that it is slower than other solutions. It is not optimized for a single-node workstation the way some of the other software is.
It is however, completely free, so instead of complaining, you can, of course, fix it yourself. I have high hopes for this product that it will continue to evolve and improve. Just recently Meshroom announced support for spherical panoramas, for example. So it is in active development and it is getting better all the time.
Agisoft Metashape
Formerly known as Agisoft Photoscan, Agisoft Metashape is a very popular photogrammetry software that has been around for over ten years. It is one of the few photogrammetry products that has a Mac version. Additionally, the price of the software is quite reasonable at $179 for the standard version and $3499 for the pro version (both offering unlimited numbers of images as input)
3DF Zephyr
3DF Zephyr is also quite a popular photogrammetry software. It has a free version, limited to 50 photos input (which can be enough in some cases to create some simple 3d models of objects or rooms) with the licenses costing anywhere from $149 to $3900 (similar to Metashape). 3DF Zephyr works well and there are lots of people who enjoy using it.
RealityCapture
RealityCapture, made by a company called Capturing Reality (yes, a bit weird) has been available since around 2015. RC is built by a small team of mostly Slovak engineers led by Michal Jancosek, Tomáš Bujnak, and Martin Bujnak, who studied at CVUT (the Czech Technical University), one of the world’s leading institutions for computer vision and related fields. This team wrote many well-known papers at that time, with their professor Tomáš Pajdla, as well as Zuzana Kukelova, an esteemed researcher and now also an assistant Professor at CVUT, who helped to write some of the underlying solvers which form a basis for the RealityCapture software.
Martin Bujnak, one of the cofounders of Capturing Reality, is clearly an accomplished and respected researcher in this field. As you can see, Bujnak, along with Kukelova and Pajdla, have published a number of papers, spanning many years, on the topics of 3d reconstruction, algebraic geometry, pose estimation, lens correction, and other topics, all relating in some way to photogrammetry / 3D reconstruction.
RealityCapture has been a bit different from the other products in a couple of ways. First of all, it has typically been faster than the others – much faster, in some cases. Compared to Meshroom for example, RealityCapture might be over 100x faster if you have a large dataset of thousands of photos (the difference might be counted in days or weeks!).
While Agisoft Metashape has caught up a lot in terms of processing time in its most recent version, RealityCapture is still faster, especially when it comes to large datasets. While the final result is slightly different for all of these products, and the character of the 3D mesh is somewhat different, the actual result of RealityCapture is very, very good (the experts might argue about which software offers better quality in high frequency vs. mid frequency details, but in the end this doesn’t matter too much for most people). And if you are doing this stuff for your job, where time is money, well, it makes sense to use the fastest product and spend less time waiting for your computer to finish rendering.
The Capturing Reality team couldn’t resist capturing this car covered in snow in Bratislava. See the original here.
The second way RealityCapture has been different is in its pricing. It has had weird pricing:
- It has a “PPI” (pay per input) version where you pay for each model, according to how many pixels there are in the input data. So if you have 50 x 50 megapixel images as input, you pay a certain amount, but if you crop those photos in just the right way, eliminating pixels which are not needed in your model, you might pay a lot less.
- Then there is a $40/month version which lets you use up to 1000 images per project, but those images have to be 24 megapixels or less.
- Or you can pay $750 monthly for unlimited usage (that’s a lot!)
- or $15,000 per year (holy smokes!).
So unless you’re a professional user, you might be stuck with the $40/month version which is actually very limiting – 1000 photos only gets you so far – you can’t do truly large areas or objects with that many photos.
Some of the reviews of RealityCapture on Steam give some more detail:
“The undeniable advantage RC holds over all other [Photogrammetry] software is it’s speed, there’s really no question that this software is a lot faster at practically every step of the process than others. For material scans it does offer one advantage also, the orthophoto mode allows exporting of a height and color map with no baking, this makes it possible to output extremely high detail height maps from 1M+ polys that would otherwise be impossible on low to mid range PCs.”
“I compared this product to Agisoft Photoscan and calculation speed of RealityCapture is unbelievable. RealityCapture can align standard 100-200 photos datasets in minutes and in Photoscan can take it hours. For me, this is top product on photogrammetry market. When i am doing simple statue, i can have preview on the spot with notebook with incredible speed. Inspection tool for analyze quality of my shooting is very very helpful. RealityCapture can handle huge datasets i never seen align/matching of thousands of photos calculated so quick on my desktop. And quality of high-resolution outputs, like textures and meshes is unbelievable. And in-app simplifying ability of high-res meshes is excellent. I can not believe what I see and i can do in this piece of software.”
But also, the rather strange pricing / licensing model obviously rubbed a lot of people the wrong way:
“An example of a loss of business acumen and, potentially, a leadership position, or maybe just greed. Best software, some time before. Now is buggy DRM and have hard limits in new license. Also, still no new features, no user-friendly interface, no documentation and support.”
“Excellent photogrammetry software, trapped within one of the most consumer-hostile business models I’ve ever seen outside of an MMO.”
“Let me start by saying that RealityCapture is the best photogrammetry software on the market. Full stop. I’ve tried plenty of other solutions (Zephyr, Metashape, COLMAP, meshroom), and none of them match the quality that RealityCapture can produce. However, that praise is overshadowed by how actively hostile their business model is to subscribers.”
You’ve made a 3D model, now what?
You can use one of the above software products to create a 3D model. It’s actually pretty easy. Take a bunch of photos of an object from all angles, drop them in your software of choice (it doesn’t matter which one – they all work fine), click a few buttons, and out comes an amazing, detailed 3-dimensional thing – it really is like magic. But then what? What do people do with this 3D data? Lots of stuff, such as:
- 3D printing
You can make a 3d print of that thing you scanned. Whether it’s your child’s face, or some statue scaled down so that it fits on your table, or a 1:1 reproduction of a car component that needs to be reverse-engineered – there are a range of reasons why people make 3d prints of 3d scans, from the hobbyist to the engineer. Here is a very popular video from Prusa, the 3D Printer company, showing how you can shoot photos of a statue, make a 3d model, and then print it.
Who wouldn’t want this printed and placed on their front stoop? See the original by Capturing Reality here.
- Video Games
Obviously, lots of video games take place in 3d worlds. Grand Theft Auto, for example, is an entire metropolitan area full of streets, houses, mountains, and everything in between. It doesn’t quite look like real life yet, but it’s getting closer and closer.
Imagine this in Grand Theft Auto. See the original 3D model here. Image credit: Jeffrey Martin
- Movies / VFX
More and more often, what you see in a movie never existed in real life, and the actors may or may not even see the virtual objects they’re sharing the frame with on their virtual sets. In fact, the production of video games and movies are starting to merge into something very similar (more on this later). VFX studios are also huge users of photogrammetry to gather environmental data and accurate recovery of the camera track.
Check out this video too. The owner won’t allow it to be played on other sites, but it’s worth 19 minutes if you’ve got it.
- Visualization
A factory, architectural site, crime scene, construction site… the list is growing longer every year: all of these places benefit from having a 3D scan which can be viewed in extreme detail on a computer screen rather than visiting it in person. 3D scanning and visualization helps crimes get solved faster and buildings get built within budget. - Street and Aerial mapping
Not only Google Maps, Apple Maps, and most other mapping companies are driving and flying all over the world to make a 3d model of the whole planet. Hundreds of smaller companies are doing this as well, for the purposes of straight mapping as well as infrastructure inspection, mobile mapping surveying, and so on.
Here’s the thing though: in all of these cases, displaying a 3D scan is not easy. This part of the toolchain is still difficult. It requires training and knowledge and practice. It’s possible to make a 3d model without really knowing anything about how the process works, but the result will be a model that’s made of 50 million triangles and 10 gigabytes in size, which is too complex to show to anyone.
This model needs to be simplified down to maybe 100,000 triangles or maybe even 10,000 triangles, which are proper UV maps and Normal Maps. This process is hard and there are university courses devoted to it for people who strive to work in the VFX or video game industry.
Shakeup in the Photogrammetry Industry
I’ve thought for a while that some company might acquire one of these 3D Photogrammetry software companies, because, well, that’s what happens eventually when a small company gets really, really good at what it does, and it starts to become relevant to the larger industry it’s a part of. And so it has happened for RealityCapture. Most likely everyone who uses RealityCapture is breathing a sigh of relief that they were not acquired by Autodesk, which famously buys companies who are potential competitors, and then kills those products. And as it turns out, the acquirer, Epic Games, is very interesting indeed.
Who is Epic Games?
Epic Games, along with its larger competitors EA, Valve, Sony, and others, is among the ten biggest video game companies in the world. Their most famous product is Fortnite, which has helped propel them into this top ten list. What is more interesting is the 3D engine that Epic Games makes, which is the software that allows them to build these games which take place in 3d worlds. This is called Unreal Engine.
What is Unreal Engine?
When you make a video game that has humans, buildings, land, water, flying objects, gravity, and interactions between all of these things, it is necessary to build software which can show these things in 3D, which can figure out how these objects are lit, and even has some knowledge of physics, so that objects can interact with each other in a plausible way. This is generally called a 3d engine. Unreal Engine is one of many – there are lots of other ones available (another very popular one is called Unity, for example). Unreal Engine is the most popular and widely used 3d engine, and it is technically very impressive. The next version of Unreal Engine, UE5, is pretty amazing. Here, watch this:
And maybe you can watch this also.
If you don’t watch the second video, just the title is enough: “Unreal Engine 5 and reality – can you spot the difference?”
Unreal Engine is not only used to making video games. It’s also becoming a valuable tool for VFX and filmmaking. A great example of this is The Mandalorian, and how it was made. Most of The Mandalorian was shot not on location but rather on a stage, in front of a huge LED screen. Using cameras with motion-control, the picture on the LED screen, containing a 3D world built in Unreal Engine, could move with the camera to create a seamless background.
If that doesn’t fully make sense, then watch this fascinating video.
So to answer the question of “what is Unreal Engine” – It is the software behind creating a simulation that can’t be distinguished from reality for the video game and movie industries, and more and more other industries as time goes on.
Who else has Epic Games acquired, lately?
Epic Games acquired Quixel at the end of 2019. Quixel builds a library called “Metascans” which is a huge library of photorealistic 3d photogrammetry scans. When Epic Games acquired Quixel, they made this whole library free to use for everyone using Unreal Engine.
Are you connecting the dots yet?
Photogrammetry + Unreal Engine
So what’s next in the video game world? What’s the next Portal, or Half life, or GTA going to look like? How real will it look?
It’s clear that the people who make video games would like it to be easier to build a 3d world, using actual real world objects, without modelling them by hand. And in fact, for ten years now, people have been scanning real world objects, and many of them are simplified and imported into Unreal Engine or other 3D engines to be used inside one of your favorite video games. But this process has been convoluted at best.
We are moving into a time where it will soon be possible to scan a room, or even a town, and then click “publish” to send it to Unreal Engine, where it will undergo sophisticated processing – simplification in a way that can’t be detected by human eyes – something that is 1/1000 of the file size, but still looks good – and can be used as the basis of a video game. In many cases, there will no longer need to be the teams of 3D modelers, painstakingly sculpting the components of a whole world.
But what will actually be the result of this acquisition?
The four fates of a Big company + Small company acquisition
In the context of “Huge company A acquiring tiny company B”, there are four general scenarios:
Scenario 1: Same company, different owner
In Scenario 1, nothing much will change. Small Company continues doing what it has always been doing, and makes money for Big Company.
Scenario 2: extermination of a potential threat
In Scenario 2, Small Company, posing a long-term existential threat to Big Company, will be slowly suffocated and eventually killed off. Its product will no longer be available, and there will be no replacement. The engineers will be subsumed into the guts of the larger company, never to be seen again.
See: most acquisitions by Autodesk
Scenario 3: Acquihire
The Product made by Small Company, while interesting, was never that interesting anyway, but the people who built it, being obviously very talented people, are the valuable part. The product, while exciting, is not nearly as exciting as having these engineers work on some other big project which Big Company is working on. The product will continue to exist, but slowly become neglected and eventually abandoned as the key engineers are pulled into Bigger Projects within the company.
One example of Scenario 3 is Apple’s acquisition of Beats. Here’s a video about that. TLDW: Beats was dismantled for its engineers and software product; the software was turned into the Apple Music streaming service and the hardware / audio engineers build Apple’s earpods and other audio equipment. Beats is still there but it’s a walking corpse.)
Another example is Gopro’s acquisition of Kolor, which allowed Gopro to build a 360º camera product, but ultimately killed the much-loved software products made by Kolor, affecting the 360º photography industry as a whole
Scenario 4: the Sum is more than the parts
In Scenario 4, Small Company, while not a present or future competitor, is strategically relevant to the larger company in the grand scheme of things, and the technology of the Small Company will help the Big Company continue whatever vertical integration it has been planning, for increased domination of its industry. The product from the Small Company will continue to evolve but with deeper integration into the main products of Big Company.
What is the fate of RealityCapture?
Scenario #1 is unlikely
Epic Games does not care about the relatively tiny revenue of RealityCapture – In 2019, Epic Games reported $4.2 billion in revenue and $730 million in earnings. While RC is an industry leader in photogrammetry software, their earnings are probably in the low tens of millions of dollars, which is a rounding error for a company like Epic Games.
Scenario #2 is also quite unlikely…
If the acquiring company was Autodesk, we would definitely have a #2 scenario, because that seems to be what Autodesk does. Thankfully for many people who use this software on a daily basis for their jobs, this is probably not the case.
That leaves us with Scenario #3 and #4.
In scenario #3, RC slowly dies or slides into irrelevance as its key talent is pulled into other, tangentially-related projects in Epic Games. In scenario #4, RC is given a blank check to improve its product, and is given a team of engineers from Epic Games to work on bridging the gap between what RC does, and what Unreal Engine does.
My money is on scenario #4.
Without having any inside knowledge of either company, or knowing any of the people involved, I would venture to say that the people involved in RealityCapture are not in it just for the money (of course they like money, and of course when someone offers you a great deal, you take it). I don’t expect that the cofounders of Capturing Reality, who are esteemed scholars and engineers themselves, would be interested in the idea of shutting down their product, which is currently an industry standard, in favor of “working for the man”. Of course, it is possible that they agreed to terms in this deal which no longer allows them the ability to control this – however, I find this idea unlikely.
So, the end game that I, and others in the industry are hoping for, is that Capturing Reality is now free from the mundane day-to-day stress of running a business and making money, now being owned by infinitely wealthy overlords who have much bigger fish to fry: the RealityCapture product, which is of extreme relevance to the core business of Epic Games, especially when it’s used by normal people who might ultimately create 3D assets that are useful for video games, will continue to evolve and thrive. Eventually the RealityCapture software will be free to use for hobbyists, who now have a more direct path to making money from their 3d scans than ever before (shoot, scan, publish on Quixel, profit!)
Conclusion: Unreal Engine + Quixel Metascans + RealityCapture = the Ultimate Simulation
Looking at all the pieces, it seems like an obvious outcome. Unreal Engine 5 is so good, that it’s no longer possible to sculpt 3d models by hand – they have to be scanned, there is no other way around it. How do we scan them? We need maximum quality, time is money; RealityCapture is most likely already being used extensively by Epic Games to create assets to put inside Unreal Engine. People will always need new assets to put into new games, so it makes sense to acquire Quixel, and to make all of those models free to use (Epic Games charges a 5% royalty when you use Unreal Engine, so they’ll make the money back eventually). For the next 3d models of objects, rooms, places, and even entire cities, it makes sense to own RealityCapture, and to integrate it as seamlessly as possible with Unreal Engine – the easier it is to create 3D worlds in Unreal Engine, the better the video games (and films, and other stuff) will be.
Special thanks to Alexandre Jenny and Arie Kai-Browne for providing their thoughts and feedback while I was writing this article; thanks to Lynn Puzzo for editing.
