At the F8 developer’s conference this week, Facebook made multiple announcements around VR and 3D. And this year they’re also starting to focus on 3D capture which means they finally have my attention.
In my previous post I wrote about 3D Photos, Facebook’s upcoming new photo format that adds parallax depth effects to photos that you can take with a regular, 2D smartphone camera. Although it’s not a new concept (Seene did it in 2013 before being acquired by Snapchat) it’s certainly a nice feature that adds interaction an immersion to photos. And since Seene no longer exists it’s nice to have the feature back.
The other 3D capture related feature Facebook announced this week is called 3D Memories. This feature turns multiple regular photos and videos of an event into a interactive 3D world that you can experience in VR. Before I dive into this, let’s take a look at the announcement.
Creating 3D models of rooms from 2D photos or video isn’t new, of course. It’s called photogrammetry (or structure from motion). I actually have written a tutorial about how you can turn your smartphone videos into 3D models for free. But then you’re doing it on purpose. And if you’d like to capture a room, you can’t have any moving people in it because photogrammetry algorithms hate motion.
So it looks like Facebook has created some kind of AI algorithms that will remove people from a room, or even multiple, interlinked rooms, and then recreates environment in 3D. That’s quite impressive.
The result is indeed 3D. But it’s still a point cloud. That’s basically a collection of points in 3D space (colored in this case) that’s the raw output of almost every 3D scanner or photogrammetry application. When 3D capturing objects and people, this point cloud is then turned into a 3D mesh, which is an actual 3D surface. And it’s then textured with the color information to look as realistic as possible.
It’s pretty hard to turn 3D captures of environments into 3D meshes. It’s usually too much data to process and often not complete enough to fill all holes realistically. And for most 3D scanning use cases that capture large areas such as interiors, exteriors or entire outdoor areas, point clouds are a good enough representation of reality. These purposes are usually about measurements for building and engineering purposes and you can perform measurements just as well on points than on polygons.
But for consumer entertainment, I don’t think point clouds are immersive enough. Below is an exampje of a point cloud of a library interior I found on Sketchfab.
It’s fun to virtually navigate the shelves in the little window I’ve embedded above. It also loads pretty fast and is doesn’t require a lot of 3D graphics resources making it very suitable for mobile devices. But it already looks like a 3D version of 8-bit pixel art. Now hit the little VR icon in the bottom right corner and put on your VR headset to view the scene.
Oh, wait, you probably don’t have one…
If you do have one, you’ll find yourself surrounded by colorful balls. It’s fun but mainly because it’s new. But it’s far from realistic. And this is a point cloud that has been captured for this specific purpose. Not based on a few random photos of the library of which most contain people in the foreground. I’m against point clouds for use in real time 3D experiences. But they need to be a lot denser than the ones in the video to be immersive in my opinion.
Yes, I’m skeptical about Facebook’s 3D Memories idea. Not only because I doubt that the average consumer has enough photos or videos that cover a large area but also that the result will be far less complete than the one in the demo. Such detail is only possible if someone has done a video walk through of a house. And even in that case, I very much doubt that consumers are going to experience their “memories” in this way.
I totally understand that Facebook has to experiment with VR. They acquired Oculus (some say for $3 billion) and just released the Oculus Go standalone VR headset. But this new headset only tracks rotational movement of the head — it won’t let you move around freely like the Rift of HTC Vive. And if you can only look around, a 360-degrees photo of an environment looks a lot more pleasing than a point clouds.
I’ll give my final judgement once this feature is released. Although I don’t have a Facebook profile anymore to test it myself. I do still have a Facebook Page though with a lot of 3D memories.
Edit: This page was updated after publishing to correct a statement that the Oculus Go features positional tracking while it only has head tracking.
“The new “killer feature” of this headset is inside-out tracking which allows users to walk around instead of just look around like other mobile VR headsets are limited to”.
The Oculus Go doesn’t feature inside-out tracking (only 3-DOF) . It’s a simple GearVR without the phone part. Windows MR HMDs, Vive Focus and the Google Daydream Lenovo Mirage are the only headsets with 6-DoF inside-tracking.
You’re right. I think I was hoping too much ?
Corrected the relevant paragraph.
You’re right. I think I was hoping too much 🙂
Corrected the relevant paragraph.
This combined with the Oculus Quest is a killer app in construction, engineering, and real estate. Right now a dense and accurate-enough point cloud takes many setups of a tripod thingy of some sort that costs thousands of dollars connected to an expensive cloud service. After all that, you then can look at it loaded on your PC and take measurements.
This may not have a superdense cloud and it may not be super accurate, but it is good enough for lots of uses. If you really can take a short room video in an app and not do a lot of processing yourself and then pop on the Quest headset and walk around in the room and look at stuff and measure stuff, you would sell thousands of these just to construction people. Who needs a tape measure? I have a phone and I will take a video.