If you’re into 3D scanning or photogrammetry and haven’t heard of 3D Gaussian Splatting (3DGS), where have you been! Since research on the technology was presented at last year’s SIGGRAPH computer graphics summit, it has become one of the industry’s hottest topics.
Without delving into too much technical detail, 3DGS represents a novel way of turning entire scenes into lifelike 3D models. Unlike 3D scanning and photogrammetry, which transform objects or areas as they are (at a fixed moment in time) into dense 3D meshes via laser capture or photo alignment, 3DGS yields models that are dynamic, lightweight, and mesh-free.
How does the technology do this? Well, like structure-from-motion photogrammetry, it gathers data points from images or videos, but rather than converting these into meshes via feature extraction, reconstruction, and filtering, it uses these to initialize blob-like clouds of 3D ‘gaussians.’
3D Gaussian Splatting – an emerging method of creating lifelike digitizations from extracted photo or video data – allows for rapid, mesh-free 3D modeling
Special algorithms then optimize and rasterize gaussian points in real time to form ultra-realistic visualizations. Real-time neural radiance field (NeRF) rendering does something similar, but 3DGS’ creators say their approach is different, as it doesn’t rely on neural networks or render empty space, making it less GPU-intensive, while still learning dynamic scenes at pace.
This is because NeRF methods rely on implicit color and density representation, whereas 3DGS utilizes finite clouds of 3D gaussians. Switching to an ‘explicit representation’ format essentially accelerates training and rendering, while lifting the limit on scene size – albeit with more detailed scenes requiring a greater amount of gaussians – and processing power.
Compared to existing NeRF-based approaches, 3DGS has already proven much faster, in one case learning a scene that took Google mip-NeRF 360 48 hours, in 14 minutes, while yielding a quality beyond that of Nvidia InstantNGT.
3D Although it remains the subject of intense R&D, the technology has already begun to enter the mainstream.
With apps like Polycam and KIRI Engine rolling out 3DGS features, and programs such as Volinga establishing 3D modeling pipelines via Unreal Engine integration, it appears certain the technology will carve itself a niche in TV, movie, and video game modeling.
But what is the true potential of 3DGS? To find out, we spoke to innovators across this emerging space, not just with early adopters, but a member of the research team who actually invented it.
Why do we need 3D Gaussian Splatting
It may have been honed via FUNGRAPH, a project backed with €2.5 million in European Research Council funding, but 3DGS is the result of 20 years’ image-based rendering research.
In the world of 3D modeling, ‘image-based rendering’ simply means the process of generating models from images taken of a given scene. Given that photogrammetry has been capable of this for some time, you may well ask: ‘Why do we need this technology?’ Well, photogrammetry has its issues – notably around scale (it only yields static snapshots), accuracy, and file size.
Structured light 3D scanning does offer a faster, more accurate alternative, yielding highly detailed mesh data, ideal for industrial-grade reverse engineering and inspection. However, with not all applications requiring such detailed data, significant research is being poured into identifying a less computing-intensive modeling method.
3D Gaussian Splatting turns real environments into dynamic, photorealistic virtual ‘scenes’.
NeRF vs 3D Gaussian Splatting
Before 3DGS, NeRF was the poster child of this movement. Using machine learning, the technology is capable of generating a radiance field of an object in 3D space that realistically captures shape, texture, and fine surface details. But, FUNGRAPH researcher George Drettakis says 3DGS takes this idea further, yielding data that’s “much easier to manipulate.”
“NeRFs are implicit. They give you a continuous density, so they optimize well. But they’re also wasteful. In most 3D model space, you don’t have anything,” said Drettakis.
“You don’t know where stuff is either, so you need to do ‘ray-casting,’” he added. “3D Gaussian Splatting is based on explicit representation, so primitives are placed in exactly the right position in 3D space, making them easier to interpret. That’s one of the reasons the technology has become extremely popular.”
With many app developers adopting the technology, clearly the industry agrees. KIRI Engine Co-Founder Jack Wang says its ability to render in real time with minimal artifacts means the technology “has everything cool about NeRFs, with none of the bad stuff.”
Doly developer aniML Co-founder Pierre Pontevia goes further, adding that it yields such a “high level of quality and performance, media, entertainment, and filmmaking professionals love it.” But these industries aren’t even Doly’s main target market – and our other interviewees agree – the technology’s speed, resolution and mesh-free nature also lend it a broader appeal.
3D Gaussian Splats are gaining popularity as they don’t require editing like 3D meshes and render faster than NeRF models
3D Gaussian Splatting pioneers
Wang, for instance, agrees that 3DGS has potential in CGI, but he believes it can also be a game-changer for everyday app users. For example, KIRI data shows that the majority of high-quality models are created by just 20% of users. Wang says this is due to photogrammetry’s high learning curve and tips 3DGS’ looser tolerances to democratize 3D data capture.
Since adopting the technology last year, he adds that his firm’s app has been applied in “all-new applications,” and suggests 3DGS mapping and video game modeling may not be too far away.
Seeking to make a splash in the PC modeling space, Volinga has also developed a desktop 3D Gaussian Splatting platform, which exports directly to Unreal Engine and Pixoto. Growing in popularity among CGI developers, these integrations are potentially a real coup, given that they represent one of the first clear pipelines for professional end-users.
Volinga, Doly, and KIRI Engine are now bringing the technology to market for applications in CGI, e-commerce, and more
Offering the platform as a cloud-based solution is also an interesting design choice. This means opening 3DGS to all users (including those with less powerful PCs). However, Volinga CEO Fernando Rivas-Manzaneque is keen to add that the offline version of his firm’s software is there to act as a ‘blackbox’ for sensitive data, joking that “producers don’t want the location of the next Game of Thrones episode going through our public cloud.”
Elsewhere, away from the worlds of CGI and mainstream smartphone capture, Doly is targeting a very specific type of customer: Merchants and social media stars. With the rise of thrifting and personal branding on platforms like TikTok, its CEO Remi Rousseau says 3DGS can take advantage, by creating content for clients who can’t afford help from an agency.
“We’re seeing more and more products introduced to help monetize your online presence,” said Rousseau. “We also truly believe in second-hand. In future, you will be able to examine such products in 3D before buying them. It’s the kind of situation where you definitely need to inspect the product, so it’s a very interesting use case for us!”
Is a rising tide lifting all boats
Each of our interviewees see 3DGS progressing slightly differently, but they all agree on one major point: It’s not designed to challenge more geometrically accurate mesh-based modeling.
On the contrary, they broadly believe that with the right development path, 3DGS models can one day be used alongside those captured with professional 3D scanning. Even when it comes to other developers utilizing the same technology to bring competitive products to market, the prevailing attitude is one of ‘a rising tide lifting all boats’ – as a certain JFK once said.
Tech giants Google and Nvidia continue to develop NeRF rendering methods as 3D Gaussian Splatting alternatives
That’s not to say that the field isn’t competitive. Take Nvidia and Google, who continue to develop NeRF-based methods undeterred. When approached for comment, Google Senior Staff Research Scientist Jon Barron even went to the trouble of pointing out that while “3DGS is a promising step forward” in terms of speed, it “reduces the overall quality of rendered images.”
It’s worth highlighting that there are now faster NeRF approaches to Google mip-NeRF 360 out there that deliver high quality results, and FUNGRAPH researcher Georgios Kopanas is a Google Research Scientist too. But with Google itself admitting to “investigating this new research direction,” it’s clear how highly 3DGS is being regarded in the industry.
As the technology progresses, Drettakis admits there’s a lot of work to be done to optimize 3D Gaussian Splats for end-usage in CGI. But, as 3DGS becomes the darling of the 3D scanning start-up world, he believes there are enough bright minds working on it to improve results significantly, and maybe one day tiptoe into another field: ‘Generative 3D.’
“Technically 3D is built right into [text-to-video platform] Sora,” he concluded. “Getting it to do exactly what you want may take a little more fine tuning. But it has the potential to solve a lot of the hard problems in 3D generative AI. Generative video is going to open the door to generative 3D in a way that was not previously imaginable. And that could happen relatively quickly!”
