Pixie – Slides

So, we have these amazing photorealistic 3D models from methods like NeRF and Gaussian Splatting. They look great. Now, you can take your phone around, take a bunch of photos of your living room, put them in a computer, and we can lift your 2D images into a 3D virtual world.
They're great but they're static, The core question we're tackling is: how can we bring these scenes to life?
So can we integrate physics? Maybe if they can also understand physics, we can then make them move and interact realistically?

Why learn physics fields?

Photorealistic 3D reconstructions (NeRF, GS) capture static geometry & appearance but lack physics.
How can we make the scene move?
Can we integrate physics?

Rendering + Physics

The answer is yes! By tagging physics params to each 3D coordinate.
The key idea is to augment the existing 3D representation.
Just like in NeRF or GS where each point has color and opacity,
we can also tag it with physical properties—like its stiffness or mass density.
Once we have these physics-aware points, we can hand them off to a physics simulator like MPM to actually compute the motion. This gives us a dynamic, interactive scene.

Yes! By tagging physics params to each 3D coordinate.
Can simulate the scene with a physics solver (e.g., MPM).

So, how have people tried to solve this before? One popular approach is test-time optimization. The idea is to take a single scene, make a random guess about its physics, and then iteratively refine that guess using a differentiable physics engine. As the animation shows, you run a simulation, get a score, and backpropagate the error to update your parameters.

While reasonable, this sort of method is
slow. can take hours for one scene
—and the results don't generalize to new objects. You have to start from scratch every time.
They're also not very accurate. Typically need good initialization for it to converge to a reasonable result.

Related Work: Test-time Optimisation

Test-time optimization methods are:

slow.
scene-specific.
inaccurate (need good initialization).

Enter Pixie!

Pixie: predict dense material fields in a single forward pass and generalize across scenes.
Using (pretrained) visual features!
Via supervised learning!

Ok so how does it work?

We start with multi-view images of a scene. Then, we use a NeRF-like model to reconstruct the 3D shape and appearance, and we distill rich visual features from a model like CLIP into this representation.
Next, a 3D U-Net takes this feature grid and directly predicts a dense field of physical parameters at every point in space.
Finally, we can use these predicted parameters in any downstream physics engine, like an MPM solver with Gaussian Splats, to get real-time, realistic simulations.

Method Overview

Multi-view RGB encoded by NeRF with distilled CLIP features.
3D U-Net predicts dense material fields.
Gaussian splats + MPM solver yield real-time simulations.

PixieVerse Dataset

Assets

Super-classes

Material Models

Annotations

E, ν, ρ, ID

So, how well does it work?

Quantitatively, Pixie achieves state-of-the-art results on our benchmark. We use a large vision model like Gemini Pro to rate the realism of the resulting animations from competing methods, and found that Pixie outperforms prior methods around 2 to 4.
And because it's a single forward pass, it's also about 3 order of magnitude faster than test-time optimization methods.
Pixie also outperforms other baselines on traditional metrics like PSNR and SSIM, which measures the rendering quality of the predicted animations against the groundtruth videos from our dataset.
Here's a picture, Pixie runs in around 2 seconds while other methods can take hours (on A6000 GPU), and still perform much worse.

Quantitative Results

2.2–4.6× higher Gemini-Pro realism than baselines.
Runs 10³× faster than optimisation-heavy methods.
State-of-the-art PSNR / SSIM gains on PixieVerse.

Let's look at some qualitative results. Here you can see that Pixie is able to accurately recover not just the material class, but also the continuous physical parameters across the object, like Young's modulus. Notice the fairly smooth and accurate distribution of values. For example, in the vasedeck, the material models basically segment the vase into the leaves and the pot. Values like young modulus, poisson ratio, and density are also consistent, with the leaves being more elastic and less stiff than the pot. Across different objects, you can also see accurate prediction. For example, the density of the metal can is much higher than the other objects.

Qualitative Results

Pixie simultaneously recovers discrete material class , E, ν, ρ with a high degree of accuracy.

Qualitative Results

Pixie produces stable, physically plausible motion and correct material attribution.
Baselines suffer from stiffness errors, collapse, or noisy artefacts.

Qualitative Results

Pixie produces stable, physically plausible motion and correct material attribution.
Baselines suffer from stiffness errors, collapse, or noisy artefacts.

Zero-shot Transfer to Real Scenes

So why does Pixie generalize so well to real scenes, even though it was only trained on synthetic data?

The key is the use of pretrained rich visual features from CLIP. This helps bridge the sim-to-real gap.
When we ablate this and use simpler features like raw RGB or just occupancy, the performance drops significantly. The model struggles to correctly identify materials and predict their properties, confirming that pretrained features are crucial for this task.

Ablation: Why does Pixie generalize?

Pretrained CLIP features are key for sim2real transfer.
Ablating CLIP and using RGB or occupancy features significantly degrades performance:

Conclusion

Pixie bridges 3D vision and physics with real-time, accurate and generalizable inference.
Project Website: pixie-3d.github.io

@inproceedings{le2025pixie,
  title={{Pixie}: Fast and Generalizable Supervised 3D Physics Learning from Pixels},
  author={Le, Long and Lucas, Ryan and Wang, Chen and Chen, Chuhao and Jayaraman, Dinesh and Eaton, Eric and Liu, Lingjie},
  year={2025}
}