Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting
Arxiv
Abstract
3D reconstruction and simulation, while interrelated, have distinct objectives: reconstruction demands a flexible 3D representation adaptable to diverse scenes, whereas simulation requires a structured representation to model motion principles effectively. This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to resolve such a dilemma. MaGS constrains 3D Gaussians to hover on the mesh surface, creating a mutual-adsorbed mesh-Gaussian 3D representation that combines the rendering flexibility of 3D Gaussians with the spatial coherence of meshes. Leveraging this representation, we introduce a learnable Relative Deformation Field (RDF) to model the relative displacement between the mesh and 3D Gaussians, extending traditional mesh-driven deformation paradigms that only rely on ARAP prior, thus capturing the motion of each 3D Gaussian more precisely. By joint optimizing meshes, 3D Gaussians, and RDF, MaGS achieves both high rendering accuracy and realistic deformation. Extensive experiments on the D-NeRF and NeRF-DS datasets demonstrate that MaGS can generate competitive results in both reconstruction and simulation.
Pipeline
The MaGS pipeline comprises two stages. In Stage I, we randomly initialize 3D Gaussians and utilize a deformation fields MLP (DF-MLP) to generate a deformation field for these Gaussians. The deformed Gaussians are rendered using splatting, and the loss between the rendered view and the ground-truth video frame is computed to optimize both the Gaussian parameters and DF-MLP. At the end of Stage I, we use Marching Cubes to extract a static mesh from the 3D Gaussians. In Stage II, the parameters of the DF-MLP are copied from Stage I and kept fixed, while Mesh-adsorbed Gaussians are initialized by adsorbing random 3D Gaussians to the mesh. In each iteration, the timestamp is fed into the DF-MLP and combined with the ARAP algorithm to deform the Mesh-adsorbed Gaussians. The ARAP-deformed Mesh-adsorbed Gaussians are then forwarded to a Relative Deformation Fields MLP (RDF-MLP) to compute the RDF. The Gaussians on the mesh then hover according to the RDF, resulting in the final deformed output for rendering. In this process, the parameters of RDF-MLP, mesh and 3D Gaussians are jointly optimized based on the rendering loss. For simulation, Mesh-adsorbed Gaussians are deformed using Dragging and ARAP deformation, calibrated by RDF to achieve the final deformation result. The simulation result can be directly rendered due to the 3D Gaussian representation.
Qualitative Results
MaGS achieves state-of-the-art reconstruction quality on the D-NeRF[4] dataset.
Deformation Simulating
Mesh Output
Quantitative Results
BibTex
@article{ma2024reconstructing,
title={Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting},
author={Ma, Shaojie and Luo, Yawei and Yang, Yi},
journal={arXiv preprint},
year={2024},
url={https://arxiv.org/abs/2406.01593}
}