GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects
GaussianArt jointly learns part geometry and motion decomposition with articulated 3D Gaussians. Part segmentation, L0 regularization, and trajectory regularization support digital twins of articulated objects with up to 20 parts.
Beijing Academy of Artificial Intelligence / Tsinghua University / Nanyang Technological University / Beijing Institute of Technology / Huazhong University of Science and Technology
multi-view-consistent part masksobject motion statescamera calibration / reconstruction
Reading Notes
Overview / Deep Dive
Available src/paper-notes/gaussian-art-3dv2026/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
Paper Reading Notes (Coarse): GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects
1. Basic information
Authors / affiliations: Licheng Shen et al.; BAAI, Tsinghua University, NTU, Beijing Institute of Technology, Huazhong University of Science and Technology.
GaussianArt jointly optimizes part geometry and motion using articulated 3D Gaussians, rather than reconstructing shapes separately at multiple states and then inferring articulation through post-hoc alignment. It also introduces MPArt-90, covering objects with up to 20 parts.
3. Problem setting
Inputs: multiview RGB-D of the same articulated object in two states, known camera intrinsics / extrinsics, and generated part-level masks consistent across views.
Outputs: canonical Gaussian geometry, part assignments, articulation axes / motion, and an interactive digital twin.
The task concerns complex multipart articulation, rather than directly estimating an object’s 6DoF pose from one RGB image.
4. Method overview
Initialize canonical Gaussians using multiview-consistent part segmentation.
Associate each Gaussian with part weights and articulation parameters, jointly fitting observations across states.
L0 regularization strengthens part assignments, while trajectory regularization constrains motion learning with matched 3D points across states; soft-to-hard optimization improves part interpretability.
5. Experimental overview
MPArt-90: 90 objects, 20 categories, and 2–20 parts per object.
On All-90, Axis Angle Error is 12.17 (ArtGS: 24.34); static / moving-part CD is 2.68 / 5.46 (ArtGS: 11.57 / 380.29).
Without part segmentation, Axis Angle Error becomes 28.60 and moving-part CD reaches 186.67, showing that initialization is decisive.
6. Strengths, Limitations, and Insights
Strengths
Joint geometry and motion learning structurally avoids drift from post-hoc motion decomposition.
MPArt-90 specifically exposes existing methods’ failures with 6+ parts, beyond comparisons on only 2–3 parts.
Limitations / risks
Requires multiview part masks and observations across states; a lack of direct constraints on intermediate motion states can distort extreme opening / closing transitions.
Its outputs support digital twins / motion estimation and do not directly replace a single-frame object-pose system.
7. Points Worth Reading Closely
Table 2 shows that part segmentation affects motion error much more than L0 / trajectory regularization. For articulated pose, getting part segmentation right is often more important than the downstream optimizer.
Available src/paper-notes/gaussian-art-3dv2026/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects
1. Paper Information
Paper: GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects
Problem: given multiview RGB-D of the same object in two states and camera intrinsics / extrinsics, jointly recover geometry, part segmentation, and rigid motion, then interpret motion as joint axes and states.
Main finding: representing an articulated object with part-identified 3D Gaussian Splatting and jointly optimizing appearance, motion states, and kinematic constraints substantially reduces moving-part reconstruction error. Stable multiview part segmentation is the most important prerequisite.
2. Method in Detail
2.1 From Static to Articulated Gaussians
Each Gaussian learns part-assignment weights. Part-level rigid rotation / translation bases transform canonical Gaussians: soft training blends bases, while hard training selects the highest-weight part’s motion. Differentiable rendering supervises transformed Gaussians with RGB-D and part masks. Geometry is shared across states; joint axes and states are interpreted from learned rigid motion afterward, rather than directly parameterizing all training motion with axes and angles.
With reprojection / rendering losses alone, multiple similar but inconsistent motions can explain a part. The paper adds two structural priors:
L0 part-motion regularization: sparsifies differences between neighboring Gaussians’ part weights, encouraging spatially consistent assignments and suppressing local drift;
Trajectory regularization: depth and camera parameters lift cross-state image matches to 3D; transformed points must approach their matches. It does not directly impose joint equations along entire intermediate trajectories.
These complement multiview part segmentation: segmentation supplies discrete part membership, while regularization keeps continuous optimization within a plausible kinematic space.
3. Experimental Results and Careful Interpretation
The authors construct MPArt-90 (90 objects, 20 categories, 2–20 parts each) to evaluate articulation estimation and reconstruction across states. Against ArtGS, the full method reduces axis-angle error from 24.34 to 12.17 and axis-position error from 1.45 to 1.06 over all 90 objects. CD-m, which more directly measures moving-part reconstruction quality, drops from 380.29 to 5.46.
The scale of the CD-m change deserves more attention than axis error alone: part-aware modeling improves moving-part geometry, but this table does not independently establish correct recovery of every unseen intermediate state. Ablations also show that removing part segmentation raises axis-angle error to 28.60 and CD-m to 186.67, revealing strong sensitivity to segmentation quality.
4. Pipeline from a Reproduction Perspective
Stage
Input
Main optimization variables
Output / role
Multiview parsing
Two-state RGB-D and known camera intrinsics / extrinsics
Cross-view part correspondences
Observations suitable for joint optimization
Part segmentation
Part masks in each view
Part labels and associations
Constrain Gaussians to rigid parts
Articulated 3DGS
RGB-D, known cameras, part associations
Gaussian attributes, part weights, rigid motion bases
Unified canonical representation
Structural constraints
Point trajectories and part labels
L0 and kinematic regularization
Eliminate local drift and stabilize joints
Novel-state rendering
Specified joint state
Part rigid transforms
Reconstruct appearance and geometry in a new configuration
5. Applicability and Limitations
The method suits articulated household objects or tools with multiple views, adequate visible texture, and segmentable part boundaries. It does not directly generate a complete articulation model from one image or guarantee correct recovery of occluded internal structures. The paper identifies a lack of direct observational constraints on intermediate motion states. Out-of-distribution appearance, inconsistent views, or similarly colored neighboring parts can also cause segmentation errors that propagate into articulation optimization.
6. Ablations and Reproduction Diagnostics
Across five multipart objects and five trials, the full model achieves Axis Angle 0.03, Axis Position 0.01, Part Motion 0.04, CD-s 0.67, and CD-m 0.14. Removing L0 gives 0.10 / 0.01 / 0.08 / 0.70 / 0.17; removing trajectory regularization gives 0.31 / 0.03 / 0.35 / 0.85 / 0.25. Both help, with trajectory regularization having a larger effect on motion parameters. Without part initialization, CD-m deteriorates to 1.34; replacing explicit part assignment with an MLP produces Axis Angle 41.57 and CD-m 478.20, an almost complete failure.
These results clarify the order of dependencies: reliable cross-view part identity must precede articulation estimation. After reasonable initialization, L0 pushes soft assignments toward interpretable rigid parts, and trajectory loss constrains motion across states. Reproduction should separately visualize segmentation, each Gaussian’s assignment entropy, joint axes, and intermediate-state trajectories, rather than tuning only the total loss after final rendering fails.
MPArt-90 training observations are not arbitrary video: each motion state has 100 randomly sampled training views and 20 test views at 800×800 resolution, with starting / ending states around 0.65–0.75 and 0.35–0.45, respectively. This provides dense view coverage, but no direct supervision of intermediate motion states, explaining the authors’ reported difficulty with extreme opening / closing transitions. These tabulated error levels should not be assumed for casual sparse video.
7. Insights for Pose Estimation Research
GaussianArt extends pose from a single rigid body’s SE(3) to part poses + low-dimensional articulation states. Two ideas transfer to 6D pose systems: part-level canonical representations can explain nonrigid appearance changes, and explicit kinematic priors can support multiframe / multiview tracking. Reliable part discovery or segmentation is the cost, so deployment evaluations should also report degradation when segmentation fails.
