Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
Dynamic 3D Gaussians keeps Gaussian appearance and identity fixed while updating position and orientation over time. Local rigidity and isometry regularization yield dense 6-DoF trajectories from dynamic multiview rendering.
27 calibrated training camerasinitial depth camerasforeground/background masksper-timestep optimization
Reading Notes
Overview / Deep Dive
Available src/paper-notes/dynamic-3d-gaussians-3dv2024/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
Dynamic 3D Gaussians: Coarse Reading
One-sentence conclusion
Dynamic 3D Gaussians keeps each Gaussian’s color, opacity, and scale persistent across time, updating only position and orientation. Local rigidity, rotation consistency, and isometry constraints let dynamic novel-view synthesis naturally yield dense 6DoF trajectories for scene elements.
Representation
The first frame initializes 3D Gaussians from multi-camera / depth point clouds. Later frames extrapolate previous velocities and optimize only Gaussian centers and quaternions, regularizing neighboring relative distances and rotations. Gaussian identities thus survive without independent per-frame densification.
One Gaussian retains its identity across frames; position and orientation changes form dense 6DoF motion. Sources: Figures 2–3 from the paper. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic commentary.
Results
Across six PanopticSports sequences, 3D median trajectory error is 2.21 cm with 100% survival; 2D MTE is 1.57 px, versus PIPs 15.7 px. Particle-NeRF synthetic-data PSNR is 39.49. However, each later timestep needs approximately 2,000 optimization iterations / 50 seconds, totaling approximately two hours for 150 frames.
View synthesis, 3D tracking, and 2D tracking are reported together. 3GS-O renders reasonably but cannot reliably track without persistent Gaussian identities. Sources: Tables 1–2 from the paper. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic commentary.
Boundaries
This produces thousands of local Gaussian 6DoF trajectories, covering people and non-rigid motion rather than one rigid-object pose. It requires 27 training cameras, an initial depth cloud, and offline optimization, and cannot recover elements invisible initially or newly entering the scene.
Available src/paper-notes/dynamic-3d-gaussians-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Dynamic 3D Gaussians: Deep Reading
1. Why Ordinary 3DGS Is Not Directly a Tracker
Independent per-frame 3DGS can reconstruct dynamic video, but densification, splitting, and pruning continuously change the Gaussian set. Good images do not identify corresponding Gaussians across frames. Persistence is the central design: fixed color, opacity, and scale preserve identities while positions and orientations change.
Tracking becomes a byproduct of the representation rather than an additional trained optical-flow head. Each Gaussian’s (xt,qt) forms a 6DoF trajectory; coordinated local-neighborhood changes express non-rigid motion.
2. Temporal Optimization
Initialize from synchronized depth-camera point clouds and optimize the first frame as ordinary 3DGS. Later timesteps:
Extrapolate initial positions / quaternions using velocities from the previous two frames.
Reset Adam momentum.
Render from 27 training cameras and minimize RGB / background losses.
Add local-rigidity, rotation-similarity, and isometry regularizers.
Fix color, opacity, and scale, preventing appearance changes from concealing incorrect motion.
Local rigidity preserves neighboring Gaussian structure over short intervals, rather than treating an entire person as rigid. Sources: Figures 2–4 from the paper. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic commentary.
3. PanopticSports Evaluation
Six Panoptic Studio motions contain 150 frames at 30 FPS, with 27 training and 4 test cameras out of 31. Evaluation uses 21 hand / face 3D keypoint trajectories and 371 projected 2D tracks. The benchmark is small but jointly evaluates metric 3D tracking and held-out-view rendering.
PIPs obtains 2D MTE 15.7 px and survival 79%. The comparison is not fully matched: Dynamic 3DGS sees 27 synchronized cameras, while PIPs sees only the monocular tracking video, as the paper acknowledges.
The main gap to 3GS-O is tracking rather than rendering; persistent identity and motion regularization are the key variables. Sources: Tables 1–2 from the paper. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic commentary.
4. Interpreting Ablations
Full-model Juggle 3D MTE is 1.90 cm; removing background loss gives 8.46, parameter fixing 30.7, and forward propagation 6.32, while ordinary 3GS-O gives 32.81. Persistent parameters and propagated initialization matter more than any single local regularizer. Removing rigidity gives MTE 4.32, supporting neighborhood geometry too.
Rendering and tracking are examined together: some ablations barely change PSNR while multiplying trajectory error. Sources: Table 3 and Figures 5–6 from the paper. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic commentary.
5. Computation and Limitations
Approximately 10k iterations initially and 2k iterations / 50 seconds per later frame; approximately two hours for 150 frames.
Requires dense synchronized, calibrated cameras and an initial depth cloud, rather than ordinary monocular video.
Low-contrast regions such as gray clothing against gray backgrounds can confuse foreground and background.
Only 21 ground-truth 3D trajectories, insufficient to characterize every dense Gaussian’s error.
6. My Assessment
The historical contribution turns 3DGS from fast rendering primitives into dynamic state primitives with identities. For object pose, it suggests tracking dense local 6DoF motion first, then aggregating rigid objects / parts as needed, rather than segmenting rigid objects before assigning each a pose. Data and optimization costs remain substantial, far from online monocular tracking.
