Deep Reading Notes: EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

1. Basic information

• Method: EgoGaussian
• Authors / affiliations: Daiwei Zhang, Gengyan Li, Jiajie Li, Mickaël Bressieux, Otmar Hilliges, Marc Pollefeys, Luc Van Gool, Xi Wang; ETH Zürich, Google, Microsoft, KU Leuven, INSAIT
• Year / venue: 2025 / 3DV
• Paper: https://arxiv.org/abs/2406.19811
• PDF: https://arxiv.org/pdf/2406.19811
• Project: https://zdwww.github.io/egogs.github.io/
• Reading mode: Deep Dive

2. Positioning in one sentence

EgoGaussian initializes from static clips and updates sequentially through dynamic clips, explicitly maintaining background Gaussians, rigidly moving object Gaussians, and per-frame object poses in egocentric RGB video. Hands and bodies are primarily masked occluders rather than shapes reconstructed jointly with the scene.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Egocentric video → background / object / trajectory	Preprocessing, static initialization, and dynamic estimation each have external dependencies
Figure 2	Sequential dynamic-object modeling	Pose propagates from the previous object state, with random historical-frame revisits for regularization
Table 1	Static / dynamic rendering on HOI4D and EPIC	Hands / bodies are masked in evaluation; scores are not full-scene 6D pose accuracy
Tables 2–3	Larger time gaps and full-scene fine-tuning ablations	Test sequential-model robustness and the need to couple background and object updates

3. Problem setting and boundaries

• Input: monocular egocentric RGB video; preprocessing also uses camera poses from COLMAP or the dataset, hand–object / body masks, and interaction onset / offset.
• Outputs: background $\mathcal G_{bg}$, object $\mathcal G_{obj}$, per-frame rigid object transforms, and a dynamic scene for free-viewpoint rendering.
• Object assumption: the manipulated object is approximately rigid within a dynamic clip; masked hands and bodies do not contribute photometric gradients.
• Task boundary: the main benchmark concerns novel-view reconstruction. Rendered consistency only indirectly supports object-pose correctness; ADD, rotation error, and translation error are not reported.

4. Motivation: why find static clips first?

Egocentric interaction often involves hand occlusion and rapid object motion, making complete geometry difficult to recover from one frame. EgoGaussian first uses clips before pickup / after placement to build visible object Gaussians and a background, then tracks rigid motion using this better initialized object shape.

Unlike directly fitting a deformable 4D representation, this keeps object shape explicitly separable from pose. My interpretation is that it makes trajectories readable and editable, but shifts difficult requirements to preprocessing: incorrect static clips, masks, or camera poses can yield internally consistent yet displaced trajectories.

5. Method details

5.1 Preprocessing and static reconstruction

1. Split the video into static and dynamic clips using interaction onset / offset. Objects stay still in static clips, though people may move.
2. Generate hand masks with EgoHOS and object / body masks with Track-Anything; apply 2-pixel dilation to improve robustness at occlusion boundaries.
3. Estimate camera poses and sparse points with COLMAP, or directly use dataset poses such as those from EPIC Fields.
4. Train 3DGS on static clips without color gradients in body-mask regions; trainable labels then divide Gaussians into object / background sets.

Object initialization here obtains an object-labeled set of 3D Gaussians, rather than regressing 6D pose from a single 2D detection. This representation enables subsequent rigid transformation.

5.2 Dynamic stage: sequential pose and shape updates

For a dynamic frame $f_t$, start from the nearby optimized object Gaussians $\mathcal G_{obj,t-k}$, estimate relative translation $t_t$ and a 6D rotation representation $\tilde R_t$, and apply the same rigid transform to all Gaussian positions, rotations, and covariances. Optimize using rendered object RGB crops and alpha masks.

To avoid fitting only the current frame, each training iteration uses a random historical frame for regularization with probability 0.5. Background $\mathcal G_{bg}$ is also retrained on regions revealed by object motion. Moving objects thus provide both pose updates and opportunities to observe the background again.

Module	Input	Output	Main role
static reconstruction	Static frames, camera poses, masks	$\mathcal G_{bg}$, initial $\mathcal G_{obj}$	Initialize trackable object geometry / appearance
object pose estimation	Previous state, current object crop	$R_t,t_t$	Predict current rigid motion
object shape refinement	Current RGB / alpha	Updated $\mathcal G_{obj}$	Correct local geometry and visibility
static scene update	Regions revealed in dynamic frames	Updated $\mathcal G_{bg}$	Complete previously occluded background

6. Experimental analysis

6.1 Data and evaluation protocol

• HOI4D: 4 videos of approximately 20 seconds, at 480×270 and 15 fps.
• EPIC-KITCHENS: 4 clips of approximately 10.43 seconds, averaging 455×256 at 60 fps.
• Training samples alternate frames. Baselines include 4DGS, Deformable 3DGS, and corresponding variants without hand-region gradients.
• Metrics are PSNR↑, SSIM↑, and LPIPS↓. Actor hands / bodies are explicitly masked to reduce human-modeling differences in object / scene comparisons.

6.2 Main findings from Table 1

Data	Frame type	EgoGaussian (SSIM / PSNR / LPIPS)	Main comparison in the paper
HOI4D	static	0.96 / 30.99 / 0.08	4DGS 0.88 / 25.33 / 0.13; Def-3DGS 0.90 / 25.85 / 0.11
HOI4D	dynamic	0.95 / 30.33 / 0.09	4DGS 0.89 / 25.34 / 0.13
EPIC	static	0.85 / 28.33 / 0.19	4DGS w/o hands performs better: 0.87 / 28.90 / 0.16
EPIC	dynamic	0.88 / 28.34 / 0.17	4DGS 0.79 / 22.54 / 0.24

The main evidence favors explicitly separating object pose and Gaussians over a general deformation field for preserving interacting objects in dynamic clips. EPIC static is an exception: EgoGaussian does not lead every column. Its strength lies in dynamic modeling and object state under occlusion, rather than superiority on all static views.

6.3 Reading the ablations

• Larger pose-estimation time gaps reduce training cost but lower object PSNR. Table 2 supports usable reconstruction under such gaps, rather than lossless robustness to fast motion.
• Omitting full-scene fine-tuning substantially degrades results; background and object must be coupled again during training despite their separate representations.
• These ablations support pipeline components, but do not directly evaluate camera-pose errors, corrupted masks, or onset/offset annotation errors. Real upstream noise remains a major gap.

7. Limitations, failures, and reproduction conditions

1. Preprocessing dependence: onset / offset, object masks, hand/body masks, and camera poses are not estimated from scratch within the method. Their errors enter all experiments as implicit priors.
2. Observability: large motion, leaving and reentering the view, weak texture, and prolonged full occlusion can prevent RGB gradients from stably constraining pose.
3. Object model: one rigid transform applies to the whole Gaussian set; soft, articulated, or opening / closing objects violate this assumption.
4. Resources: sequential per-frame optimization and densification/pruning form an offline reconstruction workflow. These results do not establish real-time AR / robotics readiness.

8. Implications for object pose research

• With a reliable explicit object representation, pose tracking can become a rendered-object / observation consistency problem.
• Human information in interaction should distinguish occlusion to mask from contact / motion priors to exploit. EgoGaussian handles the former; InterTrack attempts the latter.
• Useful next evaluations include direct 6D pose metrics, frontend mask / camera-pose perturbations, and uncertainty-aware strategies for dropped frames / reinitialization.