Deep Reading Notes: SMORE: Simultaneous Map and Object REconstruction

1. Basic information

• Method: SMORE
• Authors / affiliations: Nathaniel Chodosh, Anish Madan, Simon Lucey, Deva Ramanan; Villanova University, Carnegie Mellon University, University of Adelaide
• Year / venue: 2025 / 3DV
• Paper: https://arxiv.org/abs/2406.13896
• PDF: https://arxiv.org/pdf/2406.13896
• Project: https://anishmadan23.github.io/smore/
• Reading mode: Deep Dive

2. Positioning in one sentence

SMORE models urban spinning-LiDAR sequences through continuous-time surfaces for background and rigid actors. Deskewing, robust ICP, and neural reconstruction alternate to correct ego poses, object poses, and maps, beyond treating box annotations as final motion truth.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Annotation-only aggregation versus joint reconstruction	Pose errors create duplicated / smeared surfaces
Figure 2	Coarse actor boxes, pose optimization, then deskewing	Scan timing affects actors as well as ego
Tables 1–2	Surface quality, actor ATE, annotation subsampling	Recover states from sparse / coarse tracks
Tables 3–4	LiDAR NVS and ego-pose evaluation	Cross-check geometry and motion quality

3. Setting and meaning of object pose

• Spinning LiDAR sweeps, initial ego poses, and actor boxes / tracks; returns have different within-sweep timestamps.
• Outputs: background / actor surfaces, actor rigid-pose sequences, and corrected ego trajectories.
• Simultaneous map and object reconstruction: poses are global sequence states rather than single-image RGB object-to-camera estimates.
• Actors are approximately rigid and associable through detection / tracking; LiDAR timestamps are available.

4. Why model continuous-time LiDAR explicitly?

Spinning LiDAR is not a global shutter. Ego / object motion changes relative poses within each sweep; one sweep timestamp creates smeared cars and duplicated maps. Ego-only deskewing misses actor motion, which SMORE additionally compensates.

Wrong poses produce wrong surfaces, but reconstructed surfaces can correct poses through registration. This motivates alternating geometry and motion optimization.

5. Method details

5.1 State representation and alternating optimization

Background / actor points map to their own world / canonical frames. Continuous-time SE(3) trajectories explain exact acquisition times:

1. Initialize ego / actors from odometry and detection / association.
2. Deskew / aggregate using current trajectories; reconstruct separate neural kernel surfaces.
3. Optimize ego / object poses against surfaces using point-to-plane residuals with Huber kernels.
4. Update points / surfaces and iterate until registration converges or stops early.

Component	Problem	Pose relationship
Continuous-time deskewing	Within-sweep motion distortion	Restore each point to its correct observer / actor pose
Object-centric surfaces	Actors overwhelmed by background	Supply local geometric anchors
Point-to-plane ICP	Boxes give coarse locations	Refine rigid transforms through normal-direction residuals
Joint ego / actor updates	Coupled errors	Avoid transferring one fixed side’s errors to the other

5.2 Reconstruction and optimization beyond end-to-end learning

Neural Kernel Surface Reconstruction handles surfaces without training an end-to-end pose network. After updates, train / test sweeps undergo refinement; about 100 test refinement iterations use mean-registration early stopping. Interpretability comes with runtime / initialization costs requiring separate evaluation.

6. Experimental analysis

6.1 Evaluation protocols

• nuScenes / Argoverse 2; LT3D-like tracks or manual annotations initialize actors.
• surface: input-to-surface nearest-neighbor distance, 10 cm / 5 cm relaxed / strict coverage.
• Pose: actor-location Average Translation Error against held-out boxes.
• LiDAR NVS: Chamfer / median depth; NeuRAD PSNR / SSIM / LPIPS across ego-pose variants indirectly assess motion quality.

Public actor / ego GT annotations themselves are imperfect, without defining an absolute reconstruction upper bound. Complementary metrics help, but some pose evidence remains indirect.

6.2 Key numbers

Result	Values	Interpretation
NKSR + GT tracks	NN distance 0.071; strict coverage 0.76	Initial-track aggregation leaves poor surfaces
SMORE + GT tracks	0.048; 0.91	Joint motion / surfaces improve consistency
SMORE + LT3D tracks	0.048; 0.90	Tracker starts approach GT-track quality
Actor ATE, 1 / 0.5 / 0.25 Hz	0.20 / 0.22 / 0.52 m	More robust than interpolation under sparse labels
LiDAR NVS	CD 0.26, median depth 0.0002	Reported gains over NeuRAD and its pose variants

6.3 Interpreting results

Tables 1–2 and Figures 1–2 support improved poses from noisy annotations, especially actor ATE under sparse labels. Ego-pose evidence is less direct: the same NeRF backend fitted to nuScenes, odometry, and SMORE poses gives better SMORE rendering / geometry. This is useful evidence, distinct from independent high-precision GNSS/INS pose GT.

7. Limitations, failures, and reproduction

1. Association: missed detections, ID switches, or wrong boxes create incorrect surfaces and local ICP solutions.
2. Weak observability: distant, occluded, sparse actors lack surface constraints for pose correction.
3. Dynamics: guarantees do not extend to deforming pedestrians, articulated parts, or reflective / multipath returns.
4. Offline costs: NKSR, repeated deskewing, and ICP suit high-quality reprocessing; main results do not establish online speed.

8. Dynamic-pose takeaways

• Surface consistency can correct tracks beyond temporal interpolation when dense geometry is available.
• Model ego / actor coupling explicitly, preventing actor trajectories from absorbing observer drift.
• Combine surface residuals with learned uncertainty / association, limiting corrections when geometry is weak.