Paper Reading Notes (Deep): Living Scenes / MoRE²

1. Key assessment

Living Scenes extends object pose from location in the current camera to motion since the previous scan and increasingly complete reconstruction from long-term observations. MoRE² uses one SE(3)-equivariant latent representation for matching, registration, and reconstruction, then jointly optimizes graph poses and implicit object surfaces at test time.

This outperforms combinations of individually strong subtask baselines on end-to-end measures: 3RScan matching + registration + reconstruction recall rises from 30.77 to 40.74. Shared geometry mainly improves consistency along the error-propagation chain, beyond a small standalone registration gain.

The task assumes instance-segmented inputs and rigid motion. Outputs are relative cross-scan 6DoF, not single-frame BOP absolute pose. Identically shaped instances, nonrigid changes, and large scene-topology changes remain future work.

2. Formalizing Living Scenes

Given scans of one environment acquired at irregular times:

$$\{S^t\}_{t=1}^{T},\qquad S^t=\{X_i^t\}_{i=1}^{N_t},$$

Here $X_i^t\in\mathbb R^{3\times n_i}$ is an instance-segmented partial point cloud.

With $S^1$ as reference, for each rescan $t>1$:

1. For $X_i^1$, determine the corresponding physical instance $X_j^t$;
2. Estimate rigid transform $T_i^t=(R_i^t,t_i^t)\in SE(3)$;
3. Align and accumulate all observations:

$$X_i^{1:T}=X_i^1\circ T_i^2X_i^2\circ\cdots\circ T_i^TX_i^T;$$

4. Recover an implicit object surface / mesh from the accumulated cloud.

Added or removed objects may remain unmatched; intermediate motion is not modeled. Relocalization here estimates object motion between scan endpoints.

3. One Representation for Three Tasks

Encoder outputs:

• $F_{\text{inv}}$: global rotation-invariant embedding for object identity;
• $F_{\text{eqv}}$: rotation-equivariant Vector Neuron embedding;
• $F_s$: scale factor;
• $F_c$: centroid.

SIM(3) equivariance can be written as:

$$\Phi(sRX+t)= \left(F_{\text{inv}},RF_{\text{eqv}},sF_s,F_c+t\right).$$

For query coordinate $p$, the canonicalized latent coordinate is:

$$F_q=\left\langle F_{\text{eqv}},\frac{p-F_c}{F_s}\right\rangle.$$

DeepSDF-style decoder:

$$\operatorname{SDF}(p)=\Psi(F_{\text{inv}},F_q).$$

Training minimizes only single-shape L1 SDF reconstruction loss. Matching and registration mainly arise from representation structure and test-time optimization, rather than separately trained task heads.

4. Instance Matching: Identity Similarity and Rotational Alignability

Invariant embeddings form cosine-similarity matrix $\Lambda$ between reference and rescan objects. Using $\Lambda$ alone confuses similar shapes; MoRE² also treats equivariant embeddings as latent 3D vectors:

1. Solve latent rotation $R_{ij}$ for each candidate pair with Kabsch;
2. Compute alignment residual:

$$E(i,j)=\|R_{ij}F_{\text{eqv},i}-F_{\text{eqv},j}\|_2;$$

3. Combine scores:

$$H=\Lambda\oslash E;$$

4. Maximize global pair scores through Hungarian assignment.

Candidates must resemble the same identity and admit a consistent rotation. Unassigned objects are treated as additions / removals according to temporal order.

Hungarian one-to-one assignment is a risk: nearly identical chairs may lack enough pairwise latent evidence to establish identity. Spatial context or global motion consistency would be needed and is not addressed.

5. Registration: Equivariant Kabsch to Continuous Surface Alignment

Initial pose:

$$F_{\text{se3}}=F_{\text{eqv}}+F_c,$$

Kabsch on two latent-vector sets yields $R_i,t_i$.

Then minimize:

$$\mathcal L_{\text{reg}} = \underbrace{ \left\| \Psi\!\left( F_{\text{inv}}^{t_1}, \left\langle F_{\text{eqv}}^{t_1}, \frac{R_iX^{t_2}+t_i-F_c^{t_1}}{F_s^{t_1}} \right\rangle \right) \right\|_1 }_{\mathcal L_{\text{sdf}}} + \underbrace{\operatorname{CD}(R_iX^{t_2}+t_i,X^{t_1})}_{\mathcal L_{\text{cd}}}.$$

The two terms have different roles:

• SDF loss moves source points toward the target continuous zero-level surface without discrete point-to-point correspondences;
• Chamfer prevents learned shape priors from drifting toward surfaces unsupported by observations.

Pose is optimized on the SE(3) manifold. Compared with discrete-correspondence methods such as GeoTransformer, this is natural when partiality varies and scans expose different surfaces, at the cost of test-time iteration and shape-prior bias.

6. Joint Optimization and Multi-Time Accumulation

Build pose graph $G$ over observations of the same object and jointly optimize:

$$\mathcal L_{\text{joint}} =\mathcal L_{\text{sdf}}+\mathcal L_z,\qquad \mathcal L_z=\|F'-F\|_2.$$

• Relative transforms in $G$ on the SE(3) manifold;
• Latent equivariant embedding $F'$ with Adam;
• $\mathcal L_z$ to regularize test-time adaptation toward the training prior;
• The paper uses 200 iterations.

New scans can improve earlier scan poses, rather than independently registering pairs and stitching them once. A common latent surface geometrically reconciles multiple partial observations.

7. Data, Metrics, and Protocols

Training:

• ShapeNet subset;
• mesh watertight preprocessing;
• SDF sampling;
• Synthetic depth rendering and point-cloud back-projection;
• Synthetic objects only for the shared encoder–decoder.

Evaluation:

• FlyingShape: a synthetic living-scene benchmark derived from ShapeNet;
• 3RScan: real indoor reference / rescans with extended instance-matching and reconstruction protocols.

The official hidden 3RScan test server is no longer maintained; reproduction relies on the authors’ processed splits / evaluation code.

Metrics:

• instance matching recall;
• Scene-level matching recall $R@25/50/75/100$;
• registration recall RR;
• median rotation error MedRE;
• translation RMSE;
• median Chamfer distance MedCD;
• reconstruction L1-Chamfer, IoU, SDF recall;
• MR: matching and registration both succeed;
• MRR: matching, registration, and reconstruction all succeed.

MR / MRR combine conditional events:

$$P(R_1,M)=P(R_1|M)P(M),$$ $$P(M,R_1,R_2)=P(R_2|R_1,M)P(R_1|M)P(M).$$

Later tasks therefore receive only earlier-stage successes, revealing compounded pipeline errors.

8. Main results

End-to-end results:

Dataset	Method	MR	MRR
FlyingShape	component-wise baseline	67.32	54.30
FlyingShape	MoRE²	74.39	62.00
3RScan	component-wise baseline	44.02	30.77
3RScan	MoRE²	49.07	40.74

The 3RScan MRR gain of 9.97 exceeds the MR gain of 5.05, suggesting unified representation / joint optimization better controls error propagation into final reconstruction.

Matching:

• FlyingShape instance recall 88.75, scene R@100 72.00;
• 3RScan static / dynamic / all matching recall: 60.32 / 87.50 / 71.77;
• 3RScan scene R@25 / 50 / 75: 87.50 / 78.41 / 50.00.

Dynamic objects outperform static ones, perhaps because moved objects have more complete, isolated crops while static objects mix with background / segmentation. This should not be read as larger motion being inherently easier.

Registration:

Dataset	Method	RR	MedRE	RMSE	MedCD
FlyingShape	GeoTransformer	77.67	1.36	16.66	0.0271
FlyingShape	MoRE² full	83.83	0.74	18.47	0.0168
3RScan	GeoTransformer	51.71	3.46	6.51	0.0141
3RScan	MoRE² full	61.11	3.77	4.74	0.0030

MoRE² does not minimize every error: GeoTransformer has better FlyingShape RMSE and 3RScan MedRE. Its strengths are higher success recall and lower geometric distance, consistent with continuous implicit surfaces helping low-overlap alignment.

9. Reconstruction and Noisy Segmentation

Reconstruction:

Dataset	Method	L1-Chamfer	IoU	SDF recall
FlyingShape	ConvONet	6.64	36.99	80.67
FlyingShape	MoRE² w/o optim	6.27	49.98	78.00
FlyingShape	MoRE² full	6.11	66.73	83.33
3RScan	ConvONet	7.61	—	64.89
3RScan	MoRE² full	6.16	—	64.12

3RScan provides incomplete, non-watertight meshes, so evaluation uses one-way Chamfer and SDF recall, not directly comparable to FlyingShape two-way Chamfer. Real-data SDF recall is 64.12 for MoRE², slightly below ConvONet’s 64.89, but Chamfer is better.

Replacing GT instance segmentation with Mask3D:

Method	Match	Reg	MR	MRR
baseline + Mask3D	43.43	47.74	27.86	20.89
MoRE² + Mask3D	45.76	51.27	40.14	33.80

From GT to Mask3D, baseline MR / MRR drops by about 33% relatively, versus about 18% for MoRE². Shared latent shapes buffer missing / noisy clouds, but absolute performance still drops substantially; instance segmentation is not solved.

10. When Accumulation Helps

Increasing from 1 point cloud to 4:

• Quickly increases surface coverage;
• Substantially reduces rotation error and Chamfer;
• Reaches nearly 75% coverage at the fourth observation;
• Additional scans then provide smaller marginal gains.

This supports joint accumulation provided new views reveal new surfaces. Repeated similar viewpoints or worse segmentation need not yield monotonic improvement.

11. Relation to Pose Estimation and Tracking

Compared with BOP absolute pose:

• BOP takes a current image and known target object and predicts object→camera;
• MoRE² takes multiple point-cloud scans and predicts relative object motion between scans;
• MoRE² needs no CAD but requires reference-instance observations and segmentation.

Compared with short-term multi-object tracking:

• Tracking assumes frequent frames and continuous motion, using velocity / association history;
• Living-scene observations are far apart with unknown intermediate trajectories;
• Matching depends more on object-identity geometry, and pose changes can be large.

Compared with scene-level registration:

• Scene registration mainly aligns static background;
• MoRE² identifies independently moved objects and estimates a different SE(3) for each;
• Output is a set of object transforms and meshes rather than one global transform.

12. Limitations

• Instance segmentation is an external input;
• Rigidity excludes cloth, soft objects, and structural deformation;
• Identical objects still create identity ambiguity;
• One-to-one Hungarian assignment does not naturally handle splits / merges;
• ShapeNet shape priors may oversmooth long-tail real objects;
• Joint test-time optimization uses 200 iterations, costing more than one forward pass;
• 3RScan GT / server status complicates independent validation;
• Background changes and large-scale scene topology are not modeled;
• Geometry-only clouds omit RGB / text semantics that could aid identical-shape matching;
• Accumulation depends on views revealing new surfaces.

13. Reproduction Checklist

• Use the authors’ ShapeNet subset and watertight-mesh preprocessing;
• Reproduce SDF sampling, depth rendering, and cloud partiality;
• Preserve Vector Neuron equivariance and centroid / scale normalization;
• Store $F_{\text{inv}},F_{\text{eqv}},F_s,F_c$ separately;
• Combine cosine similarity and latent Kabsch residuals for matching;
• Permit unmatched instances in Hungarian assignment;
• Start registration from the same Kabsch initialization;
• Implement both SDF and Chamfer losses;
• Update poses on SE(3), not as unconstrained matrices;
• Use the same 200 joint-optimization iterations and latent regularizer;
• Report GT-mask and Mask3D settings separately;
• Keep FlyingShape / 3RScan Chamfer directions distinct;
• Report end-to-end MR / MRR alongside subtask errors on successful pairs;
• Show failures for identical objects, low overlap, and segmentation errors.

14. Overall assessment

MoRE² places 6DoF registration within object-centric long-term scene understanding. Shared invariant / equivariant / implicit representations let identity, pose, and shape constrain each other; 3RScan MRR of 40.74 versus baseline 30.77 demonstrates the system-level benefit.

It still needs good instance segmentation, rigid objects, and test-time optimization, and does not solve identical-instance ambiguity. The strongest conclusion is that for intermittently acquired long-term indoor scans, sharing one equivariant implicit representation for matching, relative SE(3), and accumulated reconstruction better resists error propagation than assembling independent modules.