Deep Reading Notes: DVMNet

1. Key assessment

DVMNet challenges discrete-hypothesis scoring for single-reference relative pose. RelPose / 3DAHV approximate continuous $SO(3)$ with many sampled rotations, increasing computation with hypothesis count while leaving no explicit final correspondence.

DVMNet instead:

1. Lifts reference / query 2D features into a shared 3D voxel grid.
2. Establishes soft voxel-to-voxel correspondences.
3. Weights them using 2D masks and learned 3D objectness.
4. Recovers relative rotation in one weighted Procrustes / SVD solve.

Mean angular error is 19.95° over 10 unseen CO3D categories, versus 3DAHV 28.44°, with approximately 23 ms per pair on A100. The central value is combining network predictions with closed-form geometry, beyond voxels merely looking geometric.

2. Task Boundaries

Input:

• Reference RGB crop $I_r$ of an object;
• Query RGB crop $I_q$;
• Ground-truth bounding boxes in experiments.

Output:

$$\Delta R=R_qR_r^\top.$$

If absolute reference rotation $R_r$ is known, $R_q=\Delta R R_r$. Translation is not estimated, so experimental relative object pose strictly means relative rotation.

Not required:

• CAD / mesh;
• depth;
• test-object training;
• pose hypothesis enumeration.

Object localization and shared instance identity are nevertheless assumed solved.

3. Cross-view 2D→3D autoencoder

The shared ViT extracts:

$$F_q,F_r\in\mathbb R^{C\times H\times W}.$$

Three cross-attention modules let query features attend to reference keys / values, and vice versa, encoding cross-view relationships before lifting.

The network reshapes channels into depth:

$$\mathbb R^{C\times H\times W} \rightarrow \mathbb R^{(C'+1)\times D\times H\times W}.$$

producing:

• Voxel features $V_q,V_r\in\mathbb R^{C'\times D\times H\times W}$;
• Objectness volumes $O_q,O_r\in\mathbb R^{1\times D\times H\times W}$.

Without intrinsics, depth, or multi-view triangulation, these coordinates are an object-centric latent 3D grid shaped by supervision rather than metric camera-frame geometry.

The decoder aggregates along depth, applies three self-attention modules, and reconstructs foreground image $\hat I$ and mask $\hat M$. The loss is:

$$\mathcal L_{\text{ae}}= \mathcal L_{\text{mse}}(\hat I_q,I_q^{gt}) +\mathcal L_{\text{mse}}(\hat I_r,I_r^{gt}) +\mathcal L_{\text{bce}}(\hat M_q,M_q^{gt}) +\mathcal L_{\text{bce}}(\hat M_r,M_r^{gt}).$$

Autoencoder supervision encourages object patterns in the voxels and reduces background contamination of matching.

4. Deep voxel matching

Flatten valid voxels into $v_q^i,v_r^j\in\mathbb R^{C'}$, with similarity:

$$s_{ij}= \frac{(v_q^i)^\top v_r^j} {\|v_q^i\|_2\|v_r^j\|_2}.$$

Softmax at temperature $\tau$ converts scores into soft assignments of query coordinates:

$$X'_q=p(S/\tau)X_q.$$

Each reference voxel thus obtains an expected query coordinate, allowing end-to-end gradients without hard nearest neighbors.

With reliable correspondences, relative rotation follows from covariance

$$H=\sum_i x_r^i(x_q^i)^\top=U\Sigma V^\top$$

through closed-form $\Delta \hat R=VU^\top$. Actual 2D lifting produces background and incorrect voxels, motivating WCV.

5. Weighted Closest Voxel

WCV weights combine two complementary cues:

• Repeat the predicted 2D mask along depth into a pseudo-3D mask, removing obvious background.
• Learned 3D objectness further assesses whether voxels represent the object.

The final weighted covariance is:

$$H=\sum_i \omega_i x_r^i(x_q^i)^\top.$$

Soft assignment, masks, and objectness determine $\omega_i\in(0,1)$, followed by SVD rotation recovery. Pose loss compares continuous 6D representations of predicted / ground-truth rotations:

$$\mathcal L_{\text{pose}}=\|q(\hat R)-q(R_{gt})\|, \qquad \mathcal L=\mathcal L_{\text{ae}}+\mathcal L_{\text{pose}}.$$

Lifting, matching, weighting, and the closed-form solver can therefore train end to end.

6. CO3D: Disjoint-Category Evaluation

CO3D contains 18,619 video sequences and 51 categories. Following RelPose:

• 41 training categories;
• 10 test categories;
• All test categories unseen during training;
• Geodesic angular-error evaluation:

$$e=\arccos\left(\frac{\operatorname{tr}(\hat R^\top R_{gt})-1}{2}\right).$$

Method	Mean angular error
SuperGlue	71.47°
LoFTR	78.64°
ZSP	86.66°
direct regression	47.08°
RelPose	51.80°
RelPose++	32.82°
3DAHV	28.44°
DVMNet	19.95°

DVMNet improves on the nearest comparator, 3DAHV, by 8.49°. Poor image-matching performance shows that large viewpoint gaps on one object exceed what ordinary 2D matching reliably handles.

It does not win every category: suitcase errors are 16.13° for 3DAHV and 16.85° for DVMNet. DVMNet leads the other 9 categories, so one outlier category does not drive the mean gain.

7. GROP: Real and Large-Scale Synthetic Objects

GROP includes:

• Objaverse: synthetic multi-view images of 128 test objects;
• LINEMOD: 5 real objects;
• Images containing test objects excluded from training.

Data	SuperGlue	LoFTR	RelPose++	3DAHV	DVMNet
Objaverse mean	102.40°	134.05°	33.49°	28.11°	20.19°
LINEMOD mean	64.79°	84.49°	46.67°	41.55°	36.82°

LINEMOD has a stronger domain shift and textureless household objects difficult for RGB lifting. DVMNet leads the mean, but driller error 46.29° is clearly worse than 3DAHV 32.65°; local-structure robustness is not universal.

8. WCV Ablations and Efficiency

CO3D mean:

WCV	2D mask	voxel objectness	Error
None	No	No	31.78°
Direct regression replacing WCV	No	No	21.64°
Unweighted closest voxel	Yes	No	20.92°
WCV	No	Yes	20.07°
WCV	Yes	Yes	19.95°

Interpretation:

• Removing the geometric solver entirely increases error by +11.83°.
• Direct regression already achieves 21.64°, showing strong lifted features.
• Correspondence recovery adds a further 1.69° improvement.
• Objectness contributes slightly more than repeated 2D masks.
• Combining masks and objectness improves on objectness alone by only 0.12°, a small but consistent gain.

Efficiency:

• DVMNet: 23 ms / pair;
• RelPose++: 29 ms;
• 3DAHV: 35 ms;
• Evaluation hardware: A100.

Hypothesis-based MACs grow with samples from 1k to 500k; DVMNet matches a fixed set of locations for a better error / MAC tradeoff. Its 23 ms excludes box detection and is not a complete absolute-pose pipeline.

9. Interpreting the 3D Voxel Grid

The grid has 3D coordinates and supports rotation alignment, but is not calibrated metric scene geometry:

• No intrinsics;
• No depth;
• Only two RGB images;
• Channel reshaping defines the initial depth axis;
• Cross-view, reconstruction, and pose losses supply 3D consistency.

A precise description is a learned object-centric, equivariant-like 3D latent grid. It supports relative rotation without naturally yielding metric translation. Despite weighted least squares and a 6D rotation representation, experiments therefore report only angular errors.

10. Relationship to Neighboring Approaches

• NOPE enumerates 342 viewpoints and outputs multimodal distributions. DVMNet is faster and hypothesis-free, with a closed-form single solution but no explicit symmetry uncertainty.
• SuperGlue / LoFTR lack object-centric 3D priors under large viewpoint gaps, yielding high error rates.
• RelPose / 3DAHV score rotation hypotheses directly; DVMNet operates through voxel correspondences.
• Oryon combines RGB-D, text, and robust registration to estimate $\Delta R,\Delta t$. DVMNet is lighter but limited to crop-level RGB rotation.

11. Limitations and Reproduction Checks

• Rotation only, rather than full 6DoF.
• Ground-truth boxes in the reported protocol.
• Shared reference / query instance identity is known.
• Symmetric objects receive one rotation, without calibrated multimodal uncertainty.
• Non-metric latent voxels do not directly extend to translation.
• CO3D / GROP splits must strictly exclude test categories or objects.
• Temperature $\tau=0.1$, weight temperature 1.0, and voxel-coordinate normalization affect SVD.
• Training batch 64, AdamW, learning rate $10^{-5}$, and A100 hardware.
• Correctly orthogonalize the 6D rotation representation before computing loss.
• Handle reflection correction for $\det(R)<0$ in SVD.
• Runtime comparisons should explicitly exclude detection.

12. Overall assessment

DVMNet’s strongest result is moving computation from hypothesis space into correspondence space and imposing geometric structure through a differentiable closed-form solver. The CO3D -8.49° gap to 3DAHV, WCV -11.83° gain over no solver, and 23 ms pair latency support this approach.

It remains a controlled relative-rotation module with ground-truth crops, no translation, and no symmetry distribution. Full robotic 6D pose also needs detection, absolute reference pose, and translation estimation. Within CAD-free, single-reference, cross-category rotation, however, DVMNet offers a clear and efficient design.