Paper Reading Notes (Deep): LaPose

1. Key assessment

LaPose separates two problems in RGB-only category pose:

• Shape ambiguity: a pixel’s NOCS need not be a deterministic point; predicted variance identifies unreliable correspondences;
• Metric scale ambiguity: the same image size can represent a small nearby or large distant object; gradients from this unobservable quantity should not corrupt rotation / normalized translation.

The effective combination is dual-stream supervised uncertainty + learned PnP + scale normalization. DINOv2 alone is weak: NIoU50 is 26.9 versus ConvNet’s 39.7. General features need category-supervised NOCS integration.

2. Inputs, Outputs, and Supervision

LaPose estimates category-level 9DoF for six NOCS categories:

$$(R,t,s),\quad R\in SO(3),\;t\in\mathbb R^3,\;s\in\mathbb R^3.$$

Inputs are RGB crops, Mask R-CNN detections, and intrinsics. Training supervision includes:

• visible mask;
• per-pixel GT NOCS map;
• GT rotation / translation / metric size;
• category average normalized size.

This is supervised category pose, not zero-shot use of frozen DINOv2. Under the NOCS protocol, REAL275 experiments use CAMERA25 and REAL training data together, not synthetic-only sim-to-real transfer.

3. Dual-Stream Laplacian Mixture

DINOv2 acts as a generalized 3D-information stream, offering patch semantics relatively consistent across instances and poses. The ConvNet learns local shape conventions for the six NOCS categories.

Each stream predicts per-visible-pixel NOCS mean $\mu$ and variance $\sigma^2$ using a Laplace aleatoric loss:

$$\mathcal L_{\rm 3D}= \frac{\lambda}{\sigma^2} \|M_{\rm vis}\odot(C^{3D}_{gt}-\mu)\|_1 +M_{\rm vis}\odot\log\sigma^2.$$

For a large residual, increasing variance reduces the first term, while the log term prevents unlimited variance. The paper calls variance self-supervised; more precisely, this is supervised heteroscedastic learning without variance labels, still driven by GT NOCS residuals.

The two distributions form an LMM, but mixture weights are not learned and inference performs no explicit sampling / marginalization. The solver actually receives:

$$(\mu_{\rm dino},\sigma^2_{\rm dino}, \mu_{\rm conv},\sigma^2_{\rm conv},C^{2D}).$$

The LMM thus provides parameterized evidence from two heteroscedastic experts rather than explicitly solving an interpretable mixture posterior.

4. PnP as a Learned Approximation

The theoretical objective is expected L1 reprojection under the NOCS distribution:

$$P^*=\arg\min_{P\in SE(3)} \frac1N\sum_i \mathbb E_{x_i\sim\eta_i} \|\pi(KPx_i)-u_i\|_1.$$

The authors find direct optimization difficult and use a convolutional network $\Phi$ similar to prior work:

$$(R_{\rm out},t_{\rm out}) =\Phi(\mu_d,\sigma_d^2,\mu_c,\sigma_c^2,C^{2D}).$$

Inference runs neither EPnP / RANSAC nor explicit minimization of that objective. LMM-based PnP is a pose-regression network conditioned on correspondence distributions. It is fast and end-to-end, with these costs:

• No guarantee of geometric optimality / projection consistency;
• Potential category-appearance shortcuts in the solver;
• Retraining may be needed for new cameras, crop conventions, or categories;
• Inverse-uncertainty use of predicted variance is supported only indirectly through ablations and visualizations.

Rotation predicts the first two matrix columns $r_x,r_y$ and orthogonalizes them. Axial-symmetric categories such as bowl / can supervise only $r_y$, avoiding forced single-valued regression of unidentifiable yaw.

5. Scale-Agnostic Pose

For true 3D-box size $s=(s_x,s_y,s_z)$, the diagonal is:

$$d=\sqrt{s_x^2+s_y^2+s_z^2}.$$

LaPose uses:

$$s_{\rm norm}=s/d,\qquad t_{\rm norm}=t/d.$$

The size head regresses $s_{\rm norm}-s_{\rm avg}$. Translation is split into projected center $(o_x,o_y)$ and normalized depth $t_z/d$, then converted into crop-relative translation / scale-invariant $(\delta_x,\delta_y,\delta_z)$ and back-projected with intrinsics.

This gives near-small and far-large cases with the same image a shared, identifiable normalized target. Metric robot translation still requires $d$. A separate MobileNet predicts box diagonal, then:

$$t=d\,t_{\rm norm},\qquad s=d\,s_{\rm norm}.$$

SAP is useful factorization, not new physical information resolving monocular scale. Normalized metrics measure shape ratios / orientation / relative translation; absolute metrics assess actual metric outputs.

6. REAL275 Main Results

Scale-agnostic:

Method	NIoU25	NIoU50	NIoU75	10°0.2d	10°0.5d	10°
DMSR	57.2	38.4	9.7	26.0	44.9	36.9
LaPose	70.7	47.9	15.8	37.4	57.4	60.7
LaPose (M)	66.4	48.8	20.5	39.7	55.4	60.2

For NIoU25 / 10°0.5d, the single model exceeds DMSR by +13.5 / +12.5 points; per-category models gain +10.8 at strict NIoU75. The clearest gain is rotation-only 10°, from 36.9 to 60.7.

Absolute-scale:

Method	IoU25	IoU50	IoU75	10°10cm	10cm
DMSR	37.4	16.3	3.2	25.2	40.0
LaPose	41.2	17.5	2.6	30.5	44.4
LaPose (M)	40.2	18.3	4.1	27.7	43.7

Absolute-scale advantages shrink markedly, and single-model IoU75 trails DMSR, showing the independent diagonal predictor remains a bottleneck. All baselines are recomputed with a corrected NOCS evaluation script; these scores should not be merged directly with old tables using the uncorrected script.

7. Evidence for Uncertainty Calibration

Visualizations show spatial association between variance and error, but the paper does not report:

• NLL;
• expected calibration error;
• sparsification / risk-coverage curve;
• Quantitative correlation between variance quantiles and NOCS error;
• Calibration under OOD / detection errors.

Variance can therefore be described as useful for pose, without claiming well-calibrated probabilistic uncertainty.

8. Ablation Breakdown

Configuration	NIoU25	NIoU50	10°0.2d	10°0.5d
Conv + SAP	60.3	39.7	29.0	46.1
Conv, no SAP	37.8	17.6	10.5	33.2
DINO + SAP	61.7	26.9	16.9	38.5
Conv + DINO + SAP	64.9	42.1	32.5	52.8
Conv + Laplace + SAP	65.5	43.4	33.6	51.2
Conv + Gaussian + SAP	59.1	37.6	28.3	44.7
Dual-stream LMM + SAP	70.7	47.9	37.4	57.4

SAP is the largest individual change, but it alters target parameterization and evaluation space together; the +22.5 NIoU25 cannot all be attributed to network capability.

Dual deterministic streams add +4.6 NIoU25 / +6.7 10°0.5d over Conv-only; dual Laplace adds another +5.8 / +4.6. Gaussian’s negative effect shows that adding uncertainty heads is not automatically useful; heavy-tailed Laplace better matches L1 NOCS residuals.

9. Limitations and Common Misreadings

• Scale-agnostic metrics deliberately remove metric-scale error; they do not establish full 9DoF;
• Absolute scale still comes from a monocular MobileNet appearance prior;
• Learned PnP has no analytic geometry guarantee and is not RANSAC-PnP;
• Variance is supervised through GT NOCS residuals, not unsupervised uncertainty;
• Only six NOCS categories are covered, and REAL275 test has just 2.7K images;
• REAL275 training uses REAL train + CAMERA, not zero-shot real transfer;
• Detection comes from external Mask R-CNN / prior detections;
• Symmetry uses category-level axis conventions, without covering instance textures that break geometric symmetry;
• About 10 FPS lacks an end-to-end module breakdown;
• Multi-model rows use independent category networks, with greater capacity / deployment cost than one shared model.

10. Reproduction Checklist

• Use the authors’ corrected NOCS evaluator;
• Report shared and per-category models separately;
• Distinguish CAMERA-only from CAMERA+REAL training;
• Use identical external detections;
• Keep DINOv2 frozen and ConvNet trainable;
• Train both $\mu,\sigma^2$ sets inside visible masks;
• Compare learned PnP against analytic EPnP / uncertainty-weighted RANSAC;
• Report normalized and absolute metrics together;
• Evaluate relative diagonal $d$ error separately;
• Add uncertainty calibration / risk-coverage evaluation;
• Include detection, DINO, PnP, and the MobileNet scale head in runtime.

11. Overall assessment

LaPose makes a useful statistical and geometric decomposition of RGB-only category pose: NOCS pixels need not be equally reliable, and inherently unobservable absolute scale need not dominate pose training. REAL275 normalized rotation and joint-threshold gains are substantial, with relatively complete dual-stream Laplace ablations.

The key boundary is that a learned network consumes the LMM, rather than explicit probabilistic PnP, and SAP delegates metric scale to a separate network. The method learns pose apart from scale more stably; it does not uniquely recover true physical size from one RGB image. Read normalized and absolute tables together to assess it accurately.