Deep Reading Notes: SUP-NeRF

1. Key assessment

SUP-NeRF starts from monocular NeRF’s unreliable pose optimization:

• Shape codes can scale objects;
• Translation changes depth;
• Both compensate in one view while reducing photometric loss;
• Beyond about 25° initial rotation error, NeRF gradients rarely reach the correct basin.

The authors therefore reorder the tasks:

$$\text{predict dimensions} \rightarrow \text{fix scale and refine pose} \rightarrow \text{NeRF reconstruct shape / texture}.$$

Results support this: nuScenes feed-forward gives 7.01° / 0.68 m, versus 7.07° / 0.73 m after 50 NeRF steps. Projected-box refinement handles pose; NeRF mainly handles visible appearance.

2. Inputs, supervision, and detector independence

Inputs include:

• One object RGB crop;
• occlusion / occupancy mask;
• camera intrinsics;
• category-specific learned prior.

No requirement for:

• Test-instance CAD;
• External monocular 3D detector initializations such as FCOS3D.

Mask R-CNN masks / ROIs remain necessary. Since nuScenes lacks masks, predicted instances are matched to 3D boxes. Training also uses:

• Box dimensions and poses;
• GT camera views for NeRF decoding;
• projected 3D box corners pose loss.

Thus avoiding external 3D detectors does not mean raw-RGB-only inference without detection or 3D supervision.

3. Unified encoder and object NeRF

Inputs are padded / resized to 128×128. Encoder outputs:

• 256D shape code;
• 256D texture code;
• pose code;
• Dimensions $H_B,W_B,L_B$.

CodeNeRF samples 64 points per ray, rendering only 32×32 RGB / occupancy for losses. Shape space is normalized by observed box diagonals.

Training loss:

$$\mathcal L_{\rm train} =\mathcal L_{\rm rgb} +w_{\rm occ}\mathcal L_{\rm occ} +w_{\rm pose} \left( \mathcal L_{\rm pose}^{\rm direct} +\sum_t\mathcal L_{\rm pose}^{(t)} \right),$$

Pose loss averages distances between eight projected and GT corners. Training uses 40 epochs, learning rate $10^{-4}$, and $w_{\rm occ}=0.1,w_{\rm pose}=0.01$.

NeRF trains with GT camera poses rather than pose-branch predictions. This stabilizes decoding but weakens reconstruction-to-pose feedback during training; the branches connect sequentially at inference.

4. Projected-box pose module

Projected corners encode:

• Image center / extent;
• yaw / viewpoint;
• Depth–size projection relationships;
• A current-pose state comparable with image evidence.

Axis-angle updates compose on $SO(3)$. Translation uses image updates instead of directly regressing camera-frame $\Delta x,\Delta y,\Delta z$:

$$x^{t+1}=x^t+v_x^t,\quad y^{t+1}=y^t+v_y^t,\quad Z^{t+1}=\rho^t Z^t.$$

Image offsets / depth ratios stabilize update magnitude across focal lengths and distances. Camera invariance here means intrinsics enter explicit projection / backprojection, reducing raw-parameter dependence rather than proving full invariance.

5. Random initialization and iterations

Training / inference avoid external 3D pose initialization:

• Random 2D centers inside ROIs;
• Fixed initial depth $Z=20$;
• Full-range random yaw;
• Extra random-axis rotation $(-20°,20°)$;
• Three pose iterations.

The capture range exceeds NeRF gradients, but mask-provided ROIs constrain the search; it is not unconditional full-frustum localization.

Three feed-forward updates optionally precede 50 inference-optimization steps:

$$\mathcal L_{\rm infer} =\mathcal L_{\rm rgb} +w_{\rm occ}\mathcal L_{\rm occ},$$

Freeze weights; update shape / texture / pose. Shape / texture step is 0.02, pose step 0.01.

6. In-domain and cross-domain results

nuScenes

Method	RE FF / 50it	TE FF / 50it	PSNR FF / 50it	DE FF / 50it
AutoRF+FCOS3D	9.77 / 10.17°	0.85 / 0.78 m	7.5 / 17.2	1.34 / 0.81 m
SUP-NeRF	7.01 / 7.07°	0.68 / 0.73 m	10.5 / 18.8	0.69 / 0.60 m

Pose is already estimated feed-forward: NeRF gives PSNR +8.3 and DE −0.09 m, but RE +0.06° and TE +0.05 m.

KITTI, nuScenes model zero-shot

• SUP-NeRF: 6.79° / 1.06 m;
• AutoRF+FCOS3D: 11.95° / 2.20 m.

Rotation decreases 43.2%, translation 51.8%, the strongest setting behind the approximate 50% claim.

Waymo, nuScenes model zero-shot

• SUP-NeRF: 10.01° / 1.68 m;
• AutoRF+FCOS3D: 6.65° / 3.22 m.

Translation decreases 47.8%, while rotation increases 50.5%. The authors suggest FCOS3D’s larger context helps heavy occlusion. KITTI’s roughly halved rotation / translation do not generalize to Waymo.

Cross-dataset PSNR / DE trail nuScenes substantially, showing weaker transfer for category-NeRF appearance / geometry priors.

7. Ablations and NeRF’s actual role

Pose-only Table 5:

Module	nuScenes RE / TE	KITTI RE / TE	Waymo RE / TE
MLP direct	31.96° / 5.53m	37.58° / 5.31m	30.89° / 10.10m
Corners + PnP	24.79° / 2.82m	35.10° / 3.55m	93.76° / 18.68m
FCOS3D	9.77° / 0.85m	11.95° / 2.20m	6.65° / 3.22m
SUP-NeRF	7.01° / 0.68m	6.79° / 1.06m	10.01° / 1.54m

Projected-corner PnP fails completely on Waymo, supporting learned iterative updates over hard keypoint / PnP solving.

GT initialization is more revealing:

• 50 NeRF steps move GT poses from 0° / 0m to 2.11° / 0.13m;
• SUP initialization moves from 7.01° / 0.68m to 7.07° / 0.73m.

Photometric objectives and pose metrics disagree; reconstruction refinement does not reliably refine poses.

8. Speed

One A5000:

Mode	Time
3-step feed-forward	0.018 s
FF + 20it	0.608 s
FF + 50it	1.493 s

Real-time potential applies to feed-forward poses / coarse latents; high-quality full NeRF reconstruction gives about 0.67 FPS.

9. Limitations

• Primarily driving cars rather than general multicategory objects;
• Masks / crops, intrinsics, box labels, and GT camera-pose supervision;
• ROI-constrained random initialization;
• GT-pose NeRF training versus predicted-pose inference mismatch;
• Possible pose degradation from 50 photometric steps;
• Worse Waymo rotation than FCOS3D;
• Weak cross-domain reconstruction;
• Sparse LiDAR depth evaluation rather than full-surface accuracy;
• Limited dynamic / occlusion / mask-error breakdowns;
• 1.493 s latency at 50 steps;
• Public code with substantial environment / data preparation.

10. Reproduction checklist

• Match 87,048 / 5,000 nuScenes object splits and Mask R-CNN masks;
• Restrict categories / IDs to cars;
• 128×128 encoder, 32×32 render, 64 ray samples, 256D codes;
• Reproduce random ROI centers, $Z=20$, yaw, and random axes;
• Fix three pose updates;
• Train NeRF with GT poses and compare a predicted-pose variant;
• Separate FF, 20-step, and 50-step results;
• Report RE / TE alongside PSNR / DE rather than inferring pose gains from rendering;
• Avoid KITTI / Waymo fine-tuning;
• Include mask detection / cropping in full-pipeline timing;
• Test intrinsics, masks, occlusion, and dimension errors.

11. Overall assessment

SUP-NeRF clarifies that monocular photometric optimization can explain images through wrong scales / depths and cannot handle broad pose recovery reliably. Fixed dimensions and three projected-box updates do the effective pose work; KITTI 6.79° / 1.06 m and 18 ms support engineering value.

NeRF refinement does not improve pose, Waymo rotation trails FCOS3D, and full reconstruction takes 1.493 s. The method is an efficient category-pose initializer embedded in an object-NeRF reconstruction system, rather than pose solved by NeRF alone.