Paper Reading Notes (Deep): OmniNOCS

1. Key assessment

OmniNOCS identifies three infrastructure barriers to large-vocabulary NOCS:

1. Inconsistent object axes across 3D detection datasets.
2. Sparse or missing dense depth.
3. 2D / 3D boxes without instance masks.

NOCS needs pixel-to-surface correspondence transformed into a consistent object frame; missing any component prevents map construction. Manual axis canonicalization, LiDAR / stereo / NeRF depth, and multiple mask sources complete the chain for 97 categories, 380k images, and over 450k instances.

NOCSformer uses one shared model to lift boxes into masks, NOCS, metric size, pose, and visible shape, without per-dataset / per-class heads or mean sizes. Transfer exceeds Cube R-CNN-style regression but does not eliminate domain supervision; target-trained NOCS remains much stronger for NOCS-Real275 localization.

2. Dataset composition

Ten sources:

Source	Depth	Mask	Images	Categories used
KITTI	LiDAR	Manual	7.4k	7
SUN RGB-D	depth camera	segmentation model	10k	72
Objectron	NeRF	NeRF + manual	132k	9
nuScenes	LiDAR	segmentation model	30k	9
Hypersim	synthetic	synthetic	64k	31
ARKitScenes	depth camera	segmentation model	60k	15
Cityscapes 3D	Stereo	Manual	3.4k	8
Virtual KITTI	synthetic	synthetic	4.1k	3
NOCS-Real275	Depth camera	Manual	7k	6
Waymo Open	LiDAR	Manual	62k	7

After merging / deduplication there are 97 categories and 380k images, not simple row sums. The paper additionally reports:

• 307k real, 73k synthetic;
• 107k outdoor;
• 102k complex indoor;
• 140k object-centric video;
• Over 450k instances.

Scene-type counts are not mutually exclusive exhaustive subsets and should not be forced to sum to all 380k.

3. From heterogeneous detection data to NOCS

3.1 Canonical orientation

Categories need semantically consistent axes: for cars, $X$ points forward and $Z$ upward. Internally consistent sources receive fixed dataset / class rotation offsets. For sources such as Hypersim with inconsistent instance axes, humans select canonical orientation among 6 box-axis arrangements.

Conventions are unavoidable: symmetric objects admit equivalent orientations, and one is chosen arbitrarily. Labels are single-canonical even when the physical problem has multiple solutions.

3.2 Depth augmentation

• Outdoor data prioritizes sparse LiDAR.
• Noisy original Cityscapes depth is replaced with stereo-model depth.
• Objectron lacks dense depth, so per-video NeRFs render it.
• Indoor RGB-D / synthetic data use native / synthetic depth.

NOCS density, boundary noise, and scale-error distributions therefore differ across subsets. Large scale does not imply homogeneous precision.

3.3 Instance mask augmentation

ARKitScenes / SUN RGB-D prompt SAM with projected 3D boxes. Objectron fuses NeRF depth into meshes, then uses pixel-depth-to-surface distances for multi-frame masks and occlusion. Other sources use manual, existing-model, or synthetic masks.

3.4 NOCS definition

Camera-frame point $x_{\mathrm{cam}}$ becomes:

$$x_{\mathrm{nocs}}=\frac{1}{s}\,{}^{obj}T_{cam}x_{\mathrm{cam}},$$

Here $s$ is the tight 3D-box diagonal. Unit-diagonal normalization preserves canonical orientation and relative shape.

One scalar is used, not independent XYZ scaling; metric shape needs separately predicted size.

4. NOCSformer

4.1 Backbone and RoI

• Frozen self-supervised DINOv2 ViT weights.
• Trainable DPT fuses intermediate layers and upsamples 8×.
• RoIAlign extracts 28×28 grids from external boxes.

Freezing reduces optimization difficulty, but heads still need large supervised training; NOCSformer is not a zero-shot foundation model.

4.2 NOCS / mask head

Ten self-attention blocks plus transpose convolution jointly predict NOCS / masks. Each axis predicts logits over 50 nonoverlapping bins, with softmax yielding continuous coordinates rather than direct linear regression.

Shared spatial reasoning for coordinates and foreground extent substantially exceeds small convolutional Mask R-CNN heads at scale.

4.3 Size and learned PnP

Self-attentive size heads use neither class-specific layers nor class-mean sizes. Predicted size converts normalized NOCS to metric coordinates.

Learned PnP predicts:

• Allocentric rotation using a 6D representation.
• The centroid’s 2D projection.

After converting rotation to egocentric coordinates, metric points, centroid rays, and corresponding pixels recover object depth. Classical PnP is discussed but considered more sensitive to noisy NOCS than learned solving.

4.4 Supervision

The total loss includes:

• mask L2;
• NOCS 50-bin cross-entropy + coordinate L1;
• GT-pose-based NOCS reprojection.
• normalized size CE + L1;
• Rotation-matrix L1 and centroid L2.

Cross-dataset zero-shot means only that target NOCS-Real275 was held out; other data across 97 categories densely supervise NOCS / pose.

5. Beyond 3D boxes

Beyond cuboid regression, NOCSformer also provides:

• canonical dense visible-surface coordinates;
• instance mask;
• metric visible shape;
• oriented 3D box / pose.

These clouds cover currently visible surfaces, not complete watertight reconstructions; occluded backs are not generated.

6. Direct NOCS metrics

NOCS mAE / mPSNR use GT / predicted-mask intersections; separate mask mIoU discourages sparse predictions scoring highly only on easy regions.

Test set	Method	NOCS mAE↓	NOCS PSNR↑	Mask IoU↑
NOCS-Real275	target-trained NOCS baseline	0.121	16.345	86.10
NOCS-Real275	transferred NOCSformer	0.107	18.527	89.03
OmniNOCS 75-class subset	NOCSformer	0.094	20.245	78.50

Held-out NOCS-Real275 dense coordinates / masks exceed the old target-trained model, but this does not imply superior final localization, as the next section shows.

7. Localization and transfer

7.1 nuScenes subset

Method	Multidataset	mATE m↓	mAOE rad↓	mASE↓	mIoU↑
EProPNP + TTA	No	0.653	0.319	0.255	—
Cube R-CNN	Yes	0.650	1.305	0.283	0.349
Cubeformer	Yes	0.790	0.573	0.301	0.280
NOCSformer	Yes	0.887	0.558	0.291	0.377

Canonical orientation / overlap exceed the other two multidataset baselines, but translation is worst. Target-trained EProPNP + TTA remains clearly better in mATE / mAOE.

7.2 Held-out NOCS-Real275

Method	Target-domain training	Depth	3D IoU mAP@25	mAP@50
NOCS model	Yes	Yes	79.6	72.4
Cube R-CNN	No	No	14.9	4.1
NOCSformer	No	No	43.5	10.6

Gains over transferred Cube R-CNN are large, but gaps to target-trained NOCS remain 36.1 / 61.8 points. The steep @25-to-@50 drop suggests stronger coarse-localization than precision-pose transfer.

Cube R-CNN also detects boxes, while NOCSformer receives them; detection false positives / negatives affect the former’s mAP. The paper acknowledges unequal system comparisons.

7.3 Canonical orientation

On five KITTI-val classes, gravity-axis accuracy@5° is 99.95% and heading-axis accuracy@10° 59.65%, versus Cube R-CNN 18.29% / 21.55%. This supports unified canonical axes over Chamfer-invariant cuboid supervision for stable orientation.

8. Ablations and limitations

The full head reaches 20.431 dB NOCS PSNR and 81.69 mask IoU:

• Hourglass head: -3.064 dB / -3.16 points;
• Mask R-CNN head: -5.23 dB / -21.27 points;
• Replacing 50-bin discretization with continuous regression: -4.11 dB.

Explicit paper limitations:

• Multiple equivalent canonical frames make single-label symmetry losses unnatural.
• Mirrored geometry such as left / right shoes requires reflection handling.
• Boxes must come from external detectors or humans.

Additional site observations:

• Depth / mask noise differs by source, without annotation-source error stratification.
• Manual canonicalization reaches 97 categories, with unresolved cost / semantic disputes for larger open vocabularies.
• Sharing 97 categories does not establish unseen-category open vocabulary.
• NOCS clouds cover visible surfaces only.
• No end-to-end detector + NOCSformer speed, memory, or throughput.
• Precise target localization still relies substantially on domain-specific training and depth.

9. Reproduction checklist

• Obtain each source’s licenses, versions, and train / validation splits.
• Fix class ontology, axes, and class-specific rotation offsets.
• Record manual 6-way canonicalization protocol and agreement.
• Reproduce valid pixels / scale for LiDAR, stereo, NeRF, and camera depth separately.
• Save mask sources; audit manual / SAM / model / synthetic subsets.
• Compute NOCS with tight-box diagonal scalar $s$.
• Freeze DINOv2; train DPT / heads, checking 8× upsampling and 28×28 RoIs.
• Reproduce 10-block, 50-bin joint NOCS / mask heads.
• Separate input boxes from detector predictions.
• Report NOCS mAE / mPSNR / mask IoU, 3D boxes, and canonical orientation.
• Separate held-out transfer from target-trained baselines; better transfer does not imply better target-domain performance.

10. Overall assessment

OmniNOCS contributes data engineering and task definition beyond a model increment, expanding NOCS to 97 categories, multiple cameras, indoor / outdoor scenes, and object-centric videos. Direct metrics isolate coordinate quality from final pose.

NOCSformer shows shared canonical transfer to unseen domains, exceeding transferred Cube R-CNN and old target-trained NOCS coordinate heads. Precise localization remains unresolved, with large @50 gaps to target supervision. The key lesson is that large-vocabulary pose first needs unified frames, annotation sources, and evaluation boundaries.