Paper Reading Notes (Deep): NeuSurfEmb

1. Key assessment

NeuSurfEmb contributes object-onboarding engineering rather than one network module. It connects reference-camera localization, object masking, 3D modeling, and training-image generation into a semiautomatic pipeline, enabling methods such as SurfEmb that need object surfaces and abundant synthetic data to operate without CAD.

The authors claim that few real RGB images suffice to train accurate estimators, exceeding CAD-free Gen6D and OnePose++ on custom objects. Experiments support this, while revealing costs: modeling takes tens of minutes, but per-object dense-correspondence training still takes 1–1.5 days. It removes specialist CAD / PBR production, not instance-specific training.

2. Task setting and inputs / outputs

Object onboarding

• Capture RGB video or reference images around a stationary object.
• A user supplies only the first-frame bounding box.
• Intrinsics are known or recovered by the SfM pipeline.
• Outputs are reference poses, per-frame masks, instance NeuS2, and a trained SurfEmb model.

Query stage

• One RGB image and an external detector’s object box.
• Instance-level 6D pose from object frame to query-camera frame.
• Optional query depth supports depth-based refinement.

Metric CAD is unnecessary, but monocular SfM / NeuS2 has scale ambiguity. In custom evaluation, ICP between two NeuS2 point clouds supplies scale and frame transforms from the modeling scene to the Vicon scene. Robot deployment still needs an additional metric-scale calibration source.

3. Complete pipeline

3.1 SfM and semiautomatic masks

SfM recovers reference poses. After the first-frame box is supplied:

1. Segment Anything generates the object mask within the box.
2. MixFormer propagates the box to later frames.
3. Segment each frame again and tighten its bounding box.
4. Obtain masked images aligned with SfM poses.

One user input is sufficient, but tracking drift or SAM leakage contaminates subsequent NeuS2 views. Mask errors are not systematically isolated experimentally; save per-frame masks for visual inspection during reproduction.

3.2 NeuS2 object representation

Train NeuS2 once with reference poses / masks, extract a surface cloud, and fit an oriented bounding box. Recenter the object frame, align principal axes, transform camera poses, and retrain NeuS2.

Reframing:

• Concentrates surfaces in a stable canonical range.
• Centers random-view sampling on objects rather than arbitrary SfM origins.
• Establishes consistent coordinates for SurfEmb 3D key features.

NeuS2 exports Marching Cubes meshes or directly renders coordinates. Ablations find that NeuS2 versus CAD geometry and NeuS2 versus GL rendering affect pose less than training-image domain differences.

3.3 Automatic training-data generation

Random camera poses render NeuS2 RGB, masks, and 3D surface coordinates together. Isolated neural-rendered foregrounds receive online:

• Random-noise or PASCAL-VOC backgrounds.
• Cut-and-paste with random foreground coverage.
• Color, white-balance, and affine augmentation.
• Simulated occlusion and background changes.

Each object gets 10,000 images at 224×224. This avoids materials, lighting setup, and BlenderProc PBR configuration, while leaving a substantial gap to complex real illumination.

3.4 SurfEmb dense correspondence and pose

SurfEmb learns two feature fields:

• Query CNN features $f_q(p)$ for each foreground pixel.
• A SIREN key network maps arbitrary surface point $x$ to $f_k(x)$.
• InfoNCE attracts true 2D / 3D pairs and separates other surface points.
• The query network also predicts masks with cross-entropy supervision.

Uniform surface keys and query pixels form a similarity matrix for probabilistic 2D–3D sampling. PnP+RANSAC generates pose candidates; coordinate-renderer refinement improves the best. Optional depth adds further refinement.

4. Experimental protocol

4.1 LINEMOD-Occlusion

• 8 known objects with substantial occlusion.
• BOP Average Recall (ARBOP), combining symmetry- and occlusion-related errors.
• Separate NeuS2 and correspondence models per object.
• CosyPose query boxes, aligned with SurfEmb evaluation.

Resources on one RTX 2080 Ti:

Stage	Setting / duration
NeuS2	20,000 steps, about 10 minutes
Data generation	10,000 images at 224×224, about 20–30 minutes
SurfEmb correspondence learning	50 epochs, about 1–1.5 days / object

4.2 Custom real objects

Bluebox, extinguisher, greybox, helmet, and kettle cover symmetric boxes, low texture, complex geometry, and elongated structures. Ordinary phones capture modeling video; a FLIR Firefly S captures evaluation, with Vicon camera-motion GT.

Each object appears alone in unoccluded scenes and with another object in occluded scenes. Modeling and evaluation differ substantially in background, lighting, camera color, and occlusion.

5. Results and ablations

5.1 Main LINEMOD-Occlusion results

Configuration	RGB ARBOP	RGB-D ARBOP
NeuSurfEmb: NeuS2 renderer + NeuS2 model + NeuS2 images	0.554	0.666
NeuSurfEmb: GL renderer + NeuS2 model + NeuS2 images	0.570	0.681
NeuSurfEmb: GL renderer + NeuS2 model + PBR images	0.646	0.752
SurfEmb: GL renderer + CAD + PBR	0.656	0.758
CosyPose	0.633	0.714

The key ablations go beyond whether NeuS2 replaces CAD:

• Switching the training object model from NeuS2 to CAD changes little.
• Switching pose-estimation coordinate rendering from NeuS2 to CAD also changes little.
• Replacing NeuS2 images with PBR raises RGB ARBOP from about 0.57 to 0.646.

NeuS2 geometry thus supports correspondence / PnP adequately; neural-rendering versus real-query appearance / occlusion domains are the main bottleneck.

For 5 of 8 LM-O objects, NeuS2 forward Chamfer distance is about 1.2 mm or less. Can, eggbox, and holepuncher fare worse, including failures from internal holes and bottom structures absent from reference views.

5.2 Custom-object results

Unoccluded scenes:

Method	Mean ADD(-S)	Mean 5 cm, 5°
NeuSurfEmb	0.982	0.825
OnePose++, original tracking	0.683	0.510
OnePose++, no tracking + improved recrop	0.766	0.586
Gen6D, original tracking	0.550	0.279
Gen6D, no tracking	0.606	0.185

NeuSurfEmb clearly benefits low-texture helmet, extinguisher, and kettle under occlusion. Gen6D sometimes tracks foreground occluders; OnePose++ recropping can remove important elongated-extinguisher regions. For fairness, all methods receive GT boxes, with original baseline tracking reported separately.

Treat symmetric boxes carefully: ADD-S accepts wrong symmetry-equivalent orientations, so 5 cm, 5° is also reported. Greybox ADD-S is near perfect while systematic rotation errors remain from some viewpoints.

6. Authors’ claims and site assessment

Level	Conclusion
Authors’ claim	Few real RGB images and a semiautomatic process replace CAD / manual PBR and train accurate 6D estimators
Stronger evidence	NeuS2 / CAD swaps isolate geometry from the main bottleneck; five custom objects demonstrate operation under substantial domain shifts
Qualification	Few input images do not mean low total cost: 1–1.5 days of correspondence training per object and external real-scale registration remain
More precise positioning	CAD-free, PBR-pipeline-free instance-specific onboarding, rather than training-free novel-object pose

7. Failure modes and limitations

1. Unseen surfaces: NeuS2 cannot recover bottoms / holes absent from capture.
2. Mask contamination: initial segmentation or subsequent tracking errors enter surfaces and rendered training sets.
3. Appearance gap: cut-and-paste does not fully replace material / illumination PBR; this is the largest experimental performance gap.
4. Per-object training: fast object representations do not imply fast SurfEmb training.
5. Scale / frames: monocular reconstruction has relative scale; robotics needs known dimensions, markers, depth, or external registration.
6. Detection dependence: supplied boxes exclude actual detector failures from pose scores.

8. Reproduction and engineering checklist

• Save SfM clouds, trajectories, and per-frame masks to inspect drift.
• Check bounding-box principal axes after first NeuS2 training, avoiding arbitrary flips on near-symmetric objects.
• Save meshes and coordinate renders separately; measure surface completeness.
• Fix viewpoint distribution, foreground size, backgrounds, and augmentation for the 10,000 images.
• Reproduce SurfEmb surface sampling, InfoNCE temperature, RANSAC iterations, and refinement.
• Calibrate metric scale and object-to-grasp transforms separately before robot deployment.
• Report symmetry-aware ARBOP and orientation-sensitive angular thresholds together.

9. Future directions

• Shared foundation correspondence models instead of 1–1.5 days per-object training.
• More realistic rendering through Gaussian Splatting or editable materials.
• Reconstruction uncertainty to guide active recapture of unseen surfaces.
• Joint scale estimation and reconstruction, connecting phone onboarding directly to robot frames.
• Propagate SfM, mask, NeuS2, and pose-residual uncertainty to final pose confidence.

10. Overall assessment

NeuSurfEmb makes CAD-free onboarding executable and uses ablations to locate engineering bottlenecks. NeuS2 geometry is adequate; image domains and per-object correspondence learning are the next targets. It is a strong CAD-free instance-pose baseline and a reminder to distinguish model-free, training-free, and scale-free claims.