Fit-NGP: Fitting Object Models to Neural Graphics Primitives
Fit-NGP scans a static scene with a wrist-mounted RGB camera, trains Instant-NGP, and fits known CAD surfaces directly to the density field. Joint camera-pose optimization and normal-neighborhood sampling are key to median errors of 1.6 mm / 3.3°.
A wrist-mounted RGB camera scans a tabletop scene for Instant-NGP reconstruction; known CAD models are then fitted directly to its density field. Multiview information becomes an optimizable continuous scene representation without per-frame depth sensing.
3. Method overview
Train Instant-NGP using RGB views posed by robot kinematics.
Obtain target instance masks in a reference view through open-vocabulary segmentation.
Render NeRF depth to generate multiple initial pose hypotheses.
Sample CAD surfaces and nearby normals to maximize transformed-model / density consistency.
Retain multiple hypotheses and refine camera poses in Instant-NGP to compensate hand–eye and encoder errors.
Original Figure 2: multiview RGB and instance segmentation initialize and optimize models inside a density field. Source: paper PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.
4. Main results
Custom real data: 4 scenes and 43 object instances.
Full-system medians: 1.6 mm translation and 3.3° relative rotation error.
Without camera-pose optimization: 17.3 mm / 49.4°.
Approximately 20 well-covering views are empirically needed for a density field supporting fitting.
5. Assessment
This is a known-model, active-scan, static-scene robot system rather than single-frame RGB pose estimation. Multiview wrist-camera evidence achieves millimeter-scale median errors, at the cost of scanning, scene-field training, instance masks, and sensitivity to weak-texture NeRF geometry.
Available src/paper-notes/fit-ngp-icra2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Deep Reading Notes: Fit-NGP: Fitting Object Models to Neural Graphics Primitives
1. Basic information
Authors / affiliation: Marwan Taher, Ignacio Alzugaray, Andrew J. Davison; Imperial College London, Dyson Robotics Lab
Robot-acquired RGB sequences train an Instant-NGP density field; known CAD surfaces query this continuous representation to optimize each object’s 6D pose, rather than regressing from one image.
3. Task boundaries
Input / assumption
Requirement
Known CAD and scale
Required
Static tabletop scene
Required
Multiple posed RGB views
Required, from wrist-camera scanning
Depth camera
Not required
Reference-frame instance masks
Required; Detic in the paper
Real-time single-frame output
Outside scope
It suits initial setup of a new static scene before manipulation, rather than dynamic targets or single-image benchmarks.
Original Figure 2: Instant-NGP reconstruction, reference segmentation, hypothesis initialization, and model–density optimization. Source: paper PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.
4. Method breakdown
4.1 Aggregate multiview evidence with a neural field
Robot kinematics and hand–eye calibration initialize moving wrist-camera poses. Instant-NGP learns color c(x,d) and density σ(x). Unlike direct RGB-D cloud fusion, the field is continuous, differentiable, and queryable at arbitrary positions.
4.2 Initialize hypotheses from a reference view
Detic supplies reference masks. Rendered NeRF depth back-projects masked pixels into a nearby 3D region for multiple translation / orientation hypotheses. Initialization only needs at least one candidate inside the subsequent optimization basin.
4.3 Fit model surfaces to density
For CAD surface samples XM and pose T, one might simply maximize σ(Tx). But density can form thick shells or internal peaks. Sampling both surfaces and positive / negative normal offsets favors high surface density with lower nearby density, constraining geometry better than attraction to maxima alone.
Optimize hypotheses in parallel and choose by density score. Evaluation accounts for rotational symmetry to avoid penalizing equivalent orientations.
4.4 Camera-pose refinement is essential
Kinematic, encoder, and hand–eye errors distort NeRF geometry. Instant-NGP camera-pose optimization jointly aligns images and extrinsics. Removing it nearly defeats fitting in ablations: accuracy depends on scene-coordinate calibration as well as object losses.
5. Experiments
Evaluation uses 4 custom tabletop scenes and 43 instances, including bolts, nuts, washers, containers, and cutlery at different sizes. Metrics are symmetry-adjusted relative rotation / translation errors.
Setting
Median translation error
Median rotation error
Full system
1.6 mm
3.3°
No camera-pose optimization
17.3 mm
49.4°
No normal-offset sampling
43.2 mm
70.1°
The 1.6 mm is a median across 43 instances, not millimeter accuracy for every instance or a replacement for success-threshold curves.
Original Figures 4–6: successes, a low-texture cup failure, and view-count effects on density. Source: paper PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.
5.1 View count and coverage
Approximately 20 views are an empirical lower bound for usable density. At equal counts, broadly distributed views covering multiple object sides outperform narrow-angle trajectories. Active sensing should prioritize baseline / angular coverage rather than frame count alone.
Original Figures 7–8 / Table I: dense local versus sparse broad-coverage views, and ablations of camera refinement / normal sampling. Source: paper PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.
6. Failure modes and critical analysis
Weak texture / specularity: incorrect NeRF density leaves even accurate CAD without a valid fitting surface.
Initialization: masks, rendered depth, or separation errors can put every hypothesis too far from GT.
Static-scene assumption: object motion during scanning breaks the shared field.
System time: scanning, NeRF training, segmentation, and hypothesis optimization do not suit dynamic control loops.
Data scale: 4 custom scenes do not represent warehouse bin-picking clutter / occlusion.
Model requirements: RGB-only sensing still needs high-quality metric CAD and object-category lists.
7. Extensions
Choose next views to reduce pose-hypothesis entropy rather than only improve rendering; combine density uncertainty, semantic fields, and CAD edges / normals. Fast Gaussian representations, incremental updates, and local object fields could reduce reconstruction cost in dynamic robotics.
