NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction
NCRF jointly models contact optimization fields, canonical hand–object motion fields, and radiance fields, allowing contact and multiview rendering to correct hand and object poses under occlusion.
known object meshinitial per-frame hand/object posecalibrated camerasper-sequence neural field optimization
Reading Notes
Overview / Deep Dive
Available src/paper-notes/ncrf-3dv2024/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
NCRF: Coarse Reading
One-sentence conclusion
NCRF jointly optimizes hand-object contact, pose refinement, and dynamic radiance fields. Contact constraints improve occluded hand / object poses; corrected poses sharpen canonical-space rendering, avoiding the cycle of building NeRF from wrong poses and treating resulting blur as appearance.
Three modules
Contact optimization field: fuses hand / object features to correct rigid object poses and hand joints.
Hand-object neural radiance field: models hand and object appearances separately.
Observation-to-canonical motion field: maps each frame’s dynamic hand-object deformation into canonical space, with mesh-guided ray sampling reducing color mixing.
Pose correction and neural rendering learn jointly through shared losses; contact maps and collision SDFs supply 3D constraints. Source: original Figure 2. Copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
Results
On HO3D, NCRF raises HumanNeRF PSNR from 26.06 to 29.74; object ADD-0.1D rises from initial 74.5 / S2 Contact 81.4 to 89.8. DexYCB reaches 32.16 PSNR and 83.2 ADD-0.1D. Removing rigid pose correction returns ADD to 74.5, so rendering gains are not merely from a larger NeRF.
The crop contains Figure 4 novel-view rendering comparisons on HO3D / DexYCB, Figure 5 module ablations, and Table 1 PSNR / SSIM / LPIPS rendering metrics. Source: original Figures 4–5 and Table 1. Copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
Boundaries
NCRF performs per-sequence interaction reconstruction, requiring object meshes, initial hand / object poses, and sparse multi-view video. It is not a real-time network directly predicting unknown-object poses from one RGB image.
Available src/paper-notes/ncrf-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
NCRF: Deep Reading
1. Why jointly model contact, pose, and rendering?
Contact regions are also the most mutually occluded parts of hand-object interactions. Small single-frame pose errors cause penetration or floating fingers; warping these poses into canonical space makes radiance fields absorb misalignment as blurred texture and wrong density. NCRF hypothesizes that local geometric contact constraints and global multi-view photometric consistency together stabilize refinement beyond a pose-only optimizer.
2. Module structure
2.1 Contact optimization field
The network fuses hand / object features in observation space to predict hand deformation, rigid-object corrections, and joint-rotation corrections. Corrected poses produce contact maps compared with target contact priors; an SDF penetration loss penalizes object vertices inside the hand.
2.2 Canonical radiance field
Hands are nonrigid and objects approximately rigid. A motion field maps per-frame observation samples into canonical hand / object coordinates. Mesh-guided sampling restricts samples to each mesh’s neighborhood, reducing hand-color contamination of objects at contact boundaries compared with ordinary bounding-box sampling.
Total loss combines NeRF reconstruction, contact consistency, and penetration; pose and rendering are not independent stages. Source: original Figure 2. Copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
DexYCB improves from 27.32/0.9719/27.43 to 32.16/0.9813/20.37.
For pose, HO3D object ADD-0.1D is 89.8, versus initial Hasson et al. 74.5 and S2 Contact 81.4; DexYCB reaches 83.2 versus S2 Contact’s 70.5. HO3D hand MPJPE 8.9 mm is close to S2 Contact’s 8.7: the greatest gains concern object pose and joint rendering, not every hand metric.
Figure 4 red circles mark boundary and contact-region differences in novel-view rendering; Figure 5 shows module ablations; Table 1 reports PSNR / SSIM / LPIPS rendering metrics. Source: original Figures 4–5 and Table 1. Copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
4. What the ablations support
Without rigid pose correction: MPJPE 10.9, ADD 74.5, interpenetration volume 5.1; full model: 8.9, 89.8, 3.7. Removing neural rendering also returns ADD to 74.5, supporting joint pose-correction signals from contact and multi-view appearance.
Without the pose optimization field, rendering PSNR is only 25.71, below 28.10 without mesh-guided sampling and 28.81 without nonrigid motion; the full model reaches 29.74. Pose alignment is a major source of rendering quality.
Tables 3–4 show that the pose field is decisive for ADD and PSNR. Three cameras outperform one / two, while monocular operation remains possible. Source: original Tables 3–4. Copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
5. Limitations
Object meshes, initial per-frame poses, and camera calibration are required for test-time optimization.
Monocular sequences need sufficient hand / object coverage; unseen surfaces otherwise remain unrecoverable.
Lighting changes are not explicitly modeled; color / highlights may enter the radiance field.
Per-frame priors lack explicit temporal consistency, allowing jitter to propagate through long sequences.
HO3D / DexYCB mainly contain known everyday objects; open categories and deformable objects are outside scope.
6. My assessment
NCRF’s main value is turning rendering error into pose supervision, with contact constraints keeping photometric optimization from geometrically wrong solutions. It offers a clear pattern for later Gaussian hand-object tracking: renderable representations, physical / contact priors, and pose correction should share state.
