Few-View Object Reconstruction with Unknown Categories and Camera Poses
FORGE uses 3D voxel features for both cross-view pose correlation and neural volume reconstruction. Rendering losses further jointly correct sparse-view camera poses at test time.
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
FORGE: Coarse Reading
One-sentence conclusion
FORGE jointly recovers relative camera poses and a renderable 3D representation from approximately five RGB images with unknown camera poses and unknown object categories. Its key idea is to use 3D voxel features for cross-view correlation, pose estimation, and NeRF-style rendering together, rather than estimating pose first and reconstructing afterward.
Method
A 2D encoder lifts each image into 3D voxel features; global and pairwise feature branches predict relative poses. These predictions align the views’ voxels, and a fusion module produces a unified neural volume whose volume rendering reconstructs input / novel views. At test time, 1K iterations of photometric optimization further refine the poses.
3D features provide both pose cues and a reconstruction representation, allowing the tasks to constrain each other. Source: original Figure 2. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
Results
On novel Kubric categories, FORGE achieves rotation / translation errors of 8.9°/0.23, compared with 23.9°/0.49 for 8-Point Transformer; novel-view PSNR with predicted poses is 25.57. On OmniObject3D, the errors are 5.35°/0.30 with reconstruction PSNR 26.56. The paper also demonstrates zero-shot transfer from synthetic Kubric training to real GSO objects.
The upper section compares five-view reconstruction and the lower tables compare poses. FORGE's test-time optimization further reduces the reconstruction gap between predicted and ground-truth poses. Source: original Figure 4 and Tables 2–4. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
Scope
Here, “6D pose” refers to relative camera poses within an object-centric image set, rather than object-to-camera poses in detection images. The inputs assume a centered object, approximately five views, and one static instance. The complete 1.5-minute test-time optimization is also unsuitable for real-time systems.
Available src/paper-notes/forge-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
FORGE: Deep Reading
1. Why the Task Is Difficult
Few-view object reconstruction often assumes calibrated cameras, while camera pose estimation suffers from symmetry, low texture, and wide baselines. In a serial pipeline, small front-end pose errors can produce ghosting and broken geometry during fusion. FORGE uses a shared 3D feature volume to break this one-way error propagation: 3D-aware correlation estimates poses, pose-aligned features support reconstruction, and reconstruction photometric loss feeds back to correct poses.
2. 3D Representation and Pose Estimator
A single-image encoder produces 2D features and lifts them through the viewing frustum into a 64³ voxel feature volume. The pose module uses both:
Global features: the overall appearance and viewpoint of each image;
Pairwise correlation: relationships between 3D voxels in two views.
Their combination is more robust than either branch alone. The first view is a canonical-view candidate; the system can try every frame as canonical and choose the best reference using reconstruction photometric error.
The 3D feature volume is the paper's central interface, connecting pose correlation, cross-view fusion, and volume rendering. Source: original Figure 2. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
3. Staged Training and Test-Time Optimization
Training has three stages: learn the voxel encoder / fusion / renderer with ground-truth poses, train the pose estimator, and finally fine-tune jointly. A cross-view consistency loss encourages similar 3D features for the same object regions in different views. At test time, FORGE† uses only a forward pass; full FORGE performs 1K iterations of pose optimization using the input views’ photometric loss.
This strategy prevents random poses and random reconstruction from destabilizing each other at the start, but reproduction requires consistent data, freezing policies, and coordinate conventions across stages.
4. Interpreting the Results
Kubric Test2 contains only novel categories: full FORGE achieves 8.9°/0.23, compared with 23.9°/0.49 for 8-Point Transformer. Source: original Figure 4 and Tables 2–4. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
On Kubric Test1 seen/unseen instances, full FORGE has rotation errors of 4.0°/4.3°; on Test2 novel categories it reaches 8.9°. Compared with 8-Point Transformer’s 15.0°/15.4°/23.9°, generalization beyond training categories is substantially more robust. FORGE† has 13.7° on Test2, so test-time optimization provides approximately a 35% relative reduction.
For reconstruction, FORGE* with ground-truth poses reaches PSNR 31.17 on Test2; predicted poses plus optimization achieve 25.57. Claims that it approaches the ground-truth-pose approach need qualification: the visual results are usable, but a gap of approximately 5.6 dB remains.
On OmniObject3D, FORGE has errors of 5.35°/0.30 and PSNR 26.56; PixelNeRF with ground-truth poses has PSNR 26.97. The joint system thus approaches a strong ground-truth-pose baseline, but the differing input conditions do not establish direct superiority.
The upper section covers Kubric / OmniObject3D / GSO; GSO demonstrates zero-shot transfer from synthetic Kubric training to real objects. Source: original Figures 5–7 and Tables 4–5. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
5. Ablations and Limitations
An additional 3D U-Net improves PSNR on seen Test1 but reduces novel Test2 from 31.17 to 22.33, which the paper interprets as heavier 3D convolutions overfitting training categories. Global+pairwise pose features yield 13.8°/0.31 on Test2, outperforming global-only 26.2°/0.44 and pairwise-only 22.1°/0.42.
Main limitations:
Five object-centric, static images of the same instance; detection and foreground segmentation are outside its scope;
Training relies on large-scale synthetic multiview data and ground-truth camera poses;
Predictions are relative poses within a camera set, subject to canonical-frame / scale conventions;
Full inference takes approximately 1.5 minutes; feed-forward FORGE† is faster but less accurate;
Reflective surfaces, thin structures, and symmetric objects may still produce incorrect local optima with low photometric loss.
6. Assessment
FORGE exemplifies the transition from pose estimation for reconstruction to pose estimation together with reconstruction. Its most reusable ideas are the shared 3D intermediate representation and staged training, rather than the specific NeRF renderer. Modern Gaussian / Transformer reconstruction systems can use the same interface: predict alignable 3D features first, then use rendering error as a test-time pose correction signal.
