Object Pose Estimation · 2026 · 3DV · CCF Unranked

Reconstructing Hand-Held Objects in 3D from Images and Videos

MCC-HO jointly recovers hand–object geometry from a single image and an inferred 3D hand. RAR retrieves a matching 3D object model with a VLM and rigidly aligns it to multiframe observations. The hand provides a strong geometric anchor for scale and position.

Paper Project

Research area

Object Pose Estimation

Task level

Instance-level

Object setting

Unseen objects

Input modalities

monocular RGB image / videoinferred 3D hand

Requirements

3D hand estimateVLM / text-to-3D model for RAR

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik
Affiliations: University of California, Berkeley / The University of Texas at Austin / California Institute of Technology
Conference / journal: Conference · 3DV · CCF Unranked
Research area: Object Pose Estimation
Code availability: Unavailable / not yet recorded
Reading status: Overview:Available Deep Dive:Available
Task setting: Instance-level · Unseen objects
Datasets: DexYCB MOW HOI4D 100 Days of Hands
Evaluation metrics: Chamfer Distance
Method tags: Instance-level Pose Estimation Template Matching & Render Retrieval Pose Refinement Synthetic Data & Foundation Models Multi-object Pose Estimation Novel-object Pose Estimation Absolute Pose Estimation
Requirements: 3D hand estimateVLM / text-to-3D model for RAR

Reading Notes

Overview / Deep Dive

Available src/paper-notes/hand-held-objects-reconstruction-3dv2026/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Paper Reading Notes (Coarse): Reconstructing Hand-Held Objects in 3D from Images and Videos

1. Basic information

Authors / affiliations: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik; UC Berkeley, UT Austin, Caltech.
Year / venue: 2026 / 3DV.
Paper: OpenReview; arXiv PDF; Project.
Reading mode: Overview.

2. Positioning in one sentence

Using an inferred 3D hand as a scale / position anchor, MCC-HO reconstructs a handheld object from one image. Retrieval-Augmented Reconstruction (RAR) then retrieves a 3D asset and rigidly aligns it across frames.

3. Problem setting

Input: one RGB image or RGB video and an estimated 3D hand; RAR additionally uses a VLM and text-to-3D / object assets.
Output: hand-object geometry and, for video, a unified, temporally consistent object model with transforms.
Challenges: hand occlusion, very few object pixels, and long-tail target shapes. This is not a fixed-CAD BOP pose estimator.

4. Method overview

MCC-HO predicts hand-object geometry from RGB and a 3D hand.
A VLM identifies / retrieves an observed-object-compatible 3D model; RAR rigidly aligns it with MCC-HO / image observations.
The 3D hand reduces geometric ambiguity and enables object scale / location alignment across frames.

5. Experimental overview

Datasets: DexYCB, MOW, HOI4D; demonstrations also use unlabeled 100DOH.
Single-frame MCC-HO on MOW: F5 / F10 / CD of 0.15 / 0.31 / 15.2, versus IHOI 0.13 / 0.24 / 23.1.
HOI4D video: MCC-HO+RAR F5 / F10 / CD of 0.74 / 0.91 / 0.64, versus G-HOP 0.61 / 0.89 / 0.8.

6. Strengths, weaknesses, and implications

Strengths

The hand strongly conditions object location / scale; RAR turns local single-image geometry into a reusable object asset.

Limitations / risks

Depends on 3D hand estimation, VLM retrieval, and generated-asset quality; rigid alignment cannot repair retrieval of the wrong category.
Joint reconstruction metrics measure pose only indirectly and cannot be directly compared with standard 6D pose AR.

7. Points for deeper reading

In the no-hand ablation, MOW CD rises from 15.2 to 71.6. This strongly supports explicit hand geometry as a potentially more important cue than a larger object prior in interaction scenes.

Available src/paper-notes/hand-held-objects-reconstruction-3dv2026/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Paper Reading Notes (Deep): Reconstructing Hand-Held Objects in 3D from Images and Videos

1. Basic information

Authors / affiliations: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik; UC Berkeley, UT Austin, Caltech.
Year / venue: 2026 / 3DV.
OpenReview · arXiv PDF · Project.
Reading mode: Deep Dive.

2. Positioning in one sentence

This work splits handheld-object reconstruction into two levels: MCC-HO uses a 3D hand to constrain single-frame object geometry; RAR uses VLM / text-to-3D assets to turn multiframe local predictions into a unified, rigidly aligned object model. The hand serves as a geometric scale and position anchor.

2.1 Guide to the paper’s figures and tables

Figure / table	Reading focus
Figure 2	How MCC-HO obtains local hand-object geometry from RGB + inferred hand, and how RAR retrieves and aligns an asset over frames
Figures 4–6	MOW comparisons and in-the-wild 100DOH; distinguish local point-cloud fit from consistent retrieved-model geometry
Tables 1–5	DexYCB/MOW/HOI4D, video comparisons, and hand-input / RAR ablations

Hand-held object reconstruction Figure 2: MCC-HO and RAR pipeline — Excerpt from the original method page containing Figure 2: MCC-HO reconstruction and the RAR retrieval / rigid-alignment video pipeline. Source: arXiv PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for paper analysis.

Hand-held object reconstruction Table 1: quantitative results — Excerpt from the original experiment page containing Table 1: handheld-object reconstruction on DexYCB, MOW, and HOI4D. Source: arXiv PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for paper analysis.

3. Problem setting and boundaries

Input: single-frame RGB or monocular video and an inferred 3D hand; RAR additionally uses VLM / GPT-4(V)-style object retrieval and 3D generation assets.
Output: hand-object geometry; video output includes an across-frame coherent object model and its rigid alignment.
Task boundary: joint geometry and relative alignment evaluated by CD, F-score, and intersection volume, rather than standard known-CAD object 6D pose benchmarking.
Key prior: hand geometry explains object location / scale; the space of manipulanda is smaller than the space of all objects, making retrieval tractable.

4. Method details

Module	Input	Output	Role
Inferred 3D hand	RGB video / image	Hand mesh / pose	Supplies metric scale, contact, and an explanation of occlusion
MCC-HO	RGB + 3D hand	Hand-object points / geometry	Predicts image-aligned, interaction-consistent shape from one frame
RAR	Object appearance + MCC-HO observations	Retrieved 3D asset + rigid transforms	Unifies geometry across frames with a relatively complete asset
Temporal alignment	Multiframe asset / observations	Consistent trajectory	Turns local single-frame reconstructions into a temporally consistent object model

MCC-HO follows the MCC approach but is specifically trained for handheld objects. RAR must align the object model with both images and MCC-HO observations, giving downstream applications usable pose consistency in addition to a complete mesh.

5. Experimental analysis

Data / method	F5	F10	CD ↓	Notes
MOW, IHOI	0.13	0.24	23.1	Existing hand-object baseline
MOW, MCC-HO	0.15	0.31	15.2	Hand-conditioned single-frame geometry improvement
HOI4D video, G-HOP	0.61	0.89	0.80	video baseline
HOI4D video, MCC-HO+RAR	0.74	0.91	0.64	Gains from RAR’s unified asset

Key ablations on MOW: the MCC baseline gives F5/F10/CD of 0.10 / 0.21 / 31.6; removing the hand from MCC-HO degrades this to 0.03 / 0.12 / 71.6. Adding RAR changes MCC-HO F10/CD from 0.31 / 15.2 to 0.44 / 9.41. Hand geometry and retrieval therefore serve complementary roles.

6. Critical analysis

Strengths: directly addresses internet video and hand occlusion; convincing ablations of the hand as a geometric prior; separate single-frame and video metrics; extension to unlabeled 100DOH.

Limitations: 3D hand estimation, object recognition / retrieval, and generated assets are external dependencies. If the VLM misidentifies the object category, rigid alignment cannot correct semantic / shape mismatch. Absolute metric-pose error is not reported as a separate benchmark.

Practical feasibility: useful for offline hand-object training-data creation, AR assets, or video postprocessing. Robotics use still requires converting hand-relative mesh transforms into a common camera / world frame and quantifying pose uncertainty.

Possible improvements: replace the single best retrieved asset with a posterior over candidate CAD models / assets; constrain object transforms using multiframe hand kinematics; evaluate contact stability and grasp feasibility alongside F-score/CD.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.