Object Pose Estimation · 2026 · 3DV · CCF Unranked

Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance

FollowMyHold adds multistage geometric guidance to sampling from a pretrained rectified-flow 3D generator. Joint optimization of hand / object transforms, constrained by depth, normals, silhouettes, keypoints, and SDF contact, produces physically consistent hand–object interaction reconstructions.

Paper Project

Research area

Object Pose Estimation

Task level

Instance-level

Object setting

Unseen objects

Input modalities

single RGB hand-object image

Requirements

hand / object masks and inpaintingfoundation-model geometric cuestest-time optimization

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, Christian Theobalt
Affiliations: Max Planck Institute for Informatics / Saarland University / Bielefeld University
Conference / journal: Conference · 3DV · CCF Unranked
Research area: Object Pose Estimation
Code availability: Unavailable / not yet recorded
Reading status: Overview:Available Deep Dive:Available
Task setting: Instance-level · Unseen objects
Datasets: OakInk ARCTIC DexYCB
Evaluation metrics: Chamfer Distance
Method tags: Instance-level Pose Estimation Pose Refinement Synthetic Data & Foundation Models Multi-object Pose Estimation Novel-object Pose Estimation Absolute Pose Estimation
Requirements: hand / object masks and inpaintingfoundation-model geometric cuestest-time optimization

Reading Notes

Overview / Deep Dive

Available src/paper-notes/follow-my-hold-3dv2026/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Paper Reading Notes (Coarse): Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance

1. Basic information

Authors / affiliations: Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, Christian Theobalt; MPI-INF, Saarland University, Bielefeld University.
Year / venue: 2026 / 3DV.
Paper links: OpenReview; arXiv PDF; Project.
Reading mode: Overview.

2. Positioning in one sentence

FollowMyHold adds optimization-in-the-loop guidance during sampling in a rectified-flow generator such as Hunyuan3D-2. Joint optimization of hand / object transforms, geometry, and contact directly generates more plausible single-image HOI reconstructions.

3. Problem setting

Input: a single RGB hand–object image, with geometric cues such as masks, inpainting, and hand pose supplied upstream.
Outputs: an object mesh, hand mesh / transform, and physically plausible hand–object interaction.
This is not a real-time 6D pose head: each sample takes approximately 6.03 minutes. Its main value is high-quality reconstruction under occlusion.

4. Method overview

Isolate the interaction region in the input; pretrained modules provide cues such as normals, depth / disparity, and keypoints.
Perform hand-only, object-only, and joint refinement in stages while steering the diffusion velocity field.
Constrain object geometry and relative pose with silhouette, normal / depth, 2D keypoint, SDF intersection, and proximity losses.

5. Experimental overview

OakInk / ARCTIC / DexYCB: the paper reports CD of 1.80 / 2.57 / 2.04 cm², the lowest in the table on all three datasets.
OakInk reconstruction rate is 0.87, compared with 0.39 for EasyHOI, indicating both higher accuracy and fewer outright reconstruction failures.
Phase ablation on 100 OakInk samples: full-method CD is 1.70; removing guidance, Phase 1, or Phase 2 gives 3.70, 5.67, or 3.53, respectively.

6. Strengths, Limitations, and Insights

Strengths

Image-to-3D generation obeys geometry and contact during sampling, rather than relying on corrections afterward.

Limitations / risks

Runtime is on the order of minutes per sample and depends on masks / inpainting; thin objects and upstream errors remain failure cases.
Accurate geometry does not automatically yield metric-scale 6D pose suitable for immediate robotic grasping.

7. Points Worth Reading Closely

The explicit separation into hand-first, object-first, and joint optimization stages is valuable: it reflects the hand’s role as a more stable spatial anchor than the object in single-image interaction.

Available src/paper-notes/follow-my-hold-3dv2026/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Paper Reading Notes (Deep): Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance

1. Basic information

Authors / affiliations: Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, Christian Theobalt; MPI-INF, Saarland University, Bielefeld University.
Year / venue: 2026 / 3DV.
OpenReview · arXiv PDF · Project.
Reading mode: Deep Dive.

2. Positioning in one sentence

FollowMyHold uses the diffusion generator for more than producing a mesh in one shot. During rectified-flow sampling, 2D / 3D geometry losses directly steer its velocity field, while staged optimization of hand and object transforms lets generated object geometry and grasp contact converge together.

2.1 Guide to the paper’s figures and tables

Figure / table	Reading focus
Figure 2	Hand/object cue extraction before generation, three-stage optimization, and where guidance enters latent sampling
Table 1	Object accuracy, intersection, and reconstruction rate on three datasets; all must be considered alongside CD
Tables 2–3 / Figures 5–7	The effects of phase order and each geometry loss, especially the hand-only phase

FollowMyHold Figure 2: phased geometric guidance pipeline — Excerpt from the original paper's method section: the rectified-flow formulation, cue extraction from single RGB HOI, shared-coordinate alignment, and diffusion guidance. Source: arXiv PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for paper analysis.

FollowMyHold Table 1: HOI reconstruction results — Excerpt from the original paper's experimental setup and metrics section: intersection / proximity losses for joint optimization, the HOI reconstruction evaluation setup on OakInk, ARCTIC, and DexYCB, and definitions of CD, F-score, I.V., and R.R. Source: arXiv PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for paper analysis.

3. Task Boundaries

The input is a single RGB frame; preprocessing produces hand/object masks, inpainted object appearance, normal / depth cues, and a hand pose / mesh.
Outputs are an object mesh and hand–object transforms, with contact / non-intersection constraints encouraging plausible interaction.
Inference includes test-time optimization at approximately 6.03 min/sample and should not be described as real-time pose estimation.
Training relies on pretrained generators / foundation models rather than a separate object-generation network trained specifically for OakInk, ARCTIC, and DexYCB.

4. Method details

4.1 Three-Stage Sampling Guidance

Phase 1: optimize the hand transform first to establish a reliable spatial anchor.
Phase 2: use the hand anchor to optimize the object transform and object-centric 2D cues.
Phase 3: jointly refine hand/object transforms, using 3D constraints such as SDF intersection and proximity to avoid interpenetration or unsupported contact.
Each phase’s losses feed back into the diffusion / rectified-flow velocity. The mesh adapts to geometry during sampling itself, instead of being rigidly aligned only after generation.

Constraint	Information source	Failure addressed
normal / disparity	foundation-model geometric cue	Distorted object depth and local surfaces
silhouette / keypoint reprojection	image alignment	Drift in 2D shape and position
SDF intersection / proximity	posed hand-object meshes	Hand–object penetration and implausible grasp contact

5. Experimental analysis

Dataset	This method CD ↓	This method F5 / F10	This method I.V. ↓	R.R. ↑
OakInk	1.80	0.179 / 0.322	5.96	0.87
ARCTIC	2.57	0.160 / 0.288	5.08	0.92
DexYCB	2.04	0.158 / 0.300	7.02	0.58

The paper reports the lowest tabulated CD on all three datasets. Compared with EasyHOI, OakInk R.R. rises from 0.39 to 0.87, supporting the claim of higher generation quality and fewer failures. However, HORT has higher values for some F-scores, showing that CD, local point coverage, and generation success rate do not always agree.

In the ablation on 100 OakInk samples, full-method CD / I.V. is 1.70 / 4.03; without guidance it is 3.70 / 5.83; without Phase 1, 5.67 / 7.66; and with direct joint optimization, 5.51 / 11.7. These results strongly support the phase order and hand-first design.

6. Critical analysis

Strengths: physical plausibility directly guides generative sampling; reporting three datasets and R.R. avoids evaluating only examples that are easy to generate; phase and loss ablations are thorough.

Limitations: runtime on the order of minutes; upstream mask / inpainting artifacts propagate; thin objects receive weak supervision and are also identified as failure cases in the paper; metric object-pose accuracy is not separately aligned with the BOP protocol.

Practical applications: suitable for high-quality offline HOI reconstruction, AR/VR assets, or synthetic training data, rather than per-frame localization in closed-loop control. For robotics, inference-time cues should be replaced with stable RGB-D / tactile signals, and the 6-minute optimization distilled into a feed-forward initialization.

Insight: for heavily occluded pose estimation, first determine which entity provides the more reliable frame, then order optimization by observability. This is often more stable than minimizing all degrees of freedom jointly from the outset.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.