Paper Reading Notes (Deep): Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

1. Basic information

• Title: Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
• Method / data: HOPformer / EPIC-Contact
• Authors: Siddhant Bansal, Zhifan Zhu, Shashank Tripathi, Jiahe Zhao, Michael J. Black, Dima Damen
• Affiliations: University of Bristol; Max Planck Institute for Intelligent Systems, Tübingen
• Year / venue: ECCV 2026 Poster
• Full text checked: arXiv:2606.30598 v1 (submitted 2026-06-29; 43 pages including the main paper and supplement)
• Official entry: ECCV 2026 Poster 3943
• Project / implementation: Project; HOPformer; EC-fit
• Data / weights: EPIC-Contact dataset; HOPformer models
• Reading mode: Deep Dive
• Reading focus: manual / optimization / propagation contributions to EPIC-Contact 3D annotations; how hand priors constrain object pose; evaluation protocols for success, contact error, and fully automatic inference; whether public-checkpoint scale bias changes the paper’s conclusions.

2. Main conclusions first

HOPformer uses egocentric single-image object representations as queries and left / right WiLoR hand representations as memory. Across 12 layers of self-attention and hand-object cross-attention, it updates object tokens and jointly regresses bimanual MANO, object R/T, optional articulation, and class. EPIC-Contact expands sparse manual bijective contact into 2,272 clips / 62.3K wild frames through EC-fit and hand-motion propagation.

2.1 Overall assessment

The paper has two mutually supporting contributions:

1. Data construction goes beyond automatic RGB-video pseudo-labeling. Annotators paint contact on subdivided MANO and transfer regional contact axes onto object surfaces with at most 6 clicks. EC-fit combines contact, occlusion-aware silhouettes, penetration, and hand regularization. Quantitative checks cover contact agreement, VLM scale, EC-fit pose, penetration, and noisy-label training.
2. Model design has explicit counterfactual evidence. Concatenation + MLP, generic hand features, no self-attention, no aggregation, and shallow decoders all degrade substantially. Gains are linked to repeatedly conditioning object tokens on pose-specialized hand priors, beyond backbone size alone.

Four points are especially easy to misread:

1. The abstract’s “nearly doubles / 75%” is not the strongest-baseline comparison. Against JointTransformer, SR@0.05 improves about 69.3% and CDev falls 31.2%; ArcticNet-SF gives the closer headline figures of 80.6% relative success improvement and 78.0% CDev reduction.
2. Main EPIC pose results use oracle object meshes; class accuracy is only 52.9%. The fully automatic table keeps correctly classified samples only, so it does not represent the whole test set.
3. EPIC SR is root-removed, symmetry-aware object-vertex success. Most metrics describe relative geometry or root-aligned shape / orientation and cannot validate full camera-space translation.
4. The repository discloses about 1.28× projection-scale and 0.78× absolute-depth bias in epic_epoch125.ckpt. Main-table results remain reproducible and unaffected under their protocol, but absolute-pose deployment needs extra calibration beyond Table 2.

Overall, HOPformer / EPIC-Contact provides a substantial egocentric joint hand-object pose pipeline, spanning data and architecture. Its strongest evidence is improved relative pose, contact, and root-aligned object pose from hand priors within fixed categories / mesh pools, rather than open-world CAD-free absolute 6D pose.

2.2 Claim–evidence map

Claim	Main evidence	Strength	Site assessment
EPIC-Contact supplies wild 3D supervision	κ, 30-object scale validation, ARCTIC 20% EC-fit simulation, penetration, noisy-label training	Moderately strong	Broad checks, with remaining propagation and small-scale validation limits
Hand priors improve object pose	Table 3 WiLoR / cross-attention / decoder ablations	Strong within ARCTIC	Architecture and metric changes agree; no module-by-module EPIC ablation
Nearly doubled wild success and 75% lower CDev	Table 2	Conditional	Requires ArcticNet-SF as reference; strongest JointTransformer gives smaller gains
Fully automatic inference	Cls 52.9%; Supplement Table 7 correct-class subset	Weak to moderate	Pose is usable with correct classes, but nearly half the test set is misclassified and lacks a unified pose score
Complete object pose prediction	R_o∈R^6, T_o∈R^3 outputs; MRRPE / SR	Partial	R/T are regressed, but main EPIC metrics and the checkpoint issue leave reliable absolute depth unverified
Scaling to more objects	ARCTIC 3/6/9/11 classes	Preliminary	Only 11 classes, with EPIC still at 9; larger scale remains future work

3. Task definition and coordinate conventions

3.1 External inputs and internal representations

HOPformer takes one egocentric RGB hand-object interaction image. The full scene enters the DINOv2 object encoder; local left / right crops enter WiLoR. The public README says no separate WiLoR detector.pt download is needed because HOPformer prepares its hand crops. Hand detection, object detection, and box noise are not independent evaluation variables. Thus the evidence covers joint pose given an interaction image, not a validated full-scene detection-to-pose pipeline.

The model handles right hand r, left hand l, and object o. EPIC-Contact has one hand per sequence, making hand–hand MRRPE inapplicable; ARCTIC includes both hands and articulated objects.

3.2 Output parameters

Each hand uses MANO:

$$\Theta_h=\{\theta_h,\beta_h\},\qquad H(\Theta_h)\in\mathbb{R}^{778\times3},$$

Here θ∈R^48 includes global orientation and β∈R^10 gives shape. Object pose is:

$$\omega_o=\{R_o,T_o,a_o\},$$

• R_o∈R^6: continuous 6D rotation representation, converted to a rotation matrix;
• T_o∈R^3: object translation;
• a_o∈R: one joint angle when the object is articulated.

Given class c and mesh O_c, the output is posed mesh O_c(ω). Rotation and translation are indeed regressed, but geometry comes from predefined model pool M, not image reconstruction.

3.3 Known category, novel instance, and oracle mesh

The task selects one object from known categories. ARCTIC train / validation share the same 11 object instances; EPIC-Contact tests novel instances of its known 9 categories. This is harder than same-instance testing but does not establish unseen-category / open-vocabulary pose.

The class head predicts c from DINOv2 object features and retrieves its mesh from M. For fair comparison with ArcticNet-SF and JointTransformer, Table 2 uses oracle class / mesh for pose and reports Cls=52.9% separately. It asks how well the pose head works when mesh identity is correct, leaving whole-system success with wrong meshes unanswered.

3.4 Differences from standard 6D object pose

• No BOP detection AP or combined VSD / MSSD / MSPD evaluation, nor full-scene multi-instance recall.
• EPIC SR removes the object root, then uses closest equivalent vertices for symmetry, emphasizing orientation / root-aligned shape consistency.
• CDev, MRRPE, and MDev refer to the hand, emphasizing hand-object relative geometry.
• Articulation angles and MANO metrics belong to interaction reconstruction and are not directly comparable with object-only ADD(-S) rankings.

The site includes it as an ECCV 2026 extended entry, outside the 14 core object-pose papers, because object pose is only part of joint hand-object reconstruction.

4. EPIC-Contact data construction in detail

4.1 Why stable-grasp clips

EPIC-Grasps / EPIC-Kitchens provides real kitchens, head-mounted cameras, cluttered backgrounds, frequent hand / object occlusion, and transparent glasses / bottles. Stable-grasp segments keep the hand-object contact-vertex set approximately fixed, enabling:

1. Manual contact judgment only in the center frame;
2. Fitting the center-frame hand-relative object pose;
3. Propagating the object through the clip using per-frame WiLoR hand pose.

Labels are consequently correlated across frames. The authors acknowledge hand-side errors, camera-placement jumps, and WiLoR jitter, and release per-frame confidence for filtering. Relative / root-aligned metrics resist global propagation noise, a reasonable design that also leaves absolute camera-space quality incompletely measured.

4.2 Dataset size and long tail

Total: 2,272 clips, 62.3K frames, 9 categories:

Category	Clips	Approximate share
pan	492	21.7%
plate	437	19.2%
bowl	396	17.4%
bottle	286	12.6%
glass	211	9.3%
mug	160	7.0%
cup	121	5.3%
can	97	4.3%
saucepan	72	3.2%

The first three categories account for about 58%; saucepan has only 72 clips, showing clear imbalance. The split uses 2,035 training and 237 testing videos, selecting 10% of unique videos while preserving hand-side / category distributions and preventing video overlap. The supplement specifies video disjointness, not stricter kitchen, participant, or object-identity group splits. Novel-instance claims should stay within this protocol, without assuming independent cross-environment sampling.

4.3 Hand contact painting

Standard MANO’s 778 vertices are uniformly subdivided to 3,106 for finer fingertip / palm contact. The UI plays the entire stable-grasp clip alongside a rotatable canonical hand. Annotators infer contacts invisible in the video, not just the currently visible regions.

For 12 annotators on 10 clips, κ_h=0.61, versus DECO 0.65. Fleiss’ kappa supports broadly repeatable contact identification, but only 10 clips were tested and 0.61 still permits substantial boundary disagreement. It is not a confidence interval for each annotation.

4.4 Bijective contact transfer

A hand-contact mask alone cannot recover object pose: each hand vertex needs an object-surface match. Following ContactEdit, regional contact axes parameterize thumb, four-finger, and palm groups:

• Each axis has 2 DoF, specified by two clicks for start and direction;
• At most three axes, hence up to 6 clicks per clip;
• WiLoR pose estimates finger spacing;
• Axis transfer supplies one object-surface correspondence for every contacting hand vertex.

For 4 annotators on 10 videos, object-correspondence agreement is κ_o=0.62. This supplies more information than foreground masks alone, but category-mesh versus real-instance shape differences become correspondence bias.

4.5 VLM object scaling

EPIC-Grasps meshes have fixed per-category scale, while real pans, bottles, and glasses vary in size / proportion. Gemini 2.5 receives category-specific prompts, such as pan diameter without the handle and total length with it, or glass upper / lower diameter and height. Multiple scale degrees then nonuniformly resize the template mesh.

Validation measures 30 real objects spanning 9 categories:

• absolute MAE 0.94 cm;
• relative error 5.9%;
• 82.5% of measured dimensions fall within ±10% of the real size.

VLM scale is therefore not arbitrary, but has three limits: only 30 objects, some ground truth obtained by identifying and measuring the same branded product, and about 17.5% of dimensions outside ±10%. Contact transfer, translation, and projected meshes inherit these scale errors.

4.6 EC-fit optimization

WiLoR initializes center-frame hand pose Θ_0. Objects use 6D rotation and multiple random / category-aware pose initializations. First minimize bijective contact:

$$L_{con}=\frac{1}{|\mathcal C|}\sum_{(h_i,o_i)\in\mathcal C}\|h_i-o_i\|_2.$$

Then perform image-guided refinement:

• L_om: compare rendered object and VISOR masks only in unoccluded regions;
• L_p: constrain hand-object penetration;
• L_hm: align rendered hand masks;
• L_reg=||Θ-Θ_0||²: keep MANO near the WiLoR initialization;
• Retain L_con while alternately / jointly updating hand and object.

Manual review and correction follow. EPIC-Contact 3D poses combine manual contact, known meshes, multiple optimization priors, and manual review; they are neither manual MoCap ground truth nor human-free pseudo-labels.

4.7 Annotation-quality audit

On an ARCTIC 20% training subset of 37,051 frames, object GT is removed, simulated contact annotations retained, and EC-fit run:

Configuration	Pose L2 ↓	MRRPE ↓	CDev ↓
contact alignment only	5.0 mm	9.6 mm	7.0 mm
full EC-fit	1.9 mm	8.0 mm	12.2 mm

The full pipeline greatly improves vertex pose and relative roots at the cost of contact distance. Image mask / pose alignment can conflict with pulling chosen contact vertices extremely close. EC-fit labels are multi-objective compromises rather than simultaneous optima for all physical quantities.

Penetration comparison:

Data	Volume ↓	Maximum depth ↓
ARCTIC MoCap	7.24 cm³	0.80 cm
EPIC-Contact	20.9 cm³	0.79 cm

Maximum depths are almost equal, but EPIC volume is 2.89 times larger; “comparable” applies to depth, not volume. Encouragingly, training on EC-fit labels versus GT on the ARCTIC subset changes CDev 40.9→41.1, MRRPE 35.6/37.3→35.7/37.0, and SR 71.9→71.8 only slightly. This directly supports limited training impact at the observed label-noise level.

5. HOPformer architecture in detail

5.1 Why objects are queries and hands are memory

DINOv2 ViT-G object encoder Φ_o produces dimension-1,536 generic object tokens z_o; WiLoR Φ_h produces dimension-1,280 pose-specialized tokens z_h from both hand crops. After linear projection:

$$X_o=W_oz_o,\qquad X_h=W_hz_h.$$

The hypothesis is asymmetric reliability: in the wild, hand foundation models are already more stable than joint hand-object models, so hands should condition objects. Each decoder layer applies:

1. Object self-attention: combine local and global object visual cues;
2. Hand-object cross-attention: object queries read hand keys / values;
3. Position-wise FFN: update each token.

Cross-attention at layer l is:

$$\operatorname{CA}(X_o^{l-1},X_h)= \operatorname{Softmax}\left(\frac{(X_o^{l-1}W_Q^l)(X_hW_K^l)^T}{\sqrt{d_k}}\right)(X_hW_V^l).$$

This is not an explicit contact solver: attention need not correspond to real contact vertices, and collision avoidance is not enforced at inference. Training’s CDev interaction loss supplies most physical consistency.

5.2 12-layer iteration and aggregation

The final decoder output has a residual connection to initial object tokens, followed by aggregation:

$$z_i=\operatorname{LayerNorm}(A(X_o^{L}+X_o)).$$

The supplement describes A as an MLP mapping 256 object tokens to 39 task tokens:

• tokens 1–16: right MANO joint pose / global orientation;
• token 17: right root; token 18: right shape;
• Tokens 19–36: corresponding left-hand outputs;
• Last 3 tokens: object pose / articulation.

Learnable routing outperforms fixed token selection: without aggregation, SR@0.05 falls 82.4→74.0 and CDev changes 31.9→40.4.

5.3 Object class and model pool

The pose decoder does not hardcode object geometry, but converting R/T/a into a mesh requires O_c. The class head directly predicts c from z_o, then retrieves a mesh from M. Decoupled pose regression and identity classification works well on ARCTIC (Cls 99.5%), but falls to 52.9% on EPIC novel instances, becoming a bottleneck.

Omitting a mesh as an explicit per-frame input does not make the system CAD-free; lookup follows class prediction. Supplementary qualitative comparisons with HOLD-Net, G-HOP, and SAM 3D show wrong CAD-free shapes on transparent / heavily occluded objects, but these are video-reconstruction or different tasks without a common quantitative protocol. They cannot establish universal superiority over CAD-free methods.

5.4 Loss design

For each hand:

$$L_h=\lambda_{2D}L_{2D}+\lambda_{3D}L_{3D}+\lambda_\theta L_\theta+\lambda_\beta L_\beta+\lambda_TL_T.$$

For the object:

$$L_o=\lambda_{3D}L_{3D}+\lambda_{2D}L_{2D}+\lambda_TL_T+\lambda_cL_c+\lambda_\omega L_\omega.$$

Contact frames add L_int for right-object and left-object CDev. Total loss is L=L_r+L_l+L_o+L_int. Object rotation uses geodesic loss and classification its corresponding objective; the main text describes most remaining terms as MSE.

All λ default to 1, except hand 2D at 5, right / object 3D at 5, hand pose / left 3D at 10, and hand shape / class at 0.001. Left / right 3D weights are not described symmetrically; reproduce the public configuration rather than reconstructing it from one prose statement.

5.5 Weak-perspective translation

Metric translation is decoded from regressed weak-perspective camera parameters using fixed focal lengths: ARCTIC 1,000, EPIC 5,000 in the supplement’s training convention. Inherited from ArcticNet, this suits root-relative reconstruction, but inconsistent focal / crop-scaling conventions directly become absolute-depth bias, as in EPIC’s known issue.

6. Training and data dependence

6.1 Three-stage training chain

1. ARCTIC exocentric pretraining: 8 external views, total batch 256 (4×GH200, 64 per GPU), 25 epochs; LR warms from 1e-7 to 5e-5, then follows cosine decay.
2. ARCTIC egocentric fine-tuning: initialize from p1, total batch 128, 32 per GPU, at most 30 epochs, peak LR 3e-5, with early stopping.
3. EPIC-Contact fine-tuning: initialize from the ARCTIC egocentric checkpoint, keep the batch / LR convention, and train up to 125 epochs.

DINOv2 is frozen. The public implementation requires Python 3.10, PyTorch 2.5.1, CUDA 12.6, and manual smplx changes to return 21 hand joints. WiLoR code is vendored, but noncommercial weights and MANO assets require separate acquisition.

6.2 In-the-wild data does not mean wild-only training

Table 4 is crucial:

EPIC training	CDev ↓	MRRPEro ↓	MDev ↓	MPJPE ↓	SR@0.05 ↑	SR@0.1 ↑
scratch	24.4	99.5	19.5	54.8	3.9	24.2
ARCTIC init	20.7	65.8	11.4	19.9	29.8	69.7

Success rises from 3.9% to 29.8% and hand MPJPE falls from 54.8 to 19.9 mm, making 2.1M-frame ARCTIC MoCap pretraining decisive. EPIC-Contact currently supports fine-tuning / benchmarking but cannot train an equivalent model from scratch. This limits claims that scalable contact annotations alone remove dependence on laboratory data, without diminishing the dataset’s value.

7. Metric definitions: why one SR is insufficient

7.1 Hand-object relative geometry

• CDev (mm): mean distance between corresponding contacting hand-object vertex pairs. EPIC’s symmetry-aware version finds the nearest predicted object-mesh point for each contacting hand vertex, tolerating surface sliding more.
• MRRPE rl / ro (mm): relative-position errors between right–left hand roots and right-hand–object roots. EPIC has one hand per clip, making rl inapplicable.
• MDev (mm): frame-to-frame motion differences between the hand and a fixed local object patch over stable contact. Its symmetric version allows equivalent initialization but still penalizes contact-point sliding over time.
• ACC h/o: hand / object vertex acceleration differences. EPIC object evaluation uses symmetric nearest-vertex matching.

These mainly assess plausible hand-object relative placement and stable contact, rather than fully evaluating an isolated object’s camera pose.

7.2 Hands and objects

• MPJPE: 21 hand joints root-aligned 3D error.
• AAE: average articulation error, meaningful only on ARCTIC.
• SR@0.05 / 0.1: after removing the object root, vertex success within 5% / 10% of object diameter. EPIC uses nearest GT vertices, an ADD-S-style symmetry-aware protocol.
• Cls: object class accuracy, reported separately from pose metrics.

Root removal excludes absolute translation bias from SR. MRRPEro measures relative hand-object roots and cannot isolate common hand-object scene-scale bias along camera z. This explains why EPIC’s absolute-depth checkpoint issue leaves Table 2 unaffected.

7.3 ACC and time units

The table header says m/s², but EPIC clips mix native frame rates and dropped frames. The public evaluator fixes 15 fps, and the official README explicitly treats ACC as relative smoothness. Values such as 2.5 / 4.1 are not exact SI acceleration errors for cross-dataset or physical-motion comparison.

8. Auditing the main results

8.1 ARCTIC egocentric

Metric	ArcticNet-SF	JointTransformer	HOPformer	Versus strongest baseline
CDev ↓	44.1	35.0	31.9	-3.1 mm / -8.9%
MRRPE rl / ro ↓	33.9 / 36.8	34.0 / 29.9	31.1 / 29.4	-2.9 / -0.5 mm
MDev ↓	11.8	10.4	7.3	-3.1 mm / -29.8%
ACC h/o ↓	6.3 / 11.3	6.9 / 10.1	4.8 / 6.2	Both improve
MPJPE ↓	22.9	20.0	16.1	-3.9 mm / -19.5%
AAE ↓	8.0	4.9	5.0	+0.1°, not best
SR@0.05 ↑	59.0	76.2	82.4	+6.2 points

The main text accurately says best on all metrics except AAE. Since the same ARCTIC objects appear across train / validation, this mainly tests architecture and in-distribution interaction modeling rather than novel-instance / category generalization.

8.2 EPIC-Contact

Metric	ArcticNet-SF	JointTransformer	HOPformer	Versus JointTransformer
CDev ↓	94.2	30.1	20.7	-9.4 mm / -31.2%
MRRPEro ↓	166.7	78.6	65.8	-12.8 mm / -16.3%
MDev ↓	70.1	20.0	11.4	-8.6 mm / -43.0%
ACC h/o ↓	3.9 / 5.3	3.1 / 9.5	2.5 / 4.1	Both improve
MPJPE ↓	44.8	22.9	19.9	-3.0 mm / -13.1%
SR@0.05 ↑	16.5	17.6	29.8	+12.2 points / 1.69×
SR@0.1 ↑	55.9	56.9	69.7	+12.8 points
Cls ↑	—	—	52.9	No baseline

Object success clearly improves, but remains limited: at the strict 5% threshold, 29.8% still means most samples fail. At 10%, success is 69.7%, showing strong tolerance sensitivity.

8.3 Baseline choice behind the headline

The abstract claims nearly doubled success and 75% lower contact deviation without naming the baseline alongside it. From Table 2:

• Against ArcticNet-SF: SR 16.5→29.8, 1.806×; CDev 94.2→20.7, down 78.0%, close to the abstract.
• Against JointTransformer: SR 17.6→29.8, 1.693×; CDev 30.1→20.7, down 31.2%.

The main text identifies JointTransformer as the same-backbone, best direct comparison. A topic page should therefore pair the abstract’s 75% with the strongest-baseline figures.

8.4 Oracle mesh and the correct-class subset

Table 2 uses oracle meshes. Supplement Table 7:

Scope	CDev	MRRPEro	MDev	MPJPE	SR@0.05	SR@0.1
complete data, oracle class	20.7	65.8	11.4	19.9	29.8	69.7
only correctly classified subset	20.2	64.5	11.4	18.4	30.4	68.4

The subset is slightly easier, so similar scores are unsurprising. Wrong classes have different topology, leading the authors to omit pose calculation for about 47.1% of samples. Whole-system evaluation should count wrong classes as failures or use cross-mesh surface matching / detection-pose AP; the paper does not report this.

9. Ablation and causal evidence

9.1 Interaction and object supervision

• No L_int: CDev 31.9→40.7, MDev 7.3→10.8, SR 82.4→79.2. Interaction loss mainly improves physical contact / motion consistency beyond pose-score gains.
• No L_oω: SR 82.4→72.5; no entire L_o: 66.8, AAE 5.0→8.6. Hand priors cannot replace direct object supervision.
• No 3D keypoints: MPJPE 16.1→18.7, the largest hand-accuracy loss ablation.

9.2 Cross-attention versus ordinary fusion

With the same backbone, Concat + MLP gives CDev 74.4, MDev 32.0, SR 29.4, a 53-point drop from the full model. Layerwise query–memory interaction is a main contribution beyond an added parameter head. However, parameter / FLOP matching is not documented rigorously, so optimization capacity may explain some of the gap.

9.3 Pose-specialized hand prior

Replacing WiLoR with DINOv2 hand-crop tokens changes MPJPE 16.1→20.6, CDev 31.9→44.2, and SR 82.4→70.0. Hand and object performance degrade together, supporting both positive conditioning and error propagation: strong hand priors help, wrong ones mislead.

9.4 Depth and compute

For decoder L=1/6/9/12, SR is 21.9 / 58.6 / 69.0 / 82.4 and CDev 101.6 / 59.2 / 50.0 / 31.9. Gains continue with depth, without a lightweight saturation point. The 1.83B parameters and 1.157 TFLOPs are part of attaining the main results, not incidental overhead.

9.5 Missing generalization ablations

All loss / architecture ablations use ARCTIC. Main EPIC results show full-model transfer but do not answer:

• Would removing WiLoR cause an equally large EPIC drop?
• Could contact loss overfit noisy EC-fit labels?
• How much do transparent / reflective classes benefit separately?
• Which of the 9 classes is most often misclassified, and how does that affect usability?

These questions determine why hand priors help in the wild; current answers extrapolate ARCTIC counterfactuals to EPIC.

10. Repository known issue: scale and absolute depth

As of 2026-08-23, the official HOPformer README records EPIC-checkpoint projection-scale bias:

• EPIC weak-perspective targets use height-scaled f_y=2333.33;
• Hand / object head decoding uses average focal 1822.40;
• The ratio 2333.33/1822.40≈1.28 gives median projected-mesh scale about 1.28×;
• Median absolute depth is about 0.78× GT.

The official correction sweep found no cost-free inference-only fix. A shared-scene translation workaround restores median projection scale / depth but adds about 3.4 px mean 2D hand-joint error. Square ARCTIC inputs with f_x=f_y are unaffected.

10.1 Why paper results remain reproducible

Table 2 mainly measures:

• root-aligned hand MPJPE;
• hand-object relative root MRRPE;
• root-removed object SR;
• hand-object contact / motion relative metrics.

Shared scene-scale / depth bias does not directly alter these root-aligned / relative quantities; baselines share the same convention. Prediction dumps and reference aggregate JSON reproduce Table 2 exactly. The correct interpretation is:

The comparison still supports improved interaction and root-aligned pose over baselines. The released EPIC checkpoint nevertheless has known absolute-camera-depth and projection-scale bias, outside the main table’s deployment coverage.

10.2 Practical implications for 6D pose readers

Robot grasping, AR overlays, and camera-frame object pose often require metric T=[x,y,z]. Reproducing Table 2 alone misses z and projected scale. Practical use should at least:

1. Independently validate camera-space translation / ADD(-S) without root removal;
2. Check intrinsics and focal conventions after crop resizing;
3. Report 2D reprojection, absolute depth, and relative hand-object geometry separately;
4. Distinguish workaround-adjusted poses from the original main-table checkpoint convention.

11. Failures and limitations

11.1 Data

• Each center frame needs about 3–4 minutes of annotation; 2.3K clips still entail substantial data engineering and do not imply unlimited scalability.
• One frame’s annotation propagates through its clip. Low-confidence frames can be filtered, but confidence-binned pose / training curves are not provided.
• The 30-object scale check is small, and VLM nonuniform scaling may deform templates in ways not fully consistent with real rigid objects.
• EPIC penetration volume is much larger than ARCTIC’s; similar maximum depth does not remove wider interpenetration across contact areas.
• Strong class imbalance: saucepan has only 72 clips versus pan 492.

11.2 Method

• Fixed model pool: mesh retrieval is necessary; unknown classes / wrong meshes leave pose undefined.
• EPIC class accuracy is only 52.9%, and main-table success does not incorporate this frontend bottleneck.
• Strong hand priors can override object appearance. A failure case predicts a plate as horizontal, showing plausible but wrong dataset-driven pose.
• Single-frame inference does not use genuine temporal evidence, despite temporal data and MDev.
• EPIC validates only single-hand rigid objects; wild bimanual and articulated settings remain future work.
• 1.83B parameters / 1.157 TFLOPs / GH200 profiling are far from mobile AR or onboard robot deployment.

11.3 Conclusions

• SOTA is relative to retrained ArcticNet-SF and JointTransformer under the given protocol, not a unified quantitative comparison with all hand-object / CAD-free methods.
• In-the-wild describes data origin and scene complexity, not open-world categories.
• Fully automatic requires the correct-class-subset qualification rather than implying the entire test set.
• Success rate is a root-aligned, diameter-normalized vertex metric, not whole-system task or grasp success.

12. Compute, resources, and licensing

12.1 Profiling

One NVIDIA GH200 120GB, batch 1, inputs preloaded on GPU, 30 warmup + 300 timed iterations, explicit CUDA synchronization:

• parameters: 1.83B;
• FLOPs: 1.157 TFLOPs / sample;
• peak allocated memory: 7.01 GiB;
• latency P50 / P95: 106.63 / 165.89 ms;
• throughput: 9.12 samples/s.

This is two orders of magnitude faster than >10 s EC-fit / optimization baselines, but excludes complete loading, detection, visualization, and communication. The 9.12 FPS figure cannot be directly extrapolated to consumer GPUs.

12.2 Reproduction materials

The public repository includes:

• ARCTIC / EPIC training, prediction extraction, and evaluators;
• p1 / p2 / EPIC checkpoints;
• ARCTIC p2 and EPIC per-sample prediction dumps;
• Reference aggregate metric JSON and column-key mappings;
• Pinned environments and data-directory conventions.

This is more substantial than inference-only demos. However, EPIC data, models, and prediction dumps are gated; ARCTIC, MANO, and WiLoR require separate access / downloads, leaving multiple reproduction dependencies.

12.3 Licensing boundaries

• HOPformer code: CC BY-NC 4.0;
• EPIC-Contact and checkpoints: CC BY-NC 4.0, gated;
• vendored WiLoR code: CC BY-NC-ND 4.0;
• WiLoR weights, MANO, and ARCTIC retain their respective noncommercial terms.

Public code therefore does not imply commercial permission or unrestricted modification / redistribution of every dependency. Product use requires reviewing component licenses individually.

13. Reproduction and improvement suggestions

13.1 Minimum trustworthy reproduction

1. Download official EPIC prediction dumps and align the evaluator with Table 2, checking cdev_sym/ho, mdev_sym/h, adds_success/0.05, mrrpe/r/o, mpjpe/ra/h, and obj_cls_acc.
2. Rerun the released checkpoint on the same split, checking FP precision, crop / focal conventions, and data version.
3. Compute absolute depth, camera-space ADD-S, and 2D reprojection to explicitly reproduce the known issue.
4. Report oracle mesh, correct-class subset, and whole-system scores counting wrong classes as failures separately.
5. Stratify by class, occlusion, transparency, confidence, and VLM scale error.

13.2 Research directions worth pursuing

• Mesh uncertainty: maintain joint pose hypotheses over top-k classes / shapes rather than hard-selecting a mesh after classifier argmax.
• Absolute geometry calibration: encode intrinsics-aware perspective translation and crop transforms in models / losses to avoid weak-perspective focal-convention drift.
• Temporal models: use genuine motion and contact invariance in stable-grasp videos, beyond temporal metrics and label propagation.
• Contact uncertainty: turn κ, propagation confidence, and EC-fit residuals into per-vertex / per-frame loss weights.
• CAD-free bridge: combine reliable mesh pools with retrieval + deformation / category shape bases to handle new-instance geometry before relying on pure generation.
• More categories and wild bimanual data: current 9-category, single-hand EPIC is a starting point, insufficient for household-scale open-world hand-object pose.

14. Overall assessment

Conclusions the paper establishes

• In controlled same-instance ARCTIC, a hand-conditioned deep decoder clearly exceeds two retrained baselines and leads on all main metrics except AAE.
• On EPIC-Contact novel instances of known categories, with correct meshes, HOPformer clearly improves CDev, MDev, MPJPE, and root-aligned SR over the strongest baseline.
• Contact annotation + EC-fit constructs trainable wild 3D hand-object data with relatively little manual interaction; ARCTIC simulation gives similar results from noisy and GT training labels.
• Cross-attention, WiLoR hand priors, self-attention, aggregation, and decoder depth have directionally consistent ablation support.

Broader conclusions still unproven

• Open-category, CAD-free, or arbitrary-object pose;
• Fully automatic pose over the whole test set, given only 52.9% classification accuracy;
• Reliable absolute camera-space 6D translation from Table 2;
• Mobile real-time use or large-scale category expansion;
• Itemized causal attribution of EPIC gains to individual architecture components.

For 6D object pose researchers, the main insight is: hands provide a learnable geometric condition, not just occlusion. Once hand priors, mesh identity, weak-perspective cameras, and contact metrics enter the system, oracle conditions, relative coordinates, and absolute pose must be reported separately.