Paper Reading Notes (Deep): HRDexDB: A Paired Human-Robot Dataset for Cross-Embodiment Dexterous Grasping

1. Basic information

• Dataset: HRDexDB
• Authors: Jongbin Lim, Taeyun Ha, Mingi Choi, Jisoo Kim, Byungjun Kim, Subin Jeon, Hanbyul Joo
• Affiliations: Seoul National University, RLWRLD
• Year / publication: 2026 / arXiv v2 preprint
• Paper: arXiv:2604.14944
• PDF: arXiv v2 PDF
• Project: HRDexDB Project
• Code: snuvclab/HRDexDB
• Data: HRDexDB on Hugging Face
• Data license: CC BY-NC 4.0; check the dataset repository’s current license and file documentation before use
• Reading mode: Deep Dive

2. Positioning in one sentence

In one calibrated 23-view space, HRDexDB pairs natural human grasps with semantically corresponding grasps from four dexterous robot hands across 2.1K sequences. Shared 3D hand, object 6D pose, robot-state, touch, and success interfaces support both cross-embodiment grasp-intent transfer and geometric perception under interaction occlusion.

Method inputs and outputs

The main product is a dataset, not a unified online input/output covering all experiments. Capture and downstream methods are distinguished below.

Item	Content (compiled for this site)
Capture inputs	23 synchronized RGB streams, calibration, CAD, MANO / robot kinematics, robot joint states during paired grasping, and touch on supported platforms.
Final outputs	Aligned images, 3D hand / robot states, rigid object poses, touch / success labels, and human-robot pairings, not a control-action policy.
Training inputs	Contact transfer uses paired human / robot grasp representations and object clouds; retrieval uses paired hand-object geometry. Perception baselines use their own images / annotations, not automatically every capture sensor.
Training objectives	Contact transfer supervises robot pointwise contact / part maps; contrastive retrieval aligns paired embeddings; hand / object perception supervises its own 3D outputs. Capture itself has no unified training loss.
Inference inputs	Contact transfer reads human contact / part maps and object clouds; retrieval reads human hand-object geometry and a candidate library; perception conditions depend on the baseline.
Inference outputs	Robot contact / part maps or ranked candidates; executable grasps still need CEDex / BODex optimization backends, beyond contact prediction.
Output details	Object pose is a rigid transform in calibrated space; check camera / global coordinates and length units in data files. Contact probability is a scalar on each of $N$ object points; part maps encode target-hand regions. These modalities do not form one 6D vector.
Prerequisites	Multiview images, CAD, and kinematics mainly support annotation; allowed test modalities depend on the task protocol. Pairing expresses grasp-intent correspondence, not identical framewise joints.

Basis: Capture system and Section 4 downstream tasks.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Dataset overview	100 objects, 5 embodiments, 23 RGB views, and multimodal annotations jointly supply value
Table 1	HOI / ROI / HROI comparison	Combines paired human-robot data, dexterous hands, object 6D poses, touch, and dense multiview capture
Figure 2	Capture and reconstruction	3D hands / object poses combine multiview constraints and pretrained models, not direct sensor readings
Figure 3, Table 2	Contact transfer	Robot-specific contact representations outperform directly applying human maps in real grasping
Figure 4, Tables 3—4	Cross-embodiment retrieval	Tests pairing recognition and BODex initialization; top-1 / top-5 trade precision against coverage
Tables 5—6	3D hand estimation	HRDexDB is harder than FreiHAND; small mixed-training additions provide complementary signals
Tables 7—8	Object 6D pose	All methods degrade on robot grasps; HRDexDB refiner fine-tuning helps an independent robot environment

3. What does paired mean here?

3.1 Five embodiments

HRDexDB covers humans and four dexterous robot hands:

• Allegro Hand V4
• Allegro Hand V5 Plus
• Inspire Hand RH56DFTP
• Inspire Hand RH56F1

Their DoFs, link lengths, reachable contact regions, joint limits, and sensors differ. Direct MANO-to-robot retargeting often causes penetration, unreachable configurations, or unstable grasps, so identical posture is not the pairing criterion.

3.2 Semantic pairing protocol

Each pair is captured in two stages:

1. A human naturally grasps the given object;
2. A teleoperator observes it and reproduces a semantically corresponding grasp on the target hand.

Pairing retains intent and function, such as stable side grasping of a cup, rather than identical joints or point contacts. Different morphologies rarely admit exact kinematic isomorphism. This sensible choice also limits supervision strength: pairing labels alone do not define mathematical functional equivalence.

3.3 Signals in each sequence

A human sequence can be summarized as:

$$\mathcal H = \{I_t^{1:23},\ \theta_t^{MANO},\ T_{O\rightarrow C,t},\ y\}_{t=1}^{T},$$

A robot sequence as:

$$\mathcal R = \{I_t^{1:23},\ q_t,\ T_{O\rightarrow C,t},\ F_t,\ y\}_{t=1}^{T}.$$

Here $I_t^{1:23}$ denotes synchronized RGB from 21 external and 2 ego cameras; $\theta^{MANO}$ is human pose, $q_t$ robot joints, $T_{O\rightarrow C,t}\in SE(3)$ the rigid object-to-camera / calibrated-global transform, $F_t$ available platform touch, and $y$ the success / failure label.

4. Capture system

4.1 Multiview and egocentric capture

• Camera racks on three sides carry 21 calibrated external RGB cameras.
• A custom stereo helmet captures human ego views.
• Robot over-the-shoulder stereo approximates the acting agent’s viewpoint.
• RGB resolution is 2048 × 1536; Table 1 reports about 24M total frames.

External views mainly support reconstruction / annotation; ego views support perception / policies closer to deployment. These are different input protocols: ego-only models cannot treat external 21-view annotation evidence as test information.

4.2 Robot teleoperation

Xsens inertial motion capture and MANUS gloves record operator upper-body / hand motion for robot mapping. Operators reimplement grasp intent within target kinematics and contact capabilities rather than mechanically copying the demonstration.

This has two effects:

• Pairings retain embodiment-specific adaptations useful for cross-embodiment representations;
• Operator experience shapes the data; successful samples depend on teleoperation strategy.

4.3 Interpreting scale

Statistic	Value	Meaning
Objects	100	Fixed instances with high-quality meshes / CAD
Sequences	About 2.1K	Human and robot grasp sequences combined
Embodiments	5	1 human + 4 robot hand types
RGB views	23	21 external + 2 ego
Frames	About 24M	Image total after expanding multiple views

The 24M frames include simultaneous cameras, not 24M independent action states. Group training splits by sequence, object, and paired episode to keep neighboring views of the same moment from leaking across train / test.

5. Annotation and reconstruction

5.1 Human-hand 3D reconstruction

1. HaMeR predicts 2D hand keypoints in each view;
2. Calibration enables multiview triangulation into 3D joints;
3. Optimize MANO pose to align projected keypoints with triangulated geometry;
4. SAM3 hand-mask silhouettes estimate subject-specific shape;
5. Temporal filtering reduces framewise jitter.

Markerless capture preserves natural contact, but HaMeR, SAM3, triangulation, and MANO errors accumulate. Long-occluded fingertips depend on model priors as well as image evidence.

5.2 Object 6D pose

Object-pose pipeline:

1. FoundationStereo predicts depth from a designated stereo pair;
2. SAM3 supplies object masks;
3. CAD, RGB-D, and masks enter FoundationPose for first-frame global registration and later tracking;
4. Render the mesh at current $T_{O\rightarrow C}$ into every calibrated view;
5. Minimize multiview silhouette mismatch to reduce single-view drift.

An abstract cross-view refinement objective is:

$$\mathcal L_{sil}(T)=\sum_{v=1}^{21} d\left(\Pi_v(T\mathcal M_O),\ M_v\right),$$

Here $\mathcal M_O$ is the object mesh, $\Pi_v$ the projection for camera $v$, $M_v$ its object silhouette, and $d$ silhouette mismatch. Multiple views reduce occlusion ambiguity but still depend on CAD scale, mask quality, stereo depth, and FoundationPose initialization.

Evidence boundary: the paper supplies object 6D pose annotations and demonstrates downstream utility, but reports no absolute error against external optical motion capture, calibrated robot fixtures, or manual framewise review. These are not independently measured, metrology-grade ground truth.

6. Downstream task one: human-to-robot contact transfer

Direct human-contact grasp targets ignore palm size, finger count, reachability, and joint limits. Predictions on object clouds include:

• Contact probability $C$;
• Contact-part distribution $P$;
• Robot-specific target maps for Inspire-F1 and Allegro-V5.

PointNet++ extracts object features; contact probability uses weighted $L_1$ and part maps use cross-entropy. The fixed physics-aware CEDex optimizer converts predicted robot contact maps into grasps.

Contact target	Inspire sim / real	Allegro sim / real
Human-Contact	54.6 / 66.7	60.2 / 63.3
Transferred	55.6 / 73.3	65.8 / 80.0

Simulation gains are small; real Allegro improves most, by 16.7 percentage points. This supports morphology-specific contact representations, but only 60 Inspire and 30 Allegro real trials are reported. Main tables lack confidence intervals / objectwise detail, so the aggregate ratio is not universal success.

7. Downstream task two: cross-embodiment grasp retrieval

A CLIP-style multi-branch cloud model uses separate human, Inspire-F1, and Allegro-V5 encoders with a shared object encoder. Paired human-object / robot-object grasps project into one space under symmetric contrastive loss.

Retrieval direction	R@1	R@3	R@5
Human → Inspire	36.36%	81.82%	100.00%
Human → Allegro	24.24%	63.64%	72.73%
Inspire → Allegro	8.18%	57.58%	72.73%

This uses 33 grasp candidates, not a large image-retrieval library. High R@5 can coexist with unstable top-1 ranking; Inspire → Allegro R@1 is only 8.18%, showing no automatic shared representation even between robots.

Retrieved grasps initialize BODex refinement on 7 unseen objects, 33 episodes, and 50 optimization seeds per episode:

• Retrieval-top1 has best seed-level success: Inspire 12.24%, Allegro 21.33%;
• Retrieval-top5 has best episode-level success: Inspire 75.76%, Allegro 93.94%.

Top-1 gives a stronger individual prior; top-5 uses diversity to improve at-least-one-success coverage. This is a clear precision—coverage trade-off.

8. Downstream task three: 3D hand-pose benchmarks

Model	HRDexDB PA-MPJPE / PA-MPVPE	FreiHAND	Difference
WiLoR	5.94 / 6.09	5.71 / 5.27	+0.23 / +0.82
HaMeR	6.15 / 6.16	6.11 / 5.72	+0.04 / +0.44
Hamba	6.11 / 6.10	6.14 / 5.84	-0.03 / +0.26
MeshGraphormer	8.31 / 8.10	6.64 / 6.78	+1.67 / +1.32
FrankMocap	10.61 / 12.48	9.52 / 11.64	+1.09 / +0.84

HRDexDB is harder overall, especially for mesh vertices, consistent with occlusion, object contact, and complex hand shapes. The authors then mix 6K HRDexDB samples into about 2.7M fine-tuning samples:

• HaMeR: PA-MPJPE 6.108 → 6.027, PA-MPVPE 5.718 → 5.679;
• WiLoR: PA-MPJPE 5.711 → 5.677, PA-MPVPE 5.273 → 5.260.

Small but consistent improvements suggest complementary interaction signals. Since added data is a tiny fraction, this indicates usefulness rather than a full scaling study.

9. Downstream task four: object 6D pose in human / robot grasps

All methods receive RGB + SAM3 masks + intrinsics + CAD and use BOP-style evaluation. FoundPose, GigaPose, and PicoPose localize initially; refined variants pass the top 5 coarse candidates to MegaPose.

Method	Human ADD ↓ / AR_MSSD ↑	Robot ADD ↓ / AR_MSSD ↑	Robot degradation
FoundPose	6.91 / 44.10	8.74 / 33.30	+1.83 cm / -10.80
FoundPose + MegaPose	3.35 / 70.00	4.40 / 64.10	+1.05 cm / -5.90
GigaPose	13.10 / 19.70	13.80 / 17.30	+0.70 cm / -2.40
GigaPose + MegaPose	5.99 / 54.10	8.02 / 49.20	+2.03 cm / -4.90
PicoPose	6.31 / 48.40	8.39 / 38.80	+2.08 cm / -9.60

All methods degrade on robot-grasp frames. Rigid links / fingertips cover object edges and create similar structures. Pairing controls object identity and approximate intent, making the domain gap more interpretable than comparisons across unrelated human / robot datasets.

The authors fine-tune MegaPose’s refiner on 100K GSO synthetic samples and 5.3K HRDexDB robot-grasp annotations, then test in independent OmniRobotHome:

Relative gains are 3.99% ADD and 10.2% ADD-S. Independent testing strengthens evidence beyond internal fine-tuning / testing, but this is refiner adaptation, not equal end-to-end gains for detection, segmentation, coarse pose, and tracking.

10. Do contributions match evidence?

Claim	Evidence	Assessment
Large paired human-robot dexterous-grasp data	100 objects, 2.1K sequences, 24M frames, 5 embodiments	Supported; multiview frames exceed independent action states, which are fewer than 24M
Cross-embodiment grasp transfer	Real contact transfer, retrieval, BODex initialization	Useful proof of concept with limited platforms, objects, and real trials
Interaction-centered 3D perception research	Hand / object 6D pose benchmarks	Task feasibility shown; absolute label accuracy still needs external auditing
Robot hands create harder pose conditions than humans	Consistent degradation across five methods on paired frames	Strong evidence; stratify further by visibility, material, and hand type
Improving external robot domains	MegaPose refiner on OmniRobotHome	Supported for one refiner and one external system

11. Strengths

1. Sensible pairing unit: semantic correspondence replaces invalid joint-to-joint mappings and preserves executable solutions for each morphology.
2. Unified capture / downstream interfaces: calibration, CAD, object poses, and hand / robot states support perception and manipulation from the same sequences.
3. Explicit robot-domain difficulty: pose benchmarks quantify the extra occlusion from rigid hands.
4. Benchmarks span levels: contact targets, retrieval representations, and 3D hand / object perception show utility beyond one network.
5. Substantial resources: projects, code, and public data repositories lower reproduction barriers.

12. Limitations and potential failures

1. Heterogeneous, asymmetric touch: robot-only sensing with differing layouts / ranges makes unified tactile encoders difficult.
2. Semantic pairing only: no exact timewise contacts, phase alignment, or shared functional-equivalence metric; learned correspondence may encode teleoperator preference.
3. Strong annotation-model dependence: SAM3, HaMeR, FoundationStereo, and FoundationPose biases can enter labels. Related models evaluated on them may inherit teacher-induced bias.
4. Limited objects / environments: 100 objects, controlled capture, and grasp-focused tasks do not cover 1000-scale objects, dynamic environments, or multistage functional manipulation; expansion to 1000 is future work.
5. Download / training costs: full multiview RGB is TB-scale; shard by modality / object / sequence rather than copying everything by default.
6. Split leakage: nearby views, episodes, human-robot pairs, and same-object trajectories correlate strongly; random frames exaggerate generalization.
7. Limited real success samples: contact transfer uses two robot hands and tens of trials; objectwise intervals and more repetition are needed.

13. Concrete implications for 6D pose

13.1 From object-centered to interaction-centered benchmarks

Unlike ordinary tabletop / clutter BOP data, grasped objects couple occlusion structure and motion. Robot geometry creates object-like rigid edges, causing systematic errors even for mask-conditioned estimators.

13.2 Additional conditions to exploit

• Robot joint state: render known hand kinematics and explicitly model occlusion instead of treating the hand as unknown clutter.
• Paired human sequences: easier human-frame poses can teach same-object geometry under similar intent.
• Time: more visible pre-grasp objects provide pose and continuity priors for tracking / refinement after occlusion.
• Touch: contacts constrain surfaces relative to fingers, complementing visual silhouettes / depth residuals.
• Multiview labels, single-view deployment: consistent 21-view supervision at training with ego / one external test view naturally uses privileged information.

13.3 Additional evaluations worth doing

• Stratify pose by visibility, hand type, symmetry, material, and contact region;
• Audit sampled object labels using independent external measurements;
• Quantify occlusion-modeling gains from known hand meshes / joints;
• Enforce object-, operator-, and embodiment-held-out splits;
• Separate first-frame global localization, tracking, and recovery after failure.

14. Usage recommendations

1. Start with metadata, meshes, and low-rate / few-view subsets; check frames, units, timestamps, and object IDs.
2. Keep paired episodes indivisible, placing linked human / robot sequences in the same split.
3. Add object-held-out testing for unseen-object generalization rather than framewise random splits.
4. Check reprojection, mesh scale, and continuity across views; filter visibly drifting pose segments before training.
5. Report actual inference modalities, distinguishing 21-view annotation from deployment input.
6. Check noncommercial license compatibility before releasing models or derived data.

15. Summary

HRDexDB builds a paired interface for two connected questions:

• Manipulation: how human grasp intent transfers across robot morphologies;
• Perception: why 3D hand / object 6D pose degrades under different hands’ occlusion, and how to adapt.

Multiview annotation and four benchmark types make both directions actionable. Keep semantic pairing distinct from exact trajectories and model-generated labels distinct from independently measured truth. Its most valuable use explicitly exploits human-robot pairs, robot geometry, time, and touch rather than merely adding 24M RGB frames to ordinary training.