Paper Reading Notes (Deep): InterTrack: Tracking Human Object Interaction without Object Templates

1. Basic information

• Method: InterTrack
• Authors / affiliations: Xianghui Xie, Jan Eric Lenssen, Gerard Pons-Moll; University of Tübingen, Tübingen AI Center, Max Planck Institute for Informatics
• Year / venue: 2025 / 3DV
• Paper: https://arxiv.org/abs/2408.13953
• PDF: https://arxiv.org/pdf/2408.13953
• Project: https://virtualhumans.mpi-inf.mpg.de/InterTrack/
• Reading mode: Deep Dive

2. Positioning in one sentence

InterTrack combines human correspondence, human-conditioned object rotation, shared canonical shape, and contact refinement to turn independent monocular human-object clouds into template-free, temporally consistent interaction trajectories.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 2	CorrAE, TOPNet, and joint contact refinement	Human states condition object rotation; the tracks are coupled
Table 1	Template-free BEHAVE / InterCap F-score@1 cm	Compare HDM / Ours under synthetic-only and full training
Table 2	Template-based CHORE / VisTracker comparisons	Template-free and template settings have different conditions
Tables 3–5	TOPNet, joint optimization, CorrAE ablations	Test the need for time and correspondence

3. Problem setting and boundaries

• Input: monocular RGB interaction sequences; HDM first supplies coarse human / object point reconstructions, followed by crops and visual features.
• Output: sequence-consistent SMPL parameters, canonical object shape $O$, per-frame $R_i,t_i,s_i$, and contact-aware refined states.
• Without object templates means no known CAD mesh at test time. Priors remain: SMPL, DINOv2, HDM geometry, ProciGen-Video category training, and real data.
• Main metrics are reconstruction F-scores rather than independently labeled rotation / translation errors. Pose gains are supported mainly by consistent surfaces / shape.

4. Challenges and approach

Image-based HDM clouds lack temporal correspondence; occlusion changes local shape, scale, or orientation. InterTrack first maps humans to canonical SMPL vertices, predicts object rotations to canonicalize object points, then links both trajectories through contact rather than directly registering every cloud pointwise.

In my assessment, this decomposition fits structured humans interacting with rigid objects; those priors also define its scope.

5. Method details

5.1 CorrAE: comparable human clouds across frames

PVCNN + MLP maps HDM human clouds to ordered SMPL correspondences. Per-frame SMPL pose $\theta_i$, scale $s_i$, and translation $t_i$ are optimized with Chamfer distance, pose priors, and second-order acceleration smoothing rather than latent smoothing alone. CorrAE takes about 2.553 s / image versus nonrigid ICP 72.322 s / image.

5.2 TOPNet and shared object shape

TOPNet combines DINOv2 object crops, human pose / joints / velocity, and visibility in temporal windows to predict rotations. Given $R_i$, coarse clouds enter canonical coordinates to jointly optimize one shape $O$ and per-frame translation / scale, sharing shape evidence through occlusion.

5.3 Joint contact optimization

Final contact-based refinement adjusts human / object poses under contact and temporal continuity, reducing interpenetration / drift. Contact is not unconditional truth: fast sliding, soft contacts, or wrong detection can overconstrain it.

Module	Main variables	Role	Failure behavior
CorrAE + SMPL fit	$\theta_i,s_i,t_i$	Stable, interpretable human tracks	Wrong human correspondence corrupts object conditioning
TOPNet	$R_i$	Temporal / human-conditioned rotation	Drift with full occlusion or out-of-domain appearance
Canonical shape optimization	$O,t_i,s_i$	Complete shape from sequence evidence	Deformation violates shared-shape assumptions
Contact refinement	Human-object states	Reduce penetration, retain interaction	Wrong contact assumptions overconstrain

6. Data and experimental analysis

6.1 Role of ProciGen-Video

Combining BEHAVE / InterCap motions with ShapeNet, Objaverse, and ABO shapes gives about 10 hours, 8477 sequences, 2M images, and 10 categories. CorrAE / TOPNet are pretrained there and fine-tuned on real data. Evaluation uses 66 BEHAVE test sequences (about 51k images) and 15 InterCap sequences (about 2k images).

Synthetic data supplies learnable category-level canonical rotation correspondence, beyond extra volume. The synthetic-only / full-training gap still shows domain shift.

6.2 Main results

Dataset	Training	Method	Human F-sc ↑	Object F-sc ↑	Combined ↑
BEHAVE	synthetic only	HDM	0.3574	0.4118	0.4118
BEHAVE	synthetic only	InterTrack	0.3780	0.4658	0.4488
BEHAVE	full	HDM	0.3909	0.5111	0.4622
BEHAVE	full	InterTrack	0.4113	0.5849	0.5169
InterCap	full	HDM	0.4325	0.6267	0.5362
InterCap	full	InterTrack	0.4463	0.6329	0.5555

Retain both findings: InterTrack beats HDM on both datasets / training conditions, supporting temporal joint optimization; synthetic-only remains well below full training, so ProciGen-Video does not eliminate real-video domain gaps.

6.3 Claims and evidence boundaries

• Stable tracking without object templates: directly supported by Table 1, especially improved object F-score.
• Humans help objects: human-dynamics TOPNet inputs and contact-refinement ablations support this, without proving every HOI needs equally strong SMPL priors.
• My analysis: F-score is not a pose metric. Better canonical shape, scale, or completion also improves it. Grasping / AR need independent rotation, translation, and contact-error evaluation.

7. Limitations and failures

1. HDM is a key dependency; poor clouds or scale drift can make optimization consistently wrong.
2. Main settings use rigid objects, one person, one object; multiple people / objects, soft or articulated objects are not covered.
3. BEHAVE / InterCap category overlap shapes generalization; template-free does not mean data-prior-free.
4. HDM, SMPL fitting, transformers, and test-time joint optimization suit offline analysis rather than low-latency tracking.

8. Implications for object pose

• Explore humans as pose conditions through velocity, contact, and visibility in manipulation video.
• Shared canonical shape turns tracking into geometric alignment, ideally with category uncertainty / deformation detection.
• Add pose-specific benchmarks, symmetry handling, and staged HDM / segmentation-noise ablations.