QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects
QORT-Former uses separate left-hand / right-hand / object queries, a query proposal loss, and contact features to jointly estimate both hands and object 3D poses from one RGB image in real time, while supporting interaction recognition.
hand-object image croptwo-hand and object annotations during training
Reading Notes
Overview / Deep Dive
Available src/paper-notes/qort-former/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
Overview: QORT-Former
One-sentence summary
QORT-Former replaces a large query budget with a small set of partitioned hand-object queries and uses contact features to jointly predict the 3D poses of two hands and an object, targeting real-time interaction understanding.
Task boundaries
Input: a single RGB image; video tasks run frame by frame.
Output: 3D joints for both hands, object 3D keypoints / pose, and support for downstream interaction recognition.
This is not a BOP-style rigid CAD pose benchmark; it measures joint / object end-point error during hand-object interaction.
Method at a glance
Separate queries for the left hand, right hand, and object, with a proposal loss to prevent queries from drifting into the background or favoring one hand.
The three-step decoder performs enhanced feature update, location-based feature extraction, and query update.
Contact map features provide spatial relationship cues for joint hand-object reasoning under occlusion.
Experimental anchors
H2O MEPE: left / right / object = 20.1 / 19.9 / 32.9 mm.
FPHA: right / object = 14.2 / 18.8 mm.
With 108 queries and one decoder, the paper reports 53.5 FPS (RTX 3090 Ti); H2O interaction Top-1 is 91.3.
QORT-Former treats poorly focused queries and heavy decoders as bottlenecks to real-time hand-object pose estimation: it partitions queries among the two hands and object, constrains their locations, and combines a lightweight decoder with contact maps to recover 3D poses.
2.1 Guide to the paper’s figures and tables
Figure / table
Reading focus
Figure 2
Connections between encoder features, hand-object queries, the three-step decoder, and 3D keypoint / contact outputs
Figure 3
Unpartitioned queries tend to favor the left hand; the proposal loss improves spatial coverage rather than merely adding classification
Tables 1 / 3 / 4
Read MEPE, FPS, and query / contact ablations together rather than focusing only on accuracy
Excerpt from the architecture page: partitioned hand-object queries and the three-step decoder. Source: official AAAI PDF. Image copyright belongs to the authors and publisher; reproduced here for paper analysis.Excerpt from the H2O qualitative results page: RGB, queries, contact maps, and final 3D poses. Source: official AAAI PDF. Image copyright belongs to the authors and publisher; reproduced here for paper analysis.
3. Task boundaries
A single RGB image is the input; outputs include 21 3D joints for each hand, object 3D keypoints / pose, and contact relationships.
H2O and FPHA evaluate keypoint mean end-point error in mm, rather than BOP ADD(-S), VSD, or full CAD 6D pose rankings.
Interaction recognition is an extension experiment: replace the upstream pose estimator and measure downstream Top-1 accuracy and speed.
4. Method details
Component
Mechanism
Purpose
Hand-object query division
Reserve query subsets for left hand / right hand / object
Prevent similar hand features from competing for the same queries
Query proposal loss
Supervise positions / classes for each query type
Place queries in target regions and reduce background token waste
Retain image-query interaction with fewer decoder layers
Contact map loss
Predict contact relationships at hand mesh vertices
Constrain relative hand-object spatial relationships under occlusion
The paper uses 108 queries and 1 decoder. Here, “query optimized” means more than reducing the count: without partitioning and proposal supervision, fewer queries could simply miss the right hand or object faster.
5. Experiments and numbers
Dataset
Metric / reported result
Comparison
H2O
Left / right / object MEPE = 20.1 / 19.9 / 32.9 mm
Cho et al. report 24.4 / 25.8 / 45.2 mm in the same table
FPHA
Right / object MEPE = 14.2 / 18.8 mm
Cho et al. report 15.0 / 21.0 mm in the same table
Speed
53.5 FPS (RTX 3090 Ti)
The full model is slightly slower than the version without LFE, but more accurate
Interaction
H2O Top-1 = 91.3
Replaces H2OTR’s upstream pose estimator; the reported comparison is 90.9
On H2O, ablations without hand-object queries yield left / right / object errors of 27.3 / 33.5 / 36.5 mm; partitioned queries give 22.6 / 21.4 / 33.1, and adding contact gives 20.1 / 19.9 / 32.9. Contact benefits the hands more strongly, while also improving the object estimate.
6. Critical analysis
Strengths: real-time performance is an explicit design objective; query partition failure modes are visualized; speed, accuracy, and downstream interaction recognition are all reported.
Limitations: fixed object keypoints differ from full 6D pose estimation for arbitrary CAD objects; multiple hands / objects, severe motion blur, and substantially out-of-category objects are outside the main experimental focus; 53.5 FPS depends on the GPU and frontend implementation.
Takeaway: allocate the computational budget to entities (hands / object) before modeling interactions across them. For industrial manipulation, contact maps could be replaced by grasp affordances or collision constraints, with confidence retained for each entity’s queries.
