MOTPose: Multi-Object 6D Pose Estimation for Dynamic Video Sequences Using Attention-Based Temporal Fusion
Within a DETR-style multi-object network, MOTPose's TEFM fuses historical object embeddings and TOFM fuses keypoints with pose parameters. Compared with its single-frame version, it reduces missed detections and pose outliers on dynamic SynPick while retaining competitive accuracy on YCB-Video.
MOTPose introduces two levels of temporal cross-attention inside a DETR-style multi-object pose network. TEFM fuses historical object embeddings, while TOFM fuses keypoints and pose parameters, allowing detection and pose to recover evidence missing under current-frame occlusion from previous frames.
3. Method overview
Continuous RGB frames produce a fixed-size object set at each step.
Each object predicts class, 2D box, keypoints, rotation, and translation jointly.
Relative Frame Encoding indicates how far each historical feature is from the current frame.
TEFM handles semantic embeddings; two TOFMs handle keypoints and pose separately.
Hungarian matching combines with classification, box, keypoint, symmetry-aware pose, and temporal-consistency losses for joint training.
Original Figure 2: cross-attention fuses current object queries with historical embeddings / object parameters. Source: paper PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
4. Main results
On SynPick, temporal fusion raises AUC ADD-S from 80.8 to 82.0 and AUC ADD(-S) from 74.2 to 77.1.
Full-test-set cardinality error drops from 2.06×10⁻² to 0.53×10⁻², and false-negative rate from 1.79×10⁻² to 0.48×10⁻².
Ablations find TEFM more important than TOFM alone on pose outputs.
5. Coarse-reading assessment
MOTPose makes object existence and identity depend on temporal evidence, beyond post-hoc trajectory smoothing. It suits dynamic bin videos with known YCB instances, requiring continuous frames, complete training-time 6D annotations, and an initial warm-up to establish history.
Available src/paper-notes/motpose/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Paper Reading Notes (Deep): MOTPose: Multi-Object 6D Pose Estimation for Dynamic Video Sequences Using Attention-Based Temporal Fusion
1. Basic information
Authors / affiliation: Arul Selvam Periyasamy, Sven Behnke; University of Bonn
Single-frame bin models may miss objects or produce pose outliers under transient occlusion. Rather than explicitly associating tracks, MOTPose treats each frame’s fixed-size prediction set as temporal tokens and retrieves historical evidence through cross-attention at embedding, keypoint, and pose levels.
3. Task setting
Item
Setting
Input
Monocular RGB video, using the current frame and up to 7 historical frames
Objects
21 known YCB instances, allowing multiple instances in a scene
Output
Detection set + 6D pose
CAD
Required for training / pose loss and evaluation
Explicit track ID
Tracking metrics are not the objective
Data
Dynamic SynPick and YCB-Video
SynPick contains synthetic bin-picking videos with moving objects and grippers; YCB-Video mainly has a moving camera and static multi-object scenes. Both test temporal fusion, with different scene dynamics.
Original Figure 2: TEFM aggregates historical object embeddings, while TOFM separately fuses keypoints and pose parameters; relative frame encoding preserves temporal order. Source: paper PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
4. Method details
4.1 Set-prediction backbone
A DETR-like encoder–decoder produces N 256-dimensional object embeddings per frame. N is set by the maximum object count: 30 for SynPick and 20 for YCB-Video. Each token predicts class, box, a 32-dimensional keypoint representation, 6D rotation representation, and translation. Training-time Hungarian matching uses only class and box, preventing unstable early poses from disrupting assignments.
4.2 Two-level temporal fusion
TEFM: current embeddings query historical T−1-frame embeddings as keys / values, learning which past objects supply information to the current occluded object token.
TOFM: project low-dimensional keypoints, rotation, and translation to 256 dimensions, add relative frame encoding, cross-attend, and project back to parameter space.
RFE: 1D sinusoidal relative-frame encoding explicitly represents temporal distance, preventing attention from treating frames as an unordered set.
This is prediction-level fusion rather than optical-flow warping: structurally simple, with correspondence learned implicitly by attention.
4.3 Training objectives
The total loss includes class NLL, GIoU + ℓ1 box loss, keypoint ℓ1 + cross-ratio consistency, translation ℓ2, symmetry-aware ShapeMatch rotation loss, and temporal consistency of neighboring object embeddings. Symmetric objects are not forced toward a unique rotation.
5. Experimental results
SynPick originally has 240 training / 60 test videos per action. The authors add 300 training videos each for move, targeted-pick, and untargeted-pick, calling the result SynPick-Ext. Evaluation excludes objects below 30% visibility.
Original Tables I–II: pose AUC for 21 YCB objects on SynPick and improved detection-set cardinality with temporal fusion. Source: paper PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
SynPick metric
Without temporal fusion
MOTPose
AUC ADD-S
80.8
82.0
AUC ADD(-S)
74.2
77.1
AUC ADD-S @0.1d
77.2
79.1
AUC ADD(-S) @0.1d
60.8
63.4
Cardinality error ×10−2
2.06
0.53
False negative ×10−2
1.79
0.48
Absolute pose-AUC gains are modest, but misses fall to about one quarter of the original rate. For bin picking, this matters beyond pose error on detected objects alone. The temporal model in Figure 4 also reduces isolated severe pose outliers.
Original Figure 4 / Table III: missed detections and pose outliers without temporal fusion, and improvements using historical frames. Source: paper PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
Removing TEFM gives AUC ADD(-S) 74.9 versus the full model’s 77.1, the largest drop. Removing TOFM reduces ADD-S by about 1.8 and ADD(-S) by 0.9. This supports temporal consistency in high-level object representations over directly smoothing numerical pose outputs.
6. Limitations and implications
Requires dense video 6D annotations, among the costliest real-video labels.
Fixed historical windows increase GPU memory and require warm-up for the first T−1 frames.
No explicit motion / camera geometry; rapid changes may exceed learned temporal priors.
Synthetic SynPick differs from real dynamic bin picking in materials, motion blur, and occlusion.
Prediction-set association is not formal track identity and does not establish long-term identity preservation under MOT metrics.
Extensions include pose-aware memory, SE(3)-based temporal prediction, visibility uncertainty, and synthetic-to-real video adaptation to reduce continuous 6D labeling. The key lesson is that temporal evidence can recover the detection set itself as well as smooth poses.
