Deep Reading Notes: Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds

1. Basic information

• Title: Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds
• Method: PPF-Tracker
• Authors / affiliations: Xianhui Meng, Yukang Huo, Li Zhang, Liu Liu, Haonan Jiang, Yan Zhong, Pingrui Zhang, Cewu Lu, Jun Liu; multiple Chinese universities and Shanghai AI Lab
• Year / venue: 2026 / AAAI-26 Technical Tracks 10
• Paper: https://ojs.aaai.org/index.php/AAAI/article/view/37747
• PDF: https://ojs.aaai.org/index.php/AAAI/article/download/37747/41709
• Code: https://github.com/mengxh20/PPFTracker
• Reading mode: Deep Dive

2. Positioning in one sentence

The paper reformulates category-level articulated pose tracking as learning $se(3)$ increments between consecutive frames: keyframe-based quasi-canonicalization precedes weighted PPF estimation of SE(3)-invariant part increments, followed by structured refinement with geometric and joint energies.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Framewise prediction, inter-frame fusion, and the proposed approach	The main contribution combines temporal information, keyframes, and incremental updates
Figures 2–4	Overall architecture, temporal segments, and quasi-canonicalization	Understand how frames in each segment share a keyframe transformation
Figures 5–6	Weighted point pairs and voting	Inspect how geometric point pairs produce translation / orientation votes
Tables 1–4	PM-Videos, ReArt-Videos, and RobotArm-Videos results	Read accuracy, scale IoU, and inference time together
Table 2	Kinematic constraint and keyframe selection ablations	Direct evidence on drift reduction and structural consistency

3. Problem setting and task boundaries

• Let the object have $K$ rigid parts and $J$ joints. Inputs are a point cloud sequence $\{P_t\}_{t\ge 0}$ and initial poses $T_0^k$.
• Outputs are $T_t^k\in SE(3)$ and scale $s_t^k\in\mathbb{R}^3$ for each part in every frame.
• The basic tracking relation is: $T_t^k=\Delta T_t^kT_{t-1}^k,$ Here, $\Delta T_t^k$ is the current frame’s increment relative to the preceding frame.
• The articulated structure is assumed known or definable, and initial poses are available; this is not category-level detection without initialization.
• Input may be point clouds / RGB-D. Evaluation focuses on per-part poses across frames, rather than one rigid pose for the entire object.

4. Research context

• Earlier category-level pose methods focus on single-frame canonicalization, segmentation, or keypoints; for articulated objects, these can overlook part motion and temporal continuity.
• A-NCSH, CAPTRA, ContactArt, and GAPS are the main comparisons. PPF-Tracker combines point-pair invariance, temporal keyframes, and kinematic refinement.
• PPFs are invariant to rigid transformations, but conventional PPFs weight all pairs equally; weighted PPFs increase the contribution of discriminative pairs.
• Tracking is formulated on a Lie group rather than regressing rotation matrices as Euclidean vectors, aiming to avoid orthogonality and numerical stability issues.

5. Research motivation

The authors’ central hypotheses

1. Temporal continuity of part poses should be an explicit prior.
2. Keyframes divide long sequences into locally stable temporal segments, reducing accumulated error.
3. Joint axes constrain some articulated motions; optimizing parts independently can produce physically inconsistent results.

My analysis

The contribution extends beyond a PPF variant: geometric invariance, temporal memory, and structural constraints form a hierarchical tracking system. It resembles an interpretable optimizer / tracker more than a fully end-to-end black-box estimator.

6. Method details

6.1 Overall pipeline

1. Divide the sequence into temporal segments using dynamic keyframes.
2. Transform each segment’s point clouds into quasi-canonical space using its keyframe transformation.
3. Sample point pairs, compute weighted PPFs, and use PointNet++ to predict each part’s SE(3)-invariant parameters.
4. Recover translation / orientation through point-pair voting and predict scale.
5. Represent increments in Lie algebra and apply the exponential map to obtain $\Delta T_t^k$.
6. Optimize part poses using geometric and joint kinematic energies.
7. When the current prediction agrees sufficiently with the observed point cloud, the dynamic selector makes this frame the next segment’s reference.

6.2 Key modules

Module	Input	Output	Role	Connection to the next module
Quasi-canonicalization	Current point cloud, keyframe pose	Quasi-canonical point cloud	Align a local segment to a stable reference	Feeds PPF voting
Dynamic Keyframe Selection	Predicted and observed point clouds	Keyframe update decision	Chamfer + Hausdorff energy determines when to update	Sets the next segment’s reference
Weighted PPF	Sampled point pairs	Invariant features / votes	Encode local geometry and suppress low-value pairs	PointNet++ predicts parameters
Lie algebra increment	PPF predictions	$\Delta\xi\in se(3)$	Model rotation and translation increments in tangent space	Exponential map returns to $SE(3)$
Kinematics-aware optimization	Part poses, joint axes	Refined part poses	Minimize geometric and joint consistency energies together	Produces final per-part poses

6.3 Training and supervision

• Pointwise voting uses KL divergence to supervise translation / orientation; scale uses MSE and masks use BCE.
• The total loss is: $L_{total}=0.3L_{trans}+0.3L_{orient}+0.2L_{scale}+0.2L_{mask}.$
• Kinematic refinement combines geometric energy $E_{geo}$ and joint energy $E_{kin}$ between adjacent parts as $E_{comp}=E_{geo}+E_{kin}$.
• Training runs for 200 epochs on an RTX 4090. PM-Videos is reconstructed because the original PartNet-Mobility generation scripts and configurations were not fully released.

6.4 Inference and engineering complexity

• The network estimates the current increment; geometric / kinematic optimization performs structured postprocessing.
• Dynamic keyframes reduce accumulated errors but add Chamfer / Hausdorff computations and the engineering hyperparameter $\phi=0.01$.
• Reported speed is about 0.06–0.16 s/frame depending on category and part count; this is not a high-frame-rate tracker, though it approaches interactive operation.
• Reproduction challenges include PM-Videos / ReArt-Videos generation, part and joint annotations, point-pair sampling, and energy optimization iteration counts.

7. Experimental analysis

7.1 Datasets

• PM-Videos: synthetic category-level articulated tracking dataset built from PartNet-Mobility.
• ReArt-Videos: semisynthetic tracking dataset based on ReArt-48.
• RobotArm-Videos: real robot / real-world videos including Box, Stapler, Cutter, Scissors, and Drawer categories.

7.2 Main results

On PM-Videos, PPF-Tracker generally achieves the lowest rotation / translation errors or highest 3D IoU across Laptop, Eyeglasses, Dishwasher, Scissors, Drawer, and other categories. For Eyeglasses, mean rotation error is about 3.3°, translation error about 0.036 m, and inference about 0.12 s/frame. RobotArm-Videos rotation errors span about 2.26°–19.43° and translation errors about 0.002–0.155 m, showing substantial category difficulty differences.

7.3 Ablation studies

Ablation	Result	Strength of evidence
No kinematic constraints	Rotation about 8.6/9.1°, translation about 0.109/0.122 m	Compared with full model 3.2/3.4°, 0.038/0.045 m, supports joint constraints
No keyframe	Largest errors	Supports local references for long-term tracking
Fixed keyframe	Better than no keyframe, weaker than dynamic	Supports flexible dynamic updates
Dynamic keyframe	Best	Threshold sensitivity and cross-object generalization need further study

7.4 Robustness and failures

• Semisynthetic and real-world results demonstrate some synthetic-to-real generalization.
• Kinematic constraints suppress independent part drift, but incorrect joint axes or extensive part invisibility may propagate errors through those constraints.
• Large initialization errors, unknown joint topology, transparent / reflective materials, and fast nonrigid motion are not systematically studied.

8. Critical analysis

Strengths

• Pose increments in $se(3)$ avoid the geometric issues of direct rotation matrix regression.
• Keyframes, motion increments, and joint energies offer clear paths for interpreting errors.
• Experiments cover synthetic, semisynthetic, and real-world settings and report speed.

Weaknesses and implicit assumptions

• Strong dependence on initial poses, part structure, and joint information limits fully open settings.
• Reconstructed PM-Videos may differ from previous works’ generation protocols; fair comparison requires rerunning code.
• Kinematic optimization can improve accuracy but may also hard-code incorrect structural priors into outputs.

Practical feasibility

Suitable for continuous robotic tracking of initialized articulated objects. Detection from scratch needs additional segmentation, initial pose proposals, and structural recognition.

9. Future work and possible improvements

• Unify initial pose proposals, part segmentation, joint discovery, and tracking.
• Use uncertainty-aware PPF votes for occlusion and missing local geometry.
• Learn a keyframe policy instead of using a fixed energy threshold.
• Combine differentiable kinematic solvers with end-to-end training.
• Evaluate failures under rapid motion, joint friction, soft parts, and unknown topology.

10. Implications for my research

• Dynamic keyframes and $se(3)$ increments could stabilize object pose sequences.
• For multi-object tracking, separate local object trackers from scene-level association.
• Report framewise error, accumulated drift, per-part consistency, and speed rather than only single-frame ADD / IoU.

11. Open questions

• How large can initialization errors become before tracking fails?
• Must the dynamic keyframe energy threshold be adapted to category and point cloud density?
• Are joint axes known during training or estimated from input?
• How should a fair comparison with RGB-only articulated tracking be configured?