Deep Reading Notes: Learning Distributional Demonstration Spaces for Task-Specific Cross-Pose Estimation

1. Basic information

• Authors / affiliation: Jenny Wang, Octavian Donca, David Held; Carnegie Mellon University
• Year / venue: 2024 / ICRA
• Paper: arXiv / PDF
• Project page: TAX-PoseD
• Reading mode: Deep Dive

2. Core Problem

Relative-placement tasks often have multiple correct SE(3) solutions: a mug can hang on either peg, or a plate can occupy any placemat. Unimodal regression averages modes into a pose between support points. TAX-PoseD learns a distribution of valid cross-poses.

3. Absolute Pose, Relative Pose, and Cross-Pose

Given action object $A$ and anchor object / scene $B$, predict

$$T_{AB}=f(P^A,P^B)\in SE(3),$$

so that $T_{AB}T^A$ and $T^B$ satisfy the relationship defined by demonstrations. Applying one shared world transform to both objects should preserve that relationship, so the network should focus on inter-object geometry rather than world coordinates. Inputs are segmented point clouds, without object identities or manual peg-point labels.

4. Method details

4.1 Why Ordinary cVAE Latents Are Insufficient

A standard cVAE models $z$ as a continuous Gaussian vector. For discrete placement modes, this global vector lacks an inductive bias locating modes in the scene, making smooth intermediate decoder solutions likely.

TAX-PoseD represents $z$ as a categorical distribution on the point cloud, directly selecting / weighting spatial locations. Mug–rack prior peaks can lie near each peg, making them interpretable and adaptable to rack counts and positions.

4.2 Encoder, Prior, and Decoder

• Posterior encoder $q_\psi(z\mid X,Y)$ sees current scene $X$ and successful placement $Y$, learning which spatial mode the demonstration used.
• Learned prior $p_\phi(z\mid X)$ sees only inference-available point clouds and predicts all feasible modes for the current geometry.
• Decoder $p_\theta(T_{AB}\mid X,z)$ uses TAX-Pose relational point features to output cross-pose.
• The ELBO fits demonstrations with reconstruction / pose loss while $KL(q_\psi\|p_\phi)$ encourages the prior to cover demonstrated modes.

Rotation-invariant dense features communicate latent locations to the decoder instead of directly concatenating sparse one-hot maps.

5. Experiments

Four depth cameras generate PyBullet point clouds. Each task provides only 10 demonstrations; testing uses unseen mugs and random initial poses. The robot first predicts a gripper–mug cross-pose for grasping, then a mug–rack cross-pose for placement.

Method	1 Rack	2 Racks	3 Racks (trained only on 2 racks)
TAX-Pose	0.84	0.07	0.14
TAX-PoseD	0.82	0.57	0.59

Unimodal performance is largely retained, while multimodal tasks avoid TAX-Pose’s averaging into the midpoint between racks. More compelling is 3-rack generalization: a pointwise prior can assign a new probability peak to an unseen third rack at inference.

Continuous $z$ ablations reach at most 0.41 / 0.29 success on 2 / 3 racks. Spatial $z$ + uniform prior reaches 0.60 on 2 racks but falls to 0.18 on 3. The full learned prior obtains 0.57 / 0.59: spatial representation locates modes, while the learned prior identifies genuinely feasible points.

6. Limitations and Research Implications

• Main experiments focus on mug hanging, with limited task and physics complexity.
• Clean segmented point clouds are supplied, without modeling perception uncertainty.
• Success depends on grasping, motion planning, and physical stability rather than a pure pose metric.
• Limited reporting of distribution calibration, mode coverage, and sampling efficiency.
• Three-rack generalization remains within similar geometry, without establishing cross-task composition.

For 6D pose research, symmetry and multiple solutions need not always collapse into a canonical pose. Manipulation can instead output a task-conditioned pose distribution and let the planner choose a solution using reachability, collision, and uncertainty.