Paper Reading Notes (Deep): Pose-VLA

1. Paper information and scope

• Title: Pose-VLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies
• Authors: Haitao Lin, Hanyang Yu, Jingshun Huang, He Zhang, Yonggen Ling, Ping Tan, Xiangyang Xue, Yanwei Fu
• Affiliations: Tencent Robotics X, Futian Laboratory, The Hong Kong University of Science and Technology, Fudan University, Shanghai Innovation Institute
• Venue / version: repository lists RSS 2026; this reading uses arXiv v3 (2026-07-07)
• Paper: arXiv:2602.19710
• Project: Pose-VLA
• Code: hetolin/PoseVLA
• Weights: Stage-1 VLM; RoboTwin
• Reading mode: Deep Dive

Distinguish three meanings of pose across grounding and policy:

1. Camera-frame object 6DoF for spatial pretraining;
2. Camera-frame gripper waypoints for motion alignment;
3. Continuous deployment actions, not explicit object poses.

2. Positioning in one sentence

Camera-centric tokens connect nonrobot 3D labels and robot trajectories. PaliGemma first learns grounding, then gripper motion, with a flow-matching expert adapting control. Pose prediction is auxiliary pretraining rather than an extra deployment chain.

Method inputs and outputs

Item	Content (compiled for this site)
Inference	RGB / language, optional depth / validity and rays from $K$. Experts initialize action noise; platform-specific proprioceptive fields are not uniformly detailed.
Output	Flow-matched continuous action sequences, not object $R,t$. Structured tokens are read when separately running the pretrained VLM for grounding.
Training	Spatial images / language / geometry with object labels; calibrated gripper trajectories for alignment; robot demonstrations for post-training.
Targets	Categories, anchors, discrete pose / size or trajectory tokens, then action flow fields. GT poses / future actions are supervision rather than online givens.
Details	Quantized physical camera poses / sizes during pretraining; embodiment-specific deployment action spaces. Verify dimensions, horizons, frames, and units instead of treating a pose tuple as control.
Conditions	Pretraining / embodiment adaptation and calibration for rays / camera supervision; depth / rays can be masked and are not universal requirements.

Source: Sections III-C–III-E. Pretraining and final-policy outputs are distinct interfaces.

2.1 Guide to the paper’s figures and tables

Figure / table	Role	Focus
Figure 2	Pipeline	Shared camera tokens precede expert adaptation
Figure 6	Distributions	$x/y$ differs from $z$, motivating split vocabularies / nonuniform bins
Table I / Figure 3	Grounding	AP15 and unseen-scene examples
Tables II–III	Simulation	Visual perturbations and long-horizon suites
Figures 4–5	Real tasks	About 100 demos / task; depth helps drawers
Tables IV–VI	Ablations	Geometry, data, actions, frames, tokenization
Figure 7	t-SNE	Auxiliary support for reduced feature collapse

3. Why ordinary VLM features are insufficient

3.1 Granularity mismatch

VQA emphasizes categories / relations, while manipulation needs small orientation, contact-distance, and handle-motion distinctions. Semantically identical observations can require different actions.

Recognizing targets does not ensure control-relevant 3D features. PaliGemma / expert Easy / Hard 35.4 / 33.4 is presented as an external symptom of feature collapse.

3.2 Representation mismatch

Nonrobot labels use camera object frames, robot data joints / base poses / local deltas. Mixing them creates:

• Robot vocabularies incompatible with object-centric data;
• Implicit extrinsic / kinematic transformations between images and actions.

Camera-centric 6DoF aligns supervision during pretraining, without claiming cameras are always the best control frame.

4. Formalization

Given $\mathbf O$ and instruction $\mathbf L$, autoregressively model $\mathbf S$:

$$P(\mathbf S\mid\mathbf O,\mathbf L) =\prod_{k=1}^{K}P(s_k\mid\mathbf O,\mathbf L,s_{<k}).$$

Structured tuples contain:

$$\mathbf S=(\tau_1,\tau_2,\ldots,\tau_T),\qquad \tau_t=\{c_t,b_t,p_t\}.$$

• $c_t$: category;
• $b_t\in\mathbb R^2$: 2D center / anchor;
• $p_t\in SE(3)$: camera-frame 6DoF.

Static tuples describe objects, temporal tuples grippers. One grammar covers entities and continuous paths as universal geometric primitives.

5. Backbone and explicit geometry

5.1 PaliGemma / SigLIP

PaliGemma-3B / SigLIP retains text and <loc> vocabularies, adding rotation, translation, and size.

5.2 Camera ray encoding

Given intrinsics $K$, ray for pixel $(u,v)$ is

$$r(u,v)=K^{-1}[u,v,1]^\top.$$

Patch-projected rays add to RGB before the first Transformer block: image positions locate pixels, rays specify 3D viewing directions.

This matters for off-center crops / high resolution, where similar appearance has different viewing rays.

5.3 Metric depth prior

Depth is not just $D$, but

$$[D,M]\in\mathbb R^{H\times W\times 2},$$

$M$ marks valid / sparse samples. Unnormalized physical values retain metric alignment; $[D,M]$ is embedded and added to RGB.

Sparse training depths improve handling of varied sensors / missingness.

5.4 Modality masking

Random zeroing makes depth / rays optional. Fair simulation masks both; real results show depth remains useful.

6. Unified pose representation

6.1 Object-level pose tokens

Euler angles use <rot> per principal axis; translation splits statistically:

• $x/y$: shared <trans_xy>;
• $z$: separate <trans_z>;
• Object dimensions: <size>.

$x/y$ concentrates near the optical axis, while $z$ is positive and long-tailed. Uniform bins waste near-contact resolution on sparse distant values.

Inverse-density bin widths make dense regions finer and tails coarser, approximately equalizing token frequency / increasing entropy. <rot>, <trans_xy>, <trans_z>, and <size> each receive $N=2048$ bins.

The main text calls rotation uniform, while the appendix describes spatial tokens as nonuniform. Verify tokenizer edges / checkpoint versions in code.

6.2 Trajectory-level pose tokens

Gripper actions form temporal 6DoF tokens, transformed from dataset base frames to calibrated cameras.

This aligns observed gripper motion rather than joint structure; the expert later handles embodiment control.

6.3 Beyond OpenVLA delta bins

Uniform local delta actions:

• Depend on robot definitions;
• Cannot directly encode nonrobot poses;
• Mismatch dense near-field / sparse far-field precision.

OpenVLA binning reduces Easy / Hard 79.9 / 79.1 to 68.5 / 65.9.

7. Two-stage objectives

7.1 Pre-training: spatial grounding + motion alignment

Standard next-token pretraining:

$$\mathcal L_{pre}(\theta) =-\sum_{k=1}^{K}\log p_\theta\big(s_k\mid\phi(\mathbf O),\mathbf L,s_{<k}\big),$$

$\phi(\mathbf O)$ denotes masked observations, jointly predicting 2D and 3D tokens.

Two dense supervision types:

1. Grounding: categories, anchors, camera pose / size from images / text.
2. Alignment: future camera-frame gripper paths from robot video.

The former teaches geometry, the latter control sensitivity.

7.2 Post-training: flow-matching action expert

A lightweight π0-style expert denoises action states / noise through flow matching, interacting with the VLM by self-attention.

This adapts embodiments rather than relearning geometry from scratch.

7.3 No explicit deployment pose

The paper explicitly restricts pose prediction to auxiliary pretraining. Experts map hidden features to actions without object / camera-path estimates or extra pose decoding.

Unlike modular pose-then-control with $R,t$, implicit features are less replaceable / separately diagnosable, while avoiding that extra latency.

7.4 Expert-initialization inconsistency

The main text says scratch experts without π0 weights; the appendix also has Pose-VLA (π0) using official initialization. Scratch gives 79.91 / 79.10; initialized gives 81.30 / 80.72.

Interpretation:

• 79.91 / 79.10 supports pose pretraining with scratch experts;
• 81.30 / 80.72 adds about 1.4–1.6 points from expert pretraining;
• Name initialization explicitly rather than conflating configurations.

8. Pretraining data and compute

8.1 Nonrobot 3D

1.4 million images, 6.5 million annotations:

Pillar	Sources	Learning
General detection	Omni3D: Objectron, SUN RGB-D, ARKitScenes	Indoor / object-centric box grounding
6D pose	Omni6DPose: SOPE, ROPE	Large-vocabulary real / synthetic poses and size
Manipulation perception	BOP: YCB-V, LineMOD, HOT3D	Workspace geometry / poses

Real / synthetic mixtures vary lighting, materials, scale, and density.

8.2 Robot trajectories

About 1.55 million trajectories:

• AgibotWorld Beta, real: 1 million; 100 identical robots, 200 tasks, 87 skills.
• InternData-A1, simulated: 550K; 4 embodiments including ARX, Aloha, Franka, 227 scenes.

Temporal tokens extend static spatial learning beyond more action samples.

8.3 Training configuration

• optimizer: AdamW;
• schedule: cosine;
• Peak learning rate $5\times10^{-5}$; 1,000 warm-up steps;
• Minimum $2.5\times10^{-6}$;
• precision: bfloat16;
• hardware: 16 × NVIDIA H20;
• batch: 8 / GPU;
• About 2 days each for grounding and trajectory estimation.

These cover two pretraining tasks; downloads, benchmark adaptation, simulation, and real demonstrations add separate costs.

9. Grounding evaluation

9.1 Protocol

• Omni3D SUN RGB-D / Objectron subsets;
• Category-specified text prompts;
• IoU 0.15, AP15 / mAP@0.15;
• Confidence fixed at 1.0 to reduce ranking effects.

AP15 is box grounding, not BOP AP / ADD(-S) / AR. IoU 0.15 is looser than common 2D 0.5, requiring protocol-aware comparisons.

9.2 Interpretation

• Objectron 87.3: object-centric tabletop geometry gives largest gains.
• SUN RGB-D 45.5: above open baselines / VST-SFT / RL, below Gemini Robotics-ER 48.3.
• Qwen3-VL 235B Thinking SUN 34.9 shows scale / generic QA cannot replace pose pretraining.

10. Simulation policies

10.1 RoboTwin 2.0

Settings:

• 50 tasks;
• clean demonstrations: 2,500 (50 / task);
• heavily randomized demonstrations: 25,000 (500 / task);
• Fair comparison: 80K steps, total batch 32;
• 100 trials per task for Easy / Hard;
• RGB-only, depth / rays disabled.

Hard gains 13.98 over π0; relative to π0.5, the gain is 2.94. Easy / Hard differ by only 0.81, highlighting robustness to visual / geometric randomization.

July 2026 extended training uses 100K steps, 24 GPUs × batch 12, chunk 16, giving Easy / Clean 89.4 and Hard / Random 88.6. Multiple simultaneous changes prevent attributing roughly 9.5 points to one design.

10.2 LIBERO

LIBERO has Spatial, Object, Goal, and Long suites, each with 10 tasks and 500 demonstrations within that suite. Results:

Competitive is more accurate than universal SOTA: second mean, no Spatial / Object / Goal column win, but tied best Long and above π0 85.2.

11. Real bimanual experiments

11.1 Platform and tasks

• Dual-arm Xtrainer, 1-DoF parallel grippers;
• Head-mounted Eye-on-Base D455;
• Wrist Eye-on-Hand D405;
• About 100 demonstrations per task;
• 3 seeds × 20 trials, 60 per task.

Tasks:

1. Stack three nesting bowls;
2. Hang mugs on a specified peg;
3. Open drawer, place pen, close;
4. Fold towels.

11.2 Depth

Without depth: 80.0 / 75.0 / 66.7 / 70.0; full: 88.3 / 78.3 / 81.7 / 76.7:

• Stacking: +8.3;
• Hanging: +3.3;
• Long-horizon: +15.0;
• Towel folding: +6.7.

Task 3’s precise multistage drawer geometry benefits most. RGB-only simulation proves compatibility, while real results support using available metric depth.

12. Sources of gains

12.1 Geometric modalities

• SUN: dense context raises 39.0→45.1; rays alone give 38.5.
• Objectron: RGB 86.4, rays 86.7, depth 85.4, full 87.3.

Rays are not always helpful alone. Joint rays and depth give best overall scores; rays alone lose 0.5 on SUN.

12.2 Data and actions

Increment	Easy	Hard	Change from prior row
PaliGemma + Action Expert	35.4	33.4	scratch baseline
+ 3D Data	70.2	69.1	+34.6 / +35.7, largest
+ 3D & Robotic Data	72.9	72.0	+2.7 / +2.9, motion alignment
Post-train Joint → EE	79.9	79.1	+7.0 / +7.1, aligned actions
Pre-train Cam → Base	80.4	76.7	Easy +0.5, Hard -2.4

Small robot-data increments do not imply useless trajectories: sequential action gains depend on earlier data / representations. Generic 3D is the largest marginal source at this scale.

12.3 Orthogonal ablations

Strongest conclusions:

1. Generic VLM strength does not ensure policy strength: Qwen3PI trails Hard by 23.8.
2. Nonrobot geometry matters: 3D-only 76.2 versus robot-only 68.3.
3. Representation and scaling interact: delta bins impede generic pose transfer.

12.4 t-SNE is auxiliary

13. Claims and evidence

Claim	Evidence	Assessment
Pose pretraining reduces collapse	PaliGemma 33.4 → Pose-VLA 79.1; t-SNE	Strong task evidence, moderate mechanism evidence, not causal t-SNE proof
Nonrobot 3D transfers to control	+3D 35.7; 3D-only 76.2	Strongest ablation
Universal camera interface	Cam→Base Hard -2.4; EE +7.1; shared data	Supported on current platforms, needing more cameras / mobile robots
RGB compatibility	Masking, depth-free simulation	Supported, depth still helps real tasks
Few-demo generalization	Four tasks, about 100 demos, 81.25	Preliminary, limited coverage
Long-horizon ability	LIBERO-Long 92.4, real drawer 81.7	Evidence exists, geometry versus memory not isolated

14. Relationship to 6D object pose

14.1 Pose supervision used

• Omni6DPose / BOP annotations;
• camera-centric $SE(3)$ object state;
• Structured categories, anchors, poses / sizes;
• Objectron / SUN AP15 and boxes.

14.2 What is not provided

• BOP VSD / MSSD / MSPD AR;
• An explicit pluggable deployment-pose API;
• Monotonic policy success versus independent pose errors;
• Online PnP, ICP, rendering alignment, or explicit paths.

Place it in VLA / embodied intelligence, separate from BOP FoundationPose, MegaPose, SAM-6D rankings.

14.3 Implications

1. Pose labels can supervise general representations beyond one head.
2. Shared frames / tokens connect perception and control.
3. Depth / rays / intrinsics can be optional masked priors.
4. Policy success tests downstream value, while independent pose metrics remain diagnostic.

15. Limitations and assumptions

15.1 Coverage

• Mostly rigid pretraining, downstream-only deformation;
• Fixed tabletop, parallel bimanual grippers, four real tasks;
• No systematic mobile / egocentric / dexterous tests;
• Labels / calibrated trajectories still cost effort.

15.2 Calibration

• Rays require correct $K$; multicamera transforms need extrinsics / synchronization.
• Camera motion, shake, rolling shutter, and hand-eye biases can corrupt tokens.
• Euler bins have periodic / gimbal-lock risks without dedicated rotation-topology treatment.

15.3 Tokenization

• 2048 bins may saturate outside training tails;
• Sensor-distance-dependent edges need rechecking across workspaces;
• Shared lateral vocabulary may fail under unusual mounting directions.

15.4 Semantic / geometry trade-off

The authors acknowledge stronger spatial priors may reduce general reasoning amid distractors. Semantic-preserving training remains future work.

15.5 Protocols

• Upperbound changes steps / batches / chunks together;
• Main / appendix expert initialization differs;
• Fixed 1.0 confidence omits ranking / calibration;
• Selected t-SNE projections do not quantify all retained information.

16. Release and reproduction

The repository provides:

• VLM spatial training: next-token prediction;
• action training: Flow Matching;
• 3D grounding inference;
• RoboTwin evaluation / conversion;
• HDF5 and LeRobot support;
• Stage-1 Hugging Face / RoboTwin ModelScope weights.

README specifies Python 3.10.12, PyTorch 2.7.0, CUDA 12.6, plus PaliGemma / π0 / π0.5 weights. Reproduce in stages:

1. Check Stage-1 grounding decoding / AP15;
2. Validate calibrated pose-token encode / decode round trips;
3. Check base-to-camera EE trajectories;
4. Fix scratch versus π0 experts;
5. Reproduce fair 80K separately from 100K scaling;
6. Report RGB / RGB-D rather than equating masking with depth-free optimality.

17. Final assessment

The strongest contribution connects data and representation:

$$\text{non-robotic 3D annotations} \rightarrow \text{camera-centric pose tokens} \rightarrow \text{spatially grounded VLM} \rightarrow \text{EE trajectory alignment} \rightarrow \text{continuous robot actions}.$$

Ablations identify generic 3D as the largest source, with EE / camera alignment easing control transfer. RoboTwin, LIBERO, and real tasks cover perturbations, long horizons, and few-demo adaptation.

Rigid pretraining, fixed tables, calibration, semantic trade-offs, and initialization still limit universality. Pose supervision is shown to be useful VLA pretraining, rather than proving end-to-end policies fully replace explicit 6D pose modules.