Deep Reading Notes: VAPO

1. Key assessment

VAPO’s key observation is that some predefined 3D keypoints have no direct RGB evidence under occlusion or self-occlusion. Continuing to regress them creates outlier 3D–2D correspondences and shifts the burden to RANSAC / PnP, beyond a simple keypoint-accuracy problem.

The pipeline therefore makes two decisions:

1. Which keypoints are visible in this view, and which nearby points remain useful to localize;
2. Decode only important points’ 2D positions, then recover pose.

This adds input-adaptive visibility selection beyond using fewer fixed keypoints. Fixed 256 or 512 points both average 75.3, while default dynamic selection reaches 77.2; strict 0.02d increases from 44.7 / 45.5 to 48.6, supporting the problem formulation.

2. Task boundaries and full pipeline

VAPO estimates absolute pose in an instance-level, RGB-only setting with known training objects, using CheckerPose for keypoint localization:

1. Extract image features from a detected-object RoI;
2. Predict a segmentation mask;
3. Predict external / internal visibility for $N=512$ object-surface keypoints;
4. Compute PPR importance on the object’s fixed k-NN graph;
5. Select the top $N'=256$ points to induce an image-dependent subgraph;
6. Localize their 2D coordinates with GNNs and binary codes;
7. Run Progressive-X for 400 iterations with a 2 px reprojection threshold to reject outliers and recover pose.

RGB-only describes query-time observations. Intrinsics, object detection / crops, geometry, and object-level training data remain required.

3. Visibility supervision from object-level annotations

3.1 External and internal visibility

For keypoint $p$, overall binary visibility is:

$$V(p)=V_{\mathrm{ex}}(p)V_{\mathrm{in}}(p).$$

• $V_{\mathrm{ex}}$: project $p$ with GT pose and intrinsics; assign 1 inside the visible-object mask. This models occlusion by other objects and also depends on mask-boundary quality.
• $V_{\mathrm{in}}$: in camera coordinates, $d_p^{(c)}$ points toward the camera and $n_p^{(c)}$ is the surface normal; $d_p^{(c)}\cdot n_p^{(c)}>0$ marks camera-facing points.

This inexpensive internal test approximates back-face culling without per-ray occlusion checks. It usually suits convex surfaces, but holes, concavities, or occlusion by another part of the same object can make camera-facing points actually invisible.

CAD surface keypoints / normals are sampled beforehand, then GT poses and masks generate online training labels without manual keypoint-visibility annotations.

3.2 Why symmetric objects need canonicalized poses

One symmetric-object image can have several equivalent rotations, while a dataset stores only one pose. Direct internal-visibility labels from that pose can conflict across similar appearances, giving the classifier contradictory supervision.

Choose a canonical equivalent pose maximizing internally visible points in fixed subset $P_{\mathrm{sym}}$:

• Discrete symmetry: find a stable subset offline over 2,562 SO(3) rotations, then enumerate finite equivalents during training;
• One continuous symmetry axis: parameterize equivalent rotations by axial angle $\theta$ and use the trigonometric form of $(R^\top t)^\top n_p$ to analytically orient the target normal most toward the camera;
• Continuous plus discrete symmetry: remove continuous ambiguity analytically, then enumerate discrete equivalents;
• Multiple continuous axes: apply the same derivation sequentially by axis.

This standardizes label conventions, rather than recovering unobservable symmetric angles; evaluation still needs symmetry-aware ADD(-S).

4. From binary visibility to PPR importance

Binary-visible selection alone is fragile: masks / normal labels may be wrong, and points near visible regions may retain local image evidence despite a 0 label. Build a directed k-NN graph over fixed 3D keypoints, with default $k=20$ and an adjacency-normalized transition matrix.

A uniform probability on visible points gives restart vector $s$. Personalized PageRank satisfies:

$$r=cT^\top r+(1-c)s, \qquad r=(1-c)(I-cT^\top)^{-1}s.$$

The default damping factor is $c=0.85$. Since the object-keypoint graph is fixed,

$$T_{\mathrm{ppr}}=(1-c)(I-cT^\top)^{-1}$$

can be precomputed offline; online processing forms $s$ from predicted visibility and multiplies the matrix. $r$ is the random walk’s probability at each keypoint, interpretable as proximity to the predicted visible region on the surface graph.

PPR smooths fragile 0/1 labels into a ranking signal rather than estimating true ray visibility. Table VII uses ray-traced labels as a reference: simple internal visibility gives only 60.4% / 66.0% on holepuncher / benchvise, while top-PPR points reach 67.3% / 82.8% accuracy.

5. Dynamic subgraphs in CheckerPose

After selecting the top 256 points, GNN nodes vary with each image. Unstable early visibility prediction makes the subgraph jump during training and impedes localization. VAPO uses:

• 64D positional encoding from a shallow GNN over 3D keypoint coordinates, concatenated with original keypoint embeddings;
• Two-stage training: external / internal visibility, a 1-bit indicator, and the first few 2D-coordinate binary-code bits first; the full network second;
• BCE for visibility / binary codes and $L_1$ for segmentation masks.

On LM, removing positional encoding lowers the mean from 77.2 to 75.9, and removing two-stage training lowers it to 75.1. Modest gains, but both address dynamic-subgraph training instability.

6. What CAD-free removes

Without precise CAD, Wonder3D reconstructs 8 meshes from $m=8$ object views. Each mesh contributes $N/m$ keypoints sampled uniformly from surfaces visible in its input view, combining into 512 points; synthetic training images render the reconstructed mesh nearest the randomly sampled pose’s viewpoint.

Combining visible surfaces reduces the risk of a single reconstruction missing the back. Combined diameter-normalized LM Chamfer error is best for 6 of 13 objects, rather than better than every best single view; it reduces sensitivity to reference-view selection.

CAD-free does not mean reference-free or zero-shot:

• Eight object reference images and offline reconstruction remain necessary;
• A unified estimator is still trained for 13 known LM instances;
• Object masks / poses for supervision and camera intrinsics remain required;
• Without symmetry priors, all objects are assumed asymmetric.

7. LINEMOD ablations

Setting	0.02d	0.05d	0.1d	Mean
Fixed 256 points, no selection	44.7	84.3	96.9	75.3
Fixed 512 points, no selection	45.5	84.0	96.4	75.3
Without external visibility $V_{\mathrm{ex}}$	48.1	85.6	96.9	76.9
Without internal visibility $V_{\mathrm{in}}$	36.0	76.0	93.9	68.6
Default $c=0.85$	48.6	85.9	97.0	77.2
$c=0.8$	49.2	86.1	97.0	77.4

With little external occlusion on LM, removing $V_{\mathrm{ex}}$ costs only 0.3, while removing $V_{\mathrm{in}}$ costs 8.6: self-occlusion modeling supplies the main benefit. The default is not the highest-mean configuration; 0.85 follows the common PageRank damping factor.

Increasing keypoints from 128→192→256 raises the mean from 70.9→75.6→77.2; 320 gives 77.1, indicating saturation near 256 without clear gains from further localization budget.

8. Interpreting benchmark results

8.1 LM-O

VAPO ADD(-S) recall:

Method	0.02d	0.05d	0.1d	Three-threshold mean
CheckerPose	7.3	43.5	77.5	42.8
VAPO	9.7	46.2	78.02	44.6

Gains over the backbone concentrate at stricter thresholds: +2.4 / +2.7 / +0.52. 6D-Diff reports 0.1d performance of 79.6 but no 0.02d or 0.05d result, so it cannot be compared directly by the three-threshold mean.

8.2 YCB-V

VAPO ADD(-S) is 84.9; AUC-S without / with 11-point interpolation is 92.3 / 96.4, and AUC(-S) is 87.9 / 92.7. CheckerPose gives 81.4, 91.3 / 95.3, and 86.4 / 91.1, respectively. Gains are about 1–3.5 points, rather than overwhelming dominance.

8.3 CAD-free LM

Thirteen-object mean ADD(-S)@0.1d:

• RLLG: 82.9
• OnePose: 63.6
• OnePose++: 76.9
• GS-Pose: 92.0
• VAPO: 94.6

This comparison covers only LM at 0.1d, using VAPO’s own 8-view Wonder3D representation. It supports CAD replacement, not cross-dataset zero-shot generalization.

9. Qualitative results, speed, and nonconvexity

The selected success example shows a red bowl occluded by a sugar box: CheckerPose’s blue points are dispersed, VAPO’s green points concentrate in the visible region, and its CAD prediction avoids obvious interpenetration. One author-selected case cannot replace evaluation stratified by occlusion.

On NVIDIA TITAN RTX and an Intel 2.30 GHz CPU:

Method	Correspondence	PnP	Total
ZebraPose	13.6 ms	304.2 ms	317.8 ms
CheckerPose	68.4 ms	33.4 ms	101.8 ms
VAPO	68.3 ms	31.3 ms	99.6 ms

VAPO is only 2.2 ms faster than CheckerPose; efficiency mainly means maintaining about 10 FPS without decoding every keypoint, rather than orders-of-magnitude network acceleration. PnP behavior dominates comparisons with ZebraPose.

10. Limitations and risks

The conclusion leaves efficient visibility labeling without object-level annotations and extension to zero-shot pose as future work. Further boundaries include:

• Internal visibility uses normals rather than ray tracing for nonconvex occlusion;
• External visibility depends on object masks, so detection / segmentation errors contaminate keypoint ranking;
• Symmetry canonicalization needs known transforms; CAD-free training assumes asymmetry if they are unknown;
• Trained on seen instances, without direct inference from an arbitrary new CAD;
• Validation only on LM, LM-O, and YCB-V, without BOP AR, strongly symmetric industrial T-LESS, or cross-domain results;
• Table VI excludes RoI detector time; 99.6 ms is not full scene-perception latency;
• Euclidean k-NN is not necessarily surface-geodesic: thin or nearby disconnected surfaces may acquire cross-surface edges;
• No stratification by occlusion, mask quality, or keypoint-visibility calibration.

11. Reproduction checklist

• Specify surface sampling, normal directions, and units for 512 keypoints;
• Check pose direction, projection, mask boundaries, and the sign of $d_p^{(c)}\cdot n_p^{(c)}$;
• Reproduce discrete / continuous symmetry pose canonicalization, beyond evaluation-only ADD-S;
• Fix k-NN directions, $k=20$, $c=0.85$, and PPR matrix precomputation;
• Reproduce 64D positional encoding, top-256 induced subgraphs, and the uniform fallback with too few visible points;
• Follow two-stage training and record steps and unified / per-object models separately for LM, LM-O, and YCB-V;
• Use Progressive-X with 400 iterations and a 2 px threshold;
• Reproduce CAD-free 8-view Wonder3D, $N/8$ visible points per view, and nearest-view mesh rendering;
• Record detector, network, PPR, correspondence, and PnP time separately.

12. Overall assessment

VAPO addresses a clear keypoint-pose problem: explicitly supervise visibility rather than treating invisible keypoints as ordinary regression noise, then smooth unreliable local labels into rankable importance with PPR. Internal-visibility ablations and strict-threshold gains provide the strongest evidence. The central boundary remains known-instance training with object-level annotations and geometry priors; CAD-free is also not zero-shot.