Deep Reading Notes: RKHSPose

1. Key assessment

RKHSPose embeds adaptation into an interpretable keypoint geometry pipeline rather than merely adding MMD:

$$\text{RGB-D}\rightarrow\text{radial maps}\rightarrow \text{2D keypoints}\rightarrow\text{ePnP}\rightarrow SE(3).$$

Real images provide no pose GT. Self-predicted keypoints produce pseudo-poses, CAD is rendered over real backgrounds, and an Adapter aligns inputs / intermediate features between approximately layout-matched pairs. With 320 unlabeled images, LM-O synthetic + ICP rises from 55.7 to 68.7, approaching real-GT training’s 71.3.

This is neither target-data-free zero-shot transfer nor unseen-object estimation: every evaluation domain needs CAD, synthetic PBR training, and some real RGB-D.

2. Supervision boundaries and training data flow

Training uses:

• Known CAD and four KeyGNet model keypoints per object;
• BOP PBR RGB-D with object labels and pose GT;
• 320 real target-domain RGB-D images without reading pose GT;
• Self-predicted pseudo-keypoints / pseudo-poses;
• A synthetic-over-real renderer and camera intrinsics.

Training does not use:

• 6DoF labels for those 320 real images;
• External ROIs / detection masks;
• Manually annotated real-image keypoints.

LM, LM-O, YCB-V, T-LESS, and TUD-L use real training images. ITODD / HB lack real training data and use test-disjoint validation images; IC-BIN lacks both and is excluded from this adaptation protocol. Access to unlabeled target-domain data is therefore a dependency beyond test images and CAD alone.

3. Radial regression and CNN voting

Main network $M_{rv}$ contains radial regressor $M_r$ and voter $M_v$. Input is $W\times H\times4$ RGB-D. For each predefined 3D keypoint, $M_r$ outputs radial map $V_r$: Euclidean distances from surface-pixel 3D points to that keypoint.

Radial representations offer:

• Redundant constraints from many visible surface points;
• Keypoints inside objects or at invisible locations;
• Actual 3D geometric meaning from depth;
• Occlusion redundancy beyond direct sparse 2D regression.

Instead of RCVPose’s analytical voting, CNN $M_v$ converts $V_r$ into accumulated heatmaps, detects peaks, and outputs keypoints $K$, classes $C$, and confidences $S$. Five losses cover radial regression, projection, classification, confidence, and the Adapter:

$$\mathcal L= \lambda_r\mathcal L_r+ \lambda_k\mathcal L_k+ \lambda_c\mathcal L_c+ \lambda_s\mathcal L_s+ \lambda_A\mathcal L_A .$$

Regression uses smooth-L1 and classification cross entropy. A dataset shares four KeyGNet keypoints, while classes remain its known objects; new CAD does not transfer without retraining.

4. Pseudo-poses, rerendering, and the RKHS Adapter

Stage-two data generation:

1. Synthetic-pretrained $M_{rv}$ predicts pseudo-keypoints from real RGB-D;
2. ePnP gives pseudo-poses;
3. Perturb rotations by about $\pm\pi/18$ and normalized translations by about $\pm0.1$;
4. Render perturbed CAD over its real background as $I_{\text{syn/real}},D_{\text{syn/real}}$;
5. Run it alongside $I_{\text{real}},D_{\text{real}}$ and align inputs / intermediate features.

Adapter $M_A$ maps feature pairs through convolutions into high-dimensional $h_{\text{syn/real}},h_{\text{real}}$. Its main loss is Maximum Mean Discrepancy:

$$\operatorname{MMD}^2(X,Y)= \left\| \mathbb E_{x\sim X}\phi(x)-\mathbb E_{y\sim Y}\phi(y) \right\|_{\mathcal H}^{2}.$$

A trainable linear product kernel adapts mappings and comparison scales to pose learning. MMD constrains distribution mean embeddings rather than verifying real-pixel keypoint identities. Large initial pseudo-pose errors break object co-location, allowing alignment to absorb incorrect structure.

5. Two-stage alternating optimization

• First 120 epochs: labeled synthetic PBR trains the main network;
• Next 80 epochs: alternate real and synthetic epochs; unfreeze the Adapter and add MMD during real epochs;
• The paper emphasizes classification / visibility in the first 80 epochs, then reweights toward keypoint localization; adaptation begins after 120 epochs;
• Alternation reinjects geometric GT, reducing pseudo-label drift relative to synthetic training followed by continuous real fine-tuning.

Mixed / alternating mean AR 84.4 beats sequential 81.9 by +2.5, making scheduling part of the method.

6. Main results: separate three supervision levels

Training	LM ADD(S)	LM-O ADD(S)	YCB ADD(S) AUC	YCB ADD-S AUC
Ours, synthetic only	78.2	54.3	76.5	90.2
Ours + ICP, synthetic only	87.9	55.7	78.3	91.3
Ours, 320 real unlabeled	95.8	68.6	82.8	92.4
Ours + ICP, 320 real unlabeled	95.9	68.7	83.0	92.6
Ours + ICP, real pose GT	96.8	71.3	85.6	92.4

Key observations:

• LM: unlabeled adaptation gains 8.0 over synthetic + ICP, approaching full-GT 96.8;
• LM-O: 55.7→68.7, a larger occlusion-domain gain but still 2.6 below full GT;
• YCB: ADD(S) AUC rises 78.3→83.0 but remains below full-GT 85.6;
• Self-supervised ADD-S AUC 92.6 exceeds full-GT 92.4 by 0.2 in one small fluctuation, not general superiority over GT supervision;
• ICP adds only 0.1–0.2 after adaptation, while synthetic-only LM gains 9.7, indicating different initial error structures.

7. BOP results and protocols

RKHSPose + ICP Table 3 results:

LM	LM-O	T-LESS	TUD-L	ITODD	HB	YCB-V
95.8	68.4	85.6	96.2	68.7	92.3	83.8

The authors summarize gaps to strong fully supervised column references from −11.3 to +0.2, closest on HB and largest on LM-O. Three constraints matter:

• Table 2 uses ADD(S) for LM / LM-O and AUC for YCB; Table 3 uses BOP-style results, so same-named columns are not directly comparable;
• Starred baselines use external detections; no ROI is a system advantage but changes error sources;
• ITODD / HB adapt on validation, not training images; preserve this test-disjoint split.

8. Ablations: what matters?

Variable	Control	Mean AR	Final configuration	Mean AR
Adapter density	Sparse single map	77.9	Dense multimap	84.4
Schedule	Sequential	81.9	Mixed / alternating	84.4
kernel	fixed linear	71.0	trainable linear	84.4
kernel	fixed RBF	73.2	trainable RBF	81.6
distance	KL	77.9	MMD	84.4
distance	Wasserstein	80.8	MMD	84.4

Largest ablations are fixed→trainable linear +13.4 and sparse→dense +6.5, supporting task-adaptive multilevel alignment rather than any arbitrary domain loss.

Real-image budgets range from 0 to 640. All but YCB plateau around 160 images, with YCB near saturation after 320. The chosen 320 budget lacks mean / variance across alternative 320-image subsets, leaving selection sensitivity unknown.

9. Speed and system boundaries

On Intel i7 2.5 GHz + RTX 3090 24GB, the paper reports:

• data loading: 8.7 ms;
• $M_{rv}$ forward: 4.5 ms;
• ePnP: 16.2 ms;
• Total about 29.4 ms, or 34 FPS.

ICP is absent from the breakdown, so +ICP accuracy cannot automatically be paired with 34 FPS. ePnP costs 3.6 times the network latency, locating the bottleneck in keypoint instance organization and geometric solving rather than FCN inference.

10. Limitations and failures

• Closed known-instance set: every object participates in synthetic training; new CAD / categories are unsupported;
• RGB-D dependence: holes, transparency, and reflections corrupt radial quantities;
• Target-domain dependence: new cameras, lighting, or underwater domains need unlabeled real data and readaptation;
• Pseudo-label confirmation bias: incorrect keypoints produce wrong poses and synthetic-over-real pairs;
• Marginal alignment: MMD does not explicitly align object-, keypoint-, or occlusion-conditioned distributions;
• Keypoint / class priors: four KeyGNet-preselected keypoints and a fixed class set;
• Metric distinctions: report ADD(S), ADD-S AUC, and BOP AR separately;
• Split exceptions: ITODD / HB use validation; IC-BIN cannot be adapted under this protocol;
• No dedicated implementation: the discoverable RCVPose repository belongs to the ECCV 2022 predecessor, not evidence that the RKHSPose Adapter is released.

11. Reproduction checklist

• Fix PBR versions, 320 real image IDs, and test-disjoint splits per dataset;
• Fix shared four-keypoint KeyGNet sets per dataset;
• Reproduce FCN-ResNet-18 radial maps, CNN voting, and grouping;
• Train 120 synthetic epochs, then 80 alternating epochs;
• Record pseudo-pose perturbations, rendering, overlays, and depth composition;
• Cover inputs and all specified intermediate maps with the Adapter;
• Use trainable linear kernels + MMD, retaining fixed / RBF / KL / Wasserstein controls;
• Report accuracy and speed both with and without ICP;
• Test multiple 320-image subsets for selection variance;
• Avoid overlap between validation adaptation images and test scenes / videos.

12. Overall assessment

RKHSPose uses few unlabeled real RGB-D images to raise LM-O from 55.7 to 68.7 while dense / sparse, mixed / sequential, kernel, distance, and data-budget ablations locate the gains. Keypoints and ePnP retain geometric interpretability within a clearly bounded sim-to-real study.

Self-supervised and near-fully-supervised claims still require known CAD, GT-labeled synthetic PBR, real target images, and domain-specific fine-tuning. YCB ADD(S) AUC and LM-O remain below real-GT references. Its precise scope is closed-set instance-level RGB-D pose adaptation using 320 target images without pose labels to recover much of real supervision’s benefit.