Paper Reading Notes (Deep): OP-Align

1. Key assessment

OP-Align decomposes articulated variation into two levels constrained by unlabeled geometry, beyond multiplying rigid pose heads:

1. Object-level alignment removes overall SE(3) variation.
2. Part-level alignment simulates joints, matching each part to canonical reconstruction.

Global alignment alone cannot align different joint states; directly supervised part segmentation is expensive. Correct part transforms instead bring that part closer to reconstruction while leaving others distant. Distances supervise membership indirectly, and membership weights distances in return.

The loop works on clean synthetic clouds at 41 FPS. Real missing points, outliers, and small parts weaken distance discrimination; results and failures reveal the supervision limit.

2. Inputs, outputs, and self-supervision boundaries

2.1 Inputs and outputs

Input $X\in\mathbb R^{3\times N}$ is a single-frame partial cloud back-projected from RGB-D, intrinsics, and masks. Training / inference sample $N=1024$ points without RGB features.

For categories with $P$ parts and $J=P-1$ joints, outputs are:

• Part probabilities $W\in\mathbb R^{P\times N}$.
• Joint pivots $c_j\in\mathbb R^3$ and directions $d_j\in\mathbb R^3$.
• Revolute angles or prismatic displacements.
• Per-part $R_i,t_i$.

Part count and joint types are known, with each joint joining two distinct parts. This is not topology discovery or a universal articulated-structure model.

2.2 Self-supervision does not remove all evaluation labels

Training losses use no manual pose, part segmentation, pivot, or direction labels, consistent with self-supervised training.

Emergent canonical frames need not match annotated frames. Evaluation runs trained models on training samples, combines reconstruction-to-input poses with GT, and uses RANSAC for a common pose per reconstruction part. Test predictions then enter annotated coordinates through this calibration.

Distinguish:

• Learning: no pose / shape / part / joint labels.
• Benchmark frame calibration: training-set GT.
• Test inference: only one cloud and known structural priors.

This preserves the training contribution without claiming label independence across the entire quantitative protocol.

3. Two-level alignment

3.1 Object-level phase

E2PN produces $D\times60$ equivariant features; 60 icosahedral-group rotations discretize SO(3). PoseHead predicts $R_i,t_i$ per anchor, selecting

$$(R_o,t_o)=\arg\min_i \operatorname{CD}(R_iX+t_i,Y).$$

where $Y$ is canonical reconstruction:

$$Y=\operatorname{ReconHead}(f_{\mathrm{inv}})+Y_{\mathrm{base}},$$

Here $Y_{\mathrm{base}}$ is a learnable category-shared base, stabilizing structure across instances while ReconHead expresses detail.

The authors state that $R_o,t_o$ need not be precise before joint motion is handled; it reduces global-pose variance for subsequent nonequivariant PointNet processing.

Object reconstruction uses bidirectional Density-aware Chamfer Distance with sensitivities $\alpha_L=30,\alpha_R=120$. Additional terms:

• $L_{\mathrm{regS}}$ limits deformation from the base shape.
• $L_{\mathrm{regD}}$ constrains neighbor-distance variance against clustering / sparsity.

3.2 Part-level phase

Shared PointNets process $R_oX+t_o$ and $Y$ for each side’s part probabilities, pivots, directions, and states.

For joint $j$:

• $R_d$ aligns input / reconstruction joint directions.
• Revolute states rotate by $a_y-a_x$ around that direction.
• Prismatic states translate by $d_y(a_y-a_x)$ along it.
• Pivot translation moves input pivots to reconstruction pivots.

Each $Z_{j,i}$ aligns only one side’s part, leaving others displaced. This selective alignment makes reconstruction distances informative for segmentation.

For shared parts, legal joint-side-to-label assignments $\sigma(j,i)$ are selected through segmentation-weighted Chamfer minimization. This matters for basket / eyeglasses parts connected to multiple joints.

3.3 Regularization against degeneration

Beyond object-level losses, part terms include:

• $L_p$: segmentation-weighted alignment distances.
• $L_{\mathrm{regW}}$: prevents every point joining one part.
• $L_{\mathrm{regP}}$: shared parts aligned by different joint sides should coincide.
• $L_{\mathrm{regA}}$: fixes reconstruction joint states at canonical zero.
• $L_{\mathrm{regJ}}$: keeps pivots near objects rather than outside.

Weights are $(\lambda_o,\lambda_p,\lambda_{\mathrm{regS}},\lambda_{\mathrm{regD}},\lambda_{\mathrm{regW}},\lambda_{\mathrm{regP}},\lambda_{\mathrm{regA}},\lambda_{\mathrm{regJ}})=(10,10,100,10,10,10,10,10)$. Per-category training uses 20,000 iterations, batch 24, Adam LR $10^{-4}$ halved every 5,000 iterations.

4. Datasets and evaluation

4.1 Synthetic data

EAP renders HOI4D / Shape2Motion meshes into single-view clouds for laptop, safe, oven, washer, and eyeglasses.

Metrics cover segmentation IoU, part rotation / translation, joint directions, and pivot-to-GT-axis point-to-line distances.

4.2 New real data

Captured with ASUS Xtion RGB-D:

Category	Train instances	Test instances	Train images	Test images	Joints
basket	4	2	974	449	2 revolute
drawer	4	2	884	452	1 prismatic
laptop	4	2	740	412	1 revolute
scissors	4	2	922	421	1 revolute
suitcase	4	2	813	381	1 revolute

Eight random states per instance, about 30 frames each; SAM / Mask R-CNN masks. Segmentation, part poses, pivots, and directions are supplied for evaluation.

Real mAP uses multiple thresholds. All part rotations / translations must pass together; joints test directions / pivots, segmentation IoU@50 / 75. Symmetric categories maximize mean IoU across label permutations, adding part-label tolerance.

5. Synthetic results: better than EAP, not every metric

Method	Supervision	Seg. IoU↑	Rot.↓	Trans.↓	Pivot↓	Direction↓	Memory↓	FPS↑
3DGCN	pose + seg. + joint	94.05	11.61°	0.093	0.084	9.78°	—	—
EAP	None	68.46	10.44°	0.121	0.162	23.09°	9.23 GB	<1
OP-Align	None	80.70	8.10°	0.129	0.110	6.63°	2.31 GB	41

Versus EAP:

• segmentation +12.24 points;
• Rotation improves 2.34°.
• Direction improves 16.46°.
• About one-quarter memory; speed rises from below 1 FPS to 41.
• Translation worsens slightly, 0.121→0.129.

Aggregate improvements are substantial, not universal across columns. Versus supervised 3DGCN, rotation / directions improve but segmentation, translation, and pivots worsen; unequal supervision prevents equal-cost ranking.

6. Real results: closer part pose, weaker joints / segmentation

Method	Supervision	Seg. IoU@75 / 50	Joint strict / medium / loose	Part strict / medium / loose
3DGCN	pose + seg. + joint	83.31 / 95.83	47.51 / 85.79 / 94.59	13.07 / 46.77 / 68.66
PointNet++	pose + seg. + joint	19.83 / 42.20	21.06 / 57.38 / 75.56	4.47 / 23.25 / 39.82
OP-Align	None	23.79 / 50.42	12.57 / 63.59 / 74.04	14.79 / 46.09 / 59.76

Strict part mAP 14.79 slightly exceeds 3DGCN 13.07; medium thresholds nearly tie, but loose mAP trails by 8.90 points. Larger segmentation / joint gaps remain, especially joint strict 12.57 vs 47.51.

An equivalent pivot sufficient to align parts need not coincide with the physical mechanism. Geometric alignment objectives and semantic joint parameters are not identical.

7. Ablations, failures, and limits

• Without shape-variance regularization, joint / part loose mAP falls to 42.79 / 35.39 and canonical joint state is unstable.
• Without density regularization, reconstruction points cluster locally.
• Without segmentation regularization, multipart objects collapse to one part and invalid joints.
• Without pivot regularization, pivots drift outside and part loose mAP falls to 20.49.

Basket handles have median 16.9% of object points and little effect on overall shape. Reconstruction can already fit well without handle alignment, removing the incentive to discover it.

Broader risks:

• Unseen joint regions allow multiple plausible structures from one partial cloud.
• Depth holes / mask outliers directly corrupt distance supervision.
• Per-category training lacks cross-category validation.
• Known $P$, joint types, and topology exclude unknown counts / variable topology.
• 41 FPS measures pose-network inference, excluding detection, SAM / Mask R-CNN, RGB-D capture, and back-projection.
• No cross-camera / dataset, heavy-occlusion strata, or confidence outputs.

8. Reproduction checklist

• Verify 60 E2PN anchors, rotation conventions, and $R_oX+t_o$ direction.
• Use 1,024 points, no RGB, and matching normalization.
• Separate global CD/DCD, part-weighted distances, and assignment $\sigma$.
• Share PointNet weights.
• Check revolute / prismatic composition order of $R_d,R_a,T_a$.
• Implement all eight losses / weights, especially joint / segmentation regularizers.
• Calibrate evaluation canonical frames with training GT + RANSAC and state this explicitly.
• Reproduce symmetric part-label permutation.
• Separate network FPS from detection, segmentation, and cloud-construction latency.
• Stratify small-part ratio, visibility, missing depth, and mask quality.

9. Overall assessment

OP-Align uses global alignment to reduce pose variation and part alignment to generate segmentation signals, learning single-frame category articulated pose without training labels and increasing EAP’s sub-1 FPS to 41 FPS.

Alignment success is not physical joint recovery. Some real part-pose thresholds approach supervised baselines, while segmentation / pivots lag; known topology and evaluation-frame calibration remain required. Within these boundaries, it is a useful entry point to self-supervised articulated pose.