Paper Reading Notes (Deep): Oryon

1. Key assessment

Oryon changes object onboarding from CAD, scans, or category IDs to language. It must jointly solve:

1. Which regions in two scenes match the text.
2. Which local positions correspond between those regions.
3. How noisy 2D correspondences yield reliable relative SE(3).

Shared feature learning handles the first two; explicit RGB-D geometry / PointDSC handles the third. Feasibility is supported, but predicted-mask REAL275 AR 32.2 versus oracle 46.5 exposes localization limits. This is a language-driven CAD-free relative-pose baseline, rather than a universal replacement for strong CAD-based absolute-pose systems.

2. Task and frame relations

Input:

• Anchor scene $A=(I^A,D^A,K^A)$.
• Query scene $Q=(I^Q,D^Q,K^Q)$.
• Object description $T$, e.g. “a brown open laptop.”

Output:

$$T_{A\rightarrow Q}= \begin{bmatrix} R & t\\ 0 & 1 \end{bmatrix}\in SE(3),$$

aligning transformed anchor-target points with the same query object.

Model-free means no test-object CAD / video. Requirements remain:

• The same rigid object in both images.
• Back-projectable depth in both.
• Camera intrinsics.
• Sufficiently discriminative scene descriptions.
• ShapeNet6D correspondence / segmentation supervision during training.

This differs from single-image category pose, CAD-based novel-object pose, and RGB-only relative rotation.

3. Shared text-visual local representations

Three feature types:

• Frozen CLIP ViT-L/14 $\phi_V$ extracts $E^A,E^Q$.
• Frozen CLIP text $\phi_T$ extracts prompt tokens $e^T$.
• Frozen guidance backbone $\phi_G$ supplies local visual structure.

Core $\phi_{TV}$ modulates cost features with text and aggregates shifted-window Transformer context, preserving:

• Object semantics: the concept being matched.
• Local appearance: cross-scene pixel / patch correspondence.
• Structure compatible with later depth-based geometry.

Decoder $\phi_D$ upsamples to $192\times192$, 32-channel $F^A,F^Q$. The same features yield masks $M^A,M^Q$, sharing representations / training instead of cascading independent boxes.

4. Losses, correspondence, and geometry

Hardest-negative contrastive learning attracts GT matches and separates difficult alternatives; Dice supervises masks. Table 4 shows joint-training effects:

• Segmentation only: AR 24.8, mIoU 63.3.
• segmentation only + Oracle mask: AR 33.2;
• Matching only with oracle masks: AR 40.3.
• Joint Oryon: predicted-mask AR 32.2, oracle AR 46.5.

Matching improves segmentation and segmentation focuses features on targets, beyond a separate open-vocabulary frontend.

Inference:

1. Filter background using $M^A,M^Q$.
2. Match nearest neighbors in $F^A,F^Q$.
3. Sample up to 500 pairs.
4. Back-project with each depth map / intrinsics:

$$x=D(u,v)K^{-1}[u,v,1]^\top;$$

5. PointDSC rejects outliers in $\{x_i^A,x_i^Q\}$ and solves $R,t$.

Robust registration is central: replacing PointDSC with RANSAC lowers REAL275 AR from 32.2 to 15.0, a 17.2 drop. VLM features alone do not explain performance.

5. Data and evaluation

Training:

• ShapeNet6D;
• About 20k pairs.
• Same models across different views / scenes.
• 20 epochs;
• Generalizable text-visual correspondence, rather than memorized REAL275 classes.

Evaluation:

• REAL275: 2k scene pairs derived from NOCS real splits.
• Toyota-Light: 2k scene pairs;
• 34 instances, 4k pairs total.
• Substantial differences in layout, background, and camera positions.

Metrics:

• VSD: visible-surface discrepancy.
• MSSD: symmetry-aware maximum surface distance.
• MSPD: symmetry-aware maximum projection distance.
• AR: mean recall across the three error-threshold ranges.
• ADD(S)-0.1d;
• segmentation mIoU.

Anchor→query GT composes poses from two scenes. Its AR is related to, but not identical with, standard BOP target-list / absolute-camera detection protocols.

6. REAL275 and Toyota-Light main results

REAL275:

Method / mask	AR	VSD	MSSD	MSPD	ADD	mIoU
SIFT / Oryon mask	24.4	12.2	27.3	33.8	12.8	66.5
ObjectMatch / Oryon mask	22.4	14.1	27.9	25.2	13.2	66.5
Oryon / Oryon mask	32.2	23.6	36.6	36.4	24.3	66.5
Oryon / Oracle	46.5	32.1	50.9	56.7	34.9	100

With shared predicted masks, AR exceeds SIFT by 7.8 and ObjectMatch by 9.8. ADD exceeds the nearest competitor by 11.1, supporting semantic-local features beyond mask benefits.

Toyota-Light:

Method / mask	AR	VSD	MSSD	MSPD	ADD	mIoU
SIFT / Oryon mask	27.2	5.7	35.4	40.6	9.9	68.1
ObjectMatch / Oryon mask	8.3	2.2	10.5	12.1	3.8	68.1
Oryon / Oryon mask	30.3	12.1	37.5	41.4	20.9	68.1
Oryon / Oracle	34.1	13.9	42.9	45.5	22.9	100

Toyota-Light’s stronger lighting / domain shift narrows SIFT’s gap through scale invariance, while ObjectMatch fails substantially. Oracle-to-predicted AR differs by only 3.8 versus REAL275’s 14.3, suggesting more appearance / geometry error than mask error there.

7. Good masks do not guarantee good poses

Two observations:

• Reasonable target masks can coexist with ObjectMatch rotation / translation failures; localization and local geometry differ.
• Small / slender objects offer few valid depth points, limiting PointDSC despite correct semantics.

Joint segmentation / matching reduces error coupling without eliminating degeneracy.

8. Architecture, prompts, and sensitivity

Architecture:

• No cost-aggregation guidance: AR 24.8.
• No decoder guidance: 21.2.
• One fewer upsampling layer, $96\times96$ features: 25.1.
• RANSAC replacing PointDSC: 15.0.
• Full model: 32.2.

prompt:

• Prompt only “object”: predicted-mask mIoU 3.0, AR 3.0.
• Misleading attributes with correct name: mIoU 56.4, AR 25.4.
• Name only / generic description: mIoU 63.4, AR 30.0.
• Full description: mIoU 66.5, AR 32.2.

Names drive category selection; attributes help same-class instance / state disambiguation. Arbitrary language is not uniformly robust, making prompts a measurable system variable.

Sensitivity:

• AR rises rapidly as low IoU approaches 0.4.
• It plateaus around 0.7.
• High mIoU can still yield low AR; no deterministic equivalence.
• Longer camera distances reduce matches and AR.

9. Related approaches

FoundationPose / GigaPose:

• CAD / reconstructed models and absolute pose, usually higher accuracy.
• Oryon substitutes text + anchor RGB-D for onboarding, opening the task.

NOPE / DVMNet:

• All estimate relative pose.
• NOPE uses one RGB reference, mainly relative rotation.
• DVMNet solves RGB-pair pose through voxel matching.
• Oryon additionally handles language reference / in-scene localization, requiring RGB-D.

Open-vocabulary segmentation:

• Oryon’s jointly trained masks share correspondence representations, beyond OVSeg preprocessing.
• Oryon mask mIoU 66.5 versus OVSeg 56.4 / 75.5 across different data; geometry / registration still control pose.

10. Limitations and failures

• Depth is mandatory for 3D–3D registration.
• Both images must contain the same object.
• Severe dual-view occlusion leaves too few matches.
• Same-named instances need attributes for disambiguation.
• Natural yet discriminative prompts are hard for low-texture mechanical parts.
• CLIP concept biases propagate to masks / matching.
• Oracle gaps expose segmentation as a major bottleneck.
• PointDSC adds dependencies and runtime despite robustness.
• Two sources / 34 instances limit open-world coverage.
• No complete depth-free / predicted-depth experiments.

11. Reproduction checklist

• Use paper-version REAL275 / Toyota-Light scene pairs, not original single-image splits.
• Verify anchor→query composition and frames.
• Align RGB / depth resizing with intrinsics.
• Freeze CLIP / guidance, training fusion / decoder / masks only.
• Preserve $192\times192\times32$ output features.
• Reproduce contrastive and Dice objectives separately.
• Match nearest-neighbor, mask-filter, and 500-pair sampling order.
• Use identical PointDSC checkpoints / parameters.
• Report predicted / oracle masks to separate perception and geometry.
• Report AR, VSD, MSSD, MSPD, ADD, and mIoU together.
• Save prompts for matched reference conditions.
• Isolate same-name instances, distance, small targets, and depth holes.

12. Overall assessment

Oryon brings linguistic target selection into local matching, beyond frontend selection alone. REAL275 gains over SIFT / ObjectMatch, substantial guidance / PointDSC ablations, and quantified prompt effects support the design.

The 32.2 predicted versus 46.5 oracle AR gap shows that VLMs do not remove pose difficulty. Reliable systems still need disambiguation, missed-mask recovery, valid depth, and low-overlap registration. Oryon establishes reproducible text-specified, CAD-free cross-scene RGB-D relative pose while exposing semantic and geometric error limits.