Deep Reading Notes: CODERS

1. Key assessment

CODERS starts from the tradeoffs between sensors:

• Monocular RGB lacks metric scale.
• RGB-D works for diffuse objects, but transparent / specular surfaces cause missing or distorted depth.
• Calibrated stereo can, in principle, recover metric geometry from the baseline while relying only on visible light.

Rather than first estimating stereo disparity and feeding noisy depth into a pose network, CODERS exposes image appearance and candidate 3D positions along each pixel ray jointly in feature space. This “implicit stereo” architecture lets the same object queries learn detection, pose, size, and shape reconstruction together.

The 99.5–100 3D50 scores on TOD are striking, but the benchmark is small, covers few categories, and nearly saturates at loose thresholds. Bottle 3D75, a stricter metric, is only 31.5, and the system runs at just 3 Hz. The central contribution is therefore task unification and the stereo representation; near-100 scores alone give an incomplete picture.

2. Inputs, Outputs, and Generalization Boundaries

Input:

• Full left / right RGB images;
• Intrinsics of both cameras;
• stereo extrinsics / interocular distance;
• Fixed image geometry.

Outputs:

• Classes / matching scores for multiple objects;
• continuous 6D rotation;
• metric 3D center;
• width / height / length;
• 64D category shape codes and meshes.

Not required at test time:

• object mask / box;
• target CAD;
• depth map.

The model vocabulary is nevertheless fixed to SS3D’s 16 categories. “Unseen” / “zero-shot” means unseen instances within known categories. The official code also explicitly cautions that the pretrained model targets a fixed camera resolution and baseline: directly changing the stereo cameras has not been validated and generally requires custom fine-tuning.

3. Implicit Stereo Matching

For a camera-frustum grid $P_c$:

$$P_w=[R,T]K^{-1}P_c,$$

Then:

$$F_{3D}=\psi(F_{2D},P_w)$$

produces stereo-aware features. Here, $P_w$ represents world-coordinate encodings of discrete depth hypotheses along each ray, rather than already-triangulated surface points. The Transformer can discover implicit correspondences through left / right appearance and consistent 3D coordinate hypotheses.

Advantages:

• Individual explicit-disparity failures do not directly dictate the result.
• The metric baseline enters the feature geometry.
• Evidence can be allocated across objects throughout the full image.
• Transparent surfaces can still provide contours, refraction patterns, and stereo context.

Risks:

• Computation grows with depth bins, resolution, and decoder size.
• Calibration / baseline distribution shifts alter the entire position code.
• The network may learn shortcuts specific to synthetic cameras.
• No explicit depth output is available for a geometric sanity check.

4. DETR Queries and Multi-Task Heads

ConvNeXt-B + FPN produces stereo feature maps. In the six-layer decoder, 150 queries pass through self-attention, cross-attention, and FFNs; each final object embedding feeds the following heads simultaneously:

• focal-loss class head;
• location / size L1 heads;
• 6D continuous rotation head;
• shape-code L1 head.

Rotation is obtained by orthogonalizing $r_1,r_2$:

$$R_{\cdot1}=\phi(r_1),\quad R_{\cdot3}=\phi(R_{\cdot1}\times r_2),\quad R_{\cdot2}=R_{\cdot3}\times R_{\cdot1}.$$

Hungarian matching assigns predicted queries one-to-one to ground-truth objects, avoiding NMS and two-stage cropping. The total loss is:

$$\mathcal L= \lambda_{\rm cls}\mathcal L_{\rm cls} +\lambda_{\rm pose}\mathcal L_{\rm pose} +\lambda_{\rm shape}\mathcal L_{\rm shape},$$

The weights are 2, 0.06, and 0.02. Multi-task ablations show that the shape head also improves pose, indicating shared benefits from geometric supervision of the embedding.

5. Category DeepSDF

For each SS3D mesh, a latent code $z_i$ and decoder are first learned:

$$\operatorname{SDF}(x)=f(z_i,x).$$

The authors add a category contrastive loss so that shapes in the same category share structure while embeddings from different categories separate. The main network then only needs to regress $z$ from the stereo object embedding and extract a mesh through marching / SDF decoding.

This retrieves or regresses onto a learned category shape manifold, rather than reconstructing every visible surface pixel online from the stereo pair. The prior may smooth thin structures or objects outside the training shape distribution.

6. Data and Training

SS3D:

• Derived from OmniObject3D scans;
• 16 categories of graspable everyday objects;
• 427 instances;
• 363 training and 64 test instances, with 4 test instances per category;
• Synthetic diffuse / specular / transparent / mixed materials.

The categories are banana, book, bottle, bowl, carrot, corn, cucumber, cup, dish, fork, knife, large box, orange, small box, scissors, and spoon.

Training:

• 24 epochs;
• AdamW with weight decay $10^{-2}$;
• Learning rate $2\times10^{-4}$ with cosine decay;
• batch size 8;
• 8×RTX 3090 in the paper;
• Inference on a single RTX 3090, without test-time augmentation.

TOD:

• Approximately 36K stereo pairs;
• 12 instances;
• 10 backgrounds;
• bottle 3, mug 7, cup 2;
• Bottle / mug category splits following the StereoPose protocol, with unseen instances at test time.

7. TOD Pose: Loose Thresholds Saturate, Stricter Thresholds Discriminate

Category	3D25	3D50	3D75	5°2cm	10°5cm
Bottle	100	99.5	31.5	73.4	99.8
Mug	100	100	92.8	56.2	81.9

Compared with StereoPose:

• bottle 3D50: 22.2→99.5;
• bottle 10°5cm: 57.8→99.8;
• mug 3D50: 77.4→100;
• mug 10°5cm: 34.4→81.9.

The very large gains may come from implicit metric stereo and joint size / detection learning, but may also reflect training data, full-image context, baseline implementations, and metric code. The paper takes SPD / SGPA numbers from StereoPose; it does not rerun every baseline with the same code and detector. CODERS’ own stricter 3D75 / 5°2cm results are more informative than the saturated 3D25 / 50 scores.

8. Reconstruction: Fairness When Comparing with Instance-Fine-Tuned Baselines

Method	Bottle	Bowl	Cup	Knife	Soup
Stereo2Point, instance fine-tuned	0.700	0.509	0.563	0.140	0.695
CODERS no contrastive	1.411	0.516	0.612	0.272	0.482
CODERS	0.542	0.352	0.593	0.211	0.458

The contrastive shape space helps bottle most (1.411→0.542), while improving cup by only 0.019. Beating per-instance Stereo2Point on three categories is a strong result; thin-structure reconstruction still lags on knife.

9. SS3D and Stereo Ablations

Representative SS3D categories:

• book: 3D50 90, 3D75 51, 5°5cm 65;
• bottle: 71 / 16 / 80;
• cucumber: 34 / 9 / 38;
• dish: 88 / 37 / 64;
• large box: 52 / 9 / 16;
• orange: 35 / 6 / 56.

The variation across categories shows that performance is not equally stable across all materials; size outliers and shape imbalance remain significant.

TOD ablation:

Configuration	Bottle 3D75 / 5°2cm	Mug 3D75 / 5°2cm
no position	25.8 / 53.5	88.4 / 52.5
2D position	28.6 / 70.8	91.7 / 55.3
stereo-aware 3D position	31.5 / 73.4	92.8 / 56.2
monocular	16.1 / 48.0	86.1 / 54.7
stereo	31.5 / 73.4	92.8 / 56.2

Bottle sizes vary widely, making them most sensitive to metric stereo. Mug shapes are more readily explained by the shape prior, so their stereo gains are smaller.

10. Speed, Open Source, and Deployment

The paper reports 3 Hz on an RTX 3090. The bottlenecks include:

• ConvNeXt-B on two images;
• dense 3D position hypotheses;
• Six query-decoder layers;
• shape-code / mesh decoding.

The paper described release as future work; the official project now provides:

• Training / inference code in xingyoujun/coders;
• pretrained CODERS / DeepSDF checkpoints;
• SS3D Hugging Face dataset;
• single-pair inference interface.

However, the README states that the model has only been validated at a fixed resolution / baseline; a different stereo rig will likely require fine-tuning. This calibration-domain restriction matters more for practical deployment than the absence of a depth-sensor requirement alone.

11. Limitations

• A known vocabulary of 16 categories, without open-category support;
• Main TOD results cover only bottle / mug, with limited instances and backgrounds;
• Near-saturation at loose thresholds, while strict bottle 3D75 is still only 31.5;
• A domain gap between synthetic materials / cameras and real stereo rigs;
• Sensitivity to calibration, baseline, and synchronization errors;
• Shape outputs come from a learned DeepSDF manifold, with likely failures on out-of-distribution thin structures;
• Real-world robot experiments are mainly qualitative images / demos, without a grasp success rate;
• Baseline protocols are not fully consistent;
• Training uses 8×3090, with inference at 3 Hz;
• The official code warns that new camera setups have not been validated.

12. Reproduction Checklist

• Fix stereo resolution, intrinsics, extrinsics, baseline, and synchronization.
• Use SS3D’s instance-disjoint 363 / 64 split.
• Report the 16 categories separately instead of only reporting the mean.
• ConvNeXt-B + FPN, 150 queries, 6 decoder layers;
• Specify the number and range of depth hypotheses.
• Train category DeepSDF / contrastive shape codes first, then the multi-task heads.
• Use the same object splits and metric code as StereoPose on TOD.
• Report full-image detection failures separately from matched-object pose results.
• Analyze sensitivity to baseline / calibration perturbations.
• Report detection recall, pose success, mesh error, and grasp success on real robots.
• Include mesh extraction in end-to-end latency and report memory usage.

13. Overall assessment

CODERS is one of the relatively few complete systems from 2024 that use stereo to unify detection, pose, and shape, extending beyond transparent-object pose estimation alone. Implicit Stereo Matching connects left / right appearance through calibration-aware 3D coordinates without an explicit depth map. TOD and SS3D ablations show particular value for metric bottle size / depth, while public code and data increase its long-term reference value.

Its results must still be read within the protocol: TOD is small and saturates at loose thresholds; real-robot success is not quantified; the model covers only 16 categories, depends on a fixed stereo rig, and runs at a non-real-time 3 Hz. The most defensible conclusion is that calibrated stereo plus learned 3D position encoding is an effective route to metric category-level pose estimation / reconstruction for transparent and specular objects on which RGB-D often fails.