Deep Reading Notes: FastCAD

1. Key assessment

The key is using a training encoder to teach the scene detector shape embeddings, then removing per-object encoding at test time:

• Conventional: scene detection → crop each object → encode crops → CAD retrieval → alignment optimization;
• FastCAD: one scene pass predicts detection, alignment, and distilled CAD embeddings → nearest neighbors.

This removes crop encoders scaling with object count and uses scene context for shape, such as similar chairs around a table. The 50× speedup mainly comes from reformulating tasks, rather than merely a faster backbone.

2. Task boundary: 9DoF CAD alignment versus 6D pose

Outputs are category / CAD identity and object transforms including:

• 3D center translation;
• oriented box rotation;
• Three-axis size / anisotropic scale;
• Four-way front-facing side.

Gravity-aligned indoor Scan2CAD objects use four horizontal box sides and up-axis 2-fold / 4-fold / full rotational symmetry. This is upright indoor 9DoF object-to-scene alignment, rather than arbitrary SO(3) tabletop BOP pose.

Video first reconstructs a scene mesh / cloud rather than regressing from one RGB frame. The 100 ms pipeline comprises reconstruction 50 ms + FastCAD 50 ms.

3. Single-stage scene network

MinkowskiEngine sparse convolutions and generative transposed convolutions produce multiscale features, inspired by TR3D. Each detection location outputs:

• $\hat p\in\mathbb R^{35}$: object class;
• $\hat b\in\mathbb R^7$: oriented-box parameters;
• $\hat f\in\mathbb R^4$: front-facing side;
• $\hat w\in\mathbb R^{256}$: shape embedding.

Small / large categories use feature levels 2 / 3. Total loss:

$$\mathcal L=\mathcal L_{\rm cls} +\mathbf 1_{\rm matched} (\mathcal L_{\rm bb}+\mathcal L_{\rm ff}+\mathcal L_{\rm emb}),$$

Focal loss handles category, DIoU boxes, cross-entropy front side, and MSE regresses $\hat w$ to GT CAD embeddings. Each GT is assigned once among its nearest $k=6$ detection locations.

Separating front orientation from embeddings offers two benefits:

• Avoid four 90° embeddings per CAD, reducing library size to 1/4;
• Encode symmetry directly as soft front labels:
  • non-symmetric: one-hot;
  • 2-fold: probability 1/2 on each opposite face;
  • 4-fold / full up-axis symmetry: 1/4 on every face.

4. Learning and distilling shape embeddings

Anchors are GT-box scan points with background, occlusion, and missing data; positives are corresponding clean CADs; negatives are randomly sampled same-category CADs. A shared Perceiver produces 256D embeddings.

4.1 Triplet

$$\mathcal L_{\rm tri}=\max(0,\|A-P\|_2^2+m-\|A-N\|_2^2).$$

Same-category negatives require distinguishing shapes within chairs / tables, rather than learning only category semantics.

4.2 Foreground segmentation auxiliary task

Anchor crops retain background. Pointwise BCE predicts foreground, balancing classes so 80–90% foreground does not overwhelm background supervision.

4.3 Chamfer-distance regression auxiliary task

A shallow MLP regresses positive / negative CAD-cloud Chamfer distance from their embeddings. This preserves continuous same-category geometric similarity rather than directly optimizing retrieval rank.

After encoder training:

1. Encode all CADs offline;
2. Retrieve GT CAD embedding $w_i$ for each training annotation;
3. Train FastCAD by MSE to regress $w_i$;
4. At inference, use $\hat w$ for nearest neighbors within the predicted category.

This distills target representations, rather than logits / teacher probabilities.

5. Data and metrics

Scan2CAD aligns ShapeNet CADs to 1,201 training / 312 validation ScanNet scenes, with 14K+ object annotations, 3K+ unique CADs, and 35 categories.

5.1 Alignment accuracy

Alignment counts as correct only when all conditions hold:

• Correct class;
• translation error < 20 cm;
• rotation error < 20°;
• scale error < 20%.

Each GT matches once. Retrieved-shape similarity is not checked, so correct alignment can still place a poor same-category CAD.

5.2 Reconstruction accuracy

Place predicted / GT CADs using predicted / GT transforms and sample surfaces. Normalize the largest GT side to 10; F-score at $\tau=0.5$ must exceed $\mu=0.7$. This penalizes both shape retrieval and alignment.

5.3 Shape accuracy

Shape accuracy instead places both CADs with perfect GT alignment before the same F-score test. All three metrics matter:

• Alignment: transform and category;
• Shape: retrieved geometry;
• Reconstruction: both jointly.

6. Alignment accuracy and speed

Input	Method	Instance alignment	Time
RGB-D scan	SceneCAD	61.2	2,600 ms
RGB-D scan	FastCAD	61.7	50 ms
RGB video	ODAM	29.2	366 ms
RGB video	Vid2CAD	38.6	3,200 ms
RGB video	RayTran	43.0	Not reported
RGB video	FastCAD	48.2	100 ms

Scan accuracy exceeds SceneCAD by only 0.5 point with approximately 52× faster inference: this is the tradeoff. Video accuracy exceeds RayTran by 5.2 points, but at-least-threefold speed evidence mainly comes from ODAM / Vid2CAD, since RayTran lacks timing.

Front-side ablation:

• Orientation inside embeddings: 56.2;
• Separate front head: 60.1;
• • symmetry-aware target: 61.7.

The head adds +3.9 and symmetry another +1.6, supporting separation of orientation supervision from retrieval rather than only larger embeddings.

7. Shape quality and key ablations

Main settings:

Input	Alignment	Reconstruction	Shape	Time
FastCAD scan	61.7	41.7	83.1	50 ms
FastCAD video	48.2	24.7	79.8	100 ms
FastCAD video, Vid2CAD-matched retrieval pool	48.2	29.6	87.7	100 ms

The abstract’s reconstruction 22.9→29.6 compares both methods restricted to GT scene retrieval pools. Main open-library FastCAD is 24.7; summaries must state this rather than treating 29.6 as default.

Embedding distillation:

Design	Reconstruction	Shape
2-stage, predicted bbox crop	15.6	51.0
2-stage, nearest GT bbox crop	30.6	78.1
distilled one-stage	41.7	83.1

Two-stage retrieval lags even with near-GT crops. The authors suggest scene receptive fields exploit shape correlations, but scene / category co-occurrence shortcuts may also contribute and need cross-building checks.

Auxiliary encoder tasks improve triplet-only 38.3 / 81.1 to 41.7 / 83.1. Segmentation alone has slightly higher shape accuracy 84.3, while joint training leads reconstruction. Perceiver versus PointNet++ gives reconstruction / shape 38.3 / 81.1 versus 29.6 / 74.0; encoder quality transfers to the distilled detector.

ScanNotate shape accuracy 83.5 is close to FastCAD 83.1, but GT initialization and 660,000 ms per scene make it an offline annotation reference rather than an equivalent online baseline.

8. Online video behavior

Evaluation includes only GT whose centers have entered an observed frame’s FOV. Those objects may still be occluded and unreconstructed, lowering early recall. Approaching and observing more views improves scene meshes and CAD predictions.

Independent inference on each updated cloud provides no tracking / temporal association, so:

• CAD identities can switch;
• Alignment can jitter;
• The same object may receive different categories across detections;
• Previous poses cannot stabilize refinement.

Throughput of 10 FPS does not establish temporal consistency.

9. Limitations and failures

• One ScanNet / Scan2CAD domain, 35 classes, upright indoor objects;
• Finite CAD coverage only approximates out-of-library shapes;
• Video reconstruction propagates camera-tracking / fusion errors;
• No layout, support, collision, or physical plausibility constraints;
• No iterative refinement for residual small pose errors;
• Independent online estimates jitter;
• Scene context can become a training shortcut;
• 50 ms / 100 ms concern the paper’s implementation, without mobile deployment;
• PyTorch, MMDetection3D, and MinkowskiEngine are specified, without an official FastCAD code link in the paper.

10. Reproduction checklist

• Use official Scan2CAD 1,201 / 312 scene splits;
• Limit clouds to 50K points;
• Separate scan models from video models trained on reconstruction outputs;
• Sample CAD negatives within categories;
• Use triplet + balanced foreground segmentation + Chamfer regression for the encoder;
• Distill 256D GT embeddings rather than cropping at inference;
• Separate front sides from shape, with 2 / 4 / full symmetry soft labels;
• Report alignment, shape, and reconstruction together;
• Distinguish open CAD and GT scene pools;
• Split runtime into reconstruction, FastCAD, NN search, and preprocessing;
• Add identity switches, pose jitter, and time to stable detection in online evaluation.

11. Overall assessment

FastCAD compresses expensive per-object retrieval encoders into one scene detector through embedding distillation, reducing SceneCAD’s 2.6-second pipeline to 50 ms while slightly improving alignment. Beyond 20°/20cm/20% alignment, new shape / reconstruction metrics prevent a correct pose with incorrect CAD geometry from counting as unqualified success.

Its relation to 6D pose is gravity-aligned indoor category / CAD retrieval plus 9DoF alignment, not BOP-style arbitrary-object SE(3). Within that scope it is a strong real-time digitization, AR, and robot scene-model baseline.