Paper Reading Notes (Deep): PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6 DoF Tracking

1. Basic information

• Method: PIZZA (Powerful Image-only Zero-Shot Zero-CAD Approach)
• Authors: Van Nguyen Nguyen, Yuming Du, Yang Xiao, Michaël Ramamonjisoa, Vincent Lepetit
• Affiliation: LIGM, Ecole des Ponts, Université Gustave Eiffel, CNRS, France
• Year / venue: 2022 / International Conference on 3D Vision (3DV, Oral)
• Paper: arXiv:2209.07589
• PDF: arXiv PDF
• Code: nv-nguyen/pizza (MIT License)
• Reading mode: Deep Dive

2. Positioning in one sentence

The first frame defines a gauge; later frames estimate relative motion. Unseen-object masks and shared crops remove background interference, $(\Delta u,\Delta v,Z_t/Z_{t-1}-1)$ maintains common translation scale without CAD, and multiframe context reduces recursive drift.

Method inputs and outputs

Item	Content (compiled for this site)
Inference	RGB, intrinsics $K$, initial target box; segmentation / RAFT propagate masks, while shared background-masked crops feed adjacent or historical frames.
Output	Interframe rotation $\Delta R$ and translation changes accumulate from initial conditions. Without physical initialization, the initial frame and common scale remain free.
Training	Other-object CAD renderings, masks, and GT poses; target CAD, depth, and GT motion are absent from ordinary inference.
Objective	Axis-angle rotation plus image-plane displacement / depth ratio, rather than unknown absolute object size.
Details	Crop transforms, intrinsics, and previous depth convert $(\Delta u,\Delta v,s)$ into 3D translation; $s$ is relative depth, not size. Without metric initialization, $t$ cannot be labeled in centimeters / meters.
Conditions	Boxes, calibration, masks, and flow remain essential. Absolute benchmarks add first-frame GT pose, distinct from box-only relative tracking.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Question
Figure 1	Task and real sequences	Roles of models and initial GT pose
Figure 2	Mask propagation / resegmentation	Why the 2D front end bottlenecks tracking
Figure 3	Shared crops	Avoid removing actual motion through independent centering
Figure 4	Translation	Preserve a common unknown scale
Figure 5	Two-frame / multiframe networks	Transformer context is image features or pose history?
Tables 2—5	Synthetic / real results	Compare unequal information fairly
Figure 7	Laval / MOPED examples	Interpret red / blue / green contours
Figure 11	Drift	Does history eliminate accumulation?
Figure 13	Segmentation failure	Changing foreground corrupts rigid-motion cues

3. Problem setting and task boundaries

3.1 Formal definition

Given RGB sequence:

$$\mathcal{S}=\{I_1,\ldots,I_N\},$$

Recover camera-relative object pose $(R_t,T_t)$. The network predicts adjacent motion:

$$\Delta R_{t-1,t}=R_tR_{t-1}^{\top},\qquad \Delta T_{t-1,t}=T_t-T_{t-1}.$$

Then recursively compose:

$$R_t=\Delta R_{t-1,t}R_{t-1},\qquad T_t=T_{t-1}+\Delta T_{t-1,t}.$$

Its absolute interpretation depends on initialization:

• Metric $R_1,T_1$ permits benchmark absolute trajectories;
• Without target CAD / frames, $R_1$ may be Identity;
• Without metric depth / size, arbitrary positive $Z_1$ leaves translation up to common scale.

3.2 Inputs and dependencies

Input / module	Required?	Role / boundary
RGB video	Yes	Only target-sequence sensor modality
Initial target box / center	Yes	Instance selection, not automatic discovery
Intrinsics $K$	Yes	Projected centers and depth gauges to 3D
Class-agnostic segmentation	Yes	Predicted foreground masks, not supplied GT
RAFT flow	Yes	Mask propagation / next ROI
Target CAD / mesh	No	Other-object CAD can train the model
Target depth	No	Image-only setting
Target references	No	No Gen6D / OnePose-style onboarding
Initial 6D pose	Conditional	Needed for metric absolute trajectories

3.3 Instance or category level?

Generalization to unseen categories does not make this category-level pose:

• No NOCS, category frame, or shape prior;
• The initial box selects one instance;
• The first frame defines its gauge rather than a category-wide frame.

A precise label is instance-level unseen / open-world relative 6D tracking.

3.4 Three title qualifiers

• Zero-Shot: no target retraining, with shared pretraining still needed.
• Zero-CAD: no target CAD at inference, with other CAD used synthetically.
• Image-only: no target depth; intrinsics, boxes, learned masks, and flow remain dependencies.

4. Research context

PIZZA connects temporal trackers DeepIM, MoveIt, Laval-generic, and LatentFusion with novel-object methods Gen6D / OnePose:

Method	Target information	Modality	Output	Relationship
DeepIM / MultiPath	CAD	RGB	Iterative initialized refinement	Two-frame ModelNet baselines
Laval-generic	CAD	RGB-D	15-frame tracking	Stronger-input Laval baseline
MoveIt	target CAD	RGB	learned frame-to-frame motion	Laval baseline
LatentFusion	registered target references	RGB-D	latent 3D reconstruction + pose	ModelNet / MOPED baseline
Gen6D / OnePose	Posed / registered references	RGB	Absolute novel-object pose	CAD-free with more onboarding
PIZZA	Initial box; initial pose for absolute evaluation	RGB video	Recursive relative motion	Minimal target priors

The novelty combines unseen categories, no target CAD, RGB-only, and no target-reference acquisition in an evaluable setting, rather than introducing the first relative regressor.

5. Research motivation

5.1 Why avoid single-frame absolute pose?

An unseen CAD-free image cannot define:

• Object origin;
• canonical orientation;
• Physical size / metric depth.

Absolute regression would supervise unidentifiable quantities. Relative prediction leaves those freedoms in the first-frame gauge.

5.2 Background interference

Known-object weights can ignore irrelevant appearance, whereas unseen-object regressors face competing background, camera, and clutter motion. Complex-background randomization alone is insufficient; explicit masks stabilize predictions.

5.3 Why multiple frames?

Small pairwise biases accumulate. The Transformer uses historical image evidence to reduce each current step’s error, without directly optimizing a pose graph.

6. Method details

6.1 Overall pipeline

1. Supply initial box $B_0$.
2. Segment unseen object mask $M_0$.
3. RAFT warps $M_t$ into $I_{t+1}$, producing $\hat M_{t+1}$.
4. From $\hat M_{t+1}$ derive $B_{t+1}$, then resegment $M_{t+1}$.
5. Share one crop across $K$ historical frames and remove backgrounds.
6. ResNet-18 encodes each crop into 256 dimensions.
7. Concatenate the last two directly, or contextualize them with a Transformer first.
8. Two MLPs predict $\omega$ and $(\Delta u,\Delta v,s)$.
9. Recover $\Delta R,\Delta T$ using crop scales, intrinsics, and prior pose, then update the trajectory.

6.2 Translation parameterization

At frame $t$, let center projection be $(U_t,V_t)$ and camera depth $Z_t$:

$$T_t=Z_tK^{-1} \begin{bmatrix} U_t\\V_t\\1 \end{bmatrix}.$$

Thus:

$$\Delta T_{t-1,t}= Z_tK^{-1} \begin{bmatrix} U_t\\V_t\\1 \end{bmatrix} - Z_{t-1}K^{-1} \begin{bmatrix} U_{t-1}\\V_{t-1}\\1 \end{bmatrix}.$$

Instead of direct $\Delta T$ regression, predict:

$$\Delta U=U_t-U_{t-1},\qquad \Delta V=V_t-V_{t-1},\qquad S_{t-1,t}=\frac{Z_t}{Z_{t-1}}-1.$$

Resized crops require converting outputs $(\Delta u,\Delta v,s)$ back:

$$\Delta U=\alpha_uW^{\text{crop}}\Delta u,\qquad \Delta V=\alpha_vH^{\text{crop}}\Delta v,$$ $$S=\frac{\alpha_u+\alpha_v}{2}s.$$

The key is propagating all $Z_t$ from one $Z_1$. Even arbitrary $Z_1$ gives one common scale instead of unrelated per-frame scales.

Translation uses smooth-$L_1$:

$$\mathcal{L}_{\Delta T} =\ell_1(\Delta u,\Delta u^\star) +\ell_1(\Delta v,\Delta v^\star) +\lambda\ell_1(s,s^\star), \qquad \lambda=1.$$

6.3 Rotation parameterization

Axis-angle rotation:

$$\omega\in\mathbb{R}^3,\qquad \|\omega\| \text{ is the angle},\quad \omega/\|\omega\| \text{ is the rotation axis}.$$

Loss:

$$\mathcal{L}_{\omega}=\|\omega-\omega^\star\|_2^2.$$

Euler, axis-angle, quaternion, and rotation-6D perform similarly for small tracking motion in the appendix; compact axis-angle is selected, without explicit symmetric alternatives.

6.4 Two-frame and multiframe

Module	Structure	Output	Role
$\Phi$	ResNet-18, 224×224	256-D frame features	Shared appearance / motion encoding
$\mathcal{T}$	1 Transformer layer, 12 heads, FFN 512, no dropout	Contextualized sequence	Aggregate arbitrary history
$\Psi_R$	800—400—200 MLP	3-D axis-angle	Relative rotation
$\Psi_T$	800—400—200 MLP	$\Delta u,\Delta v,s$	Relative translation

MLP middle layers use BatchNorm / ReLU. Learnable motion modules train jointly; segmentation / RAFT are separate. These are published 256-D / 12-head settings: 256 is not divisible by 12, so standard equally split attention cannot be configured directly this way. Verify hidden dimensions / projections in implementation; no replacement values are assumed here.

6.5 Two gauges

Algorithm 1 explicitly permits:

• $R_1=I$: arbitrary initial orientation;
• Arbitrary $Z_1>0$: a chosen translation scale.

Initial boxes alone do not unconditionally yield absolute 6D pose:

• Boxes initialize relative motion tracking;
• Metric first-frame pose initializes absolute metric sequences.

7. Training and supervision

7.1 Three synthetic protocols

Data	Scale (Table 1)	Purpose
ModelNet	420,400 train / 35,000 test images	Two-frame refinement / tracking comparisons
ShapeNet	400,000 train / 100,000 test images, video	15-frame temporal evaluation
BOP CAD renderings	22,500 train images, video	Training for real evaluation

Real tests:

• Laval: 14,190 test images, 297 sequences, 11 objects;
• MOPED: 3,079 test images, 11 household objects.

BlenderProc renders LINEMOD, HomebrewedDB, HOPE, RU-APC, and other BOP CAD. Training / real-test categories deliberately differ, jointly testing category and domain generalization.

7.2 Motion sampling

ModelNet pairs:

• Initial translation $(0,0,500\text{ mm})$;
• Random rotation;
• Gaussian per-axis angular perturbations, standard deviation 15°, total <45°;
• $\Delta x,\Delta y$ standard deviation 10 mm; $\Delta z$ standard deviation 50 mm.

ShapeNet videos:

• Initial $Z$ sampled uniformly over 0.4—2 m;
• Per-axis rotation motion standard deviation 20°;
• Per-axis translation standard deviation 20 mm;
• 100 frames per CAD.

These define the regressor’s range. Fast motion, severe blur, and large out-of-view excursions are not well covered.

8. Experimental analysis

8.1 Metrics

1. 5°/5cm recall: rotation and translation jointly satisfy thresholds.
2. ADD-0.1d: mean transformed point distance below 10% of diameter.
3. ADD-S: nearest neighbors handle symmetry.
4. Proj2D-5px: projected point error below 5 pixels.
5. Sequence error: rotation / translation every 15 frames.

Models used by ADD / ADD-S / Proj2D evaluation are not inference inputs.

8.2 ModelNet pairs

Method	Extra inputs	5°/5cm	ADD-0.1d	Proj2D-5px
DeepIM	CAD	64.3	83.6	73.3
MultiPath	CAD	84.8	90.1	81.6
LatentFusion	depth + references	85.5	94.3	94.7
PIZZA 2F	RGB pair	66.1	87.5	69.5

Authors’ claim: competitive with less information, beating DeepIM at 5°/5cm and ADD.

My analysis: supported, but not table-wide SOTA. Input-prior efficiency is the main value; weaker Proj2D highlights crop / center and rotation effects on contours.

8.3 ShapeNet history gains

Translation improves about 18.5%, rotation 7.5%. History helps center / scale more, while short small rotations are already easier from adjacent frames.

8.4 Laval occlusion

High-occlusion translation stays near 10 mm, supporting short-gap history / masks, with two limits:

• GT pose resets every 15 frames, not unlimited free-running trajectories;
• Unequal inputs test competitiveness with fewer priors rather than strict same-modality rankings.

8.5 MOPED

Single-reference LatentFusion gives 23.9 ADD AUC / 78.7 ADD-S; 8 references give 68.3 / 90.9. PIZZA improves single-reference ADD but trails ADD-S by 6.5—12.1 points, with larger gaps to 8 references. Interpret comparable-to-one-reference per metric.

8.6 Ablations and evidence

Claim	Evidence	Sufficiency	Assessment
History reduces drift	Table 3, Figure 11	Fairly strong	Seven categories and long-sequence trends agree
Competitive without CAD / depth	Tables 2, 4, 5	Broad	Unequal modalities / protocols, beyond architecture alone
Occlusion tolerance	Table 4	Moderate	Four levels in 15-frame Laval
Rotation representation matters little	Figure 8	Limited	Small synthetic motions only
Masks needed for unseen objects	Motivation / failures	Incomplete	No clear quantitative no-mask table

9. Accumulation and failure modes

9.1 Monocular $Z$ motion

$\Delta T_z$ dominates translation error: optical-axis motion can cause little appearance / scale change, especially after resizing, with weak texture or symmetry.

9.2 Segmentation-induced motion ambiguity

9.3 Other risks

• No recovery: prior-mask / flow dependence lacks global redetection / identity recovery.
• Large-motion shifts: fast motion may exceed Gaussian training perturbations.
• Long-term gauge drift: scale / center bias accumulates without BA / loop closure.
• Symmetric alternatives: ADD-S evaluates symmetry; regression lacks explicit symmetry-aware loss.
• Deformable attachments: changing foreground violates rigidity.
• Transparency / reflection / blur: no dedicated robustness evidence.

10. Engineering and reproduction

10.1 Inference chain

The system requires more than one network pass:

1. segmentation;
2. RAFT optical flow;
3. Mask warping and box updates;
4. re-segmentation;
5. shared crop / resize;
6. ResNet + Transformer + MLP;
7. Pose reconstruction and composition.

MIT code relies on older PyTorch / MMCV / CUDA and external segmentation / RAFT. Modern reproduction can fail through front-end versions, crop conventions, or transform composition rather than architecture.

10.2 Details to align

• Pixel conventions for $K$ and resized intrinsics;
• Shared crop origin, width / height, and $\alpha_u,\alpha_v$;
• Forward / backward mask-warp flow;
• Left composition $\Delta R=R_tR_{t-1}^{\top}$;
• Rodrigues units and numerical stability;
• Update direction $S=Z_t/Z_{t-1}-1$;
• GT versus arbitrary $R_1,Z_1$ initialization;
• Laval 15-frame GT resets.

11. Critical analysis

Strengths

• Honest treatment of undefined absolute CAD-free pose; translation matches observability.
• Masks, crop coordinates, and temporal context form a complete system.
• Multiple synthetic / real tests cover category and domain shifts.
• Comparisons disclose stronger CAD / depth / reference inputs.

Weaknesses

• Title / abstract can imply metric absolute pose from RGB alone; gauge conditions are clearer in Algorithm 1.
• Incomplete quantitative mask, flow, crop, and length ablations.
• Pairwise outputs lack trajectory loss / global consistency.
• Short real protocols and few objects underrepresent long, fast, out-and-back, cluttered tracking.
• Heavy segmentation without end-to-end timing.

Assumptions

• Accurate initialization boxes;
• Rigid, identity-consistent objects;
• Known consistent intrinsics;
• Motion covered by synthetic perturbations;
• Stable unseen-object masks;
• External metric initial pose when absolute trajectories are required.

Practical feasibility

Useful for AR, robotic tracking, or later manipulation when users / detectors select a target and only continuous motion is needed. Reliable long-term absolute tracking also needs:

• Calibrated initial pose / scale;
• Loss detection and relocalization;
• uncertainty-aware depth motion;
• sliding-window global optimization;
• Lighter, steadier video masks;
• Hardware / real-time measurements.

12. Extensions

1. Video foundation segmentation: memory-based masks with uncertainty replace warp / resegmentation.
2. SE(3) temporal modeling: predict distributions in Lie algebra rather than independent $\omega,\Delta u,\Delta v,s$, with trajectory losses.
3. Multiple $Z$ hypotheses: preserve alternative $\Delta T_z$ depth ratios for later disambiguation.
4. Keyframe reanchoring: high-confidence local gauges reduce drift.
5. Pose graphs / BA: nonadjacent appearance adds loops.
6. Lost-and-found: global redetection, identity embeddings, and recovery alignment.
7. Fair protocols: fixed boxes / masks / initial poses across RGB, RGB-D, CAD, and reference trackers.
8. Broader robustness: reflective / transparent objects, blur, camera motion, flexible attachments, reentry, and minute-long sequences.

13. Research and implementation lessons

• Representation: projected displacement plus depth ratio suits unknown-scale monocular tracking better than metric XYZ.
• Preprocessing: shared temporal crops preserve true position / scale changes.
• Baseline scope: adjacent research rather than direct single-frame absolute, category-canonical, or CAD-refinement ranking.
• Reporting: separate target CAD, references, depth, initial boxes / poses, intrinsics, and resets.
• Promising extension: minimal target priors with modern video masks, uncertainty, windowed geometry, recovery, and long-term consistency.

14. Open questions

• What are end-to-end FPS and module timing?
• How are training / test history lengths sampled, and when do gains saturate?
• Does historical-box expansion reduce current resolution when targets approach rapidly?
• Can segmentation confidence weight losses or select frames?
• Can silhouette scale, defocus, or learned depth improve $Z$ motion?
• How can box-only initialization recover reproducible metric scale without target CAD?