Deep Reading Notes: WALDO: Where Unseen Model-based 6D Pose Estimation Meets Occlusion

1. Basic information

• Abbreviation: WALDO
• Authors: Sajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, Tongtong Cao
• Affiliation: Huawei Technologies Canada
• Venue: WACV 2026
• Published paper: CVF Open Access
• Public preprint: arXiv:2511.15874
• Reading mode: Deep Dive

2. Key conclusions first

WALDO turns occlusion into four explicit problems in a multistage model-based pipeline: frontend mask errors, pointwise invisibility, ambiguous initial poses, and benchmark bias toward visible samples. It retains detection, templates, point matching, and SVD, but coarse matching jointly supplies multiple hypotheses and background / occlusion probabilities to redistribute dense-matching computation and repeatedly refine candidates.

Three sentences summarize the main results:

1. Against the main SAM6D baseline, BOP-Core* UAR / AR rises from 0.54 / 0.71 to 0.56 / 0.73, while per-image time falls from 4.37 s to 1.53 s;
2. Gains concentrate on more occluded ICBIN (UAR 0.50→0.56) and YCB-V (0.52→0.56); TUD-L UAR does not improve;
3. WALDO remains below FRTPose’s 0.70 UAR, FreeZe’s 0.66, and Co-op’s 0.61: a speed–occlusion-robustness trade-off, rather than best accuracy everywhere.

Keep these boundaries during deep reading:

• Test-time RGB-D observations, object CAD / meshes, rendered templates, and camera information are required;
• BOP-Core* is the five selected datasets, not the complete BOP benchmark;
• UAR is the authors’ equal-visibility aggregation, distinct from official AR;
• Abstract claims of “2% / 5% improvement” refer to absolute UAR score-point changes such as 0.54→0.56 and 0.50→0.56;
• The public version has no verifiable official code link; module versions and reproduction interfaces require assembly.

2.1 Evidence matrix

Claim	Main evidence	Supports	Does not establish
Better under occlusion	ICBIN UAR +0.06, YCB-V +0.04; visibility-stratified UAR	More robust low-visibility performance than SAM6D under this five-dataset protocol	Generalization to every sensor or occlusion type
Faster	4.37→1.53 s / image; FRTPose 45.37 s	Lower reported per-image inference latency in the implementation table	Hardware independence, lower energy, or faster processing at all batch sizes
Dynamic sampling helps	LMO UAR 0.49→0.50	A modest visibility-guided sampling gain	Sole or dominant explanation of the overall +0.02
Multiple hypotheses help	$K=1$ gives 0.4933→$K=8$ gives 0.5045	Moderate candidate counts improve LMO UAR	More is always better; $K=32$ falls to 0.4989
Iterative refinement helps	$N=1$ gives 0.4933, $N=32$ gives 0.5015	Sufficient depth eventually beats one step	Every round improves; $N=2,4$ declines
Ordinary AR is optimistic	WALDO BOP-Core* AR 0.73, UAR 0.56	Visibility-stratified aggregation is more conservative than a high-visibility-dominated mean	UAR is the only reasonable deployment-risk weighting

3. Task boundaries and notation

3.1 Inputs

Each test instance requires:

• Observed RGB $I_o$;
• Observed depth $D_o$;
• Object CAD / mesh;
• From $V$ views, rendered template RGB ${I_m^v}_{v=1}^{V}$;
• Corresponding template depth ${D_m^v}_{v=1}^{V}$;
• Camera intrinsics;
• Frontend detection / segmentation and trained feature / matching networks.

The observation mask covers visible surfaces, yielding partial cloud $P_o$; templates aggregate into an approximately complete model cloud $P_m$.

3.2 Outputs

The final rigid transform is:

$$(R_f,t_f),\qquad R_f\in SO(3),\quad t_f\in\mathbb{R}^3,$$

It aligns model coordinates with camera / observation coordinates. This is instance-level model-based novel-object pose: test objects are unseen in training but supplied as models during testing.

3.3 Unseen does not mean no object prior

No pose-network retraining is needed for each test object, but complete CAD and prerendered multiview RGB-D remain available. Generalization means transferring from training assets to new geometry, rather than inferring pose without object-shape information.

Distinguish three other tasks:

• Model-free pose using a few reference images / videos;
• Category-level pose for new instances in training categories, using category shape priors;
• Open-vocabulary detection from text / categories, without guaranteed complete 6D pose.

4. How occlusion propagates through the pipeline

Write the system as:

$$(I_o,D_o,M)\xrightarrow{\text{detect/segment}} (B_o,S_o)\xrightarrow{\text{feature+lifting}} (F_o,P_o;F_m,P_m)\xrightarrow{\text{coarse}} \{R_k,t_k\}\xrightarrow{\text{dense refine}}(R_f,t_f).$$

Each arrow can introduce different errors:

1. Detection / segmentation misses visible fragments or includes occluders;
2. Feature lifting attaches background pixels to wrong 3D points, with no evidence for invisible model regions;
3. Coarse matching has multiple plausible local geometry / texture alignments, prematurely collapsed by single-pose selection;
4. Dense matching wastes uniform full-model sampling on invisible surfaces;
5. Evaluation dilutes low-visibility failures when highly visible samples dominate.

These design responses roughly match the five problems, making robustness easier to audit than attributing everything to a backbone.

5. Overall architecture

Modules:

1. MUSE performs 3D-aware detection / segmentation;
2. ViT supplies multilevel features, lifted to clouds through depth;
3. Coarse point matching yields soft assignments, occlusion probabilities, and initial hypotheses;
4. Occlusion probabilities determine dense-point sampling distributions;
5. Dense transformers and weighted SVD refine candidates;
6. Results feed back over $N$ rounds;
7. Training depth / mask / view augmentation simulates the occlusion error chain.

6. MUSE: 3D-aware detection and segmentation

6.1 Frontend components

For observations:

• Grounding DINO proposes boxes;
• SAM2 generates masks from box prompts;
• DINOv2-L extracts class-level and patch-level foreground multi-embeddings.

The same encoder processes templates. Integrated von Mises–Fisher (I-vMF) weights emphasize directionally plausible templates; 3D-aware multi-embedding similarity selects targets among proposals.

6.2 Detection and pose are coupled

Table 3 evaluates detection choice through final UAR:

Detector	BOP-Core* UAR	Time (s / image)
NIDS	0.54	1.26
Fast-SAM	0.55	4.37
MUSE	0.56	1.53
Ground Truth	0.70	—

MUSE gains only 0.02 UAR over NIDS, while GT detection exceeds MUSE by 0.14. Occlusion-aware backends cannot fully recover target regions omitted upstream; error recovery remains limited rather than eliminating detector dependence.

7. Features and 3D lifting

7.1 Crops and multilevel ViT features

Using $(B_o,S_o)$, mask the background, crop the ROI, and resize, for example to $224\times224$. ViT blocks 3, 6, 9, and 12 supply patch embeddings; a fully connected layer, reshape, and bilinear interpolation yield foreground-point features:

$$F_o\in\mathbb{R}^{N_o\times C}.$$

Each template produces $F_m^v$, aggregated over views as:

$$F_m\in\mathbb{R}^{N_m\times C},\qquad N_m=\sum_{v=1}^{V}N_m^v.$$

Multilevel features retain local and semantic cues but increase feature memory; intermediate tensor memory is not reported separately.

7.2 Background and occlusion tokens

Learnable observed-background embedding $f_o^{bg}$ and model-occlusion embedding $f_m^{oc}$ give evidence that should not match physical points a valid destination:

• Contaminated observation-mask points can match background;
• Currently invisible model points can match occlusion.

Without these destinations, bidirectional softmax forces mass among wrong physical points, contaminating SVD.

7.3 Point clouds

Observed depth / masks produce partial $P_o$; template depth / masks yield model $P_m$. Correspondence is subsequently 3D–3D, beyond 2D template retrieval.

This explains RGB-D dependence: unreliable or absent observed depth prevents direct construction of $P_o$, requiring changes to point matching and SVD.

8. Bidirectional soft correspondence

8.1 Assignment matrix

With extra tokens, feature attention / assignment is:

$$A=[f_o^{bg},F_o]\,[f_m^{oc},F_m]^T \in\mathbb{R}^{(N_o+1)\times(N_m+1)}.$$

Apply temperature-scaled softmax separately by rows and columns, then multiply elementwise:

$$\tilde A= \operatorname{Softmax}_{row}\left(\frac{A}{\tau}\right) \odot \operatorname{Softmax}_{col}\left(\frac{A}{\tau}\right).$$

Bidirectional normalization favors mutual matches and weakens one-to-many assignments unsupported in reverse.

8.2 Coarse stage

Uniform farthest-point-like sampling of $P_o,P_m$ gives:

$$P_o^c\in\mathbb{R}^{196\times3},\qquad P_m^c\in\mathbb{R}^{196\times3},$$

Then apply $T^c=3$ Geometric Transformer blocks. Coarse assignment $ilde A^c$ serves two purposes:

1. Sample correspondences for pose hypotheses;
2. Read background / occlusion probabilities from the first row / column.

Visibility thus emerges from correspondence rather than an independent segmentation head.

9. Multiple initial pose hypotheses

9.1 Generation

Repeatedly sample 3D–3D matches from $ilde A^c$ and solve each rigid transform $(R_{hyp}^k,t_{hyp}^k)$ by SVD. Candidate confidence roughly inverts the mean nearest-neighbor distance from transformed model points to observed coarse clouds:

$$s_{hyp}^k=\frac{N_o^c}{ \sum_{p_o^c\in P_o^c} \min_{p_m^c\in P_m^c} \left\|R_{hyp}^k(p_m^c-t_{hyp}^k)-p_o^c\right\|_2}.$$

Retain top-$K$ candidates for the dense stage.

9.2 Why occlusion needs several hypotheses

Partial geometry admits nearby plausible orientations; symmetric objects also have genuinely equivalent rotations. Wrong coarse initialization may lie outside the correct dense-refinement basin. Several initializations increase the chance of reaching that basin.

Nearest-neighbor geometry can still favor incorrect locally overlapping solutions. The final output is one pose, not an explicit distribution.

10. Dynamic Non-uniform Dense Sampling

10.1 Probability source

The first column / row of $\tilde A^c$ encodes observed-background / model-occlusion probabilities. Defined only at 196 coarse points, they are interpolated over original clouds by 3D neighborhood distance.

10.2 Sampling

Build a Walker–Vose alias table for the probability distribution, then use Gumbel-Top-$k$ without replacement to obtain roughly 2048-point dense sets:

$$P_o^d\in\mathbb{R}^{2048\times3},\qquad P_m^d\in\mathbb{R}^{2048\times3}.$$

The aim is useful matching evidence rather than uniform geometry: fewer high-occlusion / background points and more observation-supported regions.

10.3 A caveat

Wrong coarse matching produces wrong interpolated visibility, potentially amplifying early bias through dynamic sampling. Multiple hypotheses and iterative refinement mitigate this feedback but do not theoretically guarantee recovery.

11. Dense matching and iteration

11.1 Refining each hypothesis

For each $(R_{hyp},t_{hyp})$:

1. Transform the dense model cloud near the hypothesized alignment;
2. Extract multiscale positional encoding through Set Abstraction;
3. Add positional encoding to dense features;
4. Apply $T^d=3$ Sparse-to-Dense Point Transformer blocks;
5. Extract weighted correspondences from dense assignments;
6. Solve $(R_r,t_r)$ through weighted SVD.

11.2 Recurrent refinement

Updated $(R_r,t_r)$ initializes the next round; after $N$ repetitions, output $(R_f,t_f)$. This reuses shared modules recurrently, rather than training $N$ independent stages.

Iteration can correct early errors but adds risks:

• Inference cost grows with $N$;
• Incorrect matches can reinforce themselves;
• Training / inference refinement depths must be consistent or sufficiently robust.

12. Training supervision and augmentation

12.1 Correspondence labels

Synthetic GT pose $(\hat R,\hat t)$ transforms observations into model coordinates for nearest-model-point labels. Distances below $\delta_{dis}=0.15$ indicate correspondence; others become background. The reverse direction similarly labels invisible points as occluded.

Bidirectional cross-entropy supervises assignments:

$$L=CE(A[1:,:],\hat Y_o)+CE((A[:,1:])^T,\hat Y_m).$$

Apply and sum supervision over multiple coarse / dense transformer blocks.

12.2 Augmentation

Beyond standard color, noise, blur, scale, rotation, and perspective augmentation, three occlusion-focused groups add:

Depth

• blur;
• Gaussian noise;
• Random depth dropout;
• Missing elliptical regions.

Mask

• dilation;
• Mask replacement by bounding boxes;
• Straight-line mask cuts;
• Random interior removal.

View

• Subsets of multiview templates;
• Incomplete model clouds simulating missing model evidence.

Augmentation’s marginal UAR change is 0.54→0.55: useful, while the backbone / correspondence framework supplies the main performance gain.

13. UAR: averaging by visibility

13.1 Ordinary BOP AR

BOP AR commonly aggregates thresholded recalls for:

• VSD: visible surface discrepancy;
• MSSD: maximum symmetry-aware surface distance;
• MSPD: maximum symmetry-aware projection distance.

These errors handle visible surfaces or symmetry, but ordinary dataset averages still reflect the sample visibility distribution.

13.2 UAR computation

Split instances into 10 visibility-fraction deciles. Within each:

1. Compute VSD / MSSD / MSPD recall separately;
2. Aggregate their thresholds;
3. Average equally across three metrics and 10 deciles.

UAR approximately asks how often inference succeeds if a visibility level is sampled first, then an instance within it.

13.3 Detection / segmentation counterparts

Similarly compute and average COCO AP / AR by visibility decile to obtain mAPD / mARD. WALDO frontend results:

• AP 0.508 vs mAPD 0.218;
• AR 0.594 vs mARD 0.444.

AP exceeds mAPD by 0.290, indicating the large contribution of highly visible instances to ordinary AP.

13.4 Statistical trade-offs

Benefits:

• Numerous highly visible samples cannot dominate the score;
• Compare consistency across occlusion levels;
• Expose strong mean scores hiding heavy-occlusion failure.

Limitations:

• Ten bins are an empirical choice;
• Empty / small bins may have higher variance;
• Equal weighting need not match deployment frequency;
• Failure cost at low visibility may be nonlinear across tasks;
• Values are not directly comparable with conventional leaderboards.

14. Checking the main results

14.1 Against SAM6D

Dataset	SAM6D UAR / AR	WALDO UAR / AR	Absolute UAR change
LMO	0.49 / 0.69	0.50 / 0.71	+0.01
T-LESS	0.31 / 0.51	0.32 / 0.53	+0.01
TUD-L	0.85 / 0.90	0.85 / 0.92	0.00
ICBIN	0.50 / 0.58	0.56 / 0.63	+0.06
YCB-V	0.52 / 0.84	0.56 / 0.87	+0.04
Mean	0.54 / 0.71	0.56 / 0.73	+0.02

Gains broadly align with occlusion difficulty but do not cover every dataset. Unchanged TUD-L UAR rules out universal improvement across scenes.

14.2 More accurate but slower methods

Method	UAR	Time	Relative to WALDO
FRTPose	0.70	45.37 s	+0.14 UAR, about 29.7 times the runtime
FreeZe	0.66	24.89 s	+0.10 UAR, about 16.3 times the runtime
Co-op	0.61	6.92 s	+0.05 UAR, about 4.5 times the runtime
WALDO	0.56	1.53 s	Fastest but less accurate than the three above

Figure 1’s Pareto interpretation is reasonable, but better on both metrics applies to some baselines / the efficiency frontier, not domination of every method: FRTPose is clearly more accurate and WALDO faster.

14.3 AR–UAR gap

WALDO mean AR is 0.73 and UAR 0.56, a difference of:

$$0.73-0.56=0.17,$$

Relative to UAR, approximately:

$$\frac{0.17}{0.56}\approx30.4\%.$$

The paper reports about 30.5%. Larger YCB-V / T-LESS gaps indicate mainly highly visible test instances; ICBIN’s smaller gap reflects a more occluded distribution.

15. Ablations: where gains come from

15.1 From simple matching to full training

Setting	BOP-Core* UAR
ViT features + RANSAC	0.33
+ Point Matching & SVD	0.54
+ Occlusion-focused Augmentation	0.55

Learned point matching + SVD supplies the largest +0.21; augmentation adds +0.01. Attributing all gains to occlusion augmentation from the title alone would be incorrect.

15.2 DNDS

LMO gives 0.49 with static uniform sampling and 0.50 with DNDS. This small single-dataset gain needs more seeds, visibility bins, and datasets to establish stability.

15.3 $K$ and $N$

$K$ ablation:

$K$	1	2	4	8	16	32
UAR	0.4933	0.4901	0.4959	0.5045	0.5022	0.4989

$N$ ablation:

$N$	1	2	4	8	16	32
UAR	0.4933	0.4810	0.4853	0.4956	0.4957	0.5015

Two engineering risks emerge:

• More candidates increase dense-refinement cost and can dilute / misselect hypotheses;
• Many refinement rounds are needed for net gains, so reducing $N$ changes both accuracy and the speed frontier.

Reproducing Table 2’s 1.53 s requires the authors’ complete $K,N$ configuration; combining independently best ablation values does not guarantee the same latency.

16. Failure modes and implicit assumptions

16.1 Unrecoverable frontend errors

Missing boxes leave no observed cloud; large occluder regions in masks can overwhelm background-token filtering. GT detection at 0.70 versus MUSE’s 0.56 is direct evidence.

16.2 Depth sensor

Transparent, reflective, or distant objects often produce missing / wrong depth. Augmentation simulates noise / dropout, but structured, multipath, or material-dependent sensor errors need not resemble Gaussian noise.

16.3 CAD matches the real object

CAD templates supply model clouds. Manufacturing differences, flexibility, articulation, wear, or wrong scales systematically bias 3D–3D matching and SVD. Model mismatch is not solved here.

16.4 Bootstrapping visibility probabilities

Coarse-correspondence mistakes can label visible points as occluded and suppress their dense sampling, creating confirmation bias.

16.5 Symmetry and a single final solution

Multiple hypotheses cover candidates, but final scoring selects one pose. Even symmetry-aware single-solution evaluation cannot express distribution coverage / redundancy for genuinely ambiguous images, motivating later BOP-Distrib-style evaluation.

17. Reproduction checklist

17.1 Required components

• Grounding DINO;
• SAM2;
• DINOv2-L;
• MUSE I-vMF template weighting and multi-embedding similarity;
• RGB-D template renderer;
• Geometric Transformer;
• Sparse-to-Dense Point Transformer;
• Walker–Vose alias sampling and Gumbel-Top-$k$;
• weighted SVD;
• BOP datasets, visibility masks, CAD, and evaluators;
• Visibility-decile UAR / mAPD / mARD implementations.

17.2 Error-prone details

• Object–camera transform direction and left / right multiplication conventions for $R,t$;
• Depth units, CAD scale, and BOP mm / m conversion;
• Template directions / counts and opposing-view pairs;
• Consistent image, depth, and intrinsics transforms after mask cropping;
• Background / occlusion token positions in the assignment matrix;
• Elementwise bidirectional-softmax multiplication, not an exactly equivalent Sinkhorn algorithm;
• Distance-weight direction and normalization in coarse-to-full-cloud interpolation;
• SVD reflection correction to avoid $\det(R)=-1$;
• UAR bin boundaries, empty-bin handling, and recall-threshold aggregation;
• Whether latency includes detection, template processing, transfers, and all refinement stages.

17.3 Missing information

The public paper does not fully report:

• Official code and checkpoints;
• Unified hardware / batch / warm-up settings;
• Per-module memory and FLOPs;
• Multiple random seeds or confidence intervals;
• Measured cross-depth-sensor generalization;
• Joint $K,N$ sweeps;
• Complete numeric results for each visibility decile.

These gaps mean reproducing 1.53 s with 0.56 UAR needs the authors’ implementation or substantial engineering judgment.

18. Relationship to other approaches

18.1 SAM6D

WALDO extends SAM6D-style model-based zero-shot pose for occlusion / efficiency: stronger detection, point-matching background / occlusion tokens, multiple candidates, dynamic sampling, and iterative refinement.

18.2 FoundationPose / FreeZe / Co-op

• FoundationPose: strong render-and-compare / refinement, but high runtime in this table;
• FreeZe: foundation features and geometric matching yield higher UAR, still much slower;
• Co-op: correspondence plus flow refinement gives higher accuracy at intermediate speed.

WALDO is a lighter Pareto point rather than a complete replacement.

18.3 BOP-Distrib

UAR reweights samples across visibility levels; BOP-Distrib asks which poses explain the visible evidence in one image. They are complementary:

• UAR: does each visibility level contribute fairly?
• Pose-distribution GT: is the image intrinsically ambiguous?

Future work could evaluate all WALDO hypotheses against per-image pose distributions rather than retaining one refined result.

19. Site analysis: promising extensions

19.1 Joint frontend and pose

GT detection leaves a 0.14 UAR gap, far above DNDS’s 0.01. Pose-consistency filtering of detection / segmentation or joint visibility learning may offer more than deeper refinement.

19.2 Preserve distributions

Multiple hypotheses currently support optimization only. Weighted candidates could form a mixture over $SE(3)$, with BOP-Distrib precision / recall testing coverage of visually equivalent solutions.

19.3 Risk-weighted UAR

Ten equal deciles express a fairness choice, but robots may face greater failure costs at 10% visibility. Consider:

$$R_{risk}=\sum_{b=1}^{10}w_bR_b,$$

Here $w_b$ depends on failure cost, scene frequency, or downstream grasp sensitivity; report both macro UAR and deployment-weighted scores.

19.4 Include prior costs in Pareto comparisons

Figure 1 plots only accuracy and time. A fuller deployment comparison includes:

• CAD requirements;
• Depth requirements;
• Onboarding time;
• GPU memory / power;
• Rendering and template storage;
• Calibration / retry options after failure.

Fastest inference need not mean lowest system cost.

20. Overall assessment

20.1 What the paper does well

• Locates occlusion errors stage by stage rather than treating robustness generically;
• Connects coarse assignments to hypotheses and visibility-guided sampling;
• Explicitly shows the optimism of AR relative to UAR;
• Retains more accurate but slower comparisons and unchanged TUD-L UAR;
• Exposes nonmonotonic ablations without implying hypotheses / refinement always help.

20.2 Caveats behind the presentation

• 0.56 UAR is not the main table’s best;
• +0.02 is an absolute score-point gain over SAM6D;
• Dynamic sampling alone adds only +0.01;
• The best refinement ablation uses 32 rounds;
• RGB-D, CAD, and multiple foundation models remain required;
• Missing official code raises reproduction difficulty substantially.

20.3 Intended readers

• BOP / novel-object pose researchers;
• Readers studying occlusion-aware correspondence and refinement;
• Benchmark aggregation / robustness metric designers;
• Robot-vision engineers comparing accuracy, latency, and priors.

21. Five takeaways

1. Occlusion robustness needs candidate management and computation allocation as well as augmentation.
2. Background / occlusion tokens give nonmatching evidence a destination in soft correspondence.
3. Hypotheses and iteration can both be nonmonotonic; report full sweeps and computation costs.
4. Ordinary AR and visibility-balanced UAR can tell different stories; aggregation itself encodes a research assumption.
5. WALDO offers useful accuracy at lower latency, not the highest main-table score; interpret numbers together with baselines, time, and priors.