moving RGB-D scanCAD model database3D instance proposalsrender-and-compare objective
Reading Notes
Overview / Deep Dive
Available src/paper-notes/hoc-search-3dv2024/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
HOC-Search: Coarse Reading
One-sentence conclusion
HOC-Search treats CAD identification and 9DoF pose/scale alignment in RGB-D scans as joint search. Render-and-compare supplies a reliable but expensive objective; a hierarchical CAD tree and Monte Carlo Tree Search evaluate only a small subset of candidates.
Method
The CAD database is organized into an HOC-Tree by category, shape embeddings, and scale/pose properties. Each leaf represents a specific CAD model and discrete orientation. Differences between rendered depth / masks and scan observations update node values; search can revisit upper levels to explore other branches. The current best CAD model can also undergo gradient-based 9DoF pose refinement.
Hierarchical clustering above, with models, categories, and pose properties unified into a searchable tree below. The tree focuses expensive rendering comparisons on promising branches. Source: original Figures 2–3. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
Results
On ScanNet/Scan2CAD, 800 search iterations achieve 46.3% Top-1 and 77.9% Top-5 relative to exhaustive search, with about 11.1× speedup. With inaccurate SoftGroup 3D boxes, 800 iterations + pose refinement yield Chamfer 13.72×10⁻³, better than exhaustive render-and-compare at 14.59×10⁻³ while remaining 11.1 times faster. With unknown category, 800 comparisons replace about 96,216 exhaustive comparisons, roughly 120× fewer.
Joint refinement can recover a better CAD model from inaccurate initial boxes. Different category labels can reflect ambiguous geometric boundaries such as Chair/Sofa. Source: original Figures 4–5 and Table 4. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
Boundaries
It requires RGB-D scans, a CAD database, and approximate 3D boxes, unlike a single-RGB pose estimator. Strengths are an interpretable objective and controllable search cost; bottlenecks are database coverage, category/geometry ambiguity, and each render-and-compare evaluation’s cost.
Available src/paper-notes/hoc-search-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
HOC-Search: Deep Reading
1. Joint retrieval and alignment
CAD annotation in scanned scenes couples three questions: object category, the best-matching database CAD, and its scale / pose in the scene. If initial 3D boxes are inaccurate, fixing pose before exhaustive CAD search can penalize the correct model; refining every CAD candidate is prohibitively expensive. HOC-Search spends refinement only on valuable candidates encountered during search.
2. Objective and HOC-Tree
After comparing CAD–scan similarity functions, the paper uses a render-and-compare objective combining rendered depth and silhouettes/masks. It directly measures whether the CAD at its current pose explains the scan, aligning more closely with the final task than a global shape embedding, at higher cost per evaluation.
Hierarchical k-means first organizes the database using embeddings such as ShapeGF. Upper nodes represent shape clusters, and leaves represent specific CAD models; category and discrete orientation can be inserted as property nodes. Each internal node uses a centroid CAD to represent its subtree, allowing a few rendering comparisons to select promising branches.
Early centroids need not represent every descendant well, so pure greedy search can choose the wrong branch. MCTS revisits upper levels and explores alternatives. Source: original Figures 2–3. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
3. MCTS and pose refinement
Each iteration performs selection, expansion/evaluation, and update. Evaluated descendants’ render-and-compare scores update node values; node locking avoids revisiting exhausted subtrees. When a candidate improves the best score, local gradient optimization refines its 9DoF transform (translation, rotation, scale), and the optimized objective updates the result.
The key is a tunable budget: 200/400/600/800 iterations give approximately 44.7×/22.3×/14.9×/11.1× speedups, allowing an accuracy–latency trade-off.
4. Breaking down the experiments
At 800 iterations, Top-1/Top-5 is 46.3%/77.9%; HOC-Search Top-1 substantially exceeds nearest-neighbor re-ranking at 27.6% under the same budget. Source: original Tables 2–3. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
The seemingly modest “46.3% Top-1” is relative to exhaustive search: the Table category has 8,437 CAD models × 4 orientations, so 800 random samples hit the exhaustive optimum only about 2.37% of the time. Top-5 already reaches 77.9%, and candidate Chamfer distances are close to ground truth.
The automatic pipeline obtains 3D instance boxes from SoftGroup. Exhaustive render-and-compare with fixed, inaccurate poses gives Chamfer 14.59×10⁻³; HOC-Search with 800 iterations + refinement gives 13.72×10⁻³ and an 11.1× speedup. Coupling search and alignment also repairs objective mismatch caused by exhaustive search at a fixed, wrong pose.
With inaccurate initial boxes, joint refinement improves model-to-scan fit; the unknown-category experiment uses 800 comparisons instead of about 96,216. Source: original Figures 4–5 and Table 4. Copyright belongs to the original authors and the 3DV 2024 publisher; reproduced here for academic analysis.
5. Limitations
The database must contain geometrically similar CAD models; retrieval cannot create missing shapes.
Poor centroid representativeness can still make MCTS miss good leaves before exhausting its budget.
Categories such as Chair/Sofa are semantic rather than purely geometric; geometry-based search can retrieve the right shape with the wrong label.
RGB-D scans + approximate 3D boxes have different sensing costs from single-frame RGB object pose.
Refinement covers only a few promising candidates; speedups depend on rendering-comparison and optimization implementations.
6. My assessment
HOC-Search turns an expensive but trustworthy scorer into search with an explicit budget. Its HOC-Tree + limited-candidate refinement approach remains relevant if differentiable rendering, foundation-model features, or generative CAD retrieval replaces the original scorer/database. It also shows that tightly coupled model and pose estimation may benefit from joint identification and alignment.
