Paper Reading Notes (Deep): Generalizable Single-View Object Pose Estimation by Two-Side Generating and Matching

1. Basic information

• Abbreviation: Gen2SM
• Authors: Yujing Sun, Caiyi Sun, Yuan Liu, Yuexin Ma, Siu Ming Yiu
• Affiliations: The University of Hong Kong, ShanghaiTech University
• Venue: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025)
• Published paper page: CVF Open Access
• Preprint: arXiv:2411.15860
• Code repository: scy639/Gen2SM
• Reading mode: Deep Dive

2. Assessment First

Gen2SM estimates relative viewing direction for unseen objects from one reference RGB image, rather than full 6D object pose. It replaces unreliable direct generation from reference to query across large viewpoint gaps with mutual explanation across shared intermediate viewpoints. Pretrained Zero-1-to-3 noise-prediction residuals define a pose energy, and discrete search plus SGD refinement recover query elevation and azimuth.

The main contribution is the two-side matching reformulation, beyond simply using diffusion:

• Shorter transformations: two images separated by 150° can each move smaller angles toward intermediate viewpoints.
• Generation becomes a discriminative energy: each candidate need not be fully sampled into RGB; how well the denoiser explains noise scores the pose hypothesis.
• Results align with the motivation: on GSO subsets with δ≥120° and δ≥150°, Gen2SM substantially outperforms direct-generation / inversion methods such as E2VG and IDPose.

Its boundaries are equally clear: no translation, scale, or independent roll; known reference pose, gravity alignment, and object-centric crops are required; symmetric objects are filtered from NAVI; default inference takes approximately 7.8 seconds / query on one A100; and the paper acknowledges that errors remain above a few degrees.

2.1 Guide to the paper’s figures and tables

Figure / table	Question answered	Key interpretation
Figure 3	What enters the method and what is optimized?	The images are not directly feature-matched; only query elevation / azimuth are unknown
Table 1	Does it outperform correspondence, regression, and diffusion baselines?	Leads all four main columns, but metrics measure only rotation-threshold accuracy
Table 2	Does two-side matching address large viewpoint gaps?	Gains remain substantial on 120° / 150° subsets, the most direct evidence for the motivation
Tables 3–5	Are gains due to scaling parameters or a stable design?	N, M, and t differ in saturation and sensitivity; t=0.4 is especially important

3. Problem setting and task boundaries

3.1 Coordinates, Inputs, and Unknowns

The object lies at the object-coordinate origin. Reference image $I_r$ has elevation $heta_r$ and azimuth $phi_r$; its pose is known and the object’s up direction aligns with gravity. Query image $I_q$ contains the same object. Preprocessing follows E2VG: detect the query object, then convert its crop into an object-centric view.

The objective is to estimate the query viewpoint:

$$(\theta_q,\phi_q),$$

Or, equivalently, the relative change:

$$\Delta\theta_{rq}=\theta_q-\theta_r,\qquad \Delta\phi_{rq}=\phi_q-\phi_r.$$

Zero-1-to-3’s conditioning interface also permits radial change $\Delta r$. Gen2SM sets $\Delta r=0$, so camera–object distance is not an unknown and translation is not estimated.

3.2 This Is Not Arbitrary SO(3) or SE(3)

The variables and implementation explicitly optimize only elevation and azimuth. Known gravity alignment fixes the object’s upright direction, and roll is not solved as a third independent rotational degree of freedom. This site therefore labels it gravity-aligned relative rotation / viewpoint estimation, rather than unrestricted three-axis rotation or standard six-degree-of-freedom R,t object pose.

Rotation matrices can still evaluate this task: the authors convert GT and predictions to $R_{gt}$ and $R_{pr}$ and calculate geodesic angular error. Expressing evaluation in rotation matrices does not imply that the model estimates unrestricted SO(3).

3.3 Inputs, Outputs, and Priors

Dimension	Gen2SM
reference	One RGB image of the same object, with known reference pose
query	One RGB image of the same object, after detection and object-centric cropping
CAD / mesh	Not required
depth	Not required
mask	GT masks are not inputs to the main method, but an object detection / cropping frontend is required
camera intrinsics	Not explicit score inputs; the protocol supplies camera information for GT and preprocessing
multi-view reference	Not required
object-specific training	Not required
pretrained prior	Zero-1-to-3 is required; it is pretrained on large-scale rendered 3D object data
output	$(\theta_q,\phi_q)$ / relative rotation
missing output	translation, scale, independent roll, uncertainty distribution
inference	coarse grid search + test-time gradient refinement

3.4 What Training-Free Means Here

Gen2SM neither retrains for test objects nor trains a pose regressor on NAVI / GSO, so object-level training-free or test-object training-free is appropriate. It still depends on training: the core prior is Zero-1-to-3, pretrained on extensive viewpoint-conditioned renderings. Generalization and failure modes inherit the foundation model’s training distribution.

4. Research context

4.1 Dense-Reference Approaches

Gen6D, OnePose++, template retrieval, and many CAD-based methods find evidence close to the query viewpoint among multiple known-pose reference images or dense templates. They benefit from explicit observations at the cost of reference acquisition. Gen2SM reduces references to one image, filling missing viewpoints through a generative prior.

4.2 Image Correspondences + Geometric Solving

SIFT / LoFTR recover relative geometry through cross-image correspondences. Rich texture and sufficient overlap give strong geometric interpretability; when views show almost opposite sides, little shared visible surface rapidly reduces correspondence count and stability. The paper also tests PnP / Procrustes variants with ZoeDepth, where monocular-depth and cross-view matching errors accumulate.

4.3 Direct Relative-Rotation Regression

RPR, RelPose / RelPose++, and 3DAHV regress relative rotation or verify hypotheses from image pairs. They are typically faster than diffusion search but learn category, geometry, and viewpoint statistics from their training distributions. Gen2SM instead reuses Zero-1-to-3 as a cross-category object prior without retraining for pose.

4.4 One-Sided Generation / Diffusion Inversion

E2VG generates candidate views from the reference and matches them against the query; IDPose optimizes relative pose by inverting diffusion denoising. Their similar generative priors make them the key comparisons. Across large reference–query gaps, both must synthesize substantial unobserved structure at once, degrading identity consistency and detail.

4.5 Gen2SM’s Actual Addition

Gen2SM introduces shared intermediate viewpoints instead of requiring generation directly between endpoints. A correct query-pose hypothesis should make reference-conditioned and query-conditioned generation compatible there. Pose estimation becomes set-to-set consistency across intermediate views.

5. Motivation: Why Both Sides Meet More Reliably in the Middle

Suppose reference and query differ by 120° azimuth. Directly generating the query requires inferring much of the unseen back side. Even an image that semantically resembles the same category may have incorrect local structure, color, or accessory placement. Direct RGB similarity or feature matching can then mistake generative hallucination for pose error.

With an intermediate viewpoint at, for example, 60°, each side changes by only approximately 60°. Both retain more observed regions, helping conditional generation preserve identity. An incorrect query-pose candidate also gives an incorrect transformation to that intermediate view; the denoiser explains the noise on the reference-side intermediate image less well, increasing the score.

This resembles finding a shared geometric coordinate frame, but the shared space is a viewpoint-conditioned image manifold defined by pretrained diffusion, rather than an explicit 3D mesh.

6. Method details

6.1 Zero-1-to-3 Denoiser

Standard forward diffusion adds noise to clean image $I$:

$$I_t=\alpha_t I+\sigma_t\epsilon,$$

Here $\epsilon\sim\mathcal N(0,I)$ and $t$ controls noise strength. Zero-1-to-3’s denoiser also conditions on an input view and target viewpoint change:

$$\epsilon_\Theta(I_t; t, I_{cond}, \Delta\theta,\Delta\phi,\Delta r).$$

Gen2SM cares less about sampling an attractive final novel-view image than whether a pose-conditioned denoiser correctly predicts noise on a known intermediate image.

6.2 Naive generation-and-matching

The most direct objective generates the query viewpoint from the reference and minimizes its distance to the actual query:

$$\min_{\Delta\phi_{rq},\Delta\theta_{rq}} \mathbb E_z\left\|I_q-\mathbb G(z,I_r,\Delta\theta_{rq},\Delta\phi_{rq})\right\|_2^2.$$

$z$ denotes generation randomness. E2VG discretely generates and matches candidates; IDPose optimizes through denoising inversion. This objective places all the burden on a single $I_r\rightarrow I_q$ generation, making instance details harder to preserve as the viewpoint gap grows.

6.3 Two-side generated-image objective

The paper uniformly samples $N$ known intermediate viewpoints on the upper hemisphere with a Fibonacci sphere:

$$\{(\phi_i,\theta_i)\}_{i=1}^{N}.$$

The change from reference to intermediate viewpoint $i$ is:

$$(\Delta\phi_{ri},\Delta\theta_{ri})=(\phi_i-\phi_r,\theta_i-\theta_r),$$

The change from the current query-pose hypothesis to the intermediate viewpoint is:

$$(\Delta\phi_{qi},\Delta\theta_{qi})=(\phi_i-\phi_q,\theta_i-\theta_q).$$

The ideal set-to-set objective is:

$$\min_{\theta_q,\phi_q}\mathbb E_z \sum_{i=1}^{N} \left\| \mathbb G(z,I_q,\Delta\theta_{qi},\Delta\phi_{qi})- \mathbb G(z,I_r,\Delta\theta_{ri},\Delta\phi_{ri}) \right\|_2^2.$$

If $(\theta_q,\phi_q)$ is correct, both sides condition on the same object and reach the same intermediate viewpoints, so the output sets should agree. An incorrect hypothesis gives incorrect query-side transformations and larger set differences.

6.4 Score Distillation Approximates Image-Set Distance

To avoid fully sampling both image sets for each pose candidate, the paper first generates and caches reference-side images:

$$\mathcal G_r=\{I_{r\rightarrow i}\}_{i=1}^{N}.$$

Noise is then added to each $I_{r\rightarrow i}$, giving $I_{r\rightarrow i,t}$. Feed that noisy image to Zero-1-to-3, but condition it on query image $I_q$ and the change from the query hypothesis to the intermediate view. Under a correct hypothesis, predicted noise should approach the actual added $\epsilon$.

Drawing on DreamFusion’s score-distillation idea, the objective becomes:

$$\min_{\theta_q,\phi_q} \mathbb E_{t,\epsilon} \sum_{i=1}^{N} \left\| \epsilon_\Theta(I_{r\rightarrow i,t}\mid I_q,\Delta\theta_{qi},\Delta\phi_{qi})-epsilon \right\|_2^2.$$

The following distinction matters:

• Noise is added to intermediate images already generated from the reference;
• Denoiser conditioning comes from the query image and current pose hypothesis;
• Residuals measure whether query conditioning explains reference-side intermediate images, giving the score its two-sided nature;
• The score is not cosine similarity between two image-backbone features.

6.5 Fixed Timestep + Monte Carlo Score

An expectation over all diffusion timesteps remains expensive. The paper fixes $t$ and takes $M$ noise samples:

$$f(\theta_q,\phi_q)= \frac{1}{M}\sum_{j=1}^{M} \sum_{i=1}^{N} \left\| \epsilon_\Theta(I_{r\rightarrow i,t}^{(j)}\mid I_q,\Delta\theta_{qi},\Delta\phi_{qi})-epsilon^{(j)} \right\|_2^2.$$

Default parameters are:

• $N=64$ intermediate views;
• $M=4$ Monte Carlo noise samples;
• $t=0.4$; the paper states that the full diffusion schedule has 1,000 steps;
• refinement iterations = 3.

Fixing $t$ is an important engineering approximation rather than a full solution to the original expectation. Large accuracy differences across $t$ values show strong dependence of score discrimination on noise level.

6.6 Grid-based coarse search

Coarse search narrows candidates hierarchically instead of sweeping one uniform high-resolution grid:

1. Sweep azimuth from 0° to 360° in 45° intervals, with elevation candidates 15°, 45°, and 75°.
2. Fix the best current elevation and evaluate -30°, -15°, +15°, and +30° around the best azimuth.
3. Evaluate -10°, -5°, +5°, and +10° around the best updated azimuth.
4. Continue comparing elevation candidates within an approximately 10° local range around the initial elevation; the text specifically evaluates $(\theta_q-15°,\theta_q+15°)$ and updates the best elevation.

Discrete search first crosses large-scale local extrema in the energy landscape, reaching an angular spacing of approximately 10° before continuous optimization.

6.7 Gradient refinement

Initialize from coarse search, randomly sample noise to evaluate $f$, and compute:

$$\frac{\partial f}{\partial\theta_q},\qquad \frac{\partial f}{\partial\phi_q},$$

Then update pose with SGD, using 3 refinement steps by default. Figure 8 shows gains mainly at the strict 15° threshold, while 30° saturates earlier; naive one-sided refinement lacks comparable gains. The two-side score therefore improves both coarse ranking and useful local gradients.

6.8 Module Inputs and Outputs

Module	Input	Output	Role	Main risk
Detection and normalization	query RGB	object-centric query crop	Match the object-centric distribution familiar to the foundation model	Detector / crop errors alter appearance and viewpoint cues
Intermediate-view sampling	Upper hemisphere, N	Set of $(\theta_i,\phi_i)$	Establish shared target states	No lower-hemisphere coverage; more samples cost more time
reference generation	$I_r$, reference pose, intermediate views	$I_{r\rightarrow i}$	Cache reference-side generative evidence	Back-side hallucination and instance-identity drift
two-side score	noisy $I_{r\rightarrow i}$, $I_q$, query pose hypothesis	scalar $f(\theta_q,\phi_q)$	Assess whether the query hypothesis explains reference intermediate images	Sensitive to $t$, random noise, and foundation-model calibration
coarse search	Discrete viewpoint hypotheses	Coarse $(\theta_q,\phi_q)$	Global exploration	Many candidates and high computation
SGD refinement	Coarse pose, score gradient	Refined pose	Improve 15° accuracy	Local optima and stochastic-gradient variance

6.9 Training and Supervision

No new pose network is trained for Gen2SM, and no GSO / NAVI pose labels optimize its parameters. Dataset GT is used only for evaluation. The actual supervision belongs to Zero-1-to-3 pretraining, outside this paper’s training pipeline.

Reproduction therefore focuses on inference implementation rather than a training recipe:

• Zero-1-to-3 checkpoint and scheduler versions;
• Reproducing object-centric crops;
• Viewpoint coordinate conventions, angle signs, and periodicity;
• Caching reference intermediate images;
• Whether random noise is shared across candidates;
• The coarse-to-fine candidate expansion order;
• SGD learning rate and angular parameter constraints;
• Pruning / feature-reuse implementation in the light version.

7. Experimental analysis

7.1 Datasets and Sampling Protocol

Dataset	Type	Object count	Reference–query protocol	Key boundary
GSO	Synthetic renderings of scanned objects	Approximately 1,000 in the source library; 23 for testing	Uniform upper-hemisphere multiview renderings; randomly select 1 reference + 20 queries per object	Clean rendered inputs; useful for very large viewpoint-gap statistics
NAVI	Real objects in multiple environments and views	36 in the source library; 27 tested following E2VG	Usually 20 reference–query pairs per object; use all available pairs when images are insufficient	Symmetric and other unsuitable objects are filtered, so symmetry handling is not tested

Using E2VG’s benchmark facilitates direct diffusion-baseline comparisons, but inherits its boundaries: object-centric views, filtered symmetric objects, rotation-only outputs, and limited object counts.

7.2 Metric Definition

The paper computes:

$$R_{err}=R_{gt}^{\top}R_{pr},$$

Convert $R_{err}$ to axis-angle and take its angular magnitude as geodesic error. Report the fractions below 15° / 30°. To avoid interpreting @ as a link, this site uses:

• Acc≤15°: the stricter orientation-accuracy threshold;
• Acc≤30°: a looser coarse-orientation success rate.

These thresholds provide neither median / mean error nor calibration and exclude translation. Success at 30° does not imply suitability for precise AR overlays, assembly, or grasping.

7.3 Baselines and Fairness

Category	Methods	Main inputs / solution	Comparability with Gen2SM
Handcrafted / dense correspondence	SIFT + ZoeDepth + PnP / Procrustes, LoFTR	Image correspondences, sometimes with monocular depth	Same single reference–query setting; depends on shared visibility and is inherently disadvantaged at large gaps
Direct regression	RPR, RelPose++	Regress relative rotation from two images	Similar outputs permit direct rotation-accuracy comparison; training priors differ
hypothesis & verification	3DAHV	Learn a 3D-aware representation and verify candidates	Similar task, different training and inference paradigm
diffusion	IDPose, E2VG	Zero-1-to-3 prior + inversion / generation matching	Key comparisons with similar foundation models and one-reference settings

The paper uses implementations from reference [1] for SIFT-ZoeDepth and RPR, official repositories for other methods, and only the two-view setting for RelPose++. Overall comparability is reasonable, but crops, pretraining data, generated candidate counts, and runtime are not standardized in one comparison table.

7.4 Main results

NAVI

Method	Acc≤15°	Acc≤30°
3DAHV	28.23	48.34
RelPose++	24.33	40.05
E2VG (N=64)	42.69	64.21
E2VG (N=128)	43.16	66.47
Gen2SM	55.32	82.14

Against the strongest listed baseline, E2VG (N=128), Gen2SM gains 12.16 and 15.67 percentage points, respectively. Improvements on real data show that two-side scoring works beyond clean renderings.

GSO

Method	Acc≤15°	Acc≤30°
IDPose	20.43	40.43
E2VG (N=64)	39.30	55.86
E2VG (N=128)	40.43	57.61
Gen2SM	58.26	72.61

Against E2VG (N=128), gains are 17.83 and 15.00 points, respectively. GSO’s upper-hemisphere rendering matches the intermediate-view sampling domain more closely. GSO alone therefore does not establish generalization to arbitrary real scenes; NAVI is more important for that assessment.

7.5 Large Viewpoint Gaps

GSO subset	RelPose++	3DAHV	IDPose	E2VG	Gen2SM
δ≥120°, Acc≤15°	4.07	13.82	6.05	19.51	31.71
δ≥120°, Acc≤30°	17.89	21.38	21.14	34.14	49.59
δ≥150°, Acc≤15°	3.64	14.55	9.09	16.36	21.82
δ≥150°, Acc≤30°	16.62	20.00	20.00	32.73	47.27

Compared with E2VG:

• δ≥120°: gains of 12.20 / 15.45 points;
• δ≥150°: gains of 5.46 / 14.54 points.

The authors summarize performance as approximately 1.5 times the best baseline, but the ratio varies across columns. The more defensible statement is that every column shows a substantial lead. Yet strict-threshold accuracy at δ≥150° is still only 21.82%, indicating that foundation-model hallucination remains a major bottleneck with almost no shared visible surface.

7.6 Ablation studies

Intermediate-View Count N

N	16	32	64	128
Acc≤15°	38.67	35.33	45.58	48.39
Acc≤30°	69.81	73.14	78.69	79.37

Increasing from 64 to 128 adds only 0.68 point at 30°, while computation grows approximately linearly with N, supporting the default 64. However, 15° accuracy drops by 3.34 points from N=16 to N=32, so results are not strictly monotonic. Noise variance, viewpoint coverage, or single-run variation may contribute; the paper provides no multiseed confidence intervals.

Monte Carlo Sample Count M

M	1	2	4	8
Acc≤15°	44.16	45.53	45.58	45.87
Acc≤30°	73.94	78.64	78.69	79.93

Going from M=1 to 2 adds 4.70 points at 30°; 2 to 4 barely changes it; 4 to 8 adds only 0.29 / 1.24 points. The authors report that M=8 takes nearly twice as long as M=4, motivating 4. This supports using few noise samples to reduce score variance.

Diffusion Timestep t

t	0.2	0.4	0.6	0.8
Acc≤15°	39.78	45.58	38.67	35.33
Acc≤30°	64.81	78.69	60.36	45.64

t=0.4 is clearly best. Too little noise may overemphasize local pixel differences, while too much discards instance structure. The paper favors intermediate noise but does not further analyze score landscapes or object-specific optimal t. Adaptive selection is a promising extension.

Refinement Steps and Two-Side Score

Figure 8 gives curves without an exact numerical table. Supported conclusions are:

• Two-side refinement improves continuously from 0 to 3 steps, mainly at 15° accuracy;
• Naive generate-and-match refinement does not improve substantially;
• Gains slow after 3 steps, motivating the default 3.

This site does not estimate exact values from curve pixels, avoiding false precision.

7.7 Runtime and Scalability

Version	Hardware	Search	refinement	Total / paper report
Default	One A100	6.5 s	1.3 s	Approximately 7.8 s / query
light	Similar GPU context in the paper	Not separated	Not separated	1.12 s / query

The light version uses intermediate-viewpoint pruning and feature reuse, with reported drops of approximately 1 and 4 points at Acc≤15° and Acc≤30°. Candidate scoring can run across GPUs, reducing wall-clock time approximately with GPU count, but total FLOPs and resource use remain.

For applications:

• Offline relative-orientation initialization: acceptable;
• AR first-frame initialization: the light version may be marginally usable;
• Closed-loop robotic grasping: still needs translation, stable detection, precise refinement, and speed improvements;
• Video tracking: unsuitable as an independent per-frame estimator; better invoked for relocalization after tracking loss.

7.8 Robustness and Failure Conditions

Factor	Adequately evaluated?	Site assessment
Large viewpoint gaps	Yes, separate 120° / 150° subsets	Strong evidence for the central advantage
synthetic-to-real	GSO + NAVI	Basic evidence, but few real objects
Symmetric objects	No; NAVI filters some symmetric objects	Symmetry robustness is not established
Occlusion / clutter	No visibility bins; object-centric Zero-1-to-3	Insufficient evidence for real-scene capability
textureless / reflective	Not grouped separately	Generative priors may depend on category appearance; further experiments needed
detector / crop error	No separate perturbation	An important hidden deployment variable
Query–reference lighting gap	Multiple NAVI environments, without bins	Mixed evidence without causal analysis
scene-level multiple objects	Authors explicitly state no direct support	A current method boundary
Accuracy within a few degrees	Authors explicitly acknowledge not reaching it	Unsuitable for high-precision assembly

8. Do Experiments Support the Contributions?

Author claim	Relevant experiments	Evidence strength	Site assessment
One reference and unseen-object generalization	GSO / NAVI, no object-specific training	Moderate–strong	Holds under the given object-centric and gravity-aligned protocol
Two-side matching addresses large viewpoint gaps	Table 1, Table 2, Figure 8	Strong	E2VG / IDPose comparisons and viewpoint bins directly test the claim
More effective diffusion-prior use	Main results and intermediate-view visualizations	Moderate	Supported overall, without finer causal controls for model versions, generation quality, and scoring
Applicability to robotics, AR, SLAM, etc.	Application outlook in the conclusion	Weak	No translation, slow inference, and no system experiments; potential rather than established capability

9. Critical analysis

9.1 Strengths

1. Specific problem reformulation: shared intermediate viewpoints address one-sided large-gap generation failure instead of adding abstract modules.
2. Appropriate comparisons: E2VG and IDPose use the same type of Zero-1-to-3 prior, helping isolate the matching strategy.
3. Separate key subsets: 120° / 150° bins directly test the main claim.
4. An extensible energy-based view: denoising residuals provide differentiable scores for geometric unknowns and may transfer to other foundation models.
5. Candid limitations: the authors acknowledge foundation-model bottlenecks, insufficient precision within a few degrees, and object-centric scene restrictions.

9.2 Weaknesses

1. Object pose invites overinterpretation: only elevation / azimuth are solved, without R,t; this site must make that explicit.
2. Foundation-model priors are hard to audit: if Zero-1-to-3 has seen similar objects or categories, gains from the two-side objective and pretraining coverage are difficult to separate.
3. Symmetry is bypassed: filtering simplifies angular error without resolving single-view pose ambiguity.
4. No uncertainty output: Monte Carlo scoring naturally supports a candidate distribution, but only the best pose is returned. Top-K hypotheses are more useful under large gaps / symmetry.
5. High computation: N×M×candidates×refinement denoiser evaluations are the main bottleneck. Light-version details lie in supplementary material / code, with limited main-paper evidence.
6. Insufficient frontend auditing: detection, cropping, reference-pose errors, and gravity misalignment may affect results substantially, without perturbation experiments.
7. No statistical uncertainty: ablations contain nonmonotonic values without multiseed variance or confidence intervals.

9.3 Implicit Assumptions

• Reference and query depict the same instance; no object retrieval / identity verification is performed.
• The object’s up direction is definable and gravity-aligned.
• Reference pose is sufficiently accurate; reference-pose noise is not evaluated.
• Objects remain mainly within Zero-1-to-3’s familiar object-centric distribution.
• Upper-hemisphere intermediate views suffice to connect reference and query.
• Detection produces stable crops, and crop-scale changes preserve viewpoint conditioning.
• Intermediate views generated from both endpoints retain sufficient instance-identity consistency.
• One fixed t=0.4 works across objects.

9.4 Practical Feasibility

Gen2SM is competitive for estimating another image’s approximate orientation from one product image. For robotic 6D pose, at least the following remain missing:

1. Translation / scale solving;
2. symmetry-aware multi-hypothesis;
3. A frontend for clutter, occlusion, and multiple objects;
4. Stable detection / masks / crops;
5. Depth or geometric refinement;
6. Rotation within a few degrees and translation within millimeters;
7. Inference far below 1 second;
8. Failure detection and score calibration.

A more appropriate system role is: Gen2SM generates coarse orientation hypotheses; RGB-D registration, CAD render-and-compare, PnP, or learned refinement then solve full R,t.

10. Reproduction Checklist

10.1 Data and Protocol

• Use E2VG’s GSO 23-object test split and per-object 1+20 view sampling.
• Use NAVI’s filtered 27-object split and verify the symmetric-object exclusion list.
• Fix reference–query pairs so methods do not randomly receive samples of different difficulty.
• Specify how GT rotation converts from camera extrinsics to object coordinates.
• Wrap azimuth across the periodic 0° / 360° boundary.

10.2 Image Preprocessing

• Reproduce E2VG’s object detector and object-centric transformation.
• Record crop center, scale, padding, and image resizing.
• Keep reference and query background processing consistent.
• Count detection failures separately; do not report pose only on successful detections.

10.3 Zero-1-to-3 and Score

• Fix checkpoint, VAE, scheduler, image normalization, and precision.
• Sample upper-hemisphere intermediate views with a Fibonacci sphere, default N=64.
• Cache reference-side intermediate images instead of regenerating for each query candidate.
• Fix Δr=0 and verify elevation / azimuth sign conventions.
• Fix t=0.4 and specify its index conversion in the 1,000-step schedule.
• Default M=4; common random numbers across candidates can reduce score-ranking variance.
• Record whether the score is normalized by pixel / latent count.

10.4 Search and Refinement

• Start with a 45° azimuth grid and elevations {15°,45°,75°}.
• Follow the local azimuth search order: ±30° / ±15°, then ±10° / ±5°.
• Verify local elevation candidates and bounds.
• Default to 3 refinement steps; record SGD learning rate, gradient clipping, and angle wrapping.
• Save top-K scores rather than only argmin to expose multimodality and symmetry.

10.5 Evaluation and Logs

• Use the geodesic angle of $R_{gt}^{\top}R_{pr}$, reporting Acc≤15° and Acc≤30°.
• Add median / mean angular error, per-object results, and bootstrap confidence intervals.
• Report δ≥120° and δ≥150° separately.
• Record search / refinement / preprocessing time, peak GPU memory, and denoiser calls.
• Run at least 3 noise seeds, especially for N, M, and t ablations.

11. Improvement Directions

11.1 From a Point Estimate to a Pose Posterior

Retain the complete coarse-grid score map and normalize it into a posterior on the periodic angular domain. Multiple hypotheses under symmetry or back-side ambiguity can then be disambiguated by downstream depth / temporal evidence, more reliably than forcing one argmin.

11.2 Adaptive Intermediate Views

A fixed set of 64 upper-hemisphere points is inefficient across all objects. Use a few candidates to estimate high-probability pose regions, then retain intermediate views that shorten generation from both endpoints, or learn each view’s information gain.

11.3 Adaptive Diffusion Timesteps

The strong t=0.4 ablation suggests that noise levels carry different geometric / appearance information. Scores across multiple t values could form a coarse-to-fine schedule: high noise for broad semantic orientation, low noise for instance details. Selection could also adapt to object texture and uncertainty.

11.4 Multiview-Consistent Foundation Models

The authors propose stronger or multiview-consistent priors such as SV3D, SyncDreamer, Wonder3D, and MVDiffusion++. The useful tests go beyond changing the model name:

• Does reference-side identity drift decrease?
• Is the score landscape smoother?
• Does strict-threshold accuracy improve beyond 150°?
• Does computation increase?
• Does training-data leakage risk grow?

11.5 Integration into Full 6D Pose

Three possible routes:

1. RGB-D: Gen2SM initializes orientation, while depth correspondences / Umeyama / ICP solve translation and scale.
2. CAD available: feed coarse orientation into render-and-compare and jointly optimize R,t.
3. CAD-free multi-view: reconstruct a sparse pointmap from reference / assist views, using Gen2SM’s orientation prior to stabilize registration.

11.6 Scenes and Occlusion

SAM / open-vocabulary detection could generate object masks, normalizing both images into object-centric crops before scoring multiple crops. An additional failure detector should identify incomplete objects, identity mismatches, or foundation-model generation collapse.

11.7 Symmetry

Apply the GT symmetry group to the pose score and compare:

$$f_{sym}(R)=\min_{S\in\mathcal S} f(RS),$$

Alternatively, let the posterior remain multimodal over equivalent rotations. Without CAD symmetry annotations, reference-side generation consistency could discover approximately equivalent viewpoints.

12. Insights for 6D Object Pose Research

1. Turn foundation models into geometric energies: denoisers / decoders can supply differentiable pose-hypothesis scores beyond generation.
2. Move from endpoint matching to shared explanation: with little overlap, asking both observations to explain shared canonical states can be more stable than direct correspondence.
3. Separate coarse orientation from metric pose: foundation models provide broad category / semantic orientation; geometry modules provide local millimeter-level refinement.
4. Unseen objects do not mean no priors: no object-level training does not remove large-scale generative priors, upright-direction assumptions, or reference-pose requirements.
5. Metrics must match actual outputs: rotation-threshold accuracy cannot replace ADD(-S), BOP AR, or translation error.

13. Open questions

Conceptual Questions

• How does 5°–10° reference gravity-alignment error shift the score optimum?
• Why does 15° accuracy decrease from N=16 to 32 as intermediate views increase: random noise or viewpoint distribution?
• Is denoising residual monotonic with actual generated-image similarity?

Experimental Questions

• What is the per-category performance on symmetric, heavily occluded, transparent / reflective / textureless objects?
• Which dataset underlies the light version’s 1 / 4 point drops, and are they stable across datasets?
• How do runtime, GPU memory, and preprocessing compare under standardized baselines?
• Does the gap persist with a stronger conventional image feature matcher given the same object-centric crop?

Reproduction Questions

• The main paper omits SGD learning rate, full light-version details, and multiseed variance; code and supplementary material must be checked.
• Figure 8 contains only curves, so exact refinement-step values cannot be recovered from the main text.
• Confirm NAVI’s filtered object list and pair sampling in official code to avoid protocol drift.

14. Overall Assessment

Gen2SM has a narrow task boundary and a clear method. It neither solves full 6D pose nor establishes real-time scene-level capability, but addresses a real failure of generative one-reference pose: large viewpoint gaps cause severe identity and structural drift in one-sided novel-view synthesis. Shared intermediate viewpoints, score-distillation residuals, and coarse-to-fine optimization consistently improve real NAVI, synthetic GSO, and 120° / 150° subsets.

It is best reproduced and extended as a diffusion-based relative orientation energy. Combining symmetry-aware posteriors, RGB-D translation solving, fast candidate pruning, and scene frontends could make it part of a full 6D pose system. The current result is a substantial improvement in broad relative orientation estimation for unseen objects from one reference, rather than a solution to general 6D pose from one image.