Paper Reading Notes (Deep): iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views

1. Basic information

• Method: iFusion
• Authors / affiliations: Chin-Hsuan Wu, Yen-Chun Chen, Bolivar Solarte, Lu Yuan, Min Sun; National Tsing Hua University, Microsoft, Amazon
• Year / venue: 2025 / 3DV
• Paper: https://arxiv.org/abs/2312.17250
• PDF: https://arxiv.org/pdf/2312.17250
• Project: https://chinhsuanwu.github.io/ifusion/
• Reading mode: Deep Dive

2. Positioning in one sentence

iFusion turns pretrained novel-view diffusion into a scorer for pose-free sparse-view reconstruction: a relative camera transform is plausible if it lets diffusion reconstruct the other view. After alignment, LoRA customizes the generator to the specific object.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 3	Pose optimization → LoRA adaptation → reconstruction	Three separate stages, with later stages using estimated relative poses
Table 1	Two-view relative pose on GSO / OO3D	Rotation / translation measure camera-to-camera transforms, not object-to-camera 6D pose
Tables 2–3	NVS and reconstruction	Test whether alignment improves downstream results beyond pose-loss reduction

3. Problem setting and relation to 6D object pose

• Given two or a few object images $x^r,x^q$ with unknown relative camera transform $T_{r\rightarrow q}$.
• First output $\hat T_{r\rightarrow q}$, then an adapted target-object NVS generator that supports NeRF, NeuS, DreamGaussian, or other reconstruction backends.
• Pose means view pose / relative camera transform, treating the object as fixed in multiview registration. Table 1 is not a BOP ADD or $R,t$ object-pose result.
• Frozen Zero123 supplies optimization signals, assuming object-centric images, prior-distribution coverage, and sufficient content consistency.

4. Core idea and context

SfM / COLMAP needs many views and stable matching, which two low-texture / low-overlap object images may lack. iFusion uses learned appearance conditioned on a reference and relative camera: optimize the transform until generation reconstructs the real query. This is test-time inversion rather than supervised general relative-pose regression.

My analysis: the objective is powerful but risky. In-distribution targets may be more stable than sparse matching; hallucinated plausible backsides can instead pull camera transforms toward false geometry. Downstream geometric metrics are essential cross-checks.

5. Method details

5.1 $SE(3)$ relative pose through diffusion inversion

Twist vector $\xi\in\mathfrak{se}(3)$ represents the transform as $T= T_0\exp(\xi)$. Reconstruction gradients from the frozen diffusion model update $\xi$, with symmetric forward / inverse constraints reducing one-direction-only degeneracy.

To reduce local minima, optimization starts from front, left, right, and back canonical poses $T_0$ and anneals denoising timesteps coarse to fine. Initial directions yield distinct generative trajectories; a single $T_0$ can get trapped by symmetry or low overlap.

5.2 From two aligned images to a target-specific generator

After estimating $\hat T$, iFusion adapts only small LoRA attention parameters instead of all Zero123 weights. Stochastic multiview conditioning uses aligned views to retain general priors while matching the target identity. Generated views or SDS/reconstruction losses then guide interchangeable 3D backends.

Stage	Optimized variables	Frozen / required conditions	Output
Pose inversion	$\xi$	Frozen Zero123, two object images	$\hat T_{r\rightarrow q}$
Sparse-view fine-tuning	LoRA parameters $\phi$	Estimated $\hat T$ and few observations	More identity-consistent NVS generator
3D reconstruction	Backend geometry parameters	Generated views or SDS	3D asset

6. Experimental analysis

6.1 Data and evaluation protocol

The authors select 70 objects each from GSO and OO3D, render 5 views per object for pose experiments, and report 1400 views per dataset in §4.1. These counts do not directly agree as independent rendered frames (70×5=350), so 1400 is retained only as the paper’s reported value; the sampling / pairing count remains to be checked, see original §4.1. Reconstruction starts from the two views with greatest parallax. Pose uses median rotation / translation errors and recall satisfying both 5° thresholds; NVS uses PSNR, SSIM, LPIPS; reconstruction uses Chamfer Distance and volumetric IoU.

This controlled rendering protocol tests prior-based pose inversion. Its main results do not cover casual photos’ backgrounds, exposure, occlusion, or inconsistent focal lengths.

6.2 Main results

Dataset	iFusion rotation ↓	Translation ↓	5° recall ↑	Comparison
GSO	2.29	2.22	74.79	RelPose++ 109.89 / 90.58 / 0.21; FORGE recall 0
OO3D	2.97	2.80	69.29	RelPose++ and FORGE both have recall 0

GSO NVS also reaches PSNR 18.73, SSIM 0.836, LPIPS 0.121, showing downstream synthesis gains beyond pose-loss reduction. NVS also depends on LoRA, generation, and reconstruction backends, so gains cannot all be attributed to pose initialization.

6.3 Ablations and cost

• Four initial poses give recall 74.79 in about 70.59 s; eight give 78.21 in about 133.7 s.
• Four-way initialization gains support multimodal camera solutions; nearly doubled runtime also points toward offline asset production.
• Timestep annealing plausibly uses coarse denoising’s broad basin before fine texture alignment. Stability improves, but there is no rigorous real-image calibration study.

7. Critical analysis and failures

Authors’ claims and evidence boundaries

• Strong diffusion priors recover sparse-view pose: directly supported by synthetic GSO / OO3D Table 1.
• Better 3D reconstruction: supported by NVS, Chamfer, and IoU, including backend contributions.
• Pose-free real captures: a reasonable direction, but main quantitative evidence uses rendering and does not establish metric-accurate casual-photo reconstruction.

Potential failures

1. Homogeneous textures, approximate rotational symmetry, or tiny overlap can admit multiple visual explanations.
2. Out-of-distribution priors may invent geometry, jointly biasing pose and reconstruction.
3. LoRA and reconstruction both run at test time; practical accounting must include cumulative latency and memory.

8. Implications for pose research

• Use generative-rendering likelihood to verify pose hypotheses rather than treating diffusion as geometric truth.
• Combine image priors with epipolar, silhouette, calibration, or 3D consistency constraints, especially for metric robot-grasp pose.
• Report relative camera pose, generation quality, reconstruction quality, and real object pose separately; one visual score cannot establish all four.