Deep Reading Notes: FAFA

1. Key assessment

FAFA studies a harder but narrower setting than typical indoor pose: near-infrared depth often fails underwater and real 6D annotation is costly, so it relies on RGB, known UUV CAD, synthetic GT, and unlabeled target-real images.

It divides sim-to-real adaptation into two stages:

• Change training-input spectra first: retain synthetic phase / geometry and incorporate real amplitude / style;
• Then change refinement constraints: teacher–student consistency covers photometric warps, rendered masks, and features as well as final pose / flow.

ROV6D mean 91.39, close to labeled upper bound 92.43, is the strongest result. DeepURL reaches 73.46 versus PFC’s 68.72 but remains far below upper bound 92.68. Multilevel self-supervision is effective; unlabeled adaptation has not generally replaced real labels.

2. Precise task boundaries

Given:

• Known UUV 3D CAD;
• Synthetic RGB with pose GT;
• Unlabeled real RGB from the target water environment;
• camera intrinsics;
• Initial pose $P_0$ from an external coarse estimator.

Refine accurate $SE(3)$ from real underwater RGB. This is instance-level, model-based, monocular pose refinement, rather than:

• Full-image detection + pose estimation end to end;
• Absolute pose without initialization;
• Unseen-object / category-level pose;
• Domain generalization without target-domain samples.

FFT pretraining already reads amplitudes of random target-real images. OURS (LB), although lacking teacher–student fine-tuning, is therefore not strictly synthetic-only; its Real Data column explicitly says Unlabeled.

3. FFT augmentation: target style enters pretraining

For image $x$:

$$\mathcal F^A(x)=\sqrt{R(x)^2+I(x)^2},\qquad \mathcal F^P(x)=\arctan\frac{I(x)}{R(x)}.$$

Given synthetic $x^s$ and real $x^r$, construct:

$$\hat{\mathcal F}^A(x^s,x^r)= \begin{cases} (1-\alpha)\mathcal F^A(x^s)+\alpha\mathcal F^A(x^r), &\delta<\delta_0\\ 1,&\text{otherwise}, \end{cases}$$

Here $\alpha\sim U(0,\beta)$, with $\delta_0=0.5,\beta=1$ in main experiments. Invert this amplitude with synthetic phase:

$$x^{s\rightarrow r}= \mathcal F^{-1} \left[ \hat{\mathcal F}^A(x^s,x^r)e^{-j\mathcal F^P(x^s)} \right].$$

Amplitude mixing injects target-water color cast, lighting, and blur; dropout encourages phase / contour reliance. This is inexpensive domain randomization, but phase-as-geometry and amplitude-as-style are approximations. Strong scattering, occlusion, and caustics also change phase, while global mixing cannot model spatially varying water.

4. Iterative flow–pose refinement

Starting from coarse $P_0$, render $N=4$ synthetic images around it. Iteration pose $P_j=[R_j|t_j]$ projects CAD point $p_i$ as follows:

$$\lambda_i \begin{bmatrix}u_i\\1\end{bmatrix} =K(R_jp_i+t_j).$$

Projection differences relative to $P_0$ form shape-constrained flow. Dense 2D–2D correspondences feed decoupled pose updates $R_\Delta,t_\Delta$, with continuous 6D rotation. Updated poses generate new geometric flow for the next iteration.

The teacher uses 8 iterations, the student 4; teacher weights follow student EMA with factor 0.999. Only forward / backward geometrically consistent clean-input teacher flow becomes pseudo-labels. This rejects some underwater texture errors, but valid-pixel fractions, threshold sensitivity, and retention across turbidity are not reported.

5. Multilevel self-supervision

Image-level:

$$\mathcal L_{\text{img}}= \gamma_1\mathcal L_{\text{flow}}+ \gamma_2\mathcal L_{\text{photo}}+ \mathcal L_{\text{warp-mask}}.$$

• $\mathcal L_{\text{flow}}$: fit student flow to filtered teacher pseudo-flow.
• $\mathcal L_{\text{photo}}$: backward-warp real images with both flows and compare using Census distance.
• $\mathcal L_{\text{warp-mask}}$: L1 between teacher / student forward-warped rendered masks.

Feature-level:

• Warp real features with student flow;
• Measure weighted dissimilarity against synthetic features;
• Seek a latent metric more robust than raw photometric errors under low light, color cast, and blur.

Pose-level:

• Use teacher poses as pseudo-poses;
• Model-point matching $\mathcal L_{\text{pose}}$ constrains student rotation / translation.

Total loss:

$$\mathcal L_{\text{self}}= \gamma_3\mathcal L_{\text{pose}}+ \gamma_4\mathcal L_{\text{img}}+ \mathcal L_{\text{feat}},$$

Here $\gamma_1=\gamma_2=0.1,\gamma_3=\gamma_4=10$. Freeze BatchNorm during self-supervision to avoid mixing source / target statistics.

6. Data and evaluation protocol

ROV6D

• Object: BlueROV, treated as symmetric under the paper’s ADD-S protocol.
• synthetic: PBR.
• Target real data: 1,000 Pool-Basic images for self-supervision.
• test: Pool-Occluded.
• Excluded: the small, low-visibility open-lake Maoming subset.

The split tests adaptation from basic to occluded pool scenes, not pool-to-open-water generalization.

DeepURL

• Object: Aqua2, evaluated with ADD.
• An additional 30,000 PBR synthetic images are generated with BlenderProc.
• A random 1,000 real images support self-supervision; remaining real images evaluate without overlap.
• Non-self-supervised baselines use UE4 + CycleGAN target style; PFC and FAFA use PBR + the same 1,000 unlabeled real images.

DeepURL Table 2 mixes synthetic sources. FAFA versus PFC is comparatively fair; comparisons with DeepURL / GDRN / DeepIM / SCFlow also reflect PBR versus UE4+CycleGAN differences.

7. Main results

ROV6D

Method	Real supervision	ADD-S 0.05d	ADD-S 0.1d	5°	5cm	Mean
OURS (LB)	unlabeled, FFT only	87.68	93.54	78.93	80.10	85.06
PFC	unlabeled	86.56	93.50	81.34	72.10	83.38
OURS	unlabeled	94.03	96.96	87.87	86.71	91.39
OURS (UB)	labeled	95.08	96.21	88.81	89.60	92.43

OURS improves PFC mean by +8.01 points, mainly through 5cm +14.61 and 5° +6.53. Its 0.1d exceeds UB by 0.75, but 0.05d, rotation, translation, and mean remain lower. Winning one loose threshold does not establish superiority over the supervised upper bound.

DeepURL

Method	Real supervision	ADD 0.05d	ADD 0.1d	5°	5cm	Mean
OURS (LB)	unlabeled, FFT only	18.62	73.36	48.10	38.50	44.64
PFC	unlabeled	70.30	98.87	61.60	44.10	68.72
OURS	unlabeled	76.56	97.64	68.70	50.92	73.46
OURS (UB)	labeled	85.60	97.42	95.24	92.52	92.68

OURS improves PFC mean by +4.74 points. The 5cm change 44.10→50.92 is +6.82 points; the paper’s 15.46% is relative improvement. PFC leads loose ADD 0.1d by 1.23. Self-supervision still trails UB by 26.54 / 41.60 at 5° / 5cm, leaving precise Aqua2 rotation / translation unresolved.

8. Ablations: contributions of each stage

FFT pretraining:

Configuration	Mean
No FFT augmentation	35.10
Amplitude mix only	39.06
Amplitude dropout only	37.45
mix + dropout, LB	44.64
full pretrain + self-supervision	73.46
Continue FFT during self-supervision	67.71

Mix and dropout complement each other, but continuing real-image perturbations during target self-supervision distorts the real distribution, reducing mean by 5.75.

Self-supervised losses:

Photo	Warp mask	Feature	Mean
—	—	—	64.29
✓	—	—	69.41
✓	✓	—	70.68
—	—	✓	66.77
✓	✓	✓	73.46

Photo provides the largest individual gain; feature-only also helps, while all three levels work best. The reported 23.28% photo gain at 5° is relative, from 50.31→62.02; the absolute gain is +11.71 points.

9. Engineering settings

• PyTorch on one RTX 3090;
• AdamW, batch size 16;
• OneCycle, maximum learning rate $4\times10^{-4}$;
• Initial-pose crops resized to 256×256;
• $N=4$ rendered views;
• teacher / student iterations = 8 / 4;
• EMA = 0.999;
• Frozen BatchNorm during self-supervision.

Not reported:

• Latency including coarse estimation, 4-view rendering, and 12 teacher/student iterations;
• Per-frame GPU memory;
• Real-time suitability for UUV control loops;
• Initial-pose perturbation range or convergence basin.

Using flow refinement alone does not establish online deployability.

10. Limitations and risks

• Few objects: only BlueROV and Aqua2 across two datasets, limiting cross-geometry conclusions.
• Target data remain necessary: 1,000 real images per new water domain; FFT pretraining already uses its style.
• Coarse-pose dependence: initialization controls render / observation overlap, without systematic sensitivity analysis.
• No main open-water results: small Maoming data are excluded; principal ROV6D experiments cover pools.
• Loose symmetry convention: symmetric ADD-S for BlueROV reduces orientation ambiguity penalties; also examine 5°.
• Pseudo-flow / pose bias: shared teacher–student origins mean EMA smooths but cannot automatically correct systematic errors.
• Approximate appearance assumptions: phase / amplitude do not strictly equal geometry / style.
• Mixed protocols: DeepURL tables combine PBR and UE4+CycleGAN baselines.
• No runtime: real robot closed-loop suitability cannot be assessed.
• Code status in the source notes: the paper / ECCV page promised code, but the checked GitHub repository had only README and two commits, without training / inference implementation. Metadata therefore does not mark it as released code.

11. Reproduction checklist

• Fix BlueROV / Aqua2 CAD, intrinsics, frames, and symmetry;
• Save the exact 1,000 unlabeled-real IDs and evaluation split;
• Generate the same 30,000 DeepURL PBR images;
• Separate FFT-only LB, teacher–student self-supervision, and real-label UB;
• Record amplitude sampling with $\delta_0=0.5,\beta=1$;
• Report coarse-estimator errors in $P_0$ bins;
• $N=4$, teacher 8 / student 4 iterations;
• EMA 0.999, BN frozen;
• Reproduce flow-consistency thresholds and valid-pixel fractions;
• Report ADD(-S), 5°, and 5cm separately, without hiding precise-pose gaps in a mean;
• Include detection / coarse pose / rendering / refinement in end-to-end timing;
• Add open-water and cross-water domain-shift tests.

12. Overall assessment

FAFA’s design fits underwater constraints: replace unavailable depth with shape-constrained CAD-projection flow; constrain noisy pseudo-poses through images, masks, features, and model points; and introduce target style through FFT before formal self-supervision.

The strongest evidence is ROV6D mean 83.38→91.39, DeepURL 68.72→73.46, and full-loss ablation 64.29→73.46. Key boundaries remain two known UUVs, initialization and 1,000 target-real images, no open-lake evaluation or runtime, and no implementation in the repository when checked. This is persuasive underwater pose refinement / adaptation, rather than a general underwater 6D pose system.