Deep Reading Notes: Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects

1. Basic information

• Method: Betsu-Betsu
• Authors / affiliations: Suhas Gopal, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt; Saarland University, Max Planck Institute for Informatics
• Year / venue: 2025 / 3DV
• Paper: https://arxiv.org/abs/2502.13968
• PDF: https://arxiv.org/pdf/2502.13968
• Project: https://vcai.mpi-inf.mpg.de/projects/separable-recon/
• Reading mode: Deep Dive

2. Positioning in one sentence

Betsu-Betsu performs separable neural implicit reconstruction of two objects from static multi-view captures. Shared scene features explain occlusion, while two SDF heads and an alpha-blending regularizer discourage a 3D point from belonging to both opaque objects, suppressing geometric interpenetration in contact zones.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Multi-view inputs and separable outputs	The objective is independent geometry and renderings for each object, beyond a joint mesh
Figure 2	Shared hash grid, two SDFs, and alpha blending	Separation is constrained at the opacity / volume-rendering level, beyond 2D semantic labels
Table 1	Overall / object Chamfer on human–object data	Examine the whole scene and the small object separately; human geometry can dominate overall scores
Tables 2, 5	AffordPose and novel-view synthesis	Check whether separation improves geometry and appearance together

3. Problem setting and boundaries

• Inputs are $K$ calibrated multi-view RGB images and segmentation masks $\mathcal M_1,\mathcal M_2$ for the two interacting objects.
• Outputs are separate SDFs / meshes for objects 1 and 2, joint scene appearance, and individual novel-view renderings.
• Pose is not a predicted variable; camera poses are given. The method addresses a geometric prerequisite for pose / interaction refinement: maintaining nonoverlapping surface ownership when objects touch and occlude one another.
• Both objects are assumed opaque and the scene static throughout multi-view capture. Each scene requires lengthy optimization.

4. Research motivation

Training a separate NeRF / SDF for each object creates conflicts in occluded regions: a single-object model may misinterpret invisibility caused by another object as geometry that should be carved away. Training one joint scene instead loses object ownership. ObjectSDF++ also predicts multiple SDFs, but the authors argue that opacity supervision alone may still produce interpenetrating phantom geometry inside objects.

The key insight is that, for two opaque objects, both opacities should not be high at the same sampled point. This constraint can penalize overlap through $\alpha_1\alpha_2$ without knowing which object should own that point.

5. Method details

5.1 Shared encoding, separate shape heads

Given a spatial point $x$, a multiresolution hash-grid encoding $h(x)$ and a feature network generate shared scene features $g$. Two separate MLPs output $\Phi_1(x),\Phi_2(x)$. The joint scene SDF is combined as:

$$\Phi_s(x)=\min(\Phi_1(x),\Phi_2(x))$$

A color network then predicts scene color from position, viewing direction, normals, and $\Phi_s$. Shared encoding gives both objects access to the same scene context, while separate SDF heads preserve instance boundaries.

5.2 Why alpha blending reduces interpenetration

A NeuS-style mapping converts each SDF to $\alpha_1,\alpha_2$. Their joint opacity is combined as:

$$\alpha_s=\alpha_1+\alpha_2-\alpha_1\alpha_2$$

The defined $\mathcal L_{alpha}$ penalizes simultaneous opacity in both objects, encouraging each point to be opaque for only one object. The total loss includes individual-object color, scene color, $\mathcal L_{alpha}$, and Eikonal regularization.

A critical detail is that this is not a hard SDF nonoverlap constraint. It softly constrains opacity at rendering sample locations. Separation in unobserved interiors is therefore induced by regularization, rather than directly supervised by true geometry.

Module	Input / output	Purpose	Difference from single-object reconstruction
Shared hash grid + feature network	Position → shared features	Encode common scene information	Keep occlusion relationships within shared context
Two SDF heads	Shared features → $\Phi_1,\Phi_2$	Give each object independent geometry	Produce surfaces with instance ownership
Scene SDF / color	Two SDFs → joint rendering	Explain occlusion and overall RGB	Keep color at scene level rather than blindly duplicating it
Alpha regularizer	$\alpha_1,\alpha_2$	Reduce interpenetration in contact zones	Constrain ownership inside objects

6. Experimental analysis

6.1 Datasets and metrics

The paper captures close human–object interactions in a capture dome and also evaluates on AffordPose (hand–object), WildRGB-D (object–object), NeuralDome (human–object), and ReMoCap (human–human). Geometry is evaluated with bidirectional Chamfer Distance (lower is better); appearance uses PSNR, SSIM, and LPIPS on held-out views. 3D IoU additionally measures overlap between the two objects.

Read overall-scene human–object scores carefully. Humans are usually much larger than the interacting object, so human reconstruction quality can dominate overall Chamfer. Table 1 therefore also reports individual-object CD, which is more informative about whether separation actually works.

6.2 Key results

Task / result	Segmented NeuS2	ObjectSDF++	Betsu-Betsu	Interpretation
Human–object overall CD mean	5.74	13.89	5.51	Joint-scene quality matches or slightly exceeds separately trained NeuS2
Human–object object CD mean	59.99	15.90	11.40	Separating individual-object geometry provides the main gain
AffordPose hand CD mean	7.00	6.18	5.87	Hand reconstruction also improves
AffordPose object CD mean	9.26	6.79	3.80	Small objects benefit more strongly
Novel view mean	—	28.02 / 0.95 / 0.08	32.42 / 0.97 / 0.06	PSNR / SSIM / LPIPS all improve

The ablations in Figures 7–9 further show that removing alpha regularization or sharing the SDF head can leave apparently normal renderings while phantom blobs / interpenetration appear inside surfaces when viewed transparently. This is the paper’s most convincing point: rendered RGB alone struggles to penalize invisible interior errors, motivating an ownership-oriented prior.

6.3 Boundaries of the evidence

• “Better separated geometry than ObjectSDF++”: Supported by object-level Chamfer, novel-view synthesis, and qualitative comparisons.
• “Template-free and applicable to arbitrary objects”: Demonstrated across interaction types, but still limited to two objects, offline multi-view capture, mask supervision, and static scenes.
• My analysis: Reducing interpenetration could improve downstream visibility / contact reasoning, but a differentiable rigid transform or explicit pose head is still needed. Betsu-Betsu itself does not produce online 6D pose.

7. Limitations and potential failures

1. Demanding capture conditions: Camera intrinsics/extrinsics, multiple views, and per-object masks are required inputs. Continuous pose estimation in online video is not addressed.
2. Fixed object count: Two SDF heads and two-object alpha composition directly constrain the main method. More objects introduce combinatorial complexity, identity swaps, and growing resource requirements.
3. Material assumptions: Transparent, translucent, and specular objects violate the assumption of one opaque object per point.
4. Occluded interior surfaces: Alpha regularization reduces overlap but cannot observe persistently occluded geometry. Results still depend on neural-field priors.

8. Implications for object pose and interaction geometry

• Pose refinement can adopt per-point ownership: allocate rendering residuals by object visibility / occupancy to prevent two models from jointly explaining the same pixel.
• For contact manipulation, a physical nonoverlap prior on the joint objects can be combined with 6D rigid pose, contact distances, and symmetry-aware losses.
• Further evaluation should measure whether separable reconstruction improves pose/contact errors, beyond Chamfer and novel-view synthesis.