iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos
From RGB-D video containing handheld camera motion, object motion, and occlusion, iTACO obtains coarse joint states from static / dynamic correspondences, then optimizes joint parameters and articulated parts through differentiable refinement and part segmentation.
iTACO reconstructs articulated digital twins from casual handheld dynamic RGB-D video. Static / dynamic correspondences initialize camera pose, part motion, and coarse joints; differentiable refinement with automatic part segmentation recovers joint parameters and geometry.
3. Problem setting
Input: casual dynamic RGB-D video with simultaneous camera / object motion and frequent hand occlusion.
No carefully calibrated multiview turntable or static camera assumption; still depends on depth, static-region matching, and upstream masks.
4. Method overview
LoFTR matches static regions for relative camera poses and a common frame.
Dynamic correspondences estimate part rigid transforms and coarse joints.
Moving-part segmentation aggregates pixels for differentiable global refinement of camera pose, moving maps, and joints.
5. Experimental overview
New dataset: 784 videos, 284 objects, 11 categories, about 20 times prior comparable datasets.
S-Dataset revolute joints: axis / position / type / state / failure = 0.32 rad / 0.13 m / 15.90% / 0.25 rad / 6.81%.
Moving-part CD is 0.13, rising to 0.40 without refinement and 0.39 without segmentation.
6. Strengths, weaknesses, and implications
Strengths
Simultaneous camera / object motion better reflects smartphone capture.
Coarse-to-fine estimation makes part segmentation an optimization variable rather than preprocessing alone.
Limitations / risks
Texturelessness, wrong part membership, and occluded newly visible regions harm matching; interiors are not reconstructed.
Requires RGB-D and per-video reconstruction, not real-time single-frame pose.
7. Points for deep reading
Removing the coarse stage gives revolute type error 77.27%, showing refinement cannot recover articulation reliably from unstructured initialization.
Available src/paper-notes/itaco-3dv2026/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
iTACO: Articulated Digital Twins from Casual RGB-D Video
1. Paper information
Paper: iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos
Venue: 3DV 2026
Problem: recover camera motion, part structure, joint type / axis, and motion state from casual handheld dynamic RGB-D video without CAD templates.
Main conclusion: motion-based coarse initialization followed by joint differentiable optimization of cameras, masks, dynamic point maps, and joints turns coupled single-camera scale / motion into iterative reconstruction.
2. Method in depth
2.1 Two stages: initialize geometry, then refine kinematics
Static LoFTR matches initialize relative cameras and scene alignment; dynamic correspondences initialize moving-part rigid transforms and joints. This separates camera / object motion early enough to avoid clearly wrong joint-optimization minima.
The second stage combines automatic part segmentation, cameras, dynamic point maps, joints, and dense geometry. Reasonable coarse estimates repeatedly return to per-frame image / motion consistency constraints rather than solving everything accurately at once.
2.2 Relative motion makes joints observable
Shared invariants of relative rigid transforms constrain revolute axes and points on them; prismatic motion follows a fixed direction. Object motion links frames rather than treating them as independent reconstructions. Tiny motion, weak texture, or dynamic occlusion weakens these invariants, limiting single-camera recovery.
3. Data and results
The dataset has 784 videos, 284 objects, and 11 categories covering revolute / prismatic mechanisms. Synthetic / benchmark evaluation reports axis, position, type, state, and failure: revolute errors 0.32 / 0.13 / 15.9% / 0.25 / 6.81%; prismatic axis 0.24, type 10.34%, state 0.08, failure 0.
Full moving-part Chamfer Distance is 0.13; no refinement gives 0.40, no coarse initialization 0.19, no segmentation 0.39. Refinement handles continuous errors while segmentation assigns correct rigid membership; both are substantial components.
4. Reproduction workflow
Step
Main signal
Purpose
Common failure
Static matching
Background / static LoFTR correspondences
Initial relative cameras
Weak background texture or pure rotation
Dynamic correspondence
Moving-part trajectories
Coarse rigid transforms / joints
Tiny motion, occlusion, wrong tracking
Part segmentation
Automatic masks
Rigid membership
Merged parts, similar appearance
Joint refinement
Rendering / reprojection / motion consistency
Correct camera, geometry, and joints
Poor initialization causes local minima
Mechanism output
Type, axis, position, state
Editable articulated asset
Unobserved internals remain uncertain
5. Relationship to 6D pose
iTACO uses time to build canonical representations and motion constraints without CAD. A robot can estimate an unknown tool / household object’s articulation, then use local part frames for grasping, tracking, or relocalization.
6. Limitations and usage
Sufficient observable motion is essential. Texturelessness, heavy occlusion, wrong masks, and unopened / unextended parts weaken estimation. Hidden interiors are outside scope. Output joint confidence together with visual evidence, motion amplitude, and segmentation stability rather than presenting unobservable structure as precise.
7. Why two stages matter
Full S-Dataset whole / moving / static Chamfer Distance is 0.01 / 0.13 / 0.06. Without refinement, moving / static becomes 0.40 / 0.35: coarse correspondences initialize motion but not fine geometry / stable boundaries. Removing coarse estimation gives moving CD 0.19, a modest surface change, yet revolute type error 77.27%, showing geometrically plausible but kinematically wrong local minima. Without segmentation, moving / static CD 0.39 / 0.36 confirms rigid membership’s role in differentiable refinement.
Cameras are estimated too: supplementary rotation / translation errors are iTACO 0.08066 / 0.09246, MonST3R 0.14482 / 0.08729, CUT3R 0.14663 / 0.10536. Rotation especially improves. Generic dynamic-scene camera errors propagate into part transforms / axes, making static-region matching a core anchor.
8. Failures and improvements
Reported failures center on segmentation: new drawer interiors merge with front panels; similar parts stick together; a bin lid changes segment identity and is later filtered as absent from the first frame. Temporally inconsistent masks make Chamfer correspondence pull parts onto wrong surfaces. The authors suggest 3D segmentation such as SAMPart3D / PartField; a further direction is probabilistic part identity with temporal / adjacency priors rather than one hard mask.
Capture should ensure static background texture, identifiable joint motion, and clear views of each movable part in several frames. Active robots could choose next views / articulations from current joint uncertainty instead of passively processing video.
