CRISP: Object Pose and Shape Estimation with Test-Time Adaptation
CRISP jointly estimates implicit shape and pose-normalized points, turning an active shape model corrector into a certifiable pseudo-label generator. Test-time self-training addresses large domain gaps.
Authors / affiliation: Jingnan Shi, Rajat Talak, Harry Zhang, David Jin, Luca Carlone; MIT LIDS.
Task: jointly recover category-agnostic pose and shape from RGB-D, with self-training on unlabeled deployment-domain data.
Reading mode: Overview.
2. Positioning in one sentence
CRISP goes beyond a single pose prediction: an implicit-shape and pose-normalized-point front end is followed by a correctable, certifiable shape model that generates reliable pseudo-labels for test-time adaptation.
3. Problem and motivation
The domain gap between synthetic training and real deployment can undermine accurate pose / shape systems.
Treating every prediction as a pseudo-label amplifies errors; the system must first determine when self-training is worthwhile.
4. Method overview
A FiLM-conditioned decoder predicts implicit shape, while a DPT network predicts pose-normalized points.
Approximate the decoder within the convex hull of known shapes using an active shape model, then correct pose / shape through constrained least squares.
Accept only samples passing the object-consistency check, and self-train while keeping the decoder fixed.
5. Experiments at a glance
YCB-V mean shape error is 0.026 for CRISP-Real, versus the paper’s reported 0.099 for Shap-E.
On unseen SPE3R satellites, CRISP-ST reduces mean pose error from CRISP’s 0.292 to 0.224.
YCB-V results from the CRISP paper: the left table measures shape-reconstruction error, while the right measures pose-related ADD-S. They describe different capabilities; shape error alone does not establish pose accuracy. Source: original CVPR 2025 paper. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.
CRISP-Real’s mean YCB-V shape error is 0.026, below Shap-E’s 0.099, while mean ADD-S error is 0.011. These respectively assess shape similarity and model-point alignment under the estimated pose, and should be read together. Test-time self-training is particularly valuable for out-of-domain objects: SPE3R mean pose error falls from 0.292 to 0.224.
Key question: under a large domain gap, can unlabeled test-domain images jointly correct object shape and pose without compounding incorrect pseudo-labels?
Core answer: approximate local decoder behavior with an active shape model to obtain efficiently solvable constrained least squares, then select self-training samples through correct-and-certify.
2. Positioning in one sentence
CRISP closes a loop between a pose-and-shape front end, an explicit geometric corrector, and test-time self-training. The central question is whether predictions can be corrected reliably, beyond simply making a prediction.
2.1 Guide to the paper’s figures and tables
Figure / table
What to look for
Figure 2
Data flow through implicit shape, PNC pose, the corrector, and self-training.
Tables 1–2
Read shape error and ADD-S separately; neither substitutes for the other.
Tables 3–5
Do both unseen SPE3R satellites and category-level NOCS benefit from self-training?
Figure 5
Does the object-consistency check actually reject high-shape-error pseudo-labels?
Method-figure region from the paper: CRISP predicts shape and pose-normalized points from RGB-D, then generates test-time pseudo-labels through correct-and-certify. Source: CVF Open Access. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.
3. Problem setting and boundaries
Input: segmented RGB and depth points; output: 6D pose and shape, without target-CAD instance onboarding.
YCB-V is a seen-object setting; SPE3R and NOCS emphasize shape / category generalization. Their different challenges cannot be collapsed into one ranking.
Test-time self-training requires a buffer of target-domain images, rather than a single frame with no additional inference cost.
4. Method details
Front end: a FiLM-conditioned decoder maps a latent code to implicit shape; a DPT backbone outputs pose-normalized coordinates / points.
Corrector: approximate the decoder near the convex hull of known shape latents with an active shape model, turning pose / shape correction into constrained linear least squares. The paper compares interior-point solving and block coordinate descent.
Certify and self-train: use only corrected estimates passing the object-consistency check as pseudo-labels; freeze the decoder and update the prediction networks.
5. Experimental analysis
Setting
Key reported result
YCB-V shape
CRISP-Real mean eshape = 0.026; Shap-E = 0.099 in the paper.
Unseen SPE3R satellites
Mean eposeL1 is 0.292 for CRISP and 0.224 for CRISP-ST.
Pseudo-label certification
CRISP-Syn passes the check on 6.1% of samples without a corrector, versus 10.7% / 16.6% after LSQ / BCD.
The key evidence is that the corrector lets more samples enter training under an internal-consistency criterion, rather than a guarantee that self-training always improves performance.
Test-time adaptation evidence from the CRISP paper: curves show pseudo-label selection, while tables report SPE3R pose / shape errors and REAL275 category-level shape results separately. Source: original CVPR 2025 paper. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.
On SPE3R, CRISP-ST reduces mean shape error from 0.163 to 0.141 and median error from 0.159 to 0.125; mean pose error falls from 0.292 to 0.224 and median from 0.211 to 0.168. Both shape and pose improve, indicating gains in geometric alignment beyond appearance alone. These results nevertheless depend on the target-domain sequence entering the buffer and the certification threshold; they do not establish stateless, single-frame plug-and-play inference.
NOCS / REAL275 evaluation, corrector ablations, and runtimes from the CRISP paper. BCD allows more samples to pass the consistency check, but costs substantially more than a forward pass or LSQ. Source: original CVPR 2025 paper. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.
The efficiency table reveals an important engineering constraint: approximately 125 ms for the CRISP front end, 251 ms for the LSQ corrector, and 3.8 s / 7.6 s for 25 / 50 BCD iterations. BCD’s 16.6% pseudo-label pass rate exceeds 6.1% without correction and 10.7% with LSQ, but is better suited to low-frequency adaptation than a per-frame real-time back end. Applications targeting 10–30 FPS should consider asynchronous buffer updates, infrequent correction, or confidence-triggered activation.
6. Authors’ claims and my assessment
Authors’ claims
My assessment
Certified pseudo-labels bridge large domain gaps
Supported by SPE3R / YCB-V results and pass rates; cross-domain sensitivity to certification thresholds should also be disclosed.
Joint shape / pose estimation is more stable than pose alone
Both metrics improve, but error propagation between them still needs analysis.
7. Reproduction considerations and limitations
Reproduction must fix segmentation masks, the latent-shape training distribution, consistency thresholds, test-buffer length, and corrector iteration caps.
BCD may yield more passing samples than LSQ but is substantially slower; per-frame real-time use is inappropriate.
Transparent, strongly reflective, or severely depth-incomplete objects can violate the corrector’s RGB-D geometric assumptions and cause failure.
“Certify” here refers to the paper’s object-consistency check, not a formal global guarantee of true-pose accuracy. Passing the internal check should not be described as necessarily recovering ground truth.
