Object Pose Estimation · 2025 · IROS · CCF C

Category-Level 6D Object Pose Estimation in Agricultural Settings Using a Lattice-Deformation Framework and Diffusion-Augmented Synthetic Data

PLANTPose explicitly represents category-instance deformation with a 24D cubic B-spline lattice and removes its rigid component with Umeyama alignment. Blender physics scenes and depth-conditioned Stable Diffusion generate training images. A small ViT jointly predicts rotation, deformation, and projections from RGB, while PnP recovers camera-frame translation.

Paper Project Code

Research area

Object Pose Estimation

Task level

Category-level

Object setting

Unseen objects

Input modalities

single RGBcategory base mesh

Requirements

known object categorycategory base meshobject detector / cropcamera intrinsicscategory-specific synthetic training

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Marios Glytsos, Panagiotis P. Filntisis, George Retsinas, Petros Maragos
Affiliations: Robotics Institute, Athena Research and Innovation Center / National Technical University of Athens
Conference / journal: Conference · IROS · CCF C
Research area: Object Pose Estimation
Code availability: Code available
Reading status: Overview:Available Deep Dive:Available
Task setting: Category-level · Unseen objects
Datasets: PLANTPose banana benchmark PLANTPose synthetic training set
Evaluation metrics: Chamfer Distance Rotation Error Translation Error
Method tags: Category-level Pose Estimation rgb-only deformable-object lattice-deformation synthetic-data diffusion agricultural-robotics ransac-pnp Absolute Pose Estimation
Requirements: known object categorycategory base meshobject detector / cropcamera intrinsicscategory-specific synthetic training

Reading Notes

Overview / Deep Dive

Available src/paper-notes/plantpose-iros2025/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Paper Reading Notes (Coarse): PLANTPose

1. Basic information

Title: Category-Level 6D Object Pose Estimation in Agricultural Settings Using a Lattice-Deformation Framework and Diffusion-Augmented Synthetic Data
Authors: Marios Glytsos, Panagiotis P. Filntisis, George Retsinas, Petros Maragos
Affiliations: Robotics Institute, Athena Research and Innovation Center; National Technical University of Athens
Year / venue: 2025 / IROS
Paper: arXiv · IEEE DOI · Code and data
Reading mode: Overview

2. Positioning in one sentence

A category base mesh represents bananas, with 24 corner-coordinate offsets on a 2×2×2 lattice describing instance deformation. Blender generates physically plausible shapes / poses; depth-conditioned Stable Diffusion changes texture. From one RGB image, joint rotation, deformation, and projection constraints support PnP recovery of camera-frame translation.

3. Guide to figures and tables

PLANTPose lattice deformation, physics, Stable Diffusion, and training pipeline — Original Figure 2: lattice deformation and Umeyama remove spurious rigid motion before Blender physics. Stable Diffusion inpainting improves object texture; generated data supervises pose and deformation. Source: authors' public manuscript. Image copyright belongs to the authors and IEEE; reproduced here for paper analysis.

PLANTPose versus MegaPose errors on the banana benchmark — Original Table I: PLANTPose improves Chamfer, rotation, translation, and deformation errors over MegaPose using the same average banana mesh. Source: authors' public manuscript. Table copyright belongs to the authors and IEEE; reproduced here for paper analysis.

4. Problem setting

Input: target RGB crop; a separately trained YOLOv11 supplies deployment detections.
Outputs: 6D rotation / translation and 24D deformation relative to the category base mesh.
Setting: unseen agricultural-category instances varying in shape, size, ripeness, and texture.
Data: 4,000 deformed synthetic scenes × 3 camera views, about 12,000 training images.

5. Method overview

Deform the base mesh with 3D offsets at 8 corners (24D) and smooth cubic B-splines.
Umeyama aligns original / deformed clouds to remove global rigid components from deformation labels.
BlenderProc randomizes object drops, occlusion, cameras, and lighting.
Stable Diffusion + depth ControlNet inpaints only objects after rendering, adding unripe, ripe, or rotten textures.
A 30.9M-parameter small ViT predicts 6D rotation representation, translation, and 24D deformation.
Train jointly with rotation MSE, deformation MSE, and 2D projection loss. Map crop coordinates to full images, then use deformed 3D points and intrinsics in PnP.

6. Experiments at a glance

Custom benchmark: 6 bananas at different shapes / ripeness, 100 RealSense D435 images, iPhone 14 Pro LiDAR scans, and manual annotation.
PLANTPose versus MegaPose: Chamfer 59.8 versus 90.1 mm, mean rotation 32.6° versus 52.4°, mean translation 42.5 versus 59.9 mm, deformation 12.1 versus 29.1 mm.
Without Stable Diffusion, mean Chamfer / translation is 71.5 / 48.9 mm; without Umeyama, 73.1 / 50.3 mm; without both, 89.7 / 61.1 mm.

7. Quick assessment

Strengths: interpretable low-dimensional meshes connect category pose and instance shape, with synthetic training, scanned real evaluation, and informative SD / Umeyama ablations.
Limitations: one banana category and 100 test images; separate category training; occasional diffusion pose changes; no real grasp success.
Audience: agricultural robotics, deformable category pose, synthetic data, and low-dimensional shape–pose disentanglement.

Available src/paper-notes/plantpose-iros2025/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Paper Reading Notes (Deep): PLANTPose

1. Key assessment

Agricultural-category variation extends beyond texture and scalar size to nonrigid shape. Matching an average banana mesh can compensate shape errors with wrong pose. PLANTPose explicitly absorbs instance shape in a 24D lattice and estimates 6D pose separately.

The main design disentangles deformation labels from rigid motion. Random corners cause bending plus global translation / rotation; raw offsets would conflate these. Umeyama alignment removes rigid components, and ablations show this is essential to pose–shape separation.

2. Task boundaries

Given a category base mesh and one RGB crop, predict:

Rotation $R\in SO(3)$ ;
Translation $t\in\mathbb{R}^3$ ;
Lattice corner offsets $\delta\in\mathbb{R}^{24}$ .

No inference depth is used, but the method requires:

A known category and base mesh;
YOLOv11 detection / crop;
camera intrinsics;
A category-trained PLANTPose network.

This is not arbitrary-CAD novel-object pose or generic deformable registration. Only bananas are trained; the mesh is a fixed training prior, not dynamic input.

3. Data and method pipeline

PLANTPose deformation, data generation, and joint prediction — Original Figure 2: Deformations & Umeyama Correction generates low-dimensional shapes and removes rigid components. Physics determines pose / occlusion; Stable Diffusion changes texture domains. Source: authors' public manuscript. Image copyright belongs to the authors and IEEE; reproduced here for paper analysis.

3.1 A 2×2×2 lattice and cubic B-splines

The mesh bounding box defines a lattice. Eight corners have 3D offsets $C_{i,j,k}$ , totaling 24 parameters. Normalize vertex $p$ to $(u,v,w)\in[0,1]^3$ , then weight cubic B-spline bases:

\delta_x(u,v,w)=\sum_{i,j,k=0}^{3}W_i(u)W_j(v)W_k(w)C^x_{i,j,k},

The $y,z$ components follow similarly, giving $p'=p+\delta(u,v,w)$ . Offsets extend / clamp to a 4×4×4 control grid for cubic boundary behavior. $C^2$ continuity yields smoother deformation than vertexwise noise and fixed-length network outputs.

This suits low-frequency bending, thickness, and banana silhouettes, but not topology changes, fractures, branching foliage, or high-frequency detail.

3.2 Removing spurious pose with Umeyama

After random lattice deformation, use

(s,R,t)=\operatorname{Umeyama}(P_{\text{original}},P_{\text{deformed}})

to estimate the best similarity transform, fixing $s=1$ , then correct lattice points to remove global rotation / translation from shape labels. Otherwise, $\delta$ and $R,t$ can explain the same deformation, making supervision ambiguous.

This assumes one-to-one mesh vertex correspondence. Real scans enter only Chamfer / ICP evaluation, not training deformation labels.

3.3 Physics synthesis and diffusion texture refinement

BlenderProc builds walls, floors, materials, lighting, and distractors. Deformed objects fall and settle physically; cameras sample around a point of interest, rendering 3 views per deformation.

After rendering, Stable Diffusion inpaints object regions to cover ripeness, spots, and rot missing from PBR. Depth-conditioned ControlNet aims to preserve shape / pose; prompts span unripe, ripe, rotten, and natural colors.

This is more controlled than direct text generation, but the authors acknowledge occasional slight 6D pose changes despite depth. Retaining original annotations then introduces label noise.

3.4 Network and losses

A small ViT (30.9M parameters, ImageNet-21k pretraining, 32×32 patches) processes object crops through three heads:

Rotation: 6D continuous representation orthonormalized into $R$ ;
Translation: a crop-frame 3D vector;
Deformation: 24D lattice offsets.

Total loss:

\mathcal{L}= \lambda_r\lVert r^{pred}_{6d}-r^{gt}_{6d}\rVert^2+ \lambda_d\lVert\delta^{pred}-\delta^{gt}\rVert^2+ \lambda_p\lVert\operatorname{Proj}(P\mid R,t,\delta)-v^{gt}_{2d}\rVert^2.

Projection loss compares shape / rotation / translation projections with GT mesh-vertex pixels. Undo crop offsets / scale to full-image coordinates, then solve PnP with deformed 3D points, 2D projections, and intrinsics for camera-frame translation.

The text somewhat mixes the translation head, projection loss, and final PnP translation. Reproduction must determine whether predicted translation is a crop-projection intermediate or a PnP initializer.

4. Real banana benchmark

Acquisition includes:

6 bananas varying in shape and ripeness;
100 RealSense D435 RGB images;
Separate iPhone 14 Pro LiDAR scans;
Manual image-pose annotations.

Without vertexwise scan / base correspondence, shape uses Chamfer distance. Rotation / translation errors follow ICP scan-to-base alignment; deformation error compares shapes after removing predicted pose.

ICP helps establish GT coordinates, making scan quality / local minima relevant. With only 100 images, per-banana, visibility, and ripeness statistics are absent.

5. Main results

PLANTPose versus MegaPose banana pose and shape errors — Original Table I: lower is better throughout. MegaPose uses the same average mesh without instance deformation. Source: authors' public manuscript. Table copyright belongs to the authors and IEEE; reproduced here for paper analysis.

Method	Chamfer (mm)	Mean / median rotation (°)	Mean / median translation (mm)	Deformation (mm)
MegaPose	90.1	52.4 / 43.6	59.9 / 45.6	29.1
PLANTPose	59.8	32.6 / 23.6	42.5 / 37.9	12.1

Against MegaPose, Chamfer drops 30.3 mm, mean rotation 19.8°, mean translation 17.4 mm, and deformation 17.0 mm. One unequal-task comparator mainly supports explicit deformation over an average mesh on this benchmark, rather than general category-level SOTA.

PLANTPose versus MegaPose in 2D and 3D across banana instances — Original Figure 3: green GT, blue PLANTPose, red MegaPose. The final column shows geometry / pose differences from another view. Source: authors' public manuscript. Image copyright belongs to the authors and IEEE; reproduced here for paper analysis.

PLANTPose better follows curvature and length; average-mesh errors affect MegaPose pose as well. Some 3D rotation / shape mismatches remain, and 32.6° mean rotation error falls short of fine agricultural manipulation.

6. Ablations

Setting	Chamfer	Mean / median rotation	Mean / median translation	Deformation
Full	59.8	32.6 / 23.6	42.5 / 37.9	12.1
w/o Stable Diffusion	71.5	37.7 / 23.0	48.9 / 48.7	13.9
w/o Umeyama	73.1	36.5 / 26.2	50.3 / 45.8	12.7
w/o Umeyama and SD	89.7	36.1 / 26.7	61.1 / 61.9	26.5

Stable Diffusion mainly improves real-texture transfer; Umeyama prevents deformation labels absorbing rigid motion. Not every column worsens: without SD, median rotation is 23.0° versus full-model 23.6°. Describe overall trends rather than universal per-metric gains.

PLANTPose qualitative ablations without Umeyama or Stable Diffusion — Original Figure 4: without Umeyama, shape and pose become entangled; without Stable Diffusion, unseen texture more often disrupts overall pose. Source: authors' public manuscript. Image copyright belongs to the authors and IEEE; reproduced here for paper analysis.

7. Limitations and reproduction risks

The authors identify:

Separate training for each category;
Future dynamic base-CAD inputs for cross-category transfer;
Occasional pose changes despite depth-conditioned diffusion.

Site observations:

One category, 6 real instances, and 100 images cannot establish transfer to apples, tomatoes, strawberries, or leaves;
Low-dimensional B-splines limit local dents / nonuniform detail;
Real RGB is captured with D435; RGB-only inference still requires calibration and PnP intrinsics;
No equal-RGB comparison to NOCS / shape-prior category methods;
No runtime, detection-failure, severe-occlusion, leaf-background, or robot grasp tests;
Sparse real annotation / scan details, especially ICP frames and symmetry / orientation definitions.

8. Reproduction checklist

Fix base mesh, units, axes, and lattice bounds;
Record 8-corner offset sampling and 4×4×4 clamping;
Save $R,t,\delta$ before / after Umeyama and verify rigid-component removal;
Filter diffusion outputs by image–depth / mask consistency, rejecting changed geometry;
Reproduce 4,000 scenes × 3 views, physics, distractors, lights, and prompts;
Check ViT input size, patch 32, loss weights, and rotation orthonormalization;
Specify crop-to-PnP translation coordinate transforms;
Report per-instance errors and release scans, intrinsics, and annotation protocols.

9. Overall assessment

PLANTPose combines interpretable shape, Umeyama disentanglement, and diffusion texture in a complete data–model pipeline. Benchmark / ablations support this single-category prototype, while 32.6° rotation error, 100 test images, and separate category training leave substantial gaps to general deployable agricultural pose.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.