SHEC3R | Stationary Eye-to-Hand Calibration

Overview

Stationary eye-to-hand calibration estimates the rigid transform between a fixed camera and the robot base. Silhouette-based differentiable rendering can be highly accurate, but its non-convex objective makes the starting estimate critical.

SHEC3R learns that starting estimate. Robot-grounded prompts condition a pretrained visual-geometry backbone on the base location, projected base-axis directions, and the visible arm configuration. Cross-frame attention combines evidence from multiple observations into a shared calibration.

The feed-forward stage requires RGB frames and robot masks, without a calibration board, robot meshes, or joint readings. One depth frame recovers metric scale. The estimate can also initialize an optional differentiable-rendering stage, which additionally requires robot meshes and joint readings.

Video

2 min 53 s · Download MP4 · Read the method overview

Method

Three complementary prompts connect a visual-geometry prior to the robot base frame.

SHEC3R pipeline: RGB and mask inputs feed robot-grounded prompts and a pretrained backbone, producing a base-to-camera pose and robot-frame point cloud, followed by optional differentiable-rendering refinement. — RGB observations + robot masks → robot-grounded calibration prompts → shared base-to-camera pose and robot-frame point cloud. Click the figure to enlarge.

Base location

A base-location prompt captures where the robot base appears in the image.

Axis orientation

Projected base-axis directions provide cues for the orientation of the robot frame.

Arm configuration

Mask-guided arm prompts gather evidence from the visible robot across observations.

Initialization and refinement have different requirements.

The neural initializer uses RGB + masks; metric scale uses one depth frame. Optional DR refinement additionally uses URDF meshes and joint readings.

Results

Evaluation on the same scenes with a shared differentiable-rendering stage, varying only the initializer.

Simulation · 400 scenes

EasyHeC25.2%

EasyHeC++8.0%

SHEC3R80.2%

Percentage of scenes below 5° final rotation error, after the same DR refinement.

Final rotation success rate and median rotation error after the shared refinement stage.
Evaluation split	Initializer	S@5° ↑	Final rotation ↓
Simulation 400 scenes	EasyHeC	25.2%	80.5°
	EasyHeC++	8.0%	88.6°
	SHEC3R	80.2%	0.05°
Real 86 scenes	EasyHeC++	15.1%	80.7°
Real 86 scenes	SHEC3R	86.0%	0.74°

S@5° is the percentage of scenes with final rotation error below 5°. These are refined results, not feed-forward-only results. Figures reproduced from Table I of the manuscript.

Rotation and translation error over training epochs, comparing synthetic pretraining plus real fine-tuning against real-only training, before and after DR refinement. — Synthetic pretraining and real-world fine-tuning improve the learned initialization. Dashed curves show the errors after DR refinement.

The manuscript also evaluates an external platform: the Flexiv Rizon 4 subset of RH20T. On the combined real evaluation set of 647 scenes, SHEC3R reaches 90.3% S@5° after refinement. See the paper for split definitions and limitations.

Robot-frame reconstruction

Real-world RGB observations and reconstructed point clouds. Left to right: Franka, AIRBOT Play, OpenManipulatorX, SO-101, UR5e, Rizon 4. — Real-world RGB observations (top) and reconstructed point clouds in the robot base frame (bottom). Click the figure to enlarge.

RoboEyeCalib Dataset

Synthetic and real-world data across camera layouts, scenes, and robot morphologies.

2.83MSynthetic frames

4Simulated arms

883Real scenes

5Physical arms

RoboEyeCalib examples showing synthetic and real robot observations, depth, and 3D geometry across environments. — RoboEyeCalib spans simulation and physical capture. Camera–base layouts are held out for evaluation.

Simulation. Franka Panda, xArm6, SO-100, and WidowX-AI, across eight scene types.

Physical capture. Franka, AIRBOT Play, SO-101, OpenManipulatorX, and UR5e, across nine environments.

Citation draft

@misc{shec3r2026,
  title = {SHEC3R: A Robot-Grounded Neural Initializer
           for Stationary Eye-to-Hand Calibration},
  author = {Zhang, Hangxing and
            Dou, Zhiyang and
            Ma, Yuxiang and
            Ye, Pengfei and
            Li, Peihao and
            Guo, Yaowei and
            Yan, Mingxuan and
            Li, Jinzhou and
            Tao, Yiran and
            Liu, Yuan and
            Gan, Chuang and
            Jiang, Ying and
            Li, Jiachen and
            Ivanov, Yuri and
            Matusik, Wojciech},
  year = {2026},
  note = {Manuscript under review}
}

Update with the public preprint metadata before citing.