WorldMirrorAgentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

Tianling Xu1,*, Weijie Wan1,*, Honghao Su1,*, Kailun Su1, Junyuan Tang1, Huichi Zhou4, Haozheng Zhang4, Qi Xiong1,2, Ping Luo3, Wenbo Ding1,2,†, Tianxing Chen1,3,†, Yue Chen1,5,†1Xspark AI 2Tsinghua University 3The University of Hong Kong 4University College London 5Peking University*Equal contribution. †Corresponding authors.

PaperReplay
RGB videoWorld programInteractive twinTrajectory replay
One shared camera-anchored coordinate system
Overview

Interactive worlds from in-the-wild demonstrations.

GPT-6 Astra and WorldMirror on spatial structure, visual fidelity, and physical consistency. Astra keeps the layout and fails appearance and contact. WorldMirror holds all three.

WorldMirror turns in-the-wild demonstrations into simulation-ready twins by iteratively refining an executable world program. GPT-6 Astra is competitive in spatial structure and layout, while WorldMirror achieves better visual fidelity, collision geometry, and physical consistency. A camera-pose-anchored scene lets the recorded trajectory replay unchanged.

01 · Abstract

Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry.

We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback.

A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.

02 · Method

An agent that builds, checks, and builds again.

Instead of a fixed pipeline, a router agent maintains a structured world state, delegates to specialized tools, and uses verification feedback to decide what to revisit next.

WorldMirror framework diagram showing the agentic real-to-sim reconstruction loop
Read the demonstrationGround the world state in the observed scene, instruction, and robot trajectory.
Observe

What real-to-sim systems can do.

Table 1 marks which real-to-sim systems reconstruct the scene, replay a demonstration, and close the loop.

Table 1
Capability RoLA RoboSnap SimFoundry SAGE Astra Agentic Real2Sim WorldMirror
Scene Reconstruction
Scene Appearance ✓ ✓ ✓ ✓ ✗ ✓ ✓
Interactive Assets ✓ ✓ ✓ ✓ ✓ ✓ ✓
Robot–World Alignment
Camera Calibration ✗ ✓ ✓ ✗ ✓ ✓ ✓
Camera-Pose-Anchored World Representation ✗ ✗ ✗ ✗ ✗ ✗ ✓
Demonstration Replay ✗ ✓ ✗ ✗ ✓ ✓ ✓
Agentic Construction
Tool Orchestration ✗ ✗ ✗ ✓ ✓ ✓ ✓
Agent-based Verification ✗ ✗ ✗ ✓ ✓ ✓ ✓
Feedback-guided Refinement ✗ ✓ ✓ ✓ ✓ ✓ ✓
03 · Demos

The same trajectory, in the mirror.

Eleven DROID demonstrations. Each one pairs the real exterior and wrist views with the WorldMirror replay of that trajectory.

Real demonstrationRecorded DROID views

WorldMirror replaySame trajectory, no retargeting

Scene

The reconstructed world, from every side.

Seven DROID scenes, reconstructed the same way. The photo stays in the corner so the real first frame can be compared with the orbit. Drag to turn the full circle, and scroll to move closer.

Real first frame of the selected scene
RealFirst frame

Loading the scene…

Comparison

The same scene, four reconstructions.

First frames side by side, then the same recorded trajectory replayed in each reconstruction.

First frameAppearance and layout at t = 0

Real

RoboSnap

RoLA

GPT-6 Astra

WorldMirror

RoboSnap does not support robot-arm integration; we keep the same arm setting as WorldMirror.

Replay episodeSame recorded trajectory

Real

RoLA

GPT-6 Astra

WorldMirror

Citation

Build on the mirror.

Use the title and BibTeX below when you want to reference the WorldMirror project.

@misc{xu2026worldmirror, title={WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations}, author={Tianling Xu and Weijie Wan and Honghao Su and Kailun Su and Junyuan Tang and Huichi Zhou and Haozheng Zhang and Qi Xiong and Ping Luo and Wenbo Ding and Tianxing Chen and Yue Chen}, year={2026}, eprint={2610.31834}, archivePrefix={arXiv}, doi={10.48550/arXiv.2610.31834}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2610.31834} }