Artificial intelligence
Real-to-sim-to-real
Definition
Real-to-sim-to-real is a robotics workflow that uses measurements from a physical system to construct or calibrate a simulation, develops or evaluates robot behaviour in that simulation, and transfers the result back to the physical system.
Also known as: Real2Sim2Real, Real-to-simulation-to-real
Updated
Close the loop through a simulator
The first transfer direction turns real observations into simulation assets or parameters. A system may reconstruct scene geometry and appearance, estimate friction or stiffness, or fit dynamics from physical trajectories. Simulation can then generate trials, optimise a controller, or evaluate candidate policies. The final transfer deploys the selected behaviour back on the robot.
In Planar Robot Casting with Real2Sim2Real, physical cable trajectories tune simulator parameters, simulated and physical examples train a policy, and the policy returns to the real casting task. RoboGSim instead combines a physics engine with 3D Gaussian splatting to reconstruct and compose manipulation scenes for data generation and policy evaluation. These are two research implementations of the loop, not one standard pipeline.
More than sim-to-real alone
Sim-to-real transfer begins with a simulation and asks whether a policy or model will work in the physical world. Real-to-sim-to-real explicitly adds an earlier step in which physical evidence shapes the simulator. That step may use system identification, visual reconstruction, manual measurements, or learned estimation.
The phrase also differs from simply replaying recorded data. The intermediate model must support some new computation, such as changing an object, sampling another trajectory, or testing a policy under controlled conditions.
A reconstructed scene is still an approximation
Accurate appearance does not guarantee accurate contact, deformation, sensing, or actuator timing. Conversely, a useful dynamics fit can omit visual details that matter to a camera policy. The workflow needs separate checks for the quantities used by the downstream task.
Errors can compound around the loop. Sparse real measurements may identify several plausible simulations, and a policy may exploit whichever approximation was selected. Returning to the same physical setup can hide poor generalisation to new objects or wear. Real-to-sim-to-real reduces some manual modelling and data-collection costs, but it does not remove the need for held-out physical trials and safety controls.
Sources
Related terms
Sim-to-real transfer
Sim-to-real transfer applies a model, policy, or behavior developed in simulation to a physical system. Its central challenge is the difference between the simulated environment and the robot, sensors, and interactions encountered in reality.
System identification
System identification estimates a model of a physical system from measured inputs and outputs. In robotics, it can recover parameters such as inertia and friction or learn a more general model of how actions change the system state.
Differentiable simulation
Differentiable simulation is physical simulation that provides derivatives of simulated outcomes or losses with respect to inputs such as controls, initial states, model parameters, or robot design variables. Those gradients can drive optimisation and learning through the simulated dynamics.
3D Gaussian splatting
3D Gaussian splatting is an explicit scene-representation and rendering method that models appearance with optimised three-dimensional Gaussian primitives. The primitives are projected and blended into an image, enabling novel-view rendering and, in some robotics systems, dense visual mapping.