Artificial intelligence
Model-based reinforcement learning
Definition
Model-based reinforcement learning uses a model of environment dynamics to improve decisions or policy learning. The model may be given or learned and can predict the consequences of candidate actions before they are executed.
Also known as: Model-based RL, MBRL
Updated
Learn with predicted consequences
A model-based agent has a transition or dynamics model that predicts how state or observations change after an action. It can search over action sequences with that model, generate synthetic experience for a policy, or combine both approaches. In robotics, this can reduce the physical interaction needed to compare candidate actions.
The model may encode known mechanics, be fitted from data, or combine learned and analytical parts. System identification focuses on estimating a system model; model-based reinforcement learning adds a reward objective and uses the model to improve action selection or learning.
Account for uncertainty in learned dynamics
Chua and colleagues present PETS, which uses an ensemble of probabilistic neural-network dynamics models and sampling-based uncertainty propagation. It evaluates candidate action sequences through model predictive control, executes an action, gathers another transition, and updates the learned model.
Their reported sample-efficiency comparisons use benchmark control tasks. They support the method under those experimental conditions, not a general claim that model-based methods always need fewer robot trials or outperform model-free learning.
A planner can exploit model error
Small prediction errors compound over long rollouts. An optimiser may deliberately select actions in regions where the learned model is overconfident or wrong, producing a plan that looks successful only inside the model. Ensembles can represent some uncertainty but do not guarantee awareness of every unfamiliar state.
Model-based reinforcement learning is related to a learned world model, but the terms are not interchangeable. A world model is a predictive representation; MBRL is the learning-and-control approach that uses a model to improve expected reward.
Sources
Related terms
Reinforcement learning
Reinforcement learning trains an agent to choose actions that maximize expected cumulative reward through experience with an environment. In robotics, the learned policy can select movements or higher-level behaviors from observations.
World model
A world model is an internal predictive model of an environment and how it changes. In robot learning, it can predict future states or observations under possible actions to support planning or policy training.
Model predictive control
Model predictive control repeatedly optimizes future actions using a system model, applies the next part of the solution, and replans from updated state information. It can account for objectives and constraints over a finite prediction horizon.
System identification
System identification estimates a model of a physical system from measured inputs and outputs. In robotics, it can recover parameters such as inertia and friction or learn a more general model of how actions change the system state.