Artificial intelligence
Vision-language navigation
Definition
Vision-language navigation is the task of moving an embodied agent through an environment from visual observations and a natural-language instruction. The agent must ground the instruction in what it sees while selecting a sequence of navigation actions.
Also known as: VLN, Vision-and-language navigation
Updated
Ground route language in observations
An instruction such as "leave the room, turn right, and stop beside the table" names actions, landmarks, and spatial relations. A VLN agent must connect those words to its current visual observations, remember progress, and choose where to move next.
Anderson and colleagues introduced the Room-to-Room task and dataset using natural-language instructions grounded in indoor visual environments. Their benchmark uses panoramic viewpoints and graph transitions. Success there demonstrates instruction following within that interface, not safe continuous control on a physical robot.
Connect high-level instructions to locomotion
VLN describes a task, while a vision-language model is one possible perception-and-reasoning component. A vision-language-action model may output actions, but it is not necessarily trained for navigation or long-horizon route completion.
NaVILA applies the problem to legged robots through a two-level system. Its vision-language-action component emits mid-level spatial commands, which a visual locomotion reinforcement-learning policy executes. The paper reports simulation benchmarks and real-robot experiments for that system; it does not establish general performance for arbitrary buildings, robots, or instructions.
Language success does not guarantee motion safety
Instructions can be ambiguous, omit hazards, or refer to objects outside the camera view. Visual aliasing can make separate corridors look alike, and localisation errors can make an otherwise sensible route incorrect.
A physical robot also needs collision checking, traversability estimates, local motion planning, and a controller that respects contacts and actuator limits. A fluent explanation or plausible waypoint is not evidence that the corresponding motion is reachable or safe.
Sources
Related terms
Vision-language model
A vision-language model processes visual information and natural language in a shared system. Depending on its design, it may connect images with text representations or generate text from visual and textual inputs.
Vision-language-action model
A vision-language-action model is an AI model that uses visual observations and language instructions to produce actions for a robot. It connects what a robot sees and what it is asked to do with outputs that a robot controller can execute.
Language-conditioned policy
A language-conditioned policy selects actions using a language instruction together with observations. The instruction specifies or modifies the behavior requested from the policy.
Motion planning
Motion planning finds a robot movement from an initial state to a goal while satisfying constraints such as collision avoidance. A planner may produce a geometric path, a timed trajectory, or a sequence of controls.
Embodied AI
Embodied AI is artificial intelligence that perceives and acts through a body in a physical or simulated environment. It connects sensing, reasoning, and action rather than producing only text or images.