State hallucination

Act on a transition that did not happen#

Suppose a gripper misses an object but the policy proceeds with the motion intended to place that object in a container. The later action is consistent with an imagined successful grasp rather than the observed physical state. Lee and colleagues call this recurring VLA failure pattern state hallucination.

The term is recent and is not yet a standardized robotics failure taxonomy. It should be tied to an operational test, such as whether a required contact or object transition occurred before the next action, rather than applied to any failed manipulation.

Different from an incorrect state estimate#

A state estimator can output the wrong pose because of noise or occlusion. State hallucination, as used in the cited work, describes the policy behavior of continuing as if an unrealized state had been achieved. A bad estimate can cause that behavior, but the policy may also ignore visible contrary evidence or continue an open-loop action sequence.

It also differs from a world model producing an inaccurate future prediction. The observable failure is that the action policy does not adjust to the unrealized transition, regardless of whether it has an explicit predictive model.

Diagnosis needs closed-loop evidence#

The cited authors report weaker attention to task-relevant image regions during these failures and identify associated sparse features in tested VLA models. They also report reduced hallucinated failures after selectively unlearning those features. These are findings for the studied architectures, simulations, real-world tasks, and intervention.

A readable action sequence alone cannot establish what internal assumption caused a failure. Contact may be hidden, the success condition may be ambiguous, or latency may make the policy act on an older frame. Evaluation should synchronize observations and actions, define the missing transition, and show whether fresh evidence reaches the policy before labeling the behavior state hallucination.