Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Agents win when they can deliberate slowly; this scales fast-action settings (robotics, games) where they must act. Builds the missing real-time eval infra.
The explainer · 3-minute read · 287 words
The claim
Vision-language agents lose most of their skill when the world keeps moving while they think.
The problem
Agents already score well on math, software engineering, and desktop work.
These domains allow a long pause between one action and the next.
Games and physical devices allow no pause; state moves while the model reasons.
How it works
This paper builds visual minigames that demand real-time skills.
An agentic pipeline generates each game from a taxonomy of six capabilities.
The taxonomy covers perception and action while the environment state evolves.
Every game can run live, paused, or slowed down for comparison.
Changing only the clock separates real-time cost from task difficulty.
Figures
What changes
Most agent benchmarks freeze the world between the agent's turns.
This one keeps the clock running, like a game or a robot.
The authors release 100 minigames and the generator that built them.
Limits
Minigames stand in for full games and physical robots.
How the trained skills transfer to real devices stays untested.
The paper details one frontier model; more agents need the same test.
Source · arXiv wire