RobotAgentBench: How Far Are Robots from Open-World Autonomy?

01 — Multi-level evaluationWhat does a robot agent need to act autonomously in the real world?

“Prepare the home for guests” sounds simple—but where should a robot begin? It must identify what needs to be cleaned, select appropriate food based on user preferences, coordinate multiple activities, and recover when manipulation attempts fail. Such challenges are intrinsic to everyday physical environments.

RobotAgentBench evaluates these demands across five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness. Beyond assessing individual capabilities through dedicated tasks, we introduce four open-ended scenarios that require agents to integrate these capabilities and coordinate multiple tasks to autonomously fulfill high-level user needs.

Level 1Task-level evaluation

We design five complementary evaluation settings to examine how robots determine what to do, execute physical actions, and adapt to unexpected outcomes. Each setting comprises a suite of dedicated tasks.

Level 2Scenario-level evaluation

Unlike task-level evaluation, robots do not receive explicit task lists. Instead, they must autonomously explore the environment, identify relevant tasks, interpret user needs, and plan their execution.

• Host guestsAlex is coming over shortly. Please get some refreshments ready and make the place ready for his visit.
• Prepare dinnerRyan is joining us for dinner. He’s been working out a lot lately. Please get everything ready for him.
• Prepare to leaveEmma is going out to work away from home today, and it’s raining. Could you help her get ready?
• Tidy the roomI’m about to start a video call from the sofa. Could you tidy up the room first?

02 — Real-to-simAn agent-assisted, human-in-the-loop real-to-sim pipeline.

To enable realistic and scalable evaluation, we develop an agent-assisted, human-in-the-loop real-to-sim pipeline that starts from a physical room in a real home, reconstructs it in simulation, expands the reconstructed scene into diverse interactive configurations, and collects demonstrations for training and evaluation.

03 — LeaderboardHow far are we from open-world autonomy?

We conduct systematic experiments at both the task and scenario levels to evaluate standalone VLA/WAM-based policies and complete robot agents. Building on the unified robot agent framework we developed, we compare different harness designs and examine their effects on task performance.

• PlannerPInterprets environmental context and user requirements, decomposes tasks, and adapts plans based on execution feedback.
• ExecutorETranslates instructions into physical actions through learned policy models (e.g., VLA and WAM) or code-based control.
• VerifierVMonitors execution outcomes, identifies failures, and guides corrective actions.

Each agent is denoted by its Planner / Executor / Verifier configuration: C code-based control, L learned VLA/WAM policy, H hybrid code-based and learned execution, – absent component.

• VLA/WAM Policy + Code Verifier– / L / CA VLA/WAM policy generates actions, while a coding-agent verifier monitors execution, selects action prefixes, and applies local corrections before querying the policy again.
• Code Planner + VLA/WAM PolicyC / L / –A coding-agent planner decomposes tasks into subtask instructions for a VLA/WAM policy and revises them based on execution feedback.
• Fully Code-basedC / C / CA coding agent plans, executes, and verifies tasks by composing predefined control functions (e.g., move_eef_to()) and checking execution outcomes.
• Hybrid Code/VLA/WAM PolicyC / H / CA coding agent plans and verifies execution, dynamically combining code-based control and VLA/WAM policies as callable tools based on task requirements and feedback.
• Code-based + RSIC / C / C + RSIThe fully code-based agent iteratively improves its control tools and accumulates experience from execution feedback, retaining these improvements across attempts.

04 — AnalysisWhat we found.

01Policies stall; agents get stuck.

Standalone VLA/WAM policies rarely explore in response to context. They spend most of their time idle or making unproductive movements, with stagnation above 84%. Robot agents identify tasks more actively, but frequent execution errors leave them retrying a single subtask until the time limit.

02Who plans, and who executes, both matter.

Pairing four policy executors with three coding-agent planners, Antigravity CLI yields the largest gains as a planner, while InternVLA-A1.5 is the best executor. Its stronger robustness and failure recovery, together with more reliable following of revised instructions, make it well suited to this role.

03Concise instructions are easier to follow.

Antigravity CLI writes shorter instructions, switches them less often, and uses fewer execution-specific details and simpler vocabulary. Codex often specifies detailed procedures, even which hand to use, and learned executors struggle to interpret them.

04Recursive self-improvement keeps paying off.

Over ten consecutive episodes, all three code-based agents improve at both levels as they refine reusable control functions. For example, they learn to decompose an infeasible single-shot reach into smaller steps through feasible intermediate poses.

05From simulation to the real home.

On matched subsets, real-world scores are generally lower because physical interaction is harder and scenes are more cluttered. However, the relative ranking of models and agents is preserved.

CitationBibTeX

@misc{mao2026robotagentbenchopenworldrobotautonomy,
      title={RobotAgentBench: Towards Open-world Robot Autonomy},
      author={Lingjun Mao and Jinglin Cao and Lukun He and Wenpeng Xu and Yuchen Yan and Hengyu Jin and Zijun Zhang and Jinzhou Tang and Songyao Jin and Ziche Liu and Yifei Shao and Junbo Huang and Fang Nan and Ruobing Han and Ziqiao Xi and Ziming Xu and Shuang Liang and Sibo Zhu and Wenyi Wu and Xinyue Wang and Yiming Huang and Xinkai Zou and Gao Zhu and Kun Zhou and Biwei Huang},
      year={2026},
      eprint={2610.34649},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.34649},
}