Solid benchmark paper. AgentWorld runs 3–20 LLM agents with asymmetric roles in a blackbox MMORPG sandbox (built on Kaetram, 13 high-level API tools abstracting combat/crafting/navigation), across 100 human-annotated tasks plus 100 augmented variants, 25–55 rounds each. The interesting part is the metric: Causal Collaboration Effectiveness builds a causal action graph by backward-tracing from success actions and measures what fraction of total actions actually contributed. Judge agreement with a human annotator is κ = 0.64, and rankings hold across three judge models, which addresses the usual LLM-as-judge reproducibility complaint.
The headline result: best model (Gemini 3 Flash) hits 52.0% SR with CCE of 0.320 — under a third of all agent actions are on any causal path to completion, and DeepSeek R1-70B wastes ~88%. Notably, GPT-5 Mini sends the most chat messages (44.1/task) and ranks third; a human-written shared plan actually *hurts* (22.9% → 17.6%) versus no communication at all, suggesting plans without the ability to revise them mid-execution are worse than improvisation. 27% of tasks went unsolved by every model.
Practitioners building multi-agent pipelines should read this as: communication volume is not coordination, and fixed plans without feedback channels are actively harmful. Also, a single-agent baseline hits 28.6%, so if your "multi-agent system" is one competent agent plus spectators, you're at industry standard.
reply