Why it’s still open
Telling agents not to collude does little: a prompt-only ban had no reliable effect, while an external layer of rules, detection and sanctions cut collusion sharply. In DeepMind’s research swarm, a quarter of agents tried to police cheating but had no power to delete entries or sanction peers. Governing agent groups may be closer to designing institutions for people than to writing better instructions — but which institutions, and whether agents will route around them, is open.
What would count as an answer
Head-to-head comparisons of instruction-based and institution-based designs across tasks, including adversarial agents trying to evade the institution.
Papers that tackle it
- Bracale Syrnikov et al 2026 - Institutional AI — External governance cut severe collusion from 50% to under 6%.
- Paglieri et al 2026 - Cheating and whistleblowing in research swarms — Whistleblowers without power; a failure of institutions.
- Piatti et al 2024 - Cooperate or collapse — Communication and ‘what if everyone did this?’ reasoning save the commons.
- Horibe Itao Toyokawa 2026 - Reputation-based cooperation — How strictly defectors are excluded decides survival of cooperation.
- Chan et al 2025 - Infrastructure for AI agents — IDs, channels and incident response outside the agents.
- Kolt 2025 - Governing AI agents — Agency law applied to AI agents.
- Tomasev et al 2025 - Virtual agent economies — Market design for agent economies.
Methods in play
- Cultural-evolution simulation
- Generative agent-based modeling
- Incident forensics
- Iterated social dilemmas
- Live multi-agent deployments
- Market and pricing simulations
Who’s closest
Placed by the editor, v1 — corrections welcome.
Related questions
Know a paper that belongs here, or think this is framed wrongly? Suggest a paper or a correction (forms on the Atlas home).