How do you evaluate a multi-agent system?
Evaluate each agent against its own role contract, then evaluate the system end to end as a separate suite. Most multi-agent failures are handoff failures rather than reasoning failures, so a system that scores well agent-by-agent can still fail in composition. You need both views to locate a regression.
Give every agent an explicit role boundary — what it owns, what it may call, what it must escalate — and write per-agent evals against that contract. When the end-to-end score drops, the per-agent scores tell you which boundary leaked.
Instrument the handoffs themselves. Record every message crossing between agents so a failed run can be replayed step by step; without that trace, debugging a multi-agent system is guesswork.
On Aletheia, autonomous agents negotiate live-event decisions across attendee, organiser, and venue contexts under a sub-second response budget. The explicit role boundaries are what make that system testable at all.
Written by Binary AI Labs · Reviewed