multi-agent LLMsByzantine robustnessno labels

Liars are information

When some agents in a multi-agent LLM system lie, honest agents can still use what the liars say, as long as the lies depend on the truth. An agent that never tells the truth rules out one answer every time it speaks. RACE (Receiver-Anchored Channel Estimation) learns this from each honest agent's own unlabeled history, anchored on the one fact the agent knows: that it is itself honest.

The flagship video: a team of LLM agents solving questions with saboteurs among them

Real answers from six live open models and six saboteurs (the same models told to mislead), from the pre-registered live run. (1) The architecture: where RACE sits inside every honest agent. (2) Round 1 on one question: majority vote picks a lie. (3) Inside one honest agent: what it learned about every other agent without labels, and how the twelve votes add up as signed evidence. (4) Plain versus RACE-informed debate. (5) The whole question stream. (6) The interaction analysis: honest agents alone 53.9%, majority 65.1%, RACE 70.4% after warm-up; informed debate 64.3% vs plain 55.1%, but agents adopt RACE's suggestion 87% of the time.

Pre-registered confirmation (E10)

All 5 of 5 hypotheses registered before the run are supported. On BoolQ questions the live models had never seen, RACE v3.1 scores 81.9% against 69.1% for the frozen v3.0 and 74.0% for the agent alone. On ARC, a benchmark new to the swarm, RACE scores 58.2% against 41.0% for majority vote. RACE-informed debate raises individual accuracy by +6.5 points over plain debate (95% CI [+3.9, +9.1]).

E10 pre-registered live run — E10: v3.1 vs v3.0 on unseen BoolQ items, RACE on ARC, and each model's accuracy after plain and after RACE-informed debate.

The film: how the swarm works

Ten agents on MMLU, five of them real LLMs prompted to mislead after seeing the honest answers. (1) Every agent answers, broadcasts, and each honest agent decides with its own learned trust links. (2) The swarm's trust matrix learning over 90 unlabeled questions. (3) One question in slow motion: counting votes vs. weighing evidence. (4) Results.

Label switching, animated: the same EM with and without the anchor. With 7 of 10 agents in a liar bloc, Dawid–Skene trusts the bloc (0% on test); RACE reads it backwards (91%; 66% alone).

One honest receiver among seven liars (MMLU, the gate-aware attack that defeats AIP). Blue: TRUST. Red: INVERT.

BoolQ with 50% coordinated liars: a bloc that always says the opposite, read backwards.

MedQA with 50% real LLM deceivers that see the honest answers first.

Coherence is tunable, truth-dependence is not: sweeping the attacker's coordination.

Live six-model swarm on MMLU: honest agents, covert saboteurs and a debate round. RACE lifts every honest agent.

Live swarm on BoolQ: option-biased small models break RACE's symmetric channel model; the class-conditional variant recovers.

E10 on ARC, question by question: plain debate against RACE-informed debate on the same half-liar panel.

E10 on unseen BoolQ items: the same comparison on binary questions.

Headline results

Honest-receiver accuracy (%)	Receiver alone	Majority vote	AIP (prior work)	RACE
Coherent liars, f = 0.9	85.8	0.0	0.0	88.0
Gate-aware liars, f = 0.7	86.4	3.9	48.5	94.9
Real LLM deceivers, f = 0.5	85.7	75.9	83.0	92.7
Real LLM deceivers, f = 0.7	85.8	29.4	70.8	89.8
Independent liars, f = 0.7 (vs. an oracle that removes every liar: 89.7)	84.2	7.0	29.1	94.8
E11 attack mix, f = 0.7 (best classical crowdsourcing estimator: 17.6)	84.2	22.8	60.9	89.5

Mean over six benchmarks. The LLM-deceiver rows are a confirmation set evaluated after the method was frozen (RACE v3.1: 197 significant wins, 0 losses in 480 paired tests). Failure modes (camouflage attackers, sleepers, near-chance receivers) are reported as prominently in the paper and README.

Every study at a glance

Figures

Classical crowdsourcing estimators against RACE — E11: IWMV, MACE, GLAD, KOS and Dawid–Skene collapse once liars are the majority; the anchor does not.

Accuracy versus Byzantine fraction on six benchmarks — No breakdown point against a coherent bloc.

Attack zoo heatmap — Attack zoo: where pooling helps and where it hurts.

Channel estimation — Label-free channel estimates vs truth: r = 0.977.

Information budget — What the liars are worth: RACE beats an oracle that removes every liar when lies depend on the truth.

Swarm composition and size — Replicated swarms and swarm size.

Live six-model swarm: what each agent did, and what pooling made of it.

Online attackers — Sleepers and regime switches: with forgetting, RACE recovers within about 40 questions.

Concept — Coherence is tunable; truth-dependence is not.

Code, data and paper

GitHub: Hisernberg/liars-are-information. Builds on the original artifact by Dhruv Jyoti Das (Dhruv1000/Liars_Are_Information).