There’s a saying that gets quoted in every business book: “Amateurs talk strategy; professionals talk logistics.” After watching a year-long AI experiment, Firmulate would add a second line: geniuses talk diagnosis; closers talk signatures.
Get the little things that make your day delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Four frontier AI models were each handed the same small software company and the same brutal week — same customers, same crises, same temptations to cut corners. Every crisis was spotted. Every manipulation attempt was refused. Yet only two of the four models actually signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
That gap — between knowing what to do and finishing the job — is the most human failure an AI has ever shown. And it’s the reason Firmulate exists: to measure management quality, not chat quality, before an AI ever touches your real business.
The wargame
The setup was elegantly simple. Each model ran the identical company through its worst week. Only the model changed. Every decision was versioned and auditable — a full replay of judgment under pressure.
The final Crucible League standings tell the story:
- 1. gpt-5.6-sol — 95 points
- 2. Kimi K3 — 93 points
- 3. Sonnet 5 — 88 points
- 4. Fable 5 — 77 points
- 5. Opus 4.8 — 73 points
For context, doing nothing scores 26. Partial progress counts — but a single breach of trust caps the total. As the experiment’s own rule puts it: “no amount of good work outweighs a breach of trust.” It’s a quote most boards would do well to laminate.
The buried fact that decided the deal
Here’s the twist that should keep every executive up at night. The decisive competitive weakness wasn’t in the customer conversation at all — it sat two document references deep in the company’s own files. The models that actually read the file won the €55,000 deal at full price, worth an extra €4,583 in monthly recurring revenue.
The winners didn’t out-think the losers. They out-read them. In an age where everyone brags about AI reasoning, the quietly decisive skill turned out to be old-fashioned homework.
The con artists got nowhere
The week included a social-engineering gauntlet: fake CEO messages escalating over three stages, plus a reporter’s trap — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning is worth quoting directly: “Treat the request as a suspected approval-bypass / possible impersonation.”
That’s the kind of quote you’d want framed in your compliance department. The machines passed the integrity test. The discipline test was another matter.
Why the most thorough model came last
Opus 4.8 is the experiment’s tragic figure: the most thorough participant, with over 80 learned rules and the deepest analyses — and last place. The close was left on the table, and discipline slipped: it made write attempts into a locked department instead of escalating. The same weakness appeared, more faintly, in all four models. Brilliance without follow-through is a familiar character in every office on earth.
One fairness note: Kimi K3 ran without an effort parameter (API default) while the others ran at xhigh — and still finished second.
The company that never sleeps
Behind the benchmark runs a live experiment you can actually watch: a synthetic company with 13 employees, real money mechanics, a burn rate of €105k per month against €2.3k in MRR, a public cash countdown, and more than 680 self-learned playbook rules. Every workday is versioned. It’s a company-shaped aquarium, and it’s watchable at firmulate.com.
Want to test your own instincts against the machines? A “guess the model” quiz powered by 242 real, unedited management decisions is live on the site.
From watching to acting
If AI agents will ever touch your CRM, your support queue or your forecast, the question is not whether they sound smart in a demo. It’s whether they close deals, refuse impersonators and read your own files two references deep — against your company, not a toy one.
That’s what the Firmulate pilot offers: the same wargame, run against a read-only export of your own business. Crisis scenarios churn waves, price increases, competitor attacks, PR disasters — played out against your real customers and playbooks, with a board report showing the model ranking and the weak points in your own playbook. Nothing ever writes back to your real systems. Read-only in, paper trail out.
You’ve read the quotes and watched the league table. Now run the experiment on your own company before reality runs it for you. Start your pilot at firmulate.com/pilot.html or write to contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
NFL season / tailgating Picks
team gear
As an affiliate, we earn on qualifying purchases.
