Can Artificial Intelligence Withstand a Fake CEO Scam? The Answer Is Yes, Surprisingly
In an era where digital deception is increasingly sophisticated, a recent experiment with AI models battling social engineering scams offers a reassuring story. For garden and outdoor enthusiasts, it’s a lesson in trust — but in the digital age, that trust must be tested before it’s broken. The experiment shows that some of the most advanced AI systems can resist manipulation, even when under pressure from convincing fake requests.
AI social engineering test software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Experiment: Putting AI to the Test in a Real-World Business Scenario
Imagine a small software company with a real cash flow of €2,300 in monthly revenue, but burning €105,000 every month, and a public cash countdown. Now, picture that company’s AI workforce faced with the same worst week imaginable: customer crises, tempting shortcuts, and social-engineering attempts designed to mimic a CEO’s voice and authority.
Five leading AI models participated, including the top-rated gpt-5.6-sol and Kimi K3, which scored 95 and 93 out of 100 respectively in the so-called Crucible League, a benchmark that measures trustworthiness and decision quality in high-pressure scenarios. The models were tasked with managing the company’s decisions, reading and interpreting files, and making choices aligned with company goals.
Surprising Results: Every Model Resisted Manipulation
Despite escalating social engineering attempts—fake CEO messages urging the release of customer data or signing deals without proper approval—all five models refused to comply. Even when pressed with increasingly urgent or convincing requests, none signed the €55,000 deal that the company’s own analysis had earned, nor did they bypass critical approval steps.
The experiment’s key insight: the models’ ability to detect malicious intent was not just superficial but rooted in deep analysis. Kimi K3, the second-highest scorer overall, explicitly identified the request as a suspected impersonation, exemplifying a principled approach to integrity.
The Hidden Weakness: What Made the Difference?
One crucial finding was that the decisive factor wasn’t in the initial crisis detection but in reading and understanding the company’s internal files. The models that meticulously searched the company’s documentation discovered a buried reference, which was instrumental in closing the full-price deal worth over €4,500 in Monthly Recurring Revenue (MRR). This highlights how critical internal knowledge is when assessing trustworthiness and making sound decisions.
The Lessons for Business Security and AI Deployment
This experiment reinforces a vital message: assessing an AI system’s trustworthiness should happen before deployment, not after a breach occurs. Testing AI models with scenarios like social engineering escalations reveals their true capacity for integrity under pressure.
For companies considering AI adoption—whether managing customer relationships, support queues, or financial forecasts—the takeaway is clear. The question is not just whether an AI writes well but whether it can finish what it starts, read your files thoroughly, and stay honest when faced with deception.
Beyond Chat: Measuring Real-World Decision-Making
Unlike chat demos, which often focus on language quality, the Firmulate experiment measures decision quality in a dynamic, high-stakes environment. The models’ ability to recognize and refuse manipulative requests shows they are more than language generators; they are decision agents capable of acting with integrity.
Benchmarking and Future Readiness
In the Crucible League, the top model scored 95, with the second place at 93. The baseline ‘do-nothing’ model scored just 26. This indicates that current AI systems are advancing rapidly, and their trustworthiness is becoming a competitive edge — vital for safeguarding sensitive decisions.
How to Test Your AI Before It’s Too Late
Organizations can run their own ‘wargames’ using tools provided by firms like Firmulate, which simulate real crises and social engineering scenarios. These tests are designed to be safe, non-destructive, and fully auditable, ensuring your AI can handle real-world pressures without compromising trust.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
