We’ve all been there. Just before a launch, or the latest release, an email circulates around the organisation: “Here’s the new AI agent. Try and break this, please.” Or a room gets booked, everyone gets a script, and the day is spent checking whether it meets the standard we expect.
I’ve sat in a number of those rooms across various markets, and involving colleagues has a real place before you go live.
The trouble starts when it becomes the backbone of your testing strategy. It feels thorough, but doing a lot isn’t the same as achieving a lot. Volume can’t rescue weak testing.
That’s why I call it vanity testing, and I mean it in both senses of the word. It looks good, but it’s empty.
What it actually is
In most organisations I’ve seen, this began as user acceptance testing (UAT), the final check that an agent meets the needs of the business and its users before it’s signed off. It has since swollen into part UAT, part bug bash, part attempt to break the agent, and quietly replaced the testing that should come first.
I often hear it called red teaming, which is misleading. Red teaming is a deliberate, structured hunt for failures by people who know how to find them (ideally independent of the team that built the agent). A room of colleagues isn’t that.
Why it falls short
First, the testing drifts. Colleagues are at their best when they test what they know. Customer service managers understand better than almost anyone how customers behave and talk, but most aren’t trained in jailbreaks or prompt injection, nor should they be. Hand them a task to hack the agent, and you’re asking them to test beyond their expertise. And some attacks never touch the chat window, like indirect prompt injection, where instructions are hidden in a document the agent reads. The testing grows in volume without growing in depth.
Second, it becomes a performance. The headcount, the conversations and the hours logged become the headline, rather than the substance behind them.
Third, it can blur accountability. When the whole organisation has taken part, it’s harder to say who owned whatever slipped through. If everybody’s accountable, nobody is.
Finally, the hidden cost. Senior people’s hours rarely appear in the business case, but they eat into the return you were hoping for.
Testing with purpose
Start by agreeing what good looks like before any testing begins, across its full breadth rather than through one narrow lens, whether that’s functionality, customer experience or security. Then work out which methods cover which part.
Let UAT do its job, and let simulation carry the scale. An agent that rarely gives the same answer twice has an almost endless number of possible conversations, so a fixed script won’t give you confidence. What will is simulating conversations at a volume no room could match, modelled on how your customers actually write, using science to aim each test where failure is most likely and matters most. Automated checks, specialist red teaming and independent testing all play their part too.
When you bring people in, play to their expertise. Security specialists probe the safeguards. Customer service managers bring the questions customers really ask. And start from what customers actually said to your previous agent, or to your human agents.
So keep the room, but give it a purpose.
Vanity testing is scale for show. The answer is scale with intent.