All posts

The bar to go live

Why we built Mantia Essentials, and what every customer-facing AI agent should prove before launch.

Doodle of a chat-bubble chatbot clearing a high-jump bar, watched by a colleague with a clipboard and a crowd cheering from the stands.

Over the last couple of months, we’ve run in-depth assessments of clients’ AI agents before they go live. We learned early on that “ready” means something different to everyone you ask, and it’s never just one thing.

Beyond its job, an agent has a whole host of things to prove, from security to how it sounds and, where relevant, the nuances of how it sells. Which ones matter depends on who’s asking, and on their sector, country and regulator. That breadth usually slows organisations down, and helping them move faster with less risk is why we built Mantia.

Our managed assessments are shaped around what good looks like for each organisation. Along the way, other businesses asked for a simpler place to start, and to know exactly what their agent had to prove before going live. So we built Mantia Essentials.

What Mantia Essentials is

Mantia Essentials is the pre-deployment assessment for customer-facing AI agents, covering five areas: security, safety, conduct, truthfulness and privacy. When we designed it, we asked ourselves, “What’s the bar to go live?”

My background is in product management, where you want your first version in front of customers fast, to learn how they really use it. It doesn’t need to be the best customer experience ever, but it does need the basics covered. Essentials doesn’t test the happy path, the job your agent was built to do, which you’ll have been through yourself. It covers the risks around that job, with the tests already decided.

What it tests

We test the conversations nobody wants their agent to have. Someone tries to jailbreak it, talking it out of its rules or into leaking what it shouldn’t. A customer turns abusive, insists on a policy that doesn’t exist, or pushes for someone else’s details. We check that it holds its ground, the way your best people would.

Many of these failures have already happened. DPD’s chatbot was talked into swearing at a customer and writing a poem slating the company. A Chevrolet dealer’s chatbot agreed to sell a new car for a dollar, and when Air Canada’s chatbot invented a refund policy, a tribunal held the airline liable. Each time, one conversation became a problem for the whole brand.

Who it’s for

Essentials is for any business with a customer-facing chatbot or AI agent, whether you’re about to launch or want a quick check on a live one. For now, it tests agents that chat in English, in text, through an API, whatever the channel. When you want testing built around your own business, sector or regulator, that’s what a managed assessment is for.

How it works

Setup takes about 10 minutes on the Mantia Essentials page:

  1. Tell us what your agent should and shouldn’t do, and share the documents that hold the right answers. They’re our source of truth for shaping the conversations and spotting hallucinations.
  2. Give us the API endpoint for a test version of your agent, so no customer data is involved.
  3. Pay, and we’ll email you when testing starts.

We run over 100 realistic, multi-turn conversations, playing customers and attackers. Every conversation then goes through our tried-and-tested judgement framework, so the findings you get are accurate and backed by evidence.

What you get

Within 24 hours of testing starting, you get a full report with every conversation. Each failure is pulled out in your agent’s own words and graded as critical, high, medium or low, with why it failed and what it should have done instead.

From there, you either have the proof to release with confidence, or know exactly what to fix before you do.

Is your AI agent ready for the real world?

See how Mantia tests it before it goes live.