AgentTesting sends Claude, GPT, Gemini and open-weight agents through real tasks on your MCP server or website. You get each agent's pass rate, every place they got stuck, and the fix for each one.
| Agent | T1Find a flight | T2Hold a seat | T3Change a date | T4Refund quote | T5Compare hotels | T6Add a bag | Score |
|---|---|---|---|---|---|---|---|
| Claude | 94 | ||||||
| ChatGPT | 83 | ||||||
| Gemini | 74 | ||||||
| Codex | 86 | ||||||
| Browser agent | 69 |
People now ask Claude, ChatGPT and their coding agents to book, buy, compare, cancel and configure things for them. Those agents reach your product through your MCP server, your API or your website.
When a tool name is unclear, an error is vague or a field is missing, the agent doesn’t file a ticket. It retries, guesses, or tells the person something that isn’t true, and you never hear about it. Your logs show which tools were called, not what the agent was trying to do or why it gave up.
AgentTesting puts those agents in front of your product on purpose, before your customers’ agents do.
| Question | Readiness scan | Agent test |
|---|---|---|
| What it looks at | Files and markup: llms.txt, schema, robots rules, headers, an MCP manifest | Real agents doing real tasks against your live product |
| What you learn | Whether agents can find and read you | Whether they finish, how many steps it takes, where they get stuck, and what to change |
| Surfaces | Websites | MCP servers, APIs and websites |
| Agents involved | A crawler | Claude, GPT, Gemini and an open-weight model, side by side |
| Over time | A score you re-run by hand | Nightly and on every deploy, with an alert when a task starts failing |
What the first real run found
We ran an early version on three live MCP servers of our own. All three would pass a readiness scan. The agents still found a verification job that had quietly stopped running, a “proven $6 a month” price for a stack that couldn’t hold the data, and a device lookup that dead-ended without suggesting anything else. None of that shows up in markup.
Paste your MCP server URL, OpenAPI spec or website. Add a test account if any task needs one.
We draft tasks from your tools and docs, phrased the way your users ask. You edit them. Each task gets success criteria, safety rules and an ideal number of steps, fixed before anything runs.
Every task runs through every agent, three times, in isolated sandboxes. Anything that buys, sends, files or deletes is blocked unless you allow it.
A separate model checks each run against its criteria and turns every problem into a friction report: what broke, which agents hit it, and what would have helped.
Hand the fix to your coding agent, ship it, and the panel re-runs to confirm. From then on it runs nightly and on every deploy.
Score by agent
change_bookingSince the October 27 deploy, change_booking requires fare_class, but the error only says "invalid request". Agents retried three to five times. In two runs the agent then told the user the date change had gone through.
Fix: Return a field-level error that names fare_class and its allowed values, and default it to the booking’s current fare.
Sample data from a fictional product. Open the full sample report
Tests show you what our agents hit. Your customers’ agents hit things we didn’t think of. Early access includes a feedback tool you add to your MCP server, so real agents can report problems as they happen, into the same queue as your test results.
// Example. The package ships with early access.
import { feedbackTool } from "@agenttesting/mcp"
server.registerTool(
...feedbackTool({ key: process.env.AGENTTESTING_KEY })
)We're onboarding a few MCP servers, APIs and sites at a time.