Evaluating an MCP server with models

Evaluation of an MCP server is a run of a fixed set of tasks through a model that has the server connected, repeated several times per task, with automatic grading of the outcome and measurement of tool calls, tokens, time and errors. It shows whether models complete the tasks the server is built for, using its tools, their descriptions and their results. The place of evaluation among the other levels of testing is described in the article Testing MCP servers: the basics.

The MCP specification, current version 2026-07-28, defines the protocol and does not define evaluation: the documentation of the protocol has no section on it. The method for agent tools is described by Anthropic in two engineering posts: "Writing effective tools for agents" (11 September 2025) and "Demystifying evals for AI agents" (9 January 2026). Anthropic also publishes an evaluation guide and a harness for MCP servers as part of the mcp-builder skill in the anthropics/skills repository.

Overview

The problem evaluation solves

Unit tests and conformance testing check the code and the protocol, but do not show what a model does with the server: it can call the wrong tool, fill in the arguments incorrectly or stop before the task is done.

A manual check in a chat assistant gives one trial of one scenario. It cannot be repeated exactly, and its result cannot be compared between two versions of the descriptions or two models.

Evaluation replaces this with a fixed set of tasks, automatic grading and repeated trials. Its numbers can be compared between versions of the server, of the descriptions and between models.

Parts of an evaluation

The "Demystifying evals for AI agents" post defines the vocabulary below. The third column shows what each part means for an MCP server.

The post stresses that evaluating an agent means evaluating the model and the agent harness together. For an MCP server the server itself is the third component.

Tasks

What makes a good task

Anthropic recommends tasks based on real use and real data, not on a simplified sandbox. A strong task requires several tool calls, sometimes dozens. A weak task names everything directly: schedule a meeting with a given email address. A strong task requires finding the data first: schedule a meeting about a project, attach the notes from the previous meeting and book a room.

The "Demystifying evals" post adds practical rules:

• Source. A set of 20 to 50 simple tasks drawn from real failures is enough to start. Bug trackers and support requests supply such tasks.
• Unambiguity. Two experts reach the same verdict on a result. Each task has a reference solution.
• Balance. The set contains tasks where a tool should be called and tasks where calling it is a mistake.
• Broken tasks. A task that no trial passes usually points to a defect in the task or the grader, not in the agent.

The guide of mcp-builder adds that questions are independent, use only read-only operations and have answers that do not change over time: a count of reactions to a message is unsuitable, a fact about a closed project is suitable.

Verifiable outcomes

The table shows the three kinds of graders used for MCP servers.

An exact answer works for questions with one value: a name, a date, a number. A grader must not reject a correct answer because of formatting or an equivalent wording, so the task states the format, for example "use YYYY/MM/DD".

A state check grades the outcome, not the agent's report: the grader looks for the reservation in the database instead of trusting the agent's statement that it was made. For tools that change data this is the main kind of grader.

LLM-as-judge is used where the result is free text. The general method, rubrics and calibration are described in the article Testing LLM applications.

A task can also list the expected tools, but Anthropic warns against overspecifying: a task often has several valid solution paths, and the result is graded, not the path.

Task definition

The specification and the posts do not define a file format for tasks. The example below shows the fields that a task definition usually needs; the field names are arbitrary.

{
  "id": "issues-017",
  "split": "train",
  "prompt": "Find the bug that was reopened most often in the Billing project during March 2026 and add the label needs-review to it. Reply with the issue key only.",
  "grader": [
    { "type": "exact_answer", "expected": "BILL-482" },
    { "type": "state_check", "query": "labels of BILL-482", "expected_contains": "needs-review" }
  ],
  "expected_tools": ["search_issues", "get_issue_history", "update_issue"],
  "tags": ["search", "write"]
}

The split field separates tasks used for tuning from tasks kept aside. The expected_tools field is compared with the transcript but does not decide the result.

Running an evaluation

The agent loop

Anthropic recommends running evaluations through direct calls to the model API, not through a chat interface. Each task gets its own loop: the model calls tools and receives their results until it gives a final answer.

Evaluation harness
 |
 +-- tools/list --------------------------> MCP server (version under test)
 |     name, description, inputSchema
 |
 +-- task prompt + tool definitions ------> model API
 |                                            |
 |   <------------------------------------ tool call (name, arguments)
 +-- tools/call --------------------------> MCP server
 |   <------------------------------------ result or isError: true
 +-- tool result ------------------------> model API
 |     ... repeated until a final answer ...
 |
 +-- final answer + environment state ----> graders
 +-- transcript, counters, timings -------> result record

The tool definitions passed to the model come from tools/list of the server version under test, so a changed description reaches the model without changes to the harness.

The system prompt asks the agent for a structured answer that a grader can parse, and for reasoning and feedback before tool calls; these show in the transcript why the agent chose or skipped a tool.

Isolated trials

Each trial starts from a clean environment: shared state makes failures correlated, and in an example from the "Demystifying evals" post Claude read the git history left by previous trials. For an MCP server this means a reset of test data before each trial and state handles that do not survive between trials.

Result record

Each trial produces a record that keeps the inputs of the run together with the results. The example below is one possible format; the field names are arbitrary.

{
  "task_id": "issues-017",
  "trial": 3,
  "model": "model-a",
  "agent_harness": "eval-loop-1.2",
  "server_version": "1.5.0",
  "passed": false,
  "graders": [
    { "type": "exact_answer", "passed": true },
    { "type": "state_check", "passed": false }
  ],
  "tool_calls": 14,
  "tool_errors": 3,
  "protocol_errors": 0,
  "input_tokens": 48210,
  "output_tokens": 2315,
  "duration_ms": 41870,
  "transcript": "runs/2026-11-03/issues-017-3.jsonl"
}

Tool execution errors (results with isError: true) and protocol errors (JSON-RPC errors) are counted separately: the first kind reaches the model, the second usually does not.

Repeated trials and success rates

The same task gives different sequences of tool calls in different trials. A single passed trial shows that the task can be solved, not that it is solved reliably, so the result of a task is the share of successful trials. Two metrics describe the results of k trials of one task.

pass@k is the probability that at least one of k trials succeeds. The metric was introduced in the paper "Evaluating Large Language Models Trained on Code" (Chen et al., July 2021) for code generation. The paper estimates it from n ≥ k trials, of which c succeeded, as 1 - C(n-c, k) / C(n, k), averaged over tasks; C is the binomial coefficient.

pass^k is the probability that all k trials succeed. The metric was proposed in the τ-bench paper (Yao et al., June 2024) to measure the reliability of agents.

At k = 1 both metrics equal the per-trial success rate. As k grows, pass@k approaches 100 percent and pass^k approaches zero. The "Demystifying evals" post gives the example of a 75 percent per-trial success rate: the probability that three trials in a row succeed is 0.75³, about 42 percent.

The table shows both metrics for independent trials with a fixed per-trial success rate p, computed as 1 - (1 - p)^k and p^k.

pass@k fits a scenario where one success is enough, for example when a developer can repeat the request. pass^k fits a scenario where every run must succeed, for example an agent that serves users without supervision.

Metrics

The table shows the metrics Anthropic recommends collecting and what they indicate for an MCP server.

The metrics are comparable only on the same set of tasks.

The model and host matrix

Models differ in how they choose tools. Anthropic reports that even the naming scheme of tools, a prefix or a suffix with the service name, gives different results on different models.

The host also affects the result, because it builds the context of the model: which tools the model sees, what system prompt surrounds them, how long results are handled. The instructions field of the server/discover result is an example: according to the schema, a client can use it, for example by adding it to the system prompt, but is not required to. Anthropic mentions another host-specific behavior: Claude Code limits a tool response to 25,000 tokens by default.

The evaluation is therefore run as a matrix: groups of tasks in rows, combinations of a model and a harness in columns. The example shows the structure; the values are illustrative.

The number of trials grows with every axis: thirty tasks with five trials each on four combinations is 600 trials. The combinations are chosen by the models and hosts that the users of the server work with.

Transcript analysis

The numbers show that something is wrong; the transcripts show why. Anthropic recommends reading the reasoning and feedback of the agent and then the raw tool calls and results: what the agent leaves out can matter more than what it includes. According to the "Demystifying evals" post, scores are not trusted until the transcripts have been read.

The table lists patterns described by Anthropic and the changes on the server side that they point to.

One of the examples in the post is the web search tool of Claude: the model appended "2025" to the query parameter without need, which distorted the results. A change in the tool description fixed it.

A failure can also be caused by the grader. In the CORE-Bench example from the "Demystifying evals" post, Claude Opus 4.5 scored 42 percent until grading errors were fixed, and 95 percent after; one error rejected the answer "96.12" against the reference "96.124991". Transcripts of failed trials separate errors of the agent from errors of the grader.

The design of tools and results that these patterns lead to is described in the articles Tools in MCP and MCP server over an existing API.

Held-out tasks

Tuning descriptions on the same tasks that measure the result leads to overfitting: the descriptions improve for those tasks and not in general. Anthropic therefore keeps a held-out test set that is not used while the tools are changed.

The "Demystifying evals" post separates two purposes of evaluation:

• Capability evaluation asks what the agent can do well. Its success rate starts low and leaves room for improvement.
• Regression evaluation asks whether the agent still does what it did before. Its success rate stays close to 100 percent.

A capability task that passes consistently moves to the regression set. A set where every task passes shows only regressions, not improvements; the post calls this saturation. Re-running the evaluation after changes to the server is described in the article Regression testing of an MCP server.

Small samples

Evaluation sets of MCP servers are usually small. Anthropic's post "A statistical approach to model evaluations" (19 November 2024) and the accompanying paper "Adding Error Bars to Evals" recommend:

• reporting the standard error of the mean with each score; the 95 percent confidence interval is the score plus or minus 1.96 standard errors;
• clustering standard errors when results are not independent, for example several trials of one task; according to the post, clustered errors on popular evaluations can be more than three times as large as naive ones;
• running several trials per task and averaging them;
• comparing two versions on the same tasks with paired differences, which removes the variance of task difficulty;
• choosing the number of tasks with a power analysis.

For a share of successes p on n tasks, the standard error is the square root of p(1 - p) / n. At 80 percent on 20 tasks it is about 8.9 percentage points, and the 95 percent interval is about plus or minus 17.5 points. On 20 tasks one task changes the share by 5 points, so 80 against 90 percent on such a set does not by itself show that one version is better.

Evaluation, conformance testing and unit tests

The table compares three kinds of checks that are often run on the same server.

Unit tests and conformance testing answer whether the server is correct; evaluation answers whether a model can use it.

The kinds of checks do not replace each other. A defect in code is found faster and cheaper by a unit test, a protocol defect by conformance testing. Evaluation is therefore run after the deterministic checks have passed.

Terms

Sources

• Writing effective tools for agents, Anthropic, 11 September 2025(opens in a new tab)
• Demystifying evals for AI agents, Anthropic, 9 January 2026(opens in a new tab)
• A statistical approach to model evaluations, Anthropic, 19 November 2024(opens in a new tab)
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, E. Miller, arXiv:2411.00640(opens in a new tab)
• Evaluating Large Language Models Trained on Code, M. Chen et al., arXiv:2107.03374(opens in a new tab)
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, S. Yao et al., arXiv:2406.12045(opens in a new tab)
• MCP Server Evaluation Guide, mcp-builder skill, anthropics/skills, GitHub(opens in a new tab)
• Tools, specification 2026-07-28(opens in a new tab)
• Discovery, specification 2026-07-28(opens in a new tab)