Regression testing of an MCP server

Regression testing of an MCP server is checking that a new version of the server has not broken what worked in the previous one: the contract that clients rely on (the lists of tools, resources and prompts with their schemas and error semantics), the protocol versions the server supports, and the behavior of models that choose tools by their descriptions. The general concept of regression testing is described in the article Test run; the basics of testing MCP servers are in Testing MCP servers: the basics.

The current MCP specification is version 2026-07-28. It defines how protocol versions are identified and negotiated and, since that version, a formal deprecation policy for features of the protocol. It does not define versioning of the tools of a particular server or rules of their compatibility: these remain the responsibility of the server author. Since SEP-3398, accepted on 14 October 2026, new versions of the specification are planned every six months.

Overview

The problem it solves

Between releases, tools of an MCP server are added and renamed, descriptions are edited, the SDK and the supported protocol versions are updated. Each kind of consumer of the server depends on a different part of it.

• Clients cache lists. The results of tools/list, prompts/list, resources/list and resources/templates/list carry ttlMs, and a client may use a cached list until it expires.
• Host settings refer to tools by name. In Claude Code, for example, a permission rule names a tool as mcp__<server>__<tool>.
• Scripts and agents call tools with fixed arguments and parse structured results.
• Models choose tools by name, description and schema.

The protocol has no version field on a tool: a tool definition consists of name, title, description, icons, inputSchema, outputSchema, annotations and _meta. A breaking change becomes visible only when two versions of the contract are compared.

Contract of an MCP server

The contract is the part of the server that a client can observe without reading its code. The table shows which lists it includes and which fields are compared.

Error semantics belong to the contract too, although no list describes them: which failures return a JSON-RPC error and which return a tool result with isError: true, and with which codes. They are checked with tests of error cases, not with a snapshot. The design of the lists themselves is described in the articles Tools in MCP and Prompts in MCP.

Contract snapshots

Taking a snapshot

A contract snapshot is the saved and normalized result of the list methods of one server version. The snapshot of the last release is stored in the repository and serves as the baseline; the snapshot of the candidate version is compared with it.

The MCP Inspector CLI runs one request per call and prints the result as JSON with --format json. The --protocol-era flag selects the protocol era: legacy (the default, an initialize handshake), auto (probe server/discover first, fall back to initialize) or modern (exactly 2026-07-28, no fallback). A snapshot of a modern server is taken with --protocol-era modern; without the flag the CLI connects in the legacy era.

set -euo pipefail
mkdir -p snapshot

inspect() {
  npx @modelcontextprotocol/inspector --cli node build/index.js \
    --protocol-era modern --method "$1" --format json
}

inspect tools/list | jq -S '.result.tools | sort_by(.name)' > snapshot/tools.json
inspect resources/list | jq -S '.result.resources | sort_by(.uri)' > snapshot/resources.json
inspect resources/templates/list \
  | jq -S '.result.resourceTemplates | sort_by(.uriTemplate)' > snapshot/resource-templates.json
inspect prompts/list | jq -S '.result.prompts | sort_by(.name)' > snapshot/prompts.json

Normalization keeps only the lists and sorts them by a stable key, so a change in order alone does not look like a change of the contract. Fields such as nextCursor, ttlMs and cacheScope describe delivery, not the contract, and are left out.

Two rules of the specification affect what a snapshot contains. First, the list of tools may vary by the authorization of the request, for example by the scopes of the token. A server with such behavior gets a snapshot per authorization profile. Second, lists can be paginated, and there is no consistency guarantee across pages. A client that needs a consistent full list re-fetches it from the beginning, and a snapshot follows the same rule.

Comparing snapshots

A text diff of the normalized files (diff baseline/tools.json snapshot/tools.json) shows every change but does not tell a breaking change from a harmless one. For that, a script classifies the changes by the rules in the sections below. The format of its report is not defined by the specification; the example shows one possible format.

{
  "baseline": "1.4.0",
  "candidate": "1.5.0",
  "changes": [
    {
      "list": "tools",
      "name": "search_issues",
      "kind": "removed",
      "breaksClients": true,
      "affectsModels": true
    },
    {
      "list": "tools",
      "name": "create_issue",
      "path": "inputSchema.required",
      "kind": "required_parameter_added",
      "before": ["project", "title"],
      "after": ["project", "title", "priority"],
      "breaksClients": true,
      "affectsModels": true
    },
    {
      "list": "tools",
      "name": "get_issue",
      "path": "outputSchema.properties.assignee.type",
      "kind": "type_changed",
      "before": "string",
      "after": "object",
      "breaksClients": true,
      "affectsModels": false
    },
    {
      "list": "tools",
      "name": "update_issue",
      "path": "description",
      "kind": "description_changed",
      "breaksClients": false,
      "affectsModels": true
    }
  ]
}

In this scheme, a CI job fails on changes with breaksClients: true, and changes with affectsModels: true start the evaluation described below.

Changes that break clients

A breaking change makes a call that worked against the previous version fail against the new one, or makes a client misread the result.

• Removed or renamed tool. A tools/call with the old name gets a protocol error; the example in the specification answers an unknown tool with the JSON-RPC code -32602. A client with a cached list keeps offering the old name until ttlMs expires, and host settings that name the tool stop matching it.
• New required parameter. Calls that worked before lack the parameter and fail validation. Since SEP-1303 such errors are returned as tool execution errors with isError: true.
• Narrowed type. A string becomes an enum, a maximum decreases, a pattern appears, additionalProperties: false is added. Values that were valid are rejected. Boundary values of the old and the new schema are the natural test data; the technique is described in the article Equivalence partitioning and boundary values.
• Changed outputSchema. A property is removed or renamed, its type changes, it leaves required. The specification requires the server to return structuredContent that conforms to outputSchema and recommends that clients validate it. A client that relies on the old structure breaks.
• Changed error semantics. A failure moves from isError: true to a JSON-RPC error or back, or an error code changes. The specification itself made such a change in 2026-07-28: the code for a resource that is not found changed from -32002 to -32602 (SEP-2164). According to SEP-2164, the official SDKs used at least four different codes for this case before, so an SDK update can change the codes of a server without changes to its own code.
• Changed resource URIs or templates. A client that stored a URI or built it from a template no longer finds the resource.
• Removed capability or extension. A client that relies on listChanged or on an extension goes back to the fallback behavior or gets an error.

A client on Streamable HTTP must exclude from the list a tool whose x-mcp-header value violates the constraints of the specification. An invalid annotation added to one parameter therefore removes the tool from the client without an error on the server side.

Changes that affect models

Some changes keep every call valid and still change what the model does, and with it the result of tasks.

• Edited description. The description of a tool and of its parameters is the main signal for the model. Anthropic reports that even small refinements of tool descriptions can change the results considerably; the same holds in the other direction.
• Edited title. The specification intends title for display to users. Whether a host also passes it to the model, the specification does not define.
• New tool. A tool with a purpose close to an existing one makes the choice harder.
• Changed instructions. The instructions field of server/discover can be added by a client to the system prompt.
• Changed annotations. Hints such as readOnlyHint and destructiveHint do not change the protocol. The specification requires clients to treat them as untrusted unless the server is trusted, and does not define how a host uses them.

Summary of changes

The table brings together the kinds of changes, their effect on clients and models, and the check that detects each of them.

Protocol version support

Version identifiers and negotiation

A protocol version is a string in the form YYYY-MM-DD: the date of the last backwards incompatible change. Compatible changes do not increase the version. A revision of the specification is Draft, Current or Final; the current revision is 2026-07-28.

Every request declares its version in _meta, in io.modelcontextprotocol/protocolVersion; on Streamable HTTP the same value is also sent in the MCP-Protocol-Version header. The server accepts or rejects each request independently. For a version it does not support, it returns UnsupportedProtocolVersionError with the supported versions and the requested one; on HTTP the status code is 400 Bad Request.

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32022,
    "message": "Unsupported protocol version",
    "data": {
      "supported": ["2026-07-28", "2025-11-25"],
      "requested": "1900-01-01"
    }
  }
}

The list of supported versions is part of the contract. It is returned by server/discover in supportedVersions, and a regression test checks it in two ways: the field contains the expected versions, and a request with an unsupported version, such as 1900-01-01 above, gets code -32022 with the same list in data.supported.

Legacy and modern versions

The specification calls versions up to 2025-11-25, which open a session with an initialize handshake, legacy, and versions from 2026-07-28 on, which pass the version and capabilities in every request, modern. A dual-era implementation supports both. Its compatibility matrix shows what dropping a version means:

• a legacy client and a dual-era server work;
• a legacy client and a server that supports only modern versions do not work, and a legacy client has no way to move to a newer version;
• a modern client and a legacy server do not work.

Removing legacy support from a dual-era server is therefore a breaking change for every legacy client. The changes of 2026-07-28 and migration from 2025-11-25 are described in the article Stateless MCP.

A dual-era server is tested on both wires. The conformance test suite has a requirement set for each revision (--requirements 2025-11-25, --requirements 2026-07-28), and each set runs at the wire version of its revision. According to the suite's documentation, a scenario that belongs to both revisions has to be run twice, because passing it on one wire says nothing about the other. MCP Inspector checks the same server in both eras with --protocol-era legacy and --protocol-era modern.

Deprecation policy

Since 2026-07-28, features of the specification have three states: Active, Deprecated and Removed (SEP-2596). The deprecation window, the time a deprecated feature stays in the specification, is at least twelve months, counted from the release of the revision that first marked the feature as deprecated. The feature becomes eligible for removal in the first revision released on or after the end of that window. For a feature with an active security risk, the window can be shortened, but not below ninety days.

The specification keeps a registry of deprecated features. In 2026-07-28 it lists:

A server that uses a deprecated feature therefore has a known date after which a new revision may not support it. Tier 1 SDKs must mark deprecated APIs with the native mechanism of the language in their next release and should emit a runtime warning when a deprecated feature is used, so new deprecation warnings in build logs and test output are a signal to check.

Release cadence

SEP-3398 plans releases of the specification generally on the last Tuesday of March and September, and moves the release planned for December 2026 to late March 2027. Tier 1 SDKs publish beta versions about one and a half months before a release for testing in the ecosystem. A run of the regression suite against an SDK beta shows the effect of the next revision before it is released.

Regression of model behavior

A change that affects models is checked by running the evaluation set again: the same tasks, the same models and hosts, the same number of trials. The method is described in the article Evaluating an MCP server with models; for regression testing four points matter.

• The regression set. Anthropic separates capability evaluations, with a low success rate, from regression evaluations, whose success rate stays close to 100 percent. After a description edit the regression set runs first.
• Paired comparison. The old and the new version run on the same tasks, and the results are compared task by task.
• More than success. The number of tool calls, tokens and tool errors are compared along with the success rate. A description that keeps the success rate but doubles the calls is a regression too.
• Tasks without a tool. Tasks where the right behavior is not to call a tool detect a new or edited tool that is called where it is not needed.

The trigger for this run comes from the snapshot diff: a changed description or title, a new tool, changed instructions.

Regression testing in CI

The diagram shows a typical pipeline: from cheap deterministic checks to expensive statistical ones.

build
 |
 +-- unit tests of tool handlers
 +-- start the server
 +-- Inspector smoke test: tools/list --strict, tools/call
 +-- contract snapshot + diff with the baseline
 |     breaking change -> fail
 |     description change -> mark for evaluation
 +-- conformance suite per revision, with expected failures
 |
 +-- evaluation set (marked changes, schedule)

The conformance suite runs against a server over HTTP. A YAML file with known failures is passed with --expected-failures:

server:
  - tools-call-with-progress
  - server-stateless:sep-2575-server-implements-discover

An entry names either a whole scenario or a single check in the form <scenario>:<check-id>. The second form excuses one check and keeps the others of the scenario enforced. The exit code follows four rules:

By the third rule, CI fails after a fix until the entry is removed. For a dual-era server the suite runs once per requirement set:

npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp \
  --requirements 2026-07-28 --expected-failures ./conformance-baseline.yml

npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp \
  --requirements 2025-11-25 --expected-failures ./conformance-baseline.yml

According to the suite's documentation, a requirement set is frozen at the release of its revision and lives in the suite's repository, while an expected-failures file lives in the repository of the implementation. A failure listed in the file is still a failure against the requirement set. The organization of test runs in CI pipelines in general is described in the article Automated tests in CI.

Contract checks, conformance and evaluation

The table compares the three checks of the pipeline, which answer different questions.

The contract check compares the server with itself; conformance testing compares it with the specification. A server can pass conformance and still break its clients by renaming a tool, or keep its contract and fail conformance after an SDK update. Only the evaluation shows how models react to an edited description.

Terms

Sources

• Versioning and Compatibility, specification 2026-07-28(opens in a new tab)
• Versioning, MCP documentation(opens in a new tab)
• Deprecated Features, specification 2026-07-28(opens in a new tab)
• Feature Lifecycle and Deprecation Policy(opens in a new tab)
• Key Changes, specification 2026-07-28(opens in a new tab)
• Tools, specification 2026-07-28(opens in a new tab)
• Caching, specification 2026-07-28(opens in a new tab)
• Schema of the specification 2026-07-28, schema.ts, GitHub(opens in a new tab)
• SEP-1303: Input Validation Errors as Tool Execution Errors(opens in a new tab)
• SEP-2164: Standardize Resource Not Found Error Code(opens in a new tab)
• SEP-2596: Specification Feature Lifecycle and Deprecation Policy(opens in a new tab)
• SEP-3398: Release Cycle Updates(opens in a new tab)
• MCP Inspector, CLI client(opens in a new tab)
• MCP Inspector, protocol eras(opens in a new tab)
• MCP conformance test suite, GitHub(opens in a new tab)
• Configure permissions, Claude Code documentation(opens in a new tab)
• Writing effective tools for agents, Anthropic, 11 September 2025(opens in a new tab)
• Demystifying evals for AI agents, Anthropic, 9 January 2026(opens in a new tab)