Testing LLM applications

Testing an LLM application is the verification of software in which a large language model (LLM) produces part of the result: an answer to a user, a classification, extracted data, a decision to call a tool. The code around the model is tested with ordinary tests. The behavior of the model is measured with evaluation sets: fixed tasks with expected outcomes, run repeatedly and graded by code, by another model or by people.

No international standard defines a method for testing LLM applications. The vocabulary and the practices come from research papers and from the documentation of model providers. The method of an LLM judge was studied in the paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., June 2023). Anthropic describes evaluation in its guide "Define success criteria and build evaluations" and in the engineering post "Demystifying evals for AI agents" (9 January 2026). The ISTQB Certified Tester AI Testing (CT-AI) syllabus, released in October 2021, covers testing of AI-based systems in general.

The problem it solves

A classic test assumes that the same input gives the same output and that the expected result can be written in advance. An LLM application breaks both assumptions: two correct answers can be worded differently, and the same request can give different answers on two runs.

Behavior also changes without any change in the code. A new wording of the system prompt, a new model version or a new document in the knowledge base can improve one group of answers and break another. A manual check of a few requests in a chat window does not show this.

Testing an LLM application separates what can be checked exactly from what can only be measured. The code is covered by deterministic tests. The model's behavior is measured on a fixed set of tasks, and the results of two versions are compared as shares of successful trials.

Parts of an LLM application

The table lists the parts of an LLM application with their typical defects.

Retrieval is the basis of retrieval-augmented generation (RAG), described by Lewis et al. (2020): the application puts the documents it finds into the model's context. A wrong answer can then come from the model or from retrieval, and a separate test of retrieval with the expected documents for each query shows which of the two failed.

Non-determinism

Sampling and temperature

A language model computes a probability for each possible next token and samples one from this distribution. The temperature parameter controls how flat the distribution is: a higher temperature makes unlikely tokens more probable, a lower one concentrates the choice on the most probable tokens.

In the OpenAI API the temperature is a number between 0 and 2 with a default of 1. Anthropic's glossary states that even at temperature 0 the results are not fully deterministic, and identical inputs can give different outputs across API calls.

Seeds

The OpenAI Chat Completions API has a seed parameter: the system makes a best effort to sample deterministically for the same seed and parameters, without a guarantee. The parameter is marked as beta and deprecated, as is the system_fingerprint field that reported backend changes. The newer Responses API has no seed parameter.

Causes outside sampling

The Thinking Machines Lab post "Defeating Nondeterminism in LLM Inference" (Horace He, 10 September 2025) explains why temperature 0 is not enough: the computation depends on the size of the batch in which the server processes the request, and the batch size depends on the load from other users. In the experiment from the post with the open model Qwen3-235B, 1,000 completions of the same prompt at temperature 0 gave 80 different texts. With batch-invariant kernels all 1,000 were identical, at the cost of slower inference.

Anthropic's page on model IDs adds another cause. The weights behind a model ID are fixed, but the serving infrastructure around them can change: the request router, safety classifiers and sampling logic. Such updates can cause minor differences in behavior under the same ID.

What is deterministic

The table separates the parts of an application by whether their result repeats.

The left column is tested with ordinary tests, the right column is measured with evaluation sets.

Evaluation sets

An evaluation set (eval) is a fixed collection of tasks with inputs and expected outcomes, together with graders that score each answer and metrics that summarize the scores. It plays the role of a test suite: the same set is run on every version.

Success criteria

An evaluation starts from success criteria. Anthropic's guide asks for criteria that are specific, measurable, achievable and relevant: "good performance" is not a criterion, "an F1 score of at least 0.85 on held-out posts" is. Most applications need several criteria, such as task fidelity, consistency, tone, privacy preservation, latency and price.

Tasks and expected outcomes

A task contains the input, the context it needs and the expected outcome. The expected outcome is not always a single string. It can be a label, a value, a list of facts that the answer must contain or must not contain, or a state of the system after the run. The example shows one task; the field names are arbitrary.

{
  "id": "returns-policy-004",
  "input": "Can I return a laptop 40 days after delivery?",
  "context_documents": ["policies/returns-v3.md"],
  "expected": {
    "label": "not_eligible",
    "must_mention": ["30 days"],
    "must_not_call": ["issue_refund"]
  },
  "graders": ["json_schema", "exact_label", "tool_calls", "rubric_judge"],
  "tags": ["returns", "edge-case"]
}

Anthropic's guide gives three principles for the set: tasks mirror the real distribution of requests and include edge cases, grading can be automated, and volume is preferred over quality.

Graders

A grader is the logic that scores one aspect of an answer. The "Demystifying evals" post divides graders into three kinds; the table shows their strengths and weaknesses as the post states them.

A task often has several graders. Cheap code-based checks run first, and a model-based grader then scores what code cannot: correctness in substance, completeness, tone.

Metrics

Because answers vary, each task is run in several trials, and the result of a task is the share of successful trials, not a single verdict. The metrics pass@k and pass^k for repeated trials and the statistics of small samples are described in the article Evaluating an MCP server with models. Metrics of classic testing, such as the pass rate and its denominators, are described in the article Test metrics.

Deterministic checks

The output of the model is random, but the rules it must follow are not, and many of them are checked exactly.

• Schema. The answer is valid JSON and matches a JSON Schema.
• Exact match. A label, a number or an identifier equals the expected value after normalization.
• Substring rules. The answer contains a required phrase or does not contain a forbidden one.
• Tool calls. The model called the expected tool and did not call a forbidden one.
• State. After the run the system is in the expected state.

The example below implements several of these checks in Python with the jsonschema library. It returns a list of failures; an empty list means that the task passed the deterministic graders.

import json

import jsonschema

ANSWER_SCHEMA = {
    "type": "object",
    "properties": {
        "label": {"enum": ["eligible", "not_eligible", "unknown"]},
        "answer": {"type": "string"},
    },
    "required": ["label", "answer"],
    "additionalProperties": False,
}


def check_answer(raw_output: str, expected_label: str, tool_calls: list[dict]) -> list[str]:
    try:
        answer = json.loads(raw_output)
    except json.JSONDecodeError as error:
        return [f"not JSON: {error}"]
    try:
        jsonschema.validate(answer, ANSWER_SCHEMA)
    except jsonschema.exceptions.ValidationError as error:
        return [f"schema: {error.message}"]
    failures = []
    if answer["label"] != expected_label:
        failures.append(f"label: {answer['label']} != {expected_label}")
    called = [call["name"] for call in tool_calls]
    if "search_policy" not in called:
        failures.append("search_policy was not called")
    if "issue_refund" in called:
        failures.append("issue_refund must not be called")
    return failures

Model APIs can enforce part of this at generation time. Anthropic's structured outputs use constrained decoding so that the answer follows a JSON Schema given in output_config.format; OpenAI offers a json_schema response format. Anthropic's documentation lists cases where the output can still differ from the schema: a refusal and an answer cut off by max_tokens. The schema check therefore stays in the tests.

LLM-as-judge

Judge types

LLM-as-judge is grading in which a language model scores the answers of the application against instructions. The paper by Zheng et al. describes three variants:

• Pairwise comparison. The judge sees a question and two answers and chooses the better one or declares a tie.
• Single answer grading. The judge assigns a score to one answer.
• Reference-guided grading. The judge receives a reference answer, which helps with questions such as math problems.

According to the paper, GPT-4 as a judge reached over 80 percent agreement with human preferences, the same level of agreement as between humans.

Known biases

The same paper describes systematic errors of LLM judges. The table summarizes them with the numbers reported for the models of 2023.

The paper proposes countermeasures. The pair is judged twice with the positions swapped, and a win counts only if both verdicts agree. Examples in the judge prompt raised GPT-4's consistency to 77.5 percent, and a reference answer reduced the failure rate on math questions from 70 to 15 percent.

Rubrics

A rubric is the set of criteria and allowed verdicts that the judge applies to an answer. Anthropic's guide gives three rules: the rubric is detailed and clear, the judge returns a constrained result such as "correct" or "incorrect" or a score from 1 to 5, and the judge reasons before giving the score. The guide also recommends grading with a different model from the one that produced the answer.

The "Demystifying evals" post adds two rules: the judge can answer "Unknown" when the information is insufficient, and each dimension of an answer gets its own isolated judge.

Calibration against human labels

A judge is a measuring instrument and is checked like one: people label a sample of answers with the same rubric, the judge grades the same answers, and the verdicts are compared. The post requires judges to be closely calibrated with human experts; after that, occasional human review is enough.

The example shows 200 answers labeled by people and by a judge.

The raw agreement is (120 + 50) / 200 = 85 percent. Part of it is expected by chance, so Cohen's kappa (Cohen, 1960) corrects for it: kappa = (p_o - p_e) / (1 - p_e), where p_o is the observed agreement and p_e the agreement expected from the shares of each verdict. Here p_e = 0.65 x 0.70 + 0.35 x 0.30 = 0.56, and kappa = (0.85 - 0.56) / 0.44, about 0.66.

The judge also passed 20 of the 70 answers that people failed, 29 percent. This is the more dangerous error, and the rubric is refined on those 20 answers before the judge is used.

Human review

People write the expected outcomes and rubrics, label the calibration sample and read transcripts. The "Demystifying evals" post states that scores are not taken at face value until someone has read transcripts of the trials, because a failure can come from the grader and not from the application.

Human grading is also used where no other grader is reliable, for example for legal or medical content. Anthropic's guide calls it the most flexible method and recommends avoiding it where possible because it is slow and expensive.

Regression after a model or prompt change

Model versions

Providers publish models under identifiers that fix a version. A model snapshot is such a fixed version of a model under its own identifier. On Anthropic's platform each model ID identifies a pinned version with fixed weights; older models have dated IDs, such as claude-sonnet-4-5-20250929, and aliases that move to the newest snapshot. OpenAI also lists dated identifiers, such as o3-2025-04-16.

A pinned version is retired after a notice period: at least 60 days at Anthropic for publicly released models, at least 6 months at OpenAI for generally available models. A retirement forces a model change, and the regression run for it is planned in advance.

A pinned model ID fixes the weights, not the whole behavior. Anthropic states that changes in the serving infrastructure can cause minor differences under the same ID, so the regression set also runs on a schedule, not only after changes.

Versioned prompts and a fixed set

A prompt is versioned with the code. Each run of the evaluation records everything that affects the answer: the prompt version, the model ID, the parameters, the retrieval index, the tool definitions and the graders. Without this record two runs cannot be compared.

The "Demystifying evals" post separates two purposes. A capability evaluation measures what the application cannot yet do well, and its success rate starts low. A regression evaluation checks what already worked, and its success rate stays near 100 percent. Tasks that pass consistently move from the first set to the second.

Comparing runs

A change is accepted after a comparison of two runs on the same tasks, with the same graders and the same number of trials. The comparison is made task by task: a total that stays the same can hide ten tasks that broke and ten that improved. Tokens, latency and tool calls are compared too.

Regression of the tools that an application uses through the Model Context Protocol is described in the article Regression testing of an MCP server. Attacks through instructions hidden in input data are a separate kind of test, described in the article Prompt injection.

Classic software and LLM applications

The table compares testing of classic software and of LLM applications.

The two kinds of testing are combined in one application. The deterministic parts are tested like any code on every change. The evaluation set runs on changes of the prompt, the model or the data and on a schedule.

A non-deterministic result is not the same as a flaky test: a flaky test fails because of a defect in the test or the environment and is fixed, while the variation of a model's answers is a property of the system and is measured. Flaky tests are described in the article Flaky tests.

Terms

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, L. Zheng et al., arXiv:2306.05685, NeurIPS 2023(opens in a new tab)
• Define success criteria and build evaluations, Claude Platform documentation(opens in a new tab)
• Demystifying evals for AI agents, Anthropic, 9 January 2026(opens in a new tab)
• Glossary: Temperature, Claude Platform documentation(opens in a new tab)
• Model IDs and versions, Claude Platform documentation(opens in a new tab)
• Model deprecations, Claude Platform documentation(opens in a new tab)
• Structured outputs, Claude Platform documentation(opens in a new tab)
• Create chat completion, OpenAI API reference(opens in a new tab)
• Deprecations, OpenAI API documentation(opens in a new tab)
• Defeating Nondeterminism in LLM Inference, H. He, Thinking Machines Lab, 10 September 2025(opens in a new tab)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, P. Lewis et al., arXiv:2005.11401(opens in a new tab)
• A Coefficient of Agreement for Nominal Scales, J. Cohen, Educational and Psychological Measurement, 1960(opens in a new tab)
• ISTQB releases Certified Tester AI Testing (CT-AI), ISTQB, October 2021(opens in a new tab)
• jsonschema: Schema Validation, Python library documentation(opens in a new tab)