[{"data":1,"prerenderedAt":91},["ShallowReactive",2],{"wiki-ai-testing-testing-llm-applications-en":3},{"meta":4,"html":8,"headings":9,"words":90},{"title":5,"description":6,"published":7,"updated":7},"Testing LLM applications","How to test applications built on language models: non-determinism, evaluation sets (evals), LLM-as-judge and regressions after a model or prompt change.","2026-11-03","\u003Cp>\u003Cstrong>Testing an LLM application\u003C/strong> is the verification of software in which a large language model (LLM) produces part of the result: an answer to a user, a classification, extracted data, a decision to call a tool. The code around the model is tested with ordinary tests. The behavior of the model is measured with evaluation sets: fixed tasks with expected outcomes, run repeatedly and graded by code, by another model or by people.\u003C/p>\n\u003Cp>No international standard defines a method for testing LLM applications. The vocabulary and the practices come from research papers and from the documentation of model providers. The method of an LLM judge was studied in the paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., June 2023). Anthropic describes evaluation in its guide "Define success criteria and build evaluations" and in the engineering post "Demystifying evals for AI agents" (9 January 2026). The ISTQB Certified Tester AI Testing (CT-AI) syllabus, released in October 2021, covers testing of AI-based systems in general.\u003C/p>\n\u003Ch2 id=\"the-problem-it-solves\" class=\"wiki-heading\">The problem it solves\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-it-solves\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A classic test assumes that the same input gives the same output and that the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"expected-result\">expected result\u003C/span> can be written in advance. An LLM application breaks both assumptions: two correct answers can be worded differently, and the same request can give different answers on two runs.\u003C/p>\n\u003Cp>Behavior also changes without any change in the code. A new wording of the system prompt, a new model version or a new document in the knowledge base can improve one group of answers and break another. A manual check of a few requests in a chat window does not show this.\u003C/p>\n\u003Cp>Testing an LLM application separates what can be checked exactly from what can only be measured. The code is covered by deterministic tests. The model's behavior is measured on a fixed set of tasks, and the results of two versions are compared as shares of successful trials.\u003C/p>\n\u003Ch2 id=\"parts-of-an-llm-application\" class=\"wiki-heading\">Parts of an LLM application\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"parts-of-an-llm-application\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table lists the parts of an LLM application with their typical \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defects\u003C/span>.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Part\u003C/th>\n\u003Cth>What it does\u003C/th>\n\u003Cth>Typical defects\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Prompt\u003C/td>\n\u003Ctd>System prompt, instructions, examples, templates\u003C/td>\n\u003Ctd>Ambiguous instructions, conflicting rules, missing edge cases\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Model\u003C/td>\n\u003Ctd>Generates the answer\u003C/td>\n\u003Ctd>Wrong facts, ignored instructions, changed behavior after an update\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Retrieval\u003C/td>\n\u003Ctd>Finds documents for the context\u003C/td>\n\u003Ctd>Irrelevant or outdated documents, missing documents\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Tools\u003C/td>\n\u003Ctd>Functions the model calls\u003C/td>\n\u003Ctd>Wrong tool, wrong arguments, unnecessary calls\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Output parsing\u003C/td>\n\u003Ctd>Turns the answer into data\u003C/td>\n\u003Ctd>Invalid JSON, missing fields, unexpected values\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Guardrails\u003C/td>\n\u003Ctd>Input and output filters, policy checks\u003C/td>\n\u003Ctd>Harmful content passes, legitimate requests are blocked\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Retrieval is the basis of retrieval-augmented generation (RAG), described by Lewis et al. (2020): the application puts the documents it finds into the model's context. A wrong answer can then come from the model or from retrieval, and a separate test of retrieval with the expected documents for each query shows which of the two failed.\u003C/p>\n\u003Ch2 id=\"non-determinism\" class=\"wiki-heading\">Non-determinism\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"non-determinism\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"sampling-and-temperature\" class=\"wiki-heading\">Sampling and temperature\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sampling-and-temperature\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A language model computes a probability for each possible next token and samples one from this distribution. The temperature parameter controls how flat the distribution is: a higher temperature makes unlikely tokens more probable, a lower one concentrates the choice on the most probable tokens.\u003C/p>\n\u003Cp>In the OpenAI API the temperature is a number between 0 and 2 with a default of 1. Anthropic's glossary states that even at temperature 0 the results are not fully deterministic, and identical inputs can give different outputs across API calls.\u003C/p>\n\u003Ch3 id=\"seeds\" class=\"wiki-heading\">Seeds\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"seeds\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The OpenAI Chat Completions API has a \u003Ccode>seed\u003C/code> parameter: the system makes a best effort to sample deterministically for the same seed and parameters, without a guarantee. The parameter is marked as beta and deprecated, as is the \u003Ccode>system_fingerprint\u003C/code> field that reported backend changes. The newer Responses API has no \u003Ccode>seed\u003C/code> parameter.\u003C/p>\n\u003Ch3 id=\"causes-outside-sampling\" class=\"wiki-heading\">Causes outside sampling\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"causes-outside-sampling\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The Thinking Machines Lab post "Defeating Nondeterminism in LLM Inference" (Horace He, 10 September 2025) explains why temperature 0 is not enough: the computation depends on the size of the batch in which the server processes the request, and the batch size depends on the load from other users. In the experiment from the post with the open model Qwen3-235B, 1,000 completions of the same prompt at temperature 0 gave 80 different texts. With batch-invariant kernels all 1,000 were identical, at the cost of slower inference.\u003C/p>\n\u003Cp>Anthropic's page on model IDs adds another cause. The weights behind a model ID are fixed, but the serving infrastructure around them can change: the request router, safety classifiers and sampling logic. Such updates can cause minor differences in behavior under the same ID.\u003C/p>\n\u003Ch3 id=\"what-is-deterministic\" class=\"wiki-heading\">What is deterministic\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"what-is-deterministic\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The table separates the parts of an application by whether their result repeats.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Deterministic\u003C/th>\n\u003Cth>Not deterministic\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Prompt templates and their rendering\u003C/td>\n\u003Ctd>Text generated by the model\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Parsers, validators, business rules\u003C/td>\n\u003Ctd>Choice of a tool and its arguments\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Tool handlers for given arguments\u003C/td>\n\u003Ctd>Wording, length and order of the answer\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Retrieval over a fixed index for a fixed query vector\u003C/td>\n\u003Ctd>Answers of an LLM judge\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The left column is tested with ordinary tests, the right column is measured with evaluation sets.\u003C/p>\n\u003Ch2 id=\"evaluation-sets\" class=\"wiki-heading\">Evaluation sets\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"evaluation-sets\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>An \u003Cstrong>evaluation set\u003C/strong> (eval) is a fixed collection of tasks with inputs and expected outcomes, together with graders that score each answer and metrics that summarize the scores. It plays the role of a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-suite\">test suite\u003C/span>: the same set is run on every version.\u003C/p>\n\u003Ch3 id=\"success-criteria\" class=\"wiki-heading\">Success criteria\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"success-criteria\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>An evaluation starts from success criteria. Anthropic's guide asks for criteria that are specific, measurable, achievable and relevant: "good performance" is not a criterion, "an F1 score of at least 0.85 on held-out posts" is. Most applications need several criteria, such as task fidelity, consistency, tone, privacy preservation, latency and price.\u003C/p>\n\u003Ch3 id=\"tasks-and-expected-outcomes\" class=\"wiki-heading\">Tasks and expected outcomes\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"tasks-and-expected-outcomes\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A task contains the input, the context it needs and the expected outcome. The expected outcome is not always a single string. It can be a label, a value, a list of facts that the answer must contain or must not contain, or a state of the system after the run. The example shows one task; the field names are arbitrary.\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"json\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">JSON\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>{\n  "id": "returns-policy-004",\n  "input": "Can I return a laptop 40 days after delivery?",\n  "context_documents": ["policies/returns-v3.md"],\n  "expected": {\n    "label": "not_eligible",\n    "must_mention": ["30 days"],\n    "must_not_call": ["issue_refund"]\n  },\n  "graders": ["json_schema", "exact_label", "tool_calls", "rubric_judge"],\n  "tags": ["returns", "edge-case"]\n}\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Anthropic's guide gives three principles for the set: tasks mirror the real distribution of requests and include edge cases, grading can be automated, and volume is preferred over quality.\u003C/p>\n\u003Ch3 id=\"graders\" class=\"wiki-heading\">Graders\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"graders\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A grader is the logic that scores one aspect of an answer. The "Demystifying evals" post divides graders into three kinds; the table shows their strengths and weaknesses as the post states them.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Grader\u003C/th>\n\u003Cth>Strengths\u003C/th>\n\u003Cth>Weaknesses\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Code-based\u003C/td>\n\u003Ctd>Fast, cheap, objective, reproducible\u003C/td>\n\u003Ctd>Brittle to valid variations, lacks nuance\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Model-based\u003C/td>\n\u003Ctd>Flexible, scalable, handles free text\u003C/td>\n\u003Ctd>Non-deterministic, more expensive, needs calibration\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Human\u003C/td>\n\u003Ctd>Gold-standard quality, expert judgment\u003C/td>\n\u003Ctd>Slow, expensive, hard to scale\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>A task often has several graders. Cheap code-based checks run first, and a model-based grader then scores what code cannot: correctness in substance, completeness, tone.\u003C/p>\n\u003Ch3 id=\"metrics\" class=\"wiki-heading\">Metrics\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"metrics\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Because answers vary, each task is run in several trials, and the result of a task is the share of successful trials, not a single verdict. The metrics \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"pass-at-k\">pass@k\u003C/span> and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"pass-caret-k\">pass^k\u003C/span> for repeated trials and the statistics of small samples are described in the article \u003Ca href=\"/en/wiki/mcp/evaluating-mcp-servers\" class=\"wiki-link wiki-link--article\">Evaluating an MCP server with models\u003C/a>. Metrics of classic testing, such as the \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"pass-rate\">pass rate\u003C/span> and its denominators, are described in the article \u003Ca href=\"/en/wiki/test-management/test-metrics\" class=\"wiki-link wiki-link--article\">Test metrics\u003C/a>.\u003C/p>\n\u003Ch2 id=\"deterministic-checks\" class=\"wiki-heading\">Deterministic checks\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"deterministic-checks\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The output of the model is random, but the rules it must follow are not, and many of them are checked exactly.\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Schema.\u003C/strong> The answer is valid JSON and matches a JSON Schema.\u003C/li>\n\u003Cli>\u003Cstrong>Exact match.\u003C/strong> A label, a number or an identifier equals the expected value after normalization.\u003C/li>\n\u003Cli>\u003Cstrong>Substring rules.\u003C/strong> The answer contains a required phrase or does not contain a forbidden one.\u003C/li>\n\u003Cli>\u003Cstrong>Tool calls.\u003C/strong> The model called the expected tool and did not call a forbidden one.\u003C/li>\n\u003Cli>\u003Cstrong>State.\u003C/strong> After the run the system is in the expected state.\u003C/li>\n\u003C/ul>\n\u003Cp>The example below implements several of these checks in Python with the \u003Ccode>jsonschema\u003C/code> library. It returns a list of failures; an empty list means that the task passed the deterministic graders.\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"python\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">python\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import json\n\nimport jsonschema\n\nANSWER_SCHEMA = {\n    "type": "object",\n    "properties": {\n        "label": {"enum": ["eligible", "not_eligible", "unknown"]},\n        "answer": {"type": "string"},\n    },\n    "required": ["label", "answer"],\n    "additionalProperties": False,\n}\n\n\ndef check_answer(raw_output: str, expected_label: str, tool_calls: list[dict]) -> list[str]:\n    try:\n        answer = json.loads(raw_output)\n    except json.JSONDecodeError as error:\n        return [f"not JSON: {error}"]\n    try:\n        jsonschema.validate(answer, ANSWER_SCHEMA)\n    except jsonschema.exceptions.ValidationError as error:\n        return [f"schema: {error.message}"]\n    failures = []\n    if answer["label"] != expected_label:\n        failures.append(f"label: {answer['label']} != {expected_label}")\n    called = [call["name"] for call in tool_calls]\n    if "search_policy" not in called:\n        failures.append("search_policy was not called")\n    if "issue_refund" in called:\n        failures.append("issue_refund must not be called")\n    return failures\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Model APIs can enforce part of this at generation time. Anthropic's structured outputs use constrained decoding so that the answer follows a JSON Schema given in \u003Ccode>output_config.format\u003C/code>; OpenAI offers a \u003Ccode>json_schema\u003C/code> response format. Anthropic's documentation lists cases where the output can still differ from the schema: a refusal and an answer cut off by \u003Ccode>max_tokens\u003C/code>. The schema check therefore stays in the tests.\u003C/p>\n\u003Ch2 id=\"llm-as-judge\" class=\"wiki-heading\">LLM-as-judge\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"llm-as-judge\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"judge-types\" class=\"wiki-heading\">Judge types\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"judge-types\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>\u003Cstrong>LLM-as-judge\u003C/strong> is grading in which a language model scores the answers of the application against instructions. The paper by Zheng et al. describes three variants:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Pairwise comparison.\u003C/strong> The judge sees a question and two answers and chooses the better one or declares a tie.\u003C/li>\n\u003Cli>\u003Cstrong>Single answer grading.\u003C/strong> The judge assigns a score to one answer.\u003C/li>\n\u003Cli>\u003Cstrong>Reference-guided grading.\u003C/strong> The judge receives a reference answer, which helps with questions such as math problems.\u003C/li>\n\u003C/ul>\n\u003Cp>According to the paper, GPT-4 as a judge reached over 80 percent agreement with human preferences, the same level of agreement as between humans.\u003C/p>\n\u003Ch3 id=\"known-biases\" class=\"wiki-heading\">Known biases\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"known-biases\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The same paper describes systematic errors of LLM judges. The table summarizes them with the numbers reported for the models of 2023.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Bias\u003C/th>\n\u003Cth>What happens\u003C/th>\n\u003Cth>Reported finding\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Position bias\u003C/td>\n\u003Ctd>The judge favors an answer because of its position\u003C/td>\n\u003Ctd>GPT-4 gave the same verdict after swapping the answers in 65 percent of cases\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Verbosity bias\u003C/td>\n\u003Ctd>The judge favors longer answers\u003C/td>\n\u003Ctd>Answers padded with repeated items fooled Claude-v1 and GPT-3.5 in 91.3 percent of cases, GPT-4 in 8.7 percent\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Self-enhancement bias\u003C/td>\n\u003Ctd>The judge favors answers of its own model\u003C/td>\n\u003Ctd>GPT-4 favored itself by about 10 percent in win rate, Claude-v1 by about 25 percent; the authors call the data too limited to confirm the bias\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Limited reasoning\u003C/td>\n\u003Ctd>The judge misgrades math and reasoning answers\u003C/td>\n\u003Ctd>Judges were misled by the answers they graded\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The paper proposes countermeasures. The pair is judged twice with the positions swapped, and a win counts only if both verdicts agree. Examples in the judge prompt raised GPT-4's consistency to 77.5 percent, and a reference answer reduced the failure rate on math questions from 70 to 15 percent.\u003C/p>\n\u003Ch3 id=\"rubrics\" class=\"wiki-heading\">Rubrics\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"rubrics\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A \u003Cstrong>rubric\u003C/strong> is the set of criteria and allowed verdicts that the judge applies to an answer. Anthropic's guide gives three rules: the rubric is detailed and clear, the judge returns a constrained result such as "correct" or "incorrect" or a score from 1 to 5, and the judge reasons before giving the score. The guide also recommends grading with a different model from the one that produced the answer.\u003C/p>\n\u003Cp>The "Demystifying evals" post adds two rules: the judge can answer "Unknown" when the information is insufficient, and each dimension of an answer gets its own isolated judge.\u003C/p>\n\u003Ch3 id=\"calibration-against-human-labels\" class=\"wiki-heading\">Calibration against human labels\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"calibration-against-human-labels\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A judge is a measuring instrument and is checked like one: people label a sample of answers with the same rubric, the judge grades the same answers, and the verdicts are compared. The post requires judges to be closely calibrated with human experts; after that, occasional human review is enough.\u003C/p>\n\u003Cp>The example shows 200 answers labeled by people and by a judge.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Judge: pass\u003C/th>\n\u003Cth>Judge: fail\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>People: pass\u003C/td>\n\u003Ctd>120\u003C/td>\n\u003Ctd>10\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>People: fail\u003C/td>\n\u003Ctd>20\u003C/td>\n\u003Ctd>50\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The raw agreement is (120 + 50) / 200 = 85 percent. Part of it is expected by chance, so Cohen's kappa (Cohen, 1960) corrects for it: kappa = (p_o - p_e) / (1 - p_e), where p_o is the observed agreement and p_e the agreement expected from the shares of each verdict. Here p_e = 0.65 x 0.70 + 0.35 x 0.30 = 0.56, and kappa = (0.85 - 0.56) / 0.44, about 0.66.\u003C/p>\n\u003Cp>The judge also passed 20 of the 70 answers that people failed, 29 percent. This is the more dangerous error, and the rubric is refined on those 20 answers before the judge is used.\u003C/p>\n\u003Ch2 id=\"human-review\" class=\"wiki-heading\">Human review\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"human-review\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>People write the expected outcomes and rubrics, label the calibration sample and read transcripts. The "Demystifying evals" post states that scores are not taken at face value until someone has read transcripts of the trials, because a failure can come from the grader and not from the application.\u003C/p>\n\u003Cp>Human grading is also used where no other grader is reliable, for example for legal or medical content. Anthropic's guide calls it the most flexible method and recommends avoiding it where possible because it is slow and expensive.\u003C/p>\n\u003Ch2 id=\"regression-after-a-model-or-prompt-change\" class=\"wiki-heading\">Regression after a model or prompt change\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"regression-after-a-model-or-prompt-change\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"model-versions\" class=\"wiki-heading\">Model versions\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"model-versions\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Providers publish models under identifiers that fix a version. A \u003Cstrong>model snapshot\u003C/strong> is such a fixed version of a model under its own identifier. On Anthropic's platform each model ID identifies a pinned version with fixed weights; older models have dated IDs, such as \u003Ccode>claude-sonnet-4-5-20250929\u003C/code>, and aliases that move to the newest snapshot. OpenAI also lists dated identifiers, such as \u003Ccode>o3-2025-04-16\u003C/code>.\u003C/p>\n\u003Cp>A pinned version is retired after a notice period: at least 60 days at Anthropic for publicly released models, at least 6 months at OpenAI for generally available models. A retirement forces a model change, and the regression run for it is planned in advance.\u003C/p>\n\u003Cblockquote>\n\u003Cp>A pinned model ID fixes the weights, not the whole behavior. Anthropic states that changes in the serving infrastructure can cause minor differences under the same ID, so the regression set also runs on a schedule, not only after changes.\u003C/p>\n\u003C/blockquote>\n\u003Ch3 id=\"versioned-prompts-and-a-fixed-set\" class=\"wiki-heading\">Versioned prompts and a fixed set\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"versioned-prompts-and-a-fixed-set\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A prompt is versioned with the code. Each run of the evaluation records everything that affects the answer: the prompt version, the model ID, the parameters, the retrieval index, the tool definitions and the graders. Without this record two runs cannot be compared.\u003C/p>\n\u003Cp>The "Demystifying evals" post separates two purposes. A capability evaluation measures what the application cannot yet do well, and its success rate starts low. A regression evaluation checks what already worked, and its success rate stays near 100 percent. Tasks that pass consistently move from the first set to the second.\u003C/p>\n\u003Ch3 id=\"comparing-runs\" class=\"wiki-heading\">Comparing runs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"comparing-runs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A change is accepted after a comparison of two runs on the same tasks, with the same graders and the same number of trials. The comparison is made task by task: a total that stays the same can hide ten tasks that broke and ten that improved. Tokens, latency and tool calls are compared too.\u003C/p>\n\u003Cp>Regression of the tools that an application uses through the Model Context Protocol is described in the article \u003Ca href=\"/en/wiki/mcp/mcp-server-regression-testing\" class=\"wiki-link wiki-link--article\">Regression testing of an MCP server\u003C/a>. Attacks through instructions hidden in input data are a separate kind of test, described in the article \u003Ca href=\"/en/wiki/ai-testing/prompt-injection\" class=\"wiki-link wiki-link--article\">Prompt injection\u003C/a>.\u003C/p>\n\u003Ch2 id=\"classic-software-and-llm-applications\" class=\"wiki-heading\">Classic software and LLM applications\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"classic-software-and-llm-applications\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table compares testing of classic software and of LLM applications.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>Classic software\u003C/th>\n\u003Cth>LLM application\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Output for the same input\u003C/td>\n\u003Ctd>Same\u003C/td>\n\u003Ctd>Varies between runs\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Expected result\u003C/td>\n\u003Ctd>Exact value\u003C/td>\n\u003Ctd>Label, facts, rubric, state\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Verdict of a test\u003C/td>\n\u003Ctd>Passed or failed\u003C/td>\n\u003Ctd>Share of successful trials\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Grader\u003C/td>\n\u003Ctd>Assertion in code\u003C/td>\n\u003Ctd>Code, model, human\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Cause of a regression\u003C/td>\n\u003Ctd>Change in code\u003C/td>\n\u003Ctd>Change in code, prompt, model, data\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Cost of a run\u003C/td>\n\u003Ctd>Seconds, no model\u003C/td>\n\u003Ctd>Minutes to hours, model usage\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The two kinds of testing are combined in one application. The deterministic parts are tested like any code on every change. The evaluation set runs on changes of the prompt, the model or the data and on a schedule.\u003C/p>\n\u003Cp>A non-deterministic result is not the same as a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"flaky-test\">flaky test\u003C/span>: a flaky test fails because of a defect in the test or the environment and is fixed, while the variation of a model's answers is a property of the system and is measured. Flaky tests are described in the article \u003Ca href=\"/en/wiki/test-automation/flaky-tests\" class=\"wiki-link wiki-link--article\">Flaky tests\u003C/a>.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>LLM application\u003C/td>\n\u003Ctd>Software in which a large language model produces part of the result\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Evaluation set (eval)\u003C/td>\n\u003Ctd>Fixed collection of tasks with expected outcomes, graders and metrics\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Temperature\u003C/td>\n\u003Ctd>Sampling parameter that controls how random the choice of the next token is\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>LLM-as-judge\u003C/td>\n\u003Ctd>Grading in which a language model scores answers against instructions\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Rubric\u003C/td>\n\u003Ctd>Set of criteria and allowed verdicts that a judge applies to an answer\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Position bias\u003C/td>\n\u003Ctd>Tendency of an LLM judge to favor an answer because of its position\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Cohen's kappa\u003C/td>\n\u003Ctd>Measure of agreement between two raters corrected for chance agreement\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Model snapshot\u003C/td>\n\u003Ctd>Fixed version of a model under its own identifier\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2306.05685\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, L. Zheng et al., arXiv:2306.05685, NeurIPS 2023\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://platform.claude.com/docs/en/test-and-evaluate/define-success\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Define success criteria and build evaluations, Claude Platform documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Demystifying evals for AI agents, Anthropic, 9 January 2026\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://platform.claude.com/docs/en/about-claude/glossary\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Glossary: Temperature, Claude Platform documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Model IDs and versions, Claude Platform documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://platform.claude.com/docs/en/about-claude/model-deprecations\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Model deprecations, Claude Platform documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://platform.claude.com/docs/en/build-with-claude/structured-outputs\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Structured outputs, Claude Platform documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Create chat completion, OpenAI API reference\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://developers.openai.com/api/docs/deprecations\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Deprecations, OpenAI API documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Defeating Nondeterminism in LLM Inference, H. He, Thinking Machines Lab, 10 September 2025\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2005.11401\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, P. Lewis et al., arXiv:2005.11401\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://doi.org/10.1177/001316446002000104\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">A Coefficient of Agreement for Nominal Scales, J. Cohen, Educational and Psychological Measurement, 1960\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://istqb.org/istqb-releases-certified-tester-ai-testing-v10-ct-ai/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB releases Certified Tester AI Testing (CT-AI), ISTQB, October 2021\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://python-jsonschema.readthedocs.io/en/stable/validate/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">jsonschema: Schema Validation, Python library documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87],{"id":11,"level":12,"text":13},"the-problem-it-solves",2,"The problem it solves",{"id":15,"level":12,"text":16},"parts-of-an-llm-application","Parts of an LLM application",{"id":18,"level":12,"text":19},"non-determinism","Non-determinism",{"id":21,"level":22,"text":23},"sampling-and-temperature",3,"Sampling and temperature",{"id":25,"level":22,"text":26},"seeds","Seeds",{"id":28,"level":22,"text":29},"causes-outside-sampling","Causes outside sampling",{"id":31,"level":22,"text":32},"what-is-deterministic","What is deterministic",{"id":34,"level":12,"text":35},"evaluation-sets","Evaluation sets",{"id":37,"level":22,"text":38},"success-criteria","Success criteria",{"id":40,"level":22,"text":41},"tasks-and-expected-outcomes","Tasks and expected outcomes",{"id":43,"level":22,"text":44},"graders","Graders",{"id":46,"level":22,"text":47},"metrics","Metrics",{"id":49,"level":12,"text":50},"deterministic-checks","Deterministic checks",{"id":52,"level":12,"text":53},"llm-as-judge","LLM-as-judge",{"id":55,"level":22,"text":56},"judge-types","Judge types",{"id":58,"level":22,"text":59},"known-biases","Known biases",{"id":61,"level":22,"text":62},"rubrics","Rubrics",{"id":64,"level":22,"text":65},"calibration-against-human-labels","Calibration against human labels",{"id":67,"level":12,"text":68},"human-review","Human review",{"id":70,"level":12,"text":71},"regression-after-a-model-or-prompt-change","Regression after a model or prompt change",{"id":73,"level":22,"text":74},"model-versions","Model versions",{"id":76,"level":22,"text":77},"versioned-prompts-and-a-fixed-set","Versioned prompts and a fixed set",{"id":79,"level":22,"text":80},"comparing-runs","Comparing runs",{"id":82,"level":12,"text":83},"classic-software-and-llm-applications","Classic software and LLM applications",{"id":85,"level":12,"text":86},"terms","Terms",{"id":88,"level":12,"text":89},"sources","Sources",2767,1793718935190]