[{"data":1,"prerenderedAt":79},["ShallowReactive",2],{"wiki-ai-testing-ai-test-generation-en":3},{"meta":4,"html":8,"headings":9,"words":78},{"title":5,"description":6,"published":7,"updated":7},"Generating test cases and automated tests with AI","AI test generation: inputs from requirements, UI, API specifications and code, known limits, the oracle problem, human review and verification by execution.","2026-11-03","\u003Cp>\u003Cstrong>AI test generation\u003C/strong> is the creation of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-case\">test cases\u003C/span>, automated test code or test data by a generative AI model, usually a large language model (LLM), from requirements, the user interface, API specifications or existing tests and code. The result is a draft. A person reviews it, and generated code is also built and executed, before the draft becomes part of a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-suite\">test suite\u003C/span>.\u003C/p>\n\u003Cp>The practice is described in the ISTQB syllabus Certified Tester Testing with Generative AI (CT-GenAI), a specialist-level certification introduced in July 2025. Version 1.0 of the syllabus is dated 25 July 2025; version 1.1, announced in April 2026, is a minor update with the same structure and learning objectives. Research on the subject is summarized in the survey "Software Testing with Large Language Models: Survey, Landscape, and Vision" (Wang et al., IEEE Transactions on Software Engineering, 2024). An industrial example is TestGen-LLM, a tool that Meta described in February 2024.\u003C/p>\n\u003Ch2 id=\"the-problem-it-solves\" class=\"wiki-heading\">The problem it solves\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-problem-it-solves\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Test design turns requirements into \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-condition\">test conditions\u003C/span>, test cases and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"expected-result\">expected results\u003C/span>. Test automation then turns test cases into code with \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"locator\">locators\u003C/span>, test data and assertions. Both steps are manual work that grows with the product, and both lag behind changes in requirements and in the interface.\u003C/p>\n\u003Cp>A language model produces text from text. It can draft steps from a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"user-story\">user story\u003C/span>, a test from an API description or a new \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"unit-test\">unit test\u003C/span> next to existing ones in minutes. The CT-GenAI syllabus lists such tasks: generation of test cases from functional and non-functional requirements, of synthetic test data and of automation scripts from structured test cases. The syllabus also states that generated output must be checked carefully, depending on the associated risk.\u003C/p>\n\u003Ch2 id=\"inputs\" class=\"wiki-heading\">Inputs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"inputs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The quality of a generated test depends on what the model receives. The table lists the usual inputs and what each gives the model.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-3\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Input\u003C/th>\n\u003Cth>What the model gets\u003C/th>\n\u003Cth>Limits\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Requirements, user stories, acceptance criteria\u003C/td>\n\u003Ctd>Expected behavior, rules, conditions\u003C/td>\n\u003Ctd>Ambiguities and gaps are copied or filled with guesses\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>User interface and its accessibility tree\u003C/td>\n\u003Ctd>Elements, roles, names, current texts\u003C/td>\n\u003Ctd>Shows one state of the page, not the rules behind it\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>API specification, for example OpenAPI\u003C/td>\n\u003Ctd>Operations, parameters, schemas, status codes\u003C/td>\n\u003Ctd>Describes the interface, not the business logic\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Existing tests and code\u003C/td>\n\u003Ctd>Style, fixtures, helpers, current behavior\u003C/td>\n\u003Ctd>Tests from code inherit the defects of the code\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The CT-GenAI syllabus names similar inputs: requirements, user stories, acceptance criteria, technical specifications, GUI wireframes, screenshots, code and existing test cases.\u003C/p>\n\u003Cp>The \u003Cstrong>accessibility tree\u003C/strong> is, in the definition of WAI-ARIA 1.2, a tree of accessible objects that represents the structure of the user interface. Each node has a role and an \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"accessible-name\">accessible name\u003C/span>, the same data that locators by role use, as described in the article \u003Ca href=\"/en/wiki/test-automation/ui-test-locators\" class=\"wiki-link wiki-link--article\">Locators in UI tests\u003C/a>. Tools that let a model drive a browser use this tree instead of screenshots: the Playwright \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"mcp-server\">MCP server\u003C/span>, for example, gives the model structured accessibility snapshots of the page.\u003C/p>\n\u003Cp>An OpenAPI document describes HTTP operations in a machine-readable form; the latest published version of the OpenAPI Specification is 3.2.1. A model receives exact paths, parameters and response schemas from it instead of guessing them. Expected values that depend on the data and the business rules are not in the specification.\u003C/p>\n\u003Ch2 id=\"outputs\" class=\"wiki-heading\">Outputs\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"outputs\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"test-cases\" class=\"wiki-heading\">Test cases\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-cases\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A generated test case has the same structure as a manual one: title, \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"precondition\">preconditions\u003C/span>, steps, expected results and a link to the requirement. The structure is described in the article \u003Ca href=\"/en/wiki/test-management/test-case\" class=\"wiki-link wiki-link--article\">Test case\u003C/a>. The example shows a draft as a model can return it; the field names are arbitrary.\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"json\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">JSON\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>{\n  "title": "Expired password forces a password change after sign-in",\n  "requirement": "AUTH-112",\n  "preconditions": ["User expired@example.com exists", "The password expired 3 days ago"],\n  "steps": [\n    { "action": "Open the sign-in page", "expected_result": "The sign-in form is shown" },\n    { "action": "Sign in as expired@example.com", "expected_result": "The Change password page is shown" },\n    { "action": "Open any other page", "expected_result": "The user is returned to Change password" }\n  ],\n  "status": "ai_generated_draft",\n  "generated_by": { "model": "model-a", "prompt_version": "testcase-from-story@4" }\n}\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>The \u003Ccode>status\u003C/code> field marks the test case as unreviewed. The \u003Ccode>generated_by\u003C/code> field records the model and the prompt version, so that drafts from a changed prompt can be found later.\u003C/p>\n\u003Ch3 id=\"automated-test-code\" class=\"wiki-heading\">Automated test code\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"automated-test-code\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Generated code includes unit tests, API tests and UI tests. The example is a UI test for the same requirement, written with Playwright Test.\u003C/p>\n\u003Cdiv class=\"wiki-code\" data-language=\"typescript\">\u003Cdiv class=\"wiki-code__header\">\u003Cspan class=\"wiki-code__language\">typescript\u003C/span>\u003Cbutton type=\"button\" class=\"wiki-code__copy\" data-wiki-copy data-copied-label=\"Copied\">\u003Csvg class=\"wiki-code__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M216,28H88A12,12,0,0,0,76,40V76H40A12,12,0,0,0,28,88V216a12,12,0,0,0,12,12H168a12,12,0,0,0,12-12V180h36a12,12,0,0,0,12-12V40A12,12,0,0,0,216,28ZM156,204H52V100H156Zm48-48H180V88a12,12,0,0,0-12-12H100V52H204Z\"/>\u003C/svg>\u003Csvg class=\"wiki-code__icon wiki-code__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003Cspan class=\"wiki-code__label\">Copy\u003C/span>\u003Cspan class=\"wiki-code__label wiki-code__label--done\">Copied\u003C/span>\u003C/button>\u003C/div>\u003Cpre>\u003Ccode>import { test, expect } from '@playwright/test';\n\ntest('expired password forces a password change', async ({ page }) => {\n  await page.goto('/login');\n  await page.getByLabel('Email').fill('expired@example.com');\n  await page.getByLabel('Password').fill('Old-password-1');\n  await page.getByRole('button', { name: 'Sign in' }).click();\n  await expect(page.getByRole('heading', { name: 'Change password' })).toBeVisible();\n  await expect(page.getByText('Your password expired 3 days ago')).toBeVisible();\n});\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Two questions decide whether the test is usable. Do the label "Email" and the button "Sign in" exist on the page? Is the text in the last assertion required by the product or only copied from its current state? The first question is answered by running the test, the second only by a person who knows the requirement.\u003C/p>\n\u003Ch3 id=\"test-data\" class=\"wiki-heading\">Test data\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"test-data\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The CT-GenAI syllabus describes generation of representative synthetic test data that preserves data privacy. A model produces records that follow a schema and cover boundary cases named in the prompt. The values are checked like any other output: against the schema, the business rules and the boundaries that the test needs.\u003C/p>\n\u003Ch2 id=\"evidence-from-practice-and-research\" class=\"wiki-heading\">Evidence from practice and research\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"evidence-from-practice-and-research\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"testgen-llm-at-meta\" class=\"wiki-heading\">TestGen-LLM at Meta\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"testgen-llm-at-meta\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The paper "Automated Unit Test Improvement using Large Language Models at Meta" (Alshahwan et al., FSE 2024) describes TestGen-LLM, a tool that adds test cases to existing Kotlin test classes. Each candidate passes a chain of filters:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Build.\u003C/strong> The test must build; code that does not build is discarded.\u003C/li>\n\u003Cli>\u003Cstrong>Pass.\u003C/strong> The test must pass; a failing test is discarded because the tool cannot tell a real \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"defect\">defect\u003C/span> from a wrong assertion.\u003C/li>\n\u003Cli>\u003Cstrong>Reliability.\u003C/strong> The test is run five times and must pass every time; otherwise it is treated as flaky.\u003C/li>\n\u003Cli>\u003Cstrong>Coverage.\u003C/strong> The test must increase line coverage over the existing tests.\u003C/li>\n\u003C/ul>\n\u003Cp>In the evaluation on Instagram Reels and Stories, 75 percent of the generated test cases built correctly, 57 percent passed reliably and 25 percent increased coverage. In test-a-thons on Instagram and Facebook the tool improved 11.5 percent of the classes it was applied to, and engineers accepted 73 percent of its recommendations for production. The authors call the approach assured LLM-based software engineering: the model proposes, and deterministic checks guarantee improvement and absence of regressions.\u003C/p>\n\u003Ch3 id=\"research-overview\" class=\"wiki-heading\">Research overview\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"research-overview\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The survey by Wang et al. analyzes 102 studies. It names test case preparation and program repair as the most representative uses of LLMs in testing; the tasks it covers include unit test case generation, test oracle generation and system test input generation. The survey lists four open challenges: high coverage, the test oracle problem, rigorous evaluation and real-world application.\u003C/p>\n\u003Ch2 id=\"known-limitations\" class=\"wiki-heading\">Known limitations\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"known-limitations\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"hallucinated-apis-packages-and-locators\" class=\"wiki-heading\">Hallucinated APIs, packages and locators\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"hallucinated-apis-packages-and-locators\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>The CT-GenAI syllabus defines hallucinations as output that appears factually incorrect or irrelevant to the task, and reasoning errors as misinterpretation of logical structures such as conditions or cause and effect. In generated tests both appear as references to things that do not exist: methods of a library, endpoints of an API, fields of a response, elements of a page.\u003C/p>\n\u003Cp>The paper "We Have a Package for You!" (Spracklen et al., USENIX Security 2025) measured one form of this. In 576,000 generated code samples from 16 models, on average at least 5.2 percent of the packages suggested by commercial models and 21.7 percent of those suggested by open-source models did not exist; there were 205,474 unique invented names. The authors describe package hallucination as a new form of package confusion attack on the software supply chain, so dependencies of generated tests are checked like any other dependencies.\u003C/p>\n\u003Cp>Locators are invented in the same way when the model has not seen the page. A test written from a user story alone guesses labels and names. A model that works with the accessibility tree of the running application selects existing names; the generator agent of Playwright, for example, verifies selectors and assertions on the live page while it writes the test.\u003C/p>\n\u003Ch3 id=\"the-oracle-problem\" class=\"wiki-heading\">The oracle problem\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"the-oracle-problem\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>A \u003Cstrong>test oracle\u003C/strong> is the source that decides whether the observed behavior is correct for a given input. The survey by Barr et al. (2015) describes the test oracle problem as the challenge of distinguishing the desired, correct behavior of a system from potentially incorrect behavior.\u003C/p>\n\u003Cp>Generation does not remove this problem; it moves it. A test generated from the code asserts what the code does. TestGen-LLM keeps only tests that pass, so the assertions of the kept tests describe the current behavior, correct or not. Such tests protect against future regressions but do not find existing defects. A test generated from a requirement has an independent oracle, but only as good as the requirement and its reading by the model.\u003C/p>\n\u003Cblockquote>\n\u003Cp>A generated test that passes on the first run shows that its assertions match the current behavior. It does not show that the current behavior is correct.\u003C/p>\n\u003C/blockquote>\n\u003Ch3 id=\"coverage-versus-meaning\" class=\"wiki-heading\">Coverage versus meaning\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"coverage-versus-meaning\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Coverage measures which code a test executes, not what it checks. A test that calls a function and asserts nothing increases line coverage. The TestGen-LLM paper notes that a test without assertions relies on an implicit oracle: the code must not throw an exception. Coverage is therefore read together with the assertions, and a generated test is reviewed for what it checks, not only for what it executes. The limits of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"requirements-coverage\">requirements coverage\u003C/span> are described in the article \u003Ca href=\"/en/wiki/test-management/requirements-traceability\" class=\"wiki-link wiki-link--article\">Requirements traceability\u003C/a>.\u003C/p>\n\u003Ch2 id=\"human-review\" class=\"wiki-heading\">Human review\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"human-review\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Ch3 id=\"review-statuses\" class=\"wiki-heading\">Review statuses\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"review-statuses\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Generated test cases enter the same life cycle as manual ones, with one extra status at the start. The life cycle is described in the article \u003Ca href=\"/en/wiki/test-management/test-case\" class=\"wiki-link wiki-link--article\">Test case\u003C/a>; the table shows a typical sequence for generated drafts.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Status\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>AI-generated draft\u003C/td>\n\u003Ctd>Created by a model, not yet read by a person, excluded from test runs\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>In review\u003C/td>\n\u003Ctd>A reviewer checks the draft against the requirement\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Active\u003C/td>\n\u003Ctd>Approved and used in test runs\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Rejected\u003C/td>\n\u003Ctd>Not suitable; kept or deleted by the rules of the team\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>The separate status keeps unreviewed drafts out of \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-run\">test runs\u003C/span> and reports. It also makes their share visible: how many drafts were approved, edited or rejected for each prompt version.\u003C/p>\n\u003Ch3 id=\"what-a-reviewer-checks\" class=\"wiki-heading\">What a reviewer checks\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"what-a-reviewer-checks\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cul>\n\u003Cli>\u003Cstrong>Traceability.\u003C/strong> Each test case refers to a real requirement and checks what it says.\u003C/li>\n\u003Cli>\u003Cstrong>Expected results.\u003C/strong> Each expected result follows from the requirement, not from the current behavior.\u003C/li>\n\u003Cli>\u003Cstrong>Completeness.\u003C/strong> Negative cases and boundaries are present; test design techniques show which are missing.\u003C/li>\n\u003Cli>\u003Cstrong>Duplicates.\u003C/strong> The draft does not repeat an existing test case.\u003C/li>\n\u003Cli>\u003Cstrong>Data.\u003C/strong> Test data contains no real personal data.\u003C/li>\n\u003C/ul>\n\u003Ch3 id=\"verification-by-execution\" class=\"wiki-heading\">Verification by execution\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"verification-by-execution\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h3>\n\u003Cp>Generated code is executed before review, following the filters of TestGen-LLM: it must build, pass and pass repeatedly. A test that fails on its first run is examined, because the failure can be a defect of the product or of the test. A test that passes is reviewed for its assertions. Repeated runs detect \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"flaky-test\">flaky tests\u003C/span>, described in the article \u003Ca href=\"/en/wiki/test-automation/flaky-tests\" class=\"wiki-link wiki-link--article\">Flaky tests\u003C/a>; running generated tests in a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"ci-pipeline\">pipeline\u003C/span> is described in the article \u003Ca href=\"/en/wiki/test-automation/automated-tests-in-ci\" class=\"wiki-link wiki-link--article\">Automated tests in CI\u003C/a>.\u003C/p>\n\u003Ch2 id=\"data-privacy\" class=\"wiki-heading\">Data privacy\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"data-privacy\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>Requirements, test cases and code sent to a hosted model leave the organization. The CT-GenAI syllabus names the risks: unintentional exposure of sensitive data, lack of control over how data is used, and compliance risks under regulations such as the GDPR.\u003C/p>\n\u003Cp>The conditions are set by the terms of the model provider. OpenAI, for example, states that data sent to its API is not used to train its models unless the customer opts in, and that abuse monitoring logs are kept for up to 30 days by default; zero data retention is available subject to approval. A model run on the organization's own infrastructure keeps the data inside it. In either case personal data is removed from requirements and replaced with synthetic data before it is sent.\u003C/p>\n\u003Ch2 id=\"access-to-test-data-through-mcp\" class=\"wiki-heading\">Access to test data through MCP\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"access-to-test-data-through-mcp\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>A model generates better drafts when it can read existing test cases, requirements and results, and when it can write drafts directly where they are kept. The Model Context Protocol (MCP), described in the article \u003Ca href=\"/en/wiki/mcp/what-is-mcp\" class=\"wiki-link wiki-link--article\">What is MCP\u003C/a>, provides this: a \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-management-system\">test management system\u003C/span> offers an MCP server whose tools search test cases, read requirements and create drafts.\u003C/p>\n\u003Cp>Such access is limited like any other access. A token with read access and the right to create drafts is enough for generation. Changing statuses and deleting stay with people. Instructions hidden in the data that a model reads are a known risk, described in the articles \u003Ca href=\"/en/wiki/mcp/mcp-security\" class=\"wiki-link wiki-link--article\">MCP security\u003C/a> and \u003Ca href=\"/en/wiki/ai-testing/prompt-injection\" class=\"wiki-link wiki-link--article\">Prompt injection\u003C/a>.\u003C/p>\n\u003Ch2 id=\"ai-generation-test-design-techniques-and-record-and-playback\" class=\"wiki-heading\">AI generation, test design techniques and record-and-playback\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"ai-generation-test-design-techniques-and-record-and-playback\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cp>The table compares three ways to obtain tests.\u003C/p>\n\u003Cdiv class=\"wiki-table wiki-table--cols-4\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C/th>\n\u003Cth>AI generation\u003C/th>\n\u003Cth>Test design techniques\u003C/th>\n\u003Cth>Record-and-playback\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Source of tests\u003C/td>\n\u003Ctd>Requirements, UI, specifications, code\u003C/td>\n\u003Ctd>Requirements and models of behavior\u003C/td>\n\u003Ctd>Actions of a person in the interface\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Who chooses the cases\u003C/td>\n\u003Ctd>The model, then a reviewer\u003C/td>\n\u003Ctd>The test designer, by a technique\u003C/td>\n\u003Ctd>The person who records\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Coverage criterion\u003C/td>\n\u003Ctd>None of its own\u003C/td>\n\u003Ctd>Defined by the technique\u003C/td>\n\u003Ctd>None\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Expected results\u003C/td>\n\u003Ctd>Proposed by the model\u003C/td>\n\u003Ctd>Derived from the requirement\u003C/td>\n\u003Ctd>Captured from the current state\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Repeatability\u003C/td>\n\u003Ctd>Different output on each run\u003C/td>\n\u003Ctd>Same cases from the same requirements\u003C/td>\n\u003Ctd>Exact replay of the recording\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Cp>Test design techniques, such as \u003Ca href=\"/en/wiki/test-design/equivalence-partitioning-and-boundary-values\" class=\"wiki-link wiki-link--article\">equivalence partitioning and boundary values\u003C/a>, \u003Ca href=\"/en/wiki/test-design/decision-table-testing\" class=\"wiki-link wiki-link--article\">decision tables\u003C/a> and \u003Ca href=\"/en/wiki/test-design/state-transition-testing\" class=\"wiki-link wiki-link--article\">state transitions\u003C/a>, define which cases are needed and when the set is complete. AI generation has no such criterion. The two are combined: the technique defines the cases, and the model writes their steps and code, or the technique serves the reviewer as a check of completeness.\u003C/p>\n\u003Cp>Record-and-playback tools, such as Selenium IDE or Playwright codegen, record the actions of a person and turn them into a script; codegen prefers locators by role, text and \u003Cspan class=\"wiki-term\" tabindex=\"0\" data-wiki-term=\"test-id\">test id\u003C/span>. A recording captures one path and its current results, so it shares the oracle problem of generation from code. Its place in a test suite is discussed in the article \u003Ca href=\"/en/wiki/test-automation/automation-levels-and-test-pyramid\" class=\"wiki-link wiki-link--article\">Test automation levels and the test pyramid\u003C/a>.\u003C/p>\n\u003Ch2 id=\"terms\" class=\"wiki-heading\">Terms\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"terms\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cdiv class=\"wiki-table wiki-table--cols-2\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C/th>\n\u003Cth>Meaning\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>AI test generation\u003C/td>\n\u003Ctd>Creation of test cases, test code or test data by a generative AI model\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Hallucination\u003C/td>\n\u003Ctd>Model output that appears factually incorrect or irrelevant to the task\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Accessibility tree\u003C/td>\n\u003Ctd>Tree of accessible objects that represents the structure of the user interface\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Test oracle\u003C/td>\n\u003Ctd>Source that decides whether the observed behavior is correct for a given input\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Oracle problem\u003C/td>\n\u003Ctd>Difficulty of distinguishing correct from incorrect behavior of a system\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>AI-generated draft\u003C/td>\n\u003Ctd>Status of a test case created by a model and not yet reviewed by a person\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\u003C/div>\n\u003Ch2 id=\"sources\" class=\"wiki-heading\">Sources\u003Cbutton type=\"button\" class=\"wiki-heading__anchor\" data-wiki-anchor=\"sources\" aria-label=\"Copy link to section\" data-copied-label=\"Link to section copied\">\u003Csvg class=\"wiki-heading__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M117.18,188.74a12,12,0,0,1,0,17l-5.12,5.12A58.26,58.26,0,0,1,70.6,228h0A58.62,58.62,0,0,1,29.14,127.92L63.89,93.17a58.64,58.64,0,0,1,98.56,28.11,12,12,0,1,1-23.37,5.44,34.65,34.65,0,0,0-58.22-16.58L46.11,144.89A34.62,34.62,0,0,0,70.57,204h0a34.41,34.41,0,0,0,24.49-10.14l5.11-5.12A12,12,0,0,1,117.18,188.74ZM226.83,45.17a58.65,58.65,0,0,0-82.93,0l-5.11,5.11a12,12,0,0,0,17,17l5.12-5.12a34.63,34.63,0,1,1,49,49L175.1,145.86A34.39,34.39,0,0,1,150.61,156h0a34.63,34.63,0,0,1-33.69-26.72,12,12,0,0,0-23.38,5.44A58.64,58.64,0,0,0,150.56,180h.05a58.28,58.28,0,0,0,41.47-17.17l34.75-34.75a58.62,58.62,0,0,0,0-82.91Z\"/>\u003C/svg>\u003Csvg class=\"wiki-heading__icon wiki-heading__icon--done\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M232.49,80.49l-128,128a12,12,0,0,1-17,0l-56-56a12,12,0,1,1,17-17L96,183,215.51,63.51a12,12,0,0,1,17,17Z\"/>\u003C/svg>\u003C/button>\u003C/h2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://istqb.org/certifications/gen-ai/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Certified Tester Testing with Generative AI (CT-GenAI), ISTQB\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://atsqa.org/assets/documents/CT-GenAI-Syllabus-v1.0.pdf\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">CT-GenAI Syllabus v1.0, ISTQB, 2025, copy on the ATSQA website\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://istqb.org/istqb-announces-minor-update-to-certified-tester-testing-with-generative-ai-ct-genai/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">ISTQB announces minor update to CT-GenAI, ISTQB, April 2026\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2402.09171\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Automated Unit Test Improvement using Large Language Models at Meta, N. Alshahwan et al., arXiv:2402.09171\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2307.07221\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Software Testing with Large Language Models: Survey, Landscape, and Vision, J. Wang et al., arXiv:2307.07221\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://arxiv.org/abs/2406.10279\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs, J. Spracklen et al., arXiv:2406.10279\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://discovery.ucl.ac.uk/1471263/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">The Oracle Problem in Software Testing: A Survey, E. T. Barr et al., IEEE Transactions on Software Engineering, 2015\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.w3.org/TR/wai-aria-1.2/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Accessible Rich Internet Applications (WAI-ARIA) 1.2, W3C\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://spec.openapis.org/oas/latest.html\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">OpenAPI Specification, latest version\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://github.com/microsoft/playwright-mcp\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Playwright MCP, Microsoft, GitHub\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/test-agents\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Playwright Test Agents, Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://playwright.dev/docs/codegen\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Test generator (codegen), Playwright documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.selenium.dev/selenium-ide/\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Selenium IDE, Selenium project\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://developers.openai.com/api/docs/guides/your-data\" target=\"_blank\" rel=\"noopener\" class=\"wiki-link wiki-link--external\">Data controls in the OpenAI platform, OpenAI API documentation\u003Csvg class=\"wiki-link__icon\" viewBox=\"0 0 256 256\" width=\"16\" height=\"16\" fill=\"currentColor\" aria-hidden=\"true\" focusable=\"false\">\u003Cpath d=\"M228,104a12,12,0,0,1-24,0V69l-59.51,59.51a12,12,0,0,1-17-17L187,52H152a12,12,0,0,1,0-24h64a12,12,0,0,1,12,12Zm-44,24a12,12,0,0,0-12,12v64H52V84h64a12,12,0,0,0,0-24H48A20,20,0,0,0,28,80V208a20,20,0,0,0,20,20H176a20,20,0,0,0,20-20V140A12,12,0,0,0,184,128Z\"/>\u003C/svg>\u003Cspan class=\"wiki-visually-hidden\">(opens in a new tab)\u003C/span>\u003C/a>\u003C/li>\n\u003C/ul>\n",[10,14,17,20,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75],{"id":11,"level":12,"text":13},"the-problem-it-solves",2,"The problem it solves",{"id":15,"level":12,"text":16},"inputs","Inputs",{"id":18,"level":12,"text":19},"outputs","Outputs",{"id":21,"level":22,"text":23},"test-cases",3,"Test cases",{"id":25,"level":22,"text":26},"automated-test-code","Automated test code",{"id":28,"level":22,"text":29},"test-data","Test data",{"id":31,"level":12,"text":32},"evidence-from-practice-and-research","Evidence from practice and research",{"id":34,"level":22,"text":35},"testgen-llm-at-meta","TestGen-LLM at Meta",{"id":37,"level":22,"text":38},"research-overview","Research overview",{"id":40,"level":12,"text":41},"known-limitations","Known limitations",{"id":43,"level":22,"text":44},"hallucinated-apis-packages-and-locators","Hallucinated APIs, packages and locators",{"id":46,"level":22,"text":47},"the-oracle-problem","The oracle problem",{"id":49,"level":22,"text":50},"coverage-versus-meaning","Coverage versus meaning",{"id":52,"level":12,"text":53},"human-review","Human review",{"id":55,"level":22,"text":56},"review-statuses","Review statuses",{"id":58,"level":22,"text":59},"what-a-reviewer-checks","What a reviewer checks",{"id":61,"level":22,"text":62},"verification-by-execution","Verification by execution",{"id":64,"level":12,"text":65},"data-privacy","Data privacy",{"id":67,"level":12,"text":68},"access-to-test-data-through-mcp","Access to test data through MCP",{"id":70,"level":12,"text":71},"ai-generation-test-design-techniques-and-record-and-playback","AI generation, test design techniques and record-and-playback",{"id":73,"level":12,"text":74},"terms","Terms",{"id":76,"level":12,"text":77},"sources","Sources",2413,1793718940877]