Generating test cases and automated tests with AI

AI test generation is the creation of test cases, automated test code or test data by a generative AI model, usually a large language model (LLM), from requirements, the user interface, API specifications or existing tests and code. The result is a draft. A person reviews it, and generated code is also built and executed, before the draft becomes part of a test suite.

The practice is described in the ISTQB syllabus Certified Tester Testing with Generative AI (CT-GenAI), a specialist-level certification introduced in July 2025. Version 1.0 of the syllabus is dated 25 July 2025; version 1.1, announced in April 2026, is a minor update with the same structure and learning objectives. Research on the subject is summarized in the survey "Software Testing with Large Language Models: Survey, Landscape, and Vision" (Wang et al., IEEE Transactions on Software Engineering, 2024). An industrial example is TestGen-LLM, a tool that Meta described in February 2024.

The problem it solves

Test design turns requirements into test conditions, test cases and expected results. Test automation then turns test cases into code with locators, test data and assertions. Both steps are manual work that grows with the product, and both lag behind changes in requirements and in the interface.

A language model produces text from text. It can draft steps from a user story, a test from an API description or a new unit test next to existing ones in minutes. The CT-GenAI syllabus lists such tasks: generation of test cases from functional and non-functional requirements, of synthetic test data and of automation scripts from structured test cases. The syllabus also states that generated output must be checked carefully, depending on the associated risk.

Inputs

The quality of a generated test depends on what the model receives. The table lists the usual inputs and what each gives the model.

Input	What the model gets	Limits
Requirements, user stories, acceptance criteria	Expected behavior, rules, conditions	Ambiguities and gaps are copied or filled with guesses
User interface and its accessibility tree	Elements, roles, names, current texts	Shows one state of the page, not the rules behind it
API specification, for example OpenAPI	Operations, parameters, schemas, status codes	Describes the interface, not the business logic
Existing tests and code	Style, fixtures, helpers, current behavior	Tests from code inherit the defects of the code

The CT-GenAI syllabus names similar inputs: requirements, user stories, acceptance criteria, technical specifications, GUI wireframes, screenshots, code and existing test cases.

The accessibility tree is, in the definition of WAI-ARIA 1.2, a tree of accessible objects that represents the structure of the user interface. Each node has a role and an accessible name, the same data that locators by role use, as described in the article Locators in UI tests. Tools that let a model drive a browser use this tree instead of screenshots: the Playwright MCP server, for example, gives the model structured accessibility snapshots of the page.

An OpenAPI document describes HTTP operations in a machine-readable form; the latest published version of the OpenAPI Specification is 3.2.1. A model receives exact paths, parameters and response schemas from it instead of guessing them. Expected values that depend on the data and the business rules are not in the specification.

Outputs

Test cases

A generated test case has the same structure as a manual one: title, preconditions, steps, expected results and a link to the requirement. The structure is described in the article Test case. The example shows a draft as a model can return it; the field names are arbitrary.

{
  "title": "Expired password forces a password change after sign-in",
  "requirement": "AUTH-112",
  "preconditions": ["User expired@example.com exists", "The password expired 3 days ago"],
  "steps": [
    { "action": "Open the sign-in page", "expected_result": "The sign-in form is shown" },
    { "action": "Sign in as expired@example.com", "expected_result": "The Change password page is shown" },
    { "action": "Open any other page", "expected_result": "The user is returned to Change password" }
  ],
  "status": "ai_generated_draft",
  "generated_by": { "model": "model-a", "prompt_version": "testcase-from-story@4" }
}

The status field marks the test case as unreviewed. The generated_by field records the model and the prompt version, so that drafts from a changed prompt can be found later.

Automated test code

Generated code includes unit tests, API tests and UI tests. The example is a UI test for the same requirement, written with Playwright Test.

import { test, expect } from '@playwright/test';

test('expired password forces a password change', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('expired@example.com');
  await page.getByLabel('Password').fill('Old-password-1');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Change password' })).toBeVisible();
  await expect(page.getByText('Your password expired 3 days ago')).toBeVisible();
});

Two questions decide whether the test is usable. Do the label "Email" and the button "Sign in" exist on the page? Is the text in the last assertion required by the product or only copied from its current state? The first question is answered by running the test, the second only by a person who knows the requirement.

Test data

The CT-GenAI syllabus describes generation of representative synthetic test data that preserves data privacy. A model produces records that follow a schema and cover boundary cases named in the prompt. The values are checked like any other output: against the schema, the business rules and the boundaries that the test needs.

Evidence from practice and research

TestGen-LLM at Meta

The paper "Automated Unit Test Improvement using Large Language Models at Meta" (Alshahwan et al., FSE 2024) describes TestGen-LLM, a tool that adds test cases to existing Kotlin test classes. Each candidate passes a chain of filters:

Build. The test must build; code that does not build is discarded.
Pass. The test must pass; a failing test is discarded because the tool cannot tell a real defect from a wrong assertion.
Reliability. The test is run five times and must pass every time; otherwise it is treated as flaky.
Coverage. The test must increase line coverage over the existing tests.

In the evaluation on Instagram Reels and Stories, 75 percent of the generated test cases built correctly, 57 percent passed reliably and 25 percent increased coverage. In test-a-thons on Instagram and Facebook the tool improved 11.5 percent of the classes it was applied to, and engineers accepted 73 percent of its recommendations for production. The authors call the approach assured LLM-based software engineering: the model proposes, and deterministic checks guarantee improvement and absence of regressions.

Research overview

The survey by Wang et al. analyzes 102 studies. It names test case preparation and program repair as the most representative uses of LLMs in testing; the tasks it covers include unit test case generation, test oracle generation and system test input generation. The survey lists four open challenges: high coverage, the test oracle problem, rigorous evaluation and real-world application.

Known limitations

Hallucinated APIs, packages and locators

The CT-GenAI syllabus defines hallucinations as output that appears factually incorrect or irrelevant to the task, and reasoning errors as misinterpretation of logical structures such as conditions or cause and effect. In generated tests both appear as references to things that do not exist: methods of a library, endpoints of an API, fields of a response, elements of a page.

The paper "We Have a Package for You!" (Spracklen et al., USENIX Security 2025) measured one form of this. In 576,000 generated code samples from 16 models, on average at least 5.2 percent of the packages suggested by commercial models and 21.7 percent of those suggested by open-source models did not exist; there were 205,474 unique invented names. The authors describe package hallucination as a new form of package confusion attack on the software supply chain, so dependencies of generated tests are checked like any other dependencies.

Locators are invented in the same way when the model has not seen the page. A test written from a user story alone guesses labels and names. A model that works with the accessibility tree of the running application selects existing names; the generator agent of Playwright, for example, verifies selectors and assertions on the live page while it writes the test.

The oracle problem

A test oracle is the source that decides whether the observed behavior is correct for a given input. The survey by Barr et al. (2015) describes the test oracle problem as the challenge of distinguishing the desired, correct behavior of a system from potentially incorrect behavior.

Generation does not remove this problem; it moves it. A test generated from the code asserts what the code does. TestGen-LLM keeps only tests that pass, so the assertions of the kept tests describe the current behavior, correct or not. Such tests protect against future regressions but do not find existing defects. A test generated from a requirement has an independent oracle, but only as good as the requirement and its reading by the model.

A generated test that passes on the first run shows that its assertions match the current behavior. It does not show that the current behavior is correct.

Coverage versus meaning

Coverage measures which code a test executes, not what it checks. A test that calls a function and asserts nothing increases line coverage. The TestGen-LLM paper notes that a test without assertions relies on an implicit oracle: the code must not throw an exception. Coverage is therefore read together with the assertions, and a generated test is reviewed for what it checks, not only for what it executes. The limits of requirements coverage are described in the article Requirements traceability.

Human review

Review statuses

Generated test cases enter the same life cycle as manual ones, with one extra status at the start. The life cycle is described in the article Test case; the table shows a typical sequence for generated drafts.

Status	Meaning
AI-generated draft	Created by a model, not yet read by a person, excluded from test runs
In review	A reviewer checks the draft against the requirement
Active	Approved and used in test runs
Rejected	Not suitable; kept or deleted by the rules of the team

The separate status keeps unreviewed drafts out of test runs and reports. It also makes their share visible: how many drafts were approved, edited or rejected for each prompt version.

What a reviewer checks

Traceability. Each test case refers to a real requirement and checks what it says.
Expected results. Each expected result follows from the requirement, not from the current behavior.
Completeness. Negative cases and boundaries are present; test design techniques show which are missing.
Duplicates. The draft does not repeat an existing test case.
Data. Test data contains no real personal data.

Verification by execution

Generated code is executed before review, following the filters of TestGen-LLM: it must build, pass and pass repeatedly. A test that fails on its first run is examined, because the failure can be a defect of the product or of the test. A test that passes is reviewed for its assertions. Repeated runs detect flaky tests, described in the article Flaky tests; running generated tests in a pipeline is described in the article Automated tests in CI.

Data privacy

Requirements, test cases and code sent to a hosted model leave the organization. The CT-GenAI syllabus names the risks: unintentional exposure of sensitive data, lack of control over how data is used, and compliance risks under regulations such as the GDPR.

The conditions are set by the terms of the model provider. OpenAI, for example, states that data sent to its API is not used to train its models unless the customer opts in, and that abuse monitoring logs are kept for up to 30 days by default; zero data retention is available subject to approval. A model run on the organization's own infrastructure keeps the data inside it. In either case personal data is removed from requirements and replaced with synthetic data before it is sent.

Access to test data through MCP

A model generates better drafts when it can read existing test cases, requirements and results, and when it can write drafts directly where they are kept. The Model Context Protocol (MCP), described in the article What is MCP, provides this: a test management system offers an MCP server whose tools search test cases, read requirements and create drafts.

Such access is limited like any other access. A token with read access and the right to create drafts is enough for generation. Changing statuses and deleting stay with people. Instructions hidden in the data that a model reads are a known risk, described in the articles MCP security and Prompt injection.

AI generation, test design techniques and record-and-playback

The table compares three ways to obtain tests.

	AI generation	Test design techniques	Record-and-playback
Source of tests	Requirements, UI, specifications, code	Requirements and models of behavior	Actions of a person in the interface
Who chooses the cases	The model, then a reviewer	The test designer, by a technique	The person who records
Coverage criterion	None of its own	Defined by the technique	None
Expected results	Proposed by the model	Derived from the requirement	Captured from the current state
Repeatability	Different output on each run	Same cases from the same requirements	Exact replay of the recording

Test design techniques, such as equivalence partitioning and boundary values, decision tables and state transitions, define which cases are needed and when the set is complete. AI generation has no such criterion. The two are combined: the technique defines the cases, and the model writes their steps and code, or the technique serves the reviewer as a check of completeness.

Record-and-playback tools, such as Selenium IDE or Playwright codegen, record the actions of a person and turn them into a script; codegen prefers locators by role, text and test id. A recording captures one path and its current results, so it shares the oracle problem of generation from code. Its place in a test suite is discussed in the article Test automation levels and the test pyramid.

Terms

Term	Meaning
AI test generation	Creation of test cases, test code or test data by a generative AI model
Hallucination	Model output that appears factually incorrect or irrelevant to the task
Accessibility tree	Tree of accessible objects that represents the structure of the user interface
Test oracle	Source that decides whether the observed behavior is correct for a given input
Oracle problem	Difficulty of distinguishing correct from incorrect behavior of a system
AI-generated draft	Status of a test case created by a model and not yet reviewed by a person