AI in testing
Testing applications built on language models, generating test cases and automated tests with AI, prompt injection.
3 articles in reading order
- Testing LLM applicationsHow to test applications built on language models: non-determinism, evaluation sets (evals), LLM-as-judge and regressions after a model or prompt change.
- Generating test cases and automated tests with AIAI test generation: inputs from requirements, UI, API specifications and code, known limits, the oracle problem, human review and verification by execution.
- Prompt injectionPrompt injection: direct and indirect injection, injection through tool results, jailbreaking, OWASP LLM01, how to test it, mitigations and their limits.