How to Write Tests for AI Applications: Test Pyramid, Golden Dataset and CI/CD for LLMs

Testing AI applications requires different tools from classical code - an LLM is non-deterministic, so instead of checking string equality you test semantics, behaviour and edge-case coverage. A properly designed test suite gives you confidence before deployment: semantic tests check whether the response is meaningfully correct, contract-schema tests verify output format, and regression tests detect whether a model or prompt update has broken existing behaviour.

How to design a test suite for LLM applications: unit tests with mocking, integration tests with a golden dataset, LLM-as-judge evaluation and a quality gate in CI/CD. With real Python code and GitHub Actions.

You change a prompt from "reply in Polish" to "always respond in the Polish language". Tests pass - they were written for the old version. Three weeks later a client calls: "your app started responding in English for some users." It turns out a model update changed default behaviour and the new prompt stopped working for edge cases. You had no regression tests, so you had no idea.

Testing AI applications is a different problem from testing classic code. LLM output is non-deterministic - identical input can produce different output on every call. But you can design a test suite that gives you confidence before deployment.

Why won't classic assertEquals work for LLMs?

Classic tests check equality: assert result == expected. For LLMs this makes no sense - the model says the same thing in different words every time.

Three fundamental differences: - Non-determinism - temperature > 0 produces different output on every call, even with identical input - Semantics instead of syntax - "You have 14 days" and "The return window is two weeks" are the same answer - assertEqual rejects both versions as wrong - No oracle - there is no single "correct" answer the way there is for a sorting algorithm or a parser

Result: tests that look like tests but don't catch real regressions.

What to test - and what not to?

Test:

• the logic of prompt construction - deterministic, testable like any other code
• response parsing correctness - does the JSON have the required fields and correct types?
• output format - does the result fit within character limits, does it have the required structure?
• semantic quality criteria - faithfulness, relevance - on a sample of 5-10% of calls
• semantic regression against a baseline after every model update

Don't test:

• the base model itself - OpenAI, Anthropic and Google test that for you
• exact content word-for-word - the model will rephrase and the test will fail
• model performance on general benchmarks - that's not your code
• internal framework mechanics (LangChain, LlamaIndex) - you'd be testing their code, not yours

Layer 1: Unit tests - mock the model, test the code

Unit tests for AI do not call the model. They test application logic around the model: prompt construction, response parsing, error handling. Use unittest.mock - the framework returns a predefined response. Tests are fast (< 5 seconds), free and fully deterministic.

# test_unit.pyfrom unittest.mock import patch, MagicMockimport pytestfrom myapp.rag_chain import build_prompt, RAGChaindef test_prompt_contains_context():    context = "Returns policy: 14 days no questions asked."    question = "How long do I have to return?"    messages = build_prompt(question, context)    user_content = " ".join(m["content"] for m in messages if m["role"] == "user")    assert "14 days" in user_content    assert question in user_contentdef test_prompt_has_system_role():    messages = build_prompt("test?", "ctx")    assert messages[0]["role"] == "system"    assert "assistant" in messages[0]["content"].lower()@patch("myapp.rag_chain.client.chat.completions.create")def test_chain_passes_context_to_model(mock_create):    mock_create.return_value = MagicMock(        choices=[MagicMock(message=MagicMock(content="You have 14 days to return."))]    )    chain = RAGChain()    result = chain.run("How long do I have to return?", context="14 days")    assert "14 days" in result    msgs = mock_create.call_args[1]["messages"]    assert any("14 days" in str(m) for m in msgs)@patch("myapp.rag_chain.client.chat.completions.create")def test_chain_raises_on_empty_context(mock_create):    chain = RAGChain()    with pytest.raises(ValueError, match="empty context"):        chain.run("question", context="")

Run: pytest tests/unit/ -v - zero API costs, results in < 5 seconds.

Layer 2: Integration tests - golden dataset as the oracle

Integration tests call the real model with a set of golden test cases. A golden dataset is a collection of input → quality criterion examples - created once, version-controlled in the repository, extended after every production bug.

In short: start with 30-50 cases from logs and reported bugs, add edge cases, store a criterion rather than an exact answer ("must contain the number 14"), keep the set in tests/fixtures/golden.json and add a new case after every production bug. Over time the set grows to 200-500 examples. How to build it from real failures and why not to generate it with a model is covered in detail in the LLM evaluation guide. # test_integration.py import json, pytest from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric from myapp.rag_chain import RAGChain @pytest.fixture(scope="session") def chain(): return RAGChain() @pytest.mark.parametrize("case", [ {"q": "How many days to return?", "ctx": "Returns: 14 days.", "exp": "14"}, {"q": "Can I return a used product?", "ctx": "New products only.", "exp": "new"}, {"q": "Return shipping cost?", "ctx": "Free returns on orders over $50.", "exp": "50"}, ]) def test_golden_set(case, chain): answer = chain.run(case["q"], context=case["ctx"]) test_case = LLMTestCase( input=case["q"], actual_output=answer, retrieval_context=[case["ctx"]] ) assert_test(test_case, [ AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.8) ]) assert case["exp"].lower() in answer.lower()

Cost: 100 cases × about 500 tokens is about 50k input tokens, so roughly $0.005 on a cheap model (GPT-6 Luna, $0.10 per million) and about $0.10 on a mid-tier model (GPT-6.1 Sol, $2 per million), plus the cost of model-graded metrics. Still pennies compared with one bug in production.

Layer 3: Evaluation - when the criterion is "good", not "correct"

For complex outputs (summaries, generated content, multi-turn conversations) we use LLM-as-a-judge: a second model evaluates output quality against criteria. DeepEval has the G-Eval metric for this, where you describe the criterion in plain language. The judge should come from a different model family than the model being evaluated, because models overrate their own answers.

Here I only show how to plug a judge into your tests. How to write the rubric, calibrate the judge against human ratings and limit its biases is covered in the post on LLM evaluation and LLM-as-a-judge.

# test_eval.pyimport pytestfrom deepeval import assert_testfrom deepeval.test_case import LLMTestCase, LLMTestCaseParamsfrom deepeval.metrics import GEvalfrom myapp.summarizer import summarize_documentACCURACY = GEval(    name="Accuracy",    criteria="Is the summary factually consistent with the document and free of information not present in it?",    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],    threshold=0.7)CONCISENESS = GEval(    name="Conciseness",    criteria="Is the summary concise and free of repeated information?",    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],    threshold=0.6)@pytest.mark.parametrize("doc,min_len,max_len", [    ("Q1 report: revenue up 15%. Costs stable. Margin 32%.", 20, 200),    ("Returns policy: 14 days, new products only, shipping at customer expense.", 15, 150),])def test_summarizer_quality(doc, min_len, max_len):    summary = summarize_document(doc)    assert min_len <= len(summary) <= max_len    assert_test(        LLMTestCase(input=doc, actual_output=summary),        [ACCURACY, CONCISENESS]    )

How to configure CI/CD - a quality gate before deployment?

Goal: every PR that changes prompts or AI logic must pass a quality evaluation. If results fall below the threshold - merge is automatically blocked.

Two stages in the pipeline: - unit tests - run on every PR (< 30 seconds, $0 API cost) - eval gate - runs only when changed files match AI paths (a few minutes, ~$0.01)

name: AI Evaluation Gateon:  pull_request:    paths:      - 'myapp/prompts/**'      - 'myapp/chain.py'      - 'tests/fixtures/**'jobs:  unit-tests:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - uses: actions/setup-python@v5        with: {python-version: '3.12'}      - run: pip install -r requirements.txt      - run: pytest tests/unit/ -v --tb=short  eval-gate:    needs: unit-tests    runs-on: ubuntu-latest    env:      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}    steps:      - uses: actions/checkout@v4      - uses: actions/setup-python@v5        with: {python-version: '3.12'}      - run: pip install -r requirements.txt deepeval      - run: deepeval test run tests/integration/ --min-success-rate 0.85      - run: python tests/regression/check_baseline.py --threshold 0.80

Evaluation results appear directly in PR comments - the developer sees which metrics failed without digging through CI logs.

How to detect regression after a model update?

Model aliases (for example versions marked "latest") point to newer versions over time, and older versions are retired on the vendor's schedule. To know when something has changed, you need a baseline - saved evaluation results on the golden dataset tied to a specific model version. That's why in production you pin a specific model version and treat a change like any other deployment.

1. 1.After every deployment save results as baseline_vN.json with the date and model version
2. 2.After any suspected model update run the eval again and compare against the baseline
3. 3.Alert when faithfulness or relevance drops by more than 5 percentage points
4. 4.If regression is confirmed - roll back to the previous prompt version or block the model update

Tools - what to choose and when?

Common mistakes - what not to do

• Testing the base model instead of your own code - you're checking the model's intelligence, not your code. Test only the logic you wrote yourself
• One giant test instead of a pyramid - if every test calls LLM-as-judge, you pay $0.05 per commit and CI takes 20 minutes
• Golden dataset without edge cases - 50 happy path examples will catch 30% of bugs. Add 20% hard questions and 20% out-of-scope cases
• No dataset versioning - a changed golden.json without code review is a recipe for false positives
• Exact content comparison - assert result == "14 days" fails when the model says "two weeks". Use assert "14" in result or DeepEval metrics
• Ignoring evaluation cost - 1000 cases × LLM-as-judge = $1-5 per run. Budget it and run full evaluation on merge, not on every commit

Checklist

1. 1.Unit tests mock the LLM - zero API cost, deterministic, run on every commit
2. 2.Golden dataset: 30-100 cases including edge cases and production bugs, versioned like code
3. 3.Integration tests use DeepEval or RAGAS with thresholds - faithfulness threshold ≥ 0.75
4. 4.G-Eval evaluation for complex outputs - judge from a different model family than production
5. 5.Baseline after every deployment, alert when metrics drift by more than 5 percentage points
6. 6.CI/CD: unit on every PR, eval gate only on AI changes - paths filter in GitHub Actions
7. 7.Evaluation cost monitored and budgeted: < $0.05 per merge to main
8. 8.Results visible in PR comments - no digging through CI logs required

---

I build testing systems for AI applications - from the unit test pyramid through golden datasets to CI/CD with a quality gate. If your LLM app is in production without a test suite, get in touch - I start with a code audit and designing the minimal sensible pyramid.