Ask AI for tests that express a contract

2225. Lesson identity

Module

5.9 — AI-generated tests

Lesson

Ask AI for tests that express a contract

Academic type

Workflow

Schema type

mixed

Order

1

Estimated time

30–40 minutes, including practice

2226. Learning objective

After this lesson, you can produce and edit a focused test plan for one system by stating its behavior, acceptance criteria, invariants, boundaries, expected outcomes, and regression risks.

2227. Why this matters

A generated test is useful only when it checks a behavior that the game actually promises. If the request to AI contains only an implementation description, the result may repeat the implementation instead of protecting the player-facing contract. Acceptance criteria make the contract precise enough to evaluate: they state the observable conditions that must be true for the behavior to count as correct. A focused test plan gives you a basis for accepting, editing, or rejecting generated tests. It also connects the root-cause and regression-risk reasoning from the previous lesson to a concrete verification task.

2228. Prior knowledge

You should be able to:

• describe a system's intended behavior and observable outcome;
• distinguish an observation from a root-cause hypothesis;
• identify ownership and regression risk from 5.8 — Find root cause, ownership, and regression risk;
• read the relevant implementation or interface well enough to name inputs, state changes, and outputs.

2229. Core concept

A test expresses a contract: under a defined condition, the system must produce an observable result and preserve any stated invariant. Acceptance criteria are the explicit, observable conditions used to decide whether that contract is satisfied. They should describe what must be true from the system or player's perspective, not merely which internal function or branch executed.

For one system, record five elements and turn them into acceptance criteria:

1. Behavior: what the system is supposed to do.
2. Inputs and state: the condition, data, or setup that makes the behavior observable.
3. Invariant: what must remain true before and after the operation.
4. Boundary: the limit, empty case, invalid case, threshold, or transition where behavior may change.
5. Oracle: the evidence that lets you decide pass or fail.

A useful acceptance criterion has a defined condition and an observable expected result. For example: “When the player has exactly enough currency, a successful purchase grants the item and reduces currency by exactly the listed cost.” “The purchase succeeds” alone is too vague because it does not identify the evidence required for acceptance.

A test plan is focused when each case protects a distinct contract rather than accumulating similar examples.

2230. Mental model

Contract-to-test ladder

Acceptance criteria connect the contract to a test through this ladder:

Use the ladder as a completeness checklist, not as a set of categories for classifying cases. Every proposed case should identify its intent or acceptance-criterion ID, preconditions and starting state, action, and observable oracle. Then record separately whether the case covers a boundary and which regression risk it addresses. Ask AI to draft candidate cases only after the acceptance criteria are explicit. If a case has no supported criterion or intent, mark it for removal or clarification; if a criterion has no case, identify the coverage gap.

2231. Concrete example

Suppose the system under review is a reward claim operation. Its contract is:

• If a reward is available and the player meets the requirement, the reward is granted once.
• A failed claim does not remove currency or progress.
• Repeating a successful claim does not grant the reward twice.

The corresponding acceptance criteria are:

• Given an available reward and a player who meets the requirement, when the player claims it, the reward is granted and the claim is recorded.
• Given an unavailable reward or unmet requirement, when the player claims it, no reward is granted and existing progress and currency remain unchanged.
• Given a previously successful claim, when the player claims again, no second reward is granted.
• Given the exact requirement and a value one below it, the system produces the specified success or failure result without violating the relevant invariant.

A focused plan could contain these cases:

1. Eligible first claim: the reward is granted, the claim becomes recorded, and the relevant state changes once.
2. Ineligible claim: the reward is not granted and existing progress and currency remain unchanged.
3. Repeat claim: the second request is rejected or treated as already claimed, with no duplicate reward.
4. Threshold boundary: test the exact requirement and one value below it.

Each case is now traceable to an acceptance criterion. Notice that “the method returns without an error” is not enough. The oracle must check the game-relevant state and the invariant. AI may suggest additional cases, but you keep a case only when you can name the acceptance criterion and contract it protects.

2232. AI-native workflow

Use AI as a test-design assistant, not as the authority on expected behavior. This lesson ends with an approved, contract-based test plan. Define required preconditions and starting state declaratively, but leave test-code generation, executable fixture and environment construction, execution, and failure diagnosis to L2.

1. Prepare context. Provide the system name, public behavior, acceptance criteria, relevant inputs and outputs, invariants, boundaries, and known regression risk. Exclude unrelated files and do not ask for a whole-project test suite.
2. Request a plan. Ask AI to return a table with case name, acceptance-criteria ID, preconditions and starting state, action, observable oracle, preserved invariant, boundary coverage, regression risk, and reason for inclusion.
3. Challenge the draft. Ask which cases are duplicates, which proposed checks merely restate implementation details, which acceptance criteria lack coverage, which boundaries are missing, and which expected results are unsupported by the contract.
4. Edit for truth. Correct assumptions, remove speculative cases, add missing failure paths, and make each acceptance criterion and oracle observable and deterministic. If an expected result is not specified, record it as an unresolved design decision rather than selecting an AI suggestion.
5. Approve the handoff artifact. Select and edit the cases that the contract supports, verify that each case is complete, map every selected case to an acceptance criterion, and record coverage gaps and rejected suggestions.

A useful request is specific: “Derive a six-case test plan for these reward-claim acceptance criteria. Separate normal, failure, repeat, and boundary cases. For each case, reference the criterion it verifies and state the preconditions and starting state, action, observable oracle, preserved invariant, boundary coverage, regression risk, and reason for inclusion. Do not infer behavior that is not listed.”

2233. Common mistake

The common mistake is accepting a large generated suite because it looks thorough, without checking whether its cases map to explicit acceptance criteria. Quantity does not establish coverage. A test can be redundant, implementation-coupled, nondeterministic, or based on an invented expectation. Reject any candidate whose acceptance criterion, contract, oracle, or reason for inclusion cannot be stated clearly.

2234. Guided practice

Choose one small system you have already inspected, such as a purchase, reward claim, cooldown, inventory transfer, or progression check. Do not choose the entire game.

1. Write one sentence describing the intended behavior.
2. List the relevant inputs and starting state.
3. Write one invariant that must survive the operation.
4. Identify at least two boundaries, including one failure or invalid case.
5. Convert the behavior, invariant, and boundaries into explicit acceptance criteria. Give each criterion a short ID, such as AC-1 or AC-2.
6. Define an observable oracle for each criterion.
7. Ask AI for a test plan using the contract-to-test ladder as a completeness checklist. Require every case to state an acceptance-criterion ID, preconditions and starting state, action, and observable oracle. Also require it to identify boundary coverage and regression risk separately.
8. Compare the AI plan with your own list. Mark each candidate as keep, edit, remove, or needs decision.
9. Select four to six cases and write the final plan in your own words. For each case, include its acceptance-criterion ID, preconditions and starting state, action, observable oracle, whether it covers a boundary, and the regression risk it addresses.

The required decision is which AI-generated case to reject or edit, and why it is redundant, unsupported, implementation-coupled, incomplete, or otherwise unsafe. Do not accept the plan until every selected case is complete and traceable to a criterion, and every important criterion has coverage or an explicitly recorded gap.

2235. Validation / evidence

Submit a focused test plan for one system as the scored practical assessment. Score the submission against the evidence below: acceptance-criterion traceability, observable oracles, boundary coverage, one justified rejection or edit of an AI suggestion, and an explicitly recorded gap or unresolved design decision. The submission must contain:

• a named behavior and numbered acceptance criteria written as observable conditions and expected results;
• at least one invariant;
• four to six selected cases covering normal, failure or invalid, repeat or idempotency, and boundary behavior where applicable;
• for every case, an acceptance-criterion ID, preconditions and starting state, action, and observable oracle;
• for every case, a separate statement of whether it covers a boundary and which regression risk it addresses;
• a reason for keeping each case;
• a note identifying any unresolved design decision or uncovered criterion;
• an edit record showing at least one AI suggestion that you rejected or changed.

The plan is ready when another developer can trace each selected case to an acceptance criterion and understand what “correct” means without guessing. If a case lacks a supported criterion, precondition, action, or observable oracle, edit it, remove it, or return it for clarification. If an acceptance criterion has no test case, record the gap rather than implying that coverage exists.

2236. Key takeaways

• A test protects a contract, not merely an implementation path.
• Acceptance criteria state the observable conditions that make a contract pass or fail.
• Use the contract-to-test ladder to check case completeness, not to assign each case to one layer.
• Record boundary coverage and regression risk separately from the case's preconditions, action, and oracle.
• Use AI to expand and challenge test intent; verify every expected result yourself.

2237. Next lesson

Next: 5.9 L2 — Make a test fail for the right reason. Carry forward the approved contract-based test plan. L2 covers test-code generation and implementation, executable fixtures and environment setup, execution, and diagnosis of assertion failures.

2238. Knowledge check