Make a test fail for the right reason

2239. Lesson identity

Module

5.9 — AI-generated tests

Lesson

Make a test fail for the right reason

Academic type

Debugging Lab

Schema type

Practical

Order

2

Estimated time

40–50 minutes

This lab follows Ask AI for tests that express a contract. You will challenge a generated test by introducing a controlled defect, observing the result, and diagnosing whether the test actually protects the contract.

2240. Learning objective

After this lesson, you can demonstrate, diagnose, and correct a misleading or ineffective test by showing that it fails when the intended behavior is broken and passes when that behavior is preserved.

2241. Why this matters

A passing test is not automatically useful evidence. A test can pass because it asserts the wrong value, never reaches the relevant branch, or reproduces the same defect as the implementation. AI can generate tests that look precise while providing false confidence. In a game system, that can leave an economy rule, progression threshold, or input response unprotected. Deliberately making a test fail exposes whether its contract is real.

2242. Prior knowledge

You should have completed Ask AI for tests that express a contract. You should be able to identify the intended behavior, observable acceptance criteria, boundaries, invariant, and oracle for a small game rule. You should also be able to run the project test command and inspect the failure output.

2243. Core concept

The core concept is failure validity: a test is trustworthy only when its failure is caused by a violation of the intended contract, not by a broken setup, an unrelated exception, an incorrect fixture, or a mismatch between the test and the implementation.

Use two controlled changes:

1. Break the behavior: introduce a small defect that violates the contract. The test should fail with an assertion that points to that violation.
2. Break the test: alter the test or its setup so that it no longer checks the contract. The test may pass, fail for an unrelated reason, or stop exercising the target path.

The first change demonstrates detection. The second teaches diagnosis. Both are necessary because a test that fails is not necessarily a good test.

2244. Mental model

The failure-validity chain

A useful test should produce this sequence:

Correct behavior → pass; controlled contract violation → meaningful fail; corrected behavior → pass.

If the sequence does not hold, stop treating the test result as evidence and inspect the test itself.

2245. Concrete example

Suppose a safehouse reward rule says: when a run succeeds, the player receives the base reward plus the approved bonus; a failed run receives no success bonus.

The intended contract for a successful run with a base reward of 100 and a bonus of 25 is 125. An AI-generated test might contain an assertion such as:

expect(resolveReward({ success: true, base: 100, bonus: 25 })).toBe(125);

To test failure validity, change the implementation temporarily so that it returns only base. A valid test should fail with an expected value of 125 and an actual value of 100. That failure is connected to the contract.

Now consider two misleading alternatives:

• The test calls resolveReward({ success: false, base: 100, bonus: 25 }) but still expects 125. It may expose a different rule than the one being discussed.
• The test calculates the expected value with the same helper used by the implementation: toBe(calculateReward(input)). If both contain the same missing bonus, the test can pass while the contract is broken.

The test is not validated by its appearance or by a green result. It is validated by its behavior under a controlled, contract-relevant defect.

2246. AI-native workflow

Use AI as a test challenger and diagnostic assistant, not as the authority that declares the test valid.

1. Provide AI with the contract, the test, the relevant implementation, and the exact controlled defect you introduced.
2. Ask it to predict the expected failure and name the assertion that should fail.
3. Run the test yourself and compare the actual output with the prediction.
4. If the result differs, ask AI for competing explanations, then inspect the test setup, execution path, oracle, and implementation independently.
5. Ask AI to propose a correction only after you can state why the current test is misleading.
6. Apply or reject the suggestion based on the contract and rerun the controlled-defect sequence.

Do not ask AI only, “Does this test look good?” Ask questions that require evidence: “What contract violation would make this test fail?”, “What defect could this test miss?”, and “Which line proves that the target behavior was exercised?”

2247. Common mistake

The common mistake is treating any red test as proof that the test works. A test can fail because a fixture is malformed, a mock is not configured, an import is broken, or the assertion checks an incidental implementation detail. Before interpreting the failure, identify the executed path, the expected value, and the contract clause that the failure represents.

A second mistake is changing several things at once. If the implementation, fixture, and assertion all change together, you cannot determine which change caused the result. Make one controlled change at a time.

2248. Guided practice

Use one small test generated or revised in the previous lesson. Choose a contract with a clear observable result. Do not use a broad end-to-end scenario for this lab.

Part A — Establish the baseline

1. Write the contract in one sentence, including the input condition and expected observable result.
2. Run the test without changes.
3. Record the result, the assertion being evaluated, and the path or function the test is intended to exercise.
4. Mark the source of the expected value. It should come from the contract or an independent example, not from the implementation under test.

Part B — Introduce one intended defect

1. Make one temporary implementation change that violates the contract. Examples include omitting a required reward component, using the wrong threshold comparison, or returning the pre-update state.
2. Run only the target test.
3. Capture the failure message and identify the assertion that failed.
4. Decide whether the failure directly represents the contract violation. If it does not, inspect reach, oracle, and setup before changing the test.
5. Restore the implementation and confirm that the test passes again.

Part C — Challenge the test itself

Choose one challenge and make it temporary:

• Change the fixture so the target branch cannot be reached.
• Replace the independent expected value with a value calculated by the implementation.
• Alter the input so it tests a neighboring case rather than the stated contract.
• Remove or weaken the assertion while leaving the test runnable.

Run the test and classify the result as meaningful failure, false pass, or unrelated failure. Explain the classification in one or two sentences. Then correct the test and repeat the baseline → controlled defect → repair sequence.

During the lab, use AI to generate hypotheses about the result, but verify each hypothesis by reading the test and examining the runner output.

2249. Validation / evidence

Your evidence is complete when you can point to all of the following:

• The contract and its independent expected result.
• A baseline passing run.
• One controlled implementation defect that produces a meaningful assertion failure.
• The exact failure output and the contract clause it represents.
• A restored implementation that passes the same test.
• One misleading-test challenge and a diagnosis of why its result was a false pass or unrelated failure.
• A corrected test whose reach, oracle, and expected result are defensible.

A test does not pass this lab merely because the final run is green. The required evidence is the complete failure-validity chain.

2250. Key takeaways

• A useful test must fail when the behavior named by its contract is broken.
• Failure validity depends on reach, an independent oracle, and a contract-relevant defect.
• A red test can indicate a broken test setup rather than a broken implementation.
• Controlled defects are a practical way to test whether a test is actually protective.
• AI can suggest diagnoses, but the learner must verify the cause against the contract and runner evidence.

2251. Knowledge check

2252. Next lesson

Next: 5.10 — AI art pipeline. Continue to the next module, where you will begin the AI art pipeline and address a different production concern.

2253. Knowledge check