Evaluate variation without worshiping quantity

2327. Lesson identity

Module

5.12 — Procedural content

Lesson

Evaluate variation without worshiping quantity

Academic type

Debugging Lab

Schema type

practical

Order

2 of the module sequence

Estimated time

40–50 minutes, including practice

This lesson turns procedural generation into an evaluation problem. You will inspect a sample set, separate useful variation from noise, and revise generator constraints using evidence rather than output count.

2328. Learning objective

After this lesson, you can evaluate a generated sample set against playability and purpose criteria, identify failure patterns, and revise the generator’s constraints or rejection gates accordingly.

2329. Why this matters

A generator can produce hundreds of outputs while producing very few usable ones. Quantity does not compensate for broken traversal, repetitive decisions, irrelevant variation, or layouts that undermine the intended player experience. In an AI-assisted workflow, the ability to evaluate samples is more important than asking for more generation. The goal is a generator that reliably produces candidates worth reviewing, not a large folder of unexamined results.

2330. Prior knowledge

You should already be able to:

• Define procedural invariants and distinguish them from preferences.
• Describe the generator’s inputs, constraints, and acceptance conditions from Design a generator around invariants.
• Inspect a generated result and record whether it passes a required property.

2331. Core concept

Procedural variation needs two separate controls:

1. Sampling creates a deliberate exploratory set of candidates across selected inputs and risk conditions. A small sample can reveal patterns, but it is not automatically representative of the generator’s full output distribution.
2. Quality gates reject candidates that violate hard requirements or fail the intended play-purpose.

A useful evaluation does not ask, “How many different outputs did we get?” It asks:

• Which outputs are valid?
• Which outputs create meaningfully different player decisions?
• Which failures repeat often enough to reveal a generator problem?
• Which accepted outputs are technically valid but still not worth using?

Rejection is not wasted generation. It is part of the generator’s design. However, a high rejection rate can indicate that the generator explores a poor search space or that its constraints are contradictory. Measure rejection and inspect representative failures instead of hiding them.

Three evaluation categories

2332. Mental model

Use the sample → classify → diagnose → constrain model:

Generate a deliberate sample
          ↓
Classify each result: pass, reject, or questionable
          ↓
Diagnose patterns across results
          ↓
Change one constraint or quality gate
          ↓
Generate a new comparison sample

For each sample, record four fields:

Compare samples by meaningful behavior, not by visual difference alone. Two layouts may look different but force the same route, timing, and decision. Conversely, small spatial changes may create an important tactical distinction.

2333. Concrete example

Suppose a generator creates five-room exploration layouts. Its invariants require a connected path from entrance to objective and at least one optional branch. The intended purpose is to make the player choose between a safer, longer route and a riskier, shorter route.

A sample of 20 candidates produces three non-overlapping classifications:

• All 20 pass the hard connectivity and optional-branch invariants.
• 14 also pass the play-purpose gate by offering a credible safer route and a credible riskier route.
• 4 fail the branch-separation criterion because the optional branch rejoins the main path immediately.
• 2 fail the route-cost contrast criterion because the objective can be reached only through the most dangerous area, making the supposed safe route irrelevant.

Counting 20 hard-valid outputs would suggest healthy productivity. Evaluating purpose shows a different result: only 14 support the intended route decision. The other six are not hard-invariant failures; they are purpose failures identified by separate quality criteria. The next revision should not simply request more rooms. It should add or tune a quality gate for branch separation or route-cost contrast, then inspect whether danger placement still permits a credible alternative.

The gate should remain specific. “Make the layout more interesting” is not a useful test. “The optional route must differ from the shortest route by at least two traversal segments and have a lower exposure score” is testable, even if the exact thresholds require tuning.

2334. AI-native workflow

Use AI as an analysis and revision partner, not as the final judge.

1. Export or collect a labeled sample with reproducible seeds. Include both accepted and rejected results.
2. Ask the AI to group failures by observable pattern. Require it to quote the evidence for each group and distinguish facts from hypotheses.
3. Review the grouping yourself. Correct any category that confuses visual difference with meaningful gameplay variation.
4. Ask the AI for two or three narrowly scoped constraint or gate changes. Do not accept a broad rewrite of the generator.
5. Choose one change, apply it, and generate a comparison sample using new seeds plus at least one previously failing seed as a regression input. Preserve the original artifact, generator version, and configuration for exact before-and-after evidence.
6. Compare pass rate, failure categories, and play-purpose results. Keep the change only if it improves the target without breaking an existing invariant.

A useful request is:

Here is a labeled sample of generated layouts. Classify each result by hard validity, intended play-purpose, and meaningful variation. Cite the sample IDs for every claim. Then propose one quality gate that addresses the most frequent purpose failure. Do not modify the generator yet, and do not treat visual novelty as gameplay value.

The learner remains responsible for the acceptance criteria. AI can accelerate classification and suggest hypotheses; it cannot establish that a result is fun or purposeful from a description alone.

2335. Common mistake

The common mistake is treating every non-identical output as valuable variation. Randomized positions, colors, or room shapes may create visual novelty without changing the player’s decisions. The opposite mistake is rejecting unusual outputs because they depart from a familiar pattern, even when they satisfy the invariants and create a useful alternative. Evaluate the behavior and purpose produced by the variation, not the amount of randomness used to create it.

2336. Guided practice

Use the generator from the previous lesson or the ready-to-inspect training pack below. Both routes assess the same classification and revision capability; an executable generator is not required. If you use visual layouts, provide each one with a structured text alternative, such as an adjacency list or ordered route list that records connections, costs, and observed failures.

Ready-to-inspect training pack

This fixed pack uses entrance E, objective O, and intermediate rooms A–C. A route is written as an ordered node sequence followed by [cost, exposure]. Hard validity requires a connected E-to-O path and an optional branch. The play-purpose gate requires credible safer-longer and riskier-shorter routes with sufficient branch separation and contrasting exposure.

Every original row has the same reproducibility record: generator version TP-1.0; configuration C0. Configuration C0 means the supplied graph, route, cost, and exposure records are used exactly as written, with no repair or omitted seed. Cite each original label as seed | TP-1.0 | C0 | label | evidence.

Completed reference subset:

• S01 — keep: passes both hard invariants and provides a longer route with lower exposure than the shorter route.
• S04 — reject: fails hard validity because no connected route runs from E to O.
• S02 — investigate: passes hard validity, but equal exposure removes the intended risk contrast; confirm the purpose threshold before rejecting it.

Classify the remaining outputs yourself. Preserve this original pack as the before-change artifact; any revised interpretation or output belongs in a separate comparison record.

For the no-generator route, produce that comparison record with this deterministic paper-based revision:

• Revised version: TP-1.1
• Revised configuration: C1 = C0 + route-choice gate
• Route-choice gate: retain a candidate only when it passes both original hard invariants, contains two distinct routes from E to O that do not rejoin before O, and the longer route costs at least one unit more while having at least one unit less exposure than the shorter route.
• Transformation order: for each seed from S01 through S12, copy its original structured output; test connectivity, optional branch, two distinct routes, route separation, cost contrast, and exposure contrast in that order; then append either ACCEPT or REJECT: first failed check. Do not alter nodes, costs, or exposure values.
• Required revised label: seed | TP-1.1 | C1 | ACCEPT/REJECT | first failed check or passing evidence.

This procedure creates the complete after-change pack from the supplied seeds without a live generator. Keep both versions so every comparison can be traced to the same recorded input.

Part A — Build a sample

Use the supplied pack or collect 12–20 candidates from your generator. Complete this reproducibility checklist before classification:

• Freeze and record the generator version and configuration.
• Choose and record seeds before inspecting outputs.
• Use every selected result rather than keeping only favorites.
• When inputs have known ranges or risk boundaries, cover low, typical, high, and boundary conditions.
• Record every exclusion and its reason.
• Include at least two rejected results and one hard-valid result whose play-purpose is uncertain.
• Preserve each original artifact or structured output with its seed, version, and configuration.

Treat this as an exploratory sample, not proof of statistical reliability. Keep it distinct from regression seeds retained to check known failures. Claims about output frequencies or reliability require broader distribution testing beyond this lab.

Part B — Classify the candidates

Use the supplied procedural evaluation worksheet with these columns:

Use only three actions: keep, reject, or investigate. Enter these as textual labels rather than distinguishing states by color alone. Complete the table in a keyboard-accessible document or spreadsheet with explicit column headers. An investigate result is appropriate when the candidate passes hard checks but its play-purpose is uncertain.

Part C — Make one constraint decision

Identify the most important repeated failure. If you are using a live generator, choose one response:

• Add or tighten a hard quality gate.
• Change a generator constraint so the failure is less likely.
• Keep the current rule and document why the observed variation is acceptable.

If you are using only the fixed pack, treat the C1 route-choice gate as the selected change. Use your C0 classifications to explain which repeated failure it addresses and which existing invariants it must preserve.

Write the decision in this form:

Because [evidence from the sample], I will [change or preserve one rule]. I expect this to improve [playability or purpose] without weakening [existing invariant]. I will verify it by [specific comparison].

Part D — Run a comparison

Apply only the selected change. With a live generator, generate a second sample with new seeds and reuse at least one previously failing seed as a regression input; preserve both versions and configurations. With the fixed pack, apply the deterministic C1 transformation to all 12 original seeds and compare each TP-1.1 | C1 record with its matching TP-1.0 | C0 record. Compare:

• Original hard-validity rate and revised overall gate-acceptance rate, reported separately.
• Number and type of repeated failures before and after the added gate.
• Number of meaningful variation classes retained.
• Whether the intended player decision is present in every retained candidate.

Do not declare success merely because the revised records contain more accepted outputs. State whether C1 should be kept or reverted, including any valid variation it rejects and any purpose failure it still permits.

2337. Validation / evidence

You have completed the lab when you can point to:

1. A reproducible sample containing both accepted and rejected candidates.
2. A classification table with evidence for each keep, reject, or investigate decision.
3. One documented generator constraint or quality-gate decision tied to a repeated failure pattern.
4. A before-and-after comparison that reports pass rate and failure categories.
5. A written conclusion stating whether the revision improved play-purpose, not just output quantity.

A strong result may include rejected candidates. Rejection is acceptable when the reason is explicit, the gate measures a real requirement, and the generator can still produce a useful range of valid results.

Practical assessment rubric

The quiz is a knowledge check. Submit the worksheet, preserved before-and-after artifacts, reproducibility record, and final decision for the practical assessment. Score each dimension from 0 to 2.

Submission standard: earn at least 9 of 12 points, with no zero in evidence traceability or invariant preservation. Revise and resubmit the practical evidence if the standard is not met.

2338. Key takeaways

• Sampling reveals patterns that a single favorite output hides.
• Hard validity, play-purpose, and variation value are different evaluation categories.
• Rejection is a designed part of procedural generation, not automatically a defect.
• A meaningful revision changes one constraint or gate and compares evidence before and after.
• Visual difference is not enough; variation must affect play in a purposeful way.

2339. Next lesson

Proceed to Module 5.13 — Systems thinking. Use the evidence from this evaluation—especially the relationship between constraints, rejection gates, and player purpose—as input for analyzing the wider system rather than treating the generator as an isolated feature.

2340. Knowledge check