Data is evidence, not a verdict

1611. Lesson identity

Module

4.5 — Balancing with data

Lesson

Data is evidence, not a verdict

Academic type

Concept

Schema type

text

Order

Lesson 1 in the module

Estimated time

30–40 minutes, including practice

1612. Learning objective

After this lesson, you can identify which conclusions a balance dataset supports, which conclusions it cannot support, and what additional evidence is needed before changing a design.

1613. Why this matters

A dataset can reveal a pattern without explaining its cause. If a weapon has a low completion rate, the problem may be its cost, its availability, the encounter where it is used, or the player's understanding of its role. Changing the wrong variable can conceal the original problem and create a new one. A developer working with AI must be especially precise when turning observations into instructions: an ambiguous diagnosis produces an efficient implementation of the wrong change.

1614. Prior knowledge

You should already be able to specify a minimal gameplay event set and distinguish observable events from interpretations. This lesson builds directly on 4.4 L2 — Specify a minimal gameplay event set. You should also be familiar with the game's intended player experience and the difference between an economy value, a progression threshold, and a difficulty contract.

1615. Core concept

Balance data is evidence for a decision, not the decision itself. A useful analysis triangulates three things:

1. Observed pattern: What happened, and how often?
2. Possible explanation: What mechanisms could produce that pattern?
3. Design intent: What experience or trade-off was the system supposed to create?

A strong conclusion keeps these levels separate. For example:

• Observation: Players who acquire the shield use it less often than players who acquire the medkit.
• Possible explanations: The shield may be too expensive, its use case may be unclear, or the situations that reward it may be rare.
• Design question: Is the shield intended to be a frequent defensive tool, a situational counter, or a high-value emergency option?

The observation is supported by the dataset. The explanations are hypotheses. A plausible mechanism, such as the shield being too expensive to carry into a fight, can make one hypothesis more credible than another. A repeated observational pattern can also strengthen a hypothesis if it appears across relevant contexts. Neither a plausible mechanism nor repeated observation establishes causation by itself. Stronger causal support requires a controlled or otherwise well-designed comparison that reduces competing explanations. The design question requires intent and further investigation.

Observational data has specific limits: players choose what to buy and use, situations differ, and important variables may not be recorded. It can show associations and help generate hypotheses, but it usually cannot isolate the effect of a balance variable on its own.

1616. Mental model

Use the Evidence–Explanation–Intent triangle before making a balance change:

Then classify the relationship between variables:

• Correlation: Two things vary together. This supports an association in the observed context, but does not establish that one caused the other.
• Causal hypothesis: A proposed explanation that one factor produces a change in another. A credible mechanism or a repeated observational pattern can make this hypothesis more plausible, but neither proves it.
• Causal support: Evidence from a controlled or otherwise well-designed comparison that reduces competing explanations. This provides stronger support for a causal claim, while still requiring attention to scope and limitations.
• Design intent: The experience the system is meant to create. Intent can justify investigating a pattern, but it cannot be inferred from the numbers alone or substitute for causal evidence.

A practical statement format is:

“The data shows X in Y context. This is consistent with A and B. A plausible mechanism or repeated pattern may make A worth testing, but the observational data does not establish that A caused the result. The intended experience is Z. We should collect or compare C before changing D.”

1617. Concrete example

Consider a fictional balance dataset for a defensive item:

• 100 recorded acquisitions
• 62 uses
• 38 unused acquisitions
• Players who used the item completed the next encounter 78% of the time
• Players who did not use it completed the next encounter 55% of the time

The data supports the claim that item use and next-encounter completion are correlated in this sample. It does not prove that the item caused the higher completion rate. Players may choose to use the item when they already have more resources, more experience, or a better position.

Suppose the same association appears across several comparable encounter types, and the item is designed to mitigate the specific threats present in those encounters. That repeated pattern and plausible mechanism make the causal hypothesis—that the item contributes to better outcomes—more plausible. They still do not establish causation because players may continue to differ in unmeasured ways.

Stronger causal support would come from a controlled or otherwise well-designed comparison, such as comparing similar encounter states where item availability or use is assigned or constrained while relevant conditions are kept comparable. Even then, the conclusion should remain limited to the contexts represented by the comparison.

Possible next investigations include:

• Compare players with similar health, resources, and encounter state.
• Check whether the item was available before the encounters where it would be useful.
• Review whether players recognized the item as a response to the threat.
• Examine whether the intended role is frequent prevention or rare recovery.

A premature verdict would be: “The item is too powerful; reduce its effect.” A disciplined conclusion is: “Item use is associated with better next-encounter completion. Repeated observations and the item's role make a causal contribution plausible, but the current observational data does not isolate or establish that effect. First compare equivalent encounter contexts and verify the intended role.”

1618. Common mistake

The common mistake is treating a plausible story as proof. A mechanism that sounds convincing, or a pattern that repeats in observational data, can increase confidence in a causal hypothesis without establishing causation. A low pick rate does not automatically mean an item is weak. A high win rate does not automatically mean it is overpowered. Metrics must be interpreted within their population, context, sample limitations, unmeasured differences, and intended design role. Stronger causal claims require a controlled or otherwise well-designed comparison rather than observation alone.

1619. Guided practice

Analyze both the labeled dataset and the accompanying playtest notes below. Keep three layers separate: what the dataset records, what the notes suggest, and what the design intends. Do not use either source as automatic proof of causation, and do not introduce a balance intervention or implementation model in this exercise.

Labeled dataset

Accompanying playtest notes

• Two testers said the expensive tool looked useful but that they were unsure when to spend it.
• One tester bought the expensive tool immediately before a high-damage encounter; another saved it through several low-threat rooms.
• Testers with fewer starting resources described the purchase as risky.
• The notes cover five observed sessions and are not a representative sample.

Write a short analysis using the Evidence–Explanation–Intent triangle.

1. State one claim that D1–D5 directly support, and limit it to the dataset's scope.
2. State one claim the playtest notes support, while naming their small and non-representative scope.
3. List at least three possible explanations for the higher abandonment rate, combining evidence from both sources without presenting any explanation as proven.
4. Identify one plausible mechanism or repeated observational pattern that could make one explanation more credible, and explain why it still would not establish causation.
5. State one design-intent question that must be answered.
6. Choose one additional comparison, event, or qualitative check that would help separate the explanations.
7. Decide whether the correct immediate action is change the balance value, collect more evidence, or inspect the player-facing communication. Justify the decision from both the dataset and the playtest notes.

Your decision must distinguish what is measured from what is inferred. Do not recommend a numerical change merely because one outcome is worse than another. Do not describe a mechanism or repeated pattern as proof unless the evidence comes from a controlled or otherwise well-designed comparison.

1620. Validation / evidence

Your work is sufficient when it contains:

• One dataset observation limited to D1–D5's actual scope.
• One playtest observation limited to the notes' small sample.
• Three plausible explanations, clearly labeled as hypotheses.
• One explicit distinction between correlation and causation.
• One mechanism or repeated observational pattern identified as support for plausibility, not proof.
• One design-intent question.
• One additional comparison, event, or qualitative check that could reduce uncertainty.
• A justified decision about whether to change, investigate, or clarify.

A strong answer can say “the current evidence is insufficient” and explain exactly what evidence is missing. It should reserve a stronger causal claim for a controlled or otherwise well-designed comparison and state what that comparison would and would not show.

1621. Key takeaways

• Data describes patterns; it does not automatically explain them.
• Correlation is not proof of causation.
• A plausible mechanism or repeated observational pattern can make a causal hypothesis more credible, but cannot establish it by itself.
• Controlled or otherwise well-designed comparisons provide stronger support for causal claims than observational data alone.
• Playtest notes add context and hypotheses, but their sample and selection limits must remain visible.
• Design intent is a criterion for evaluation, not a conclusion hidden inside the dataset.
• A balance change should follow a justified diagnosis, not merely a concerning metric.

1622. Next lesson

Continue to 4.5 L2 — Make one reversible balance decision.

1623. Próxima lección

La siguiente lección del módulo 4.5 — Equilibrio con datos es L2 — Toma una decisión de balance reversible.

1624. Knowledge check