Measure a decision, not a fantasy dashboard

1581. Lesson identity

Module

4.4 — Gameplay telemetry

Lesson

Measure a decision, not a fantasy dashboard

Academic type

Concept

Schema type

text

Order

1

Estimated time

30–40 minutes, including practice

This lesson establishes a decision-first standard for gameplay telemetry. A useful event is not merely data that can be collected; it is evidence that can inform a stated design decision.

1582. Learning objective

After this lesson, you can reject telemetry events that do not support a stated design decision, or that exceed the scope of what the event can legitimately show.

1583. Why this matters

A large dashboard can create the appearance of control without helping anyone choose an action. In a game, a designer may want to know whether players avoid a risky route, misunderstand an objective, or cancel an interaction after a cost is presented. Each question requires different evidence. If an event has no decision attached to it, AI can help generate names and queries, but it cannot give the event a valid purpose. Decision-first telemetry keeps measurement connected to gameplay judgment rather than to data collection for its own sake.

1584. Prior knowledge

You should already be able to:

• distinguish an observable signal from an interpretation;
• define a small observability plan with an interpretation, owner, and response;
• separate a gameplay question from a presentation preference;
• identify the relevant player action and game state in a decision.

The immediate prerequisite is 4.3 L2 — Design a small observability plan.

1585. Core concept

Every proposed telemetry event needs a decision contract:

1. Decision: What design decision could this evidence inform?
2. Event purpose: What player action or state does the event record?
3. Scope: Which players, session segment, version, context, or time window does it describe?
4. Interpretation limit: What can this event suggest, and what can it not establish?

Reject the event when the decision is missing, the event does not measure the relevant behavior, the scope is undefined, or the interpretation claims more than the evidence supports.

A useful test is:

If this event changed, what would we consider changing in the game?

If the answer is “nothing,” the event may be interesting, but it is not yet decision-supporting telemetry.

1586. Mental model

Use the Decision → Evidence → Scope → Limit chain.

The chain is not a funnel for collecting more events. It is a filter for removing events that cannot support a defensible decision.

1587. Concrete example

Suppose a stealth encounter has a risky shortcut. The design question is:

Should the shortcut be made more visible, or is its low use an intentional choice by players who understand the risk?

Consider three proposed events:

• encounter_started: records that the player entered the encounter.
• shortcut_seen: records that the game presented the shortcut cue while the player was in the encounter state defined for that cue.
• shortcut_used: records that the player committed to the shortcut.

encounter_started provides context but cannot answer whether players noticed the shortcut. shortcut_used measures the choice, but by itself it cannot show whether the player saw and rejected the option. shortcut_seen adds evidence that the game presented the cue, but it still cannot prove that the player noticed it or understood the risk.

A bounded telemetry plan might compare shortcut_seen and shortcut_used for players who reached the encounter, within a named game version and encounter context. A high exposure rate with low usage may support a visibility decision only when combined with other evidence, such as observed interaction attempts or a controlled presentation change. It does not prove that players disliked the shortcut or found it confusing.

The point is not to select all three events automatically. The point is to state what decision each event can support and reject any interpretation that exceeds its evidence.

1588. Common mistake

The common mistake is treating a metric as valuable because it is precise, easy to aggregate, or visually impressive. A count such as “average actions per session” may be accurate while remaining irrelevant to the decision about a particular encounter. Another mistake is treating correlation as intent: players who cancel an interaction after a warning may have misunderstood the warning, changed priorities, encountered a technical problem, or simply chosen another strategy. Telemetry narrows possibilities; it does not read player motivation directly.

1589. Guided practice

Evaluate the following proposed events for a single question:

Should the cost of a risky interaction be changed because players are avoiding it?

For each event, write one sentence for each field in the decision contract:

1. session_duration
2. interaction_prompt_shown
3. interaction_confirmed
4. interaction_cancelled_after_prompt
5. resource_spent_on_interaction

Then classify each event as keep, keep as context, or reject for this decision.

Use these criteria:

• Keep: the event directly measures behavior relevant to the decision.
• Keep as context: the event helps define scope or a competing explanation but cannot answer the question alone.
• Reject for this decision: the event has no plausible effect on the decision, or its interpretation would be too broad.

Your most important decision is not the classification itself. It is the explanation of what would change if the evidence moved in an unexpected direction. For example, if interaction_cancelled_after_prompt is high while interaction_confirmed is low, identify at least two explanations that remain possible before recommending a design change.

1590. Validation / evidence

Your evidence is a completed decision contract for at least three proposed events. It must include:

• one explicit design decision;
• the behavior or state each event records;
• a defined scope, such as encounter, session segment, version, or comparison group;
• one interpretation limit for each event;
• a keep, contextualize, or reject judgment;
• a statement of what additional evidence would be needed before changing the game.

A successful submission rejects at least one event for a reason tied to the decision, rather than rejecting it because it is generally “bad data.” It also distinguishes evidence about what players did from claims about why they did it.

1591. Key takeaways

• Telemetry earns its place by supporting a stated design decision.
• Event purpose and event meaning are not the same: a recorded action still needs interpretation boundaries.
• Scope prevents a local observation from being mistaken for a universal player pattern.
• A useful event can narrow a decision without proving player motivation.
• Rejecting irrelevant telemetry is part of observability design, not a loss of information.

1592. Next lesson

Next in module 4.4: 4.4 L2 — Specify a minimal gameplay event set. You will specify a minimal, precise telemetry event set without implementing a tracker.

1593. Knowledge check