Ask an operational question first

1553. Lesson identity

Module

4.3 — Observability

Lesson

Ask an operational question first

Academic type

Systems

Schema type

Text

Order

1 in module 4.3

Estimated time

30–40 minutes, including practice

1554. Learning objective

After this lesson, you can convert a product risk into a specific operational question and identify the signals, symptoms, and evidence needed to answer it.

1555. Why this matters

Observability is not the practice of collecting every available metric. It is the practice of making important product behavior answerable when the team needs to make a decision. A dashboard chosen before the question often produces attractive information without operational value. Starting with the question keeps instrumentation connected to player impact, system behavior, and a concrete response. It also gives AI a bounded reasoning task instead of asking it to invent a monitoring system without context.

1556. Prior knowledge

You should already be able to:

• distinguish a product promise from an implementation detail;
• identify ownership, evidence, and capacity limits in the smallest responsible operating model from 4.2 L2 — Choose the smallest responsible operating model;
• describe a product risk without treating every risk as an emergency.

You do not need prior experience with a specific dashboard or observability platform.

1557. Core concept

The core concept is question-first observability.

A product risk is not yet an observability specification. For example, “players may be unable to complete a session” names a concern, but it does not tell the team what to inspect. Convert it into an operational question:

When this risk occurs, what do we need to determine, for whom, over what time window, and in order to take which action?

A useful operational question has five properties:

1. Specific subject: what player journey, system behavior, or promise is at stake?
2. Relevant population: which players, sessions, builds, platforms, or environments matter?
3. Time boundary: when must the answer be available, and over what period should the behavior be compared?
4. Decision connection: what decision will the answer support?
5. Evidence standard: what observation would distinguish a real problem from a suspicion?

Only after the question is clear should you select signals, define symptoms, or decide how evidence will be collected.

1558. Mental model

Use the Question → Signal → Symptom → Evidence chain.

Keep the layers distinct:

• A question is not a dashboard.
• A signal is not automatically a problem.
• A symptom is not proof of a cause.
• Evidence is not a collection of unrelated numbers; it must help answer the question.

This separation prevents a common failure mode: selecting familiar metrics first and then inventing a story around whatever they show.

1559. Concrete example

Consider this generic product risk:

“A change to a daily challenge may leave some players unable to complete it.”

A dashboard-first response might request session count, average playtime, memory usage, frame rate, error count, and several demographic breakdowns. That list is not yet a monitoring design because it does not say what decision any item supports.

A question-first response is narrower:

“During the first 24 hours after a build change, are sessions that begin the daily challenge completing it at a lower rate, and is the change large enough to trigger rollback or investigation?”

Now the chain can be specified:

• Question: Are daily-challenge completions declining for the relevant session population after the build change?
• Signals: daily challenge started, daily challenge completed, session outcome, and build identifier.
• Symptom: completion rate declines for one build while the number of starts remains comparable.
• Evidence: the comparison window, population definition, event ordering, build label, and representative failed-session records.

Notice what this example does not claim. A lower completion rate does not prove that the build change caused the problem. It identifies a symptom that justifies investigation or a predefined operational response.

1560. Common mistake

The most common mistake is treating “more telemetry” as equivalent to observability.

A large dashboard can still fail to answer the important question because it lacks a population definition, a comparison window, event context, or an agreed response. Another mistake is labeling a suspected cause as evidence: “the new flow is broken” is a hypothesis, not an observation. Preserve the distinction between what was asked, what was measured, what changed, and what has been established.

1561. Guided practice

Use the following invented risk. Do not choose tools or draw a dashboard yet.

“Players may spend resources on an action without receiving the promised result.”

Complete this worksheet:

1. Risk: Rewrite the risk in one sentence without naming a presumed cause.
2. Operational question: Specify the player action, relevant population, time boundary, decision, and evidence standard.
3. Signals: List the minimum events or measurements needed to answer the question.
4. Symptoms: Describe one observable pattern that would justify investigation.
5. Evidence: List the context required to distinguish a real product issue from a logging gap, duplicate event, or player misunderstanding.
6. Response: State the decision the answer should support, such as investigate, pause a change, or continue monitoring. Do not invent a threshold unless you can justify it.

One possible question structure is:

“For sessions in [population] during [window], are actions that [start condition] followed by the promised result at the expected rate, and what evidence is required before [decision]?”

Your task is not to copy this wording. Your task is to make the question operationally answerable.

1562. Validation / evidence

Your work is adequate when you can point to all of the following:

• one risk stated without an unverified causal claim;
• one operational question tied to a decision;
• a clear population and time boundary;
• a minimum set of signals that maps directly to the question;
• at least one symptom that is not presented as proof of cause;
• evidence that includes enough context for investigation;
• no dashboard, alert, or metric included solely because it is conventional or easy to collect.

A reviewer should be able to remove any proposed signal and ask what part of the question becomes unanswerable. If nothing changes, the signal is not yet justified.

1563. Key takeaways

• Start observability design with an operational question, not a dashboard.
• Separate questions, signals, symptoms, and evidence.
• Tie the question to a population, time boundary, and decision.
• A symptom indicates that investigation may be needed; it does not establish the cause.
• Collect the minimum evidence that allows the team to answer and act.

1564. Next lesson

Continue to 4.3 L2 — Design a small observability plan.

1565. Knowledge check