Phase 3 of 3 in the IQLSteps 10-15 · Observation

Late-Game

«How does it behave in the real world?»

Observation phase: focused on monitoring and ensuring reliability in production

Lead role
QA + DevOps
Focus
Observation
Approaches integrated
Shift-Right · Chaos Engineering · Production Monitoring · AI Ops
Tools
Sentry / Grafana / Datadog · Progressive delivery / feature flags · k6 · UptimeRobot (or equivalent uptime monitor)
View the full IQLPractice in the Workspace
~$

Late-Game: where you are inside the IQL

The Integrated Quality Lifecycle runs through the 15 steps in three phases. Each one has its question, its leading role and its own way of looking at quality.

  1. Early-Game

    «Let's build it right from the start»

    Prevention — QA Analyst

    COMPLETED
    Scope
    Steps 1-4
    Approaches
    Shift-Left, BDD, Risk-Based
    Main role
    QA Analyst
    Tools
    Jira (or equivalent tracker), Confluence (or team wiki), API client (Postman, Bruno, curl), Agentic harness (Claude Code, OpenCode, Codex)

  2. Mid-Game

    «Does the software meet the requirements?»

    Detection — QA Automation Engineer

    COMPLETED
    Scope
    Steps 5-9
    Approaches
    Continuous Testing, Agile Testing, AI-Driven
    Main role
    QA Automation Engineer
    Tools
    Playwright, GitHub Actions (or equivalent CI), Docker, Xray

  3. Late-Game

    «How does it behave in the real world?»

    Observation — QA + DevOps

    CURRENT PHASE
    Scope
    Steps 10-15
    Approaches
    Shift-Right, Chaos Engineering, Production Monitoring, AI Ops
    Main role
    QA + DevOps
    Tools
    Sentry / Grafana / Datadog, Progressive delivery / feature flags, k6, UptimeRobot (or equivalent uptime monitor)

Agentic contract

The stages of this phase

The stage is what gets executed and gated: it has a name, never a number. Each one declares what the agent does, what the person signs, what evidence it demands and with how much autonomy. The steps stay inside, as its breakdown.

  1. Close and release

    Regression

    Late-GameL3 · Continuous Detection

    The suite runs and every failure gets classified: none stays as "it broke". From there comes the release verdict and the sprint's STR.

    The contract, the guardrails, the deliverables and the breakdown1 step

    Agentic contract

    The agent does
    Runs the suite, classifies every failure (REGRESSION, FLAKY, KNOWN, ENVIRONMENT or NEW TEST), calculates pass-rate and trend without truncating, proposes GO, CAUTION or NO-GO and builds the STR.
    The person signs
    Decides every CAUTION. And never lets the sprint number get invented: if it's missing, it gets asked.
    Evidence required
    Run report with the classification of every failure, a history of at least five runs to be able to call something FLAKY (with fewer, it's declared insufficient history) and the STR linked to the STP.

    Autonomy

    3 on a 0-5 scale

    Separate verifier

    not needed

    Owning skill
    regression-testing
    Guardrails that govern it
    ReadApprovalSecretsValidationTraceabilityCheckpoints
    Roles
    QA Automation EngineerQA LeadExecutorGuardian

    Comes out of here

    • STR
    • Regression report with each failure classified
    • GO / CAUTION / NO-GO verdict
    Steps it is made of · 1

    1. Step 10: Continuous Maintenance

      Late-GameProduction Ops

      Keep the application stable for release and keep it that way after deployment. The sprint's regression is planned in the STP and its execution is recorded in the STR.

      Deliverable

      US deployed to production with confidence + the sprint's STR (Sprint Test Results)

      Activities
      Regression suiteSmoke testingTest maintenance
      Tools
      PlaywrightXrayGitHub Actions (or equivalent CI)
      Boilerplate
      Regression/regression-testing

  2. Production

    Observation

    Late-GameL4 · Production ObservationPanorama

    Production as the source of the strategy. It doesn't have a skill yet: its operating unit is the agentic routine, and that's why the stage is declared panorama until one exists.

    The contract, the guardrails, the deliverables and the breakdown5 steps

    Agentic contract

    The agent does
    Watches the signals the product already emits (service objectives, error budget, real user monitoring, canary cohorts), opens backlog items when a signal falls out of range and feeds what it learned back into the next cycle's analysis.
    The person signs
    Defines the service objectives and the error budget policy, and decides when a release gets stopped. No threshold gets inherited from someone else's benchmark.
    Evidence required
    The product's own metrics against the targets that product declared; industry benchmarks count as a reference, never as a target.

    Autonomy

    3 on a 0-5 scale

    Separate verifier

    not applicable yet

    Owning skill
    no skill yet: the operating unit is the agentic routine
    Guardrails that govern it
    ReadApprovalSecretsValidationCheckpoints
    Roles
    QA LeadGuardian

    Comes out of here

    • Backlog items opened from production
    • Product metrics against its own targets
    Steps it is made of · 5

    1. Step 11: Canary Release Monitoring

      Late-GameShift-RightPanorama

      Roll new functionality out to a small percentage of users to monitor behaviour

      Deliverable

      Safe validation with minimal risk

      Tools
      Sentry / Grafana / DatadogProgressive delivery / feature flags
      Boilerplate
      no stage: the boilerplate doesn't cover Shift-Right

    2. Step 12: A/B Testing

      Late-GameExperimentationPanorama

      Test different versions of a feature to optimise the user experience

      Deliverable

      Decisions based on real user data

      Tools
      Progressive delivery / feature flags
      Boilerplate
      no stage: the boilerplate doesn't cover Shift-Right

    3. Step 13: Real User Monitoring (RUM)

      Late-GameObservabilityPanorama

      Monitor the real experience of users in production to spot performance problems

      Deliverable

      Full visibility into real UX

      Metrics
      Core Web VitalsMTTDMTTR
      Tools
      k6Sentry / Grafana / DatadogUptimeRobot (or equivalent uptime monitor)
      Boilerplate
      no stage: the boilerplate doesn't cover Shift-Right

    4. Step 14: Chaos Engineering

      Late-GameResiliencePanorama

      Introduce controlled failures in production to validate the resilience of the system

      Deliverable

      Resilient system with validated recovery

      Tools
      Chaos tools (Gremlin, Litmus, custom scripts)
      Boilerplate
      no stage: the boilerplate doesn't cover Shift-Right

    5. Step 15: Feedback Loop

      Late-GameContinuous LearningPanorama

      Analyse user feedback and production metrics to feed the next Early-Game cycle

      Deliverable

      Continuous product improvement

      Tools
      Jira (or equivalent tracker)Slack / TeamsSentry / Grafana / Datadog
      Boilerplate
      no stage: the boilerplate doesn't cover Shift-Right

The agentic routine is the operating unit for stages that don't fit inside a session: it runs on its own, on a schedule or on a trigger, and its product is evidence with a recipient. It's what lets Regression and Observation exist without anyone sitting down to watch them.

  1. 01

    Trigger

    What wakes it up: a schedule, a push, a crossed threshold. Explicit and singular; a routine without a named trigger is a script somebody runs by hand.

  2. 02

    Scoped permissions

    What it can read and what it can write, and nothing else. It is the `read`/`write` guardrail instantiated for this routine, not the agent's general trust level.

  3. 03

    Evidence

    What it leaves behind when it finishes, whether it runs well or not: the command, the output, the artifact. Without reproducible evidence there is no conclusion, not even when nobody is watching.

  4. 04

    Report

    Who it talks to and through which channel. A routine that does not report to a person is not autonomy: it is a silenced alarm.

Steps 10-15

The 6 steps of Late-Game

Every step with its lifecycle stage, its deliverable, its Jira transition, and the note that prevents the typical misunderstanding. It is the full detail of the syllabus, without summarizing.

  1. Step 10: Continuous Maintenance

    Late-GameProduction Ops

    Keep the application stable for release and keep it that way after deployment. The sprint's regression is planned in the STP and its execution is recorded in the STR.

    Deliverable

    US deployed to production with confidence + the sprint's STR (Sprint Test Results)

    Activities
    Regression suiteSmoke testingTest maintenance
    Boilerplate
    Regression/regression-testing

    Tools

    • PlaywrightRECOMMENDEDImplementation standard

      Default when UPEX automates from scratch: the boilerplate and KATA's layer 2 assume it. It isn't mandatory, because the IQL doesn't require migrations without technical justification: an existing mature stack (Cypress, WebdriverIO) gets evaluated at the start and is kept, adapted, or migrated in stages.

    • XrayADAPTABLEImplementation standard

      The TMS mode UPEX operates with (`jira-xray`): the story's coverage comes natively from the ATS. The boilerplate starts in `jira-native` (`tms_cli: null`) and that mode is supported; another TMS (TestRail, Zephyr) gets evaluated as an adaptation. The initial assessment decides the mode, not the method.

    • GitHub Actions (or equivalent CI)CLIENT-CONTROLLEDThe environment brings it

      The CI comes from the client (GitHub Actions, GitLab CI, Jenkins). The standard requires the ATCs to run on every PR; the boilerplate's reference pipeline is written for GitHub Actions.

    Maturity

    Continuous Detection · suggested adoption: # 1

    PrerequisiteGreen CI with a regression suite that runs on every PR.

  2. Step 11: Canary Release Monitoring

    Late-GameShift-RightPanorama

    Roll new functionality out to a small percentage of users to monitor behaviour

    Deliverable

    Safe validation with minimal risk

    Boilerplate
    no stage: the boilerplate doesn't cover Shift-Right

    Tools

    • Sentry / Grafana / DatadogCLIENT-CONTROLLEDThe environment brings it

      Late Game observes with whatever the client already has. The IQL defines what to observe; the tool is defined by the environment. Prerequisite for the Production Observation level.

    • Progressive delivery / feature flagsCLIENT-CONTROLLEDThe environment brings it

      Prerequisite for canary and A/B. Without this, steps 11 and 12 don't get implemented.

    Maturity

    Production Observation · suggested adoption: # 4

    PrerequisiteProgressive rollout (canary or feature flags) with metrics per cohort.

  3. Step 12: A/B Testing

    Late-GameExperimentationPanorama

    Test different versions of a feature to optimise the user experience

    Deliverable

    Decisions based on real user data

    Boilerplate
    no stage: the boilerplate doesn't cover Shift-Right

    Tools

    • Progressive delivery / feature flagsCLIENT-CONTROLLEDThe environment brings it

      Prerequisite for canary and A/B. Without this, steps 11 and 12 don't get implemented.

    Maturity

    Production Observation · suggested adoption: # 5

    PrerequisiteFeature flags and product analytics.

  4. Step 13: Real User Monitoring (RUM)

    Late-GameObservabilityPanorama

    Monitor the real experience of users in production to spot performance problems

    Deliverable

    Full visibility into real UX

    Metrics
    Core Web VitalsMTTDMTTR
    Boilerplate
    no stage: the boilerplate doesn't cover Shift-Right

    Tools

    • k6ADAPTABLEImplementation standard

      Only when the performance attribute enters by risk in Step 1. It is not part of the Dojo and is not promised in the offer to companies.

    • Sentry / Grafana / DatadogCLIENT-CONTROLLEDThe environment brings it

      Late Game observes with whatever the client already has. The IQL defines what to observe; the tool is defined by the environment. Prerequisite for the Production Observation level.

    • UptimeRobot (or equivalent uptime monitor)CLIENT-CONTROLLEDThe environment brings it

      Any equivalent uptime monitor.

    Maturity

    Production Observation · suggested adoption: # 2

    PrerequisiteError tracking and RUM (Core Web Vitals) in production.

  5. Step 14: Chaos Engineering

    Late-GameResiliencePanorama

    Introduce controlled failures in production to validate the resilience of the system

    Deliverable

    Resilient system with validated recovery

    Boilerplate
    no stage: the boilerplate doesn't cover Shift-Right

    Tools

    • Chaos tools (Gremlin, Litmus, custom scripts)CLIENT-CONTROLLEDThe environment brings it

      Prerequisite for step 14.

    Maturity

    Production Observation · suggested adoption: # 6

    PrerequisiteDistributed architecture, defined SLOs and an SRE team.

  6. Step 15: Feedback Loop

    Late-GameContinuous LearningPanorama

    Analyse user feedback and production metrics to feed the next Early-Game cycle

    Deliverable

    Continuous product improvement

    Boilerplate
    no stage: the boilerplate doesn't cover Shift-Right

    Tools

    • Jira (or equivalent tracker)CLIENT-CONTROLLEDThe environment brings it

      The tracker belongs to the client. The IQL requires a workflow with the states and transitions of `iql.jiraStates`; today the Implementation Standard only has Jira implemented (`.agents/jira-workflows.json`), so Azure DevOps or another tracker is adaptation work evaluated at the start.

    • Slack / TeamsCLIENT-CONTROLLEDThe environment brings it

      CI notification and alert channel. The IQL doesn't depend on which one.

    • Sentry / Grafana / DatadogCLIENT-CONTROLLEDThe environment brings it

      Late Game observes with whatever the client already has. The IQL defines what to observe; the tool is defined by the environment. Prerequisite for the Production Observation level.

    Maturity

    Production Observation · suggested adoption: # 3

    PrerequisiteAt least one production source (RUM or alerts) and a ritual that feeds what's learned back into step 1.

Maturity

How far to go

This phase doesn't get adopted whole: every step needs a prerequisite from the environment, and the maturity level says which ones make sense today. The highlighted levels are the ones that contain steps from the Late-Game.

  1. L1 · Steps 1, 2

    Prevention

    Requires
    A backlog in a tracker and QA sitting in refinement.
    Outcome
    Better-defined stories.

  2. L2 · Steps 3, 4

    Early Detection

    Requires
    A test environment before the merge (QA or staging).
    Outcome
    Feedback before the PR.

  3. L3 · Steps 5-10

    Continuous Detection

    Requires
    A repository with pull requests and a CI that runs on every PR.
    Outcome
    Reliable regression in CI.

  4. L4 · Steps 11-15

    Production Observation

    Requires
    Production observability; progressive rollout for canary and A/B; SLOs and an SRE culture for chaos.
    Outcome
    Production feeding the strategy.

Measurement

The 6 Late-Game metrics

The earlier phases are evaluated by what they produce: a plan, a case, a merged pull request. This one is evaluated by what happens in production once nobody is watching anymore, and that is only known with numbers and a goal declared in advance.

  • MTTD

    Mean Time To Detect

    How long it takes the team to find out that something broke in production.

    Target< 5 minutesreference benchmark

    Everything that happens before detection is impact nobody is containing.

  • MTTR

    Mean Time To Resolution

    How long it takes to resolve a problem once it gets detected.

    Target< 30 minutesreference benchmark

    It's the half of the incident that actually shortens with process, not luck.

  • Error Rate

    Application Error Rate

    What percentage of requests ends in an error.

    Target< 0.1%reference benchmark

    It's the most direct indicator of stability, and it moves before any complaint does.

  • CSAT

    Customer Satisfaction Score

    The satisfaction declared by the people who use the product.

    Target> 4.5/5reference benchmark

    The only one on the list that doesn't measure the system, but the perception of the system.

  • SLO Compliance

    Service Level Objective Compliance

    What portion of the time the committed service objectives get met.

    Target> 99.9%reference benchmark

    Turns availability into a number that can be negotiated against.

  • Performance Score

    Core Web Vitals Score

    Perceived performance, measured with Google's web metrics.

    Target> 90/100reference benchmark

    Affects ranking and conversion, not just the feeling of speed.

`lateGame` targets are reference benchmarks (`targetKind: example`), not universal goals: a bank and an internal HR app don't need the same SLO. Each implementation declares its own in `implementationTargets` based on criticality, baseline, architecture and business.

The method metrics

  • Change failure rate

    Change failure rate

    Percentage of deployments that end in an incident, rollback, or hotfix. It's the DORA metric that says whether the method is sustaining velocity or just pushing it.

  • Escape rate

    Escape rate to production

    Bugs found in production over the total bugs of the period. It measures the one thing the method promises: that the defect gets found earlier.

  • Ratio Deferred

    Proportion of deferred scenarios

    Share of the explored scenarios that the ROI gate leaves out of regression. It's the health of the filter: a suite grows healthy when most of what got explored does NOT get persisted.

    AlarmIf more than half fall into Candidate or Manual, the filter wasn't applied: the discard phase runs again.

  • Flaky rate

    Flaky test rate

    Tests that change their result without the code changing, over the total suite. It's what decides whether the team trusts the regression.

  • Lead time AC → ATP

    Optional

    Time from acceptance criterion to its plan

    How long a story with criteria takes to have its acceptance plan written. It measures whether Shift-Left arrives on time or arrives late.

Integrated approaches

What lens Late-Game works with

The IQL does not apply the same approaches across the whole cycle. These are the ones the syllabus assigns to this phase, and they explain why the work here does not look like the other two.

  • Shift-Right

    Extend quality validation into production, where the system meets real users.

  • Chaos Engineering

    Introducing controlled failures to validate resilience before the failure arrives on its own.

  • Production Monitoring

    Continuous observability of the system in production, to catch anomalies early.

  • AI Ops

    Artificial intelligence applied to telemetry: predictive analysis and anomaly detection.

Tools

What you work with in Late-Game

The same pieces the syllabus assigns to the phase. They change from one phase to another because the work changes: analyzing does not need the same as automating, nor the same as monitoring production.
  • Sentry / Grafana / Datadog logo

    Sentry / Grafana / Datadog

    Production errors with their trace, and the dashboards where the metrics are read together. That is where the Error Rate and much of the MTTD come from.

  • Progressive delivery / feature flags logo

    Progressive delivery / feature flags

    Percentage rollouts and feature flags: what makes it possible to watch a cohort before everyone else.

  • k6 logo

    k6

    Load and performance testing against the real system, not a mock.

  • UptimeRobot (or equivalent uptime monitor) logo

    UptimeRobot (or equivalent uptime monitor)

    Availability measured from the outside: the check that supports the service objective.

Master the Late-Game

Become the QA + DevOps that companies need: the one who works quality starting from observation and knows how to answer «How does it behave in the real world?».

See available courses
See the whole IQLImplement it with UPEX

The IQL is practiced end to end in the DOJO Program, with Jira, the TMS and the real repository.