Skip to content

Incident-response analytics: metrics and Grafana dashboard catalogue ​

Research date: 2026-09-10. Status: dashboards 01–10 and the section 5 corrections were implemented on 2026-09-10. Dashboards 11–13 are not implemented; they need additional data sources.

This document maps Google SRE, DORA and incident-management practices to actual nxs-anomaly events. It explains the design rather than reporting on a running installation. SQL and every panel query were executed against ClickHouse using a synthetic stream. Production KPI values and real-volume performance have not been measured, and the dashboards were not imported into Grafana in that study.

Recommendation: ten dashboards using the existing Enterprise stream and three extensions with new sources. Correct semantics and calculation quality before splitting the former shared dashboard JSON.

Sections 2, 5–7 retain the original assessment and implementation criteria. Where they describe proposed work already listed as completed in section 0, section 0 is the implementation-status summary; the original design rationale is retained for review.

0. What has been implemented ​

Implemented on 2026-09-10. A real ClickHouse 24.8 instance was tested with a synthetic stream covering section 7: 22 value assertions, three rollup assertions, and all 120 dashboard-panel queries passed.

Completed:

  • Dashboards 01–10 in docs/enterprise/grafana/, one file per dashboard, sharing team / service / severity variables on every panel where the dimension actually exists.
  • Views nxs_episode_dimensions, nxs_episode_fact, nxs_notification_fact, nxs_group_state_latest, nxs_episode_objectives, nxs_response_attainment, nxs_analytics_health, and the nxs_response_objectives table.
  • Section 5 corrections: a matured cohort instead of only respondents; mergeable quantile states alongside computed percentiles; superseded instead of permanent backlog; resolution / actor_kind on episode facts; delivery episode_id / team_id; separate firing / resolving; coverage report_id; stage-order guards and negative-interval flags; dirty-day response-rollup rebuilds for episodes that changed.
  • Two product changes needed for correct SQL: purpose on notification.delivery_attempted distinguishes pages from resolve, shift-change, test and log notifications; alert_group.silenced with reason=maintenance is emitted when a group is silenced at creation.

Not completed, and why:

  • 11–13 (user impact, reviews and DORA) require an incident registry, SLI aggregates and CI/CD events. They cannot be invented from an alert journal; section 3 explains this boundary.
  • oncall_intervals and analytics_checkpoints in section 6 need historical shifts and heartbeat events, which the product does not publish.
  • There is no envelope installation_id; until added, every installation uses a separate database.
  • Delivery completeness cannot represent notifications that never reached an attempt: there is no notification-created event for every scheduled notification. Panel descriptions state this limitation explicitly.

1. Good practice ​

Metrics must answer different questions ​

  • Does the platform deliver pages? Measure attempts, notification outcomes, first successful-send latency and missing recipients.
  • How does response proceed? Measure ACK time, objectives, unfinished work, escalations and reopening.
  • Has the service become more reliable for users? That requires SLI/SLO and verified impact/recovery intervals, not simply closed alerts.

MTTR is ambiguous: recovery, repair and resolution use different boundaries. Name the interval in panel titles; use abbreviations only with definitions. Atlassian: incident metrics.

Google SRE notes that rare, heterogeneous incidents make MTTR trends highly variable by chance. Medians and p95 do not remove that problem. Use distributions for investigation, include sample size and individual cases, and evaluate a process change against the stage it targets. An MTTR chart alone does not prove improvement. Google SRE: Incident Metrics in SRE.

Alerting quality can use precision, recall, detection time and reset time. Precision/recall require independent knowledge of significant incidents; an alert journal alone is insufficient. Google SRE: Alerting on SLOs.

On-call analysis should consider interruption frequency and distribution alongside time on duty. Sent-message count is not engineering effort or the number of times somebody was woken. Google SRE: Being On-Call.

DORA's five software-delivery metrics include failed deployment recovery time, which concerns failed deployments. Substituting the closure time of all alert groups is invalid. Compare within one service's context; avoid ranking individuals or incomparable teams. DORA: software delivery performance metrics.

Calculation rules ​

  1. Response grain is (alert_group_id, episode_id): an alert-handling episode, not necessarily a business incident. One outage can generate several groups; one group may have several episodes.
  2. Separate opened in the period, closed in the period and active at T cohorts. State the cohort on every panel.
  3. Show mean, p50, p90/p95, N and unknown count. p99 requires enough volume. For small N show individual cases and a small-sample label, without a green/red trend verdict. Agree a threshold for each metric rather than claiming an industry standard.
  4. NULL means unknown or unreached, not zero. Negative intervals indicate clock/attribution defects; show them separately instead of clamping silently.
  5. Never average precomputed percentiles. Attainment is summed numerators divided by summed denominators, not the average of team percentages.
  6. Include overdue unfinished cases. “Fast among respondents” is not the share of cases answered on time.
  7. Fix clocks/timezone: elapsed 24×7 time in UTC by default. Business-hours SLA and night interruptions require a calendar and IANA timezone.
  8. Filter by installation, team, integration, service, environment, severity, channel and chain only where each dimension is actually available.
  9. Agree targets by severity and service class. Current 300-second ACK and 3600-second close targets are chart defaults, not industry norms.
  10. Separate production, tests, drills, maintenance, automated and human actions. Missing classification is unknown, not inferred from absent ACK.

2. The actual nxs-anomaly data foundation ​

Stream: PostgreSQL transactional outbox → Kafka → Vector → ClickHouse. Analytics is Enterprise. Full coverage requires Kafka, consumer and KAFKA_ANALYTICS_LIFECYCLE_ENABLED=true (chart: kafkaConsumer.analytics.lifecycleEvents=true). It is off by default; earlier history lacks the full lifecycle.

The following table records the original assessment; completed semantic fixes are listed in section 0.

SourceAvailable dataLimits identified in the assessment
nxs_events_raw_deduplicatedv2 envelope, event_id, timestamps, identifiers, payloadRead instead of physical raw for KPIs; at-least-once delivery
nxs_group_episodesopened, first ACK/resolve/successful page, reopening, steps, silences, timestampsNot complete current state; service/environment/severity were not ready dimensions
nxs_delivery_attemptsnotification_id, user_id, integration_id, channel, attempt/notification status, provider code, retry, duration_ms, wait_secondsAssessment found episode_id/team_id missing from the view despite being present in newer raw events; no notification.created for every planned notification
nxs_schedule_coveragesampled overall/problematic schedules, gap_count, attached, disabled, ghost_usersNo actual on-call intervals or future-gap durations
nxs_daily_alert_qualitydaily accepted events and encountered groupsNot new-episode count; includes resolved payloads
nxs_daily_delivery_qualitychannel attempts and distributionsNot final success of unique notifications
nxs_daily_response_qualitydaily team percentiles and ratiosFinal quantiles cannot be correctly combined across team-days
nxs_alerts_loglegacy narrow alert.ingested journalNot full lifecycle or a response-KPI foundation

Events: alert.ingested; alert_group.opened, reopened, acknowledged, unacknowledged, silenced, resolved, escalation_step, escalation_exhausted; notification.delivery_attempted; schedule.coverage_reported.

Lifecycle events carry actor_kind, sometimes actor_role, reopen reasons and resolution=source/operator. ACK has no acknowledging person's identifier in the envelope; delivery user_id identifies the recipient. They are not interchangeable.

service, environment, cluster, host, alertname come from alert.ingested labels; lifecycle columns are usually empty. Severity is in payload. Response slices need an unambiguous episode-dimension snapshot, for example first firing labels and opening severity with a flagged fallback. Joining all alerts to episodes multiplies rows and corrupts KPIs.

Schema and semantics were checked against:

  • Analytics event names.
  • Event envelope and emission.
  • ClickHouse schema and views.
  • Vector mapping.
  • The former docs/grafana-analytics-dashboard.json: 23 data panels, excluding row dividers; split into ten dashboards and removed.
  • Pipeline description.

3. Metric dictionary ​

Notation: o = episode opening; a = first valid ACK; p = first successful paging send; r = first closure; s = source-reported start; T = observation time. Every mean uses the valid observations of that specific interval as its denominator and displays N.

Metric / panel titleFormulaAvailability and interpretation
Time to acknowledge / MTTAa − o; mean over valid episodesmtta_seconds; ACK kinds combined. Human ACK requires verified actor_kind and clarification of automated-client provenance
Time to close alert episoder − o; mean/p50/p95Current mttr_seconds; label this interval explicitly, not service recovery
Time to first successful pagep − oApproximation time_to_page_seconds; clarify channel/purpose and exclude resolve/test/log events
Page-to-ACKa − p, only o ≤ p ≤ aDerived after page-semantics correction; does not prove the message was read
ACK-to-closer − a, only o ≤ a ≤ rDerived; not active engineering effort
Source-to-open lago − sdetection_seconds; not full MTTD because s need not be user-impact start
ACK within targettimely ACK count / eligible episodes whose deadline expiredMatured denominator logic described below
Close within targetanalogous for rWorkflow closure target, not an availability SLO
Active backlog / agecurrent unclosed groups; T − o for the current episodeReconstruct state, reopen/unack/silence; resolved_at IS NULL alone is insufficient
Reopen shareopenings with reopened / all openings in periodSeparate new_alert and unresolved; not proof of recurring root cause
Alert grouping factoraccepted firing events / unique nonempty groups with firing in same windowRaw SQL; exclude resolved_without_group. Not precision or eliminated false positives
Attempt success ratedelivered / (delivered + failed)Available; show skips separately. Adapter/provider success, not confirmed reading
Observed notification successunique notification_id with successful attempt / ids with known terminal outcomeDerived for observed notifications; no-attempt queues absent. Show pending separately
Retry burdenrepeated real calls / all real callsUse attempt and outcome; a skip is not a network call
Provider duration / enqueue-to-attempt-finishduration_ms; occurred_at − notification_created_atLatter includes waiting, retry/backoff and invocation duration, not pure queue time
Escalation exhaustion shareexhausted episodes / opened cohortAccount for cohort maturity; does not prove any recipient was reached
Paging loadunique (user_id, group_id, episode_id) with successful paging sendRaw-derived potential interruptions, not actual awakenings
Forecast coverage degradationshare/count of schedules with detected future gaps at a checkSamples; not covered on-call-hour percentage
Pipeline processing lagingested_at − occurred_atApproximation until Vector transform; ingested_at precedes actual INSERT and may change on replay

Response objectives use a cohort whose waiting window has completed:

text
eligible = episodes whose policy requires ACK
matured  = eligible opening cohort where o + target <= T
on_time  = matured episodes where o <= a <= o + target
attainment = count(on_time) / count(matured)

No ACK after deadline is a violation; a deadline not yet reached is pending, neither success nor violation. Historical slices exclude ACK after T. Source-closed-before-ACK is a separate outcome; exclusion requires an agreed policy, never a retrospective choice to improve percentages. Apply the same rule to close targets. Missing history is unknown/incomplete, not an automatic team violation.

Metrics requiring new facts:

MetricMissing data
True MTTDimpact_started_at, detected_at and an independent significant-incident registry
Time to mitigate / MTTMverified mitigated_at and a start definition
Time to restore service / MTTR recoveryservice_restored_at, impact_started_at and business-incident link
Availability, error budget, burn rategood/total SLI events or actual unavailability intervals; alert correlation is not SLI
MTBFindependent confirmed failures and operational exposure; inter-alert intervals are different
Alert precision / recallactionable/false_positive labels and incidents alerting missed
Pages per on-call hour, coverage %historical shifts/overrides and coverage requirements
Responsible person's time to engage, handoffsactor_id, assignment/ownership history and notification/step correlation
Postmortem and remediation qualityincident_id, action-item deadlines/status, causes and recurrence
DORAdeployment/commit history, failed-deployment markers, recovery and incident↔deployment links

MTTF is not part of the core response set: it better suits nonrepairable components than an alert-handling process.

4. Dashboard catalogue ​

P0 = first priority; P1 = deeper operations analysis; P2 = new sources/events. “Current stream” means sufficient event types, not correct old views without section 5's fixes.

01. Response overview · P0 ​

Audience / decision: SRE lead or service owner; choose services and stages for investigation, not rank employees.

Panels: episode openings by severity; ACK/close attainment with N/pending; mean/p50/p95 time-to-ACK and time-to-close; reopen share; exhausted episodes; source/operator outcomes; current-backlog age link; freshness banner.

Visuals: stat + sparkline, weekly trend, service table with N, outcome distribution. Source: episodes plus normalised dimensions and raw outcomes. Window: 7/30/90 days, opening cohort. Drill-down: 02, 03, 06, 09. Readiness: current stream plus new semantics.

02. Response latency and objectives · P0 ​

Decision: locate delay before paging, after paging or after ACK.

Panels: time-to-page, MTTA, page-to-ACK, ACK-to-close, time-to-close; duration distributions and longest-case table; source-to-open lag with valid-s coverage; matured/on-time/late/pending/unknown; severity objectives; separate ACK actor kinds and source/operator closure.

Visuals: histogram/heatmap, time series, violations table. Source: episodes, delivery raw and episode dimensions. Filters: team, service, severity, integration, objective, completion kind. Do not sum stage p95s into an overall p95: intervals must be combined per episode, and not every episode traverses the stages in that order. Readiness: current stream plus corrected target denominator/page semantics.

03. Active backlog and aging · P1 ​

Decision: groups requiring attention now or at T.

Panels: open/acknowledged/silenced, oldest age and buckets, overdue ACK, exhausted-but-open, unacknowledged events, reopened current episodes and groups without expected delivery.

Visuals: stat, age bars, group-linked table. Source: reconstructed raw group state and current episode. Read openings before the selected range: last-24-hour openings alone miss old backlog. Show insufficient historical depth separately. ClickHouse is eventually consistent: check current UI/API before operational action. Readiness: current stream; incomplete maintenance and silence-expiry events need snapshots/reconciliation for exact live state.

04. Alert volume and signal quality · P1 ​

Decision: rules/integrations generating excess events and pages.

Panels: separate firing/resolved; new episodes versus encountered groups; alerts-per-group; top alertname/service; minute bursts; reopen reasons; source-resolved-without-ACK; missing key labels; potential recurrence by stable rule/service fingerprint.

Visuals: stacked series, Pareto/top-N, hour×day heatmap. Source: raw alert.ingested plus episode facts, by event time. Source closure without ACK and frequent reopen are investigation candidates, not automatically false positives. Readiness: current stream; precision/recall/actionability require classification and an incident registry.

05. Notification delivery · P0 ​

Decision: channel/transport preventing pages from arriving.

Panels: delivered/failed/skipped attempts; attempt success; terminal unique notification_id outcomes and pending; retry burden; provider p50/p95/p99; enqueue-to-attempt-finish; first-success time; failure classes by channel; notification_status after errors; no successful send before episode deadline.

Visuals: stat, distributions, channel×outcome table, retry trend. Source: delivery view and raw episode/team. Window: 1 hour/24 hours/7 days. Separate skips, logs, resolve lifecycle messages and actual pages. Complete delivery coverage requires notification.created and terminal events even for notifications that never reached an attempt. Readiness: current attempts; delivery completeness is partial.

06. Routing and escalation · P1 ​

Decision: chains delaying response or ending without a result.

Panels: executed steps by kind; WAIT separate from notification steps; repeat_count; exhaustion share/count; steps until first ACK; inter-step time; high-fan-out chains; successful sends/ACK alongside steps.

Visuals: chain bars, route/chain table, episode timeline. Source: raw escalation_step/exhausted, episodes and delivery. step is a WAIT/REPEAT-inclusive index, not people or support tiers. Exact “step → recipient → ACK” conversion requires step_execution_id and explicit links; temporal proximity proves nothing. Readiness: stream supports chain behaviour; exact attribution needs extensions.

07. On-call load · P1 ​

Decision: reduce interruptions and balance load.

Panels: unique paged episodes per recipient; pages per episode; recipients affected; hour/weekday distribution; closely spaced pages; night/weekend share once a calendar is defined.

Visuals: heatmap, load distribution, team trend. Source: delivery raw (user_id, group_id, episode_id). Ids may be pseudonymised; agree mapping and access. Per-duty-hour normalisation requires historical shifts; without it do not judge fairness. Night uses recipient timezone, not ClickHouse timezone. The recipient is not the ACK author. Readiness: proxy load from current stream; normalisation needs a new source.

08. Schedule coverage · P1 ​

Decision: fix known future gaps before the next incident.

Panels: latest schedules_total/degraded/degraded_attached; problematic schedule_id/gap_count; disabled/ghost_users; latest-check age; observed degraded duration; sample-based share of degraded checks.

Visuals: stat, state timeline, schedules list. Source: schedule.coverage_reported / coverage view. ARRAY JOIN repeats overall counts: take them once per source report rather than summing schedule rows. gap_count is not gap minutes. A missing sample is unknown; do not carry state forward forever. Readiness: stream available; covered-hour share needs intervals.

09. Episode explorer · P1 ​

Decision: explain a delay and verify aggregates against primary events.

Panels: episode/group/integration; observed lifecycle; escalation steps; attempt outcomes/durations; ACK/resolution kinds; alerts; previous/next episodes; occurred/published/ingested; completeness and ordering ambiguity.

Visuals: state timeline and millisecond event table. Source: deduplicated raw, exact group/episode, bounded LIMIT. Construct group-UI links from a configurable base URL after verifying frontend routes. Do not display entire payloads by default: legacy alert labels/title lack lifecycle-data redaction. Readiness: current stream; ACK-author name is absent.

10. Analytics data quality · P0, deploy first ​

Decision: whether other panels can be interpreted.

Panels: last arrival per event type; latest coverage sample; event-to-Vector lag from ingested_at; late arrivals by occurred day; lifecycle completeness since enabling; missing opening/episode/team/service; negative intervals; ACK/resolve without opening; superseded episodes; physical rows versus unique event_id in diagnostic window; rollup rebuilt_at.

Visuals: statuses, lag histogram, series, defects table. Source: raw/semantic views/rollups. Pre-merge physical duplicate ratio is point-in-time diagnostics, not a stable replay-history KPI. No alerts may be normal: use an expected heartbeat, not only now − max(occurred_at). Completed-event lag cannot detect events stuck before ClickHouse. Kafka lag, outbox backlog and Vector rejected/errors belong to operational Prometheus dashboards; a strictly ClickHouse-only version needs heartbeat/checkpoint/health events or metric export. Readiness: partial; complete end-to-end coverage needs extensions.

11. Service impact and recovery · P2 ​

Decision: actual harm and service restoration time.

Panels: confirmed incidents/severity; detect/mitigate/restore time; affected users/requests; user-impact minutes; availability SLI; error budget; overlapping impact intervals and linked nxs-anomaly episodes.

Source: new incident/impact feed plus ClickHouse SLI aggregates. Union overlapping per-service downtime intervals rather than summing all alerts. Customer-minutes require time-varying affected-user counts, not a single count multiplied by the entire incident. Visuals: SLI trend, impact timeline, incident↔episodes table.

12. Incident learning · P2 ​

Decision: whether causes are addressed and commitments completed.

Panels: eligible incidents with postmortems; time-to-review; open/overdue/ completed action items; on-time completion; confirmed recurrence; causes and contributing factors.

Source: incident registry plus tracker/postmortem feed into ClickHouse. Preserve due-date/status history so moving a deadline cannot improve old KPIs. Visuals: aging table, completion trend, cause Pareto. Automatic reopened does not establish repeated root cause.

13. Deployment reliability / DORA · P2 ​

Decision: relationship between software changes, failures and recovery.

Panels: deployment frequency, change lead time, failed deployment recovery time, change fail rate, deployment rework rate; incident↔deployment links and share without confirmed links.

Source: CI/CD and VCS events plus verified incident-registry links in ClickHouse. Context is service, production environment and rollout. Visuals: five trends and failed-deploy→recovery table. Proximity is not causation; change fail rate requires all production deployments as denominator.

5. Corrections identified before implementation ​

These are code/SQL findings from the original assessment, not production KPI measurements. Implemented corrections are listed in section 0.

FindingWhy it affects interpretationProposed correction
Old panels 7, 8, 20, 22 use avg(p95_...); panel 8 also averages within_targetMean team p95 is not combined p95; small/large teams receive equal weightEpisode calculation or mergeable quantile states after dedup; ratios use summed counts
Daily response denominator includes only acknowledged/closed episodesOverdue unanswered cases disappearMatured eligible cohort with late/pending/unknown
mttr_seconds = resolved − opened; detection_seconds = opened − source_startedClosure does not prove recovery; source time does not prove impact startAccurate panel labels; real incident timeline later
first_delivered_at uses any delivered attemptLogs/resolve notifications resemble pages; provider success is not human contactNormalise purpose; filter channel/reason and episode boundaries
Backlog filters opened_at and tests resolved_at IS NULLMisses old work; reopen after ACK can leave old episode unresolved foreverLatest group/current episode; supersede previous episode
First acknowledged_at remains after unackFirst ACK is not current acknowledged stateLatest operation for state; first ACK for MTTA
Old episode gets no separate terminal event on reopenCompleted-response and backlog denominators differ incorrectlyepisode.superseded or infer next opening; outcome separate from resolved
Source/operator closure combinedAutomatic closure resembles fast human responseresolution and actor_kind in episode facts
Team filter not applied to all old JSON panelsChosen team displays global delivery/load dataJoin team through episode/raw or label panels global
Delivery view loses team_id/episode_id; load uses uniq group_idRepeated episodes and contacts collapseExpose snapshot ids; consistent paging-load grain
Multi-topic producer but single chart consumer topicGroup events may not be co-observed with deliveryCover every used topic; verify id correlation/completeness
Legacy no-episode events use anyLast by occurred_at,event_idTies/no sequence cause ambiguity; late old delivery affects fallbackPrefer explicit episode; flag unresolved legacy attribution; add monotonic entity sequence later
Episode views lack service/environment/severityFiltering empty lifecycle labels loses eventsUnambiguous dimension snapshot and quality flag
Raw TTL and rebuildDays=3Late closure of old episode leaves its opening-day rollup staleDirty-day rebuild for affected episodes, retain complete episode facts
published_at is outbox creation; ingested_at is Vector transformNeither is actual Kafka publish/ClickHouse commitHonest labels; broker/checkpoint/commit timestamps for detailed SLI

Aggregation example: team A has one response at 1000 s, B has 99 at 1 s. Average team p95 is 500.5 s; nearest-rank p95 of all 100 observations is 1 s. A's 1/1 successes and B's 0/99 average to 50%, but combined attainment is 1%.

The assessment also found creation-time silenceForMaintenance after opening without a corresponding emitGroupEvent(...silenced...). A silenced event alone could not exclude all maintenance episodes. Explicit suppression metadata and, where needed, state snapshots were required; section 0 records the added event.

6. Data preparation and Grafana ​

The original proposed datasets below are design descriptions. Section 0 lists which have since been implemented with nxs_ names.

DatasetGrain / purpose
episode_factinstallation×group×episode; stages, outcome, next_episode_at, completeness, dimensions, policy version
group_state_asofgroup state at T; separate latest operational view
notification_factinstallation×notification_id; purpose, episode, team, user, created_at, first_success, terminal outcome, attempts
response_objectivesservice/severity/valid_from/valid_to; ACK/close targets, eligibility, calendar
oncall_intervalshistorical user/team/schedule shifts and overrides for normalisation
analytics_checkpointsinstallation/topic/partition, heartbeat, checkpoint and watermark

Multi-installation is not automatic: the envelope lacks installation_id. Use a separate database/datasource per installation until it is added to every key and dimension; probabilistic id uniqueness is not a substitute.

ReplacingMergeTree removes duplicates during background merges; reading its physical table does not guarantee deduplication. Use FINAL views or explicit equivalent deduplication. Incremental aggregates over raw at-least-once data can retain duplicates forever. ClickHouse: ReplacingMergeTree.

Build episode transitions before cohort filtering. Filtering raw openings first loses ACK/resolve outside that interval. Historical-as-of excludes occurred events after T; decide whether to show knowledge at T or revised history including late arrivals.

Use Grafana's official ClickHouse datasource, DateTime64 time macros and time series with a time column. SQL-quote string filters and align enumerations with multi-select/All semantics. Grafana: ClickHouse query editor.

Use folder nxs-anomaly / Incident analytics, shared variables, seconds/percent/count units, numerator/denominator descriptions, overview → filtered detail → episode explorer navigation, and refresh aligned with lag. Do not colour No data / incomplete green or connect missing samples into continuous lines. Grafana's ClickHouse user must be read-only with bounded query time; the team variable is not authorization. Configure row/view access separately from dashboard filters.

7. Rollout and acceptance criteria ​

  1. Metric/fact contract: agree grain, close/recovery terminology, eligibility, source/operator, deadlines, superseded, purpose and timezone.
  2. Data quality (10): verify lifecycle, all topics, history, dedup, retention, negative times and opening completeness.
  3. First package (01, 02, 05): build overview, response and delivery on verified facts; compare sampled cases with PostgreSQL/UI.
  4. Diagnostics (03, 04, 06–09): backlog, noise, escalation, load, schedules, explorer. Do not fabricate missing dimensions.
  5. Impact/learning (11–13): incident/SLI, tracker and CI/CD are separate integrations; introduce recovery and DORA only afterwards.

Before trusting dashboard KPIs, verify:

  • repeated event_id does not change values;
  • 1/1 and 0/99 combine to 1%, not 50%; total p95 comes from observations;
  • unanswered after deadline is late, before deadline is pending;
  • source closure without ACK never becomes human acknowledgment;
  • ACK→unack and ACK→reopen reconstruct current state correctly;
  • superseded old episodes never create permanent backlog;
  • resolution outside the selected window updates its opening cohort;
  • resolution older than rebuildDays recalculates the old response rollup;
  • pipeline outage is incomplete data, not “no incidents”;
  • resolve/log/skip cannot improve first-page latency;
  • team/service filters affect all corresponding panels;
  • absent source time, unknown timezone and missing denominator are explicit.

These were the original implementation criteria. The research itself did not change the then-current JSON, schema or product code; the subsequent completed implementation and tests are recorded in section 0.

Dashboard downloads ​

nxs-anomaly · Documentation