Incident-response analytics: metrics and Grafana dashboard catalogue
Research date: 2026-09-10. Status: dashboards 01–10 and the section 5 corrections were implemented on 2026-09-10. Dashboards 11–13 are not implemented; they need additional data sources.
This document maps Google SRE, DORA and incident-management practices to actual nxs-anomaly events. It explains the design rather than reporting on a running installation. SQL and every panel query were executed against ClickHouse using a synthetic stream. Production KPI values and real-volume performance have not been measured, and the dashboards were not imported into Grafana in that study.
Recommendation: ten dashboards using the existing Enterprise stream and three extensions with new sources. Correct semantics and calculation quality before splitting the former shared dashboard JSON.
Sections 2, 5–7 retain the original assessment and implementation criteria. Where they describe proposed work already listed as completed in section 0, section 0 is the implementation-status summary; the original design rationale is retained for review.
0. What has been implemented
Implemented on 2026-09-10. A real ClickHouse 24.8 instance was tested with a synthetic stream covering section 7: 22 value assertions, three rollup assertions, and all 120 dashboard-panel queries passed.
Completed:
- Dashboards 01–10 in docs/enterprise/grafana/, one file per dashboard, sharing
team/service/severityvariables on every panel where the dimension actually exists. - Views
nxs_episode_dimensions,nxs_episode_fact,nxs_notification_fact,nxs_group_state_latest,nxs_episode_objectives,nxs_response_attainment,nxs_analytics_health, and thenxs_response_objectivestable. - Section 5 corrections: a matured cohort instead of only respondents; mergeable quantile states alongside computed percentiles;
supersededinstead of permanent backlog;resolution/actor_kindon episode facts; deliveryepisode_id/team_id; separatefiring/resolving; coveragereport_id; stage-order guards and negative-interval flags; dirty-day response-rollup rebuilds for episodes that changed. - Two product changes needed for correct SQL:
purposeonnotification.delivery_attempteddistinguishes pages from resolve, shift-change, test and log notifications;alert_group.silencedwithreason=maintenanceis emitted when a group is silenced at creation.
Not completed, and why:
- 11–13 (user impact, reviews and DORA) require an incident registry, SLI aggregates and CI/CD events. They cannot be invented from an alert journal; section 3 explains this boundary.
oncall_intervalsandanalytics_checkpointsin section 6 need historical shifts and heartbeat events, which the product does not publish.- There is no envelope
installation_id; until added, every installation uses a separate database. - Delivery completeness cannot represent notifications that never reached an attempt: there is no notification-created event for every scheduled notification. Panel descriptions state this limitation explicitly.
1. Good practice
Metrics must answer different questions
- Does the platform deliver pages? Measure attempts, notification outcomes, first successful-send latency and missing recipients.
- How does response proceed? Measure ACK time, objectives, unfinished work, escalations and reopening.
- Has the service become more reliable for users? That requires SLI/SLO and verified impact/recovery intervals, not simply closed alerts.
MTTR is ambiguous: recovery, repair and resolution use different boundaries. Name the interval in panel titles; use abbreviations only with definitions. Atlassian: incident metrics.
Google SRE notes that rare, heterogeneous incidents make MTTR trends highly variable by chance. Medians and p95 do not remove that problem. Use distributions for investigation, include sample size and individual cases, and evaluate a process change against the stage it targets. An MTTR chart alone does not prove improvement. Google SRE: Incident Metrics in SRE.
Alerting quality can use precision, recall, detection time and reset time. Precision/recall require independent knowledge of significant incidents; an alert journal alone is insufficient. Google SRE: Alerting on SLOs.
On-call analysis should consider interruption frequency and distribution alongside time on duty. Sent-message count is not engineering effort or the number of times somebody was woken. Google SRE: Being On-Call.
DORA's five software-delivery metrics include failed deployment recovery time, which concerns failed deployments. Substituting the closure time of all alert groups is invalid. Compare within one service's context; avoid ranking individuals or incomparable teams. DORA: software delivery performance metrics.
Calculation rules
- Response grain is
(alert_group_id, episode_id): an alert-handling episode, not necessarily a business incident. One outage can generate several groups; one group may have several episodes. - Separate opened in the period, closed in the period and active at T cohorts. State the cohort on every panel.
- Show mean, p50, p90/p95, N and unknown count. p99 requires enough volume. For small N show individual cases and a small-sample label, without a green/red trend verdict. Agree a threshold for each metric rather than claiming an industry standard.
- NULL means unknown or unreached, not zero. Negative intervals indicate clock/attribution defects; show them separately instead of clamping silently.
- Never average precomputed percentiles. Attainment is summed numerators divided by summed denominators, not the average of team percentages.
- Include overdue unfinished cases. “Fast among respondents” is not the share of cases answered on time.
- Fix clocks/timezone: elapsed 24×7 time in UTC by default. Business-hours SLA and night interruptions require a calendar and IANA timezone.
- Filter by installation, team, integration, service, environment, severity, channel and chain only where each dimension is actually available.
- Agree targets by severity and service class. Current 300-second ACK and 3600-second close targets are chart defaults, not industry norms.
- Separate production, tests, drills, maintenance, automated and human actions. Missing classification is
unknown, not inferred from absent ACK.
2. The actual nxs-anomaly data foundation
Stream: PostgreSQL transactional outbox → Kafka → Vector → ClickHouse. Analytics is Enterprise. Full coverage requires Kafka, consumer and KAFKA_ANALYTICS_LIFECYCLE_ENABLED=true (chart: kafkaConsumer.analytics.lifecycleEvents=true). It is off by default; earlier history lacks the full lifecycle.
The following table records the original assessment; completed semantic fixes are listed in section 0.
| Source | Available data | Limits identified in the assessment |
|---|---|---|
nxs_events_raw_deduplicated | v2 envelope, event_id, timestamps, identifiers, payload | Read instead of physical raw for KPIs; at-least-once delivery |
nxs_group_episodes | opened, first ACK/resolve/successful page, reopening, steps, silences, timestamps | Not complete current state; service/environment/severity were not ready dimensions |
nxs_delivery_attempts | notification_id, user_id, integration_id, channel, attempt/notification status, provider code, retry, duration_ms, wait_seconds | Assessment found episode_id/team_id missing from the view despite being present in newer raw events; no notification.created for every planned notification |
nxs_schedule_coverage | sampled overall/problematic schedules, gap_count, attached, disabled, ghost_users | No actual on-call intervals or future-gap durations |
nxs_daily_alert_quality | daily accepted events and encountered groups | Not new-episode count; includes resolved payloads |
nxs_daily_delivery_quality | channel attempts and distributions | Not final success of unique notifications |
nxs_daily_response_quality | daily team percentiles and ratios | Final quantiles cannot be correctly combined across team-days |
nxs_alerts_log | legacy narrow alert.ingested journal | Not full lifecycle or a response-KPI foundation |
Events: alert.ingested; alert_group.opened, reopened, acknowledged, unacknowledged, silenced, resolved, escalation_step, escalation_exhausted; notification.delivery_attempted; schedule.coverage_reported.
Lifecycle events carry actor_kind, sometimes actor_role, reopen reasons and resolution=source/operator. ACK has no acknowledging person's identifier in the envelope; delivery user_id identifies the recipient. They are not interchangeable.
service, environment, cluster, host, alertname come from alert.ingested labels; lifecycle columns are usually empty. Severity is in payload. Response slices need an unambiguous episode-dimension snapshot, for example first firing labels and opening severity with a flagged fallback. Joining all alerts to episodes multiplies rows and corrupts KPIs.
Schema and semantics were checked against:
- Analytics event names.
- Event envelope and emission.
- ClickHouse schema and views.
- Vector mapping.
- The former
docs/grafana-analytics-dashboard.json: 23 data panels, excluding row dividers; split into ten dashboards and removed. - Pipeline description.
3. Metric dictionary
Notation: o = episode opening; a = first valid ACK; p = first successful paging send; r = first closure; s = source-reported start; T = observation time. Every mean uses the valid observations of that specific interval as its denominator and displays N.
| Metric / panel title | Formula | Availability and interpretation |
|---|---|---|
| Time to acknowledge / MTTA | a − o; mean over valid episodes | mtta_seconds; ACK kinds combined. Human ACK requires verified actor_kind and clarification of automated-client provenance |
| Time to close alert episode | r − o; mean/p50/p95 | Current mttr_seconds; label this interval explicitly, not service recovery |
| Time to first successful page | p − o | Approximation time_to_page_seconds; clarify channel/purpose and exclude resolve/test/log events |
| Page-to-ACK | a − p, only o ≤ p ≤ a | Derived after page-semantics correction; does not prove the message was read |
| ACK-to-close | r − a, only o ≤ a ≤ r | Derived; not active engineering effort |
| Source-to-open lag | o − s | detection_seconds; not full MTTD because s need not be user-impact start |
| ACK within target | timely ACK count / eligible episodes whose deadline expired | Matured denominator logic described below |
| Close within target | analogous for r | Workflow closure target, not an availability SLO |
| Active backlog / age | current unclosed groups; T − o for the current episode | Reconstruct state, reopen/unack/silence; resolved_at IS NULL alone is insufficient |
| Reopen share | openings with reopened / all openings in period | Separate new_alert and unresolved; not proof of recurring root cause |
| Alert grouping factor | accepted firing events / unique nonempty groups with firing in same window | Raw SQL; exclude resolved_without_group. Not precision or eliminated false positives |
| Attempt success rate | delivered / (delivered + failed) | Available; show skips separately. Adapter/provider success, not confirmed reading |
| Observed notification success | unique notification_id with successful attempt / ids with known terminal outcome | Derived for observed notifications; no-attempt queues absent. Show pending separately |
| Retry burden | repeated real calls / all real calls | Use attempt and outcome; a skip is not a network call |
| Provider duration / enqueue-to-attempt-finish | duration_ms; occurred_at − notification_created_at | Latter includes waiting, retry/backoff and invocation duration, not pure queue time |
| Escalation exhaustion share | exhausted episodes / opened cohort | Account for cohort maturity; does not prove any recipient was reached |
| Paging load | unique (user_id, group_id, episode_id) with successful paging send | Raw-derived potential interruptions, not actual awakenings |
| Forecast coverage degradation | share/count of schedules with detected future gaps at a check | Samples; not covered on-call-hour percentage |
| Pipeline processing lag | ingested_at − occurred_at | Approximation until Vector transform; ingested_at precedes actual INSERT and may change on replay |
Response objectives use a cohort whose waiting window has completed:
eligible = episodes whose policy requires ACK
matured = eligible opening cohort where o + target <= T
on_time = matured episodes where o <= a <= o + target
attainment = count(on_time) / count(matured)No ACK after deadline is a violation; a deadline not yet reached is pending, neither success nor violation. Historical slices exclude ACK after T. Source-closed-before-ACK is a separate outcome; exclusion requires an agreed policy, never a retrospective choice to improve percentages. Apply the same rule to close targets. Missing history is unknown/incomplete, not an automatic team violation.
Metrics requiring new facts:
| Metric | Missing data |
|---|---|
| True MTTD | impact_started_at, detected_at and an independent significant-incident registry |
| Time to mitigate / MTTM | verified mitigated_at and a start definition |
| Time to restore service / MTTR recovery | service_restored_at, impact_started_at and business-incident link |
| Availability, error budget, burn rate | good/total SLI events or actual unavailability intervals; alert correlation is not SLI |
| MTBF | independent confirmed failures and operational exposure; inter-alert intervals are different |
| Alert precision / recall | actionable/false_positive labels and incidents alerting missed |
| Pages per on-call hour, coverage % | historical shifts/overrides and coverage requirements |
| Responsible person's time to engage, handoffs | actor_id, assignment/ownership history and notification/step correlation |
| Postmortem and remediation quality | incident_id, action-item deadlines/status, causes and recurrence |
| DORA | deployment/commit history, failed-deployment markers, recovery and incident↔deployment links |
MTTF is not part of the core response set: it better suits nonrepairable components than an alert-handling process.
4. Dashboard catalogue
P0 = first priority; P1 = deeper operations analysis; P2 = new sources/events. “Current stream” means sufficient event types, not correct old views without section 5's fixes.
01. Response overview · P0
Audience / decision: SRE lead or service owner; choose services and stages for investigation, not rank employees.
Panels: episode openings by severity; ACK/close attainment with N/pending; mean/p50/p95 time-to-ACK and time-to-close; reopen share; exhausted episodes; source/operator outcomes; current-backlog age link; freshness banner.
Visuals: stat + sparkline, weekly trend, service table with N, outcome distribution. Source: episodes plus normalised dimensions and raw outcomes. Window: 7/30/90 days, opening cohort. Drill-down: 02, 03, 06, 09. Readiness: current stream plus new semantics.
02. Response latency and objectives · P0
Decision: locate delay before paging, after paging or after ACK.
Panels: time-to-page, MTTA, page-to-ACK, ACK-to-close, time-to-close; duration distributions and longest-case table; source-to-open lag with valid-s coverage; matured/on-time/late/pending/unknown; severity objectives; separate ACK actor kinds and source/operator closure.
Visuals: histogram/heatmap, time series, violations table. Source: episodes, delivery raw and episode dimensions. Filters: team, service, severity, integration, objective, completion kind. Do not sum stage p95s into an overall p95: intervals must be combined per episode, and not every episode traverses the stages in that order. Readiness: current stream plus corrected target denominator/page semantics.
03. Active backlog and aging · P1
Decision: groups requiring attention now or at T.
Panels: open/acknowledged/silenced, oldest age and buckets, overdue ACK, exhausted-but-open, unacknowledged events, reopened current episodes and groups without expected delivery.
Visuals: stat, age bars, group-linked table. Source: reconstructed raw group state and current episode. Read openings before the selected range: last-24-hour openings alone miss old backlog. Show insufficient historical depth separately. ClickHouse is eventually consistent: check current UI/API before operational action. Readiness: current stream; incomplete maintenance and silence-expiry events need snapshots/reconciliation for exact live state.
04. Alert volume and signal quality · P1
Decision: rules/integrations generating excess events and pages.
Panels: separate firing/resolved; new episodes versus encountered groups; alerts-per-group; top alertname/service; minute bursts; reopen reasons; source-resolved-without-ACK; missing key labels; potential recurrence by stable rule/service fingerprint.
Visuals: stacked series, Pareto/top-N, hour×day heatmap. Source: raw alert.ingested plus episode facts, by event time. Source closure without ACK and frequent reopen are investigation candidates, not automatically false positives. Readiness: current stream; precision/recall/actionability require classification and an incident registry.
05. Notification delivery · P0
Decision: channel/transport preventing pages from arriving.
Panels: delivered/failed/skipped attempts; attempt success; terminal unique notification_id outcomes and pending; retry burden; provider p50/p95/p99; enqueue-to-attempt-finish; first-success time; failure classes by channel; notification_status after errors; no successful send before episode deadline.
Visuals: stat, distributions, channel×outcome table, retry trend. Source: delivery view and raw episode/team. Window: 1 hour/24 hours/7 days. Separate skips, logs, resolve lifecycle messages and actual pages. Complete delivery coverage requires notification.created and terminal events even for notifications that never reached an attempt. Readiness: current attempts; delivery completeness is partial.
06. Routing and escalation · P1
Decision: chains delaying response or ending without a result.
Panels: executed steps by kind; WAIT separate from notification steps; repeat_count; exhaustion share/count; steps until first ACK; inter-step time; high-fan-out chains; successful sends/ACK alongside steps.
Visuals: chain bars, route/chain table, episode timeline. Source: raw escalation_step/exhausted, episodes and delivery. step is a WAIT/REPEAT-inclusive index, not people or support tiers. Exact “step → recipient → ACK” conversion requires step_execution_id and explicit links; temporal proximity proves nothing. Readiness: stream supports chain behaviour; exact attribution needs extensions.
07. On-call load · P1
Decision: reduce interruptions and balance load.
Panels: unique paged episodes per recipient; pages per episode; recipients affected; hour/weekday distribution; closely spaced pages; night/weekend share once a calendar is defined.
Visuals: heatmap, load distribution, team trend. Source: delivery raw (user_id, group_id, episode_id). Ids may be pseudonymised; agree mapping and access. Per-duty-hour normalisation requires historical shifts; without it do not judge fairness. Night uses recipient timezone, not ClickHouse timezone. The recipient is not the ACK author. Readiness: proxy load from current stream; normalisation needs a new source.
08. Schedule coverage · P1
Decision: fix known future gaps before the next incident.
Panels: latest schedules_total/degraded/degraded_attached; problematic schedule_id/gap_count; disabled/ghost_users; latest-check age; observed degraded duration; sample-based share of degraded checks.
Visuals: stat, state timeline, schedules list. Source: schedule.coverage_reported / coverage view. ARRAY JOIN repeats overall counts: take them once per source report rather than summing schedule rows. gap_count is not gap minutes. A missing sample is unknown; do not carry state forward forever. Readiness: stream available; covered-hour share needs intervals.
09. Episode explorer · P1
Decision: explain a delay and verify aggregates against primary events.
Panels: episode/group/integration; observed lifecycle; escalation steps; attempt outcomes/durations; ACK/resolution kinds; alerts; previous/next episodes; occurred/published/ingested; completeness and ordering ambiguity.
Visuals: state timeline and millisecond event table. Source: deduplicated raw, exact group/episode, bounded LIMIT. Construct group-UI links from a configurable base URL after verifying frontend routes. Do not display entire payloads by default: legacy alert labels/title lack lifecycle-data redaction. Readiness: current stream; ACK-author name is absent.
10. Analytics data quality · P0, deploy first
Decision: whether other panels can be interpreted.
Panels: last arrival per event type; latest coverage sample; event-to-Vector lag from ingested_at; late arrivals by occurred day; lifecycle completeness since enabling; missing opening/episode/team/service; negative intervals; ACK/resolve without opening; superseded episodes; physical rows versus unique event_id in diagnostic window; rollup rebuilt_at.
Visuals: statuses, lag histogram, series, defects table. Source: raw/semantic views/rollups. Pre-merge physical duplicate ratio is point-in-time diagnostics, not a stable replay-history KPI. No alerts may be normal: use an expected heartbeat, not only now − max(occurred_at). Completed-event lag cannot detect events stuck before ClickHouse. Kafka lag, outbox backlog and Vector rejected/errors belong to operational Prometheus dashboards; a strictly ClickHouse-only version needs heartbeat/checkpoint/health events or metric export. Readiness: partial; complete end-to-end coverage needs extensions.
11. Service impact and recovery · P2
Decision: actual harm and service restoration time.
Panels: confirmed incidents/severity; detect/mitigate/restore time; affected users/requests; user-impact minutes; availability SLI; error budget; overlapping impact intervals and linked nxs-anomaly episodes.
Source: new incident/impact feed plus ClickHouse SLI aggregates. Union overlapping per-service downtime intervals rather than summing all alerts. Customer-minutes require time-varying affected-user counts, not a single count multiplied by the entire incident. Visuals: SLI trend, impact timeline, incident↔episodes table.
12. Incident learning · P2
Decision: whether causes are addressed and commitments completed.
Panels: eligible incidents with postmortems; time-to-review; open/overdue/ completed action items; on-time completion; confirmed recurrence; causes and contributing factors.
Source: incident registry plus tracker/postmortem feed into ClickHouse. Preserve due-date/status history so moving a deadline cannot improve old KPIs. Visuals: aging table, completion trend, cause Pareto. Automatic reopened does not establish repeated root cause.
13. Deployment reliability / DORA · P2
Decision: relationship between software changes, failures and recovery.
Panels: deployment frequency, change lead time, failed deployment recovery time, change fail rate, deployment rework rate; incident↔deployment links and share without confirmed links.
Source: CI/CD and VCS events plus verified incident-registry links in ClickHouse. Context is service, production environment and rollout. Visuals: five trends and failed-deploy→recovery table. Proximity is not causation; change fail rate requires all production deployments as denominator.
5. Corrections identified before implementation
These are code/SQL findings from the original assessment, not production KPI measurements. Implemented corrections are listed in section 0.
| Finding | Why it affects interpretation | Proposed correction |
|---|---|---|
Old panels 7, 8, 20, 22 use avg(p95_...); panel 8 also averages within_target | Mean team p95 is not combined p95; small/large teams receive equal weight | Episode calculation or mergeable quantile states after dedup; ratios use summed counts |
| Daily response denominator includes only acknowledged/closed episodes | Overdue unanswered cases disappear | Matured eligible cohort with late/pending/unknown |
mttr_seconds = resolved − opened; detection_seconds = opened − source_started | Closure does not prove recovery; source time does not prove impact start | Accurate panel labels; real incident timeline later |
first_delivered_at uses any delivered attempt | Logs/resolve notifications resemble pages; provider success is not human contact | Normalise purpose; filter channel/reason and episode boundaries |
| Backlog filters opened_at and tests resolved_at IS NULL | Misses old work; reopen after ACK can leave old episode unresolved forever | Latest group/current episode; supersede previous episode |
| First acknowledged_at remains after unack | First ACK is not current acknowledged state | Latest operation for state; first ACK for MTTA |
| Old episode gets no separate terminal event on reopen | Completed-response and backlog denominators differ incorrectly | episode.superseded or infer next opening; outcome separate from resolved |
| Source/operator closure combined | Automatic closure resembles fast human response | resolution and actor_kind in episode facts |
| Team filter not applied to all old JSON panels | Chosen team displays global delivery/load data | Join team through episode/raw or label panels global |
| Delivery view loses team_id/episode_id; load uses uniq group_id | Repeated episodes and contacts collapse | Expose snapshot ids; consistent paging-load grain |
| Multi-topic producer but single chart consumer topic | Group events may not be co-observed with delivery | Cover every used topic; verify id correlation/completeness |
| Legacy no-episode events use anyLast by occurred_at,event_id | Ties/no sequence cause ambiguity; late old delivery affects fallback | Prefer explicit episode; flag unresolved legacy attribution; add monotonic entity sequence later |
| Episode views lack service/environment/severity | Filtering empty lifecycle labels loses events | Unambiguous dimension snapshot and quality flag |
| Raw TTL and rebuildDays=3 | Late closure of old episode leaves its opening-day rollup stale | Dirty-day rebuild for affected episodes, retain complete episode facts |
| published_at is outbox creation; ingested_at is Vector transform | Neither is actual Kafka publish/ClickHouse commit | Honest labels; broker/checkpoint/commit timestamps for detailed SLI |
Aggregation example: team A has one response at 1000 s, B has 99 at 1 s. Average team p95 is 500.5 s; nearest-rank p95 of all 100 observations is 1 s. A's 1/1 successes and B's 0/99 average to 50%, but combined attainment is 1%.
The assessment also found creation-time silenceForMaintenance after opening without a corresponding emitGroupEvent(...silenced...). A silenced event alone could not exclude all maintenance episodes. Explicit suppression metadata and, where needed, state snapshots were required; section 0 records the added event.
6. Data preparation and Grafana
The original proposed datasets below are design descriptions. Section 0 lists which have since been implemented with nxs_ names.
| Dataset | Grain / purpose |
|---|---|
episode_fact | installation×group×episode; stages, outcome, next_episode_at, completeness, dimensions, policy version |
group_state_asof | group state at T; separate latest operational view |
notification_fact | installation×notification_id; purpose, episode, team, user, created_at, first_success, terminal outcome, attempts |
response_objectives | service/severity/valid_from/valid_to; ACK/close targets, eligibility, calendar |
oncall_intervals | historical user/team/schedule shifts and overrides for normalisation |
analytics_checkpoints | installation/topic/partition, heartbeat, checkpoint and watermark |
Multi-installation is not automatic: the envelope lacks installation_id. Use a separate database/datasource per installation until it is added to every key and dimension; probabilistic id uniqueness is not a substitute.
ReplacingMergeTree removes duplicates during background merges; reading its physical table does not guarantee deduplication. Use FINAL views or explicit equivalent deduplication. Incremental aggregates over raw at-least-once data can retain duplicates forever. ClickHouse: ReplacingMergeTree.
Build episode transitions before cohort filtering. Filtering raw openings first loses ACK/resolve outside that interval. Historical-as-of excludes occurred events after T; decide whether to show knowledge at T or revised history including late arrivals.
Use Grafana's official ClickHouse datasource, DateTime64 time macros and time series with a time column. SQL-quote string filters and align enumerations with multi-select/All semantics. Grafana: ClickHouse query editor.
Use folder nxs-anomaly / Incident analytics, shared variables, seconds/percent/count units, numerator/denominator descriptions, overview → filtered detail → episode explorer navigation, and refresh aligned with lag. Do not colour No data / incomplete green or connect missing samples into continuous lines. Grafana's ClickHouse user must be read-only with bounded query time; the team variable is not authorization. Configure row/view access separately from dashboard filters.
7. Rollout and acceptance criteria
- Metric/fact contract: agree grain, close/recovery terminology, eligibility, source/operator, deadlines, superseded, purpose and timezone.
- Data quality (10): verify lifecycle, all topics, history, dedup, retention, negative times and opening completeness.
- First package (01, 02, 05): build overview, response and delivery on verified facts; compare sampled cases with PostgreSQL/UI.
- Diagnostics (03, 04, 06–09): backlog, noise, escalation, load, schedules, explorer. Do not fabricate missing dimensions.
- Impact/learning (11–13): incident/SLI, tracker and CI/CD are separate integrations; introduce recovery and DORA only afterwards.
Before trusting dashboard KPIs, verify:
- repeated event_id does not change values;
- 1/1 and 0/99 combine to 1%, not 50%; total p95 comes from observations;
- unanswered after deadline is late, before deadline is pending;
- source closure without ACK never becomes human acknowledgment;
- ACK→unack and ACK→reopen reconstruct current state correctly;
- superseded old episodes never create permanent backlog;
- resolution outside the selected window updates its opening cohort;
- resolution older than rebuildDays recalculates the old response rollup;
- pipeline outage is incomplete data, not “no incidents”;
- resolve/log/skip cannot improve first-page latency;
- team/service filters affect all corresponding panels;
- absent source time, unknown timezone and missing denominator are explicit.
These were the original implementation criteria. The research itself did not change the then-current JSON, schema or product code; the subsequent completed implementation and tests are recorded in section 0.