Skip to content

Prometheus Alerting Rules ​

Example alerting rules for monitoring nxs-anomaly. Adapt the thresholds to your own load.

rules.yml ​

yaml
groups:
  - name: nxs-anomaly
    interval: 60s
    rules:

      # Ingest errors, by source.
      - alert: IngestErrorsHigh
        expr: >
          sum by (source) (
            rate(nxs_anomaly_ingest_errors_total[5m])
          ) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High ingest error rate from {{ $labels.source }}"
          description: >
            {{ printf "%.2f" $value }} errors/s from source {{ $labels.source }}.
            Check integration key validity and payload format.

      # Notifications keep failing.
      - alert: NotificationFailuresHigh
        expr: >
          rate(nxs_anomaly_notifications_failed_total[10m]) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High notification failure rate on {{ $labels.instance }}"
          description: >
            {{ printf "%.2f" $value }} permanent failures/s. Check delivery
            adapter credentials (Telegram, SMTP, Asterisk).

      # Delivery is slow — the retry queue is large.
      - alert: NotificationRetryQueueDeep
        expr: nxs_anomaly_notifications_retry_queue_depth > 200
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Notification retry queue backlog on {{ $labels.instance }}"
          description: >
            {{ $value }} notifications in retry queue. Delivery adapters may
            be slow or unreachable.

      # No worker cycle for N minutes.
      # NOTE: this compares against the timestamp of the last cycle, NOT its
      # duration. An earlier version subtracted duration_seconds (0.05s, say)
      # from time() and therefore fired permanently. The metric
      # *_last_cycle_timestamp_seconds exists precisely for stall detection.
      - alert: WorkerCycleStuck
        expr: >
          time() - nxs_anomaly_worker_last_cycle_timestamp_seconds > 120
          and nxs_anomaly_worker_last_cycle_timestamp_seconds > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Worker cycle not completing on {{ $labels.instance }}"
          description: >
            Last worker cycle completed {{ $value | humanizeDuration }} ago.
            Escalations and notifications may be delayed. Check the worker's
            /ready endpoint and logs.

      # The database is unreachable from the process (worker or API): the ping
      # fails. This is the root cause rather than a symptom — a stuck cycle or a
      # growing backlog are the symptoms.
      - alert: DatabaseUnavailable
        expr: nxs_anomaly_db_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Database unreachable from {{ $labels.instance }}"
          description: >
            The last DB ping from this process failed. Ingest, escalation and
            delivery all stop until the database is reachable again.

      # The escalation backlog is ageing: the worker cannot keep groups moving.
      - alert: EscalationBacklogAge
        expr: nxs_anomaly_oldest_due_escalation_age_seconds > 120
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Escalation backlog aging on {{ $labels.instance }}"
          description: >
            Oldest due alert group is {{ $value | humanizeDuration }} past its
            next_run_at. Worker is behind: check it is running and DB is healthy.

      # The delivery backlog is ageing: notifications are stuck in the queue.
      - alert: DeliveryBacklogAge
        expr: nxs_anomaly_oldest_pending_delivery_age_seconds > 120
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Delivery backlog aging on {{ $labels.instance }}"
          description: >
            Oldest delivery_scheduled notification is {{ $value | humanizeDuration }}
            old. Delivery is falling behind ingest.

      # Claims are being reclaimed: workers are dying mid-delivery, or the
      # claim timeout is set too aggressively.
      - alert: StaleClaimsReclaimed
        expr: increase(nxs_anomaly_stale_claims_reclaimed_total[15m]) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Stale delivery claims being reclaimed on {{ $labels.instance }}"
          description: >
            {{ $value }} notifications were reclaimed from a stale 'delivering'/
            'retrying' claim. A non-zero rate means workers crash mid-delivery or
            the claim timeout is too short for the provider latency.

      # Many open alert groups — possibly an incident storm.
      - alert: AlertGroupsBurst
        expr: nxs_anomaly_alert_groups_open > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High number of open alert groups on {{ $labels.instance }}"
          description: >
            {{ $value }} open groups. This may indicate an alert storm.

      # Delivery errors per provider — this is what identifies a broken channel.
      - alert: DeliveryProviderErrors
        expr: >
          sum by (provider) (
            rate(nxs_anomaly_delivery_errors_by_provider_total[10m])
          ) > 0.02
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Delivery errors on {{ $labels.provider }}"
          description: >
            {{ printf "%.3f" $value }} errors/s on provider {{ $labels.provider }}.

      # The worker is behind: cycles are skipped because the previous one is still running.
      - alert: WorkerCyclesSkipped
        expr: rate(nxs_anomaly_worker_cycles_skipped_total[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Worker cycles being skipped on {{ $labels.instance }}"
          description: >
            The worker is behind the load — a cycle starts while the previous
            one is still running. Raise the concurrency or the poll interval, or
            move the worker into separate replicas.

      # A process is gone, not degraded. Every rule above reads metrics the
      # process itself exports, so a dead API or worker silently takes them
      # with it; only the scrape result says so. The job label is the Service
      # name under the chart's ServiceMonitor (<release>-nxs-anomaly-api,
      # -worker) or your job_name in a static config — adjust the regex if you
      # use fullnameOverride.
      - alert: NxsAnomalyTargetDown
        expr: up{job=~".*nxs-anomaly.*"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "nxs-anomaly target {{ $labels.job }} down on {{ $labels.instance }}"
          description: >
            Prometheus cannot scrape this nxs-anomaly process. If it is the
            worker, nobody is being paged. Route this alert around nxs-anomaly.

      # No worker anywhere completes cycles: all are down, crash-looping or not
      # scraped at all. "> 0" matters: an API started with --no-scheduler
      # exports this gauge too, at 0, and would otherwise mask the absence.
      - alert: WorkerAbsent
        expr: absent(nxs_anomaly_worker_last_cycle_timestamp_seconds > 0)
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "No nxs-anomaly worker is completing cycles"
          description: >
            No scraped process reports a completed worker cycle. Escalation and
            delivery are stopped. Route this alert around nxs-anomaly.

      # Always firing, by design. It proves the Prometheus → Alertmanager →
      # receiver path works: send it to an external dead-man's switch, which
      # alarms when it STOPS arriving. Never route it into nxs-anomaly.
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Alerting pipeline heartbeat (always firing)"
          description: >
            Send to a dead-man's-switch receiver with a short repeat_interval.
            Its silence means Prometheus or Alertmanager is down.

      # A delivery channel has degraded in latency (p95), possibly without errors.
      - alert: DeliveryLatencyHigh
        expr: >
          histogram_quantile(0.95,
            sum by (channel, le) (
              rate(nxs_anomaly_notification_delivery_duration_seconds_bucket[10m])
            )
          ) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow delivery on channel {{ $labels.channel }}"
          description: >
            p95 delivery latency on channel {{ $labels.channel }} =
            {{ printf "%.1f" $value }}s. The provider is alive but slow.

      # Paging-latency SLI: pages leave the queue late (SLO: p95 < 10 s).
      - alert: PagingLatencyHigh
        expr: >
          histogram_quantile(0.95,
            sum by (channel, le) (
              rate(nxs_anomaly_notification_first_attempt_delay_seconds_bucket[10m])
            )
          ) > 10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pages leave late on channel {{ $labels.channel }}"
          description: >
            p95 from queued to first delivery attempt = {{ printf "%.1f" $value }}s.
            The worker is behind: check cycle duration and the delivery backlog.

      # Responder SLI: acknowledge/resolve is slow (SLO: p95 < 2 s).
      - alert: RespondLatencyHigh
        expr: >
          histogram_quantile(0.95,
            sum by (le) (
              rate(nxs_anomaly_http_request_duration_seconds_bucket{handler="respond"}[10m])
            )
          ) > 2
          and sum(rate(nxs_anomaly_http_request_duration_seconds_count{handler="respond"}[10m])) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Acknowledge/resolve is slow"
          description: >
            p95 responder action latency = {{ printf "%.1f" $value }}s; the page
            keeps escalating while the responder waits on the button.

      # Notifications are being lost for good (dead letter).
      - alert: DeadLetterEvents
        expr: >
          sum by (channel) (
            rate(nxs_anomaly_dead_letter_events_total[15m])
          ) > 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Dead-letter notifications on channel {{ $labels.channel }}"
          description: >
            Notifications on channel {{ $labels.channel }} exhausted their
            retries and became permanently failed. Check the provider and the
            dead-letter webhook.

A ready-to-apply PrometheusRule manifest with all of these rules is in prometheus-rules.yaml; an importable Grafana dashboard is in grafana-dashboard.json.

Wiring it into Prometheus ​

The API and a separate worker (run-worker) export metrics independently: the worker brings up its own telemetry port (/live, /ready, /metrics, :8081 by default — see NXS_ANOMALY_WORKER_ADDR). Without it, the serve --no-scheduler plus run-worker arrangement would lose every delivery metric: the worker produces them, and the registry used to live only in the API's HTTP server. Scrape both targets.

yaml
# prometheus.yml
scrape_configs:
  - job_name: nxs-anomaly-api
    static_configs:
      - targets: ['nxs-anomaly-api:8080']
    metrics_path: /metrics
    scrape_interval: 15s
  - job_name: nxs-anomaly-worker
    static_configs:
      - targets: ['nxs-anomaly-worker:8081']
    metrics_path: /metrics
    scrape_interval: 15s

Monitoring around nxs-anomaly ​

An alert that nxs-anomaly is broken cannot be delivered through nxs-anomaly: it would arrive at the very system that is down. Route the three rules above — NxsAnomalyTargetDown, WorkerAbsent and Watchdog — through a separate Alertmanager receiver straight to an independent channel, and Watchdog to an external dead-man's switch:

yaml
# alertmanager.yml
route:
  routes:
    - matchers: ['alertname="Watchdog"']
      receiver: deadmans-switch
      repeat_interval: 1m
    - matchers: ['alertname=~"NxsAnomalyTargetDown|WorkerAbsent|DatabaseUnavailable|WorkerCycleStuck"']
      receiver: ops-direct          # Telegram/e-mail/SMS directly, not nxs-anomaly
      continue: false
receivers:
  - name: deadmans-switch
    webhook_configs:
      - url: https://hc-ping.com/<uuid>
  - name: ops-direct
    telegram_configs:
      - chat_id: <id>
        bot_token_file: /etc/alertmanager/telegram-token

Watchdog checks the Prometheus → Alertmanager path, not nxs-anomaly itself. For a signal straight from the worker, set NXS_ANOMALY_WORKER_HEARTBEAT_URL: after a successful cycle the worker sends a GET there, at most once a minute. A cycle that failed on the database sends nothing, so one such check covers the worker, its database and the network — without Prometheus. Give the check a generous period in the external service, 5 minutes or more. Every worker replica pings, so the check goes quiet only when none of them works.

In the Helm chart, set the variable through the application Secret or worker.extraEnv. The URL carries the check's token, so it is never logged.

SLI / SLO ​

The indicators fixed for the beta. The thresholds are a starting point, to be calibrated against your load (see CAPACITY.md).

SLIDefinitionPromQLTarget SLO
Ingest availabilitythe share of ingest webhook requests without a 5xx1 - (sum(rate(nxs_anomaly_http_requests_total{handler="webhook",code=~"5.."}[5m])) / sum(rate(nxs_anomaly_http_requests_total{handler="webhook"}[5m])))≥ 99.9%
Ingest→first-attempt latencyp95 of the delay from accepting an alert to the first delivery attempthistogram_quantile(0.95, sum by (le) (rate(nxs_anomaly_ingest_duration_seconds_bucket[5m]))) + nxs_anomaly_oldest_pending_delivery_age_secondsp95 < 10s
Successful delivery ratio (per provider)the share of successful deliveries per channelsum by (provider) (rate(nxs_anomaly_notifications_delivered_by_provider_total[30m])) / (sum by (provider) (rate(nxs_anomaly_notifications_delivered_by_provider_total[30m])) + sum by (provider) (rate(nxs_anomaly_delivery_errors_by_provider_total[30m])))≥ 99%
Acknowledge API latencyp95 of acknowledge requestshistogram_quantile(0.95, sum by (le) (rate(nxs_anomaly_http_request_duration_seconds_bucket{handler="api"}[5m])))p95 < 2s

A direct measurement of ingest→first-attempt, without decomposing it into two metrics, comes from the load harness: it times the gap between ingest and the first delivery_attempt and prints p50/p95/p99. See CAPACITY.md.

The ingest SLI's label is handler="webhook". That is the category classifyHandler (internal/server/server_internal.go) assigns to the /integrations/v1/ and /v2/alert/ paths; there is no handler="ingest" value, and a query using one silently returns nothing.

Error budget and burn rate ​

Every threshold alert above compares an instantaneous rate against a constant. Such a threshold cannot tell a two-minute spike from a week of slow bleeding: the spike wakes somebody and passes on its own, while the bleeding never crosses the threshold and quietly eats the whole SLO. So the nxs-anomaly.slo group in prometheus-rules.yaml alerts not on errors but on the rate at which the error budget is being spent (Google SRE Workbook, chapter 5).

The budget is 1 - SLO. The burn rate is the observed error ratio divided by the budget. A burn rate of 1 spends a monthly budget in exactly 30 days; 14.4 spends 2% of it in an hour.

TierBurn rateWindowsBudget exhausted inseverity
Fast14.45m and 1h~2 dayscritical — wake somebody
Slow630m and 6h~5 dayswarning — a ticket

Both windows are required at once: the short one gives a fast trigger and a fast reset, and the long one keeps a brief blip from waking anybody.

SLISLOBudgetFast thresholdSlow thresholdAlerts
Ingest availability99.9%0.1%1.44% 5xx0.6% 5xxIngestErrorBudgetBurnFast / Slow
Delivery success (per provider)99%1%14.4% errors6% errorsDeliveryErrorBudgetBurnFast / Slow

The ratios are computed by the recording rules nxs_anomaly:ingest_5xx:ratio_rate* and nxs_anomaly:delivery_errors:ratio_rate*, which double as ready dashboard panels.

What these rules do not catch: with no traffic the ratio is 0/0 = NaN, every comparison against NaN is false, and a quiet instance stays quiet instead of alerting about a division by zero. The flip side is that ingest stopping altogether is invisible to a burn rate; WorkerCycleStuck and DatabaseUnavailable cover that.

DeliverySuccessRatioLow is kept deliberately: it catches a dip below 99% over a 30-minute window, so it fires earlier than the slow tier but without reference to the budget. The burn-rate alerts answer a different question — "will we make it to the end of the month" rather than "is it bad right now".

Grafana dashboard (a quick start) ​

The main panels:

PanelPromQL
Ingested alerts/srate(nxs_anomaly_alerts_ingested_total[1m])
Ingest errors/s by sourcesum by(source)(rate(nxs_anomaly_ingest_errors_total[1m]))
Open alert groupsnxs_anomaly_alert_groups_open
Skipped notificationsnxs_anomaly_notifications_skipped_total
Schedule coveragenxs_anomaly_schedules_degraded
Shifts nobody confirmednxs_anomaly_duty_without_checkin
Sources gone quietnxs_anomaly_sources_silent
Retention deletions by categorysum by(category)(increase(nxs_anomaly_retention_deleted_total[24h]))
Notification retry queuenxs_anomaly_notifications_retry_queue_depth
Delivery errors by providersum by(provider)(rate(nxs_anomaly_delivery_errors_by_provider_total[5m]))
Worker cycle durationnxs_anomaly_worker_cycle_duration_seconds
Time since last worker cycletime() - nxs_anomaly_worker_last_cycle_timestamp_seconds
DB upnxs_anomaly_db_up
Escalation backlog agenxs_anomaly_oldest_due_escalation_age_seconds
Delivery backlog agenxs_anomaly_oldest_pending_delivery_age_seconds
Delivered/s by providersum by(provider)(rate(nxs_anomaly_notifications_delivered_by_provider_total[5m]))
Per-provider success ratiosum by(provider)(rate(nxs_anomaly_notifications_delivered_by_provider_total[30m])) / (sum by(provider)(rate(nxs_anomaly_notifications_delivered_by_provider_total[30m])) + sum by(provider)(rate(nxs_anomaly_delivery_errors_by_provider_total[30m])))
Worker cycles skipped/srate(nxs_anomaly_worker_cycles_skipped_total[5m])
Delivery latency p95 by channelhistogram_quantile(0.95, sum by(channel,le)(rate(nxs_anomaly_notification_delivery_duration_seconds_bucket[5m])))
Dead-letters/s by channelsum by(channel)(rate(nxs_anomaly_dead_letter_events_total[5m]))
Short-circuited deliveries/ssum by(channel)(rate(nxs_anomaly_delivery_short_circuited_total[5m]))

There is no universal alert rule for retention: a zero increase can mean either that the sweep has stopped or that there are no rows past the cutoff. Read it against the horizons you chose and the size of the tables; readiness warns separately when no policy has been chosen. | Ingest latency p95 | histogram_quantile(0.95, rate(nxs_anomaly_ingest_duration_seconds_bucket[5m])) |

nxs-anomaly · Documentation