High availability (HA)
This document describes the reference nxs-anomaly HA topology, the failures it survives, and the drill that verifies it. Recovery from backup after an entire site is lost is covered in BACKUP_RESTORE.md. HA and DR complement one another: HA keeps the service running after a node failure; DR brings it back after data or site loss.
Principle
The application is stateless. API and worker processes may be terminated at any time; all information is in PostgreSQL. Workers coordinate through the database itself: notification claims and escalation-shard advisory locks. Two or more workers are safe, and there is no single scheduler as a failure point.
Therefore, service HA means PostgreSQL HA plus application replicas in multiple zones.
┌──────── zone A ────────┐ ┌──────── zone B ────────┐ ┌── zone C ──┐
Ingress ───▶ │ api ×1 worker ×1 │ │ api ×1 worker ×1 │ │ api ×1 │
│ frontend ×1 │ │ frontend ×1 │ │ │
└───────────┬────────────┘ └───────────┬────────────┘ └─────┬──────┘
│ DSN: primary only (target_session_attrs) │
┌───────────▼────────────┐ ┌───────────▼────────────┐
│ PostgreSQL primary │══│ PostgreSQL sync standby│ + WAL archive (PITR)
└────────────────────────┘ └────────────────────────┘Installation
helm install nxs-anomaly oci://registry.nixys.ru/<helm-project>/nxs-anomaly-enterprise \
-f values-production.yaml -f values-ha.yaml \
--set existingSecret.name=nxs-anomaly-env \
--set networkPolicy.extraIngress[0].namespaceSelector.matchLabels.kubernetes\.io/metadata\.name=monitoringvalues-ha.yaml configures:
| Component | Replicas | PDB minAvailable | Placement |
|---|---|---|---|
| API | 3 | 2 | across zones (ScheduleAnyway), strictly across nodes (DoNotSchedule + nodeTaintsPolicy: Honor) |
| Worker | 2 | 1 | the same |
| Frontend | 2 | 1 | the same |
Strict placement across nodes is safe only together with nodeTaintsPolicy: Honor and matchLabelKeys: [pod-template-hash] (Kubernetes ≥ 1.27). The latter counts only pods from the current revision: otherwise rolling updates place new pods relative to old ones, and removing the old pods can leave two API replicas on one node. Without the taint policy, a drained or failed node still counts as a domain with zero pods, and an evicted pod's replacement cannot be scheduled on the remaining nodes: it remains Pending throughout maintenance or the outage.
The preset replaces failed-node pods after 30 s (tolerations for node.kubernetes.io/unreachable and not-ready), rather than the default 300 s. Requests the Service sends to the dead node before it is marked NotReady (~40–50 s, controlled by the cluster's node-monitor-grace-period) fail. Sender retries cover them: in kind this was 14 of 1612 requests, with no lost alerts.
The preset also configures a 120 s database startup wait, connection rotation every 10 minutes, and a pool health check every 15 s (see failover behaviour below).
PostgreSQL
PostgreSQL must support automatic failover. The DSN (NXS_ANOMALY_DB_DSN in the Secret) must point only to the primary.
| Option | DSN |
|---|---|
| CloudNativePG | postgres://nxs_anomaly@<cluster>-rw.<ns>.svc:5432/nxs_anomaly?sslmode=require&target_session_attrs=primary |
| Patroni / multiple addresses | postgres://nxs_anomaly@pg1:5432,pg2:5432,pg3:5432/nxs_anomaly?sslmode=require&target_session_attrs=primary |
| Managed (with standby) | the provider's writer endpoint plus target_session_attrs=primary |
target_session_attrs=primary is useful even behind CloudNativePG's -rw service: if its endpoints briefly lag behind a primary change, a new connection will not remain on a standby.
PgBouncer is supported only with session pooling (see SETUP.md).
Synchronous replication and RPO
| Replication | RPO after primary failure | Cost |
|---|---|---|
Synchronous (synchronous_standby_names; CNPG: minSyncReplicas: 1) | 0 for every alert acknowledged with HTTP 2xx | Write latency equals the RTT to the standby. Writes wait without a live standby, so at least 2 standbys are needed (CNPG: instances: 3) |
| Asynchronous | replication lag (usually < 1 s, longer under load) | Recently acknowledged alerts may be lost |
A 2xx response tells the sender it may forget the alert. Synchronous replication is therefore recommended for production. With asynchronous replication, an alert accepted just before primary failure may not reach the new primary, and Alertmanager will not retry it.
Failure matrix
| Failure | What happens | Loss |
|---|---|---|
| API pod | Ingress sends traffic to the others. The PDB prevents draining below 2 | none |
| Worker pod | The other worker takes the shards on its next cycle. The reaper releases the failed worker's claims | none; delay ≤ one cycle |
| Node or zone | As above, with pods spread across zones | none |
| PostgreSQL primary | The operator promotes a standby. Until one is available, ingest returns 503 + Retry-After, not 500; senders retry | none with synchronous replication |
| Old primary returns as standby | Its connections are reset on the first write error (see below) | none |
| Kafka / ClickHouse (analytics) | The outbox accumulates in PostgreSQL and drains after recovery. Alerts and escalations do not depend on analytics | none |
| Entire site lost | DR applies: BACKUP_RESTORE.md | backup RPO (≤ 5 min with PITR) |
Application behaviour during failover
- Connection errors and
57P01/08xxxare classified as transient; ingest returns 503 + Retry-After. This behaviour already existed. 25006 read_only_sql_transaction(a write reaches a standby) is also transient. The first such error resets the pool (at most once every 5 s); all connections reopen through the DSN and reach the new primary. Otherwise connections to the demoted node would survive untilMaxConnLifetime: standby pings succeed, so pool health checks do not discard them.- Shard advisory locks are not acquired on standbys. PostgreSQL grants advisory locks on hot standbys too. A worker retaining a lock connection to a demoted node could “hold” a shard there while another worker acquired it on the primary, duplicating processing.
TryAdvisoryLockcheckspg_is_in_recovery()and refuses the lock on a standby. - LISTEN connections are not established on standbys. A standby does not receive the primary's NOTIFY messages; otherwise workers would rely solely on interval polling until restarted.
What the application does not do: fence the old primary. If both nodes accept writes after a network partition (split brain), the HA manager must prevent that (Patroni through its DCS; CNPG through the Kubernetes API).
Drill: tests/failover_drill.sh
CI runs this as test:failover-drill (Docker-in-Docker, on tags and changes to internal/store/**). The release retains failover-drill.json as an artifact.
primary + synchronous streaming standby
→ API and worker on a multi-host DSN (target_session_attrs=primary)
→ continuous ingest; sender retries non-2xx for up to 300 s (like Alertmanager)
→ FAILURE: docker kill primary → after 10 s detection delay, pg_promote
→ measure: readiness database=ok and worker=ok (RTO), and first 2xx (ingest downtime)
→ verify: EVERY alert acknowledged with HTTP 2xx exists on the new primaryThe drill fails if any acknowledged alert is missing, the sender never receives 2xx, or RTO exceeds the budget (120 s by default).
bash tests/failover_drill.sh
NXS_ANOMALY_DRILL_REPORT=/tmp/failover.json bash tests/failover_drill.sh # with reportParameters: NXS_ANOMALY_FAILOVER_DETECT_DELAY (detection delay, 10 s), NXS_ANOMALY_FAILOVER_RTO_BUDGET (120 s), NXS_ANOMALY_FAILOVER_SENDER_RETRY (300 s).
Scope. The drill verifies application behaviour during a primary change. The actual HA manager has its own detection and promotion times. Acceptance on the target site requires a normal switchover followed by a forced failover through the operator under tests/loadtest load, checking loss and readiness as in drill step 7.
Targets
| Metric | Target | Evidence |
|---|---|---|
| RPO after primary failure | 0 (synchronous replication) | failover-drill.json: lost = 0 |
| RTO after primary failure | ≤ 2 min (including detection) | failover-drill.json: rto_seconds |
| RPO after site loss | ≤ 5 min | pitr-drill.json, BACKUP_RESTORE.md |
| RTO after site loss | ≤ 30 min | restore-drill.json, BACKUP_RESTORE.md |
Runbook: planned switchover
- Confirm readiness is green and the standby is
sync(pg_stat_replication). - Switch over through the operator (
kubectl cnpg promote/patronictl switchover). - Watch
nxs_anomaly_ingest_errors_totaland thedb_pool_reset_standbylog. A few seconds of 503 responses are expected and covered by sender retries. - Check readiness (
databaseandworker= ok) and that notification queues drain (nxs_anomaly_oldest_pending_delivery_age_seconds,nxs_anomaly_notifications_retry_queue_depth).