Skip to content

High availability (HA) ​

This document describes the reference nxs-anomaly HA topology, the failures it survives, and the drill that verifies it. Recovery from backup after an entire site is lost is covered in BACKUP_RESTORE.md. HA and DR complement one another: HA keeps the service running after a node failure; DR brings it back after data or site loss.

Principle ​

The application is stateless. API and worker processes may be terminated at any time; all information is in PostgreSQL. Workers coordinate through the database itself: notification claims and escalation-shard advisory locks. Two or more workers are safe, and there is no single scheduler as a failure point.

Therefore, service HA means PostgreSQL HA plus application replicas in multiple zones.

              ┌──────── zone A ────────┐  ┌──────── zone B ────────┐  ┌── zone C ──┐
 Ingress ───▶ │ api ×1   worker ×1     │  │ api ×1   worker ×1     │  │ api ×1     │
              │ frontend ×1            │  │ frontend ×1            │  │            │
              └───────────┬────────────┘  └───────────┬────────────┘  └─────┬──────┘
                          │   DSN: primary only (target_session_attrs)     │
              ┌───────────▼────────────┐  ┌───────────▼────────────┐
              │ PostgreSQL primary     │══│ PostgreSQL sync standby│   + WAL archive (PITR)
              └────────────────────────┘  └────────────────────────┘

Installation ​

bash
helm install nxs-anomaly oci://registry.nixys.ru/<helm-project>/nxs-anomaly-enterprise \
  -f values-production.yaml -f values-ha.yaml \
  --set existingSecret.name=nxs-anomaly-env \
  --set networkPolicy.extraIngress[0].namespaceSelector.matchLabels.kubernetes\.io/metadata\.name=monitoring

values-ha.yaml configures:

ComponentReplicasPDB minAvailablePlacement
API32across zones (ScheduleAnyway), strictly across nodes (DoNotSchedule + nodeTaintsPolicy: Honor)
Worker21the same
Frontend21the same

Strict placement across nodes is safe only together with nodeTaintsPolicy: Honor and matchLabelKeys: [pod-template-hash] (Kubernetes ≥ 1.27). The latter counts only pods from the current revision: otherwise rolling updates place new pods relative to old ones, and removing the old pods can leave two API replicas on one node. Without the taint policy, a drained or failed node still counts as a domain with zero pods, and an evicted pod's replacement cannot be scheduled on the remaining nodes: it remains Pending throughout maintenance or the outage.

The preset replaces failed-node pods after 30 s (tolerations for node.kubernetes.io/unreachable and not-ready), rather than the default 300 s. Requests the Service sends to the dead node before it is marked NotReady (~40–50 s, controlled by the cluster's node-monitor-grace-period) fail. Sender retries cover them: in kind this was 14 of 1612 requests, with no lost alerts.

The preset also configures a 120 s database startup wait, connection rotation every 10 minutes, and a pool health check every 15 s (see failover behaviour below).

PostgreSQL ​

PostgreSQL must support automatic failover. The DSN (NXS_ANOMALY_DB_DSN in the Secret) must point only to the primary.

OptionDSN
CloudNativePGpostgres://nxs_anomaly@<cluster>-rw.<ns>.svc:5432/nxs_anomaly?sslmode=require&target_session_attrs=primary
Patroni / multiple addressespostgres://nxs_anomaly@pg1:5432,pg2:5432,pg3:5432/nxs_anomaly?sslmode=require&target_session_attrs=primary
Managed (with standby)the provider's writer endpoint plus target_session_attrs=primary

target_session_attrs=primary is useful even behind CloudNativePG's -rw service: if its endpoints briefly lag behind a primary change, a new connection will not remain on a standby.

PgBouncer is supported only with session pooling (see SETUP.md).

Synchronous replication and RPO ​

ReplicationRPO after primary failureCost
Synchronous (synchronous_standby_names; CNPG: minSyncReplicas: 1)0 for every alert acknowledged with HTTP 2xxWrite latency equals the RTT to the standby. Writes wait without a live standby, so at least 2 standbys are needed (CNPG: instances: 3)
Asynchronousreplication lag (usually < 1 s, longer under load)Recently acknowledged alerts may be lost

A 2xx response tells the sender it may forget the alert. Synchronous replication is therefore recommended for production. With asynchronous replication, an alert accepted just before primary failure may not reach the new primary, and Alertmanager will not retry it.

Failure matrix ​

FailureWhat happensLoss
API podIngress sends traffic to the others. The PDB prevents draining below 2none
Worker podThe other worker takes the shards on its next cycle. The reaper releases the failed worker's claimsnone; delay ≤ one cycle
Node or zoneAs above, with pods spread across zonesnone
PostgreSQL primaryThe operator promotes a standby. Until one is available, ingest returns 503 + Retry-After, not 500; senders retrynone with synchronous replication
Old primary returns as standbyIts connections are reset on the first write error (see below)none
Kafka / ClickHouse (analytics)The outbox accumulates in PostgreSQL and drains after recovery. Alerts and escalations do not depend on analyticsnone
Entire site lostDR applies: BACKUP_RESTORE.mdbackup RPO (≤ 5 min with PITR)

Application behaviour during failover ​

  • Connection errors and 57P01/08xxx are classified as transient; ingest returns 503 + Retry-After. This behaviour already existed.
  • 25006 read_only_sql_transaction (a write reaches a standby) is also transient. The first such error resets the pool (at most once every 5 s); all connections reopen through the DSN and reach the new primary. Otherwise connections to the demoted node would survive until MaxConnLifetime: standby pings succeed, so pool health checks do not discard them.
  • Shard advisory locks are not acquired on standbys. PostgreSQL grants advisory locks on hot standbys too. A worker retaining a lock connection to a demoted node could “hold” a shard there while another worker acquired it on the primary, duplicating processing. TryAdvisoryLock checks pg_is_in_recovery() and refuses the lock on a standby.
  • LISTEN connections are not established on standbys. A standby does not receive the primary's NOTIFY messages; otherwise workers would rely solely on interval polling until restarted.

What the application does not do: fence the old primary. If both nodes accept writes after a network partition (split brain), the HA manager must prevent that (Patroni through its DCS; CNPG through the Kubernetes API).

Drill: tests/failover_drill.sh ​

CI runs this as test:failover-drill (Docker-in-Docker, on tags and changes to internal/store/**). The release retains failover-drill.json as an artifact.

primary + synchronous streaming standby
  → API and worker on a multi-host DSN (target_session_attrs=primary)
  → continuous ingest; sender retries non-2xx for up to 300 s (like Alertmanager)
  → FAILURE: docker kill primary → after 10 s detection delay, pg_promote
  → measure: readiness database=ok and worker=ok (RTO), and first 2xx (ingest downtime)
  → verify: EVERY alert acknowledged with HTTP 2xx exists on the new primary

The drill fails if any acknowledged alert is missing, the sender never receives 2xx, or RTO exceeds the budget (120 s by default).

bash
bash tests/failover_drill.sh
NXS_ANOMALY_DRILL_REPORT=/tmp/failover.json bash tests/failover_drill.sh   # with report

Parameters: NXS_ANOMALY_FAILOVER_DETECT_DELAY (detection delay, 10 s), NXS_ANOMALY_FAILOVER_RTO_BUDGET (120 s), NXS_ANOMALY_FAILOVER_SENDER_RETRY (300 s).

Scope. The drill verifies application behaviour during a primary change. The actual HA manager has its own detection and promotion times. Acceptance on the target site requires a normal switchover followed by a forced failover through the operator under tests/loadtest load, checking loss and readiness as in drill step 7.

Targets ​

MetricTargetEvidence
RPO after primary failure0 (synchronous replication)failover-drill.json: lost = 0
RTO after primary failure≤ 2 min (including detection)failover-drill.json: rto_seconds
RPO after site loss≤ 5 minpitr-drill.json, BACKUP_RESTORE.md
RTO after site loss≤ 30 minrestore-drill.json, BACKUP_RESTORE.md

Runbook: planned switchover ​

  1. Confirm readiness is green and the standby is sync (pg_stat_replication).
  2. Switch over through the operator (kubectl cnpg promote / patronictl switchover).
  3. Watch nxs_anomaly_ingest_errors_total and the db_pool_reset_standby log. A few seconds of 503 responses are expected and covered by sender retries.
  4. Check readiness (database and worker = ok) and that notification queues drain (nxs_anomaly_oldest_pending_delivery_age_seconds, nxs_anomaly_notifications_retry_queue_depth).

nxs-anomaly · Documentation