Skip to content

Kafka analytics integration ​

nxs-anomaly asynchronously sends alerts to Kafka using the Transactional Outbox pattern. Kafka is the data bus for loading ClickHouse or other analytics systems.

Architecture ​

webhook → IngestAlert()
               │
               ▼ (one PostgreSQL transaction)
     ┌──────────────────────────────────┐
     │  nxs_anomaly_alerts              │
     │  nxs_anomaly_alert_groups        │
     │  nxs_anomaly_kafka_outbox  ← NEW │
     └──────────────────────────────────┘
               │
               ▼ (worker cycle, every poll-interval seconds)
     PublishKafkaOutbox()
               │
               ▼
     Kafka topic: nxs-anomaly.alerts
               │
               ▼
     ClickHouse Kafka Engine → MergeTree

Delivery guarantees ​

  • Crash-safe: the outbox row is written atomically with the alert. If the process crashes before publishing, the row stays and is published next cycle.
  • At-least-once: a crash after publish but before deleting the outbox row causes republication. The payload is byte-for-byte identical, so downstream deduplication is possible, but only with an appropriate table (ReplacingMergeTree, below). Plain MergeTree doubles the analytics.
  • No ingest blocking: the Kafka event is written to PostgreSQL in the same batch as the alert; no broker round trip is added to HTTP request latency.

Configuration ​

Environment variables ​

VariableDefaultDescription
KAFKA_BROKERS—Comma-separated brokers, e.g. kafka:9092. Unset disables Kafka.
KAFKA_TOPIC_ALERTSnxs-anomaly.alertsDefault topic for alert.ingested events.
KAFKA_BATCH_SIZE100Maximum producer batch size.
NXS_ANOMALY_ANALYTICS_PSEUDONYMISEfalseReplace user_id and team_id with stable one-way digests before writing the outbox.

Without KAFKA_BROKERS, the outbox is not populated and the worker skips its Kafka step.

Who creates the topic ​

The producer permits the broker to create missing topics. The broker decides: with auto.create.topics.enable=true (Kafka's default and the bundled chart broker's setting), the first publish creates it; with false, publishing fails, the worker logs kafka_publish_failed, and events remain in the outbox until fixed.

Production topics are normally created beforehand to control partitions, replication factor and retention. Automatic creation uses broker defaults.

Historically the producer prohibited automatic topic creation regardless of broker configuration. Installations without a pre-created topic published nothing: nxs_anomaly_kafka_outbox grew and analytics remained empty. This is fixed (see CHANGELOG). After upgrading from an affected release, accumulated rows publish on subsequent worker cycles; do not move events manually.

Topic routing (multi-topic) ​

Each integration may set its own Kafka topic using kafka_topic:

bash
curl -X PUT /api/v1/integrations/{id} \
  -H "Content-Type: application/json" \
  -d '{"kafka_topic": "nxs-anomaly.pagerduty"}'

Without it, global KAFKA_TOPIC_ALERTS applies. Separate source streams (alertmanager, pagerduty, grafana-alerting) can therefore have independent analytics.

Multiple replicas: advisory lock ​

PublishKafkaOutbox takes a session-level PostgreSQL advisory lock (key 72544106) through pg_try_advisory_lock, preventing multiple worker replicas from concurrently publishing the same events.

  • Worker A acquires the lock → reads outbox → publishes → deletes → unlocks.
  • Worker B cannot acquire it → skips the cycle (return 0, nil) → retries next cycle.

Delivery remains at-least-once: a crash between Kafka publish and outbox deletion republishes the event next cycle. The receiver table must accommodate that; see ClickHouse ingest below.

Running ​

bash
# API + built-in worker (development)
KAFKA_BROKERS=localhost:9092 \
KAFKA_TOPIC_ALERTS=nxs-anomaly.alerts \
nxs-anomaly serve --host 0.0.0.0 --port 8080

# Production: API without scheduler
KAFKA_BROKERS=kafka:9092 nxs-anomaly serve --no-scheduler

# Production: separate worker
KAFKA_BROKERS=kafka:9092 nxs-anomaly run-worker --poll-interval 5

Kafka message schema ​

Each topic message is a JSON object:

json
{
  "event_id": "evt_...",
  "event": "alert.ingested",
  "schema_version": 2,
  "occurred_at": "2026-06-06T12:00:00Z",
  "published_at": "2026-06-06T12:00:00Z",
  "entity": {
    "alert_id": "alr_...",
    "alert_group_id": "grp_...",
    "episode_id": "epd_...",
    "integration_id": "int_...",
    "team_id": "",
    "route_id": "rte_...",
    "escalation_chain_id": "esc_...",
    "notification_id": "",
    "user_id": ""
  },
  "data": {},
  "alert": {
    "id": "alr_...",
    "alert_group_id": "grp_...",
    "integration_id": "int_...",
    "status": "firing",
    "severity": "critical",
    "source": "alertmanager",
    "title": "High CPU on web-01",
    "labels": { "host": "web-01", "alertname": "HighCPU" },
    "received_at": "2026-06-06T12:00:00Z"
  },
  "group": {
    "id": "grp_...",
    "status": "open",
    "severity": "critical",
    "alert_count": 3,
    "created_at": "2026-06-06T11:55:00Z",
    "last_received_at": "2026-06-06T12:00:00Z",
    "resolved_at": null
  }
}

group is null for resolved_without_group: a resolved alert received when no active group existed.

Message key: alert.id, providing per-alert partition affinity.

Envelope v2 ​

v2 adds an envelope around the old message. alert and group retain their names, shapes and positions, so existing consumers can continue shaping them. Four fields are new:

FieldMeaning
event_idGenerated once when creating the outbox row and retained in its payload. At-least-once publish can send the same row twice; downstream deduplicates by this id. That works because the stored payload is resent verbatim rather than rebuilt.
occurred_atDomain-transition time. published_at is outbox-row creation time. A Kafka retry an hour later changes neither, distinguishing backlog from slow acknowledgment.
entityAll dimensional identifiers, always with the same keys, even when empty. Consumers map them to fixed columns; intermittently missing keys would give NULL two meanings.
dataEvent-specific fields, filtered through a central denylist: no target, tokens, URLs, email, phone numbers or provider bodies. Analytics is read by more people and retained longer than PostgreSQL delivery diagnostics.

Only alert.ingested carries alert / group; other events do not.

Episodes ​

episode_id identifies one group lifecycle. A group may be worked more than once: a new alert after acknowledgment, or Unresolve after closure, starts a new episode. Otherwise a group reopened four times becomes one “slow incident”, and its last resolution rewrites previous MTTR values.

The field is in alert-group Extra, without a migration or typed column. Groups predating episodes receive an id lazily at their next transition: the groups still being worked are what matter.

Lifecycle events ​

alert_group.opened/reopened/acknowledged/unacknowledged/silenced/resolved and notification.delivery_attempted are enabled by KAFKA_ANALYTICS_LIFECYCLE_ENABLED (off by default). Rollout order matters: the old Vector configuration shapes every message through object!(.alert); a lifecycle event sent to it is rejected after its offset is committed. Deploy the schema and routing first, then enable emission.

ClickHouse ingest ​

The chart deploys a ready consumer with kafkaConsumer.enabled=true, a source (kafka.enabled or externalKafka.enabled) and a destination (clickhouse.enabled or externalClickHouse.enabled). Rendering fails without both ends: a pod destined to CrashLoop is not released as a “successful” install.

A post-install hook creates the table alongside the consumer:

sql
CREATE TABLE IF NOT EXISTS nxs_alerts_log
(
    received_at     DateTime,
    integration_id  String,
    alert_id        String,
    alert_group_id  String,
    status          LowCardinality(String),
    severity        LowCardinality(String),
    source          LowCardinality(String),
    title           String,
    labels          String,
    group_status    LowCardinality(String),
    group_count     UInt32,
    published_at    DateTime
)
ENGINE = ReplacingMergeTree(published_at)
PARTITION BY toYYYYMM(received_at)
ORDER BY (received_at, integration_id, alert_id);

ReplacingMergeTree is essential: the producer is at-least-once, and a republished event matches all three sorting-key fields, so duplicates collapse. Merges run asynchronously. For exact counts, query nxs_alerts_log_deduplicated (SELECT * ... FINAL) or add FINAL yourself.

Event journal: nxs_events_raw ​

nxs_alerts_log is a narrow projection of alert.ingested. Other events (group transitions and delivery attempts) go to a separate table created by the same hook:

sql
CREATE TABLE IF NOT EXISTS nxs_events_raw
(
    event_id            String,
    event_type          LowCardinality(String),
    schema_version      UInt16,
    occurred_at         DateTime64(3, 'UTC'),
    published_at        DateTime64(3, 'UTC'),
    ingested_at         DateTime64(3, 'UTC'),
    alert_id            String,
    alert_group_id      String,
    episode_id          String,
    integration_id      String,
    team_id             String,
    route_id            String,
    escalation_chain_id String,
    notification_id     String,
    user_id             String,
    alertname           LowCardinality(String),
    service             LowCardinality(String),
    environment         LowCardinality(String),
    cluster             LowCardinality(String),
    host                String,
    payload             String
)
ENGINE = ReplacingMergeTree(ingested_at)
PARTITION BY toYYYYMM(occurred_at)
ORDER BY (toDate(occurred_at), event_type, integration_id, alert_group_id, episode_id, event_id);

Details worth understanding:

  • Three timestamps: occurred_at is the domain transition, published_at is outbox creation, and ingested_at is row insertion. The first two are stamped once and survive republication; the third does not. This distinguishes backlog from slow acknowledgment.
  • event_id completes the sorting key: uniqueness makes deduplication exact. The version column is ingested_at, excluded from the key because it is the only difference between copies. Dashboards read nxs_events_raw_deduplicated (SELECT * ... FINAL), not the table.
  • payload retains the whole message: unanticipated questions can be answered without redeploying. Producer-side redaction has already removed targets, provider bodies and credentials.
  • Labels appear twice: fully in the payload, plus five promoted columns used by dashboards. Per-row JSON extraction across a month's alerts is the difference between a usable dashboard and one people stop opening.

Routing: one topic, two destinations ​

Vector explicitly classifies messages in the shared topic:

  • classify.v2 (schema_version >= 2) → the complete event table;
  • ingested_only → only alert.ingested reaches nxs_alerts_log (the legacy transform begins with object!(.alert) and would reject other types);
  • classify.legacy_ingested → pre-v2 messages arriving during upgrades have no event_id, so cannot be deduplicated into the event journal; they still go to the legacy table;
  • everything else → classify._unmatched → rejected sink: stdout and counters. A silent drop would lose events from analytics used for decisions.

Semantic views ​

The same hook builds regular views over nxs_events_raw_deduplicated using CREATE OR REPLACE, so chart upgrades update semantics too:

ViewOne row representsPurpose
nxs_episode_dimensionsone groupfirst firing alert's labels; lifecycle events have empty labels, while joining every alert to episodes would multiply rows
nxs_group_episodesone group episoderesponse stages, superseded, resolution, eligibility flags
nxs_episode_factone group episodethe same plus severity/service/environment and a single outcome; dashboards read this
nxs_delivery_attemptsone provider invocation or deliberate skipparsed JSON, purpose, episode_id, team_id
nxs_notification_factone notificationoutcome of the unique notification, not individual attempts
nxs_group_state_latestone groupstate now, for backlog
nxs_response_objectivesan objective per (service, severity, valid_from)a table, not a view; '*' is a wildcard
nxs_episode_objectivesone episodethe objective applied to it
nxs_response_attainmentday × team × service × severitymatured / on time / late / pending
nxs_schedule_coverageschedule × checkrotation gaps; report_id avoids counting shared totals repeatedly
nxs_daily_alert_qualityday × integrationvolume/noise, firing and resolving separately
nxs_daily_delivery_qualityday × channel × purposedelivered/failed/skipped, retries, percentiles and mergeable states
nxs_daily_response_qualityday × teamstage percentiles and mergeable states
nxs_analytics_healthevent typefreshness, lag and completeness; inspect before the others

Three interpretation traps:

  • Percentiles cannot be combined: p95_mtta_seconds is correct for one team-day. Across teams or a week, use the neighbouring mtta_p95_state with quantileMerge; an average p95 is not a p95. Ratios likewise use sum(numerator)/sum(denominator) so one-episode and hundred-episode teams do not receive equal weight.
  • Objective attainment uses a matured cohort: an expired unacknowledged episode is late; one not yet due is pending, not success. Counting only “fast among those who responded” would improve as incidents drag on.
  • superseded is not unfinished work: an unresolved episode followed by another episode of the same group was superseded, not abandoned.

There are no incremental materialized views without a separate design. At-least-once duplicates increment an MV immediately; later raw-table merges never subtract them, leaving KPIs silently and permanently wrong. Preaggregates are justified only when dashboards exceed two seconds, and must retain deduplication rather than replace it.

Further semantics:

  • Episode, not group: reopen after a new alert or manual unresolve starts another attempt; otherwise reopening rewrites the first response's MTTR.
  • Delivery events are attributed by time: the delivery worker transaction does not load the group, leaving older events' episode_id empty. The view carries forward the group's last episode (anyLast(...) OVER (...)). Group events are keyed by group id and ordered within one partition.
  • An unfinished episode is not “slow”: absent ACK/resolve yields NULL for that metric, excluded from percentiles but exposed as never_acknowledged / unresolved. Threshold ratios must not improve merely because an incident remains unfinished.
  • delivery_rate excludes skips: no-transport means no provider was tried; merging that with failures/successes hides configuration problems. A separate column exposes skips.
  • First page means successfully delivered: failures and skips never count toward first_delivered_at.

Thresholds are chart values kafkaConsumer.analytics.mttaTargetSeconds (default 300) and mttrTargetSeconds (3600). Views expose proportions, not a verdict.

Escalation, coverage and detection ​

Three additions beyond the first iteration:

  • Escalation depth: every executed step emits alert_group.escalation_step (index, step kind, REPEAT count). Final group state cannot distinguish an acknowledgment after one page from one after four steps and a repeat. alert_group.escalation_exhausted records a chain ending while the group remains open: everyone named by the policy was tried, with no answer. This cannot be inferred from step events: a chain waiting at its final WAIT looks identical externally. nxs_group_episodes includes escalation_steps, max_escalation_step and escalation_exhausted.
  • Schedule coverage: the worker's existing once-a-minute Prometheus coverage report also emits schedule.coverage_reported. This preserves history: a gauge cannot say how long a gap lasted after it has been fixed. nxs_schedule_coverage has one row per schedule-check. Schedule names are not exported (they may name clients or people), only ids and counts.
  • MTTD: episode-open events carry source_started_at, supplied by the source (startsAt from Alertmanager, starts_at from generic payloads). detection_seconds measures until group opening. Source clocks may be wrong; missing source time produces NULL, never an artificial zero that improves KPIs. MTTD stays separate from MTTA; episodes_with_source_time shows the eligible coverage.

All three use kafkaConsumer.analytics.lifecycleEvents. Account for volume: escalation steps are the most frequent lifecycle event, multiplied during a storm by chain length.

Rollups (off by default) ​

kafkaConsumer.analytics.rollups.enabled=true adds daily preaggregated tables nxs_rollup_daily_response, …_delivery, …_alerts and a rebuilding CronJob.

They are rebuilt, not incremented. An incremental MV over at-least-once data adds a duplicate and never removes it. Recomputing closed days from the deduplicated source cannot drift; repeated runs collapse by (day, dimension).

  • rebuildDays (3) controls how many recent days to rebuild. Cover the longest plausible Kafka/ClickHouse catch-up: days rebuilt before late events arrive stay wrong until rebuilt again.
  • concurrencyPolicy: Forbid avoids overlapping rebuilds writing duplicate rows and stealing memory needed by consumer inserts.
  • Tables use AS SELECT … WHERE 0; their columns follow the views initially. Changing a view does not migrate an existing rollup table. Drop it and let the hook recreate it.

Enable only after measuring dashboards exceeding two seconds. Ordinary views read the deduplicating view directly and are correct by construction; each preaggregate is a second copy that can disagree.

Grafana dashboards ​

docs/enterprise/grafana/ contains ten dashboards, one question each:

FileQuestion
01-response-overview.jsonwhere response is going wrong at service-owner level
02-response-latency.jsonwhich stage is slow: before page, after page, or after ACK
03-active-backlog.jsonwhat remains unfinished now
04-alert-volume.jsonwhich rules/integrations generate volume and pages
05-notification-delivery.jsonwhether notifications arrive, and which channel breaks
06-routing-escalation.jsonwhich chains run long or end without a response
07-oncall-load.jsonhow often people are interrupted
08-schedule-coverage.jsonknown schedule gaps
09-episode-explorer.jsonone episode explained from primary events
10-analytics-data-quality.jsonwhether the other nine can be trusted

Import 10-analytics-data-quality.json first. Both a quiet week and a consumer stopped three days ago produce empty response charts; only this dashboard distinguishes them.

At import, select the ClickHouse datasource and db constant (from clickhouse.auth.database, default nxs_anomaly). team, service and severity apply wherever those dimensions exist; panel descriptions identify where they do not.

Rationale, metrics and missing data are explained in INCIDENT_ANALYTICS_DASHBOARDS.md.

Grafana access: read-only ​

Create a separate ClickHouse user rather than reusing Vector's DDL/INSERT user. Dashboards need SELECT on views.

sql
CREATE USER grafana IDENTIFIED BY '...' SETTINGS max_execution_time = 30;
GRANT SELECT ON nxs_anomaly.* TO grafana;

max_execution_time is part of the contract: views use FINAL, and an unfortunate query range can occupy a server the consumer needs for inserts. Configure ClickHouse TLS in the datasource; store the password in a Grafana secret, never dashboard JSON.

Beta analytics boundaries ​

  • MTTD requires source time: absent starts_at leaves detection_seconds NULL, not zero. episodes_with_source_time shows coverage.
  • Schedule coverage is a periodic sample: it describes measurement time, not gaps already fixed before sampling.
  • Delivery episode ids are snapshotted on notifications: old notifications without the field use temporal attribution; new ones do not move episodes when their group reopens.
  • Rollups are opt-in rebuilds: not incremental MVs over an at-least-once stream. Semantic-view shape changes require corresponding rollup migration.
  • Pre-flag data is incomplete by construction: only alert.ingested was published. Compare the first day after enabling with PostgreSQL rather than treating earlier data as complete history.

Retention ​

kafkaConsumer.analytics.rawRetentionDays is explicit. 0 (default) retains everything: unchosen retention can otherwise erase a year of history precisely when needed. Positive values add TTL by occurred_at.

This is not a backup policy; a ClickHouse replica is not a backup.

Pseudonymising ids does not remove personal information from alert text or labels: sources can still supply names, email and hostnames. See the data map and erasure procedure in DATA_INVENTORY.md.

Rollout order ​

  1. helm upgrade deploys schema and Vector, leaving kafkaConsumer.analytics.lifecycleEvents=false.
  2. Confirm ordinary alert.ingested events populate nxs_events_raw.
  3. Enable kafkaConsumer.analytics.lifecycleEvents=true. Both API and worker restart because both Deployments have ConfigMap checksum annotations.
  4. Reconcile PostgreSQL and ClickHouse for a day before treating dashboards as supported.

Old Vector configurations reject lifecycle events after committing offsets: wrong ordering irretrievably loses events.

VRL trap: vector validate --no-environment does not compile the config. It loads files and prints “Validated” even for invalid VRL. Validate by building the topology (vector validate without that flag, or a real start); CI does this in deploy/helm/nxs-anomaly/tests/e2e/kind-kafka-consumer.sh.

Vector is recommended over Kafka Engine because stopped consumers and corrupt message skips in the latter appear only in system.kafka_consumers and server logs. Vector exports Prometheus metrics via the existing ServiceMonitor, keeps a disk buffer while ClickHouse is unavailable, and logs/counts rejected events (component_discarded_events_total) instead of silently dropping them.

The minimum disk-buffer size is 268435488 bytes. Smaller values pass vector validate but fail on topology startup; chart preflight checks it.

ClickHouse sink startup health checks are deliberately disabled (healthcheck.enabled: false). A failed startup check prevents a sink starting at all: events remain buffered forever after one “Healthcheck failed” log. Pods start in arbitrary order, so ClickHouse still starting is routine; checks for two sinks milliseconds apart make either one fail intermittently. Detect vector_buffer_events increasing with vector_component_received_events_total = 0 for the same component_id, rather than relying on startup checks.

Option 2 — ClickHouse Kafka Engine (no separate service) ​

ClickHouse reads the topic directly. The cost is observability: stopped consumers and corrupt-message skips are visible only in system.kafka_consumers and server logs, without an alertable metric.

Deduplication is equally necessary. The short example below uses MergeTree, but at-least-once delivery requires ReplacingMergeTree(published_at) and ORDER BY (received_at, integration_id, alert_id), as in option 1.

sql
-- 1. Intermediate Kafka Engine table
CREATE TABLE nxs_alerts_kafka_queue
(
    event           String,
    schema_version  UInt8,
    published_at    DateTime,
    alert_id        String        MATERIALIZED JSONExtractString(alert_raw, 'id'),
    alert_group_id  String        MATERIALIZED JSONExtractString(alert_raw, 'alert_group_id'),
    integration_id  String        MATERIALIZED JSONExtractString(alert_raw, 'integration_id'),
    status          LowCardinality(String) MATERIALIZED JSONExtractString(alert_raw, 'status'),
    severity        LowCardinality(String) MATERIALIZED JSONExtractString(alert_raw, 'severity'),
    source          LowCardinality(String) MATERIALIZED JSONExtractString(alert_raw, 'source'),
    title           String        MATERIALIZED JSONExtractString(alert_raw, 'title'),
    labels          String        MATERIALIZED JSONExtractString(alert_raw, 'labels'),
    received_at     DateTime      MATERIALIZED toDateTime(JSONExtractString(alert_raw, 'received_at')),
    group_status    LowCardinality(String) MATERIALIZED JSONExtractString(group_raw, 'status'),
    group_count     UInt32        MATERIALIZED JSONExtractUInt(group_raw, 'alert_count'),
    alert_raw       String        ALIAS JSONExtractRaw(message_value, 'alert'),
    group_raw       String        ALIAS JSONExtractRaw(message_value, 'group'),
    message_value   String
) ENGINE = Kafka('kafka:9092', 'nxs-anomaly.alerts', 'clickhouse-consumer', 'JSONEachRow');

-- 2. Destination table
CREATE TABLE nxs_alerts_log
(
    received_at     DateTime,
    alert_id        String,
    alert_group_id  String,
    integration_id  String,
    status          LowCardinality(String),
    severity        LowCardinality(String),
    source          LowCardinality(String),
    title           String,
    labels          String,
    group_status    LowCardinality(String),
    group_count     UInt32
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(received_at)
ORDER BY (received_at, integration_id, severity);

-- 3. Materialized view transfers rows from queue to table
CREATE MATERIALIZED VIEW nxs_alerts_mv TO nxs_alerts_log AS
SELECT
    received_at, alert_id, alert_group_id, integration_id,
    status, severity, source, title, labels,
    group_status, group_count
FROM nxs_alerts_kafka_queue;

Option 3 — Vector outside the chart ​

Use ConfigMap templates/enterprise/kafka-consumer-config.yaml as the model. Two common custom-configuration mistakes:

  • group may be null: a resolved alert without an open group (resolved_without_group). object!(.group) fails and silently loses the alert from resolved statistics. The chart uses group = object(.group) ?? {}.
  • labels may be null: use encode_json(object(alert.labels) ?? {}).

Dead-letter webhook ​

After permanent delivery failure (retries exhausted, status = "failed"), nxs-anomaly can emit a dead-letter event:

bash
NXS_ANOMALY_DEAD_LETTER_WEBHOOK_URL=https://hooks.example.com/dead-letter

Payload:

json
{
  "event": "notification.failed",
  "notification_id": "ntf_...",
  "alert_group_id": "grp_...",
  "user_id": "usr_...",
  "channel": "telegram",
  "target": "123456789",
  "retry_count": 3,
  "last_error": "HTTP 502",
  "failed_at": "2026-06-07T12:00:00Z"
}

Sending is fire-and-forget and does not block the worker cycle. Failures log ERROR dead_letter_delivery_failed.

Monitoring ​

Worker-cycle logs include published_kafka_events:

INFO worker_cycle_complete published_kafka_events=12 processed_alert_groups=3 ...

Broker failures log:

ERROR kafka_outbox_publish_failed error="dial tcp kafka:9092: connection refused"

Rows stay in PostgreSQL and are retried next cycle. Check queue depth with:

sql
SELECT COUNT(*) FROM nxs_anomaly_kafka_outbox;

Growth means an unavailable broker or a stopped worker.

Local development ​

bash
# Start Kafka (official ASF image, KRaft, no ZooKeeper)
docker run -d --name kafka \
  -p 9092:9092 \
  -e KAFKA_NODE_ID=0 \
  -e KAFKA_PROCESS_ROLES=controller,broker \
  -e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=0@localhost:9093 \
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
  -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
  apache/kafka:3.9.0
# Without the three KAFKA_..._REPLICATION_FACTOR/_MIN_ISR settings above,
# __consumer_offsets (the consumer-group system topic) never gets created:
# upstream defaults require 3 replicas, but there is one broker. Producing works,
# but every consumer (including Vector in kafkaConsumer) hangs on its first fetch.

# Pre-create the topic (optional, useful to control partitions instead of
# relying on broker defaults). Image binaries are not in PATH.
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic nxs-anomaly.alerts --partitions 3 --replication-factor 1

# Run with Kafka
KAFKA_BROKERS=localhost:9092 \
NXS_ANOMALY_DB_DSN=postgres://... \
nxs-anomaly serve

nxs-anomaly · Documentation