Skip to content

feat(audit): stream audit events to OTLP and syslog sinks - #582

Merged
Bccorb merged 14 commits into
mainfrom
feat/audit-sinks
Nov 8, 2026
Merged

Bccorb merged 14 commits into
mainfrom
feat/audit-sinks

Conversation

@Bccorb

@Bccorb Bccorb commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #580 (feat/signed-webhooks); review and merge that first. Built against the published @seamless-auth/types 0.40.0.

Closes #420.

What it adds

  • Admin API per AuditSinkRoutes: list, create, get, update, delete, and POST /admin/audit-sinks/:id/test. Reads take requireAdmin('read'). Writes, including the test, take requireAdmin('write') and step-up. Create, update and delete are audited as admin_audit_sink_created, _updated and _deleted, with the destination recorded without its query string. Status reports cursorSeq, lagEvents (chain head minus cursor), lastDeliveredAt, lastFailure and lastFailureAt.
  • audit_sinks table and model. Without startAfterSeq a sink starts at the chain head. A replay may not start past the head or before the oldest event still held.
  • Delivery loop (startAuditSinkDelivery, started in server.ts, 1 s tick like webhooks). Each sink takes pg_try_advisory_xact_lock(72641003, hashtext(id)), reads up to 100 events past its cursor with readExportBatch, sends them, and moves the cursor only when the whole batch was taken. A failure records lastFailure and backs off 1 s, 2 s, 4 s and so on up to 60 s, retrying forever. There is no dead letter and nothing is audited per failure. One sink catches up at most 5,000 events per tick.
  • OTLP/HTTP JSON to <endpoint>/v1/logs through calloutRequest: resourceLogs / scopeLogs (scope seamless-auth.audit) / logRecords. The body is the export line as stringValue. seamless.audit.seq, .id, .type, .hash, .prev_hash, user.id, seamless.audit.actor_user_id and session.id are also sent as attributes, and service.name comes from settings. headersFromEnv is read at send time. A missing or empty var, a name outside the pattern, or a value with CR/LF fails the send as misconfigured, and nothing is sent.
  • Syslog: RFC 5424 with octet counting, over tls.connect (TLSv1.2+, servername, lookup: calloutLookup(env), and an optional CA, client cert and key from the named env vars). Plain TCP is refused in production when the sink is written, and fails as misconfigured at connect. An unset CA, cert or key env var also fails as misconfigured; a refused host fails as refused. MSG is the export line. seq and hash are also in SD [seamless@32473 seq=".." hash=".."], and MSGID is audit, or test for a test record. PRI uses the configured facility, with severity 6, or 4 for *_suspicious events.
  • Export line reuse: exportLine() in auditExport.ts is now the single serializer, used by the export, the retention archive and every sink.
  • Test record: {"type":"test","test":true,"sinkId":..,"generatedAt":..} has no seq or hash. It is marked seamless.audit.test=true in OTLP and test="true" in syslog. It moves no cursor and records no failure.

Deviations from ADR 0005

  1. Retention is held by sink cursors (now recorded in E7 and "Found while building", decided 2026-11-07). Retention used to delete by age alone, so a sink that fell behind for longer than the retention period would lose events it never sent. runAuditRetention now never deletes past the lowest sink cursor, counting enabled and disabled sinks. An audit event deleted before a SIEM receives it cannot be recovered, so a disabled sink holds retention until it is deleted. While retention is held it logs a warning that names the holding sink's id and says "delete the sink to release it". The cursor is read on the application connection, since the retention role may not see audit_sinks.
  2. Syslog success is the collector closing its side after our end of stream, not just the socket flushing. This also catches a TLS 1.3 client cert being rejected after the handshake. A collector that never closes looks like a timeout, and the batch is resent.
  3. Delivery is at least once. The send happens inside the transaction that holds the sink's lock. If the commit fails after a successful send, the batch is sent again, and seq lets a collector drop the repeat.
  4. Two extra columns on audit_sinks, failureCount and nextAttemptAt, hold the backoff state.

Checks (real output, on 0.40.0, at a1b1408)

  • Pre-commit hook: npm run lint, npm run format, npm run typecheck, npm run coverage and npm run build, all exit 0.
  • npm run coverage: 188 files, 3245 passed, 1 todo. All files: 99.1% statements, 95.81% branches, 99.36% functions, 99.29% lines.
  • npm run generate:api: "Generated openapi.json, src/generated/api.ts and adapter-manifest.json from 144 paths." Committed with the change; no further diff.
  • Postgres (TEST_DB=postgres, postgres:18-alpine): 15 files, 264 passed. That includes the new tests/postgres/auditSinks.spec.ts: OTLP and syslog collectors receive the events in order with the chain verifying, the cursor holds on a 503 and then advances, a second replica stands down on the lock, and retention stops at a disabled sink's cursor.
  • Reproduce-first: each security rule was removed one at a time (missing header env var, env name pattern at send, header CR/LF, plain TCP in production, syslog host rules, the guarded lookup on the TLS socket, cursor advancing on failure, the retention hold). The new specs fail every time. The retention warning that names the sink is pinned by a unit test.
  • TLS tests use a committed, test-only CA, server and client certificate under tests/fixtures/auditSinkTls/ (P-256, valid until 2126, SAN localhost and 127.0.0.1).

Not run: /security-review, end-to-end tests against a real SIEM (Datadog, Splunk, rsyslog), and the conformance workflow.

Decisions (2026-11-07)

  • Syslog SD-ID. RFC 5424 requires a private SD-ID to be name@<IANA Private Enterprise Number>, and Fells Code has none yet. The placeholder seamless@32473 (32473 is RFC 5612's documentation number) stays in the single constant SYSLOG_SD_ID, with TODO(#586). Brandon is applying to IANA. Ship syslog audit sinks in 0.23.1 with the Fells Code IANA enterprise number #586 (milestone v0.23, P1) tracks the swap: the release must not ship with 32473.
  • A disabled sink holds retention until it is deleted, as built. This is recorded in ADR 0005.
  • OTLP partialSuccess. Any 2xx counts as delivered, even when the collector reports rejected records, as built. This is recorded in ADR 0005.

Still open:

  • No BOM on MSG. The line is sent as MSG-ANY, so JSON consumers parse it as-is. RFC 5424 only requires a BOM when a sender declares UTF-8.

DELETE /users/credentials removed a passkey without writing an audit
event, so a change to what an account can sign in with left no trace.
It now writes credentials_deleted naming the passkey.
Adds webhook endpoints and deliveries behind an audited admin API with
step-up on every write. Payloads use the Standard Webhooks signature
scheme, are built from committed audit rows past a cursor, and are
retried eight times over just under 42 hours before they are dead
lettered. An endpoint that answers 410 or fails for three days is
disabled.

Every callout to an adopter URL goes through one transport that checks
the address it connects to, never reaches link-local, multicast or the
API itself, never follows redirects, and in production refuses private
addresses unless CALLOUT_PRIVATE_HOSTS lists the host. Signing secrets
are encrypted under CALLOUT_SECRET_ENCRYPTION_KEY.

Writes session_created once per completed sign-in, which user.signed_in
is built from, since login_success precedes any factor.

Records the design in ADR 0005. Closes #418.
Adds audit sinks behind an audited admin API under /admin/audit-sinks,
with step-up on every write. A sink streams committed audit rows past its
own cursor to an OTLP/HTTP JSON collector at <endpoint>/v1/logs, or to a
syslog collector as RFC 5424 over TLS with octet counting. Plain TCP is
refused in production.

Every record carries the event's seamless-auth-events/1 export line, so
the hash chain verifies downstream as an export does. The cursor moves
only when the collector takes the whole batch, and a failing sink retries
with backoff capped at a minute, with no dead letter. Status reports the
lag behind the chain head. A test record is sent outside the chain and
moves nothing.

Destinations follow the callout reachability rules when written and at
connect time, including the guarded DNS lookup for syslog sockets.
Credentials are named as _AUDIT_SINK_SECRET env vars and read at send
time; a missing one fails the send rather than sending it empty.

Retention now stops at the oldest sink cursor, so a sink that falls
behind never loses an event to the archive.

Closes #420.
Retention's warning now names the audit sink holding it back and says
to delete the sink to release it. A disabled sink keeps holding
retention, since an audit event deleted before a SIEM receives it cannot
be recovered.

Records in ADR 0005 the decisions of 2026-11-07: disabled sinks hold
retention, an OTLP 2xx with partialSuccess counts as delivered, and the
syslog SD-ID stays a placeholder until Fells Code has an IANA enterprise
number (#586).
A disabled endpoint was given no deliveries, so events that happened
while it was off were never sent, including after the API disabled it
for a 410 or three days of failures. Its deliveries are now queued on
hold, and turning it back on releases them, back as far as the 30 days
deliveries are kept. The release uses the database clock, which the due
check compares against.
@Bccorb
Bccorb changed the base branch from feat/signed-webhooks to main November 8, 2026 03:29
# Conflicts:
#	openapi.json
#	src/generated/api.ts
#	src/schemas/authEvent.types.ts
#	src/server.ts
@Bccorb
Bccorb marked this pull request as ready for review November 8, 2026 04:17
@Bccorb
Bccorb merged commit ac82c6b into main Nov 8, 2026
9 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: stream audit events to a SIEM over OTLP and syslog

1 participant