Skip to content

RUNBOOK.md

Latest commit

 

History

History
740 lines (554 loc) · 28.7 KB

RUNBOOK.md

File metadata and controls

740 lines (554 loc) · 28.7 KB
Raw

Production Runbook — Stellar Mainnet

1. Network & Environment Alignment

Verify deployment target

# Current env values (must match mainnet)
echo "STELLAR_NETWORK=$STELLAR_NETWORK"
echo "STELLAR_RPC_URL=$STELLAR_RPC_URL"
echo "VAULT_CONTRACT_ID=$VAULT_CONTRACT_ID"
Variable Mainnet value
STELLAR_NETWORK mainnet
STELLAR_RPC_URL https://soroban-mainnet.stellar.org
Network passphrase Public Global Stellar Network ; September 2015
NODE_ENV production

Contract IDs, token addresses, and the STELLAR_AGENT_SECRET_KEY must be mainnet instances. A testnet key on mainnet will sign invalid operations.

Pre-flight alignment checks

# 1. Confirm network in env matches deployment context
grep STELLAR_NETWORK .env | grep -q mainnet || echo "WARN: not mainnet"

# 2. Verify RPC connection returns mainnet ledger
curl -s -X POST "$STELLAR_RPC_URL" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"getLatestLedger"}' | \
  jq '.result.sequence'

# 3. Confirm agent key controls the vault on this network
#    (validate via a read-only contract call — getVaultInfo or similar)

# 4. Verify Prisma migration status matches schema
npx prisma migrate status

Boot sequence validation

On startup, src/config/env.ts validates:

  • STELLAR_NETWORK ∈ {mainnet, testnet, futurenet}
  • STELLAR_AGENT_SECRET_KEY starts with S, length 56
  • WALLET_ENCRYPTION_KEY is exactly 64 hex chars
  • All required env vars are set (throws if missing)

The GET /health/ready endpoint reports three subsystems: database, eventListener, agentLoop. All must be ready: true before the load balancer marks the instance healthy.

2. Key Custody

Secrets under management

Secret Source Purpose Rotation
STELLAR_AGENT_SECRET_KEY env var Signs Soroban contract calls (rebalance, update total assets) On key compromise or quarterly
WALLET_ENCRYPTION_KEY env var AES-256-GCM key encrypting custodial wallet secrets in custodial_wallets table Coordinated re-encryption migration
JWT_SEED env var Signs session JWTs Every 90 days (invalidates all sessions)
DATABASE_URL env var/env file PostgreSQL connection DB password rotation per provider policy

Agent key rotation

  1. Generate new Stellar keypair: 
    stellar keys generate neurowealth-agent-v2  # or via SDK
  2. Fund the new public key with XLM on mainnet.
  3. If the vault contract maintains an operator allowlist, update it to include the new key.
  4. Set STELLAR_AGENT_SECRET_KEY in your secret manager to the new secret.
  5. Redeploy all instances (rolling update).
  6. Verify agent loop health: GET /health/ready → agentLoop: ready.
  7. Keep the old key funded for 30 days in case a rollback is needed.
  8. Drain and discard the old key after the rollback window.

Wallet encryption key rotation

  1. Provision WALLET_ENCRYPTION_KEY_NEW in the secret manager alongside the current key.
  2. Run a one-off migration script that:
    • Reads every row from custodial_wallets
    • Decrypts encryptedSecret with the old key
    • Re-encrypts with the new key
    • Writes back the new encryptedSecret, iv, authTag
  3. Swap the env var to the new key.
  4. Verify a sample of users can still sign operations.
  5. Remove the old key from the secret store.

Custodial wallet recovery

Losing WALLET_ENCRYPTION_KEY permanently destroys all custodial wallet keys.

  • Backup: Regular DB snapshots preserve encrypted key material.
  • Audit: The custodial_wallets table stores (publicKey, encryptedSecret, iv, authTag) — never plaintext secrets.
  • Disaster: If the DB is restored from a backup, the encryption key at backup time must still be available.

Secret storage policies

  • Never commit secrets to git. Use .env.example as a template.
  • Production: AWS Secrets Manager / HashiCorp Vault with access audit logging.
  • CI/CD: GitHub Environments secrets, injected as env vars in deploy workflows.
  • Local dev: .env file (gitignored).

3. RPC Failover

Current architecture

src/stellar/client.ts creates a single rpc.Server(STELLAR_RPC_URL) singleton. There is no built-in automatic failover. A mainnet RPC outage halts event ingestion and agent operations.

Failover strategy

Option A: Load-balanced endpoint (recommended)

Configure a single URL that routes across multiple RPC providers:

STELLAR_RPC_URL=https://soroban-mainnet.stellar.org

Replace this with a load balancer or provider that pools:

  • https://soroban-mainnet.stellar.org (SDF)
  • https://mainnet.sorobanrpc.com (public)
  • https://rpc.stellar.org/mainnet (alternative)

Option B: Multi-provider fallback (not yet implemented)

If you need resilience without a LB, wrap getRpcServer() to fall back:

const RPC_URLS = [
  'https://soroban-mainnet.stellar.org',
  'https://mainnet.sorobanrpc.com',
]
let currentIndex = 0

export function getRpcServer(): rpc.Server {
  // Returns current server; call rotateRpc() on failure
  if (!rpcServer) rpcServer = new rpc.Server(RPC_URLS[currentIndex])
  return rpcServer
}

export function rotateRpc(): void {
  currentIndex = (currentIndex + 1) % RPC_URLS.length
  rpcServer = new rpc.Server(RPC_URLS[currentIndex])
  logger.warn(`[RPC] Failed over to ${RPC_URLS[currentIndex]}`)
}

Wire rotateRpc() into error handlers in fetchEvents and submitTransaction.

RPC outage playbook

Symptom Action
fetchEvents fails with connection error Rotate RPC URL (manual or automated)
sendTransaction hangs or times out Rotate RPC; retry tx via getTransaction
Persistent RPC failures Switch to backup RPC provider entirely
All known RPCs down Pause event listener; set agentLoop to degraded; page on-call

Verify RPC health

# Check latest ledger via RPC
curl -s -X POST "$STELLAR_RPC_URL" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"getLatestLedger"}' | \
  jq '.result.sequence'

# Monitor via /metrics
curl -s http://localhost:3001/metrics | grep cursor_lag

4. Ledger Lag Alerts

Metrics

Metric Type Description
cursor_lag_ledgers Gauge latest_ledger - last_processed_ledger
last_processed_ledger Gauge Last ledger successfully processed
events_processed_total Counter Events processed, labelled by type and status

Alert rules are defined in docs/OBSERVABILITY.md and deployed to Prometheus.

Alert thresholds

Severity Lag Action
Info > 10 ledgers Note — may be normal during low traffic
Warning > 50 ledgers for 5 min Investigate within 1 hour
Critical > 100 ledgers for 2 min Page immediately

Investigation steps

# 1. Check current lag
curl -s http://localhost:3001/metrics | grep cursor_lag

# 2. Check last processed ledger in DB
psql "$DATABASE_URL" -c "SELECT * FROM event_cursors WHERE \"contractId\" = '$VAULT_CONTRACT_ID';"

# 3. Check listener logs for errors
grep "Event Listener" /var/log/app/*.log | tail -50

# 4. Check RPC connectivity
curl -s -X POST "$STELLAR_RPC_URL" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"getLatestLedger"}' | \
  jq '.result.sequence'

# 5. Check for backpressure (DLQ growth)
curl -s http://localhost:3001/metrics | grep dlq_size

# 6. Check database connection pool
psql "$DATABASE_URL" -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"

Common causes & remediation

Cause Signal Fix
RPC outage fetchEvents errors in logs Rotate RPC endpoint (see §3)
DB slow / locked High db_operation_duration_seconds Check locks, pool size, index usage
Schema validation failures DLQ growth, event_validation errors Inspect DLQ, fix event format or validator
Listener crashed cursor_lag rising, agent_loop_status == 0 Container restart, check OOM killer
Network partition RPC timeouts Check DNS, firewall, egress rules

Recover from lag

# If lag < 1000 ledgers — automatic backfill runs on restart
# If lag > 1000 ledgers — manual backfill recommended via admin endpoint

# Manual backfill (from a specific ledger)
# Restart the service; backfill runs automatically up to latest
# If auto-backfill is too slow, consider:
#   1. Stop the listener
#   2. Update event_cursors to an earlier ledger
#   3. Restart the listener to trigger backfill
psql "$DATABASE_URL" -c "UPDATE event_cursors SET \"lastProcessedLedger\" = $EARLIER_LEDGER WHERE \"contractId\" = '$VAULT_CONTRACT_ID';"

5. DLQ Replay Procedure

Overview

Events that fail processing (validation error, DB error, missing user) are stored in the dead_letter_events table with status PENDING. The DLQ module (src/stellar/dlq.ts) manages retries through three admin API endpoints.

Inspect DLQ

# Via admin API (requires ADMIN_API_TOKEN)
curl -s -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  http://localhost:3001/api/admin/dlq/inspect | jq

# Filter by status
curl -s -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  "http://localhost:3001/api/admin/dlq/inspect?status=PENDING" | jq

# Via direct DB query
psql "$DATABASE_URL" -c "
  SELECT id, \"eventType\", \"txHash\", ledger, status, \"retryCount\", error, \"createdAt\"
  FROM dead_letter_events
  ORDER BY \"createdAt\" DESC
  LIMIT 50;
"

Dry-run retry

curl -s -X POST -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}' \
  http://localhost:3001/api/admin/dlq/retry | jq

Dry run simulates the retry loop without persisting status changes.

Full retry

curl -s -X POST -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' \
  http://localhost:3001/api/admin/dlq/retry | jq

Returns:

{
  "resolved": 5,
  "failed": 2,
  "totalRemaining": 2
}

Resolve a specific event

If an event cannot be processed (e.g. user deleted), manually resolve it:

curl -s -X POST -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"id": "uuid-of-event"}' \
  http://localhost:3001/api/admin/dlq/resolve | jq

Automatic retry behavior

  • retryAll() processes all PENDING and RETRIED events sequentially.
  • Success → status set to RESOLVED, count +1.
  • Failure → status set to RETRIED, count +1, logged.
  • There is no automatic scheduled retry. All retries are manual via the admin API.
  • When DLQ size reaches 50, a critical log line is emitted and the Prometheus dlq_size gauge crosses the critical threshold.

DLQ replay decision matrix

Event type Common failure Retry likely? Notes
deposit User not found No until user exists Resolve after user registers
deposit Schema validation Depends Fix validator or event source
withdraw Position not found No May indicate data integrity issue
rebalance DB constraint Yes Transient — retry typically succeeds
Any RPC/DB timeout Yes Transient — retry typically succeeds

6. Database Migration Rollback

Prisma migrations are forward-only — there is no built-in down. Every migration in prisma/migrations/<name>/ therefore ships a hand-written rollback.sql alongside its migration.sql. CI enforces this via scripts/check-migration-rollback.sh.

For a migration that failed mid-apply (P3009, CONCURRENTLY in a transaction, duplicate column, stuck rehearsal), follow MIGRATIONS.md Appendix A: Common failure modes.

When to roll back

Roll back when a freshly deployed migration is itself the problem (broken schema, failed constraint, performance regression). If the application code is the problem, prefer redeploying the previous app version over a schema rollback.

WARNING: Rollbacks can be destructive — DROP TABLE/DROP COLUMN discard data. Confirm a recent backup/snapshot exists before proceeding. Some rollbacks are flagged partially irreversible inside their rollback.sql (e.g. 20260617000000_fix_agent_log_attribution cannot restore NOT NULL if system-generated rows with a null userId exist).

Procedure

# 1. Take / confirm a database snapshot first.

# 2. Identify the migration to reverse (most recent applied is the usual target)
npx prisma migrate status

# 3. Run the rollback for that migration (applies rollback.sql, then marks it
#    rolled back in _prisma_migrations and runs a health check).
DATABASE_URL=$DATABASE_URL bash scripts/rollback-migration.sh <migration-name>

#    Optionally verify the live app afterwards:
HEALTHCHECK_URL=http://localhost:3001/health/ready \
  DATABASE_URL=$DATABASE_URL bash scripts/rollback-migration.sh <migration-name>

# 4. Re-deploy the previous application version if the schema change was paired
#    with code changes.

After a successful rollback the migration is marked rolled_back_at in Prisma's history, so a later prisma migrate deploy (with a fixed migration) re-applies it.

Authoring rollbacks for new migrations

Every new migration PR must add a rollback.sql that reverses its migration.sql: drop what it created, recreate what it dropped, and document any irreversible steps as comments. The Migration rollback check workflow blocks merge otherwise.

7. Incident Contacts

Alert-by-alert steps, ack timers, and the postmortem workflow are in INCIDENT_RESPONSE.md. This section is the contact list those steps use.

Escalation tiers

Tier Role Responsibility Contact
T1 On-call engineer Ack, triage, restart, DLQ retry, RPC rotation, sponsor top-up PagerDuty (PAGERDUTY_ROUTING_KEY)
T2 Backend lead Code fix, data reconciliation, migration rollback Slack @backend-lead. Page if they have not answered inside the severity window
T3 Engineering manager Stakeholder updates, priority, SEV1 postmortem acceptance Slack @eng-mgr
T4 Security officer Key compromise, wallet recovery, audit Slack @sec-officer. Page immediately for SEV1 security incidents, do not wait for T2

Communication channels

Channel Purpose
#neurowealth-alerts Prometheus alert notifications
#neurowealth-incidents Incident coordination thread. The acknowledging engineer is the lead until they hand off in the thread
PagerDuty T1 page, then T2 at 15 min (SEV1) or 1 hour (SEV2) if still unacked or unmitigated
Email: ops@neurowealth.io Backup when PagerDuty or Slack is unreachable

Incident severity definitions

Severity Definition Response time Escalation
SEV1 Event processing halted, global agent breaker open, funds at risk, data loss, sponsor XLM exhausted while deposits fail < 15 min T1 → T2 at 15 min → T3 at 30 min. T4 immediately for key compromise
SEV2 Lag > 100 ledgers, DLQ > 50, agent loop degraded, readiness failing < 1 hour T1 → T2 if not mitigated in 1 hour
SEV3 Lag > 50 ledgers, DLQ > 20, elevated latency < 8 hours T1
SEV4 Minor anomalies, informational alerts Next business day None

After the incident

SEV1 and SEV2 are not closed until the postmortem in INCIDENT_RESPONSE.md is merged. That workflow covers the timeline, the root cause, and the action items. Update this runbook in the same follow-up when a step here was wrong or missing.

Quick Reference Commands

# Health
curl http://localhost:3001/health/live
curl http://localhost:3001/health/ready
curl http://localhost:3001/health

# Metrics
curl http://localhost:3001/metrics | grep -E "(cursor_lag|dlq_size|events_processed|agent_loop)"

# DLQ inspect
curl -s -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  http://localhost:3001/api/admin/dlq/inspect | jq '. | length'

# DLQ retry (dry run)
curl -s -X POST -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}' \
  http://localhost:3001/api/admin/dlq/retry

# DLQ retry (live)
curl -s -X POST -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' \
  http://localhost:3001/api/admin/dlq/retry

# DB — cursor status
psql "$DATABASE_URL" -c "SELECT * FROM event_cursors;"

# DB — DLQ count by status
psql "$DATABASE_URL" -c "
  SELECT status, count(*) FROM dead_letter_events GROUP BY status;
"

# DB — recent processed events
psql "$DATABASE_URL" -c "
  SELECT \"eventType\", ledger, \"txHash\", \"createdAt\"
  FROM processed_events
  ORDER BY ledger DESC LIMIT 10;
"

8. Sponsor Account Top-Up

Sponsored reserves move the XLM cost from user to sponsor accounts (STELLAR_SPONSOR_KEYS). Monitor GET /api/v1/admin/reserves (admin-scoped, audit-logged) for outstandingXlm and perSponsor[].availableXlm. Alert SponsorLowXlm fires when any sponsor < 10 XLM for 5m.

Top-up:

# Check
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" http://localhost:3001/api/v1/admin/reserves | jq

# Fund sponsor from treasury/ops hot wallet via Stellar Laboratory or
stellar account fund --destination <sponsorPublicKey> --amount 100 --network public

# Verify
curl -s http://localhost:3001/metrics | grep sponsor_available_xlm
psql "$DATABASE_URL" -c "SELECT \"sponsorAccount\", count(*), sum(\"xlmReserved\") FROM reserve_sponsorships WHERE status='ACTIVE' GROUP BY \"sponsorAccount\";"

No auto top-up — operational runbook only. Reconciliation job (reserveReconciliation hourly) flags drift where on-chain sponsor ≠ ledger.

9. Agent Circuit Breaker (#345)

The agent circuit breaker halts agent-initiated rebalancing when the market, a protocol, or a user's account shows risk. It never touches withdrawals. Scopes: GLOBAL (halt everything), PROTOCOL (halt a target protocol + batches leaving it), USER (halt that user's batches). Breakers are CLOSED → OPEN → HALF_OPEN → CLOSED; an OPEN breaker auto-probes after its cooldown and needs BREAKER_DEPEG_SUSTAINED_CHECKS clean evaluations before recovering to HALF_OPEN, then one clean probe to close.

Rules

Rule Env Default Trips when
abnormal_loss BREAKER_LOSS_PCT, BREAKER_LOSS_WINDOW_HOURS 5% / 24h mark-to-market drawdown over the window exceeds the pct
depeg BREAKER_DEPEG_ENABLED (=false) off reported stablecoin price deviates > BREAKER_DEPEG_BPS (150) from $1
oscillation BREAKER_MAX_FLIPS, BREAKER_FLIP_WINDOW_HOURS 3 / 24h same batch rebalances ≥ N times in the window
stale_data BREAKER_STALE_MINUTES, BREAKER_STALE_CONSECUTIVE_FAILURES 120m / 3 APY table older than limit, never scanned, or ≥ N consecutive failures

BREAKER_COOLDOWN_MS (1h) is the base cooldown; a repeated HALF_OPEN trip doubles it up to BREAKER_MAX_COOLDOWN_MS (24h).

Known limitation — de-peg price feed

The de-peg rule is a pure consumer of a stablecoin spot price. As of this change no live price feed exists in this codebase: the fee oracle (src/stellar/feeOracle.ts) publishes only fees, and src/stellar/routing.ts is a stub. getStablecoinPrice() (src/agent/breakerService.ts) is the single integration point and currently returns null (fails safe — the rule never trips); the rule is disabled by default. Wire the oracle there, keep the pure rule unchanged, flip BREAKER_DEPEG_ENABLED=true, and re-run the de-peg unit tests.

Inspector

# All breakers with current state
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" http://localhost:3001/api/v1/admin/agent/breakers | jq

# Agent status (incl. cached global breaker summary)
curl -H "X-Internal-Token: $INTERNAL_SERVICE_TOKEN" http://localhost:3001/api/v1/agent/status | jq

Manual trip

POST /api/v1/admin/agent/breakers — body {"scope":"PROTOCOL","scopeKey":"blend","reason":"..."}. GLOBAL needs no scopeKey; USER/PROTOCOL require it. reason is always required. Written to the admin audit log (TRIP_AGENT_BREAKER).

curl -X POST http://localhost:3001/api/v1/admin/agent/breakers \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"scope":"PROTOCOL","scopeKey":"blend","reason":"incident-4821 protocol outage"}'

A manual trip can only be cleared manually (rule=manual breakers never auto-reset). Manual resets are audit-logged too:

curl -X POST http://localhost:3001/api/v1/admin/agent/breakers/<id>/reset \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"reason":"incident-4821 resolved, APY table verified fresh"}'

A breaker skips its tick when evaluation itself fails: the agent alerts (critical, agent-breaker:eval-failed) and halts all rebalancing for that tick rather than trading blind.

Response

  1. Global halt: investigate the trip rule (agent_breaker_trips_total{scope="GLOBAL"}), check the [Breaker] logs for the lastEvaluation detail, fix root cause, then either wait for auto-recovery or reset manually.
  2. Protocol halt: verify the protocol's APY/status independently before resetting; compareProtocols already refuses it as a target while OPEN.
  3. User halt: confirm with the user before resetting.
  4. Evaluation-failed halt: the breaker could not decide — check DB connectivity and the logged error before the next tick.

Verify

curl -s http://localhost:3001/metrics | grep -E "agent_breaker_(state|trips_total)"
psql "$DATABASE_URL" -c "SELECT scope, \"scopeKey\", state, \"trippedRule\" FROM agent_circuit_breakers ORDER BY \"updatedAt\" DESC;"

10. Feature Flags & Emergency Rollback Runbook (#494)

Detailed reference and architectural specifications are documented in docs/FEATURE_FLAGS.md.

Inspect Active Feature Flags

# List all feature flags and runtime evaluation for the current environment
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  http://localhost:3001/api/v1/admin/feature-flags | jq

# Inspect single flag
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  http://localhost:3001/api/v1/admin/feature-flags/smart_dca_engine | jq

Staged Rollout Adjustment

# Advance rollout percentage (0 to 100)
curl -X POST http://localhost:3001/api/v1/admin/feature-flags/smart_dca_engine/staged-rollout \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"percentage": 25}'

Fast Emergency Rollback (< 1s)

When an anomaly or regression is detected, trigger an immediate rollback. This clamps the rollout to 0%, engages the rollback lock, increments the Prometheus rollback metric, logs structured SRE audit events, and broadcasts the event across Redis:

# Fast rollback with mandatory reason
curl -X POST http://localhost:3001/api/v1/admin/feature-flags/smart_dca_engine/rollback \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "High slippage observed in Horizon liquidity pools"}'

Platform-Wide Emergency Maintenance Mode

To halt all state-mutating requests platform-wide during an active exploit or catastrophic dependency failure:

curl -X POST http://localhost:3001/api/v1/admin/feature-flags/emergency-disable-all \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Halting mutating requests due to protocol exploit"}'

Environment Variable Emergency Disable

If the admin endpoint is unreachable, override in the environment without deploying new code:

# Disable specific flags via rollback
FEATURE_FLAGS_ROLLBACK=smart_dca_engine,fast_fiat_onramp

# Or engage individual kill switch
FEATURE_FLAG_SMART_DCA_ENGINE_KILL_SWITCH=true

Reset to Default Environment Configuration

curl -X POST http://localhost:3001/api/v1/admin/feature-flags/smart_dca_engine/reset \
  -H "Authorization: Bearer $ADMIN_API_TOKEN"

13. Telegram & WhatsApp Message Delivery Recovery (#493)

Alert: telegram_dlq_high / whatsapp_dlq_high

Triggered when dead-lettered message count exceeds MESSAGING_DLQ_ALERT_THRESHOLD (default: 10).

Diagnostic commands

# Check queue metrics and dead-letter count
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" http://localhost:3001/api/v1/admin/messages/stats | jq

# List failed / dead-lettered messages
curl -H "Authorization: Bearer $ADMIN_API_TOKEN" "http://localhost:3001/api/v1/admin/messages?status=DEAD_LETTER&limit=20" | jq

Recovery actions

  1. Investigate Root Cause: Inspect lastError on failed deliveries:
    • 429 Rate Limits: Verify rate-limit configuration and exponential backoff settings.
    • 403 Forbidden (Telegram): User blocked the bot; cancel message or re-route.
    • 21211 (WhatsApp/Twilio): Invalid phone number format.
  2. Bulk Retry Dead Letters: Once downstream connectivity is restored: 
    curl -X POST http://localhost:3001/api/v1/admin/messages/retry-dead-letters \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"channel": "TELEGRAM"}'
  3. Single Message Manual Retry: 
    curl -X POST http://localhost:3001/api/v1/admin/messages/<message-id>/retry \
  -H "Authorization: Bearer $ADMIN_API_TOKEN"
  4. Cancel Poison Messages: 
    curl -X DELETE http://localhost:3001/api/v1/admin/messages/<message-id> \
  -H "Authorization: Bearer $ADMIN_API_TOKEN"

DLQ Backoff Schedule (#610)

The Stellar event DLQ (src/stellar/dlq.ts) retries failed events with exponential backoff and jitter. Every retriable entry carries a retryCount; on each failed attempt the count increments and the entry becomes eligible again only after BACKOFF_BASE_MS * 2^retryCount milliseconds (capped at BACKOFF_MAX_MS) multiplied by a uniform(0.5, 1.5) jitter factor. After MAX_RETRY_ATTEMPTS attempts the entry transitions to the terminal POISON state and is never auto-retried again — only admin action can resurrect it.

Defaults (src/stellar/dlq.ts):

  • BACKOFF_BASE_MS = 30_000 (30 s)
  • BACKOFF_MAX_MS = 3_600_000 (1 h)
  • MAX_RETRY_ATTEMPTS = 10

Approximate schedule (central value; actual spread is ±50% jitter):

retryCount after failure Next eligible after Cumulative wall-clock
1 30 s ~30 s
2 60 s ~1.5 min
3 2 min ~3.5 min
4 4 min ~7.5 min
5 8 min ~15 min
6 16 min ~31 min
7 32 min ~63 min
8 60 min (cap) ~2 h
9 60 min ~3 h
10 → POISON terminal

Poison entries never auto-retry

An entry at status === 'POISON' is skipped by retryAll forever. The dlq_poison_size Prometheus gauge tracks how many such entries exist; alerting on dlq_poison_size > 0 surfaces them for triage.

Admin actions (all audited via AdminAuditLog)

  • POST /api/admin/dlq/retry — trigger retryAll now, honouring the backoff gate (entries whose window has not elapsed are skipped, not forced).
  • POST /api/admin/dlq/resolve — mark one entry resolved by id (operator verified the side effect landed by another path).
  • POST /api/admin/dlq/requeue — reset one entry's retryCount to 0 and status to PENDING, restarting the backoff schedule. Required { "eventId", "reason" }. Use after the underlying cause has been fixed (an RPC endpoint restored, a bad migration rolled back).
  • POST /api/admin/dlq/discard — delete one entry permanently. Required { "eventId", "reason" }. Audit log records who, what, and why.

Operator playbook

  1. GET /api/admin/dlq/inspect — see what is piling up and in which status.
  2. Group by eventType/contractId; most storms are one-cause.
  3. If the cause is fixed and the entries are RETRIED/POISON, requeue them (not retry — retry still honours backoff and does nothing for POISON).
  4. If the entries are structural garbage that will never process (schema mismatch, stale testnet events, an irrecoverable bug), discard with a written reason naming the triage ticket.
  5. If the queue size alert keeps firing but pending stays flat and poison grows: the retriers are working as designed on real failures. Investigate the upstream fault, not the queue.