Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@
# WG_RECEIVER_PORT=9000 # demo receiver
# WG_ALLOW_INSECURE_HTTP=true # allow http:// endpoint URLs (set false in production)
# WG_ALLOW_PRIVATE_NETWORKS=true # allow endpoints on private networks (set false if customers choose URLs)
# WG_API_DOCS=true # Swagger pages at /docs on Ingest and the operator API (false hides them)
# WG_ENFORCE_QUOTA=false # reject events over the plan's monthly quota with 429
2 changes: 2 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.TimeProvider.Testing" Version="10.0.0" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
<PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.13" />
<PackageVersion Include="Swashbuckle.AspNetCore.SwaggerUI" Version="10.3.1" />
<PackageVersion Include="AWSSDK.DynamoDBv2" Version="4.0.109" />
<PackageVersion Include="AWSSDK.KeyManagementService" Version="4.0.100.19" />
<PackageVersion Include="AWSSDK.S3" Version="4.0.104.5" />
Expand Down
21 changes: 20 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,22 @@ When the logs settle, in a second terminal:

The script creates a customer, an API key and an endpoint, sends an event, and prints it arriving at the demo receiver with `"signature":"valid"`. Open <http://localhost:9000> to see the receiver's page.

### Try the APIs in your browser (Swagger)

Both APIs publish an interactive page where you can read every call and run it with **Try it out**, no tools needed:

| API | Swagger page | OpenAPI definition (for code generators, Postman, Insomnia) | Authorize with |
|---|---|---|---|
| **Events API** (your apps send events here) | <http://localhost:8080/docs> | <http://localhost:8080/openapi/v1.json> | a customer API key `wgk_live_...` |
| **Operator API** (create customers, keys, endpoints) | <http://localhost:8081/docs> | <http://localhost:8081/openapi/v1.json> | the admin key (see below) |

1. Open <http://localhost:8081/docs>, click **Authorize**, paste the admin key, then run **Create a customer**, **Create an API key** and **Register an endpoint** (the endpoint URL `http://receiver:8080/webhook` is the demo receiver).
2. Open <http://localhost:8080/docs>, click **Authorize**, paste the customer API key from step 1, open **Send an event**, click **Try it out** and **Execute**. You get `202` and an `eventId`.
3. Open <http://localhost:9000> to see the event arrive.

Your admin key is printed by the `init` container on first start: `docker compose logs init`, or any time with `docker compose exec controlplane sh -c '. /secrets/env; echo $ControlPlane__AdminKey'`.
Import the OpenAPI URL into Postman or Insomnia to get every call as a ready request, or generate a client in your language with [OpenAPI Generator](https://openapi-generator.tech/). Hide the pages with `WG_API_DOCS=false`.

That is the whole product running on your machine. Stop it with `Ctrl+C` (or `docker compose down`; add `-v` to delete all data).

## Use it from your app
Expand Down Expand Up @@ -93,12 +109,14 @@ curl -s -X POST http://localhost:8080/v1/events \
# {"eventId":"evt_..."} <- HTTP 202: accepted, delivery happens asynchronously
```

Prefer clicking to typing? Open [`samples/api.http`](samples/api.http) in VS Code (extension "REST Client") or any JetBrains IDE: every call above, in order, with the values passed along automatically.
Prefer clicking? Use the [Swagger pages](#try-the-apis-in-your-browser-swagger), or open [`samples/api.http`](samples/api.http) in VS Code (extension "REST Client") or any JetBrains IDE: every call above, in order, with the values passed along automatically.

**6. Receive it** at your endpoint and verify the signature, see [Receiving webhooks](#receiving-webhooks). More operator calls (list, disable, change subscriptions, revoke keys, suspend a customer): [`docs/admin-api.md`](docs/admin-api.md).

## Sending events (API)

> Interactive version of this section: <http://localhost:8080/docs> (Swagger, while the stack is running).

`POST /v1/events` · header `Authorization: Bearer <api key>` · `Content-Type: application/json`

| Field | | |
Expand Down Expand Up @@ -191,6 +209,7 @@ make images # build the container images
|---|---|
| `src/Ingest`, `src/FanOut`, `src/Dispatcher`, `src/ControlPlane` | the four services |
| `src/Abstractions` | **the ports**: small interfaces such as `IEventPublisher`, `IDeliverySource`, `IDeliveryStateStore`, `IArchive` that every storage backend implements |
| `src/ApiDocs` | OpenAPI + Swagger UI shared by the two APIs |
| `src/Postgres` | the default backend (queues, state, archive on PostgreSQL) and the config database stores |
| `src/Aws` | optional AWS backend: SQS, DynamoDB, S3, KMS |
| `src/Valkey` | optional shared rate limiting |
Expand Down
3 changes: 3 additions & 0 deletions WebhookGateway.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
<Folder Name="/src/Abstractions/">
<Project Path="src/Abstractions/WebhookGateway.Abstractions.csproj" />
</Folder>
<Folder Name="/src/ApiDocs/">
<Project Path="src/ApiDocs/WebhookGateway.ApiDocs.csproj" />
</Folder>
<Folder Name="/src/Aws/">
<Project Path="src/Aws/WebhookGateway.Aws.csproj" />
</Folder>
Expand Down
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ services:
environment:
<<: *environment
Ingest__Quota__Enforce: ${WG_ENFORCE_QUOTA:-false}
ApiDocs__Enabled: ${WG_API_DOCS:-true}
ports: ["${WG_INGEST_BIND:-0.0.0.0}:${WG_INGEST_PORT:-8080}:8080"]
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8080 && printf \"GET /healthz HTTP/1.0\\r\\n\\r\\n\" >&3 && grep -q \" 200 \" <&3'"]
Expand Down Expand Up @@ -107,6 +108,7 @@ services:
<<: *environment
ControlPlane__AllowInsecureHttp: ${WG_ALLOW_INSECURE_HTTP:-true}
ControlPlane__AllowPrivateNetworks: ${WG_ALLOW_PRIVATE_NETWORKS:-true}
ApiDocs__Enabled: ${WG_API_DOCS:-true}
# Local only by default: the operator API must not be reachable from the network (see docs/selfhost.md).
ports: ["${WG_ADMIN_BIND:-127.0.0.1}:${WG_ADMIN_PORT:-8081}:8080"]
healthcheck:
Expand Down
2 changes: 2 additions & 0 deletions docs/admin-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Status: **v1, implemented** (D46). Tenants are created by an operator in v1 (no self-service signup). The customer-facing portal API (`/portal/v1`, Cognito login) is separate and comes later (P2-2, P2-3).

**Interactive docs:** with the stack running, <http://localhost:8081/docs> (Swagger UI, click Authorize and paste the admin key) and the OpenAPI definition at `/openapi/v1.json`. Turn them off with `ApiDocs__Enabled=false` (`WG_API_DOCS=false` in Compose).

Base path `/admin/v1`, header `X-Admin-Key: <operator key>` on every call. Without `ControlPlane:AdminKey` configured the API answers `503 admin_disabled`; a wrong key is `401` and logged as a warning. Errors use `{ "error": { "code", "message" } }`. Changes become visible to Ingest, Fan-out and Dispatcher within their cache time (30 s).

| Call | What it does |
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Images: `docker build --build-arg PROJECT=<Ingest|FanOut|Dispatcher|ControlPlane
| `Ingest__Quota__Enforce` | Ingest | `true` rejects over-quota events with 429. Default `false` until plan quotas are set (D42). |
| `Secrets__KmsKeyId` | Dispatcher, ControlPlane | KMS key (id or alias) protecting endpoint signing secrets. Required in production. |
| `Secrets__LocalKey` | dev/tests | Base64 AES key instead of KMS. **Never in production.** One of the two must be set or the service refuses to start. |
| `ApiDocs__Enabled` | Ingest, ControlPlane | `true` (default) serves the OpenAPI definition at `/openapi/v1.json` and the Swagger UI at `/docs`; `false` hides both. |
| `ControlPlane__AdminKey` | ControlPlane | Operator key for `/admin/v1`. Unset means the admin API answers 503. |
| `ControlPlane__AllowInsecureHttp` | ControlPlane | Allow `http://` endpoint URLs (dev only). |
| `Dispatcher__Consumer__MaxConcurrencyPerTier` | Dispatcher | Deliveries in flight per tier queue (default 100). |
Expand Down
1 change: 1 addition & 0 deletions docs/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,4 @@ Format: ID, date, decision, why. Add new entries at the bottom; do not rewrite h
- **D53 (2026-10-21) Self-host mode uses PostgreSQL for queues, delivery state, rollups and the event archive; Valkey is optional; AWS adapters stay as an alternative backend.** Why: a self-hoster wants one `docker compose up` with as few moving parts as possible. Today's local stack uses Moto, an AWS test emulator that is not durable or meant for traffic, so it is demo-only. Postgres is already required for configuration (D21), supports a durable queue with `FOR UPDATE SKIP LOCKED` at a few thousand messages per second on one instance, and gives backup/restore for everything in one place. Valkey stays optional for shared rate limits (without it Ingest uses in-process limits, which is exact for one Ingest instance). The event archive defaults to Postgres (compressed batches with a retention job) and can use any S3-compatible store; MinIO is not bundled because of its license and maintenance status. KMS is replaced by a master key from an environment variable or file (envelope encryption, rotation documented); KMS stays available. Trade-offs accepted: a Postgres queue will not reach the 30,000 events/s Stage 2 design (that remains the AWS backend's job, `architecture.md` section 10), and the self-host guide states the supported throughput honestly. Port interfaces in `src/Abstractions` (D37) make this an additional adapter project (`src/Postgres` grows the queue/state/archive adapters), not a rewrite.
- **D54 (2026-10-21) Self-host backend implemented: `Backend:Mode=Postgres` (migration `0003_selfhost_runtime.sql`, `src/Postgres/Backend`).** Queue: one table `queue_messages` with SQS-like behavior (delay, visibility timeout, five receives then `<queue>-dlq`), receiving with `UPDATE ... WHERE id IN (SELECT ... FOR UPDATE SKIP LOCKED)` so workers never block or double-receive; waiting is a 200 ms poll up to the long-poll time (simple and reliable; `LISTEN/NOTIFY` is a later optimization if latency matters). Why a lease with a receipt UUID: a worker that finished late cannot delete a message that already reappeared and was leased by someone else. State, rollups and archive mirror the DynamoDB/S3 layouts (same column names, same gzip JSONL format, pointer `pg://archive/<key>#<line>`) so documentation and the Athena-style tooling concepts carry over. Idempotency keys and the monthly count live in PostgreSQL by default in this mode (durable across restarts and exact with several Ingest instances) instead of memory; Valkey is therefore optional and only speeds up the rate limit, which stays per-process without it (exact for a single Ingest instance). A maintenance job deletes expired rows hourly (state, archive, rollup minutes, idempotency) and dead-letter messages after 14 days. Retention values are options (`Backend:Postgres:*`). Known limits, stated honestly in the self-host docs: one PostgreSQL instance bounds throughput (target a few thousand events per second; the AWS backend is the path to tens of thousands), the archive grows the database (30-day default retention), and queue polling adds up to ~200 ms latency when idle.
- **D55 (2026-10-22) One-command Docker Compose stack: generated secrets in a volume, operator API and demo receiver on localhost only, private-network delivery on by default, samples tested in CI.** Why generated secrets: `docker compose up` must work with no setup, yet no installation may share a default admin key or encryption key; a one-shot `init` container writes random values into a `secrets` volume on first start (or uses `WG_ADMIN_KEY`/`WG_SECRET_KEY` if the operator provides them) and services source that file at startup through the image entrypoint, so secrets never sit in compose files or images. The generated key encrypts endpoint secrets (`Secrets:LocalKey`, AES-GCM envelope, D45); the docs say plainly to back it up. Why localhost for the operator API and the demo receiver: whoever can call the operator API can create customers, so a fresh install must not expose it to the network; only Ingest listens on all interfaces. Why private-network and http delivery are on by default in Compose (`WG_ALLOW_PRIVATE_NETWORKS`, `WG_ALLOW_INSECURE_HTTP`): the first thing a self-hoster does is deliver to a service on their own network, and the demo receiver is one; the cloud-metadata address stays blocked in all modes, and the docs say to turn both off when untrusted people can register endpoints. Why a demo receiver in the stack: it proves the whole loop (including signature verification) in one command and its code is the reference for the sample verifiers. Why six tested languages: samples that are not run rot, so only languages whose toolchain is available to CI and to the author are shipped (curl, Python, C#, Java, Ruby, Go); Node.js and PHP are welcome as contributions once they can be tested. A CI job starts the Compose stack from scratch, runs the quick start and the samples against it. Status fixes: `database.md` and related docs no longer say "in review".
- **D56 (2026-10-22) Both HTTP APIs publish OpenAPI and an interactive Swagger UI, on by default.** Owner asked for Swagger because it is friendly for developers. Choices: first-party `Microsoft.AspNetCore.OpenApi` generates the definition at `/openapi/v1.json` (no third-party generator to track .NET releases), and only the UI assets come from `Swashbuckle.AspNetCore.SwaggerUI`, which bundles them in the package so the page works offline in a self-hosted install (loading from a CDN would break air-gapped setups). It lives in its own small project (`src/ApiDocs`) used by Ingest and ControlPlane so the worker services do not pick up the web framework. Security is declared in the definition (Bearer for the events API, `X-Admin-Key` for the operator API) so the page's Authorize button works; documentation is readable without a key, calls need one, so enabling it exposes only the shape of the API, which the README already documents. It is switched off with `ApiDocs:Enabled=false` for operators who prefer not to publish it. The events endpoint reads the raw body itself, so its schema comes from description-only records (`SendEventRequest`, `EventAccepted`, `ApiErrorResponse`) and a test checks the definition lists the responses and the Idempotency-Key header; the operator API's response shapes are described the same way. This also delivers the OpenAPI document that roadmap P0-5 and the docs site (S-4) need.
2 changes: 1 addition & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Status: **v1 (2026-10-20)**, derived from the approved architecture v4 (D20-D28)
| P0-2 | Solution scaffold in the monorepo: `src/{Contracts,Ingest,FanOut,Dispatcher,ControlPlane}`, `tests/`, `Directory.Build.props` (.NET 10, nullable, warnings as errors), `.editorconfig` | M | | |
| P0-3 | `Makefile` with `verify` and `verify-fast`; GitHub Actions: build, test, format check | M | risky (CI) | P0-2 |
| P0-4 | Local dev environment: docker-compose with PostgreSQL, LocalStack (SQS, DynamoDB, S3, KMS), Valkey; run script | M | | P0-2 |
| P0-5 | Create repo `webhookgateway-portal` (empty) and OpenAPI generation setup in this repo | S | | P0-2 |
| P0-5 | Create repo `webhookgateway-portal` (empty) and OpenAPI generation setup in this repo. **OpenAPI part DONE 2026-10-22 (D56): each API serves `/openapi/v1.json` and Swagger at `/docs`; publishing it as a release artifact is open.** | S | | P0-2 |

## Phase 1: core pipeline (local, tested against LocalStack and Testcontainers)

Expand Down
4 changes: 4 additions & 0 deletions docs/worklog.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ Resume point for agents after a sudden stop, crash, or new session. **Read this
- **Outcome:** `architecture.md` v4 approved; decisions D20-D28 logged; `contracts.md` v2; `database.md` rewritten config-only (still **in review**); `docs/roadmap.md` lists the work items; plan numbers still provisional.
- **Resume from:** see the next entry (roadmap execution).

### 2026-10-22 Swagger / OpenAPI (D56)
- **Status:** PR open
- **Outcome:** `src/ApiDocs`; `/docs` and `/openapi/v1.json` on Ingest (:8080) and ControlPlane (:8081); events and admin calls described with responses and security; tests; README section "Try the APIs in your browser (Swagger)"; `WG_API_DOCS` switch. The running local stack was rebuilt with it.

### 2026-10-22 One-command stack, samples, README (O-5..O-8, O-11 part)
- **Status:** PR open; owner asked to be given the running app to test and a go/no-go for making the repo public
- **Outcome:** `docker compose up` runs postgres, init (secrets), migrator, ingest, fanout, dispatcher, controlplane, demo receiver; `scripts/quickstart.sh`; six tested sample languages; README as the full guide; `docs/selfhost.md`; CI job `compose-smoke`; private-network option; stale "in review" statuses fixed. Verified from a clean state: quick start, 11 sample checks, and the retry path (failed endpoint leaves a scheduled retry).
Expand Down
2 changes: 2 additions & 0 deletions samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ python3 python/send_event.py

`./test.sh` checks the verify code of every installed language against `test-vector.json` (a known-good signature, and a tampered body that must be rejected) and, when the stack is running and `WEBHOOKGATEWAY_API_KEY` is set, sends an event with every sample.

Prefer a browser? Both APIs have Swagger pages with "Try it out": <http://localhost:8080/docs> (events) and <http://localhost:8081/docs> (operator). `api.http` has every call for VS Code/JetBrains.

Rules the samples follow, and you should too:

- Use **one** `Idempotency-Key` per logical event and **reuse it on retries**, so a retry never creates a duplicate.
Expand Down
Loading
Loading