Skip to content

Commit 1af9dbc

Browse files
authored
Merge pull request #169 from DecaturMakers/esb-integration
ESB support: machine status API + status-change webhook
2 parents 99a3d65 + 0f486ab commit 1af9dbc

20 files changed

Lines changed: 1380 additions & 75 deletions

‎CLAUDE.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ On every push to `main`, `.github/workflows/release.yml` compares the `pyproject
110110
- `MACHINES`: MachinesConfig instance managing all machine configurations
111111
- `USERS`: UsersConfig instance managing all user data
112112
- `SLACK_HANDLER`: Optional SlackHandler for Slack integration
113+
- `WEBHOOK_NOTIFIER`: Optional WebhookNotifier (see Status-Change Webhook below); `None` unless `STATUS_WEBHOOK_URL` is set
113114
- `START_TIME`: Server start timestamp for uptime tracking
114115

115116
**Configuration System**:
@@ -196,8 +197,30 @@ as `/api/machine/*`):
196197

197198
**Admin APIs** (`/api/*`):
198199
- `POST /api/reload-users`: Hot-reload users.json without restart
200+
- `GET /api/machines`: List all machines and their current status as JSON
201+
(sorted by name). Read-only; intended for external consumers such as the
202+
Equipment Status Board. Each entry is `Machine.status_dict` (name,
203+
display_name, derived `status`, relay, oops, locked_out, current_user,
204+
last_checkin, last_update).
199205
- `GET /metrics`: Prometheus metrics endpoint
200206

207+
### Status-Change Webhook
208+
209+
When `STATUS_WEBHOOK_URL` is set, `WebhookNotifier` (`src/dm_mac/webhook.py`)
210+
POSTs a JSON webhook to that URL on every *meaningful* machine status change —
211+
never on ordinary MCU heartbeats. Events: `login`, `logout`, `unauthorized`,
212+
`unknown_fob`, `override_login`, `oops`, `unoops`, `lockout`, `unlock`,
213+
`reboot`. The payload is `Machine.status_dict` plus `event`, `timestamp`, and a
214+
distinct `user` field (the event actor, e.g. who logged out — differs from
215+
`current_user`). Delivery is fire-and-forget (an `asyncio` task, like the Slack
216+
notifications) with a per-attempt timeout and bounded exponential-backoff
217+
retry, so the MCU response is never blocked; failures are logged and dropped.
218+
`notify()` is called only from the meaningful status-change code paths in
219+
`models/machine.py`, so heartbeats never fire it. The notifier is resolved via
220+
`current_app` in request-context (MCU/API) paths and via the passed Slack
221+
handler's app (`slack.quart`) in the Slack-command path, which runs without a
222+
request context.
223+
201224
### Logging
202225

203226
Custom `RequestFormatter` adds request context (`remote_addr`, `url`) to all logs when available. The `AUTH` logger is used specifically for authentication/authorization decisions.
@@ -218,6 +241,7 @@ Optional for MAC server:
218241
- `SLACK_SIGNING_SECRET`: Slack Signing Secret
219242
- `SLACK_CONTROL_CHANNEL_ID`: Private admin channel ID
220243
- `SLACK_OOPS_CHANNEL_ID`: Public channel for oops/maintenance notices
244+
- `STATUS_WEBHOOK_URL`: If set, URL to POST status-change webhooks to (see Status-Change Webhook above); disabled when unset
221245

222246
## Testing Notes
223247

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# ESB Support
2+
3+
> Status: **completed**. This document is the original feature request plus the
4+
> implementation plan and per-milestone progress notes recorded as the work was
5+
> carried out.
6+
7+
## Overview
8+
9+
Goal: implement the required minimal changes in this application to support https://github.com/DecaturMakers/equipment-status-board/issues/10.
10+
11+
At a high level, this means:
12+
13+
1. If we don't already have one, a simple API for retrieving a list of all machines and their current status.
14+
2. Ability to fire a webhook (non-blocking/async or from a background worker) when a machine's status is changed (user login/logout, unauthorized RFID fob, oops/lockout/clear), using a webhook destination configured via environment variable. The webhook should include information on the currently-logged-in user if there is one. This should only fire on meaningful status changes, not on every update from the MCU.
15+
3. If we don't already have them, API endpoints to oops a machine, put it in maintenance lockout, or clear oops/lockout.
16+
17+
Our approach is to minimize change in this application, and let ESB do the heavy lifting; we just need to send events to it via webhook and give it the appropriate API endpoints to call.
18+
19+
## Implementation Plan
20+
21+
Commit-message prefix for this feature: ``ESB Support - {Milestone}.{Task}``.
22+
23+
### Findings / Scope
24+
25+
Mapping the three high-level requirements onto the current codebase:
26+
27+
1. **Machine list/status API** — *does not exist as JSON.* Today status is only
28+
available via the Prometheus ``/metrics`` endpoint and the Slack ``status``
29+
command. We will add a small read-only JSON endpoint.
30+
2. **Status-change webhook** — *does not exist.* The meaningful status changes
31+
already have well-defined chokepoints in ``models/machine.py`` where Slack
32+
notifications fire (login, logout, unauthorized/unknown fob, override login,
33+
oops, un-oops, lockout, unlock, reboot). We will fire the webhook from those
34+
same sites so it only fires on meaningful changes, never on ordinary MCU
35+
heartbeats.
36+
3. **Oops / lockout / clear API endpoints** — *already exist* in
37+
``views/machine.py``: ``POST``/``DELETE /api/machine/oops/<name>`` and
38+
``POST``/``DELETE /api/machine/locked_out/<name>``. ESB "clear" = ``DELETE``
39+
on whichever of the two is set. No new control endpoints are required; we
40+
only verify and document them.
41+
42+
The ESB issue also mentions storing 100–500 recent activity events per machine.
43+
Per the "let ESB do the heavy lifting" directive, **ESB stores that history from
44+
the webhook events**; MAC does not add history storage.
45+
46+
### Design Decisions (confirmed with maintainer)
47+
48+
* **Webhook destination:** ``STATUS_WEBHOOK_URL`` environment variable. When
49+
unset, webhook firing is disabled entirely (mirrors how Slack is disabled
50+
without its tokens).
51+
* **Authentication:** none. The POST is sent with no auth header.
52+
* **Delivery:** fire-and-forget (``asyncio.create_task``, like Slack) so the MCU
53+
response is never blocked, with a bounded **retry-with-backoff** in the
54+
background task and a per-attempt timeout. Failures are logged after retries
55+
are exhausted; ESB can reconcile via the status API.
56+
57+
### Shared status representation
58+
59+
A single ``status_dict`` (and derived ``status`` string) will be used by **both**
60+
the list API and the webhook payload so they never drift:
61+
62+
```json
63+
{
64+
"name": "planer",
65+
"display_name": "Planer",
66+
"status": "in_use", // one of: idle | in_use | oops | locked_out | unknown
67+
"relay": true,
68+
"oops": false,
69+
"locked_out": false,
70+
"current_user": { "account_id": "123", "full_name": "Jane Doe" }, // or null
71+
"last_checkin": 1720000000.0, // epoch seconds, or null if never
72+
"last_update": 1720000000.0 // epoch seconds, or null if never
73+
}
74+
```
75+
76+
The webhook payload is this dict plus an ``event`` field and a firing
77+
``timestamp``. Event values: ``login``, ``logout``, ``unauthorized``,
78+
``unknown_fob``, ``override_login``, ``oops``, ``unoops``, ``lockout``,
79+
``unlock``, ``reboot``.
80+
81+
### Milestone 1 — ESB-facing status & control API — ✅ COMPLETE
82+
83+
Added ``Machine.status`` and ``Machine.status_dict`` (shared representation),
84+
the ``GET /api/machines`` endpoint with ``MachineStatus`` / ``MachinesListResponse``
85+
schemas, and unit tests for both the model properties and the endpoint. Verified
86+
the existing oops/lockout endpoints already satisfy requirement #3 and are
87+
well-covered (no code change needed). All ``nox -s tests`` pass at 97% coverage.
88+
89+
* **1.1** Add a ``status`` property (derived string) and a ``status_dict``
90+
property on ``Machine`` (reading ``self.state``), producing the shared
91+
representation above. ``current_user`` serializes to ``{account_id, full_name}``
92+
or ``null``.
93+
* **1.2** Add ``GET /api/machines`` returning ``{"machines": [status_dict, ...]}``
94+
for all configured machines, sorted by name. Add ``pydantic`` response schemas
95+
in ``models/api_schemas.py`` and document the route with ``quart_schema``
96+
(tagged ``Admin``). Read-only and unauthenticated, consistent with the existing
97+
API surface.
98+
* **1.3** Confirm the existing oops/lockout endpoints satisfy ESB's control
99+
needs; add any missing test coverage. No behavior change expected.
100+
* **1.4** Unit tests for ``status``/``status_dict`` and the new endpoint
101+
(idle/in-use/oops/locked-out, with and without a current user).
102+
103+
### Milestone 2 — Status-change webhook — ✅ COMPLETE
104+
105+
Added ``src/dm_mac/webhook.py`` (``WebhookNotifier``: fire-and-forget delivery
106+
via ``aiohttp`` with bounded exponential-backoff retry, enabled by
107+
``STATUS_WEBHOOK_URL``), wired it into ``create_app`` as ``WEBHOOK_NOTIFIER``,
108+
and fired ``notify(...)`` from the meaningful status-change sites in
109+
``models/machine.py`` (login / logout / unauthorized / unknown_fob /
110+
override_login / oops / unoops / lockout / unlock / reboot), including the
111+
always-enabled RFID-tracking path. The notifier is resolved via
112+
``current_app`` in the request-context (MCU/API) paths and via the passed
113+
Slack handler's app (``slack.quart``) in the Slack-command path, which runs
114+
without a request context. Payloads reuse ``Machine.status_dict`` plus
115+
``event``, ``timestamp``, and a distinct ``user`` (event actor) field. Unit
116+
tests cover the notifier (payload/retry/backoff/disabled) and the wiring
117+
(each event fires; heartbeats do not; Slack path; disabled no-op). All
118+
``nox -s tests`` and ``nox -s mypy`` pass; ``webhook.py`` at 100% coverage.
119+
120+
* **2.1** New module ``src/dm_mac/webhook.py`` with a ``WebhookNotifier`` class:
121+
* Constructed from ``STATUS_WEBHOOK_URL``.
122+
* ``notify(machine, event, user=None)`` builds the payload from
123+
``machine.status_dict`` + ``event`` + ``timestamp`` and spawns a
124+
fire-and-forget ``asyncio.create_task`` delivery coroutine.
125+
* Delivery coroutine POSTs JSON via ``aiohttp`` (already a dependency) with a
126+
per-attempt timeout and retry-with-backoff (a small, bounded number of
127+
attempts with exponential backoff); logs a single error after exhaustion.
128+
* **2.2** In ``create_app``/``main`` (``__init__.py``), instantiate the notifier
129+
when ``STATUS_WEBHOOK_URL`` is set and store it as
130+
``app.config["WEBHOOK_NOTIFIER"]`` (default ``None``), exactly like
131+
``SLACK_HANDLER``.
132+
* **2.3** Fire ``notify(...)`` from the same async sites that emit Slack
133+
messages, so webhooks track meaningful changes only:
134+
* ``MachineState._handle_rfid_insert`` → ``login`` / ``unauthorized`` /
135+
``unknown_fob`` / ``override_login``
136+
* ``MachineState._handle_rfid_remove`` → ``logout``
137+
* ``MachineState._handle_reboot`` → ``reboot`` (and logout of any prior user)
138+
* ``Machine.oops`` / ``MachineState._handle_oops`` → ``oops``
139+
* ``Machine.unoops`` → ``unoops``; ``Machine.lockout`` → ``lockout``;
140+
``Machine.unlock`` → ``unlock``
141+
Each site fetches the notifier via ``current_app.config.get("WEBHOOK_NOTIFIER")``
142+
and no-ops when ``None``.
143+
* **2.4** Unit tests: payload shape per event, disabled-when-unset, retry/backoff
144+
on failure (mock ``aiohttp``), and non-firing on ordinary heartbeat updates.
145+
146+
### Milestone 3 — Acceptance Criteria — ✅ COMPLETE
147+
148+
Updated documentation (`docs/source/configuration.rst` env-var table;
149+
`docs/source/http-api.rst` with the Machine Status API and Status-Change
150+
Webhook sections; `CLAUDE.md` app-config, API endpoints, env vars, and a new
151+
Status-Change Webhook subsection). `README.rst` is a pointer to the full docs
152+
and needed no change. `GET /api/machines` is picked up automatically by the
153+
generated OpenAPI spec, and `dm_mac.webhook` by `sphinx-apidoc`. New code has
154+
unit-test coverage (`webhook.py` at 100%). All `nox` sessions pass: `tests`,
155+
`mypy`, `pre-commit`, `typeguard`, `docs`, and `safety`.
156+
157+
Note: `nox -s safety` was failing on `main` due to a pre-existing transitive
158+
`msgpack==1.2.0` advisory (GHSA-6v7p-g79w-8964), unrelated to this feature. To
159+
satisfy the "all nox sessions passing" criterion, `msgpack ^1.2.1` is pinned in
160+
`pyproject.toml`. Because `msgpack` is only pulled in via `cachecontrol` (a
161+
dev-group dependency) and is not used at runtime, the pin lives in
162+
`group.dev.dependencies` so it does not expand the production dependency set.
163+
164+
* **3.1** Update documentation: ``README.md`` (env var + endpoints if listed),
165+
``docs/source/configuration.rst`` (add ``STATUS_WEBHOOK_URL`` to the env-var
166+
table), ``docs/source/http-api.rst`` (document ``GET /api/machines`` and the
167+
webhook payload/events), and ``CLAUDE.md`` (env vars + architecture notes).
168+
Match the existing style and verbosity.
169+
* **3.2** Ensure all new code has appropriate unit-test coverage.
170+
* **3.3** All ``nox`` sessions pass (``tests``, ``mypy``, ``pre-commit``,
171+
``safety``, ``typeguard``, ``docs``).
172+
* **3.4** Move this file from ``docs/features/`` to ``docs/features/completed/``.
File renamed without changes.

‎docs/source/configuration.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,9 @@ Environment Variables
9999
* - ``SLACK_OOPS_CHANNEL_ID``
100100
- no
101101
- If using the Slack integration, the Channel ID of of the public channel where Oops and maintenance notices will be posted, and where machine status can be checked.
102+
* - ``STATUS_WEBHOOK_URL``
103+
- no
104+
- If set, the URL to POST a JSON status-change webhook to on every meaningful machine event (login, logout, unauthorized or unknown fob, override login, oops, un-oops, lockout, unlock, reboot). Disabled when unset. See :ref:`http-api.status-webhook`.
102105

103106
.. _configuration.machine-state-dir:
104107

‎docs/source/dm_mac.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,4 @@ Submodules
2727
dm_mac.neongetter
2828
dm_mac.slack_handler
2929
dm_mac.utils
30+
dm_mac.webhook

‎docs/source/dm_mac.webhook.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
dm\_mac.webhook module
2+
======================
3+
4+
.. automodule:: dm_mac.webhook
5+
:members:
6+
:private-members:
7+
:show-inheritance:
8+
:undoc-members:

‎docs/source/http-api.rst‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,93 @@ The ``display`` field is byte-identical between pre-feature and post-feature
8282
servers for any given (machine, operator, machine state) tuple. ``second_relay``
8383
configuration never causes LCD changes.
8484

85+
Machine Status API
86+
------------------
87+
88+
``GET /api/machines``
89+
90+
Returns the current status of every configured machine as JSON, sorted by
91+
machine name. This read-only endpoint is intended for external consumers (such
92+
as the `Equipment Status Board
93+
<https://github.com/DecaturMakers/equipment-status-board>`_) to poll or
94+
reconcile machine state. It is included in the OpenAPI spec above. Each machine
95+
entry has the shape:
96+
97+
::
98+
99+
{
100+
"name": "planer",
101+
"display_name": "Planer",
102+
"status": "in_use",
103+
"relay": true,
104+
"oops": false,
105+
"locked_out": false,
106+
"current_user": {"account_id": "123", "full_name": "Jane Doe"},
107+
"last_checkin": 1720000000.0,
108+
"last_update": 1720000000.0
109+
}
110+
111+
``status`` is a derived summary — one of ``locked_out``, ``oops``, ``in_use``,
112+
``idle``, or ``unknown`` (never checked in). ``current_user`` is ``null`` when
113+
no user is logged in. ``last_checkin`` and ``last_update`` are epoch seconds:
114+
``last_checkin`` is ``null`` until the machine's first check-in, and
115+
``last_update`` is ``null`` until its first *meaningful* state change. These are
116+
independent — a machine that has only sent idle heartbeats has a ``last_checkin``
117+
while ``last_update`` is still ``null``.
118+
119+
.. _http-api.status-webhook:
120+
121+
Status-Change Webhook
122+
---------------------
123+
124+
When the ``STATUS_WEBHOOK_URL`` environment variable is set (see
125+
:ref:`configuration.env-vars`), the server POSTs a JSON webhook to that URL on
126+
every *meaningful* machine status change — never on ordinary MCU heartbeats.
127+
This lets an external consumer (such as the Equipment Status Board) maintain
128+
its own activity history and live status without polling.
129+
130+
The webhook fires for these ``event`` values: ``login``, ``logout``,
131+
``unauthorized`` (known user lacking authorization), ``unknown_fob``,
132+
``override_login``, ``oops``, ``unoops``, ``lockout``, ``unlock``, and
133+
``reboot`` (MCU reboot detected).
134+
135+
The request body is the same per-machine object as ``GET /api/machines`` (so
136+
``current_user`` means the same thing) plus three fields:
137+
138+
* ``event`` — the status-change event name (see above).
139+
* ``timestamp`` — epoch seconds when the event fired.
140+
* ``user`` — the user *involved in this event* (the actor), as
141+
``{"account_id", "full_name"}`` or ``null``. This differs from
142+
``current_user`` for events like ``logout`` and ``unauthorized`` where a user
143+
acted but is not (or is no longer) logged in.
144+
145+
::
146+
147+
POST <STATUS_WEBHOOK_URL>
148+
Content-Type: application/json
149+
150+
{
151+
"name": "planer",
152+
"display_name": "Planer",
153+
"status": "idle",
154+
"relay": false,
155+
"oops": false,
156+
"locked_out": false,
157+
"current_user": null,
158+
"last_checkin": 1720000000.0,
159+
"last_update": 1720000000.5,
160+
"event": "logout",
161+
"timestamp": 1720000000.5,
162+
"user": {"account_id": "123", "full_name": "Jane Doe"}
163+
}
164+
165+
Delivery is fire-and-forget so it never blocks the MCU response: each webhook
166+
is sent on a background task that retries with exponential backoff and a
167+
per-attempt timeout, giving up (and logging an error) after a few attempts. No
168+
authentication header is sent. Consumers should treat delivery as best-effort
169+
and reconcile via ``GET /api/machines`` when needed. See
170+
:py:mod:`dm_mac.webhook` for details.
171+
85172
Prometheus Metrics
86173
------------------
87174

0 commit comments

Comments
 (0)