Skip to content

Send events asynchronously - #758

Open
teta2k wants to merge 8 commits into
mainfrom
refactor/ipc-request-reply-threadless
Open

teta2k wants to merge 8 commits into
mainfrom
refactor/ipc-request-reply-threadless

Conversation

@teta2k

@teta2k teta2k commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Previously, every IPC call created a thread. When communication stalled, the caller could time out while that thread continued holding its socket and request payload.

This replaces per-call threads with socket deadlines for synchronous calls and one persistent sender for events. Event reporting enqueues locally without waiting for IPC delivery. Each application process can buffer up to 1,000 pending events, with a five-second IPC timeout per delivery attempt. When the queue is full, new events are dropped. Forked workers initialize their own empty queue and sender state.

Rate-limit and firewall-list checks remain synchronous with their existing 10 ms timeouts. Configuration/statistics synchronization now uses a five-second IPC timeout for both the initial and periodic sync, allowing larger batches to finish. The first request can perform the initial sync synchronously and wait for it; periodic synchronization runs in a background thread. This synchronization exchanges local data with the background process, while cloud communication runs separately. Other query timeouts are unchanged. Connections and completed payloads are released on success and failure.

Local Unix-socket benchmarks on CPython 3.14, using six alternating paired batches against main (e70fdfbb), measured the current PR implementation (741671ea):

Case Main This change
Median sender caller time, 1 KiB event 223 µs 1 µs
Completed event throughput, 1 KiB 4,082/s 10,294/s
Completed property-read query throughput 4,544/s 13,146/s
Completed event throughput, 1 MiB 534/s 591/s

The table reports medians across the six batches. Measurements cover IPC transport with prepared messages. Event throughput includes draining the sender and confirming receiver counts. Application throughput depends on the workload.

Validation:

  • 223 local tests passed: 214 focused SDK tests and nine startup/framework tests. Current Python 3.8 unit-test CI and Windows FFI smoke-test CI also passed.
  • With a three-second receiver pause, all 20 events arrived in order for both 1 KiB and 1 MiB payloads. Callers returned after local enqueue, and no original event payloads remained retained after delivery.
  • In an artificial stalled-query probe, 20 calls retained 20 threads and 5 MiB of original payloads on main; the current implementation retained neither after timeout.
  • Tests cover queue overflow, concurrent startup, inherited locks, disconnects, partial I/O, missing replies, and payload release while the sender is idle.

Delivery remains best effort: failed sends are not replayed, and pending events can be lost when the worker process exits. An IPC acknowledgement confirms acceptance by the background process, not cloud delivery. Socket deadlines do not interrupt serialization or deserialization.

@teta2k
teta2k marked this pull request as ready for review October 19, 2026 14:32
Comment thread aikido_zen/background_process/comms.py Outdated
Comment thread aikido_zen/background_process/comms.py Outdated
Comment thread aikido_zen/background_process/comms.py Outdated
@teta2k teta2k changed the title Remove per-call IPC threads and fix stalled-call retention Replace IPC worker threads with socket timeouts Oct 19, 2026
@teta2k teta2k changed the title Replace IPC worker threads with socket timeouts Prevent IPC thread leaks and send events without blocking requests Oct 19, 2026
@teta2k teta2k changed the title Prevent IPC thread leaks and send events without blocking requests Send events asynchronously Oct 19, 2026
@teta2k
teta2k force-pushed the refactor/ipc-request-reply-threadless branch from 5917ecc to c748f14 Compare October 19, 2026 23:09
@teta2k
teta2k force-pushed the refactor/ipc-request-reply-threadless branch from c748f14 to 2aa0cbc Compare October 19, 2026 23:17
Comment thread aikido_zen/background_process/commands/put_event.py
@teta2k
teta2k force-pushed the refactor/ipc-request-reply-threadless branch from 769b1f1 to 741671e Compare October 20, 2026 11:03
Comment thread aikido_zen/background_process/ipc.py
Comment thread aikido_zen/background_process/__init__.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants