[jlewi]

Summary

• add a Runme-owned kernel manager and a direct Go-to-kernel ZeroMQ bridge
• replace the Jupyter Server registry and proxy routes with runner-owned /v1/jupyter/kernels lifecycle and channels routes
• let an authorized client provide the complete kernelspec-style argv used to start a kernel, including a Python virtual-environment executable
• generate and own the Jupyter connection file inside Runme, substitute its path for exactly one {connection_file} placeholder, and launch without a shell
• protect kernel lifecycle and channels routes with RunnerUserRole
• support Jupyter JSON channels messages while explicitly rejecting binary buffers and the v1.kernel.websocket.jupyter.org binary subprotocol
• instrument kernel lifecycle and bridge health with OpenTelemetry instruments and bounded attributes

Why

The old model depended on a separately managed Jupyter Server, token discovery, and an HTTP/WebSocket proxy. Runme now owns the kernel process and bridges the browser channels WebSocket directly to the kernel's loopback ZeroMQ sockets, so the server registry and proxy are unnecessary.

Kernel commands are selected by the client instead of being fixed in Runme server configuration. This lets a user start ipykernel from a particular Python or virtual environment without reconfiguring or restarting the runner.

Design document

The architecture, protocol decisions, threat model, rollout constraints, and implementation goal are documented in the direct WebSocket-to-kernel Runme notebook.

Related work and issues

• runmedev/web#151 — Jupyter server management is very brittle: primary motivation for removing the external Jupyter Server registry/proxy lifecycle.
• runmedev/web#63 — ipykernel/REPL support: original product use case for persistent kernel-backed execution.
• runmedev/web#164 — Jupyter Server token cached in memory: an example of token/proxy coupling removed by the direct design.
• runmedev/web#317: companion Web change for authenticated lifecycle calls, kernel selection and management, and client-supplied launch commands.

API

POST   /v1/jupyter/kernels
GET    /v1/jupyter/kernels
GET    /v1/jupyter/kernels/{kernel_id}
DELETE /v1/jupyter/kernels/{kernel_id}
POST   /v1/jupyter/kernels/{kernel_id}/interrupt
POST   /v1/jupyter/kernels/{kernel_id}/restart
WS     /v1/jupyter/kernels/{kernel_id}/channels

The legacy /v1/jupyter/servers/... routes are removed.

The start request contains a display name and the complete argument vector:

{
  "name": "project-venv",
  "argv": [
    "/workspace/.venv/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ]
}

argv[0] is resolved on the runner host. Runme validates the argument count and size, rejects NUL bytes, requires exactly one {connection_file} placeholder outside the executable, and invokes the process directly without shell expansion. Restart reuses the original launch specification.

Decisions that deserve careful review

These are the API, security, and compatibility choices that will become expensive to change after clients depend on them:

1. Authorization is also process-launch authority. RunnerUserRole permits the caller to supply an arbitrary executable and arguments that run as the runner's OS user. This is intentional; it must not be exposed to a weaker role without introducing a separate policy or allowlist.
2. The client owns argv; Runme owns connectivity. The client selects the environment and kernel command. Runme selects ports, creates the HMAC key and connection file, controls its path and permissions, and only substitutes {connection_file}.
3. No shell is involved. Arguments are passed directly to exec.Command. Shell strings, interpolation, pipelines, and environment-variable expansion are not part of the API.
4. Kernel identity is server-generated and opaque. A client-supplied name is only a display name. Lifecycle and channels operations address the kernel by generated kernel_id, allowing multiple kernels with the same name.
5. Restart preserves identity and launch intent. Restart reuses the stored launch specification and keeps the same kernel ID while replacing the process generation.
6. JSON channels are the compatibility boundary for this PR. Arbitrary binary buffers and widget parity are not claimed. The binary Jupyter WebSocket subprotocol is rejected explicitly rather than partially implemented.
7. The transport boundary is internal. Browser clients speak the standard channels WebSocket shape; the Runme-owned bridge handles Jupyter multipart framing, HMAC verification, routing identities, heartbeat, backpressure, and ZeroMQ sockets.

Reviewer guide

• pkg/agent/server/jupyter_kernels.go: public HTTP/WebSocket API shape, strict JSON decoding, lifecycle dispatch, status mapping, upgrade rules, and rejection of the binary subprotocol.
• pkg/agent/server/server.go: route registration and the RunnerUserRole authorization boundary.
• pkg/agent/jupyter/kernel_manager.go: launch-spec validation, connection-file ownership, direct process execution, state transitions, restart semantics, and cleanup.
• pkg/agent/jupyter/bridge.go: long-lived WebSocket-to-ZeroMQ bridge behavior, reconnects, backpressure, heartbeat handling, and close reasons.
• pkg/agent/jupyter/channel_json.go: JSON channels compatibility boundary and explicit binary-buffer rejection.
• pkg/agent/jupyter/message.go: Jupyter multipart framing limits and HMAC signing/verification.
• pkg/agent/jupyter/connection.go and socket.go: connection-file schema, loopback endpoint construction, and the narrow ZeroMQ socket abstraction.
• pkg/agent/jupyter/process_unix.go and process_windows.go: process-group isolation, interrupt, termination, and platform-specific guarantees.
• pkg/agent/jupyter/metrics.go: OpenTelemetry lifecycle and bridge instruments; reviewers should check attribute cardinality and whether each signal has an operational consumer.

Validation

• SHELL=/bin/bash ./runme run check
• focused Jupyter kernel-manager, protocol, bridge, server-route, and configuration tests
• direct authenticated Runme Web browser verification: list, start with a client-supplied virtual-environment command, restart, execute against, and stop runner-owned kernels
• comparison against the previous PR head for the locally flaky reconnect soak

Current limitations

• binary WebSocket buffers and widgets that depend on them are not supported yet
• Windows uses the isolated fallback process-control implementation
• the reconnect soak can time out under constrained local macOS runs; the same timeout reproduces at the previous PR head and is not introduced by the OpenTelemetry conversion

[jlewi]

Should we just allow the client to send the start command?

[sourishkrout]

The Jupyter kernel API runs commands chosen by the client and is always on. It needs a switch, off by default and dependent on runnerService, so operators choose whether to allow it. Mind adding a assistantServer.jupyterService or something along those lines, please, @jlewi?

[jlewi]

@sourishkrout Done in 194b88c. I added assistantServer.jupyterService, which is disabled by default. Enabling it requires assistantServer.runnerService: true; invalid configurations fail validation/server startup. When disabled, Runme does not initialize the kernel manager or register /v1/jupyter/kernels. The existing RunnerUserRole authorization remains in place when the service is enabled. I also added config/route tests and updated the agent documentation.