Skip to content

fix(api): Redact container command lines from the served debug info - #396

Draft
yuechao-qin wants to merge 1 commit into
masterfrom
ycq/redact-command-line-on-the-read-path
Draft

yuechao-qin wants to merge 1 commit into
masterfrom
ycq/redact-command-line-on-the-read-path

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A component input whose value comes from the secret store is passed to the container on its
command line, so the launched object a launcher stores in ContainerExecution.launcher_data can
hold that value in plaintext. get_container_execution_state serves that dict straight back as
debug_info, so any authenticated user reading an execution's container state can read it.

#395 stops the Kubernetes launchers writing it. This is the other half: a filter on the way out,
for the rows that fix cannot reach.

Before / after

GET /api/executions/{execution_node_id}/container_state

Kubernetes shape — .debug_info.kubernetes_job.debug_job.spec.template.spec.containers[0],
and the same again for every .debug_info.kubernetes_job.debug_pods[*].spec.containers[0]:

   "name": "main",
   "image": "alpine",
-  "command": ["sh", "-c"],
-  "args": ["echo", "plaintext-secret-value"]
+  "command": ["<REDACTED_ON_READ>", "<REDACTED_ON_READ>"],
+  "args": ["<REDACTED_ON_READ>", "<REDACTED_ON_READ>"]

Docker shape — .debug_info.docker.debug_container:

   "Path": "python",
-  "Args": ["-c", "print(1)", "plaintext-secret-value"],
+  "Args": ["<REDACTED_ON_READ>", "<REDACTED_ON_READ>", "<REDACTED_ON_READ>"],
   "Config": {
     "Image": "python:3.12-slim",
-    "Entrypoint": ["python", "-c"],
-    "Cmd": ["print(1)", "plaintext-secret-value"]
+    "Entrypoint": ["<REDACTED_ON_READ>", "<REDACTED_ON_READ>"],
+    "Cmd": ["<REDACTED_ON_READ>", "<REDACTED_ON_READ>"]
   }

Everything that is not a command line list is served unchanged — image, Path,
Config.Image, job_name — and the stored row is not touched, so the plaintext is still in
the database until a backfill runs.

(shapes and values from the tests in this PR, not a live capture)

Fix

  WRITE PATH                                  READ PATH
  launcher to_dict                            GET /api/executions/{id}/container_state
      |  redaction #1 (#395, selective)               |
      v                                               v
  launcher_data column  -------------------->  _redact_command_lines
      a row may be:                                   |  redaction #2 (this PR, blanket)
       - new, already clean                            v
       - older, still plaintext  ----------->   debug_info (clean either way)
       - from a launcher that does not redact

At serving time the stored dict is all there is, and nothing in it records which parts came from
a secret — so every element of every command line list goes, at any depth.

Depth and spelling both matter:

Launcher shape Where the command line sits
Pod debug_pod.spec.containers[*].command / .args
Job debug_job.spec.template.spec.containers[*], plus a copy per Pod under debug_pods
Docker a raw docker inspect dict: Args, Config.Cmd, Config.Entrypoint

Hence a recursive walk and a case-insensitive match on command, args, cmd, entrypoint.

Notes

  • The marker is <REDACTED_ON_READ>, deliberately different from a launcher-side marker: it
    says the serving layer hid this, and therefore that the row may still hold the value. It
    claims nothing about the element having been a secret, because at this point that is unknown.
  • Element count is preserved — the argument count is most of the debugging value that remains.
  • The helper returns a copy and never mutates its input, so a read request cannot mark the ORM
    instance's JSON column dirty. Asserted by a test.
  • This is a backstop, not a cleanup: the database still holds the plaintext of rows already
    written, which needs a separate one-off pass.
  • Trade-off accepted: operators lose the real command line from this endpoint.
  • Not covered: env lists. No launcher here stores a secret in one — the Kubernetes launchers
    pass values on the command line, and the launcher that does deliver secrets by environment
    persists no spec at all. Easy to extend if that changes.

Tests

tests/test_execution_nodes_api_service.py, in TestGetContainerExecutionState

  • a Job-shaped row: the pod template and the per-Pod copy under debug_pods are both
    blanked, while image and job_name are served unchanged
  • the docker inspect shape: Args, Config.Cmd, Config.Entrypoint blanked; Path (a
    string) and Config.Image untouched
  • serving does not modify the stored row: the row keeps its plaintext and session.dirty is
    empty

Each fails on the code without this change. Full suite: 537 passed. black clean.

A container that consumes a secret as an input value receives it on its command
line, so the launched object a launcher stores in `ContainerExecution.launcher_data`
can hold that value in plaintext, and `get_container_execution_state` serves that
dict to any authenticated user as `debug_info`.

Launchers can blank it on the way in, but a row written before they did, or by a
launcher that does not, still holds it. At serving time the stored dict is all there
is and nothing records which parts came from a secret, so the filter is deliberately
non-selective: every element of every command line list, at any depth, is replaced
with `<REDACTED_ON_READ>`. Depth and spelling both matter - a Pod keeps containers
at `spec.containers`, a Job at `spec.template.spec.containers` plus a copy per Pod,
and the Docker launcher stores a raw `docker inspect` dict spelling the same fields
`Args`, `Config.Cmd` and `Config.Entrypoint`.

The helper returns a copy, so a read request cannot mark the JSON column dirty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant