Skip to content

fix(kubernetes): Do not store secret values in the serialized launcher data - #395

Draft
yuechao-qin wants to merge 2 commits into
masterfrom
ycq/redact-secret-values-in-serialized-launcher-data
Draft

yuechao-qin wants to merge 2 commits into
masterfrom
ycq/redact-secret-values-in-serialized-launcher-data

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A component input whose value comes from the secret store is passed to the container on its
command line. The container needs the real value, but Tangle also serialises the launched
Kubernetes object into ContainerExecution.launcher_data, which is stored and later served by
the execution API — so the plaintext value is persisted and readable there.

Before / after

GET /api/executions/{execution_node_id}/container_state

  .debug_info.kubernetes_job.debug_job.spec.template.spec.containers[0]     # Job launcher
  .debug_info.kubernetes.debug_pod.spec.containers[0]                       # Pod launcher
   "image": "alpine",
   "command": ["curl"],
   "args": [
     "-H",
-    "Authorization: Bearer s3cr3t-token-value",
+    "<REDACTED_SECRET_VALUE>",
     "https://example.test/api"
   ]

The Pod Kubernetes runs is unchanged — it still receives the real value, which is how the
process gets it. Only the serialised copy changes, in the row and therefore in the response.
The positions come from a Pod annotation written at launch:

   "metadata": {
     "annotations": {
+      "tangleml.com/launchers.kubernetes.redacted_command_line_parts": "{\"args\": [1]}",
       "cloud-pipelines.net/execution_node.id": "…"
     }
   }

(values from a local run of the launcher against the test component; the jq paths are the ones a real run produces)

Fix

Blank the secret-derived entries in every serialised copy of the object, leaving the object
sent to the cluster untouched. Detection and redaction cannot be the same place: the launcher
knows which inputs are secrets but must still send the real values, and the serialiser is the
only point both writes pass through but no longer knows which entries were secret.

A Pod annotation carries positions — never values — between the two:

  tangleml.com/launchers.kubernetes.redacted_command_line_parts: {"command": [3], "args": [1, 3]}
  _prepare_kubernetes_pod
     collect the resolved value of every input whose `dynamic_data` has a "secret" kind
     resolve_container_command_line(...)
     record the positions of parts that contain one of those values  ->  pod annotation
        |
        |  the real values go to the cluster; the Job's pod template carries the annotation,
        |  so every Pod the Job controller creates (including retries) inherits it
        v
  _serialize_kubernetes_object_to_compact_dict
     read the annotation off the object in hand, replace those parts with
     "<REDACTED_SECRET_VALUE>"

Working on the annotation rather than on the handle keeps this to four small hunks in one file:
no change to to_dict/from_dict, to either handle's constructor, or to the orchestrator. It
also means a Pod first seen on a later poll — a retry attempt, which Tangle never creates and
cannot anticipate — is redacted by the same code path, because the annotation arrives on the
object itself.

Both sides of the orchestrator's "has anything changed?" comparison are redacted by the same
deterministic rule, so a settled container still produces no write.

Behaviour notes

  • Detection is by value and fail-closed: a part that contains a secret value is blanked, so
    an identical non-secret argument is blanked too.
  • Granularity is the whole part — a ConcatPlaceholder building --token=<secret> loses the
    flag name with it.
  • An empty secret value records nothing, rather than matching every part.
  • Only the main container is touched.
  • Unlike the task annotations in this module, which are Tangle's own, this key is set on the Pod
    and therefore has to be a Kubernetes qualified name — one / after the prefix, dots after.
  • Rows written before this change are unaffected.

Tests

tests/launchers/kubernetes/test_kubernetes_secret_redaction.py

  • the launched Pod keeps the real value
  • a serialised Pod blanks only the secret part
  • a serialised Job blanks it in the pod template
  • the annotation omits a field that holds no secret ({"args": [1]}, not {"command": [], …})
  • multiple distinct secrets across both command and args, with one value used twice
  • the same multi-secret container through the Job path
  • every annotation key the launcher puts on the Pod is a valid Kubernetes annotation key
  • a concatenated --token=<secret> part is redacted whole
  • an empty secret value redacts nothing
  • a Pod without secrets gets no annotation

Full suite: 544 passed. black clean.

…r data

Secret input values reach the container on its command line, which also put them
into the serialized Pod/Job stored in `ContainerExecution.launcher_data` and served
by the execution API.

The launcher now records the positions of the secret-derived command line entries in
a Pod annotation, and the serializer blanks those entries on the way out. The object
sent to the cluster is unchanged, and Pods created later by a Job (retry attempts)
inherit the annotation from the pod template, so they are covered by the same path.
Two defects in the previous commit, both found by review:

A Kubernetes annotation key allows a single `/` after the optional DNS prefix,
so `tangleml.com/launchers/kubernetes/redacted_command_line_parts` would have
been rejected by the API server - failing every launch that consumes a secret.
The task annotations in this module are Tangle's own and have no such limit;
this one is set on the Pod. Key is now dotted, like the other Pod annotations.

Detection compared a command line part to the secret value for equality, so a
`ConcatPlaceholder` part such as `--token=<secret>` was not recorded and the
value was still stored. A part that contains a secret value is now redacted
whole, and an empty secret value records nothing rather than everything.

Both are covered by tests that fail on the previous code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant