Repository navigation
fix(kubernetes): Do not store secret values in the serialized launcher data - #395
Draft
yuechao-qin wants to merge 2 commits into
Draft
yuechao-qin wants to merge 2 commits into
yuechao-qin wants to merge 2 commits into
Conversation
…r data Secret input values reach the container on its command line, which also put them into the serialized Pod/Job stored in `ContainerExecution.launcher_data` and served by the execution API. The launcher now records the positions of the secret-derived command line entries in a Pod annotation, and the serializer blanks those entries on the way out. The object sent to the cluster is unchanged, and Pods created later by a Job (retry attempts) inherit the annotation from the pod template, so they are covered by the same path.
Two defects in the previous commit, both found by review: A Kubernetes annotation key allows a single `/` after the optional DNS prefix, so `tangleml.com/launchers/kubernetes/redacted_command_line_parts` would have been rejected by the API server - failing every launch that consumes a secret. The task annotations in this module are Tangle's own and have no such limit; this one is set on the Pod. Key is now dotted, like the other Pod annotations. Detection compared a command line part to the secret value for equality, so a `ConcatPlaceholder` part such as `--token=<secret>` was not recorded and the value was still stored. A part that contains a secret value is now redacted whole, and an empty secret value records nothing rather than everything. Both are covered by tests that fail on the previous code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A component input whose value comes from the secret store is passed to the container on its
command line. The container needs the real value, but Tangle also serialises the launched
Kubernetes object into
ContainerExecution.launcher_data, which is stored and later served bythe execution API — so the plaintext value is persisted and readable there.
Before / after
GET /api/executions/{execution_node_id}/container_state"image": "alpine", "command": ["curl"], "args": [ "-H", - "Authorization: Bearer s3cr3t-token-value", + "<REDACTED_SECRET_VALUE>", "https://example.test/api" ]The Pod Kubernetes runs is unchanged — it still receives the real value, which is how the
process gets it. Only the serialised copy changes, in the row and therefore in the response.
The positions come from a Pod annotation written at launch:
"metadata": { "annotations": { + "tangleml.com/launchers.kubernetes.redacted_command_line_parts": "{\"args\": [1]}", "cloud-pipelines.net/execution_node.id": "…" } }(values from a local run of the launcher against the test component; the jq paths are the ones a real run produces)
Fix
Blank the secret-derived entries in every serialised copy of the object, leaving the object
sent to the cluster untouched. Detection and redaction cannot be the same place: the launcher
knows which inputs are secrets but must still send the real values, and the serialiser is the
only point both writes pass through but no longer knows which entries were secret.
A Pod annotation carries positions — never values — between the two:
Working on the annotation rather than on the handle keeps this to four small hunks in one file:
no change to
to_dict/from_dict, to either handle's constructor, or to the orchestrator. Italso means a Pod first seen on a later poll — a retry attempt, which Tangle never creates and
cannot anticipate — is redacted by the same code path, because the annotation arrives on the
object itself.
Both sides of the orchestrator's "has anything changed?" comparison are redacted by the same
deterministic rule, so a settled container still produces no write.
Behaviour notes
an identical non-secret argument is blanked too.
ConcatPlaceholderbuilding--token=<secret>loses theflag name with it.
maincontainer is touched.and therefore has to be a Kubernetes qualified name — one
/after the prefix, dots after.Tests
tests/launchers/kubernetes/test_kubernetes_secret_redaction.py{"args": [1]}, not{"command": [], …})commandandargs, with one value used twice--token=<secret>part is redacted wholeFull suite: 544 passed.
blackclean.