Repository navigation
Implement SecretsMasker for Variables and Connections in the Go SDK #67599
Description
Dependency
Wait until both #67317 and #67318 merge. #67317 lands the coordinator-side GetVariable / GetConnection TODO sites; #67318 lands the pkg/worker/runner.go task log handler that the masker plugs into. Until both are on main, the consumer-side hook this feature needs does not exist.
Background
Python's airflow.models.variable.Variable.get and airflow.models.connection.Connection.get register sensitive values with a SecretsMasker so any subsequent task log line that contains the secret is automatically redacted. The Go SDK currently does not — secrets retrieved via GetVariable / GetConnection flow straight into task logs unredacted.
The gap is documented as four coordinated TODOs across the SDK and the runtime. None of them is useful in isolation: log redaction needs both a producer (the lookup site registering the secret) and a consumer (the log handler running the redaction). Until all four land, the SDK ships without log-side masking parity.
HTTP-backed client (already on main):
// go-sdk/sdk/client.go - GetVariable
// TODO: register secret-named variables with a SecretsMasker so the
// returned value is automatically redacted from subsequent task logs,
// matching Python's airflow.models.variable.Variable.get behaviour.
// Pairs with the "TODO: mask secrets here" hook in
// pkg/worker/runner.go's task log handler — both halves are needed
// before secret masking actually works end-to-end.// go-sdk/sdk/client.go - GetConnection
// TODO: register Connection.Password and sensitive-keyed entries of
// Connection.Extra with a SecretsMasker so they are auto-redacted from
// subsequent task logs, matching Python's
// airflow.models.connection.Connection.get behaviour. Pairs with the
// "TODO: mask secrets here" hook in pkg/worker/runner.go's task log
// handler and the matching TODO on GetVariable above.Coordinator-mode client (lands with #67317):
// go-sdk/pkg/execution/client.go - GetVariable and GetConnection
// (same TODO text, same SecretsMasker hook)Log-handler hook (lands with #67318): the // TODO: mask secrets here comment in go-sdk/pkg/worker/runner.go is where the consumer side of the redaction belongs.
What needs to happen
- Design and add a
SecretsMaskertype to the Go SDK with a register-and-redact API roughly equivalent toairflow.utils.log.secrets_masker.SecretsMasker(case-insensitive secret-name detection, replace-with-***semantics, per-task scoping). - Register secret-named variables in both
sdk.client.GetVariableandexecution.CoordinatorClient.GetVariable. - Register
Connection.Passwordand sensitive-keyed entries ofConnection.Extrain bothsdk.client.GetConnectionandexecution.CoordinatorClient.GetConnection. - Wire the masker into the task log handler at the
// TODO: mask secrets heresite inpkg/worker/runner.goso emitted log lines are redacted before they leave the runtime. - Remove the four coordinated TODOs above.
Acceptance criteria
- Looking up a variable named e.g.
my_passwordvia eithersdk.clientorCoordinatorClientcauses any subsequent task log line containing the variable's value to be redacted to***(or the project's chosen sentinel). Connection.Passwordand sensitive-keyedConnection.Extraentries produce the same redaction behaviour from both clients.- Tests cover: (a) registration from each client, (b) redaction in the log handler, (c) negative case where a non-sensitive variable name is not registered.
- All four TODOs cross-referencing this work are removed from the codebase.
Context
- Originating review: self-review thread on Go-SDK: Add concurrent-safe coordinator comms, log handler, and client #67317 (coordinator comms + client PR).
- HTTP-side TODOs (already on
main):go-sdk/sdk/client.goGetVariableandGetConnection. - Coordinator-side TODOs (land with Go-SDK: Add concurrent-safe coordinator comms, log handler, and client #67317):
go-sdk/pkg/execution/client.goGetVariableandGetConnection. - Log-handler hook (lands with Go-SDK: Implement coordinator-mode runtime entry point and task runner #67318):
go-sdk/pkg/worker/runner.go. - Python parity:
airflow.utils.log.secrets_masker.SecretsMasker,airflow.models.variable.Variable.get,airflow.models.connection.Connection.get. - Related project board: Go-SDK (view)
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
uranusjr commentedon Jun 5, 2026
I think instead of re-implementing this in each language, we should move secrets and secret masking to the supervisor, so it’s implemented only once in Python. This is probably not strictly required yet, so we can move this beyond 3.3. Keeping this here for now so we remember to make a decision.
neochaotic commentedon Sep 20, 2026
I hit this exact fork building a Go, Airflow-compatible orchestrator, so here's a data point on "re-implement the masker per language SDK vs. centralize it in the supervisor", including a case that I think complicates the supervisor-side option.
What I landed on was to not do value-scanning log masking in either place, and lean on two things instead:
Channel separation. Keep the data channel (return value / XCom) strictly separate from the log channel, with no automatic data→log crossing. Task return values are never auto-journaled to the log (no
Done. Returned value was: <repr>line), and the runtime logs kwarg keys, not values. That removes the most common leak vectors structurally, with no per-line scanning anywhere.Name-based masking on the operator-facing surfaces, centrally. The API/UI responses that render variables and connection
extraredact by key name (is_sensitive_key→***) at serialization time. It's centralized and, importantly, works without the central component ever needing the secret value.To be precise, since it matters for the argument below: this covers the operator-facing read surfaces, not the task's. The task's own fetch necessarily returns the real value, so nothing is masked there.
Two things from that experience that bear on the decision here:
Supervisor-side value-scanning has a coverage hole for externally-resolved secrets. When a secret is resolved task/pod-side (e.g. an external secrets backend the control plane deliberately never sees), the supervisor doesn't hold the value and so can't scan-and-redact it. Centralizing in the supervisor only fully works if the supervisor is guaranteed to know every secret value, which stops being true once external/pod-side resolution is in play.
Structural separation does not cover derived secrets. The one vector neither channel-separation nor name-based masking catches is user code that derives/transforms a secret and then prints it. That specific case is exactly what a value-scanning
SecretsMaskerexists to close. So if it's in scope, some component that knows the value has to scan, and per the point above that component often has to be the runtime, not the supervisor.So my takeaway isn't "per-SDK" or "supervisor": it's that the decision hinges on two prior questions.
If it's helpful I'm happy to write up the channel-separation design in more detail.