Repository navigation
Go SDK: bring the coordinator-protocol models back to the supervisor schema - #73963
Merged
jason810496 merged 2 commits intoOct 1, 2026
Merged
Conversation
…schema The models are generated from the supervisor wire-schema snapshot the Python Task SDK owns, and they had fallen two fields behind it: a snapshot can grow a property without the api_version moving, and the version constant's test is the only thing that was watching. Nothing else regenerated or verified the committed file, so the drift survived two releases in silence — msgpack drops a field the struct does not declare, so the supervisor sent multi_team and retry_reason and the Go SDK could not see them. The Dag specs had the same exposure and got a check for it. Making that check cover both generated surfaces is what keeps this from happening again, and is cheaper than watching one of them and hoping about the other. closes: apache#73954
henry3260
marked this pull request as ready for review
September 30, 2026 15:55
henry3260
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl and
potiuk
as code owners
September 30, 2026 15:55
henry3260
requested review from
guan404ming
and removed request for
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jscheffl and
potiuk
September 30, 2026 15:55
The models target pointed `git diff` at the whole genmodels package, but that package also holds hand-written code: gen.go carries the //go:generate directives, gen/ the custom generator, plus two test files. The hook's own files: regex fires on edits to exactly those, so editing a generator made the check report the author's own uncommitted gen.go edit as generator drift, and told them to `git add` the package directory, staging unrelated work with it. A directory also defeats the missing-file guard, which the specs target gets for free from naming a file: a directory outlives every generated file in it, so a moved or deleted models.gen.go passed silently — the drift the guard exists to catch. Naming the files also lets a report say which one drifted rather than which package, which is the one thing it has to get across. Three of them do not fit the singular wording the reports had, and a comma-joined list is not a runnable `git add`, so both follow from the same change.
guan404ming
approved these changes
Sep 30, 2026
guan404ming
left a comment
Member
There was a problem hiding this comment.
LGTM, thanks! Verified locally:
go generateoutput is stable across runs and matches the committed files.- The drift hook reports both targets up to date.
- New fields match the schema; no leftover references to the old hook id.
Just verified the function wise. However I'm not that familiar with Go, it would be better to have another eyes on this for coding style wise in this pr.
jason810496
approved these changes
Oct 1, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
Thanks for the improvement
jason810496
reviewed
Oct 1, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
I think we need to vendor the core schema to go-sdk dir and just only need to make sure the vendored schema and the gen spec won't be drift.
Since the upstrem CI fail with
All changes made by hooks:
diff --git a/go-sdk/pkg/execution/genmodels/discriminators.gen.go b/go-sdk/pkg/execution/genmodels/discriminators.gen.go
index 027a512..09da311 100644
--- a/go-sdk/pkg/execution/genmodels/discriminators.gen.go
+++ b/go-sdk/pkg/execution/genmodels/discriminators.gen.go
@@ -105,6 +105,7 @@ const (
TypeTaskStateStoreResult = "TaskStateStoreResult"
TypeTaskStatesResult = "TaskStatesResult"
TypeTriggerDagRun = "TriggerDagRun"
+ TypeUpdateDagRunNote = "UpdateDagRunNote"
TypeUpdateHITLDetail = "UpdateHITLDetail"
TypeValidateInletsAndOutlets = "ValidateInletsAndOutlets"
TypeVariableKeysResult = "VariableKeysResult"
@@ -364,6 +365,9 @@ func EnsureType(m any) any {
case TriggerDagRun:
b.Type = TypeTriggerDagRun
return b
+ case UpdateDagRunNote:
+ b.Type = TypeUpdateDagRunNote
+ return b
case UpdateHITLDetail:
b.Type = TypeUpdateHITLDetail
return b
diff --git a/go-sdk/pkg/execution/genmodels/models.gen.go b/go-sdk/pkg/execution/genmodels/models.gen.go
index 2274180..dee4700 100644
--- a/go-sdk/pkg/execution/genmodels/models.gen.go
+++ b/go-sdk/pkg/execution/genmodels/models.gen.go
@@ -407,12 +407,18 @@ type DagFileParseRequest struct {
// serialized DAGs,
// import errors and warnings to send back to the scheduler to store in the DB.
type DagFileParsingResult struct {
+ // DagSourceCodes corresponds to the JSON schema field "dag_source_codes".
+ DagSourceCodes DagSourceCodes `msgpack:"dag_source_codes,omitempty"`
+
// Fileloc corresponds to the JSON schema field "fileloc".
Fileloc string `msgpack:"fileloc"`
// ImportErrors corresponds to the JSON schema field "import_errors".
ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+ // ParsedDefinitions corresponds to the JSON schema field "parsed_definitions".
+ ParsedDefinitions []string `msgpack:"parsed_definitions,omitempty"`
+
// SerializedDags corresponds to the JSON schema field "serialized_dags".
SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
@@ -634,6 +640,17 @@ const DagRunTypeManual DagRunType = "manual"
const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
const DagRunTypeScheduled DagRunType = "scheduled"
+// Raw source code and its language identifier for a DAG definition.
+type DagSourceCode struct {
+ // Language corresponds to the JSON schema field "language".
+ Language string `msgpack:"language"`
+
+ // SourceCode corresponds to the JSON schema field "source_code".
+ SourceCode string `msgpack:"source_code"`
+}
+
+type DagSourceCodes map[string]DagSourceCode
+
type Data map[string]interface{}
type Defaults []string
@@ -1839,6 +1856,17 @@ type TriggerDagRun struct {
type TriggerKwargs map[string]interface{}
+type UpdateDagRunNote struct {
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
// Update the response content part of an existing Human-in-the-loop response.
type UpdateHITLDetail struct {
// ChosenOptions corresponds to the JSON schema field "chosen_options".
@@ -1854,6 +1882,19 @@ type UpdateHITLDetail struct {
Type string `msgpack:"type,omitempty"`
}
+// Variable schema for responses with fields that are needed for Runtime.
+type VariableResponse struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value interface{} `msgpack:"value"`
+}
+
+type Warnings []interface{}
+
+type VersionData map[string]interface{}
+
type ValidateInletsAndOutlets struct {
// TIID corresponds to the JSON schema field "ti_id".
TIID string `msgpack:"ti_id"`
@@ -1873,15 +1914,6 @@ type VariableKeysResult struct {
Type string `msgpack:"type,omitempty"`
}
-// Variable schema for responses with fields that are needed for Runtime.
-type VariableResponse struct {
- // Key corresponds to the JSON schema field "key".
- Key string `msgpack:"key"`
-
- // Value corresponds to the JSON schema field "value".
- Value interface{} `msgpack:"value"`
-}
-
type VariableResult struct {
// Key corresponds to the JSON schema field "key".
Key string `msgpack:"key"`
@@ -1893,10 +1925,6 @@ type VariableResult struct {
Value interface{} `msgpack:"value"`
}
-type VersionData map[string]interface{}
-
-type Warnings []interface{}
-
// One positional stub-task argument pulled from an upstream task's XCom.
type XComArgBinding struct {
// Kind corresponds to the JSON schema field "kind".
Error: Process completed with exit code 1.
1 task done
1 task done
This was referenced Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
go-sdk/pkg/execution/genmodels/models.gen.gois generated from the supervisor wire-schemasnapshot the Python Task SDK owns, and it had fallen two fields behind it. The snapshot moved
in #73224 and #73027; the models were last regenerated in #73531, before both.
Nothing failed, which is the problem: msgpack drops a field the struct does not declare, so
the supervisor sent
multi_teamandretry_reasonand the Go SDK could not see them. Theonly thing watching was
TestSupervisorSchemaVersionMatchesSnapshot, and a snapshot can growa property without
api_versionmoving — which is exactly what happened.The Dag specs had the same exposure and got a check for it in #73936. Making that check cover
both generated surfaces is what keeps this from happening again.
closes: #73954
What
models.gen.goregenerated:TIRunContext.MultiTeamandTaskState.RetryReasonare added,nothing is removed, and the rest of the diff is go-jsonschema reordering its type
declarations.
scripts/ci/prek/check_go_sdk_spec_drift.pybecomesscripts/ci/prek/check_go_sdk_generated_drift.pyand takes a table of targets instead of onehard-coded path — the Dag specs and the coordinator-protocol models — regenerating each and
asking Git whether the committed files changed. Each target names the schema it generates from
and what to weigh before committing: for the specs, the exclusion list in
internal/genspec/authoring.go; for the models, nothing, since nothing is excluded from them.The hook (
check-go-sdk-generated-drift) watches both schemas and both generators.go-sdk/README.mdno longer says the version-constant test makes a prek hook unnecessary,since that test is what missed this.
Was generative AI tooling used to co-author this PR?