Skip to content

Go SDK: bring the coordinator-protocol models back to the supervisor schema - #73963

Merged
jason810496 merged 2 commits into
apache:mainfrom
henry3260:go-sdk-regenerate-supervisor-models
Oct 1, 2026
Merged

jason810496 merged 2 commits into
apache:mainfrom
henry3260:go-sdk-regenerate-supervisor-models

Conversation

@henry3260

@henry3260 henry3260 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

go-sdk/pkg/execution/genmodels/models.gen.go is generated from the supervisor wire-schema
snapshot the Python Task SDK owns, and it had fallen two fields behind it. The snapshot moved
in #73224 and #73027; the models were last regenerated in #73531, before both.

Nothing failed, which is the problem: msgpack drops a field the struct does not declare, so
the supervisor sent multi_team and retry_reason and the Go SDK could not see them. The
only thing watching was TestSupervisorSchemaVersionMatchesSnapshot, and a snapshot can grow
a property without api_version moving — which is exactly what happened.

The Dag specs had the same exposure and got a check for it in #73936. Making that check cover
both generated surfaces is what keeps this from happening again.

closes: #73954

What

models.gen.go regenerated: TIRunContext.MultiTeam and TaskState.RetryReason are added,
nothing is removed, and the rest of the diff is go-jsonschema reordering its type
declarations.

scripts/ci/prek/check_go_sdk_spec_drift.py becomes
scripts/ci/prek/check_go_sdk_generated_drift.py and takes a table of targets instead of one
hard-coded path — the Dag specs and the coordinator-protocol models — regenerating each and
asking Git whether the committed files changed. Each target names the schema it generates from
and what to weigh before committing: for the specs, the exclusion list in
internal/genspec/authoring.go; for the models, nothing, since nothing is excluded from them.
The hook (check-go-sdk-generated-drift) watches both schemas and both generators.

go-sdk/README.md no longer says the version-constant test makes a prek hook unnecessary,
since that test is what missed this.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

…schema

The models are generated from the supervisor wire-schema snapshot the Python
Task SDK owns, and they had fallen two fields behind it: a snapshot can grow a
property without the api_version moving, and the version constant's test is the
only thing that was watching. Nothing else regenerated or verified the committed
file, so the drift survived two releases in silence — msgpack drops a field the
struct does not declare, so the supervisor sent multi_team and retry_reason and
the Go SDK could not see them.

The Dag specs had the same exposure and got a check for it. Making that check
cover both generated surfaces is what keeps this from happening again, and is
cheaper than watching one of them and hoping about the other.

closes: apache#73954
The models target pointed `git diff` at the whole genmodels package, but that
package also holds hand-written code: gen.go carries the //go:generate
directives, gen/ the custom generator, plus two test files. The hook's own
files: regex fires on edits to exactly those, so editing a generator made the
check report the author's own uncommitted gen.go edit as generator drift, and
told them to `git add` the package directory, staging unrelated work with it.

A directory also defeats the missing-file guard, which the specs target gets
for free from naming a file: a directory outlives every generated file in it,
so a moved or deleted models.gen.go passed silently — the drift the guard
exists to catch.

Naming the files also lets a report say which one drifted rather than which
package, which is the one thing it has to get across. Three of them do not fit
the singular wording the reports had, and a comma-joined list is not a runnable
`git add`, so both follow from the same change.

@guan404ming guan404ming left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Verified locally:

  • go generate output is stable across runs and matches the committed files.
  • The drift hook reports both targets up to date.
  • New fields match the schema; no leftover references to the old hook id.

Just verified the function wise. However I'm not that familiar with Go, it would be better to have another eyes on this for coding style wise in this pr.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the improvement

@jason810496
jason810496 merged commit e6a9de5 into apache:main Oct 1, 2026
97 checks passed

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to vendor the core schema to go-sdk dir and just only need to make sure the vendored schema and the gen spec won't be drift.

Since the upstrem CI fail with

All changes made by hooks:
diff --git a/go-sdk/pkg/execution/genmodels/discriminators.gen.go b/go-sdk/pkg/execution/genmodels/discriminators.gen.go
index 027a512..09da311 100644
--- a/go-sdk/pkg/execution/genmodels/discriminators.gen.go
+++ b/go-sdk/pkg/execution/genmodels/discriminators.gen.go
@@ -105,6 +105,7 @@ const (
 	TypeTaskStateStoreResult        = "TaskStateStoreResult"
 	TypeTaskStatesResult            = "TaskStatesResult"
 	TypeTriggerDagRun               = "TriggerDagRun"
+	TypeUpdateDagRunNote            = "UpdateDagRunNote"
 	TypeUpdateHITLDetail            = "UpdateHITLDetail"
 	TypeValidateInletsAndOutlets    = "ValidateInletsAndOutlets"
 	TypeVariableKeysResult          = "VariableKeysResult"
@@ -364,6 +365,9 @@ func EnsureType(m any) any {
 	case TriggerDagRun:
 		b.Type = TypeTriggerDagRun
 		return b
+	case UpdateDagRunNote:
+		b.Type = TypeUpdateDagRunNote
+		return b
 	case UpdateHITLDetail:
 		b.Type = TypeUpdateHITLDetail
 		return b
diff --git a/go-sdk/pkg/execution/genmodels/models.gen.go b/go-sdk/pkg/execution/genmodels/models.gen.go
index 2274180..dee4700 100644
--- a/go-sdk/pkg/execution/genmodels/models.gen.go
+++ b/go-sdk/pkg/execution/genmodels/models.gen.go
@@ -407,12 +407,18 @@ type DagFileParseRequest struct {
 // serialized DAGs,
 // import errors and warnings to send back to the scheduler to store in the DB.
 type DagFileParsingResult struct {
+	// DagSourceCodes corresponds to the JSON schema field "dag_source_codes".
+	DagSourceCodes DagSourceCodes `msgpack:"dag_source_codes,omitempty"`
+
 	// Fileloc corresponds to the JSON schema field "fileloc".
 	Fileloc string `msgpack:"fileloc"`
 
 	// ImportErrors corresponds to the JSON schema field "import_errors".
 	ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
 
+	// ParsedDefinitions corresponds to the JSON schema field "parsed_definitions".
+	ParsedDefinitions []string `msgpack:"parsed_definitions,omitempty"`
+
 	// SerializedDags corresponds to the JSON schema field "serialized_dags".
 	SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
 
@@ -634,6 +640,17 @@ const DagRunTypeManual DagRunType = "manual"
 const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
 const DagRunTypeScheduled DagRunType = "scheduled"
 
+// Raw source code and its language identifier for a DAG definition.
+type DagSourceCode struct {
+	// Language corresponds to the JSON schema field "language".
+	Language string `msgpack:"language"`
+
+	// SourceCode corresponds to the JSON schema field "source_code".
+	SourceCode string `msgpack:"source_code"`
+}
+
+type DagSourceCodes map[string]DagSourceCode
+
 type Data map[string]interface{}
 
 type Defaults []string
@@ -1839,6 +1856,17 @@ type TriggerDagRun struct {
 
 type TriggerKwargs map[string]interface{}
 
+type UpdateDagRunNote struct {
+	// Note corresponds to the JSON schema field "note".
+	Note interface{} `msgpack:"note"`
+
+	// TIID corresponds to the JSON schema field "ti_id".
+	TIID string `msgpack:"ti_id"`
+
+	// Type corresponds to the JSON schema field "type".
+	Type string `msgpack:"type,omitempty"`
+}
+
 // Update the response content part of an existing Human-in-the-loop response.
 type UpdateHITLDetail struct {
 	// ChosenOptions corresponds to the JSON schema field "chosen_options".
@@ -1854,6 +1882,19 @@ type UpdateHITLDetail struct {
 	Type string `msgpack:"type,omitempty"`
 }
 
+// Variable schema for responses with fields that are needed for Runtime.
+type VariableResponse struct {
+	// Key corresponds to the JSON schema field "key".
+	Key string `msgpack:"key"`
+
+	// Value corresponds to the JSON schema field "value".
+	Value interface{} `msgpack:"value"`
+}
+
+type Warnings []interface{}
+
+type VersionData map[string]interface{}
+
 type ValidateInletsAndOutlets struct {
 	// TIID corresponds to the JSON schema field "ti_id".
 	TIID string `msgpack:"ti_id"`
@@ -1873,15 +1914,6 @@ type VariableKeysResult struct {
 	Type string `msgpack:"type,omitempty"`
 }
 
-// Variable schema for responses with fields that are needed for Runtime.
-type VariableResponse struct {
-	// Key corresponds to the JSON schema field "key".
-	Key string `msgpack:"key"`
-
-	// Value corresponds to the JSON schema field "value".
-	Value interface{} `msgpack:"value"`
-}
-
 type VariableResult struct {
 	// Key corresponds to the JSON schema field "key".
 	Key string `msgpack:"key"`
@@ -1893,10 +1925,6 @@ type VariableResult struct {
 	Value interface{} `msgpack:"value"`
 }
 
-type VersionData map[string]interface{}
-
-type Warnings []interface{}
-
 // One positional stub-task argument pulled from an upstream task's XCom.
 type XComArgBinding struct {
 	// Kind corresponds to the JSON schema field "kind".
Error: Process completed with exit code 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Go SDK: models.gen.go is behind the supervisor schema snapshot

3 participants