Skip to content

Support InputPath artifact values in @task(unwrap=...) (v0.1.33) - #78

Merged
Volv-G merged 1 commit into
masterfrom
unwrap-inputpath-artifacts
Oct 10, 2026
Merged

Volv-G merged 1 commit into
masterfrom
unwrap-inputpath-artifacts

Conversation

@Volv-G

@Volv-G Volv-G commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

What

@task(unwrap=...) now accepts artifact values. A parameter annotated dict[str, components.InputPath(T)] expands into one inputPath component input per call-site key (<param>__<key>, type T). The generated runtime wrapper re-assembles the dict with each key's local artifact path, so one inline task can consume an arbitrary number of upstream artifacts:

@task(image="python:3.12", unwrap="tables")
def merge_parquets(
    tables: dict[str, components.InputPath("ApacheParquet")],
    merged_path: components.OutputPath("ApacheParquet"),
): ...

merged = merge_parquets.named("merge")(tables={"shop": shop.table, "catalog": catalog.table})

Schema / compatibility

  • Artifact unwraps persist "kind": "input_path" on each key and "value_kind": "input_path" on the parameter in local_from_python.unwrapped_inputs, so hydrate regenerates the identical inputPath interface from the sidecar alone.
  • Value unwraps omit both fields: their persisted schema and fragment hash are byte-identical to before. Entries without kind are value inputs.
  • Artifact inputs always deserialize as str (the path), regardless of T.
  • OutputPath dict values are now rejected with a clear error (previously they silently became String value inputs).
  • Use builtin dict[...]; typing.Dict rejects non-type arguments such as InputPath(...).

Tests

  • Schema shape for value unwraps pinned (incl. a Dict[str, dict] JSON-object case) and for artifact unwraps.
  • Untyped InputPath(), OutputPath rejection, unknown persisted kind rejection.
  • Generated program end-to-end with real file paths, including two artifact unwraps + one value unwrap with hyphenated keys on one task.
  • Compile → sidecar → hydrate round trip asserting inputPath placeholders.

Version bumped to 0.1.33. README documents the new form.

`dict[str, components.InputPath(T)]` unwrap targets now expand into one
`inputPath` component input per call-site key. The generated runtime
wrapper re-assembles the dict with each key's local artifact path, so a
single inline task can consume an arbitrary number of upstream artifacts
(e.g. merging N Parquet tables).

The artifact kind is persisted in `local_from_python.unwrapped_inputs`
(`kind` per key, `value_kind` per parameter) so hydrate regenerates the
identical interface. Value unwraps omit both fields, keeping their
persisted schema and fragment hash byte-identical. `OutputPath` dict
values are now rejected instead of silently becoming String inputs.
@Volv-G
Volv-G requested a review from Ark-kun as a code owner October 10, 2026 14:22
@Volv-G
Volv-G merged commit ffee3c4 into master Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant