[rorylshanks]

What

New command partforge upload-chain -source-job-id=<finished-job-id> that starts a new job whose source is the finished output of a previous job.

partforge upload-chain \
  -source-job-id=<finished-job-id> \
  -destination-schema-file=dest_v3.sql \
  -insert-select-file=insert_v3.sql \
  -bucket=partforge

• Each non-superseded, non-empty finished artifact of the source job becomes one READY source part of the new job.
• Only a manifest.json is uploaded per part (no data copy). New manifest field s3.source_finished_key points at the parent's finished prefix.
• The worker downloads that prefix, extracts all part tarballs, and attaches every part to the source table before running the insert-select.
• The new job's source table/schema is the parent job's destination table/schema, so the insert-select reads FROM the parent's destination table.
• -clickhouse-build defaults to the parent job's build; -copy-sql-from-job is supported.

Why

Lets a second rewrite run over an earlier rewrite's output without importing into ClickHouse and re-freezing.

Reviewer notes

• The source job must be fully finished (all non-SUPERSEDED rows FINISHED/IMPORTING/IMPORTED); otherwise the command fails with per-status counts.
• When a chained artifact contains more than one physical part, the worker disables insert chunking for that item (_part_offset ranges only partition a single part) and runs one insert. Single-part sources behave as before.
• Parent artifacts are not protected: delete-job -delete-s3, reset-job, or reset-compaction on the parent before the chained job finishes rewriting will break it. This is documented, not enforced.
• The e2e chain step exercises a single-part parent artifact; the multi-part fallback is not covered by e2e.

Testing

• go mod tidy, go test ./... pass.
• ./e2e/run.sh passes. New step chains e2e-job into dst.events_v3, verifies no source data was copied, runs workers, imports, and diffs against e2e/expected-chain.tsv.

🤖 Generated with Claude Code