[zhang-arvin]

Fixes #927

[the-other-tim-brown]

@zhang-arvin can we have this target main instead? if we want to fix this on the 0.4.0 release we can start a thread to raise a 0.4.1 release and then cherry-pick this to the release branch.

[vinishjail97]

@zhang-arvin can we have this target main instead? if we want to fix this on the 0.4.0 release we can start a thread to raise a 0.4.1 release and then cherry-pick this to the release branch.

@zhang-arvin I think this needs the xtable-spark-runtime to be merged in main. If you want to take over my existing work feel on xtable-spark-runtime using hudi version 1.2.1, free to do so. I can get to it in few days.
#838
#839

[zhang-arvin]

Checked the prerequisite before retargeting: xtable-spark-runtime is not on main yet (it only exists on branch-0.4), so this PR can't be rebased onto main as-is — three of the five files it touches (xtable-spark-runtime/.../TableSyncSpec.java, XTableSparkSync.java, XTableSyncService.java) don't exist there.

@vinishjail97 the two runtime PRs you linked — #838 and #839 — are still open and in draft, targeting main, so the spark-runtime isn't in main yet. So I'm leaving the base as branch-0.4 for now: the PR is mergeable and the change itself is small (route the partition spec through ParquetSourceConfig.PARTITION_FIELD_SPEC_CONFIG for Parquet sources instead of the Hudi key).

Plan:

• Retarget to main (and rebase) as soon as the spark-runtime lands in main — whichever path [836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery #838/[836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener) #839 ends up taking. Ping me when one merges and I'll re-point it immediately.
• In the meantime I'm happy to help on the runtime work: I can review [836] xtable-spark-runtime: thin drop-in bundle (packaging) + resilient target discovery #838/[836] xtable-spark-runtime: XTableSyncListener for in-job metadata sync (listener) #839, or if you'd rather hand off part of it (e.g. the Hudi 1.2.1 bump), say the word and I'll pick it up so we can unblock the main target sooner.

@the-other-tim-brown the 0.4.x cherry-pick path is orthogonal and fine by me — this change is small enough to cherry-pick cleanly once it's on main.