[SPARK-60360][SQL] Validate file source writes before deleting existing data in static overwrite

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR keeps the existing data when a static-mode overwrite fails the driver side checks of the write. The S3A magic committer on S3 benefits the most:

• The magic committer makes new files visible only at job commit, but a static-mode overwrite deletes the existing data in Spark itself (FileCommitProtocol.deleteWithJob, which PathOutputCommitProtocol does not override) before the job starts.
• Previously, Spark ran this delete before the driver side checks. An invalid write, such as a typo in the compression codec, bulk-deleted every object under the target S3 path and then failed.
• Now such a write fails before Spark deletes anything, so the existing data stays as is and no delete request is sent to S3.

InsertIntoHadoopFsRelationCommand used to delete the matching output before calling FileFormatWriter.write, which then checks the data schema (verifySchema, checkFieldNames) and the write options (FileFormat.prepareWrite). Now FileFormatWriter.write takes a beforeSetupJob callback (a no-op by default) that runs right before committer.setupJob, and InsertIntoHadoopFsRelationCommand deletes the matching output there. The delete still happens before setupJob, where the magic committer creates its __magic_job-<uuid> directory under the destination, so valid writes are unchanged.

Why are the changes needed?

When these checks fail, the existing data has already been deleted.

spark.range(3).write.parquet(path)
spark.range(5).write.mode("overwrite").option("compression", "bogus").parquet(path)
// SparkIllegalArgumentException: [CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] ...
spark.read.parquet(path)
// AnalysisException: [PATH_NOT_FOUND] Path does not exist: ...

The same happens with an unsupported data type (e.g. a map column in CSV), with binaryFile, which does not support writes, and with INSERT OVERWRITE into tables (including converted Hive tables) and directories. This has been the behavior since Spark 1.4.

Does this PR introduce any user-facing change?

Yes. A static-mode overwrite that fails these checks keeps the existing data instead of deleting it, with any committer. Valid writes are unchanged.

How was this patch tested?

Added tests to InsertSuite of sql/core and sql/hive. They fail without this fix. The delete does not depend on the committer, so the tests use the default one.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

[dongjoon-hyun]

Thank you so much always, @HyukjinKwon ! 😄

[dongjoon-hyun]

Also, FYI, @peter-toth .

[peter-toth]

Thanks for the PR, @dongjoon-hyun!

The static-mode delete now runs in a beforeSetupJob callback inside FileFormatWriter.write. That is after verifySchema, checkFieldNames and prepareWrite, and still before setupJob. No built-in prepareWrite touches the output path, so a valid write is unchanged. The 6 new tests pass on the head, and I checked that all of them fail on the base.

Non-blocking

• 1. saveAsTable still loses the table: On an existing V1 table, spark.range(5).write.mode("overwrite").option("compression", "bogus").saveAsTable("t") fails with CODEC_NOT_AVAILABLE and leaves no t behind, on both the base and the head. SaveAsV1TableCommand drops the table at ddl.scala:1064 before the CTAS reaches FileFormatWriter. Could you note it as out of scope in the description, or file a follow-up?