[tnazarew]

Fixes the intermittent bigquery failures reported in #339 and #341 (Spark 3.5.1, OpenLineage 1.54.0):

'run_event_test_1.38.0' In .outputs: Length does not match: expected 1 result: 0

(also dataSource_test_1.38.0, schema_test_1.38.0, columnLineage_test_1.38.0)

Cause

test.py ends right after the BigQuery write. Python exits and the SparkContext stops while the OpenLineage listener is still handling the write's SQLExecutionEnd. When GCS is slow, as in the failing runs, the listener falls behind. The BigQuery connector's lineage extension then throws, and the COMPLETE event is emitted with outputs: []. From the driver log of Dataproc job 6fe0adfc-0bf0-4ca1-ae01-4f1e77e8beb1 (run 39567597991):

WARN SparkOpenLineageExtensionVisitorWrapper: Can't invoke apply method on ...SparkOpenLineageExtensionVisitor class instance
Caused by: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.

The emitted events: START has 1 input and 1 output, but the COMPLETE event for writing_to_big_query.execute_save_into_data_source_command.e2e_dataset_wordcount_output_351_1540 has 1 input and 0 outputs. It passed on 2026-10-20 and failed on 2026-10-17 and 2026-10-21.

Change

After the write, the script waits up to 60 s for Spark's listener bus to drain (listenerBus().waitUntilEmpty(60000) through py4j), then calls spark.stop(). The OpenLineage listener now finishes while the context is still alive.

• waitUntilEmpty(long) and SparkContext.listenerBus() are public in the bytecode on Spark 3.1.3 and 3.3.2 (checked with javap). Tested at runtime on Spark 3.5.1.

Notes

• Because the failure is a race, one green run proves little. Repeated nightly runs will show whether the fix holds.
• This is also a producer-side problem. Any job whose last step is a BigQuery write can lose its output dataset this way, so it is worth reporting upstream (OpenLineage Spark integration / spark-bigquery-connector).
• bigquery_to_delta, bigquery_to_iceberg, cloudsql, spanner and hive also end without stopping Spark. They are left unchanged because none of them shows this failure.

🤖 Generated with Claude Code