rand and randn with an integer seed fall back to Spark without a fallback reason #7479

Describe the bug

A Project or Filter that contains rand(<int>) or randn(<int>), for example rand(42), is
not converted, and Comet records no fallback reason on the operator or on any of its expressions.
In SQL, an integer literal seed such as rand(42) has type IntegerType. Only a LongType
seed, such as rand(42L), is converted.

With spark.comet.explain.fallback.strict.enabled=true, which CometTestBase turns on for
Comet's own suites, the query fails with:

java.lang.IllegalStateException: Comet did not convert Filter but recorded no fallback reason on
the operator or any of its expressions. Add a withFallbackReason call stating why conversion
failed.

(or ... did not convert Project ... for a projection). Without the strict check, the only
reason shown is the generic "<operator> is not supported" from
CometExecRule.reportUnexplainedFallback, which does not say what to change.

Steps to reproduce

Run on main (0f7f8e453) with the default profile (Spark 4.1.3), in a CometTestBase suite:

spark.range(0, 200, 1, numPartitions = 1)
  .selectExpr("CAST(id AS INT) AS id", "IF(id % 3 = 0, NULL, CAST(id AS INT)) AS y")
  .write.parquet(path)
spark.read.parquet(path).createOrReplaceTempView("t")

These four queries each fail with the error above:

SELECT id FROM t WHERE y IS NOT NULL AND rand(42) < 0.5;
SELECT id FROM t WHERE abs(rand(42)) < 0.5;
SELECT id, rand(1) AS r FROM t;
SELECT id, randn(1) AS r FROM t;

The same queries with a Long seed are converted:

In the analyzed plans, the seed of rand(1) and rand(42) is IntegerType, and the seed of
rand(1L) is LongType.

Cause

In spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala, the seeded serde's
getSupportLevel returns Compatible() for any Literal seed. But CometRand.convert and
CometRandn.convert build the proto from extractSeedFromExpr:

protected def extractSeedFromExpr(expr: Expression): Option[Long] = {
  expr match {
    case Literal(seed: Long, _) => Some(seed)
    case Literal(null, _) => Some(0L)
    case _ => None
  }
}

An IntegerType literal matches neither case, so convert returns None without recording a
reason.

Expected behavior

rand and randn with an integer literal seed are converted, as they are with a Long seed.
Spark accepts an IntegerType or LongType seed and widens an integer seed to a long
(RDG.seed in Spark's randomExpressions.scala). If an integer seed is not supported,
the conversion should record a fallback reason that says so.

Additional context

A possible fix: accept an Int literal in extractSeedFromExpr and widen it to a Long, or have
getSupportLevel reject what convert cannot take, with a reason.

🤖 Generated with Claude Code