[akupchinskiy]

Which issue does this PR close?

• Closes Casting Float64 / Float32 to decimal is wrong once the scaled value reaches 2^53 (295161.0 at scale 15 becomes 295160.999999999967232) #12048.

Rationale for this change

Float-to-decimal casts computed round(10^scale * x) in f64. From 2^53 on, the result is
wrong: 295161.0 at scale 15 becomes 295160.999999999967232. At scale 15 and above this
affects most values (see the issue).

What changes are included in this PR?

In arrow-cast/src/cast/decimal.rs:

Fix.

• Below 2^53 the result is unchanged: the rounded floating-point product.
• From 2^53 on, the shortest decimal representation of the input is scaled exactly in integer
  arithmetic. This is the representation arrow displays the float with, from ryu (already a
  dependency).
• An integer below 2^53 is its own shortest representation, so it is scaled without formatting.

This applies to float_to_decimal, float_to_decimal_opt and the array cast. The doc comment of
float_to_decimal describes the switch.

Array cast structure, which also makes the common casts faster (second commit).

• Pass 1 is a separate function over plain slices, 64 values at a time. It writes the values and
  two bitmaps, valid and exact (needs the exact path), as words. The kernels capture their
  invariants by value and convert with an unchecked i64 to native conversion; invalid slots are
  masked by valid. This replaces the __fixdfti call of the f64 to i128 conversion and the
  per-value precision check call.
• For precision <= 15, every valid value is below 10^15 < 2^53, so a product of 2^53 or more is a
  precision overflow either way. The kernel is the rounded product plus a range check against
  10^precision - 1, without branches, and vectorizes (on aarch64: fmul.2d, frinta.2d,
  fcvtas.2d, packed comparisons).
• From precision 16 on, the kernel takes the rounded product below 2^53. An integer below 2^53
  with scale 0 to 22 gets the exact i128 product. Other values are marked exact.
• Pass 2 visits only the exact slots that are not null, through set_indices(). In strict
  casts, the first failing row reports the same error as the scalar function.

Benchmarks: three new cases in arrow/benches/cast_kernels.rs, for Decimal128(38, 15) and
(38, 20) casts of random doubles and of integer doubles.

Are these changes tested?

Yes, in arrow-cast/src/cast/decimal.rs:

• test_float_to_decimal_large_products: the cases from the issue. It also checks that values
  below 2^53 are unchanged (for example, 1.005 at scale 2 is still 100), plus tiny, huge and
  non-finite inputs.
• test_float_to_decimal_matches_reference: 20,000 random doubles (all four decimal widths,
  scales -40 to 79) and 20,000 doubles whose product lies near 2^53. Below 2^53 they are checked
  against the old formula. From 2^53 on they are checked against an independent reference, which
  takes the significant-digit count from core's Display and rounds the exact {:.1100}
  expansion half-to-even to it. One-off runs with 1M values each also passed.
• test_cast_float_array_to_decimal_matches_scalar, test_low_precision_float_array_boundaries
  and test_float_array_integer_shortcut_boundaries check that the array cast matches the scalar
  functions:
  • safe and strict casts, with nulls, NaN and infinities;
  • mixes of large and small products, and the integer shortcut's range bound (±2^53, f64::MAX);
  • all decimal widths, precisions 1 to 76, negative and large scales;
  • non-zero array offsets;
  • in strict casts, the same first error.

Benchmarks (cast_kernels, float(32|64) (integers )?to decimal), Apple M2 Pro, median time per
array. The best of two interleaved runs of each version.

case	main	first commit (fix)	this PR	change
float64 to decimal128(32, 3), safe	34.8 µs	15.6 µs	11.0 µs	-68%
float32 to decimal32(7, 2), safe	18.4 µs	11.2 µs	5.5 µs	-70%
float32 to decimal32(7, 2), strict	20.1 µs	9.9 µs	5.6 µs	-72%
float64 to decimal128(20, 3), strict	35.7 µs	14.6 µs	11.1 µs	-69%
float64 to decimal256(40, -2), safe	375 µs	30.6 µs	14.0 µs	-96%
float64 to decimal64(3, 0), precision overflow	30.1 µs	15.3 µs	5.3 µs	-83%
float64 integers to decimal128(38, 15) (new)	34.8 µs	19.6 µs	16.8 µs	-52%
float64 to decimal128(38, 15) random (new)	34.8 µs	426 µs	424 µs	+1119%
float64 to decimal128(38, 20) random (new)	34.8 µs	426 µs	425 µs	+1120%

The only slower cases are high-scale casts of non-integers, where the previous results were
wrong (see the issue). There the cost is formatting the shortest digits with ryu. A ryu API
that returns the decimal digits and exponent without formatting would remove most of it.

Are there any significant user-facing changes?

Yes: casts whose scaled value is 2^53 or more now return the correct decimal. Results below 2^53
do not change. No public API changes.

AI disclosure

I found the bug while comparing query engines on a benchmark of mine. Claude Code helped write the
fix, the tests and this description, and Codex did review passes. I reviewed the code and checked
the results myself: tests, a comparison against an exact decimal reference on a few million
doubles, and benchmarks.

[mbutrovich]

Thanks @akupchinskiy. Could we settle on #12048 which decimal a double should map to before going further on the code? I left the evidence in a comment there.

If AI tools helped write this PR or the issue, could you say which parts, as the AI Generated Submissions section of CONTRIBUTING.md asks?

[mbutrovich]

Should the cast round the double's exact binary value at every magnitude, as I asked on #12048, instead of switching to the shortest representation at 2^53?

With the switch, the result of a cast depends on the target scale. On the head commit, 1.005 cast to Decimal128(38, 2) gives 1.00, while casting to Decimal128(38, 16) and then to Decimal128(38, 2) gives 1.01. On main both give 1.00.

The branch below 2^53 also isn't correctly rounded under either definition. On the head commit, float_to_decimal_opt::<Decimal128Type>(26.657349498176934, 38, 14) returns 2665734949817694, and both definitions give 2665734949817693. My comment on #12048 has the sample this came from.

Rounding the exact value needs no formatting. With x = m * 2^e and m < 2^53, the result is m * 10^scale * 2^e rounded half away from zero, which is one integer multiply and a shift. This sketch matches Rust's exact {:.N} formatting on 300,000 random doubles at scales 0 to 21 (except exact ties, where Rust rounds to even), and is exact for integers:

fn exact(x: f64, s: u32) -> Option<i128> {
    if !x.is_finite() { return None; }
    let bits = x.to_bits();
    let biased = ((bits >> 52) & 0x7ff) as i32;
    let frac = bits & ((1u64 << 52) - 1);
    let (m, e) = if biased == 0 { (frac, -1074) } else { (frac | (1 << 52), biased - 1075) };
    // m < 2^53 and 10^21 < 2^70, so the product fits
    let p = m as i128 * 10i128.pow(s);
    let mag = if e >= 0 {
        if p != 0 && (p.leading_zeros() as i32) <= e { return None; }
        p << e
    } else {
        let k = -e as u32;
        if k > 124 { 0 } else { (p + (1i128 << (k - 1))) >> k }
    };
    Some(if bits >> 63 == 1 { -mag } else { mag })
}

Wider scales would do the same in i256. It also handles Float32 input without a separate path, because widening to f64 is exact. With the shortest representation, 0.1f32 at Decimal128(38, 20) gives 0.10000000149011612000 on the head commit, which is not the digits arrow displays (0.1), so the doc comment's description doesn't hold for Float32.

[mbutrovich]

Could the two-pass array kernel (fast_pass, NativeFromI64, the valid and exact bitmaps) move to its own PR? It changes the common casts below 2^53 that the title doesn't cover, and it makes this fix hard to review on its own. A separate PR would let us review its speedups and its from_i64_wrapping and .expect("a failing row") paths separately, with the new benchmark cases in their own commit first so the benchmark bot can compare.

[akupchinskiy]

Thanks for the careful review, these are all fair points.

Happy to split it up. I'll strip this PR down to just the fix: compute the exact value directly like
in your sketch (with i256 for wider scales), no ryu. The two-pass kernel and the other speedups
can come later in a separate PR, with the benchmark cases in their own PR first. Fair warning, the
fix on its own will be slower in places: high-scale casts of non-integers now always take the exact
path (around 3x in my runs on Decimal128(38, 15)/(38, 20)), and depending on the semantics we
pick, the common casts may pay a small per-value check. I'll post numbers for the fix-only version.

On semantics, your 1.005 example convinced me: if the definition switches at 2^53, the result
depends on the path you cast through, and keeping the old rounding below 2^53 has the same problem
for cases like 26.657349498176934. So I lean towards the exact value everywhere.

The one thing that gives me pause is that today's behaviour below 2^53 is fairly common. DuckDB and
StarRocks round the f64 product the same way, so 0.15 at scale 1 gives 0.2 there, and about a
third of the ~1% of values that would change match the decimal string people expect. It's not
universal: ClickHouse truncates, and SQLite's round uses the exact value. Above 2^53, DuckDB,
ClickHouse and Polars all behave like arrow does today (295161.0 at scale 15 gives
295160.999999999967232). So I'd rather leave the call on below-2^53
behaviour to the maintainers. I have both versions working locally and can go either way.

On AI: yes. I found the bug myself while comparing engines on a benchmark. Claude Code helped write
the fix, tests and text, and Codex reviewed it. I verified everything myself. I've added a short
disclosure to the PR description, and I'll trim the wordier parts.