fix: array to array cast by manuzhang · Pull Request #2897 · apache/datafusion-comet

manuzhang · 2025-12-13T15:28:08Z

Which issue does this PR close?

Part of #2766

Rationale for this change

What changes are included in this PR?

Fixed native array-to-array casting by recursively casting list elements in native/spark-expr/src/conversion_funcs/cast.rs.
Added native coverage for null-only array casts and safer binary-to-string conversion in native/spark-expr/src/conversion_funcs/cast.rs.
Added generated ArrayType -> ArrayType coverage in spark/src/test/scala/org/apache/comet/CometCastSuite.scala, with shared matrix logic for flat and nested arrays.
Improved array test data generation in spark/src/test/scala/org/apache/comet/CometCastSuite.scala with edge cases, reused date/timestamp/binary sources, and bounded float/double/long values to avoid timestamp overflow.
Stabilized cast and try_cast comparisons in spark/src/test/scala/org/apache/comet/CometCastSuite.scala by ordering on a synthetic row id.
Added a temporary JVM-side guard in spark/src/main/scala/org/apache/comet/expressions/CometCast.scala to stop unsupported Array[Date] casts from reaching native.

How are these changes tested?

Added tests.

Co-authored-by: @codex

codecov-commenter · 2025-12-14T15:46:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.53%. Comparing base (f09f8af) to head (5f7b017).
⚠️ Report is 815 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2897      +/-   ##
============================================
+ Coverage     56.12%   59.53%   +3.40%     
- Complexity      976     1376     +400     
============================================
  Files           119      167      +48     
  Lines         11743    15496    +3753     
  Branches       2251     2569     +318     
============================================
+ Hits           6591     9225    +2634     
- Misses         4012     4971     +959     
- Partials       1140     1300     +160

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

manuzhang · 2025-12-16T04:34:49Z

Created #2914 for the failed test

  "...The value -96833550.[]7BD of the type "DEC..." did not equal "...The value -96833550.[0]7BD of the type "DEC..." (CometCastSuite.scala:1328)

manuzhang · 2025-12-16T15:00:48Z

Submitted #2916 to fix the failed test.

native/spark-expr/src/conversion_funcs/cast.rs

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

github-actions · 2026-03-13T02:16:52Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

coderfender · 2026-03-18T08:02:49Z

Taking a look

manuzhang · 2026-03-18T08:15:17Z

@coderfender Sorry, there are more test failures that I need to fix first.

manuzhang · 2026-03-18T15:12:14Z

@coderfender It's ready for review now.

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

native/spark-expr/src/conversion_funcs/cast.rs

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

coderfender · 2026-03-21T06:32:16Z

@manuzhang , Thank you the code / feature looks good over all to me. It would be great if we could add some more comprehensive tests both on spark and rust side (including different data types, eval modes , edge cases etc) and benchmarks (can be a follow up PR too). Thank you

coderfender

added few minor comments but PR looks good overall

parthchandra

+1 for adding the tests requested by @coderfender

manuzhang · 2026-03-26T07:40:56Z

I'll add more DataTypes to the test once #3768 is resolved.

coderfender · 2026-03-27T01:20:47Z

Sure @manuzhang

parthchandra

This is looking good. Some minor comments.

parthchandra · 2026-04-03T22:18:56Z

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

+    testArrayCastMatrix(types, ArrayType(_), generateArrays(100, _))
+  }
+
+  ignore("cast nested ArrayType to nested ArrayType") {


Is there a github issue for this? Can we reference it here to document why it is being ignored?

parthchandra · 2026-04-03T22:26:32Z

spark/src/main/scala/org/apache/comet/expressions/CometCast.scala

    (fromType, toType) match {
      case (dt: ArrayType, _: ArrayType) if dt.elementType == NullType => Compatible()
+      case (ArrayType(DataTypes.DateType, _), ArrayType(toElementType, _))
+          if toElementType != DataTypes.StringType &&


Shouldn't DateType -> IntegerType (and maybe others) be possible too?

parthchandra · 2026-04-03T22:29:40Z

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

+            CometCast.isSupported(fromWrappedType, toWrappedType, None, CometEvalMode.TRY) ==
+              Compatible()
+          castTest(
+            generateInput(fromType),


minor: we can probably call generateInput once in the outer loop

parthchandra · 2026-04-03T22:33:36Z

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

+        val fromWrappedType = wrapType(fromType)
+        val toWrappedType = wrapType(toType)
+        if (fromType != toType &&
+          testNames.contains(name) &&


manuzhang force-pushed the test-array-cast branch 4 times, most recently from 864a25f to 92a7867 Compare December 14, 2025 15:23

manuzhang mentioned this pull request Dec 16, 2025

cast_decimal_to_int16_down formats decimal value incorrectly #2914

Closed

manuzhang force-pushed the test-array-cast branch from 92a7867 to 5f7b017 Compare January 5, 2026 08:09

parthchandra reviewed Jan 10, 2026

View reviewed changes

native/spark-expr/src/conversion_funcs/cast.rs Outdated Show resolved Hide resolved

coderfender reviewed Jan 10, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Show resolved Hide resolved

github-actions bot added the Stale label Mar 13, 2026

manuzhang force-pushed the test-array-cast branch from 5f7b017 to 9c99e19 Compare March 17, 2026 07:46

github-actions bot removed the Stale label Mar 18, 2026

manuzhang force-pushed the test-array-cast branch 2 times, most recently from d9770c7 to fae74f6 Compare March 18, 2026 07:26

manuzhang requested review from coderfender and parthchandra March 18, 2026 07:39

manuzhang force-pushed the test-array-cast branch 3 times, most recently from b93600c to b73aff5 Compare March 18, 2026 09:50

manuzhang mentioned this pull request Mar 18, 2026

fix: native error message not propagated to SparkException on empty errorClass #3727

Merged

parthchandra reviewed Mar 19, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Show resolved Hide resolved

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Show resolved Hide resolved

native/spark-expr/src/conversion_funcs/cast.rs Show resolved Hide resolved

coderfender reviewed Mar 20, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Outdated Show resolved Hide resolved

coderfender reviewed Mar 20, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Show resolved Hide resolved

coderfender reviewed Mar 20, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Outdated Show resolved Hide resolved

coderfender reviewed Mar 22, 2026

View reviewed changes

parthchandra reviewed Mar 24, 2026

View reviewed changes

manuzhang mentioned this pull request Mar 27, 2026

fix: handle inf/-inf/nan in ShimSparkErrorConverter cast overflow #3768

Merged

manuzhang force-pushed the test-array-cast branch 3 times, most recently from 74a608e to 47186ca Compare March 31, 2026 04:51

manuzhang requested review from coderfender and parthchandra April 3, 2026 16:03

parthchandra reviewed Apr 3, 2026

View reviewed changes

fix: array to array cast

460aa7e

manuzhang force-pushed the test-array-cast branch from 47186ca to 460aa7e Compare April 5, 2026 16:23

Conversation

manuzhang commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

codecov-commenter commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

manuzhang commented Dec 16, 2025

Uh oh!

manuzhang commented Dec 16, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2026

Uh oh!

coderfender commented Mar 18, 2026

Uh oh!

manuzhang commented Mar 18, 2026

Uh oh!

manuzhang commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderfender commented Mar 21, 2026

Uh oh!

coderfender left a comment

Choose a reason for hiding this comment

Uh oh!

parthchandra left a comment

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Mar 26, 2026

Uh oh!

coderfender commented Mar 27, 2026

Uh oh!

parthchandra left a comment

Choose a reason for hiding this comment

Uh oh!

parthchandra Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

parthchandra Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

parthchandra Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

parthchandra Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manuzhang commented Dec 13, 2025 •

edited

Loading

codecov-commenter commented Dec 14, 2025 •

edited

Loading