Skip to content

feat: Mark array_compact as Compatible and improve test coverage #3889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove wants to merge 2 commits into
base: main
Choose a base branch
from
+28 −4
Open

feat: Mark array_compact as Compatible and improve test coverage #3889

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1248366
fix: mark array_compact as Compatible and improve test coverage
andygrove Apr 2, 2026
042a784
fix: handle KnownNotContainsNull in Spark 4.0 shim
andygrove Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion spark/src/main/scala/org/apache/comet/serde/arrays.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ object CometArrayRepeat extends CometExpressionSerde[ArrayRepeat] {

object CometArrayCompact extends CometExpressionSerde[Expression] {

override def getSupportLevel(expr: Expression): SupportLevel = Incompatible(None)
override def getSupportLevel(expr: Expression): SupportLevel = Compatible()
Copy link
Copy Markdown
Contributor

@kazuyukitanimura kazuyukitanimura Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove getSupportLevel entirely


override def convert(
expr: Expression,
Expand Down
6 changes: 6 additions & 0 deletions spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,12 @@ trait CometExprShim extends CommonStringExprs {
// val optExpr = scalarFunctionExprToProto("width_bucket", childExprs: _*)
// optExprWithInfo(optExpr, wb, wb.children: _*)

// KnownNotContainsNull is a TaggingExpression added in Spark 4.0 that only
// changes schema metadata (containsNull = false). It has no runtime effect,
// so we pass through to the child expression.
case k: KnownNotContainsNull =>
exprToProtoInternal(k.child, inputs, binding)

case _ => None
}
}
Expand Down
24 changes: 21 additions & 3 deletions spark/src/test/resources/sql-tests/expressions/array/array_compact.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,35 @@
-- specific language governing permissions and limitations
-- under the License.

-- ConfigMatrix: parquet.enable.dictionary=false,true

statement
CREATE TABLE test_array_compact(arr array<int>) USING parquet

statement
INSERT INTO test_array_compact VALUES (array(1, NULL, 2, NULL, 3)), (array()), (NULL), (array(NULL, NULL)), (array(1, 2, 3))

query spark_answer_only
-- column argument
query
SELECT array_compact(arr) FROM test_array_compact

-- literal arguments
query spark_answer_only
query
SELECT array_compact(array(1, NULL, 2, NULL, 3))

-- string element type
statement
CREATE TABLE test_array_compact_str(arr array<string>) USING parquet

statement
INSERT INTO test_array_compact_str VALUES (array('a', NULL, 'b', NULL, 'c')), (array()), (NULL), (array(NULL, NULL)), (array('', NULL, '', NULL))

query
SELECT array_compact(arr) FROM test_array_compact_str

-- double element type
query
SELECT array_compact(array(1.0, NULL, 2.0, NULL, 3.0))

-- nested array type (removes null arrays from outer, preserves null elements in inner)
query
SELECT array_compact(array(array(1, NULL, 3), NULL, array(NULL, 2, 3)))
Loading