Using validity bitmap as Arrow does #14409

JaroslavTulach · 2025-12-01T15:06:53Z

Pull Request Description

to align our column storage data with Arrow as Use Arrow format for LongStorage to make exchange between JVMs and Python possible #13851 requires
to address the isNothing vs. validity clash
let's use getValidityMap throughout our storages

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

All code follows the
Scala,
Java,
Unit tests will pass before integration
Benchmarks to verify speed is on par with previous isNothing behavior
- stdlib first run - showed some slowdown
- stdlib second run with shared allTrue and 0c07d60

Akirathan

LGTM

std-bits/table/src/main/java/org/enso/table/data/column/builder/DoubleBuilder.java

...its/table/src/main/java/org/enso/table/data/column/storage/ColumnStorageWithValidityMap.java

…mal.from.mixed.column

JaroslavTulach · 2025-12-02T14:57:50Z

/runner/_work/enso/enso/target/test-results/Table_Tests/JUnit.xml 7124✅ 60❌

… in Table_Spec

JaroslavTulach · 2025-12-02T16:33:47Z

Down to 27 ❌ failed tests. Still a lot of failures like:

    - [FAILED] should be able to replace multiple columns selected by type [19ms]
        Reason: An unexpected panic was thrown: java.lang.AssertionError: Null values found in a column, so a null key should be found
        at <Unknown Location> related to com.oracle.truffle.host.HostObject.doInvoke
        at <enso> In_Memory_Table_Implementation.merge.handle_java_errors(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:538:30-129)
        at <enso> Java_Problems.with_problem_aggregator(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:118:14-25)
        at <enso> In_Memory_Table_Implementation.merge<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:537-539)
        at <enso> In_Memory_Table_Implementation.merge.handle_java_errors<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:535:25-30)
        at <enso> Panic.catch(Internal)
        at <enso> In_Memory_Table_Implementation.merge.handle_java_errors<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:534-535)
        at <enso> Panic.catch(Internal)
        at <enso> In_Memory_Table_Implementation.merge.handle_java_errors<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:533-535)
        at <enso> Panic.catch(Internal)
        at <enso> In_Memory_Table_Implementation.merge.handle_java_errors(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:532-535)
        at <enso> In_Memory_Table_Implementation.merge(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:536-539)
        at <enso> Table.merge(distribution/lib/Standard/Table/0.0.0-dev/src/Table.enso:3052:9-123)
        at <enso> case_branch<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:60:34-199)
        at <enso> Any.if_not_error(distribution/lib/Standard/Base/0.0.0-dev/src/Error.enso:364:32-36)
        at <enso> case_branch<arg-1>(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:42-62)
        at <enso> Any.if_not_error(distribution/lib/Standard/Base/0.0.0-dev/src/Error.enso:364:32-36)
        at <enso> case_branch(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:42-62)
        at <enso> case_branch(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:23-62)
        at <enso> Replace_Helpers.replace(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:16-62)
        at <enso> case_branch.replace_columns(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:20:69-166)
        at <enso> Array_Like_Helpers.fold.f(distribution/lib/Standard/Base/0.0.0-dev/src/Internal/Array_Like_Helpers.enso:286:22-48)
        at <enso> if_then_else(distribution/lib/Standard/Base/0.0.0-dev/src/Data/Range.enso:399:31-50)
        at <enso> if_then_else.go(distribution/lib/Standard/Base/0.0.0-dev/src/Data/Range.enso:398-400)
        at <enso> if_then_else(distribution/lib/Standard/Base/0.0.0-dev/src/Data/Range.enso:401:13-30)
        at <enso> Range.fold(distribution/lib/Standard/Base/0.0.0-dev/src/Data/Range.enso:395-401)
        at <enso> Array_Like_Helpers.fold(distribution/lib/Standard/Base/0.0.0-dev/src/Internal/Array_Like_Helpers.enso:287:5-39)
        at <enso> Vector.fold(distribution/lib/Standard/Base/0.0.0-dev/src/Data/Vector.enso:372:9-50)
        at <enso> case_branch(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:20:13-167)
        at <enso> Replace_Helpers.replace(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Replace_Helpers.enso:16-62)
        at <enso> In_Memory_Table_Implementation.replace(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Table_Implementation.enso:542:9-118)
        at <enso> Table.replace(distribution/lib/Standard/Table/0.0.0-dev/src/Table.enso:3138:9-127)
        at <enso> Replace_Spec.add_replace_specs.prefix<arg-0>(test/Table_Tests/src/Common_Table_Operations/Join/Replace_Spec.enso:86:22-77)
        at <enso> Replace_Spec.add_replace_specs.prefix<arg-1>(test/Table_Tests/src/Common_Table_Operations/Join/Replace_Spec.enso:86:22-95)

JaroslavTulach · 2025-12-02T16:53:03Z

Down to 2 failures:

sbt:enso> runEngineDistribution --run test/Table_Tests
5531 tests succeeded.
2 tests failed.
118 tests skipped.
34 groups skipped.

Failed tests: 'will.fail.if.key.columns.of.the.lookup.table.contain.Nothing|should.fail.on.null.key.values.in.lookup.table'

one failure is at

    - [FAILED] will fail if key columns of the lookup table contain Nothing [70ms]
        Reason: Expected an error Null_Values_In_Key_Columns but no error occurred, instead got: (Table.Value (In_Memory_Table.Value org.enso.table.data.table.Table@acc8cbf) In_Memory_Table_Implementation) (at test/Table_Tests/src/Common_Table_Operations/Join/Lookup_Spec.enso:312:13-58).

and second failure is at

[FAILED] [In-Memory] Table.replace: [21/22, 510ms]                    
  | => enso / runEngineDistribution 347s
    - [FAILED] should fail on null key values in lookup table [21ms]
        Reason: Expected an error Null_Values_In_Key_Columns but no error occurred, instead got: (Table.Value (In_Memory_Table.Value org.enso.table.data.table.Table@4403ddef) In_Memory_Table_Implementation) (at test/Table_Tests/src/Common_Table_Operations/Join/Replace_Spec.enso:136:13-88).

Co-authored-by: Pavel Marek <pavel.marek@enso.org>

enso-bot · 2025-12-03T05:02:42Z

Jaroslav Tulach reports a new STANDUP for yesterday (2025-12-02):

Progress: .

all Table_Tests passing Using validity bitmap as Arrow does #14409
Unexpected Type error when checking a Java type, currently only seen on CI #10250 can be fixed by Reintroduce Charset return type ascription #13031
NI in Bazel: https://github.com/enso-org/enso/pull/14351/files#r2580489249
planning meeting It should be finished by 2025-12-05.

JaroslavTulach · 2025-12-03T05:22:01Z

org_enso_benchmarks_generated_Column_Arithmetic_1000000_Multiply_Constant is 2x slower

Column_Arithmetic_1000000_Multiply_Constant

org_enso_benchmarks_generated_Column_Arithmetic_1000000_Plus_Constant is 2x slower

org_enso_benchmarks_generated_Column_Arithmetic_1000000_Plus_Overflowing also 2x slower
org_enso_benchmarks_generated_Column_Cast_1000000_Float_To_Integer is up to 2x slower

org_enso_benchmarks_generated_Column_Cast_1000000_Integer_To_Float is up to 2x slower
org_enso_benchmarks_generated_Column_Cast_1000000_Integer_To_Smaller_Integer_Fitting is up to 2x slower
org_enso_benchmarks_generated_Column_Cast_1000000_Integer_To_Text few percent slower

many org_enso_benchmarks_generated_Column_Comparisons are slightly slower

jdunkerley

Looks good - a couple of small fixes please

std-bits/table/src/main/java/org/enso/table/data/column/builder/NumericBuilder.java

std-bits/table/src/main/java/org/enso/table/data/column/operation/unary/CountNothing.java

std-bits/table/src/main/java/org/enso/table/data/column/operation/unary/IsNothingOperation.java

std-bits/table/src/main/java/org/enso/table/data/column/storage/ComputedLongStorage.java

std-bits/table/src/main/java/org/enso/table/data/column/operation/unary/CountNothing.java

hubertp

If increased memory is an issue then why not use something like RoaringBitmap?

hubertp · 2025-12-03T09:46:07Z

std-bits/table/src/main/java/org/enso/table/data/column/operation/IsInOperation.java

    if (keepValue) {
      var newIsNothing =
-          boolStorage.isNegated() ? or(isNothing, values) : orNot(isNothing, values, checkedSize);
+          boolStorage.isNegated()
+              ? or(isNothing, values, checkedSize)
+              : orNot(isNothing, values, checkedSize);
+      newIsNothing.flip(0, checkedSize);
      return new BoolStorage(values, newIsNothing, checkedSize, boolStorage.isNegated());
    } else {
      var newIsNothing =
-          boolStorage.isNegated() ? orNot(isNothing, values, checkedSize) : or(isNothing, values);
+          boolStorage.isNegated()
+              ? orNot(isNothing, values, checkedSize)
+              : or(isNothing, values, checkedSize);
+      newIsNothing.flip(0, checkedSize);
      return new BoolStorage(values, newIsNothing, checkedSize, !boolStorage.isNegated());
    }


DRY (although it has been like that before)

I had touched the code in previous commits

and then I greatly regretted that

in bfa1c62 I had to return the original code back

it is probably more important to keep the semantics correct

than to fulfill any artificial code metrics

but the current version is also slower (because of calling flip two times)

thus I may give it one more try...

JaroslavTulach · 2025-12-03T14:38:29Z

If increased memory is an issue then why not use something like RoaringBitmap?

the primary goal here is to adhere to Arrow format
if the increased memory usage is a necessary to be Arrow compatible, then @jdunkerley agreed to to pay the price
if there is a way to be Arrow compatible and avoid the overhead
- 1M of LongStorage non-null long items used to take 8MB of memory
- now we need additional 1/8MB for BitSet full of 1
then I happily implement such an improvement

Validity bitmaps

Any value in an array may be semantically null, whether primitive or nested type.

All array types, with the exception of union types (more on these later), utilize a dedicated memory buffer, known as the validity (or “null”) bitmap, to encode the nullness or non-nullness of each value slot. The validity bitmap must be large enough to have at least 1 bit for each array slot.

reading this section from arrow columnar spec doesn't leave much hope we could do without the 125KB of 0xff

enso-bot · 2025-12-04T05:02:32Z

Jaroslav Tulach reports a new STANDUP for yesterday (2025-12-03):

Progress: .

benchmarks show 2x slow down: Using validity bitmap as Arrow does #14409 (comment)
- speeding Arrow like storages
bugfixed all "step over": 7698b8a
review call nodes: Call nodes: App tree roots #14303 (comment)
meetings It should be finished by 2025-12-05.

JaroslavTulach · 2025-12-04T05:10:11Z

org_enso_benchmarks_generated_Column_Arithmetic_1000000_Multiply_Constant is 2x slower

with recent changes the benchmarks got faster

otherwise we are green
let's integrate and move towards Arrow

enso-bot · 2025-12-05T05:00:04Z

Jaroslav Tulach reports a new STANDUP for yesterday (2025-12-04):

Progress: .

validity bitmap merged Using validity bitmap as Arrow does #14409 (comment)
step over is merged Improving debugging "step over" experience #14312
review of call nodes: https://github.com/enso-org/design/pull/60#pullrequestreview-3538199583
merging with latest develop and bugfixing LongStorage over buffer: e5fd524
rawData& rawValidity: d45d553 It should be finished by 2025-12-05.

enso-bot · 2025-12-06T00:45:02Z

Jaroslav Tulach reports a new STANDUP for yesterday (2025-12-05):

Progress: .

Arrow language without co-location: https://github.com/enso-org/enso/pull/14309/files#r2591462347
no --add-opens: https://github.com/enso-org/enso/pull/14309/files#r2591506561
addressing review comments: https://github.com/enso-org/enso/pull/14309/files#r2592405993
hunting problem of 9th bit being zero: 3fcf693
simplify CurryNode? https://github.com/enso-org/design/pull/60#discussion_r2591820979
standups It should be finished by 2025-12-05.

GitHub
Arrow like storage over `ByteBuffer` to allow dual JVM buffer exchange by JaroslavTulach · Pull Request #14309 · enso-org/enso
Pull Request Description
#14357 has introduced a "mean benchmark"

that benchmark was slow in dual JVM mode
the goal of this pull request is to speed it up

the idea is to reuse work do...

Using validity bit map as arrow does

d6a67ec

JaroslavTulach requested a review from Akirathan December 1, 2025 15:06

JaroslavTulach self-assigned this Dec 1, 2025

JaroslavTulach requested review from AdRiley, GregoryTravis, hubertp and jdunkerley as code owners December 1, 2025 15:06

JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 1, 2025

JaroslavTulach added 7 commits December 1, 2025 16:12

Let's hope validity has been fixed

c88bf32

Need to set validity bit when appending valid value

ea48d1d

Missing ! in LongStorage.isNothing

08870e4

Better validityMap handling in BoolBuilder and BoolStorage

d6f3888

isNothingMap still useful in FillMissingOperation

84b51ea

LongStorageIterator also uses validityMap

65f340c

Iterate over clear bits of the validity bitmap

6f1da71

enso-bot bot mentioned this pull request Dec 2, 2025

Suspended application of default arguments playing badly with warnings and causing weird behaviour #10605

Closed

test/Table_Tests/src/In_Memory/Bool_Spec.enso is passing OK

ffc465a

JaroslavTulach added this to Issues Board Dec 2, 2025

JaroslavTulach added this to the 2026.1 Release milestone Dec 2, 2025

JaroslavTulach added the -compiler label Dec 2, 2025

JaroslavTulach moved this to 🔧 Implementation in Issues Board Dec 2, 2025

Success with --run test/Table_Tests should.support.is_blank

438864a

Akirathan approved these changes Dec 2, 2025

View reviewed changes

std-bits/table/src/main/java/org/enso/table/data/column/builder/DoubleBuilder.java Outdated Show resolved Hide resolved

...its/table/src/main/java/org/enso/table/data/column/storage/ColumnStorageWithValidityMap.java Outdated Show resolved Hide resolved

JaroslavTulach added 2 commits December 2, 2025 14:31

One ! fixes runEngineDistribution --run test/Table_Tests cast.to.Deci…

6011c98

…mal.from.mixed.column

Cardinality of a map represents number of valid entries

e23ae23

JaroslavTulach added 2 commits December 2, 2025 17:18

makeValidityMap in IsInOperation

215b453

Keeping the original logic and just flipping the bits fixes all tests…

bfa1c62

… in Table_Spec

Empty valid map with non-empty storage must mean there is null value

01ad757

JaroslavTulach and others added 3 commits December 2, 2025 17:54

Keep sealed and permit InferredDoubleBuilder

8a0e205

Co-authored-by: Pavel Marek <pavel.marek@enso.org>

Rewording comment

c2676e8

Co-authored-by: Pavel Marek <pavel.marek@enso.org>

Reformatted by javafmtAll

ab49b81

JaroslavTulach changed the title ~~Using validity bit map as arrow does~~ Using validity bitmap as arrow does Dec 2, 2025

Nothing is present when validity cardinality isn't equal to size

171cded

jdunkerley approved these changes Dec 3, 2025

View reviewed changes

github-project-automation bot moved this from 🔧 Implementation to 🌟 Q/A review in Issues Board Dec 3, 2025

hubertp reviewed Dec 3, 2025

View reviewed changes

JaroslavTulach mentioned this pull request Dec 3, 2025

Implement relational NULL semantics for Nothing for in-memory Column operations #8816

Merged

5 tasks

JaroslavTulach changed the title ~~Using validity bitmap as arrow does~~ Using validity bitmap as Arrow does Dec 3, 2025

JaroslavTulach added 4 commits December 3, 2025 17:31

Enforcing immutability of validity map by using ImmutableBitSet

ef972d5

Sharing allTrue bitmap for all sizes

0621d6e

More of ImmutableBitSet

4f78cd0

Avoid allocating validityMap in the builder when all values are valid

0c07d60

JaroslavTulach force-pushed the wip/jtulach/ValidityBitMap13851 branch from a9592b4 to 0c07d60 Compare December 3, 2025 18:30

JaroslavTulach added 2 commits December 3, 2025 19:51

Unit testing IsNothingOperation

6b036af

Fixing CountNothing.allNothing for storages with validity map

58fc873

JaroslavTulach merged commit e03ad84 into develop Dec 4, 2025
78 of 80 checks passed

JaroslavTulach deleted the wip/jtulach/ValidityBitMap13851 branch December 4, 2025 05:10

github-project-automation bot moved this from 🌟 Q/A review to 🟢 Accepted in Issues Board Dec 4, 2025

enso-bot bot mentioned this pull request Dec 9, 2025

Expanding the mean benchmark to DoubleStorage #14436

Open

3 tasks

Using validity bitmap as Arrow does #14409

Using validity bitmap as Arrow does #14409

Uh oh!

Conversation

JaroslavTulach commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Checklist

Uh oh!

Akirathan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JaroslavTulach commented Dec 2, 2025

Uh oh!

JaroslavTulach commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Dec 2, 2025

Uh oh!

enso-bot bot commented Dec 3, 2025

Uh oh!

JaroslavTulach commented Dec 3, 2025

Uh oh!

jdunkerley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubertp left a comment

Choose a reason for hiding this comment

Uh oh!

hubertp Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

enso-bot bot commented Dec 4, 2025

Uh oh!

JaroslavTulach commented Dec 4, 2025

Uh oh!

Uh oh!

enso-bot bot commented Dec 5, 2025

Uh oh!

enso-bot bot commented Dec 6, 2025 • edited by unfurl-links bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JaroslavTulach commented Dec 1, 2025 •

edited

Loading

JaroslavTulach commented Dec 2, 2025 •

edited

Loading

JaroslavTulach Dec 3, 2025 •

edited

Loading

JaroslavTulach commented Dec 3, 2025 •

edited

Loading

enso-bot bot commented Dec 6, 2025 •

edited by unfurl-links bot

Loading