Core: Add JMH benchmarks for Variants

[steveloughran]

Feature Request / Improvement

There's no JMH benchmarks for variants, so no way to see if behaviour scales well.

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[steveloughran]

Report from the initial PR, which doesn't (yet) fork. It can be viewed in jmh viewer.

• Nested variants have 1/8 of their toplevel fields nested, each of which has 5 children with existing (string) keys and new strings, so there's more data to write
• There's a lot of sample variance; running a few thousand iterations on a physical machine otherwise completely idle would be better here.

You can see the cost of shredding in the build() benchmark.

This is what claude says of the numbers

  build (construct ShreddedObject + sizeInBytes):                                                                                                                                           
  - Shallow and Nested are very close at all 10k params — within noise. At 100%: 2252 vs 2190us. Construction cost is dominated by the number of put() calls and sizeInBytes() computation,
  not value complexity.                                                                                                                                                                     
  - Nested 10k scales cleanly: 353 → 1107 → 1707 → 2190us. Nearly linear with shredding %.                                                                                                
  - At 0% shredded, Nested (353us) is cheaper than Shallow (567us). No put() calls in either case, so the difference is sizeInBytes() on the unshreddedObject. Nested objects may compute   
  size differently than flat primitives.
  - 1k results are still noisy at low shredding (9-48% error) but stable at high shredding (2-9%).

  serialize (pre-built object, writeTo only):
  - Tightest results of all three benchmarks — most entries <2% error.
  - Nested overhead is consistent: ~1.8x at 10k/100% (7045 vs 3896us). At 10k/0% they're nearly identical (2356 vs 2299us) since no nested values are in the shredded portion.
  - Linear scaling with shredding % in both modes. Nested 10k: 2356 → 4392 → 5854 → 7045us. Shallow 10k: 2299 → 3161 → 3595 → 3896us. The slope is steeper for Nested — each additional
  shredded field costs more bytes to write.

  deserialize (parse buffer + iterate all fields):
  - Flat across shredding % — confirms deserialization cost depends on total data volume, not how it was shredded.
  - Nested 10k: ~4500us. Shallow 10k: ~3760us. ~1.2x overhead for Nested. Less than the 1.8x serialization overhead because deserialization just scans offsets without recursing into nested
   values.
  - Nested 1k: remarkably stable at ~390us across all shredding values (1-2% error). This is the cleanest data in the entire run.
  - Shallow 1k/0%: 421us with 16% error — the first param combination still absorbs some residual warmup even with 100 warmup iterations.

  Summary: Serialization cost scales with both shredding % and nesting depth. Deserialization is flat across shredding and only modestly affected by nesting (~20% overhead). Build cost is
  dominated by field count and shredding %, with nesting having negligible impact.

results.json

[steveloughran]

For anyone looking at this, there's no significant cost for variants without shedding when creating/loading them. It takes write when shedding, but these benchmarks don't measure cost of queries over data, where the shedded variants should show significant speedup compared to just reading all the data.

[RussellSpitzer]

This sounds expected to me. An unshredded vs shredded variant should be pretty similar size. Probably similar enough. The Variance you see may be dependent on JIT optimizations. I would expect nested writers taking more and more nested function calls which may take more runs before the JIT cleans them up.

[steveloughran]

@RussellSpitzer Nesting is creating more hashmaps, which allocate map space.

I've been doing the parquet version of this test suite too, using this file as the foundation.

apache/parquet-java#3452

There I've noticed that the granularity of ARM CPUs is nowhere as good as x86 rdtsc opcodes (ignoring core/socket variations there), so the very small operations need to be repeated multiple times within a single benchmark method. I'll apply that here too. Essentially: microbenchmarks need to take milliseconds to minimise clock granularity. jmh does at least pin the benchmark to a core so cross-socket and cross-core issues (older intel Si) don't surface. And by only working on the same struct the data should all be in $L1 so data cache issues shouldn't be an issue, hence nor is NUMA. Modern multisocket servers are the new datacentre.

what I'm doing now are:

parquet-benchmark module row write with/without shedding.

Assesses cost of write complexity and makes sure there is now problem lurking right at the bottom.

iceberg spark module: table-level benchmark

I'm creating two tables with same content then doing queries over them. I need to produce an uneven spread of numbers so that iceberg can use the shedded metadata to filter files; without that filtering shedded is coming of slower right now, which is presumably happening underneath. Spark parquet vectorization currently doesn't work with projection on shedded variants, so I'm turning vectorization off on both tables for consistency.

[steveloughran]

@RussellSpitzer and here's the results of a the spark reader benchmark I've added to the PR.
The longer lines are all the shedded ones, even when filtering or selecting on a shedded column.

human-readable-output.txt

I can think of some more tests there

• cost of adding rows
• extending the persisted structure (I'd parameterize for a simple vs complex variant)
• and optionally partition by the category:int field, both direct and via the variant.

Suggestions welcome

[steveloughran]

Full test results.json
View in https://jmh.morethan.io/