GH-3414: Add parseJson to VariantBuilder for JSON-to-Variant conversion by gaurav7261 · Pull Request #3415 · apache/parquet-java

gaurav7261 · 2026-03-07T06:50:27Z

Rationale for this change

Every consumer of parquet-variant currently has to independently implement JSON-to-Variant parsing. Apache Spark has one in its common/variant module
(source), our Kafka Connect S3 sink connector had to write one, and any other project (Flink, Trino, DuckDB-Java, etc.) would need to do the
same. Since VariantBuilder already provides all the low-level append*() primitives, parseJson() is the natural completion of that API — a canonical, reusable entry point for the most common use
case: converting a JSON string into a Variant.

What changes are included in this PR?

parquet-variant/pom.xml: Added jackson-core (compile) and parquet-jackson (runtime) dependencies, following the same pattern as parquet-hadoop.
VariantBuilder.java: Added two public static methods:
- parseJson(String json) — convenience method that creates a Jackson streaming parser internally.
- parseJson(JsonParser parser) — for callers who already have a positioned parser (e.g., reading from a stream).
- Internal helpers ported from Apache Spark's production VariantBuilder.buildJson:
  - buildJson() — recursive single-pass streaming parser handling OBJECT, ARRAY, STRING, NUMBER_INT, NUMBER_FLOAT, TRUE, FALSE, NULL.
  - appendSmallestLong() — selects the smallest integer type (BYTE/SHORT/INT/LONG) based on value range.
  - tryAppendDecimal() — decimal-first encoding for floating-point numbers; falls back to double only for scientific notation or values exceeding DECIMAL16 precision (38 digits).
TestVariantParseJson.java: 32 new tests covering all primitive types, objects (empty, simple, nested, null values, sorted keys, duplicate keys), arrays (empty, simple, nested, mixed types), and
edge cases (unicode, escaped strings, deeply nested documents, scientific notation, integer overflow to decimal, malformed JSON).

Are these changes tested?

Yes. 32 new tests in TestVariantParseJson.

Are there any user-facing changes?

Yes. Two new public static methods on VariantBuilder:

VariantBuilder.parseJson(String json) — returns a Variant
VariantBuilder.parseJson(JsonParser parser) — returns a Variant
These are additive API additions with no breaking changes to existing APIs.
Closes parseJson implementation in VariantBuilder #3414

gaurav7261 · 2026-03-07T06:54:46Z

@alamb @aihuaxu read https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ and check the feasibility of having our S3 Sink connector write variant, found out that parseJson can be a better fit here, wdyt? is it making sense

…uctured fields Write Kafka Connect JSON fields (Debezium CDC, Confluent Protobuf Struct, custom messages, maps, arrays) as native Parquet VARIANT columns instead of plain STRING, improving storage efficiency and query performance. - Upgrade parquet-java to 1.17.0 for VARIANT logical type support - https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ - Add config: parquet.variant.enabled, parquet.variant.connect.names, parquet.variant.field.names - Auto-detect recursive schemas (google.protobuf.Struct) as VARIANT - Stream JSON-to-Variant conversion ported from Apache Spark (PR also raised in parquet-java, once approved, code will be neat here in this repo: apache/parquet-java#3415) - Unwrap Protobuf Struct/Value/ListValue/map-as-array to clean JSON - Graceful fallback for non-JSON values (e.g. __debezium_unavailable_value) - Feature is fully opt-in (disabled by default), zero impact on existing connectors

gaurav7261 · 2026-03-09T16:36:41Z

@Fokko can you please review, is it looking good to you?

alamb · 2026-03-11T17:29:26Z

FWIW we have a similar method in Rust in case that is interesting

https://github.com/apache/arrow-rs/blob/d3c79006f2595e144d539f56b3054fe916ab184b/parquet-variant-compute/src/from_json.rs#L47

…onversion

gaurav7261 · 2026-03-11T17:50:39Z

@julienledem thanks for the call, I have added notice, please review

gaurav7261 · 2026-03-16T16:14:38Z

@gszadovszky can you please review as well

wgtmac · 2026-03-17T09:24:08Z

I'm not familiar with this code yet but I think it is worth adding.

@emkornfield @gene-db @rdblue WDYT?

steveloughran

I think the json parsing should be in a class alongside VariantBuilder, rather than inside it, some VariantJsonParser class.

ensures that use of jackson classes are isolated; the core variant does not need jackson on the CP
if there is anything which can be done to improve single line JSON content performance, then it can be done there as it's life will span the whole file, rather than a row

Once it's split out, a new question surfaces: should it belong in parquet-jackson? Personally, I think it should

steveloughran · 2026-03-19T10:04:51Z

NOTICE

+This project includes code from Apache Spark with the following copyright
+notice:
+
+  Apache Spark


do cross-ASF projects need this credit? What I do think is good is ensure the original authors get credit in the final commit message

steveloughran · 2026-03-19T10:06:01Z

parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantParseJson.java

+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestVariantParseJson {


nested object parsing?

what about invalid json?

empty file

not a json file

incomplete

large json with many values

steveloughran · 2026-03-19T10:07:25Z

parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java

+   * @throws IOException if the JSON is malformed or an I/O error occurs
+   */
+  public static Variant parseJson(String json) throws IOException {
+    try (JsonParser parser = JSON_FACTORY.createParser(json)) {


can be quite slow

steveloughran · 2026-03-19T10:12:16Z

parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java

+   * @return the parsed Variant
+   * @throws IOException if the JSON is malformed or an I/O error occurs
+   */
+  public static Variant parseJson(String json) throws IOException {


I was to suggest making CharSequence for feeding in from other places (string fields within avro, ...) but it looks like jackson 2 doesn't support that itself.

steveloughran · 2026-03-19T10:21:29Z

parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java

 */
 public class VariantBuilder {

+  private static final JsonFactory JSON_FACTORY = new JsonFactory();


factory should be built with some constraints so that giving it a malicious json file should be rejected rather than trigger OOM problems, etc. From the javadocs.

JsonFactory f = JsonFactory.builder() .streamReadConstraints( StreamReadConstraints.builder() .maxNestingDepth(500) .maxStringLength(10_000_000) .maxDocumentLength(5_000_000) .build() ) .build();

gaurav7261 mentioned this pull request Mar 8, 2026

feat: add native Parquet VARIANT column support for JSON and semi structured fields confluentinc/kafka-connect-storage-cloud#917

Open

6 tasks

[apacheGH-3414] Add parseJson to VariantBuilder for JSON-to-Variant c…

643ac64

…onversion

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 1274733 to 643ac64 Compare March 11, 2026 17:49

steveloughran reviewed Mar 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-3414: Add parseJson to VariantBuilder for JSON-to-Variant conversion#3415

GH-3414: Add parseJson to VariantBuilder for JSON-to-Variant conversion#3415
gaurav7261 wants to merge 1 commit intoapache:masterfrom
gaurav7261:gh-3414-variant-parse-json

gaurav7261 commented Mar 7, 2026

Uh oh!

gaurav7261 commented Mar 7, 2026

Uh oh!

gaurav7261 commented Mar 9, 2026

Uh oh!

alamb commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 16, 2026

Uh oh!

wgtmac commented Mar 17, 2026

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Mar 19, 2026

Uh oh!

steveloughran Mar 19, 2026

Uh oh!

steveloughran Mar 19, 2026

Uh oh!

steveloughran Mar 19, 2026

Uh oh!

steveloughran Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

gaurav7261 commented Mar 7, 2026

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

gaurav7261 commented Mar 7, 2026

Uh oh!

gaurav7261 commented Mar 9, 2026

Uh oh!

alamb commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 16, 2026

Uh oh!

wgtmac commented Mar 17, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants