Fix primitive type parsing in complex types for Arrow-serialized formats#1250
Open
vikrantpuppala wants to merge 1 commit intodatabricks:mainfrom
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Improves parsing of Arrow-serialized primitive values when they appear inside complex types (ARRAY/MAP/STRUCT), aligning driver behavior with Databricks/Spark’s alternate nested encodings.
Changes:
- Add fallback parsing for nested DATE (epoch days), TIMESTAMP/TIMESTAMP_NTZ (epoch micros) and BINARY (base64) in
ComplexDataTypeParser. - Handle
TIMESTAMP_NTZexplicitly inDatabricksStructandDatabricksArraysimple-value conversion. - Add unit tests for the new nested parsing behaviors and update
NEXT_CHANGELOG.md.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/main/java/com/databricks/jdbc/api/impl/ComplexDataTypeParser.java |
Adds fallback conversions for Arrow nested encodings (epoch days/micros, TIMESTAMP_NTZ component arrays, base64 BINARY). |
src/main/java/com/databricks/jdbc/api/impl/DatabricksStruct.java |
Adds TIMESTAMP_NTZ handling (and avoids re-parsing when already a Timestamp). |
src/main/java/com/databricks/jdbc/api/impl/DatabricksArray.java |
Adds TIMESTAMP_NTZ handling (and avoids re-parsing when already a Timestamp). |
src/test/java/com/databricks/jdbc/api/impl/ComplexDataTypeParserTest.java |
Adds tests covering epoch-micros timestamps, TIMESTAMP_NTZ representations, and base64 BINARY within containers. |
NEXT_CHANGELOG.md |
Documents the user-visible parsing fix for complex types with Arrow alternate formats. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
src/test/java/com/databricks/jdbc/api/impl/ComplexDataTypeParserTest.java
Outdated
Show resolved
Hide resolved
src/main/java/com/databricks/jdbc/api/impl/ComplexDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/main/java/com/databricks/jdbc/api/impl/ComplexDataTypeParser.java
Outdated
Show resolved
Hide resolved
vikrantpuppala
force-pushed
the
fix/complex-type-timestamp-time-epoch-values
branch
2 times, most recently
from
7cd5ce5 to
92b4488
Compare
6 tasks
gopalldb
reviewed
Mar 6, 2026
gopalldb
reviewed
Mar 6, 2026
Arrow serialization uses alternate formats for primitive types within complex types (ARRAY, MAP, STRUCT): DATE as epoch day integers, TIMESTAMP and TIMESTAMP_NTZ as epoch microseconds or component arrays, and BINARY as base64-encoded strings. The parser now correctly handles all these formats with appropriate fallback logic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
vikrantpuppala
force-pushed
the
fix/complex-type-timestamp-time-epoch-values
branch
from
82780d2 to
2164a4b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
java.sql.Timestamp. Also handles TIMESTAMP_NTZ serialized as[year,month,day,hour,min,sec]component arrays.convertPrimitive().TIMESTAMP_NTZcase toDatabricksStruct.convertSimpleValue()andDatabricksArray.convertValue()switch statements that were missing it.Related: #1247, #1248
Test plan
ComplexDataTypeParserTestpass🤖 Generated with Claude Code