Skip to content

feat(python/cpp): add streaming serialization support to python and c++#3449

Merged
chaokunyang merged 18 commits intoapache:mainfrom
chaokunyang:implement_python_stream_serialization
Mar 4, 2026
Merged

feat(python/cpp): add streaming serialization support to python and c++#3449
chaokunyang merged 18 commits intoapache:mainfrom
chaokunyang:implement_python_stream_serialization

Conversation

@chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Mar 3, 2026

Why?

Add output-stream serialization support so large payloads can be written incrementally instead of materializing full byte arrays first, and align Python/C++ stream APIs for end-to-end stream roundtrips.

What does this PR do?

  • Add C++ output-stream serialization APIs: Fory::serialize(OutputStream&, const T&), Fory::serialize(std::ostream&, const T&), and corresponding ThreadSafeFory overloads.
  • Unify stream input types in C++ (StreamReader/ForyInputStream to InputStream/StdInputStream) and wire stream-writer state/reset handling through WriteContext and Buffer.
  • Introduce flush barrier and incremental flush integration for chunked serializers (notably map and struct paths) to keep chunk headers valid while streaming.
  • Add Python stream-write bridge (PyOutputStream), Buffer.wrap_stream + stream writer binding, and Fory.dump(obj, stream) for direct serialization to Python writable streams.
  • Expand tests in C++ and Python for stream output roundtrip/parity, partial-write flush behavior, and map chunk streaming paths.

Related issues

Closes #3445
Closes #3444
Closes #3452

Does this PR introduce any user-facing change?

Yes. This adds new public stream serialization APIs in C++ and Python (dump to writable streams).

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

@chaokunyang chaokunyang requested review from pandalee99 and urlyy March 3, 2026 05:42
@chaokunyang chaokunyang requested a review from PragmaTwice as a code owner March 3, 2026 05:42
@chaokunyang chaokunyang merged commit 9c4510a into apache:main Mar 4, 2026
64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Support Stream Serialization [Python] Support Streaming Serialization [Python] Add support for dumping directly to file

2 participants