GH-49548: [C++][FlightRPC] Decouple Flight Serialize/Deserialize from gRPC transport

[raulcd]

Rationale for this change

Currently the Serialize/Deserialize APIs are gRPC dependent. This means that any code that needs to encode or decode Flight data must depend on gRPC C++ internals. After some discussions around trying to build a PoC using gRPC's generic API with gRPC's BidiReactor we discussed that these primitives should be made gRPC agnostic.

What changes are included in this PR?

• Move the serialization/deserialization logic from cpp/src/arrow/flight/transport/grpc/serialization_internal.{h,cc} to cpp/src/arrow/flight/serialization_internal.cc
• Create new arrow::Result<arrow::BufferVector> SerializePayloadToBuffers(const FlightPayload& msg) gRPC agnostic function.
• Create new arrow::Result<arrow::flight::internal::FlightData> DeserializeFlightData(const std::shared_ptr<arrow::Buffer>& buffer) gRPC agnostic function.
• Keep the existing serialize/deserialize functions for gRPC as simple wrappers on top of the new serialization functions to implement the grpc::ByteBuffer and grpc::Slice details.
• Add utility arrow::Result<BufferVector> SerializeToBuffers() const; to FlightPayload struct.
• Add roundtrip tests to serialize/deserialize

Are these changes tested?

Yes, both by existing tests and new tests.

Are there any user-facing changes?

No

• GitHub Issue: [C++][FlightRPC] Decouple Flight Serialize/Deserialize from gRPC transport #49548

[github-actions]

⚠️ GitHub issue #49548 has been automatically assigned in GitHub to PR creator.

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: e852d66

Submitted crossbow builds: ursacomputing/crossbow @ actions-c88270abef

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[raulcd]

CI failures are unrelated.

I've run some comparison benchmarks between the old code path and the new code path, the full info can be seen here:
https://gist.github.com/raulcd/f139ccaaeb700a3ffe23a0c914e48c7c

Basically run the following to validate:
./arrow-flight-benchmark --num_streams=4 --records_per_stream=1000000

Summary:

DoGet benchmark, 4 streams, 1M records/stream (arrow-flight-benchmark)

Build	Branch	Run 1	Run 2	Run 3	Median
Debug	main	1354.94	1394.40	1263.38	1354.94
Debug	GH-49548	1431.46	1390.77	1433.68	1431.46
Release	main	2515.47	2408.77	2461.71	2461.71
Release	GH-49548	2447.93	2213.12	2546.33	2447.93

All values in MB/s.