Using arrow and parquet_derive

[ross-paypay]

Sorry if this a basic question but I can't seem to find an answer for this.

I have a Vec of custom structs that I am using to construct a RecordBatch by manually building each of the columns.

Below is a simplified example

struct Datum { foo: String }

let data: Vec<Datum> = vec![....]

let foo = Arc::new(StringArray::from(data.iter().map(|v| v.foo).collect::<Vec<_>>())) as ArrayRef

RecordBatch::try_from_iter_with_nullable(vec![
        ("foo", foo, true),
]);

This is working, but my data has 50+ fields so the mapping code is getting a bit rough to maintain.

Looking at parquet_derive, it seems like it could solve this by allowing me to derive the the implementation.
However, the parquet ParquetRecordWriter seems quiet different than the arrow RecordBatch API.

Is there a way to have these interact in a nice way? (or perhaps another solution in arrow I am over looking)

[mbrobbel]

Maybe you can use narrow? Note that it's an experimental project (there will be breaking changes) and not all types are supported. There is a parquet write example here that includes the conversion to an Arrow RecordBatch.

Other options: arrow_convert and serde_arrow.

[ross-paypay]

Thanks! I was not aware of these other crates.
Looks like they should fit my needs