How do you write a single parquet file with a specified compression?

[theelderbeever]

As the title says? How do you just write a single parquet file with configuration? The two write_parquet methods that exist have completely different arguments and config options that hardly offer much to work with. Additionally, all the examples for

Datafusion 49.0.2

    let options = WriterProperties::builder()
        .set_compression(datafusion::parquet::basic::Compression::ZSTD(
            ZstdLevel::try_new(3)?,
        ))
        .build();

    let write_options = DataFrameWriteOptions::new().with_single_file_output(true);

    // This writes a single file but takes `TableParquetOptions` which  can't configure compression. The docs say this is tied to `ParquetWriterOptions` but there is no way to convert between the two.
    df.repartition(Partitioning::RoundRobinBatch(1))?
        .write_parquet("data/data.zstd.parquet", write_options, None) 
        .await?;

    // This accepts the `WriterProperties` but can't be configured to write a single file.
    ctx.write_parquet(
        df.repartition(Partitioning::RoundRobinBatch(1))?
            .create_physical_plan()
            .await?,
        "data/data.zstd.parquet",
        Some(options)
    )
    .await?;

[Jefffrey]

You can configure the options when using the DataFrame::write_parquet API like so:

    let mut options = TableParquetOptions::default();
    options.global.compression =
        Some(datafusion::parquet::basic::Compression::SNAPPY.to_string());

    parquet_df
        .write_parquet(
            "test_parquet1",
            DataFrameWriteOptions::default().with_single_file_output(true),
            Some(options),
        )
        .await?;

Which should achieve what you need. To note there is currently an issue with writing single files where a directory is actually being produced, see #13323

[theelderbeever]

Thank you. I did eventually figure that out. I think I was thrown by the options using mutability rather than a builder pattern. Is there any taste for more of a builder pattern around TableParquetOptions for ergonomics?

[Jefffrey]

That does sound reasonable, especially considering we seem to have it for the CSV version:

datafusion/datafusion/common/src/config.rs

Lines 2813 to 2822 in 0c6b654

	impl CsvOptions {
	/// Set a limit in terms of records to scan to infer the schema
	/// - default to `DEFAULT_SCHEMA_INFER_MAX_RECORD`
	pub fn with_compression(
	mut self,
	compression_type_variant: CompressionTypeVariant,
	) -> Self {
	self.compression = compression_type_variant;
	self
	}

Feel free to raise an issue if this is something you'd like to see 👍