Enable zstd compression format

[GilbertHan1011]

zstdandard(zstd) is a compression algorithm with higher performance than zlib. This contribution enabled fastp to handle zstd format

[bwlang]

@sfchen I tested this locally. It compiled and worked as expected for reading zstd compressed inputs - including with larger files (10k reads). I also tested interleaved fastq.zst files successfully

I think it could be merged, maybe without the .gitignore modification.

[bwlang]

I think this is a local preference

[GilbertHan1011]

Yes. This have been updated.

[KimBioInfoStudio]

we r working on this, BTW which downstream sw support consume fq.zst?

[GilbertHan1011]

@KimBioInfoStudio Precellar https://github.com/regulatory-genomics/precellar, and lots of snakemake pipeline https://github.com/regulatory-genomics/ATAC-sm

[KimBioInfoStudio]

seems, most upstream basecalling not support output r1.fq.zst even BGZF, welcome pr, could u rebase it to latest master branch, BTW, we r seeking a better format which support indexed multi members and read parallelism

[GilbertHan1011]

seems, most upstream basecalling not support output r1.fq.zst even BGZF, welcome pr, could u rebase it to latest master branch, BTW, we r seeking a better format which support indexed multi members and read parallelism

Done.
I’m also very interested in that kind of format. I'm planing work on it.

[KimBioInfoStudio]

Pls use lfs track those files?

[GilbertHan1011]

I tried Git LFS, but GitHub does not allow uploading new LFS objects to my public fork (rejected by server).
So I removed the .zst testdata files from the PR to avoid adding binary blobs.

[KimBioInfoStudio]

could u help with one group of bench as evidence

1. xxx.fq.zst win at compression ratio
2. no perf regression compare to .fq.gz -> .fq.gz
3. cover pe and se
4. ge 10M reads/pairs

[GilbertHan1011]

fastp_zstd_benchmark_report_maxcomp.pdf
@KimBioInfoStudio

[KimBioInfoStudio]

@sfchen plz kindly consider merge this pr