feat(vector sink): Add zstd compression support by jpds · Pull Request #24917 · vectordotdev/vector

jpds · 2026-03-13T13:25:01Z

Summary

Add support for zstd compression to the Vector sink, alongside the existing gzip compression. Changes include:

Update compression field from bool to Compression enum with backward compatibility via bool_or_compression deserializer
Support compression algorithms: "none", "gzip", "zstd"
Add validation that returns clear errors for unsupported compression types (zlib, snappy)
Apply appropriate tonic::codec::CompressionEncoding based on selected algorithm

The default behavior (no compression) is unchanged, ensuring full backward compatibility with existing configurations.

Vector configuration

How did you test this PR?

cargo test + wedging into https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/vector/caddy-clickhouse.nix

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes: #23030

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

pront

LLM review findings:

compression now deserializes and documents the full shared Compression enum, but the vector sink only supports none, gzip, and zstd. In this PR, generated docs still expose snappy and zlib in website/cue/reference/components/sinks/generated/vector.cue (line 73), while src/sinks/vector/service.rs (line 94) rejects them at runtime. That means users can write a schema-valid config that fails only when the sink builds.
The new config path also accepts object syntax with level via src/serde.rs (line 238), but src/sinks/vector/service.rs (line 82) ignores the level entirely because tonic only takes a CompressionEncoding. So settings like { algorithm = "zstd", level = 3 } are accepted but ineffective.

My suggestion:

For this sink, a dedicated enum is the cleaner design because the actual capability surface is smaller than the shared Compression type:

The vector sink only supports none, gzip, and zstd.
It does not support zlib or snappy.
It does not support per-codec level

jpds · 2026-03-13T20:09:17Z

@pront I also asked Claude about the extra compression options and it pointed out that greptimedb also has this issue: https://vector.dev/docs/reference/configuration/sinks/greptimedb_logs/#compression

I'll look into why the level isn't applied.

pront · 2026-03-18T18:40:43Z

@codex review

chatgpt-codex-connector · 2026-03-18T18:44:48Z

Codex Review: Didn't find any major issues. 🚀

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pront

Thanks!

pront · 2026-03-18T18:46:30Z

One last concern, the test plan states cargo test. Did you test this with real Vector instances?

Enable the zstd compression feature in the tonic dependency to support zstd compression for gRPC communication in the Vector sink and source. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Add support for zstd compression to the Vector sink, alongside the existing gzip compression. Changes include: - Update compression field from bool to Compression enum with backward compatibility via bool_or_compression deserializer - Support compression algorithms: "none", "gzip", "zstd" - Add validation that returns clear errors for unsupported compression types (zlib, snappy) - Apply appropriate tonic::codec::CompressionEncoding based on selected algorithm The default behavior (no compression) is unchanged, ensuring full backward compatibility with existing configurations. Resolves vectordotdev#23030 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Update the Vector source to accept zstd-compressed data in addition to gzip. This enables the source to receive and decompress data from Vector sinks using either compression algorithm, ensuring seamless interoperability. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Add changelog fragment documenting the new zstd compression support in the Vector sink, including usage examples for all configuration formats (legacy boolean, string, and advanced object syntax). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

jpds · 2026-03-19T01:16:47Z

@pront All done.

changelog.d/23030_vector_sink_zstd_compression.enhancement.md

pront · 2026-03-19T13:20:46Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3c0de3044

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

src/sources/util/grpc/decompression.rs

The custom DecompressionAndMetrics tower layer only supported gzip, rejecting zstd requests before tonic could handle them. Add zstd support to the decompression layer and a test for the full path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The boolean compression syntax (`compression: true/false`) in the vector sink is deprecated in favour of the string syntax (`compression: "gzip"`, `compression: "none"`, etc.), to be removed in v0.57.0. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pront · 2026-03-20T19:58:57Z

One last concern, the test plan states cargo test. Did you test this with real Vector instances?

Code looks good. Ping on this one @jpds.

jpds · 2026-03-21T09:49:52Z

@pront Oh, sorry - I'd updated the description along with my "all done" reply earlier. Yes - I had.

jpds requested a review from a team as a code owner March 13, 2026 13:25

github-actions bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks labels Mar 13, 2026

jpds changed the title ~~Feat vector sink zstd compression~~ feat(vector sink): Add zstd compression support Mar 13, 2026

jpds force-pushed the feat-vector-sink-zstd-compression branch from bf87cf3 to 4131367 Compare March 13, 2026 16:19

jpds requested a review from a team as a code owner March 13, 2026 16:19

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Mar 13, 2026

jpds force-pushed the feat-vector-sink-zstd-compression branch from 4131367 to 7c0577e Compare March 13, 2026 16:33

hestonhoffman previously approved these changes Mar 13, 2026

View reviewed changes

pront requested changes Mar 13, 2026

View reviewed changes

jpds force-pushed the feat-vector-sink-zstd-compression branch from 7c0577e to eacbc3d Compare March 15, 2026 18:11

jpds dismissed hestonhoffman’s stale review via eacbc3d March 16, 2026 14:24

pront approved these changes Mar 18, 2026

View reviewed changes

jpds and others added 4 commits March 18, 2026 23:47

chore(deps): enable tonic zstd feature

76ba138

Enable the zstd compression feature in the tonic dependency to support zstd compression for gRPC communication in the Vector sink and source. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

jpds force-pushed the feat-vector-sink-zstd-compression branch 2 times, most recently from 963067a to f3c0de3 Compare March 19, 2026 00:44

pront reviewed Mar 19, 2026

View reviewed changes

changelog.d/23030_vector_sink_zstd_compression.enhancement.md Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Mar 19, 2026

View reviewed changes

src/sources/util/grpc/decompression.rs Outdated Show resolved Hide resolved

jpds and others added 2 commits March 19, 2026 14:24

jpds force-pushed the feat-vector-sink-zstd-compression branch from f3c0de3 to dc27966 Compare March 19, 2026 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(vector sink): Add zstd compression support#24917

feat(vector sink): Add zstd compression support#24917
jpds wants to merge 6 commits intovectordotdev:masterfrom
jpds:feat-vector-sink-zstd-compression

jpds commented Mar 13, 2026 •

edited

Loading

Uh oh!

pront left a comment

Uh oh!

jpds commented Mar 13, 2026 •

edited

Loading

Uh oh!

pront commented Mar 18, 2026

Uh oh!

chatgpt-codex-connector bot commented Mar 18, 2026

Uh oh!

pront left a comment

Uh oh!

pront commented Mar 18, 2026

Uh oh!

jpds commented Mar 19, 2026

Uh oh!

Uh oh!

pront commented Mar 19, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

pront commented Mar 20, 2026

Uh oh!

jpds commented Mar 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jpds commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

jpds commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pront commented Mar 18, 2026

Uh oh!

chatgpt-codex-connector bot commented Mar 18, 2026

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

pront commented Mar 18, 2026

Uh oh!

jpds commented Mar 19, 2026

Uh oh!

Uh oh!

pront commented Mar 19, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pront commented Mar 20, 2026

Uh oh!

jpds commented Mar 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jpds commented Mar 13, 2026 •

edited

Loading

jpds commented Mar 13, 2026 •

edited

Loading