feat(nixcacheindex): implement RFC 0195 binary cache index protocol #552

kalbasit · 2026-01-15T18:33:34Z

Implementation of the Binary Cache Index Protocol as specified in RFC 0195.
This package provides a standalone, reusable library for interacting with
Nix binary cache indexes.

Key features implemented:

Base32 hash parsing and formatting with Big-Endian 160-bit interpretation.
Manifest serialization and deserialization (JSON).
Journal reading and writing (line-delimited +hash/-hash).
Golomb-Rice encoding and decoding with BitWriter/BitReader, supporting
arbitrary-precision integers (big.Int) for large deltas.
Shard File Format (Header, Sparse Index, Encoded Data) with XXH64 checksums.
Client Query Algorithm implementing Journal and Shard lookup logic.

The package is isolated and has no dependencies on other ncps internal packages,
enabling its future extraction into a standalone library.

Verified with comprehensive unit and end-to-end tests.

dosubot · 2026-01-15T18:33:50Z

Related Documentation

Checked 1 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}

kalbasit · 2026-01-15T18:33:53Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

gemini-code-assist · 2026-01-15T18:34:02Z

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a foundational nixcacheindex package, designed to fully support the Nix Binary Cache Index Protocol (RFC 0195). It provides all the necessary primitives and a client for interacting with Nix binary cache indexes, enabling efficient storage, retrieval, and querying of package hashes. The implementation focuses on correctness, performance, and reusability, laying the groundwork for robust binary cache management.

Highlights

RFC 0195 Implementation: A new nixcacheindex package has been introduced, providing a complete implementation of the Nix Binary Cache Index Protocol as specified in RFC 0195.
Core Components: The package includes robust implementations for Base32 hash parsing and formatting, Manifest serialization/deserialization, Journal reading/writing, and the Shard File Format (Header, Sparse Index, Encoded Data) with XXH64 checksums.
Efficient Encoding: Golomb-Rice encoding and decoding, including support for arbitrary-precision integers (math/big.Int), is provided for efficient storage and retrieval of hash deltas within shards.
Client Query Algorithm: A client query algorithm is implemented to efficiently check for the presence of a hash by first consulting the journal for recent mutations and then querying the relevant shards.
Isolation and Test Coverage: The new package is designed to be isolated with no dependencies on other internal packages, facilitating its future extraction. It is verified with comprehensive unit and end-to-end tests.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request provides a comprehensive implementation of the Nix binary cache index protocol as specified in RFC 0195. The code is well-structured into a new nixcacheindex package, covering base32 encoding, manifest handling, journal operations, Golomb-Rice encoding, and the shard file format. The client query logic correctly implements the two-layer lookup (journal then shards). The test coverage is good, including end-to-end tests for the client.

I've identified a few areas for improvement:

A couple of minor refactoring opportunities in the shard writing logic to improve code clarity and maintainability.
A misleading comment in the client implementation.
A significant gap in test coverage for the big.Int variant of the Golomb-Rice encoding, which is critical for correctness.

Additionally, the current client implementation reads entire shards into memory, which could be a performance bottleneck for very large shards. This is a reasonable trade-off for an initial implementation but should be considered for future optimization (e.g., using HTTP Range requests).

Overall, this is a high-quality contribution. Addressing the feedback will further improve its robustness and maintainability.

pkg/nixcacheindex/golomb_test.go

pkg/nixcacheindex/client.go

pkg/nixcacheindex/shard.go

kalbasit · 2026-01-16T09:27:15Z

/gemini review

kalbasit · 2026-01-16T10:02:45Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a comprehensive implementation of the Nix binary cache index protocol (RFC 0195) as a new, self-contained Go package nixcacheindex. The implementation is thorough, covering base32 hash handling, manifest and journal processing, Golomb-Rice encoding, the shard file format, and the client query algorithm. The code is well-structured and accompanied by extensive tests, which is excellent.

My review focuses on improving the robustness and correctness of the new library. I've identified a few areas for improvement, including a potential data loss issue in the Golomb encoder due to unflushed buffers, missing validation for encoding parameters that could lead to panics, and opportunities to improve resource management. The suggested changes aim to make the library safer and more reliable for future use.

pkg/nixcacheindex/golomb.go

pkg/nixcacheindex/shard.go

pkg/nixcacheindex/client.go

kalbasit · 2026-01-16T10:28:09Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a comprehensive implementation of the Nix binary cache index protocol (RFC 0195) in a new, standalone Go package. The implementation is well-structured, follows the RFC closely, and includes good test coverage for many of the new components like base32 encoding, manifest handling, and journaling.

My review has identified the following issues:

Critical: The client query logic for shards does not decompress fetched shard files, which will cause queries to fail against a real-world compressed cache index.
Medium: The tests for the Golomb-Rice encoding logic are missing coverage for the big.Int variants, which are crucial for handling large hash deltas.

Detailed comments and code suggestions have been provided to address these points. Once the decompression logic is fixed, this will be a solid implementation of the RFC.

pkg/nixcacheindex/client.go

pkg/nixcacheindex/golomb_test.go

kalbasit · 2026-01-16T10:43:29Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a comprehensive implementation of the RFC 0195 binary cache index protocol. The changes include robust handling of Base32 hash parsing and formatting, manifest serialization/deserialization, journal reading/writing, Golomb-Rice encoding/decoding with big.Int support, and the Shard File Format with XXH64 checksums. The client query algorithm effectively integrates journal and shard lookup logic, including handling missing shards and epoch retries. The code is well-structured, follows Go conventions, and is accompanied by thorough unit and end-to-end tests, demonstrating a high level of attention to detail and correctness. Overall, this is a solid and well-executed feature implementation.

pkg/nixcacheindex/client.go

kalbasit · 2026-01-17T01:59:17Z

Merge activity

Jan 17, 1:59 AM UTC: Graphite couldn't merge this PR because it had merge conflicts.
Jan 17, 2:00 AM UTC: Graphite couldn't merge this PR because it had merge conflicts.
Jan 17, 2:01 AM UTC: Graphite couldn't merge this PR because it had merge conflicts.

Mic92 · 2026-01-19T10:55:58Z

pkg/nixcacheindex/client.go

+	// For now, read all.
+	var reader io.Reader = rc
+	if strings.HasSuffix(shardPath, ".zst") {
+		zstdReader, err := zstd.NewReader(rc)


nice to have: I did some benchmarking in the past and noticed that creating a new instance each time, creates significant allocations. In niks3 I am using a pool allocator for zstd for this reasons, which saved me a few gigabytes of allocations for uploads.

Yep, I use pools in ncps as well. This pr is mostly antigravity coding the rfc. I haven't reviewed/changed it yet.

Mic92 · 2026-01-19T10:57:41Z

pkg/nixcacheindex/client.go

+
+	data, err := io.ReadAll(reader)
+	if err != nil {
+		return DefiniteMiss, err


not super familiar with the error handling in this project, but I would wrap the error here to add more context, when we have a connection/filesystem error.

There are probably some other places where I would add more error context for production troubleshooting.

Mic92 · 2026-01-19T11:00:04Z

pkg/nixcacheindex/client_test.go

+
+	var compressedShardBuf bytes.Buffer
+
+	enc, err := zstd.NewWriter(&compressedShardBuf)


Same here, I think a pool allocator would make sense.

Mic92 · 2026-01-19T11:04:13Z

Will do a more in-depth review later. Just quickly skimmed over the code.

kalbasit · 2026-01-20T00:13:49Z

Will do a more in-depth review later. Just quickly skimmed over the code.

Thank you! I will ping you to review once the RFC is approved (or at least settled on a design) and I update these prs with the implementation.

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request go Pull requests that update go code labels Jan 15, 2026

This was referenced Jan 15, 2026

feat: implement delta logic for binary cache index protocol #553

Draft

feat: wire up new index feature into pkg/cache/upstream #554

Draft

feat: Implement Nix Binary Cache Index Protocol #555

Draft

gemini-code-assist bot reviewed Jan 15, 2026

View reviewed changes

pkg/nixcacheindex/golomb_test.go Show resolved Hide resolved

pkg/nixcacheindex/client.go Show resolved Hide resolved

pkg/nixcacheindex/shard.go Show resolved Hide resolved

pkg/nixcacheindex/shard.go Outdated Show resolved Hide resolved

kalbasit force-pushed the nixcacheindex branch from 0a96c77 to 577c5cf Compare January 16, 2026 08:21

kalbasit force-pushed the nixcacheindex branch from ad2f920 to 30a94e8 Compare January 16, 2026 10:02

gemini-code-assist bot reviewed Jan 16, 2026

View reviewed changes

pkg/nixcacheindex/golomb.go Outdated Show resolved Hide resolved

pkg/nixcacheindex/shard.go Show resolved Hide resolved

pkg/nixcacheindex/client.go Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Jan 16, 2026

View reviewed changes

pkg/nixcacheindex/client.go Outdated Show resolved Hide resolved

pkg/nixcacheindex/golomb_test.go Show resolved Hide resolved

kalbasit force-pushed the nixcacheindex branch 2 times, most recently from 9a7194d to 11f9c88 Compare January 16, 2026 10:42

gemini-code-assist bot reviewed Jan 16, 2026

View reviewed changes

pkg/nixcacheindex/client.go Outdated Show resolved Hide resolved

pkg/nixcacheindex/client.go Show resolved Hide resolved

kalbasit mentioned this pull request Jan 16, 2026

[RFC 0195] init: Binary Cache Index Protocol NixOS/rfcs#195

Draft

kalbasit changed the base branch from main to graphite-base/552 January 17, 2026 01:34

kalbasit force-pushed the nixcacheindex branch from 8a47362 to 9011df0 Compare January 17, 2026 01:34

kalbasit changed the base branch from graphite-base/552 to 01-16-pkg_golomb January 17, 2026 01:34

kalbasit mentioned this pull request Jan 17, 2026

feat: introduce golomb-rice encoding package #564

Merged

kalbasit changed the base branch from 01-16-pkg_golomb to graphite-base/552 January 17, 2026 01:44

kalbasit force-pushed the graphite-base/552 branch from 5fadb1a to 0ea41ff Compare January 17, 2026 01:45

kalbasit force-pushed the nixcacheindex branch from 9011df0 to fa9617c Compare January 17, 2026 01:45

kalbasit changed the base branch from graphite-base/552 to main January 17, 2026 01:45

kalbasit force-pushed the nixcacheindex branch from fa9617c to c9f97b6 Compare January 17, 2026 01:45

kalbasit force-pushed the nixcacheindex branch 2 times, most recently from c5a7889 to 9622a58 Compare January 17, 2026 21:06

kalbasit and others added 2 commits January 17, 2026 16:12

feat(nixcacheindex): implement RFC 0195 binary cache index protocol

da35c96

Auto Update Vendor Hash

c3b9d41

kalbasit force-pushed the nixcacheindex branch from 9622a58 to c3b9d41 Compare January 19, 2026 04:50

kalbasit marked this pull request as draft January 19, 2026 07:58

Mic92 reviewed Jan 19, 2026

View reviewed changes


		var compressedShardBuf bytes.Buffer

		enc, err := zstd.NewWriter(&compressedShardBuf)

Uh oh!

feat(nixcacheindex): implement RFC 0195 binary cache index protocol #552

Are you sure you want to change the base?

feat(nixcacheindex): implement RFC 0195 binary cache index protocol #552

Uh oh!

Conversation

kalbasit commented Jan 15, 2026

Uh oh!

dosubot bot commented Jan 15, 2026

Uh oh!

kalbasit commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Jan 15, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kalbasit commented Jan 16, 2026

Uh oh!

kalbasit commented Jan 16, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kalbasit commented Jan 16, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

kalbasit commented Jan 16, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

kalbasit commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Mic92 Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

kalbasit Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Mic92 Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mic92 Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Mic92 Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Mic92 commented Jan 19, 2026

Uh oh!

kalbasit commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

kalbasit commented Jan 15, 2026 •

edited

Loading

kalbasit commented Jan 17, 2026 •

edited

Loading

Mic92 Jan 19, 2026 •

edited

Loading