HTTP body-hash deduplication gate, written in C, designed to sit behind NGINX.
Gatekeeper computes a SHA-256 hash of every incoming HTTP request body and uses atomic filesystem locks to ensure that only the first request with a given body is allowed through. Subsequent identical requests are rejected immediately. Locks can be explicitly released, and stale locks are automatically recovered.
It is built for high-concurrency production environments where duplicate request suppression must be reliable, fast, and operationally transparent.
Client ──POST──▶ NGINX ──proxy──▶ Gatekeeper (/gate)
│
sha256(body) streamed
│
┌─────────┴─────────┐
│ │
lock acquired? lock exists?
│ │
202 ALLOW 409 DROP
- The client sends a POST request through NGINX.
- NGINX proxies the request body to Gatekeeper's
/gateendpoint. - Gatekeeper incrementally computes
SHA-256(body)as bytes stream in (bounded memory, no buffering). - It attempts to acquire an atomic filesystem lock (
open(O_CREAT|O_EXCL)) keyed by the hex digest. - If the lock is acquired: 202 Accepted with
X-Gate-Decision: ALLOW. - If the lock already exists and is not stale: 409 Conflict with
X-Gate-Decision: DROP. - If the lock exists but its
mtimeexceeds the stale TTL (default 300s): the lock is removed and re-acquired (logged for visibility).
NGINX can then act on the response -- the recommended configuration drops duplicate connections with a 444 (closed with no response), but you can also pass the 409 through to the client.
| Method | Path | Description |
|---|---|---|
POST |
/gate |
Stream body, compute hash, acquire or reject lock |
POST |
/release |
Release a lock by SHA-256 hex (plain text or JSON {"sha256":"..."}) |
GET |
/healthz |
Health check -- verifies lock root is writable, returns version |
GET |
/metrics |
Prometheus-compatible counters (allow, drop, stale recovered, errors) |
| Header | Description |
|---|---|
X-Gate-Decision |
ALLOW, DROP, or ERROR |
X-Body-SHA256 |
The computed SHA-256 hex digest of the request body |
X-Gate-Error |
Present on internal errors (fail-open mode only) |
Locks are stored as files in a two-level sharded directory structure under the lock root (default /var/lib/gatekeeper/locks):
<lock_root>/<h0h1>/<h2h3>/<full_64_hex_hash>
Example:
/var/lib/gatekeeper/locks/2c/f2/2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
Each lock file contains metadata for debugging: unix timestamp, PID, and a random nonce.
- libmicrohttpd -- HTTP server library
- OpenSSL (EVP) -- SHA-256 hashing
On Debian/Ubuntu:
sudo apt-get install -y build-essential pkg-config libmicrohttpd-dev libssl-devmakeThe binary is produced at ./bin/gatekeeper.
| Flag | Default | Description |
|---|---|---|
GK_STALE_TTL_SECONDS |
300 |
Seconds before an unreleased lock is considered stale |
GK_FAIL_CLOSED |
0 |
Set to 1 to return 503 on internal errors instead of fail-open 202 |
Example:
make CFLAGS="-O2 -Wall -Wextra -Wpedantic -std=c11 -DGK_STALE_TTL_SECONDS=600 -DGK_FAIL_CLOSED=1"# Create lock directory
sudo install -d -m 0755 /var/lib/gatekeeper/locks
# Run
./bin/gatekeeper --listen 127.0.0.1 --port 8087 --lock-root /var/lib/gatekeeper/locks| Option | Default | Description |
|---|---|---|
--listen |
127.0.0.1 |
Bind address |
--port |
8087 |
Listen port |
--lock-root |
/var/lib/gatekeeper/locks |
Directory for lock files |
--conn-limit |
10000 |
Maximum concurrent connections |
--timeout-sec |
15 |
I/O timeout per connection |
# First request -- should return 202 ALLOW
curl -sS -D- -o /dev/null http://127.0.0.1:8087/gate -XPOST --data-binary 'hello'
# Same body again -- should return 409 DROP
curl -sS -D- -o /dev/null http://127.0.0.1:8087/gate -XPOST --data-binary 'hello'
# Release the lock (use the X-Body-SHA256 value from the first response)
curl -sS -D- http://127.0.0.1:8087/release -XPOST \
--data-binary '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'
# Same body again -- should return 202 ALLOW (lock was released)
curl -sS -D- -o /dev/null http://127.0.0.1:8087/gate -XPOST --data-binary 'hello'An example NGINX configuration is provided in nginx/gatekeeper.conf. The key setup:
- Define Gatekeeper as an upstream with keepalive connections.
- Proxy your ingestion endpoint to
/gatewithproxy_request_buffering offto stream the body. - Map the
409response to NGINX's444(silently drop the connection) or pass it through.
location = /ingest {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_request_buffering off;
proxy_pass http://gatekeeper_backend/gate;
# Drop duplicates silently
error_page 409 = @drop_duplicate;
}
location @drop_duplicate {
return 444;
}See nginx/gatekeeper.conf for the full snippet.
A unit file is provided in systemd/gatekeeper.service. It runs as a dedicated gatekeeper user with hardened sandboxing (NoNewPrivileges, ProtectSystem=strict, ProtectHome).
sudo cp systemd/gatekeeper.service /etc/systemd/system/
sudo useradd -r -s /usr/sbin/nologin gatekeeper
sudo install -d -o gatekeeper -g gatekeeper -m 0755 /var/lib/gatekeeper/locks
sudo systemctl daemon-reload
sudo systemctl enable --now gatekeeper| Test | Command | What it validates |
|---|---|---|
| SHA-256 vectors | ./tests/test_sha256_vectors.sh |
Hash correctness against OpenSSL reference |
| Concurrency | python3 ./tests/concurrency_test.py --url http://127.0.0.1:8087/gate --n 200 --c 50 --body hello |
Exactly 1 ALLOW and N-1 DROPs under concurrent load |
| NGINX smoke | See tests/nginx_smoke.md |
End-to-end NGINX proxying and lock release |
Run all automated tests:
make testA more comprehensive load test plan is documented in tests/load_test_plan.md.
By default, Gatekeeper operates in fail-open mode: if an internal error prevents lock acquisition (filesystem error, etc.), the request is allowed through with a 202 and an X-Gate-Error header. This ensures availability at the cost of potential duplicate processing.
To switch to fail-closed mode (return 503 on internal errors), compile with -DGK_FAIL_CLOSED=1. This prioritizes correctness over availability.
MIT