Crush

Crush is a high-performance parallel compression toolkit written in Rust. It combines multi-threaded CPU compression with GPU-accelerated decompression through a pluggable architecture, delivering throughput that scales from single-core workloads to discrete GPUs.

Crush is designed for high-throughput data pipelines, particularly in AI and ML workflows where large datasets are frequently compressed, transferred over the network, and decompressed. By pairing near-linear CPU scaling with GDeflate-based GPU decompression, Crush minimizes end-to-end data ingestion time at every stage of the pipeline.

Why Crush?

Modern data pipelines move massive amounts of data — training sets, model checkpoints, log archives — often across networks and under tight latency budgets. Traditional compression tools force a choice between speed and flexibility. Crush eliminates that tradeoff.

Crush is built to:

Scale with hardware — parallel CPU compression across all cores, GPU-accelerated decompression via Vulkan/Metal/DX12
Adapt to data — plugin architecture enables format-aware compression strategies; automatic entropy analysis routes data to the best algorithm
Stay out of the way — automatic GPU detection with CPU fallback, cooperative cancellation, and pipeline-friendly stdin/stdout
Be safe and predictable — memory-safe Rust, CRC32 integrity checks, RAII cleanup of partial files, no .unwrap() in production code

Features

Core Capabilities

Parallel CPU Compression: Multi-threaded compression scaling near-linearly to all available cores (983 MiB/s on 8+ cores)
GPU-Accelerated Decompression: GDeflate-based GPU decompression via Vulkan/Metal/DX12 with batched dispatch and automatic CPU fallback
Plugin Architecture: Extensible system supporting multiple compression algorithms with automatic scoring and selection
Random Access: O(1) tile-based decompression for seeking into large archives without full decompression
Metadata Preservation: Automatically preserves file modification times and Unix permissions
Pipeline Integration: Full stdin/stdout support for seamless Unix pipeline integration
Configuration Management: Per-user TOML configuration with environment variable overrides

Graceful Cancellation

Crush supports graceful cancellation of long-running operations via Ctrl+C (SIGINT):

Instant Response: Cancellation detection within 100μs for immediate feedback
Automatic Cleanup: Incomplete output files are automatically deleted on cancellation
User Hints: Large file operations (>1MB) display helpful hints about cancellation
Proper Exit Codes: Exit code 130 (Unix) / 2 (Windows) indicates cancelled operations
Clear Messaging: Displays "Operation cancelled" message for user feedback

# Start compression of a large file
crush compress large_dataset.bin
# ℹ️  Press Ctrl+C to cancel this operation

# Press Ctrl+C at any time
^C
# Output: Operation cancelled
# Exit code: 130
# Result: Incomplete large_dataset.bin.crush automatically deleted

Benefits:

No more waiting for operations you didn't want to complete
No manual cleanup of incomplete files
Safe interruption without corruption or partial files
Ideal for interactive use and long-running batch jobs

Performance

Crush delivers exceptional throughput for both compression and decompression operations.

Benchmark Results

Test Environment:

Rust: 1.93.1 (stable)
CPU: Multi-core (benchmarks use all available cores by default)
GPU: NVIDIA GeForce RTX 3060 Ti (Vulkan)
Build: Release mode with optimizations

CPU Compression (crush-core, 128 MB corpus)

Compression Scaling (64 KB blocks):

Workers	Throughput	Scaling
1	92 MiB/s	1.0×
2	181 MiB/s	1.97×
4	347 MiB/s	3.77×
8	576 MiB/s	6.26×
default (all cores)	983 MiB/s	—

Compression scales near-linearly with thread count. workers=0 (default) uses all available logical CPUs.

Compression Performance by Block Size (default workers):

Block Size	Throughput
64 KB	983 MiB/s
512 KB	957 MiB/s
1024 KB	941 MiB/s

CPU Decompression (crush-core)

Workers	Throughput
1	312 MiB/s
2	375 MiB/s
4	417 MiB/s
8	439 MiB/s
default (all cores)	435 MiB/s

GPU Decompression (crush-gpu, GDeflate)

GPU decompression uses batched dispatch — all tiles are submitted in a single GPU command to eliminate per-tile host-GPU synchronization overhead.

Corpus	GPU	CPU	Winner
log-1MB	137 MiB/s	329 MiB/s	CPU
binary-1MB	86 MiB/s	126 MiB/s	CPU
mixed-1MB	87 MiB/s	181 MiB/s	CPU
mixed-10MB	344 MiB/s	186 MiB/s	GPU 1.85x

GPU decompression outperforms CPU at larger data sizes (>2-4 MB) where the fixed dispatch overhead is amortized across many tiles. For smaller data, CPU is faster.

GPU Compression Throughput (GDeflate, CPU-side):

Corpus	Throughput
log-text-1MB	178 MiB/s
binary-1MB	70 MiB/s
mixed-1MB	99 MiB/s
mixed-10MB	104 MiB/s

Random Access Performance (64 MB corpus, 1 MB blocks):

Operation	Latency
Decompress last block	~1.12 ms
Decompress first block	~1.12 ms

Key Performance Features

Near-linear compression scaling: throughput doubles with each doubling of worker count up to CPU core count
SIMD-accelerated DEFLATE: powered by libdeflater for maximum per-thread throughput (~4x faster than pure-Rust backends)
GPU-accelerated decompression: batched GDeflate dispatch via wgpu (Vulkan/Metal/DX12) with automatic CPU fallback
Configurable parallelism: workers(n) limits CPU usage for background tasks; workers(0) (default) uses all cores
Consistent Random Access: O(1) block decompression via seekable block index (~1 ms per block)
Memory Efficient: Exact worst-case buffer sizing eliminates over-allocation

Running Benchmarks

# Run crush-core compression/decompression benchmarks
cargo bench -p crush-core --bench throughput

# Run random access benchmarks
cargo bench -p crush-core --bench random_access

# Run GPU decompression throughput benchmarks (requires compatible GPU)
cargo bench -p crush-gpu --bench throughput

# Run GPU compression ratio benchmarks
cargo bench -p crush-gpu --bench ratio

Comparing Against gzip/pigz

To compare crush against standard compression tools:

# Build crush first
cargo build --release

# Run comparison script (Linux/Mac/WSL)
bash benchmark-compare.sh

# Manual comparison example
echo "Test data..." > test.txt

# Crush
time ./target/release/crush compress test.txt
ls -lh test.txt.crush

# gzip --fast (comparable compression ratio to crush)
time gzip --fast --keep test.txt
ls -lh test.txt.gz

# pigz (parallel gzip)
time pigz --keep test.txt
ls -lh test.txt.gz

Note: Crush currently optimizes for speed over compression ratio. For fairest comparisons, compare against gzip --fast rather than default gzip, as they have similar compression ratios while crush provides significantly higher throughput.

Installation

From Source

# Clone the repository
git clone https://github.com/john-agentic-ai-tools/crush.git
cd crush

# Build release binary
cargo build --release

# Install to ~/.cargo/bin
cargo install --path crush-cli

Binary Release

Download the latest release from GitHub Releases.

Quick Start

Basic Compression

# Compress a file (creates data.txt.crush)
crush compress data.txt

# Compress with specific output name
crush compress data.txt -o compressed.crush

# Compress and keep the original file
crush compress data.txt --keep

Basic Decompression

# Decompress a file (creates data.txt)
crush decompress data.txt.crush

# Decompress to specific output name
crush decompress data.txt.crush -o output.txt

# Decompress and keep compressed file
crush decompress data.txt.crush --keep

Pipeline Usage

Crush supports stdin/stdout for seamless integration with Unix pipelines:

# Compress from stdin to file
cat data.txt | crush compress -o data.txt.crush

# Compress from stdin to stdout
cat data.txt | crush compress | ssh remote 'cat > data.txt.crush'

# Decompress from file to stdout
crush decompress --stdout data.txt.crush | grep "pattern"

# Full pipeline: compress and decompress
echo "Hello, World!" | crush compress | crush decompress

Usage Examples

File Compression

Compress a Single File

# Basic compression
crush compress data.txt
# Output: data.txt.crush

# With progress bar (shown for files > 1MB)
crush compress large_file.bin
# Compressing large_file.bin [=========>     ] 64% 450 MB/s

Compress Multiple Files

# Compress each file individually
for file in *.txt; do
    crush compress "$file"
done

# Or use shell globbing (requires bash/zsh)
crush compress *.txt  # Note: processes first file only, use loop for multiple

Force Overwrite

# Overwrite existing compressed file
crush compress data.txt --force

File Decompression

Decompress a Single File

# Basic decompression
crush decompress data.txt.crush
# Output: data.txt

# Decompress to stdout
crush decompress --stdout data.txt.crush > output.txt

Verify Decompression

# Decompress and verify file integrity (CRC32 check is automatic)
crush decompress data.txt.crush
# ✓ Decompressed: data.txt.crush → data.txt (CRC32 verified)

File Inspection

Inspect compressed files without decompressing:

# Basic inspection
crush inspect data.txt.crush
# File: data.txt.crush
# Size: 1.2 MB → 450 KB (62.5% reduction)
# Plugin: deflate
# Original name: data.txt
# Compressed: 2026-01-27 14:32:15
# CRC32: a3f5c2d1

# JSON output for scripting
crush inspect data.txt.crush --json
# {"file":"data.txt.crush","input_size":1200000,"output_size":450000,...}

# CSV output for batch processing
crush inspect *.crush --csv
# file,input_size,output_size,ratio,plugin,crc32
# data1.txt.crush,1200000,450000,62.5,deflate,a3f5c2d1
# data2.txt.crush,2400000,800000,66.7,deflate,b4e6d3f2

Plugin Management

List Available Plugins

# List all plugins
crush plugins list
# Available plugins:
#   deflate - DEFLATE compression (default)

# JSON output
crush plugins list --json

Get Plugin Information

# Get detailed plugin info
crush plugins info deflate
# Plugin: deflate
# Compression: DEFLATE (RFC 1951)
# Magic: 0x1F8B
# Features: fast, widely-supported
# Format: GZIP-compatible

Test Plugin Performance

# Benchmark a plugin
crush plugins test deflate
# Testing deflate plugin...
# Compression: 12.5 MB/s
# Decompression: 48.3 MB/s
# Ratio: 3.2x (average)

Configuration

Crush supports persistent configuration for default settings:

Set Configuration

# Set default compression level
crush config set compression.level fast

# Set default plugin
crush config set compression.plugin deflate

# Set default verbosity
crush config set general.verbose 1

Get Configuration

# Get specific setting
crush config get compression.level
# fast

# List all settings
crush config list
# compression.level = fast
# compression.plugin = auto
# general.verbose = 0

Reset Configuration

# Reset specific setting to default
crush config reset compression.level

# Reset all settings
crush config reset --all

Verbose Output

Control output verbosity with -v flags:

# Level 1: Show compression details
crush compress data.txt -v
# Compressing: data.txt
# Plugin: deflate (auto-selected)
# Compressed: 1.2 MB → 450 KB (62.5% reduction)
# Throughput: 125 MB/s

# Level 2: Show performance metrics
crush compress data.txt -vv
# Compressing: data.txt
# Plugin selection:
#   deflate: score 85.5 (compression: 90, speed: 75, ratio: 92)
# Compressed: 1.2 MB → 450 KB (62.5% reduction)
# Time: 9.6ms | Throughput: 125 MB/s
# CRC32: a3f5c2d1

# Level 3: Show debug information
crush compress data.txt -vvv
# [DEBUG] Reading input file: data.txt
# [DEBUG] File size: 1,200,000 bytes
# [DEBUG] Initializing plugin: deflate
# [DEBUG] Compression started
# ...

Logging

Enable structured logging for debugging and auditing:

# Log to file (JSON format)
crush compress data.txt --log compress.log

# Log with specific level
RUST_LOG=debug crush compress data.txt --log debug.log

# View logs
cat compress.log | jq
# {
#   "timestamp": "2026-01-27T14:32:15Z",
#   "level": "INFO",
#   "message": "Compression started",
#   "input_file": "data.txt",
#   "plugin": "deflate"
# }

Advanced Examples

Compress with Metadata Preservation

# File timestamps and permissions are preserved automatically
crush compress data.txt
crush decompress data.txt.crush
ls -l data.txt  # Original timestamp preserved

Batch Compression with Error Handling

# Compress all files, continue on error
for file in *.txt; do
    crush compress "$file" --force 2>> errors.log || echo "Failed: $file"
done

Pipeline Integration with Progress

# Download and compress on-the-fly
curl https://example.com/large-dataset.json | crush compress -o dataset.json.crush

# Note: Progress bar is hidden for stdin/stdout mode

Measure Compression Ratio

# Compare sizes
ls -lh data.txt data.txt.crush
# -rw-r--r-- 1 user user 1.2M Jan 27 14:30 data.txt
# -rw-r--r-- 1 user user 450K Jan 27 14:32 data.txt.crush

# Or use inspect command
crush inspect data.txt.crush | grep "reduction"
# Size: 1.2 MB → 450 KB (62.5% reduction)

Command Reference

`crush compress`

Compress files or stdin.

crush compress [OPTIONS] [INPUT]

Arguments:
  [INPUT]  Input file (omit for stdin)

Options:
  -o, --output <FILE>    Output file (default: INPUT.crush or stdout)
  -k, --keep             Keep input file after compression
  -f, --force            Overwrite existing output file
  -p, --plugin <NAME>    Force specific compression plugin (default: auto)
  -v, --verbose          Increase verbosity (-v, -vv, -vvv)
      --log <FILE>       Log operations to file
  -h, --help             Print help

`crush decompress`

Decompress files or stdin.

crush decompress [OPTIONS] <INPUT>

Arguments:
  <INPUT>  Input file or stdin (use '-' for stdin)

Options:
  -o, --output <FILE>    Output file (default: removes .crush extension or stdout)
  -k, --keep             Keep input file after decompression
  -f, --force            Overwrite existing output file
  -c, --stdout           Write to stdout
  -v, --verbose          Increase verbosity (-v, -vv, -vvv)
      --log <FILE>       Log operations to file
  -h, --help             Print help

`crush inspect`

Inspect compressed files.

crush inspect [OPTIONS] <FILES>...

Arguments:
  <FILES>...  Compressed files to inspect

Options:
      --json    Output as JSON
      --csv     Output as CSV
  -v, --verbose Increase verbosity
  -h, --help    Print help

`crush config`

Manage configuration.

crush config <COMMAND>

Commands:
  set     Set configuration value
  get     Get configuration value
  list    List all configuration
  reset   Reset configuration to defaults
  help    Print help

`crush plugins`

Manage compression plugins.

crush plugins <COMMAND>

Commands:
  list    List available plugins
  info    Show plugin information
  test    Test plugin performance
  help    Print help

Performance Tips

Maximize Throughput

Use pipelines: Compress/decompress in-memory without writing intermediate files
Parallel processing: Crush automatically uses multiple threads for large files
GPU decompression: For datasets larger than ~2-4 MB, GPU decompression (auto-detected) outperforms CPU
SIMD acceleration: CPU compression uses libdeflater's SIMD-optimized DEFLATE implementation

Optimize Compression Ratio vs Speed

# Fast compression (lower ratio, higher speed)
crush config set compression.level fast
crush compress large_file.bin

# Balanced (default)
crush config set compression.level balanced

# Best compression (higher ratio, slower)
crush config set compression.level best

Benchmarking CLI

# Measure compression performance
time crush compress large_file.bin

# Measure with verbose output
crush compress large_file.bin -vv
# Shows: throughput, time, ratio

Troubleshooting

Common Errors

"File already exists"

# Use --force to overwrite
crush compress data.txt --force

"Invalid compressed file"

# File may be corrupted or not a Crush file
crush inspect data.txt.crush
# Verify: Is this a .crush file? Check the magic number.

"Plugin not found"

# List available plugins
crush plugins list

# Use auto-selection instead of forcing specific plugin
crush compress data.txt  # Uses --plugin auto by default

Debug Mode

# Enable debug logging
RUST_LOG=debug crush compress data.txt --log debug.log

# View logs
cat debug.log

Getting Help

# General help
crush --help

# Command-specific help
crush compress --help
crush decompress --help
crush inspect --help

# Version information
crush --version

Integration Examples

Shell Scripts

#!/bin/bash
# backup.sh - Compress and archive logs

LOG_DIR="/var/log/app"
BACKUP_DIR="/backup/logs"

for log in "$LOG_DIR"/*.log; do
    if [ -f "$log" ]; then
        crush compress "$log" --keep -o "$BACKUP_DIR/$(basename "$log").crush"
    fi
done

CI/CD Pipeline

# .github/workflows/compress-artifacts.yml
name: Compress Build Artifacts

jobs:
  compress:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Crush
        run: cargo install --path crush-cli

      - name: Build artifacts
        run: make build

      - name: Compress artifacts
        run: |
          crush compress target/release/app -o app.crush
          crush inspect app.crush --json > artifact-info.json

      - name: Upload
        uses: actions/upload-artifact@v3
        with:
          name: compressed-build
          path: |
            app.crush
            artifact-info.json

Python Integration

import subprocess
import json

def compress_file(input_path, output_path):
    """Compress a file using Crush."""
    result = subprocess.run(
        ["crush", "compress", input_path, "-o", output_path],
        capture_output=True,
        text=True
    )
    return result.returncode == 0

def inspect_file(compressed_path):
    """Inspect a compressed file and return metadata."""
    result = subprocess.run(
        ["crush", "inspect", compressed_path, "--json"],
        capture_output=True,
        text=True
    )
    if result.returncode == 0:
        return json.loads(result.stdout)
    return None

# Usage
compress_file("data.txt", "data.txt.crush")
info = inspect_file("data.txt.crush")
print(f"Compression ratio: {info['compression_ratio']:.1%}")

Project Structure

crush/
├── crush-core/          # Core compression library
│   ├── src/
│   │   ├── lib.rs
│   │   ├── compression.rs
│   │   ├── decompression.rs
│   │   ├── inspection.rs
│   │   └── plugin/      # Plugin system
│   └── Cargo.toml
├── crush-gpu/           # GPU-accelerated compression engine
│   ├── src/
│   │   ├── lib.rs       # Plugin registration
│   │   ├── engine.rs    # Compress/decompress API
│   │   ├── backend/     # GPU backends (wgpu, CUDA)
│   │   ├── shader/      # WGSL compute shaders
│   │   ├── format.rs    # CGPU binary file format
│   │   ├── gdeflate.rs  # GDeflate algorithm
│   │   ├── scorer.rs    # GPU eligibility scoring
│   │   └── entropy.rs   # Shannon entropy analysis
│   ├── benches/         # Criterion benchmarks
│   └── Cargo.toml
├── crush-cli/           # CLI application
│   ├── src/
│   │   ├── main.rs
│   │   ├── commands/    # Command implementations
│   │   ├── config.rs    # Configuration management
│   │   └── output.rs    # Output formatting
│   └── Cargo.toml
└── Cargo.toml           # Workspace manifest

Development

Building

# Debug build
cargo build

# Release build (optimized)
cargo build --release

# Build specific crate
cargo build -p crush-core
cargo build -p crush-gpu
cargo build -p crush-cli

Testing

# Run all tests
cargo test

# Run specific test suite
cargo test -p crush-core
cargo test -p crush-gpu
cargo test -p crush-cli

# Run integration tests only
cargo test --test '*'

Benchmarking

# Run all benchmarks
cargo bench

# CPU compression/decompression throughput
cargo bench -p crush-core --bench throughput

# GPU decompression throughput (requires compatible GPU)
cargo bench -p crush-gpu --bench throughput

# CLI startup benchmarks
cargo bench -p crush-cli --bench cli_startup

Code Quality

# Format code
cargo fmt

# Lint code
cargo clippy -- -D warnings

# Generate documentation
cargo doc --no-deps --open

Contributing

See the Contributing Guide for details on how to contribute to Crush.

License

This project is licensed under [LICENSE] - see the LICENSE file for details.

Acknowledgments

Inspired by pigz by Mark Adler
Built with Rust
DEFLATE implementation via libdeflater (SIMD-optimized)
GPU compute via wgpu (cross-platform Vulkan/Metal/DX12)
GDeflate format based on Microsoft DirectStorage GDeflate

Name		Name	Last commit message	Last commit date
Latest commit History 108 Commits
.cargo-husky/hooks		.cargo-husky/hooks
.cargo		.cargo
.claude		.claude
.config		.config
.github		.github
.specify		.specify
crush-cli		crush-cli
crush-core		crush-core
crush-gpu		crush-gpu
crush-parallel		crush-parallel
specs		specs
.codecov.yml		.codecov.yml
.gitattributes		.gitattributes
.gitignore		.gitignore
.jscpd.json		.jscpd.json
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
COVERAGE.md		COVERAGE.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
TESTING.md		TESTING.md
benchmark-compare.ps1		benchmark-compare.ps1
benchmark-compare.sh		benchmark-compare.sh
clippy.toml		clippy.toml
demo.txt		demo.txt
demo.txt.crush		demo.txt.crush
rust-toolchain.toml		rust-toolchain.toml
rustfmt.toml		rustfmt.toml
test.txt		test.txt
test.txt.crush		test.txt.crush

Uh oh!

License

john-agentic-ai-tools/crush

Folders and files

Latest commit

History

Repository files navigation

Crush

Why Crush?

Features

Core Capabilities

Graceful Cancellation

Performance

Benchmark Results

CPU Compression (crush-core, 128 MB corpus)

CPU Decompression (crush-core)

GPU Decompression (crush-gpu, GDeflate)

Key Performance Features

Running Benchmarks

Comparing Against gzip/pigz

Installation

From Source

Binary Release

Quick Start

Basic Compression

Basic Decompression

Pipeline Usage

Usage Examples

File Compression

Compress a Single File

Compress Multiple Files

Force Overwrite

File Decompression

Decompress a Single File

Verify Decompression

File Inspection

Plugin Management

List Available Plugins

Get Plugin Information

Test Plugin Performance

Configuration

Set Configuration

Get Configuration

Reset Configuration

Verbose Output

Logging

Advanced Examples

Compress with Metadata Preservation

Batch Compression with Error Handling

Pipeline Integration with Progress

Measure Compression Ratio

Command Reference

crush compress

crush decompress

crush inspect

crush config

crush plugins

Performance Tips

Maximize Throughput

Optimize Compression Ratio vs Speed

Benchmarking CLI

Troubleshooting

Common Errors

"File already exists"

"Invalid compressed file"

"Plugin not found"

Debug Mode

Getting Help

Integration Examples

Shell Scripts

CI/CD Pipeline

Python Integration

Project Structure

Development

Building

Testing

Benchmarking

Code Quality

Contributing

License

`crush compress`

`crush decompress`

`crush inspect`

`crush config`

`crush plugins`

Packages