Enhance configuration and caching mechanisms, add clear_cache option, and improve binary file handling

Author: igorls

.github/workflows/ci.yml

@@ -2,9 +2,9 @@ name: CI
 
 on:
   push:
-    branches: [ master ]
+    branches: [master]
   pull_request:
-    branches: [ master ]
+    branches: [master]
 
 env:
   CARGO_TERM_COLOR: always
@@ -15,7 +15,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ ubuntu-latest, windows-latest, macos-latest ]
+        os: [ubuntu-latest, windows-latest, macos-latest]
     steps:
       - name: Checkout code
         uses: actions/checkout@v5
@@ -43,20 +43,14 @@ jobs:
       - name: Build (default features)
         run: cargo build --verbose
 
-      - name: Build (no default features)
-        run: cargo build --no-default-features --verbose
-
       - name: Build (all features)
         run: cargo build --all-features --verbose
 
       - name: Run tests
-        run: cargo test --verbose
-
-      - name: Run tests (no default features)
-        run: cargo test --no-default-features --verbose
+        run: cargo test --verbose -- --test-threads=1
 
       - name: Run tests (all features)
-        run: cargo test --all-features --verbose
+        run: cargo test --all-features --verbose -- --test-threads=1
 
   security:
     name: Security Audit

.gitignore

@@ -12,13 +12,8 @@ Cargo.lock
 .DS_Store
 Thumbs.db
 
-# Output files
-*.md
-!README.md
-!CHANGELOG.md
-!docs/**/*.md
-
 # Logs
 *.log
 
 samples/
+docs/*.md

CHANGELOG.md

@@ -5,7 +5,6 @@ All notable changes to this project will be documented in this file.
 ## v0.5.0
 
 - **BREAKING CHANGES**
-  - Environment variable `CB_DIFF_CONTEXT_LINES` is no longer used; diff configuration is now handled explicitly through `DiffConfig`
   - Cache file locations changed to project-specific paths to prevent collisions
 
 - **Critical Bug Fixes**
@@ -39,26 +38,34 @@ All notable changes to this project will be documented in this file.
   - Fixed inconsistent file tree visualization between auto-diff and standard modes
 
 - **Testing & Quality**
-  - Added comprehensive integration test suite with 11 tests covering:
-    - Determinism verification (5 tests)
-    - Auto-diff workflows (6 tests) 
+  - Added comprehensive integration test suite with tests covering:
+    - Determinism verification
+    - Auto-diff workflows
     - Cache collision prevention
     - Configuration change detection
     - Error recovery scenarios
-  - All tests use `#[serial]` attribute to prevent race conditions
+  - Fixed test race conditions by running tests serially in CI (`--test-threads=1`)
   - Added `pretty_assertions` for better test output
+  - Fixed all clippy warnings and enforced `-D warnings` in CI
 
 - **Dependencies**
-  - Added `fs2 = "0.4.3"` for file locking
-  - Added `serde_json = "1.0"` for structured cache format
-  - Added `serial_test = "3.0"` for test serialization
-  - Added `pretty_assertions = "1.4"` for enhanced test output
+  - Added `fs2` for file locking
+  - Added `serde_json` for structured cache format
+  - Added `serial_test` for test serialization
+  - Added `pretty_assertions` for enhanced test output
+  - Added `encoding_rs` for enhanced encoding detection and transcoding
 
 - **Migration**
   - Automatic detection and cleanup of old markdown-based cache files (`last_canonical.md`, etc.)
   - First run after upgrade will clear old cache format to prevent conflicts
   - CLI interface remains fully backward compatible
 
+- **Code Quality & Maintenance**
+  - Fixed all clippy warnings including type complexity, collapsible if statements, and redundant closures
+  - Updated CI workflow to prevent race conditions in tests
+  - Improved binary file detection with better encoding strategy handling
+  - Enhanced error handling for edge cases and file system operations
+
 ## v0.4.0
 
 

Cargo.lock

@@ -21,12 +21,15 @@ env_logger = "0.11.8"
 rayon = { version = "1.10", optional = true }
 serde = { version = "1.0.219", features = ["derive"] }
 toml = "0.9.5"
-similar = "2.4.0"
+similar = "2.7.0"
 tempfile = "3.22.0"
 tiktoken-rs = "0.7.0"
-once_cell = "1.19.0"
+once_cell = "1.21.3"
 fs2 = "0.4.3"
 serde_json = "1.0.143"
+crossbeam-channel = "0.5.15"
+num_cpus = "1.17.0"
+encoding_rs = "0.8.35"
 
 [features]
 default = ["parallel"]

Cargo.toml

@@ -19,31 +19,31 @@ It's a command-line utility that recursively processes directories and creates c
 ## Core Features
 
 
-- ⚡ **Blazing Fast & Parallel by Default:**  
+- ⚡ **Blazing Fast & Parallel by Default:**
   Processes thousands of files in seconds by leveraging all available CPU cores.
 
-- 🧠 **Smart & Efficient File Discovery:**  
+- 🧠 **Smart & Efficient File Discovery:**
   Respects `.gitignore` and custom ignore patterns out-of-the-box using optimized, parallel directory traversal.
 
-- 💾 **Memory-Efficient Streaming:**  
+- 💾 **Memory-Efficient Streaming:**
   Handles massive files with ease by reading and writing line-by-line, keeping memory usage low.
 
-- 🌳 **Clear File Tree Visualization:**  
+- 🌳 **Clear File Tree Visualization:**
   Generates an easy-to-read directory structure at the top of the output file.
 
-- 🔍 **Powerful Filtering & Preview:**  
+- 🔍 **Powerful Filtering & Preview:**
   Easily include only the file extensions you need and use the instant `--preview` mode to see what will be processed.
 
-- ⚙️ **Configuration-First:**  
+- ⚙️ **Configuration-First:**
   Use a `.context-builder.toml` file to store your preferences for consistent, repeatable outputs.
 
-- 🔁 **Automatic Per-File Diffs:**  
+- 🔁 **Automatic Per-File Diffs:**
   When enabled, automatically generates a clean, noise-reduced diff showing what changed between snapshots.
 
-- ✂️ **Diff-Only Mode:**  
+- ✂️ **Diff-Only Mode:**
   Output only the change summary and modified file diffs—no full file bodies—to minimize token usage.
 
-- 🧪 **Accurate Token Counting:**  
+- 🧪 **Accurate Token Counting:**
   Get real tokenizer–based estimates with `--token-count` to plan your prompt budgets.
 
 
@@ -57,6 +57,41 @@ It's a command-line utility that recursively processes directories and creates c
 cargo install context-builder
 ```
 
+
+### If you don't have Rust installed
+
+Context Builder is distributed via crates.io. We do not ship pre-built binaries yet, so you need a Rust toolchain.
+
+
+#### Quick install (Linux/macOS):
+
+```bash
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+```
+Follow the prompt, then restart your shell
+
+#### Windows (PowerShell):
+
+```powershell
+irm https://sh.rustup.rs -UseBasicParsing | Invoke-Expression
+```
+
+After installation, ensure Cargo is on your PATH:
+
+```bash
+cargo --version
+```
+
+Then install Context Builder:
+```bash
+cargo install context-builder
+```
+
+Update later with:
+```bash
+cargo install context-builder --force
+```
+
 ### From source
 
 ```bash
@@ -100,6 +135,15 @@ context-builder --token-count
 # Add line numbers to all code blocks
 context-builder --line-numbers
 
+# Skip all confirmation prompts (auto-answer yes)
+context-builder --yes
+
+# Output only diffs (requires auto-diff & timestamped output)
+context-builder --diff-only
+
+# Clear cached project state (resets auto-diff baseline & removes stored state)
+context-builder --clear-cache
+
 # Combine multiple options for a powerful workflow
 context-builder -d ./src -f rs -f toml -i tests --line-numbers -o rust_context.md
 ```
@@ -129,6 +173,9 @@ auto_diff = true
 # Set to true to greatly reduce token usage when you just need what's changed.
 diff_only = false
 
+# Number of context lines to show around changes in diffs (default: 3)
+diff_context_lines = 5
+
 # File extensions to include
 filter = ["rs", "toml", "md"]
 
@@ -137,6 +184,19 @@ ignore = ["target", "node_modules", ".git"]
 
 # Add line numbers to code blocks
 line_numbers = true
+
+# Preview mode: only show file tree without generating output
+preview = false
+
+# Token counting mode
+token_count = false
+
+# Automatically answer yes to all prompts
+yes = false
+
+# Encoding handling strategy for non-UTF-8 files
+# Options: "detect" (default), "strict", "skip"
+encoding_strategy = "detect"
 ```
 
 ---
@@ -161,10 +221,11 @@ If you also set `diff_only = true` (or pass `--diff-only`), the full “## Files
 - `--preview` - Preview mode: only show the file tree, don't generate output.
 - `--token-count` - Token count mode: accurately count the total token count of the final document using a real tokenizer.
 - `--line-numbers` - Add line numbers to code blocks in the output.
-- `--diff-only` - With `--auto-diff` + `--timestamped-output`, output only change summary + modified file diffs (omit full file bodies).
+- `-y, --yes` - Automatically answer yes to all prompts (skip confirmation dialogs).
+- `--diff-only` - With auto-diff + timestamped output, output only change summary + modified file diffs (omit full file bodies).
+- `--clear-cache` - Remove stored state used for auto-diff; next run becomes a fresh baseline.
 - `-h, --help` - Show help information.
 - `-V, --version` - Show version information.
-
 ---
 
 ## Token Counting
@@ -189,4 +250,4 @@ See **[CHANGELOG.md](CHANGELOG.md)** for a complete history of releases and chan
 
 ## License
 
-This project is licensed under the MIT License. See the **[LICENSE](LICENSE)** file for details.
+This project is licensed under the MIT License. See the **[LICENSE](LICENSE)** file for details.

README.md

@@ -208,6 +208,8 @@ fn bench_scenario(c: &mut Criterion, spec: DatasetSpec, line_numbers: bool) {
         line_numbers,
         yes: true,
         diff_only: false,
+
+        clear_cache: false,
     };
 
     let prompter = NoPrompt;
@@ -249,6 +251,8 @@ fn bench_scenario(c: &mut Criterion, spec: DatasetSpec, line_numbers: bool) {
                     line_numbers: args.line_numbers,
                     yes: true,
                     diff_only: false,
+
+                    clear_cache: false,
                 },
                 Config::default(),
                 &prompter,