feat: focused profiling — only profile changed columns

README.md

@@ -9,6 +9,7 @@ Visual data diff for dbt + BigQuery. Compare production vs development data afte
 ```bash
 dbt build --select my_model        # 1. Build your changes
 ./data-diff.sh my_model            # 2. See what changed → opens HTML report
+./data-diff.sh --full my_model     # 3. (optional) Profile ALL columns
 ```
 
 Zero configuration — reads your GCP project, dbt project name, and schemas from the manifest automatically.
@@ -53,6 +54,14 @@ Compares your dev tables against production across three layers:
 
 The HTML report includes summary with risk indicators, per-model cards with column profiles (prod vs dev side-by-side), code diffs, and sample rows.
 
+### Focused Profiling
+
+By default, only **changed columns** are profiled — using sqlglot AST analysis to detect which columns were added or had their expressions modified. For wide models (100+ columns), this is dramatically faster.
+
+CTE modifications are classified as **additive** (new LEFT JOINs, new columns — safe for focused profiling) or **structural** (WHERE/filter/JOIN changes — falls back to full profiling automatically). The `EXCEPT DISTINCT` row comparison always covers all columns regardless.
+
+Use `--full` to force profiling of every column when needed.
+
 ## AI Agent Integration
 
 Includes a `SKILL.md` for [pi](https://github.com/mariozechner/pi-coding-agent) and Claude Code. The agent suggests running data-diff after validation passes and summarises findings in chat.

SKILL.md

@@ -48,11 +48,33 @@ Suggest running it when:
 
 # Multiple models
 .agents/skills/data-diff/data-diff.sh "int_order_pricing int_product_inventory"
+
+# Full profiling (all columns — slower but complete)
+.agents/skills/data-diff/data-diff.sh --full int_order_pricing
 ```
 
 The script prints progress to stderr and the output HTML file path to
 stdout.  It automatically opens the page in the browser on macOS.
 
+### Focused profiling (default)
+
+By default, data-diff only profiles **columns that changed** (added or
+expression-modified) using sqlglot analysis. This is dramatically faster
+for wide models (e.g. a 100+ column model profiling only the ~11 that
+changed).
+
+The EXCEPT DISTINCT row comparison still covers **all columns**, so data
+changes in non-profiled columns are still caught in the sample rows
+section.
+
+Fallback to full profiling happens automatically when:
+- sqlglot analysis fails or is unavailable
+- Only CTEs changed with no identifiable output column changes
+- The model is new (no production counterpart)
+
+Use `--full` to force profiling of all columns when you need complete
+statistical comparison (e.g. investigating indirect CTE changes).
+
 ### What happens under the hood
 
 | Step | What | Cost |
@@ -61,11 +83,12 @@ stdout.  It automatically opens the page in the browser on macOS.
 | 2. Code diff | sqlglot AST parse (local) | Free |
 | 3. Schema diff | `INFORMATION_SCHEMA` query | 1 fast query / model |
 | 4. Extract primary keys | Manifest parsing (local) | Free |
-| 5. Profile columns | BQ profiling query | 1 query / model / env |
-| 6. Sample rows | `EXCEPT DISTINCT` query | 1 query / model |
+| 5. Profile columns | BQ profiling query (focused: only changed cols) | 1 query / model / env |
+| 6. Sample rows | `EXCEPT DISTINCT` query (all columns) | 1 query / model |
 | 7–8. Assemble + render | JSON → HTML injection | Free |
 
-**Performance**: ~30–60 seconds for up to 5 models.
+**Performance**: ~30–60 seconds for up to 5 models. Focused profiling
+significantly speeds up wide models (100+ columns → only changed columns).
 
 ## Interpreting the Output