feat(sync): pre-compile dbt docs for repo sources to improve centralized docs context

[Flamki]

Summary

Adds optional dbt docs pre-compilation during nao sync for repository sources, so the agent can use compiled dbt artifacts (catalog.json, manifest.json) when projects use centralized docs references in YAML.

Closes #190.

Changes

• Added repo config options:
• compile_dbt_docs: bool (default false)
• dbt_profiles_dir: Optional[str]

• Updated repository sync flow:
• after clone/pull, when enabled, run:
dbt docs generate --project-dir [--profiles-dir ...]

• Added best-effort behavior:
• if repo is not dbt, dbt is missing, or generation fails, sync continues with warning logs

• Added unit tests for:
• skip when disabled
• skip when non-dbt repo
• skip when dbt binary missing
• execute command path with profiles dir
• integration into repo sync success flow

• Updated docs:
• README.md (usage + config example + behavior notes)
• README.md (“Recent Updates” note)
• nao_config.yaml enabling this for example dbt repo

Why

Raw dbt YAML with centralized doc() references can be hard for the agent to interpret directly. Compiled docs artifacts provide richer, resolved metadata.

Backward compatibility

• No behavior change unless compile_dbt_docs: true is explicitly set.
• Existing configs continue to work unchanged.

Validation

• test_repository_provider.py
• Result: 17 passed

[Bl3f]

Mentioned this on #209 because both are related and we should find the best combination of both to make it understandable for end users.

[Flamki]

Thanks for the feedback and for linking #209.

I cleaned up #213 and force-pushed a focused branch:

• removed unrelated history
• kept only the dbt compiled-docs sync change

Scope of #213 remains intentionally minimal:

• optional compile_dbt_docs repo setting
• runs dbt docs generate during repo sync
• exposes compiled artifacts (manifest.json, catalog.json) for centralized docs use cases
• best-effort behavior (non-blocking if dbt is missing/fails)

I see #213 as complementary to #209, not competing:

• feat(sync): pre-compile dbt docs for repo sources to improve centralized docs context #213: high-fidelity compiled output (requires dbt runtime)
• feat: add dbt project indexer for fast model lookup #209: static indexing approach (no dbt runtime required)

Happy to align on whether to keep both paths, pick one as default, or gate both behind config flags.

[cubic-dev-ai]

1 issue found across 13 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cli/nao_core/commands/sync/providers/repositories/provider.py">

<violation number="1" location="cli/nao_core/commands/sync/providers/repositories/provider.py:114">
P1: Missing exception handling contradicts the "best-effort" contract. Unlike `clone_or_pull_repo` which wraps its work in `try/except`, this function lets exceptions from `subprocess.run` (e.g., `OSError`, `PermissionError`) propagate up, which would crash the sync loop and skip all remaining repositories. Wrap the body in a try/except to match the sibling function's pattern.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[Bl3f]

Be careful you added the Trino code to this PR.

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cli/nao_core/commands/sync/providers/repositories/provider.py">

<violation number="1" location="cli/nao_core/commands/sync/providers/repositories/provider.py:115">
P1: `subprocess.run` for `dbt docs generate` has no `timeout`, which can hang the sync process indefinitely if the database is unreachable or slow. Consider adding a reasonable timeout (e.g., 300 seconds) to keep this best-effort and non-blocking.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[Bl3f]

Hey @Flamki you added the Trino code from your other PR to this one, could you remove it?

[Bl3f]

Regarding the dbt, its LGTM but remove the Trino code if you can pls.

[Bl3f]

To be removed from this PR.

[Bl3f]

Use the ui.py methods to display stuff in the console.

[Bl3f]

The thing I'm the most worried about is that a user will have to forward all the necessary dbt env variables to compile correctly the dbt docs (esp. if we want it in the "production" context). It adds a small complexity, but the feature is interesting, so we can add it, but might be complex for users to make it work.

[Bl3f]

One last thing I'm afraid about is the size of the manifest.json for large project that might make it unusable and pollute the context window. Nonetheless, as it generate also the compile version of each query in the folder structure that's interesting.

[Flamki]

Thanks for the review — addressed.

Updates made in this PR:

• Removed the unrelated Trino changes from this branch (scope is now dbt pre-compile only).
• Switched repository provider output to use ui.py helpers in cli/nao_core/commands/sync/providers/repositories/provider.py.
• Kept dbt docs generation best-effort and non-blocking.
• Added timeout for dbt docs generate to avoid hangs.
• Ran formatting fix required by Ruff.

Current status:

• Diff now contains only 6 files related to this feature.
• No conflicts with base branch.
• Checks are passing:
  • CLI Lint / Ruff Checks
  • cubic · AI code reviewer

Validation:

• PYTHONPATH=cli python -m pytest cli/tests/nao_core/commands/sync/test_repository_provider.py -q → 17 passed

Happy to adjust any wording in docs around env var complexity / manifest size if you want before merge.