Skip to content

Latest commit

 

History

History
140 lines (97 loc) · 3.09 KB

verification.md

File metadata and controls

140 lines (97 loc) · 3.09 KB

Verification

Source of truth:

  • crates/core/src/verify.rs
  • crates/core/src/harness.rs
  • crates/harness-support/src/*
  • .github/workflows/ci.yml
  • CONTRIBUTING.md

Verification Philosophy

Verification combines:

  • build checks
  • tests
  • lint checks
  • optional custom project checks

Verify Runtime Model

Core types:

  • CheckKind: build|test|lint|custom
  • VerifyCheck: { kind, command }
  • Evidence: command result + stdout/stderr + timestamp + duration
  • VerifyReport: collection of evidence with summary rendering

Auto-detected Checks

detect_checks(project_root) defaults by project type:

Rust

  • cargo check
  • cargo test
  • cargo clippy -- -D warnings

Node

  • npm run build
  • npm test
  • optional npx eslint . when local eslint binary exists

Go

  • go build ./...
  • go test ./...
  • go vet ./...

Python

  • python -m pytest
  • python -m ruff check .

CI Baseline in This Repository

From .github/workflows/ci.yml:

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets (with RUSTFLAGS=-Dwarnings)
  • cargo nextest run
  • qvl harness run runtime
  • qvl harness run governance
  • qvl harness run routing
  • qvl harness run protocol
  • cargo test -p quavil-tui --test snapshots
  • cargo build --release -p quavil
  • sh -n infra/releases-worker/install.sh
  • bazel test //tools/bazel:parity

Harness and Snapshot Discipline

Quavil verification is split into two evidence classes:

  • deterministic harness cases decide pass/fail for release-gated lanes
  • observability metrics report runtime drift without silently flipping release status

Current release-blocking lanes:

  • runtime
  • governance

Currently CI-required deterministic lanes:

  • runtime
  • governance
  • routing
  • protocol

Supporting deterministic infrastructure includes:

  • scripted provider fixtures
  • mock OpenAI SSE transport
  • mock OpenAI realtime websocket transport
  • mock MCP streamable HTTP server
  • VT100 snapshot fixtures for primary TUI states
  • typed runtime perf facts persisted in SQLite for snapshot, replay, and MCP connect latency

Recommended Local Commands (This Repo)

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo run -p quavil -- harness run routing
cargo run -p quavil -- harness run protocol
cargo run -p quavil -- harness run all

Optional fast type-check:

cargo check --workspace
bazel test //tools/bazel:parity

CLI and Agent-facing Verification

  • verify tool runs checks and returns structured pass/fail evidence
  • qvl ci-fix can use CI logs to propose and apply fixes

Evidence Freshness

Evidence includes timestamps and can be freshness-checked via Evidence::is_fresh(max_age).

Failure Reporting

VerifyReport::summary() prints:

  • pass/fail status per check
  • command
  • elapsed time
  • tail of relevant stdout/stderr on failures

Best Practices

  • run checks before commit and before release
  • keep custom checks deterministic and non-interactive
  • for large changes, include both local command output and CI result links in PR description