Draft
Conversation
Update ADR-001 with unified type definitions evolution Document how Micaiah identified type duplication between runtime parser and validation infrastructure, leading to the validator-merge refactor. The evolution: - Runtime parser had Diagnostic, FileLocation types - Validation introduced ValidationError, LocatedInputRef (duplicates!) - Unified these into common type definitions in txtx-core/validation - Runtime, linter, and LSP now share the same type definitions Key insights: - Eliminates type duplication (one Diagnostic type, not two) - Rust compiler enforces consistent usage across all systems - Validation logic remains isolated (preserves original safety) - Type definitions are DRY while validation stays parallel Suggested by Micaiah, refactored collaboratively.
## Architecture Introduce a shared validation layer in txtx-core that serves as the foundation for both CLI linting and LSP real-time diagnostics. The architecture consists of: - **ValidationContext**: Central state management for all validation operations, reducing parameter passing through the validation pipeline - **HCL Validator**: Two-tier system (BasicHclValidator and FullHclValidator) that performs AST traversal and structural validation - **Runbook Collector**: Extracts domain-specific items (inputs, variables, actions, signers) from runbook files for analysis - **Rule System**: Extensible linter rule infrastructure with RuleIdentifier supporting both core and addon-scoped rules ## Changes Add validation module (txtx-core/src/validation/): - context.rs: ValidationContext with builder pattern for flexible configuration - validator.rs: High-level validate_runbook API with ValidatorConfig - hcl_validator/: AST visitor pattern with block processors and dependency graph - hcl_diagnostics.rs: Convert HCL parse errors to user-friendly diagnostics - linter_rules.rs: Core lint rules (cli-override, input-naming, sensitive-data) - manifest_validator.rs: Validate inputs against manifest environments - rule_id.rs: Typed rule identifiers with addon scope support - types.rs: Shared validation types (errors, warnings, suggestions) - file_boundary.rs: Track line mappings for multi-file runbooks Add runbook analysis (txtx-core/src/runbook/): - collector.rs: Extract RunbookItems from HCL AST via visitor pattern - variables.rs: Track variable definitions, references, and resolution Add C4 architecture annotations to core validation components: - ValidationContext: @c4-component with state management responsibilities - HCL Validator: @c4-component with two-phase validation responsibilities - Manifest Validator: @c4-component with input validation responsibilities - FileBoundaryMapper: @c4-component documenting normalization strategy pattern ## Context This foundational layer enables static analysis of txtx runbooks before execution. The shared validation context eliminates duplicate logic between CLI and LSP, while the collector enables features like "find all references", unused variable detection, flow validation with related locations, file boundary mapping for accurate multi-file error locations, and runbook-scoped references for proper LSP scoping. The rule system is designed for extensibility, allowing addons to register custom validation rules. C4 annotations document the validation architecture for auto-generation of component diagrams, establishing the foundation for architecture-as-code documentation strategy.
## Architecture Introduce txtx-test-utils as a centralized testing toolkit that consolidates validation and execution testing. The module provides two main testing paths: - **Validation Testing**: Fluent RunbookBuilder API with SimpleValidator for static analysis testing without execution overhead - **Execution Testing**: TestHarness for full runbook execution with mocked blockchain interactions (moved from txtx-core) The validation layer supports two modes: HCL-only validation for syntax and basic semantics, and full manifest validation for environment and input checking. ## Changes Add txtx-test-utils crate with: - builders/runbook_builder.rs: Fluent API for programmatic runbook construction - builders/runbook_builder_enhanced.rs: Extended builder with additional helpers - builders/parser.rs: HCL parsing utilities for test content - simple_validator.rs: Lightweight validation wrapper using ValidationContext - assertions/mod.rs: Test assertion macros (assert_error, assert_validation_error) - addon_registry.rs: Addon specification extraction for tests - test_harness.rs: Mock-based execution testing (moved from core) - README.md: Comprehensive documentation with usage patterns and limitations - examples/enhanced_builder_example.rs: Reference implementation ## Context This consolidation reduces test boilerplate across the codebase and provides consistent patterns for testing both validation rules and execution flows. The builder API enables readable, maintainable tests while the dual-mode validation allows testing at different levels of strictness. The clear documentation of limitations helps developers choose between unit tests with RunbookBuilder and integration tests with the full lint pipeline. These utilities support testing of flow validation with related locations, multi-file runbook validation with file boundary mapping, LSP workspace diagnostics, and runbook-scoped reference resolution.
- Add shared location module with SourceLocation, SourceMapper, BlockContext - Migrate RunbookCollector to use SourceLocation instead of Location - Migrate VariableReference to use SourceLocation - Replace duplicate ParentContext with unified BlockContext - Eliminate ~150 lines of duplicate position mapping code - Add thiserror dependency for validation error display - Fix test compilation errors in linter_rules and rule_id
Replace PositionMapper with shared SourceMapper in HclValidationVisitor Update block_processors.rs to use SourceMapper for all position mapping Replace ReferenceContext enum with shared BlockContext from location module Add helper functions to convert SourceMapper results to Position type Fix FlowInputReference to use BlockContext instead of local ReferenceContext Update flow validation error handling for all BlockContext variants Fix tests to use variable blocks instead of action blocks
Fix FileBoundaryMap to handle empty files correctly Account for empty files occupying at least 1 line in concatenated content Fix manifest validator tests to use correct input prefix Change env.* prefix to input.* prefix in test cases
Extract extract_block_position helper to eliminate repeated pattern for extracting position from block identifier spans Add Default impl for Position with (1,1) as default start position Replace Position::new(1,1) calls with Position::default() Replace unwrap_or_else closures with unwrap_or_default Reduces duplicate code and makes intent clearer
- Add C4 component annotations to location.rs module - Document SourceLocationMapper architectural role - Enhance extract_block_position helper documentation - Clarify span-to-position conversion behavior
Enhance existing Diagnostic type in txtx-addon-kit with fields from ValidationError/ValidationWarning to eliminate type duplication. Create txtx-addon-kit/types/diagnostic_types.rs with common types: - DiagnosticLevel (Error/Warning/Note) - DiagnosticSpan (line/column ranges) - RelatedLocation (additional context locations) Add validation-specific fields to Diagnostic: - file, line, column (simple location) - context (additional context string) - related_locations (Vec<RelatedLocation>) - suggestion, documentation, example Add builder methods: with_file(), with_line(), with_column(), etc. Update validation code to use enhanced Diagnostic directly. Remove ValidationError/ValidationWarning type aliases. This creates a single Diagnostic type used across runtime, validation, linter, and LSP. Common field types can be shared, and Diagnostic is isolated as a container for future error-stack migration. All validation tests pass.
## Synopsis ``` txtx lint [RUNBOOK] [OPTIONS] txtx lint RUNBOOK --gen-cli [--manifest-path PATH] [--env ENV] txtx lint RUNBOOK --gen-cli-full [--manifest-path PATH] [--env ENV] txtx lint [RUNBOOK] [--manifest-path PATH] [--env ENV] [--input KEY=VALUE...] txtx lint [RUNBOOK] [--format stylish|compact|json|quickfix] txtx lint --init ``` ## Architecture Implement `txtx lint` command that performs static analysis of runbooks and manifests using the validation infrastructure from txtx-core. The architecture consists of: - **Linter Engine** (validator.rs): Orchestrates validation using ValidationContext from core and coordinates rule execution - **Workspace Analyzer** (workspace.rs): Discovers txtx.yml manifests by searching upward from current directory and resolves runbook files - **Rule System** (rules.rs): Function-based validation rules with typed CliRuleId for errors/warnings/suggestions - **Formatters** (formatter.rs): Multiple output formats (stylish, compact, json, quickfix) for IDE/CI integration - **Configuration** (config.rs): Centralized LinterConfig with manifest path, environment selection, and CLI input overrides ## Changes Add lint command (txtx-cli/src/cli/lint/): - mod.rs: Main entry point with run_lint orchestration and --gen-cli support Add linter module (txtx-cli/src/cli/linter/): - validator.rs: Linter struct with validate_content and IntoManifest trait - workspace.rs: WorkspaceAnalyzer with upward manifest discovery - rules.rs: ValidationContext, ValidationIssue types, and rule implementations (input-defined, naming-convention, cli-override, sensitive-data) - rule_id.rs: CliRuleId enum for typed rule identification - formatter.rs: Format enum with stylish/compact/json/quickfix outputs - config.rs: LinterConfig for configuring validation runs - README.md: Architecture documentation and usage examples Add shared utilities: - common/addon_registry.rs: Extract addon specifications for validation Add C4 architecture annotations to linter components: - Linter Engine: @c4-component orchestrating validation pipeline - WorkspaceAnalyzer: @c4-component for manifest discovery and runbook resolution - Documents normalization strategy (multi-file → single-file) in annotations - Relationships: WorkspaceAnalyzer → Linter Engine → ValidationContext ## Testing Include 24 unit tests covering: - Manifest discovery (upward search, git root boundary, explicit paths) - Runbook resolution (direct paths, standard locations, not found cases) - Validation engine (error detection, manifest context integration) - Circular dependency detection (2-way, 3-way, false positive prevention) - Rule system (CLI rule ID display and identification) ## Context The lint command provides pre-execution validation similar to TypeScript's tsc, catching configuration errors before runbook execution. The workspace analyzer enables project-level linting without explicit manifest paths, while the formatter abstraction supports both human (stylish) and machine (json/quickfix) outputs for IDE integration. The --gen-cli flag generates CLI commands from runbook inputs, bridging declarative runbooks with imperative shell workflows. The workspace analyzer supports multi-file runbook validation by combining runbook sources and tracking file boundaries for accurate error reporting across flow definitions, variable references, and related locations.
- Add lint command to Command enum - Create LintRunbook struct with all options - Add LintOutputFormat enum for output formatting - Create common module with mod.rs - Add txtx-addon-kit, colored, and yansi dependencies - Implement handle_lint_command to parse inputs and call linter - Support all lint flags: --init, --fix, --gen-cli, --config, etc.
- Add linter_tests_builder.rs with comprehensive test suite - Add EVM linter demo fixtures with correct/problematic examples - Add demo scripts for interactive linter testing - Add list_addon_functions utility for development
…tions Add c4-generator crate to extract C4 architecture annotations from Rust source code and generate Structurizr DSL. ## Implementation - Scan all .rs files in crates/ for @c4-* annotations in doc comments - Extract component metadata: name, container, description, technology - Extract relationships: @c4-uses, @c4-relationship - Extract responsibilities: @c4-responsibility - Generate workspace-generated.dsl with Structurizr DSL format - Proper ID sanitization for view keys (a-zA-Z0-9_-) - Group components by container with component views ## Rationale Bash script approach would be too complex due to: - Nested process substitution causing hangs with heredocs - Subshell scope issues preventing associative array updates - Fragile sed/grep patterns for extracting multi-line annotations - sed inconsistencies across platforms (BSD sed on macOS vs GNU sed on Linux) - Unreliable handling of optional annotation fields - Platform-specific shell differences (bash vs zsh) Rust provides: - Type-safe data structures (Component, HashMap) - Reliable regex parsing with capture groups - Proper error handling vs silent failures - Cross-platform compatibility - Native binary performance
Add comprehensive build infrastructure and developer tooling to streamline development workflow and handle build constraints: **Build System**: - Cargo aliases for building without supervisor UI (requires privileged tooling) - Separate unit and integration test targets for CLI, linter, and LSP - Just recipes providing ergonomic task runner interface - Development RUSTFLAGS to suppress common warnings during iteration **VSCode Integration**: - Launch configurations for debugging CLI, LSP, and tests - Task definitions for common build operations - Settings for Rust development with recommended extensions **Performance Monitoring**: - Criterion benchmarks for LSP request handling **Architecture Documentation**: - C4 diagram generation from code annotations - Structurizr Lite integration for viewing diagrams - Module dependency graph generation Add Cargo configuration (.cargo/config.toml): - build-cli, build-cli-release: Build without supervisor UI - test-cli-unit, test-cli-int: Separate unit and integration test targets - test-cli-unit-linter, test-cli-int-linter: Linter-specific tests - test-cli-unit-lsp, test-cli-int-lsp: LSP-specific tests - test-hcl-diagnostics, test-lsp-validation: Focused validation tests - tokio_unstable rustflag for async runtime features Add justfile with 30 recipes (195 lines): **Build recipes**: - build, build-release: Build CLI with/without optimizations - check: Fast syntax/type checking without codegen - fmt: Format code with rustfmt - clean: Remove build artifacts **Test recipes**: - cli-unit, cli-int: CLI unit and integration tests - lint-unit, lint-int: Linter-specific unit and integration tests - lsp-unit, lsp-int: LSP-specific unit and integration tests - test <name>: Run specific test by name - test-verbose: Run tests with full output - watch: Auto-run tests on file changes **Coverage recipes**: - coverage: Generate HTML coverage report - coverage-ci: Generate JSON coverage for CI/CD pipelines - coverage-test <name>: Coverage for specific test module **Analysis recipes**: - complexity-high: Find functions with high cyclomatic complexity - complexity-file <file>: Analyze complexity of specific file **Documentation recipes**: - doc: Generate and open API documentation - doc-all: Generate docs for all packages including dependencies **Architecture recipes**: - arch-c4: Generate C4 diagrams from @c4-* code annotations - arch-view: Generate and view diagrams with Structurizr Lite (podman/docker) - arch-modules: Generate module dependency graph with cargo-modules Add architecture tooling: - scripts/generate-c4-from-code.sh: Bash reference implementation (deprecated) - Just recipes use Rust c4-generator for reliable cross-platform generation - Auto-detect podman/docker with :Z flag for SELinux compatibility - Output to workspace-generated.dsl (gitignored) Add VSCode configuration: - .vscode/launch.json: Debug configurations for CLI, LSP, tests - .vscode/tasks.json: Build task definitions - .vscode/settings.json: Rust-analyzer and editor settings - .vscode/settings.json.example: Template for team settings Add benchmarks: - benches/lsp_performance.rs: Criterion benchmarks for LSP completion and hover Update build configuration: - crates/txtx-cli/Cargo.toml: Add criterion dev-dependency, configure benches - crates/txtx-core/Cargo.toml: Update dependencies - Cargo.toml: Add c4-generator to workspace members - Cargo.lock: Update dependency resolution (257 line changes) Update gitignore: - .structurizr/: Structurizr Lite generated files - workspace.json: Structurizr workspace metadata - workspace-generated.dsl: Auto-generated from code annotations - Coverage reports, VSCode files, build artifacts Add development files: - bacon.toml: Configure bacon file watcher Update CLI structure: - src/cli/mod.rs: Add ls-envs command, refactor module organization (123 line changes) - src/cli/common/mod.rs: Add common utilities module - src/cli/docs/mod.rs, src/cli/runbooks/mod.rs: Minor updates The supervisor UI build requires npm/node and specific frontend tooling not available to all developers. The cargo aliases and justfile recipes enable CLI-only builds with --no-default-features. The granular test aliases allow running specific test suites (unit vs integration, linter vs LSP) for faster iteration during development of multi-file validation, flow tracking, and runbook-scoped reference features. Architecture tooling supports hybrid documentation strategy: - arch-c4 generates workspace-generated.dsl from @c4-* annotations - arch-view launches Structurizr Lite for interactive diagram viewing - Rust c4-generator replaces bash script for reliable cross-platform support - Bash script kept as reference but deprecated in favor of Rust utility
- Remove lsp-unit and lsp-int test recipes - Remove arch-view-lsp architecture recipe - Update help text to show only linter commands - Remove lsp_performance benchmark
Linter documentation: - Add linter architecture and C4 diagrams - Add user guides for linter configuration and usage - Add validation error examples - Add internal linter plugin system documentation - Add validation architecture documentation Development documentation: - Add developer guide with setup and contribution guidelines - Add comprehensive testing guide - Add .gitattributes for consistent line endings Update README with linter features and documentation links.
Architecture documentation: - Add main documentation index (linter-focused) - Add architecture overview and features - Add performance improvements documentation Linter Architecture Decision Records: - ADR-001: PR architectural premise - ADR-003: Capture everything, filter later pattern - ADR-004: Visitor strategy pattern with readonly iterators
Adapt linter implementation to work with enhanced txtx-addon-kit Diagnostic that replaced ValidationError/ValidationWarning. Changes: - Use Diagnostic builder methods (with_file, with_line, etc.) - Handle file as Option<String> instead of String - Update field name from documentation_link to documentation
The base branch (525ba52) includes an old LSP implementation using tower-lsp and txtx-lsp crate. Restore these dependencies so the linter branch builds successfully: - Add txtx-lsp to workspace members - Add tower-lsp dependency for old LSP implementation - Add txtx-lsp as dependency of txtx-cli (required by native_bridge.rs)
- Add LSP architecture documentation with async implementation details - Add C4 diagrams for LSP architecture - Add sequence diagrams for LSP interactions - Add state management documentation - Add use case diagrams and specifications - Add user guide for LSP features and IDE integration
## Synopsis ``` txtx lsp ``` Starts the Language Server Protocol server for IDE integration. Communicates over stdin/stdout using the LSP protocol. ## Architecture Implement LSP server for txtx runbooks providing IDE features through a handler-based architecture with AST-powered reference tracking and intelligent state management: - **Handler System**: Request handlers for completion, definition, hover, references, rename, and diagnostics with shared workspace state - **AST-Based References** (hcl_ast.rs): Uses hcl-edit parser (same as runtime/linter) for accurate reference extraction with strict and lenient modes - **Workspace State** (workspace/state.rs): Thread-safe state with validation caching, dependency tracking, and environment resolution - **State Machine** (workspace/state_machine.rs): Explicit workspace states (Ready, Validating, etc.) with audit trail for debugging - **Smart Validation**: Content hashing prevents redundant validation; dependency graph enables cascade validation through transitive dependencies (A→B→C) - **Multi-Environment Support**: Cross-file operations work across environment-specific files (e.g., signers.sepolia.tx, signers.mainnet.tx) - **Multi-File Runbooks**: Workspace diagnostics validate entire runbooks with file boundary mapping for accurate error locations across flow definitions - **Runbook-Scoped References**: References and rename operations properly scoped to runbook boundaries (variables, flows, actions, outputs) vs workspace-wide (inputs, signers) ## Changes Add LSP implementation (txtx-cli/src/cli/lsp/): - mod.rs: Main server loop with request routing - async_handler.rs: Async request handling with caching - hcl_ast.rs: AST-based reference extraction with visitor pattern Add handlers (handlers/): - completion.rs, definition.rs, hover.rs, references.rs, rename.rs - diagnostics.rs: Real-time diagnostics with multi-file runbook support - document_sync.rs, workspace.rs - environment_resolver.rs, workspace_discovery.rs, debug_dump.rs Add workspace state (workspace/): - state.rs: WorkspaceState with validation cache and dependency graph - state_machine.rs: MachineState with 50-transition audit trail - validation_state.rs: Per-document validation status tracking - dependency_graph.rs: Transitive dependency resolution with cycle detection - dependency_extractor.rs, documents.rs, manifests.rs, manifest_converter.rs Add validation integration (validation/): - adapter.rs, converter.rs, hcl_converter.rs: Adapt linter for LSP diagnostics Add multi-file support: - diagnostics_multi_file.rs: Workspace diagnostics for entire runbooks - multi_file.rs: Runbook expansion and file boundary mapping - Related locations in diagnostics for flow validation errors Add utilities: - utils/: environment.rs, file_scanner.rs - diagnostics.rs, diagnostics_hcl_integrated.rs - linter_adapter.rs, functions.rs Remove deprecated txtx-lsp crate (5039 lines) Add documentation (2158 lines): - docs/lsp-state-management.md: State architecture and implementation status - docs/lsp-sequence-diagram.md, docs/lsp-use-case-diagram.md - README.md, ASYNC_GUIDE.md ## Features - Code completion for inputs, actions, and signers - Go-to-definition across multiple files and environments - Find all references with runbook-scoped filtering - Rename refactoring with runbook scope awareness (updates references, preserves prefixes) - Real-time diagnostics using linter rules with multi-file runbook support - Flow validation with related locations showing missing inputs across files - File boundary mapping for accurate error locations in multi-file runbooks - Hover documentation for functions and actions - Environment-aware manifest resolution - Automatic cascade validation when dependencies change ## Testing Include 200+ unit and integration tests covering: - State machine transitions and audit trail - Cascade validation through dependency chains - Multi-file and multi-environment references/rename - Runbook-scoped reference filtering (variables, flows, actions vs inputs, signers) - HCL diagnostics integration - Multi-file runbook workspace diagnostics - Flow validation with related locations - Environment switching and manifest resolution - Dependency extraction and graph operations - Mock editor for end-to-end testing ## Context Provides IDE features for txtx runbooks with real-time validation feedback. AST-based reference extraction ensures consistency with runtime/linter and eliminates false positives in strings/comments. State machine enables observability for debugging. Multi-environment support handles enterprise workflows with environment-specific configurations. Multi-file runbook support enables proper flow validation with accurate error locations and related location tracking. Runbook-scoped references prevent cross-runbook pollution while maintaining workspace-wide visibility for manifest inputs. Integrates with CLI linter to share validation logic.
- Add lsp_tests_builder.rs for LSP test generation - Provides test suite for validating LSP implementations
- Add missing dependencies: serde_yml, dashmap, lru, futures - Update tokio with required features: rt-multi-thread, macros, sync - Fix lsp-server version to 0.7.6 (compatible with Cargo 1.84) - Remove legacy txtx-lsp crate from workspace members - Fix run_lsp() call: remove .await, add error conversion - Add dev dependencies for tests: criterion, txtx-test-utils, tempfile - All 49 tests passing (43 linter + 6 LSP)
Adapt LSP implementation to work with enhanced txtx-addon-kit Diagnostic that replaced ValidationError/ValidationWarning. Changes: - Update converter functions to use unified Diagnostic type - Replace documentation_link with documentation field - Add diagnostic_to_lsp as primary conversion function
Update validation_integration_test to use Diagnostic builder methods instead of removed ValidationError struct.
Consolidate duplicate filter_runbook_uris, expand_runbook_uris, and is_manifest_file functions from references and rename handlers into shared handlers/common.rs module. Changes: - Create handlers/common.rs with shared utilities - Move is_manifest_file from mod.rs to common.rs - Remove duplicate implementations from references.rs and rename.rs - Update imports to use shared functions
Consolidate duplicate diagnostic-to-LSP conversion logic into a single converter module, reducing duplication across linter_adapter and diagnostic provider. Changes: - Create diagnostics/converter.rs with unified to_lsp_diagnostic() - Reorganize diagnostics.rs into diagnostics/ module structure - Update linter_adapter.rs to use unified converter (-58 lines) - Update diagnostics provider to use unified converter (-43 lines) - Add converter tests for range calculation and message building
Replace imperative loops with functional iterator chains and update doc comments to follow Rust conventions. - Add validation_result_to_diagnostics() converter using chain/map/collect - Replace manual iteration in provider, linter_adapter, and validation adapter
Remove workspace-level state machine. Per-document ValidationStatus provides sufficient state tracking for now. Preserved in branch: archive/state-machine-impl Changes: - Remove state_machine.rs - Remove state_machine_test.rs - Remove MachineState and StateHistory fields from WorkspaceState - Remove process_event() and related methods
22 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.