Feature: add self-validation rules for automatic AIDLC workflow validation#79
Feature: add self-validation rules for automatic AIDLC workflow validation#79hmandhadpocs wants to merge 4 commits intoawslabs:mainfrom
Conversation
- Add self-validation.md in common/ with 9 validation categories - Validates always-execute stages, conditional stage logic, depth levels - Validates state tracking, audit trail, user interactions - Validates content validation compliance, plan checkboxes, two-part stages - Update core-workflow.md to load self-validation rules at workflow start - Addresses issue awslabs#44 for evaluating AIDLC methodology compliance
triggers at key checkpoints to ensure AIDLC methodology compliance. ## What Self-Validation Does Validates workflow execution across 9 categories: 1. Always-Execute Stage Validation - Verifies mandatory stages run 2. Conditional Stage Execution - Validates skip/execute logic 3. Depth Level Validation - Checks appropriate depth chosen 4. State Tracking Validation - Verifies aidlc-state.md maintained 5. Audit Trail Validation - Ensures audit.md completeness 6. User Interaction Validation - Confirms approval patterns 7. Content Validation Compliance - Checks diagram validation 8. Plan Checkbox Validation - Verifies immediate checkbox updates 9. Two-Part Stage Validation - Validates Planning + Generation flow ## Changes Made - Added 'APPLY THIS CHECK' instructions to all 9 validation categories specifying WHEN, WHERE, and HOW to apply each check - Removed stage-level validation details (answer completeness, artifact checks) keeping only workflow-level validation - Added MANDATORY SELF-VALIDATION CHECKPOINTS section to core-workflow.md defining automatic trigger points - Added CRITICAL: Self-Validation Enforcement section with blocking rules - Added 🔍 checkpoint markers at validation points in workflow stages ## Test Results ✅ PASSED (100% success rate) All 11 Self-Validation Triggers Executed Successfully: 1. ✅ Requirements Analysis (INCEPTION) 2. ✅ User Stories (INCEPTION) 3. ✅ Workflow Planning (INCEPTION) 4. ✅ Application Design (INCEPTION) 5. ✅ Units Generation (INCEPTION) 6. ✅ Functional Design (CONSTRUCTION) 7. ✅ NFR Requirements (CONSTRUCTION) 8. ✅ NFR Design (CONSTRUCTION) 9. ✅ Infrastructure Design (CONSTRUCTION) 10. ✅ Code Generation (CONSTRUCTION) 11. ✅ Build and Test (CONSTRUCTION) Key Findings: - Pattern Confirmed: Self-validation triggered AFTER each stage completion, BEFORE presenting completion message - 100% Pass Rate: All 53 validation checks passed across 11 triggers - Complete Coverage: 9 out of 9 validation categories tested - Zero Failures: No validation errors, no missing checkpoints, no audit gaps Addresses issue awslabs#44
|
Could you clarify @hmandhadpocs
|
|
Yes. 1/Issue #44 identified that we have no way to validate if agents are following the AIDLC methodology correctly. When someone changes a rule, we can't tell if it improved or broke the workflow. This PR addresses that gap. 2/Did the agent not perform workflow correctly before? Stages were skipped without validation - No check to verify skip decisions matched the criteria 3/Do these validation rules help us test changes in the workflow? The validation rules act as "guardrails" that ensure the agent follows the methodology correctly, similar to how unit tests validate code behavior. Reach out to hmandhad@amazon.com if you have more questions |
hmandhadpocs
requested review from
a team,
harmjeff,
raj-jain-aws,
scottschreckengaust and
spraja08
|
@hmandhadpocs - Thank you for this submission. This addresses an important problem. However, there are significant changes comping up on the AI-DLC workflow. These upcoming updates will directly address issue #44 in the baseline design. A complete rewrite of the rules language is in the works, and a sub-agents or SKILLs-based design is being discussed, which will require a different mechanism to tackle issue #44. Appreciate the ideas presented here. The new AI-DLC workflow upgrades will likely look quite different from the current approach and will inherently address issue #44. |
3 participants
Issue #44 , if available:
Description of changes:
Summary
This PR adds self-validation rules that automatically validate AIDLC workflow execution at key checkpoints. The AI agent now validates stage execution order, conditional stage logic, state tracking, audit trails, and workflow compliance without manual intervention.
What This Adds
1. self-validation.md
Workflow-level validation rules with 9 categories:
Each validation category includes "APPLY THIS CHECK" instructions (WHEN, WHERE, HOW).
2. core-workflow.md updates
Testing
Tested with Kiro CLI on a complete AIDLC workflow (greenfield project).
Tested against AI-DLC workshop Project - https://catalog.workshops.aws/aidlc/en-US/06-greenfield-development
Test Result: ✅ PASSED (100% success rate)
Self-validation triggered automatically at all 11 stages:
Key Findings:
Addresses
Closes #44
Checklist
aidlc-rules/aws-aidlc-rule-details/)Reached out to Alias: hmandhad@amazon.com