aiQA • AI-Powered Network Test Case Generation

Platforms
Integrations

Overview

aiQA in a nutshell: AI-powered network test case generator for multi-vendor networks.

Describe your network intent once. aiQA generates RFC-compliant test cases from it — vendor-specific CLI commands, precise assertions, and full traceability to RFC sections. Output is a framework-agnostic YAML spec rendered into ready-to-run pytest suites and Ansible playbooks.

Recommended model:

• Opus 4.6, effort Medium

Output samples:

• See output_samples

Documentation:

• WORKFLOW.md

What's new in v1.3:

• CHANGELOG.md

Tech Stack

Technology	Role
Python	Core language
FastMCP	MCP server exposing 3 tools
Claude	Reasoning, test design, spec generation
LangChain	RAG pipeline (chunking, embedding, retrieval)
ChromaDB	Vector database for knowledge base

Scope

aiQA uses a single general-purpose /qa skill that handles any protocol, any feature, and any test type from a natural language request. No per-protocol skill files needed.

All tests are active: configure a condition → wait → check the result → teardown (revert). Every test modifies device configuration and must have a complete teardown block. The agent warns the user and requires explicit confirmation before generating any tests.

Supported protocols (any protocol present in INTENT.json + KB):

• OSPF (RFCs 2328, 3101; 6 vendor implementations)
• BGP (RFCs 4271, 4760)
• EIGRP (RFC 7868)

Test Network Topology

Network diagram:

Lab environment:

• 16 devices defined in TOPOLOGY.yml
• 8 x Cisco IOS-XE, 4 x Arista cEOS, 2 x MikroTik CHR, 1 x Juniper JunOS, 1 x Aruba AOS-CX
• See lab_configs for my test network's configuration

Installation

Prerequisites: Python 3.12+

Step 1 - Install and ingest:

sudo apt install git make python3.12-venv -y
cd ~ && git clone https://github.com/pdudotdev/aiQA
cd aiQA && make setup

Step 2 - Authenticate with Claude:

claude auth login

Step 3 - Register the MCP server:

claude mcp add aiqa -s user -- /home/<user>/aiQA/aiqa/bin/python /home/<user>/aiQA/server.py

MCP Tools

Tool	Description
search_knowledge_base	Search network knowledge base (RFCs, vendor guides) with protocol/vendor/topic filters
query_intent	Retrieve network design intent and inventory from data/INTENT.json
list_devices	List inventory devices, optionally filtered by CLI style

Customization

aiQA is designed to work with your own test topology. Bring your own:

What	Where	Format
Network intent	data/INTENT.json	JSON: router roles, OSPF areas, links, neighbors, inventory fields
Protocol docs	docs/*.md	Markdown (re-ingest with make ingest)

QA Workflow

Generate tests from natural language:

claude
> /qa OSPF timer mismatch tests between A4M and D2B

The skill:

1. Parses the request — protocol, feature, device scope, failure mode
2. Resolves device intent (scoped per-device queries, not full topology dump)
3. Asks clarifying questions if needed (scope, test condition)
4. Searches the KB for RFC grounding, vendor CLI commands, and rollback patterns
5. Presents a test plan for confirmation before generating anything
6. Generates a YAML spec, pytest suite, and Ansible playbook

Output	Path	Description
YAML spec	output/spec/<protocol>_<feature>[_<scope>].yaml	Canonical, framework-agnostic test specification
Pytest suite	output/pytest/test_<protocol>_<feature>[_<scope>].py	Executable tests using Netmiko for SSH
Ansible playbook	output/ansible/playbook_<protocol>_<feature>[_<scope>].yml	Ansible tasks using cli_command and cli_config modules
Emergency rollback	output/ansible/playbook_<protocol>_<feature>_rollback.yml	Unconditional teardown playbook

Safety model: Every test has a mandatory teardown block. pytest uses try/finally (teardown always runs). Ansible uses block/always. A session-level rollback registry in conftest.py covers interrupted test runs. Cross-vendor pairs test both directions; same-vendor pairs test one direction only.

Knowledge Base

Protocol documentation lives in docs/ as Markdown files. Each file is tagged with vendor, topic, and protocol metadata during ingestion.

To update after editing docs:

make ingest

RAG optimizations in place:

• protocol metadata field — filters search by protocol (ospf, bgp, eigrp), eliminating cross-protocol noise
• Contextual chunk headers — source and protocol prepended to each chunk for better embedding quality
• Compound filtering — combine vendor, topic, and protocol filters in a single query
• Scoped intent queries — query_intent("<device>") per named device; full topology only when needed
• Lazy-loaded schema files — spec-schema.md at spec generation, spec-renderers.md at rendering

See OPTIMIZATIONS.md for the full optimization roadmap.

Project Structure

aiQA/
├── server.py                     # FastMCP server (3 tools)
├── ingest.py                     # RAG ingestion pipeline
├── CLAUDE.md                     # Agent system prompt (tools, quality standards, data model)
├── CHANGELOG.md                  # Version history
├── TOPOLOGY.yml                  # ContainerLab topology definition
├── Makefile                      # Setup automation (make setup / ingest / clean)
├── requirements.txt
├── data/
│   ├── INTENT.json               # Network design intent + inventory (16 devices)
│   └── chroma/                   # ChromaDB vector store (generated)
├── docs/                         # Knowledge base (RFCs + vendor guides)
│   ├── rfc2328_summary.md        # OSPF
│   ├── rfc3101_nssa.md           # OSPF NSSA
│   ├── rfc4271_bgp.md            # BGP-4
│   ├── rfc4760_mpbgp.md          # MP-BGP
│   ├── rfc7868_eigrp.md          # EIGRP
│   └── vendor_*_<protocol>.md    # e.g. vendor_cisco_ios_ospf.md, vendor_cisco_ios_bgp.md
├── .claude/
│   ├── spec-schema.md            # YAML spec schema — loaded at Step 7, before generation
│   ├── spec-renderers.md         # pytest + Ansible renderer guidance — loaded at Step 9, before rendering
│   └── skills/qa/
│       └── SKILL.md              # /qa general QA methodology skill (13-step workflow, QC-1 through QC-8)
├── output/                       # Generated test artifacts
│   ├── spec/                     # YAML test specifications
│   ├── pytest/                   # pytest test files
│   └── ansible/                  # Ansible playbooks
├── output_samples/               # Reference output for showcase
├── testing/                      # Manual test scenarios and RAG retrieval tests
├── metadata/
│   ├── scalability/              # RAG optimization roadmap
│   └── workflow/                 # End-to-end workflow documentation
└── lab_configs/                  # Test network device configurations

Disclaimer

You are responsible for defining your own network inventory and design intent, building your test environment, and meeting the necessary prerequisites (Python 3.11+, Claude CLI/API, network device access).

License

Licensed under GPLv3.

Collaborations

Interested in collaborating?

• LinkedIn: LinkedIn