Create Recursive Language Model skill epic

[ericelliott]

Add comprehensive task epic for building RLM skill that applies MIT's
Recursive Language Model strategies to AI agent codebase exploration.
Epic includes 7 tasks covering research, design, implementation of
fan-out search, SQLite indexing, and SudoLang skill specification.

---

Note

Lays groundwork for the RLM skill with a SQLite indexing schema and planning docs, plus project setup updates.

• Adds ai/tools/create-schema.js (with CLI) and comprehensive tests to create SQLite schema: documents + FTS5, dependencies, and ProductManager tables (personas, journeys, steps, pain_points, stories, functional_requirements), with indexes and triggers
• Introduces RLM planning docs under plan/rlm/ (spec, architecture, database design, research summary)
• Updates package.json (adds better-sqlite3, new test:bun, adjusts unit test filters) and adds .gitattributes
• Updates indexes (ai/index.md, ai/rules/index.md) and adds ai/rules/sudolang/index.md and ai/tools/index.md

^{Written by Cursor Bugbot for commit 5bb8333. This will update automatically on new commits. Configure here.}

[ericelliott]

@cursoragent please /review and diagnose failures

[cursor]

🔬 Code Review: Recursive Language Model Skill Epic

🎯 Review Context

Reviewing PR #84 which adds comprehensive RLM skill groundwork including SQLite schema creation, planning documentation, and project configuration updates. The CI is failing with bun:sqlite import errors.

---

🚨 CI Failure Diagnosis

Root Cause

Issue: The test suite uses bun:sqlite imports, but CI runs on Node.js with Vitest.

import { Database } from "bun:sqlite";
import { createSchema } from "./create-schema.js";

Failure Flow:

1. CI executes npm test (runs all tests via Vitest)
2. Vitest attempts to load create-schema.test.js
3. Node.js cannot resolve bun:sqlite (Bun-only module)
4. Test fails: Cannot find package 'bun:sqlite'

Current Mitigation (Incomplete):

• ✅ test:unit excludes this test
• ✅ test:bun runs with Bun
• ❌ npm test still includes it (what CI runs)

Solution Options

Option 1: Make Code Runtime-Agnostic (Recommended)

// Detect runtime and use appropriate SQLite library
const Database = globalThis.Bun 
  ? (await import('bun:sqlite')).Database
  : (await import('better-sqlite3')).default;

Pros: Works everywhere, uses optimal library per runtime
Cons: Adds conditional logic, slight complexity

Option 2: Update CI to Run Bun Tests Separately

- name: Run Node tests
  run: npm run test:unit
  
- name: Install Bun
  uses: oven-sh/setup-bun@v1
  
- name: Run Bun tests
  run: npm run test:bun

Pros: Clean separation, tests real Bun environment
Cons: CI complexity, longer run time

Option 3: Exclude from Main Test Script

"test": "vitest run --exclude '**/*-e2e.test.js' --exclude '**/create-schema.test.js' && npm run -s lint && npm run -s typecheck"

Pros: Quick fix, mirrors test:unit
Cons: Test doesn't run in default CI, easy to forget

Recommendation: Implement Option 1 for maximum compatibility, then add Option 2 to CI for comprehensive testing.

---

1. Code Structure and Organization

🎯 What I'm Reviewing

Code organization, file placement, and architectural decisions for the SQLite schema foundation.

💡 Observations

Strengths:

• ✅ Clear separation of concerns: schema creation vs tests
• ✅ CLI-friendly design with #!/usr/bin/env bun shebang
• ✅ Proper export of core function for testing
• ✅ Planning docs well-organized in plan/rlm/
• ✅ Follows existing project structure conventions

Issues:

• ❌ Outdated index files: ai/tools/index.md says "This directory is empty" but contains files
• ❌ Missing TypeScript definitions: No create-schema.d.ts file
• ⚠️ Inconsistent naming: File is create-schema.js but could be schema.js (simpler)

🪞 Critical Reflection

The planning documentation is thorough and well-researched. However, the implementation shows signs of rushing - index files weren't updated, type definitions are missing, and runtime compatibility wasn't considered.

🔭 Broader Context

This is foundational code for Phase 1 of a 5-phase epic. Any compatibility issues here will cascade to future phases. Getting the foundation right is critical.

⚖️ Score: 7/10

Good structure, but missing polish and completeness checks.

---

2. Adherence to Coding Standards

🎯 Checking Against

• javascript/javascript.mdc - JavaScript best practices
• tdd.mdc - Test-driven development patterns
• Project conventions

💡 Observations

Follows Project Standards: ✅

• Uses ESM imports
• Functional programming style
• Riteway + Vitest for tests
• Proper use of beforeEach/afterEach
• Named exports over default exports

JavaScript Best Practices: ✅

export function createSchema(db) {

• Clear, self-describing function name ✅
• Single responsibility ✅
• Pure function (no side effects beyond db mutation) ✅

Test Quality: ✅

  test("should return version and success status", async () => {
    const result = createSchema(db);

    assert({
      given: "schema creation",
      should: "return version 1",
      actual: result.version,
      expected: 1,
    });

• Answers the 5 questions (unit, behavior, actual, expected, debugging) ✅
• Isolated tests with fresh db per test ✅
• Explicit assertions ✅

Issues:

• ⚠️ Uses async in test functions but doesn't await anything (unnecessary)
• ⚠️ Date.now() for timestamps (not mockable for deterministic tests)
• ❌ Missing JSDoc for public API function

🪞 Critical Reflection

The code follows FP principles well. Tests are thorough and isolated. However, some async/await usage is cosmetic, and the lack of JSDoc hurts discoverability.

⚖️ Score: 8/10

Strong adherence to standards, minor improvements needed.

---

3. Test Coverage and Quality

🎯 Evaluating

Test comprehensiveness, edge cases, and TDD compliance.

💡 Analysis

Coverage: Excellent ✅

• Schema creation verification
• Table existence checks
• Column validation
• FTS5 virtual table
• Indexes verification
• Foreign key constraints
• Triggers
• CHECK constraints
• Idempotency

Test Quality: Strong ✅

  test("should enforce foreign key constraints", async () => {
    createSchema(db);

    let errorThrown = false;
    let errorMessage = "";

    try {
      db.prepare(
        `
        INSERT INTO stories (
          id, step_id, name, priority, status, created_at, updated_at
        ) VALUES (
          'story1', 'nonexistent', 'Test Story', 50, 'backlog', ?, ?
        )
      `,
      ).run(Date.now(), Date.now());
    } catch (error) {
      errorThrown = true;
      errorMessage = error.message;
    }

    assert({
      given: "invalid foreign key",
      should: "throw foreign key constraint error",
      actual: errorThrown && errorMessage.includes("FOREIGN KEY"),
      expected: true,
    });

Tests verify both happy paths and constraints. Good coverage of SQL semantics.

Missing Tests:

• ❌ No tests for CLI execution path (lines 266-290)
• ❌ No integration tests with real file system
• ❌ No performance benchmarks (important per design doc)
• ⚠️ No tests for concurrent access (SQLite locking)

🔭 Expansion

The design doc mentions "TDD-friendly in-memory tests" and this delivers. However, missing integration tests for the CLI path and performance validation against stated goals (100 files in <1s).

⚖️ Score: 8.5/10

Excellent unit test coverage, missing integration and performance tests.

---

4. Performance Considerations

🎯 Evaluating

Implementation efficiency against stated performance goals.

💡 Analysis

Design Goals (from database-design.md):

• 100 files: ~100ms ✓ (stated goal)
• Simple query: <10ms ✓ (stated goal)
• FTS5 search: <200ms ✓ (stated goal)

Implementation: Good Foundation ✅

  // Enable foreign keys
  db.exec("PRAGMA foreign_keys = ON");

  // Schema version tracking
  db.exec(`
    CREATE TABLE IF NOT EXISTS schema_version (
      version INTEGER PRIMARY KEY,
      applied_at INTEGER NOT NULL
    )
  `);

Efficient Patterns:

• ✅ Uses IF NOT EXISTS for idempotency
• ✅ Proper indexes on foreign keys
• ✅ FTS5 with Porter stemming for better search
• ✅ Triggers keep FTS5 in sync automatically

Missing Optimizations:

• ⚠️ Multiple db.exec() calls instead of single transaction
• ⚠️ No PRAGMA journal_mode = WAL for better concurrency
• ⚠️ No PRAGMA synchronous = NORMAL for speed/safety balance

Potential Issues:

  db.exec(`
    CREATE INDEX IF NOT EXISTS idx_documents_type ON documents(type);
    CREATE INDEX IF NOT EXISTS idx_documents_indexed_at ON documents(indexed_at);
    CREATE INDEX IF NOT EXISTS idx_documents_modified_at ON documents(modified_at);

Multiple indexes created separately. Could use single exec with multiple statements for better performance.

🪞 Critical Reflection

Schema is well-designed for performance, but implementation doesn't wrap everything in a transaction. For 100+ tables/indexes, this could be noticeable.

⚖️ Score: 7/10

Good design, missing production-grade optimizations.

---

5. Security Scan

🎯 Checking OWASP Top 10 + Project Security Rules

OWASP Top 10 (2021) Analysis:

1. A01: Broken Access Control - N/A (no auth layer yet)
2. A02: Cryptographic Failures - ⚠️ See below
3. A03: Injection - ✅ Safe (uses prepared statements in tests)
4. A04: Insecure Design - ✅ Good (structured schema, constraints)
5. A05: Security Misconfiguration - ⚠️ See below
6. A06: Vulnerable Components - ✅ Good (better-sqlite3 is well-maintained)
7. A07: Identification/Authentication - N/A
8. A08: Software/Data Integrity - ⚠️ See below
9. A09: Security Logging - N/A (no auth to log yet)
10. A10: SSRF - N/A

💡 Security Issues Found

A02: Cryptographic Failures ⚠️

      hash TEXT NOT NULL,

Design doc mentions SHA256 for change detection, but no validation that hashes use cryptographic functions. Should enforce in schema or docs.

A05: Security Misconfiguration ⚠️

  // Enable foreign keys
  db.exec("PRAGMA foreign_keys = ON");

Missing security-relevant PRAGMAs:

• PRAGMA trusted_schema = OFF - Prevents malicious views/triggers
• PRAGMA cell_size_check = ON - Detects corruption

A08: Software/Data Integrity ⚠️

  // Record schema version
  const version = 1;
  const now = Date.now();

  const existing = db
    .prepare("SELECT version FROM schema_version WHERE version = ?")
    .get(version);
  if (!existing) {
    db.prepare(
      "INSERT INTO schema_version (version, applied_at) VALUES (?, ?)",

No integrity checks. If database is tampered with, no detection mechanism. Consider adding:

• Schema checksums
• Version signature verification
• Corruption detection on load

Input Validation: ✅ Good

Tests show proper use of prepared statements, avoiding SQL injection.

🔭 Broader Context

This is infrastructure code, so security is primarily about data integrity and safe defaults. No immediate vulnerabilities, but missing defense-in-depth layers.

⚖️ Score: 7.5/10

No critical vulnerabilities, but missing security hardening.

---

6. Architectural Patterns & Design Decisions

🎯 Evaluating

Design decisions against project goals and best practices.

💡 Analysis

Decision 1: Bun-Specific Implementation

import { Database } from "bun:sqlite";

Reasoning (from design doc):

"Bun-native - better-sqlite3 synchronous API"

Critical Reflection: ❌

• better-sqlite3 is already a dependency
• Bun isn't mentioned in package.json engines
• Breaking CI is a red flag
• Design doc says "better-sqlite3" but code uses "bun:sqlite"

Verdict: Architectural mismatch. Should use better-sqlite3 for cross-runtime compatibility.

Decision 2: SQLite Over Alternatives

Rationale (from design doc):

• Zero infrastructure ✅
• TDD-friendly ✅
• Portable ✅
• Fast ✅

Verdict: ✅ Excellent choice for use case.

Decision 3: Hybrid Document + Structured Storage

  // Documents table - main frontmatter index
  db.exec(`
    CREATE TABLE IF NOT EXISTS documents (
      path TEXT PRIMARY KEY,
      type TEXT NOT NULL,
      frontmatter JSON,
      content TEXT,
      hash TEXT NOT NULL,
      indexed_at INTEGER NOT NULL,
      file_size INTEGER,
      modified_at INTEGER
    )
  `);

Verdict: ✅ Smart. Flexible JSON for varying schemas, structured tables for complex queries.

Decision 4: FTS5 with Triggers

  // Triggers to keep FTS5 in sync with documents table
  db.exec(`
    DROP TRIGGER IF EXISTS documents_fts_insert;
    CREATE TRIGGER documents_fts_insert AFTER INSERT ON documents BEGIN

Verdict: ✅ Excellent. Automatic sync prevents stale search results.

🪞 Critical Reflection

Strong architectural decisions overall, but critical error: Implementation uses bun:sqlite when design doc specifies better-sqlite3. This breaks cross-runtime compatibility and CI.

⚖️ Score: 6/10

Great design marred by implementation that contradicts design doc.

---

7. Documentation & Commit Quality

🎯 Checking

Documentation completeness, commit messages, inline comments.

💡 Analysis

Planning Documentation: Excellent ✅

• Comprehensive research summaries
• Clear phase breakdown
• Detailed database design
• Rationale for all decisions

Code Documentation: Weak ❌

#!/usr/bin/env bun
/**
 * SQLite Schema Creation
 * Creates the complete database schema for aidd indexing system
 */

import { Database } from "bun:sqlite";

/**
 * Creates the complete database schema
 * @param {Database} db - SQLite database instance
 */

Missing:

• Return type documentation
• Error conditions
• Usage examples
• Link to design doc

Commit Messages: Good ✅

82be9da Add SQLite schema foundation for document indexing
5bb8333 Apply prettier formatting to schema files

Follows conventional commits, clear descriptions.

Index Files: Broken ❌

# tools

This index provides an overview of the contents in this directory.

*This directory is empty.*

Says "empty" but contains 2 files. Pre-commit hook should have caught this.

🔭 Expansion

Per AGENTS.md:

The ai/**/index.md files are auto-generated from frontmatter. Do not create or edit these files manually—they will be overwritten by the pre-commit hook.

Issue: Pre-commit hook didn't update index, OR files are missing frontmatter.

Let me check:

#!/usr/bin/env bun
/**
 * SQLite Schema Creation
 * Creates the complete database schema for aidd indexing system
 */

❌ No frontmatter. Should have:

---
description: Creates complete SQLite schema for aidd indexing
---

⚖️ Score: 6/10

Excellent planning docs, weak code docs, broken index generation.

---

8. Missing TypeScript Definitions

🎯 Checking

Type safety and developer experience.

💡 Issue

Project has .d.ts files for other modules:

• utils/index.d.ts ✅
• src/server/index.d.ts ✅
• lib/index-generator.d.ts ✅

But missing:

• ai/tools/create-schema.d.ts ❌

Should export:

import type { Database } from 'better-sqlite3';

export interface SchemaResult {
  version: number;
  tablesCreated: boolean;
}

export function createSchema(db: Database): SchemaResult;

⚖️ Score: 5/10

Inconsistent with project patterns.

---

9. Comparison to Requirements

🎯 Checking

Implementation against Phase 1 success criteria.

From Epic (EPIC-rlm-database-jiron-skills.md):

Success Criteria:

• Schema creates all tables with indexes ✅ (implemented)
• Indexers handle 100+ files in <1 second ⚠️ (no indexer yet, just schema)
• FTS5 search finds relevant content ⚠️ (schema ready, no query tools)
• Fan-out search aggregates multiple strategies ❌ (not in this PR)
• All unit tests pass (>90% coverage) ❌ (tests fail in CI)

Verdict: Partial implementation. Schema foundation is solid, but missing query tools and failing CI.

---

📋 Summary & Actionable Recommendations

Critical Issues (Must Fix)

1. CI Failure - Bun/Node Compatibility 🚨

   • Problem: Uses bun:sqlite but CI runs Node.js
   • Fix: Use better-sqlite3 or add runtime detection
   • Priority: P0 - Blocks merge

2. Outdated Index Files 🚨

   • Problem: ai/tools/index.md says "empty" but contains files
   • Fix: Add frontmatter to .js files, run pre-commit hook
   • Priority: P0 - Breaks discoverability

High Priority (Should Fix)

3. Missing TypeScript Definitions

   • Problem: No .d.ts file for public API
   • Fix: Create create-schema.d.ts with proper types
   • Priority: P1 - Developer experience

4. Missing JSDoc

   • Problem: Public API lacks comprehensive documentation
   • Fix: Add complete JSDoc with examples
   • Priority: P1 - Code clarity

5. No Integration Tests

   • Problem: CLI path untested, no performance validation
   • Fix: Add integration tests, benchmark against goals
   • Priority: P1 - Quality assurance

Medium Priority (Good to Have)

6. Performance Optimizations

   • Wrap schema creation in transaction
   • Add PRAGMA journal_mode = WAL
   • Add PRAGMA synchronous = NORMAL

7. Security Hardening

   • Add PRAGMA trusted_schema = OFF
   • Add schema integrity checks
   • Document hash requirements

8. Code Quality

   • Remove unnecessary async in tests
   • Mock Date.now() for deterministic tests
   • Simplify file naming (schema.js vs create-schema.js)

---

⚖️ Overall Scores

Category	Score	Weight	Weighted
Structure	7/10	10%	0.7
Standards	8/10	15%	1.2
Tests	8.5/10	20%	1.7
Performance	7/10	10%	0.7
Security	7.5/10	15%	1.125
Architecture	6/10	20%	1.2
Documentation	6/10	10%	0.6

Overall: 7.2/10 - Good foundation with critical CI issue

---

🎯 Recommendation

DO NOT MERGE until CI passes. Fix the Bun/Node compatibility issue and update index files. The architectural foundation is solid, but implementation contradicts design doc and breaks CI.

Once CI is green, this is a strong foundation for Phase 1. The planning documentation is excellent and shows thoughtful design.

---

💬 Specific Code Changes Needed

Fix 1: Runtime Compatibility

// ai/tools/create-schema.js
let Database;
if (globalThis.Bun) {
  Database = (await import('bun:sqlite')).Database;
} else {
  Database = (await import('better-sqlite3')).default;
}

export function createSchema(db) {
  // ... rest of implementation
}

Fix 2: Add Frontmatter

---
description: Creates complete SQLite schema for aidd indexing system with FTS5, ProductManager tables, and dependency tracking
---
#!/usr/bin/env bun
/**
 * SQLite Schema Creation

Fix 3: Update Index (will auto-generate after adding frontmatter)

Run: npm run prepare (triggers pre-commit hook)

Fix 4: Add TypeScript Definitions

Create ai/tools/create-schema.d.ts:

import type { Database } from 'better-sqlite3';

export interface SchemaResult {
  version: number;
  tablesCreated: boolean;
}

export function createSchema(db: Database): SchemaResult;

---

This review follows the systematic process from review.mdc and identifies both the immediate CI blocker and the broader quality improvements needed for production readiness.