Add app-development benchmark: task function, classifiers, and sample metrics

[mongodben]

Jira: https://jira.mongodb.org/browse/EAI-1614

Changes

• Implement makeGenerateAppResponseTask — the main task function that orchestrates the app-development eval pipeline:
  1. generate response
  2. classify stack
  3. self-reflect
  4. analyze database choice
• Add sampleSize param to support running multiple samples per eval case for model non-determinism
• Add pass@k, pass%k, and pass^k sample metrics to mongodb-rag-core/eval for computing success probabilities across samples
• Update AppDevelopmentTaskOutput to use { samples: AppDevelopmentSample[] } structure
• Update PrimaryDatabaseIsMongoDb and MentionsMongoDbInGeneration scorers to return all three pass*k metrics per dimension

Notes

• pass@k, pass%k, pass^k live in mongodb-rag-core/eval/sampleMetrics so they're across benchmark packages
• When sampleSize=1 (default), all three metrics collapse to the same value — no behavioral change for existing usage
• Task function runs steps 2 (classify) and 3 (self-reflect) in parallel, then step 4 (analyze) sequentially since it depends on the classified database

Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
ai-benchmarks	Error		Mar 18, 2026 3:12pm