From the team behind Gatsby, Mastra is a framework for building AI-powered applications and agents with a modern TypeScript stack.

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

AI observability platform for production LLM and agent systems.

Evaluation and Tracking for LLM Experiments and AI Agents

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

OpenSource Production ready Customer service with built in Evals and monitoring

Evaluate your LLM-powered apps with TypeScript

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

Harbor is a framework for running agent evaluations and creating and using RL environments.

Cloud-synced dashboards for OpenCode and Claude Code. Track sessions, search with semantic lookup, export eval datasets.

[NeurIPS 2024] Official code for HourVideo: 1-Hour Video Language Understanding

Test Generation for Prompts

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

Evalica, your favourite evaluation toolkit

Run coding agents against each other. Merge the winner.

An MCP Evaluation Library

Benchmarking Large Language Models for FHIR