feat: Add OpenRAG Workbench evaluation framework#1213
Open
matanor wants to merge 34 commits intolangflow-ai:mainfrom
Open
feat: Add OpenRAG Workbench evaluation framework#1213matanor wants to merge 34 commits intolangflow-ai:mainfrom
matanor wants to merge 34 commits intolangflow-ai:mainfrom
Conversation
- Add FilenameExistsResponse model to SDK - Add filename_exists() async method to DocumentsClient - Add GET /v1/documents/check-filename endpoint with API key auth - Export FilenameExistsResponse in SDK __init__.py This enables SDK users to check if a file exists in the knowledge base before ingestion, avoiding duplicate uploads.
…to sdk_for_ragworkbench
- Add inference and ingest pipelines - Add create_boards script and utility modules (utils, logging_config) - Add .env.example for configuration - add pyproject.toml
- Add shared conftest.py with environment and logging fixtures - Enhance inference test with explicit cache hit/miss validation - Update pytest configuration with pythonpath and strict markers
- Add boards module with table_rich board configuration - Rename create_boards.py to evaluate.py for clarity - Enhance .gitignore with additional patterns - Remove unused imports from pipelines - Update utility functions
github-actions
bot
added
backend
github-actions
bot
added
enhancement
define a single configuration with multiple options update acreens to use metric names
github-actions
bot
added
enhancement
github-actions
bot
added
enhancement
github-actions
bot
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the OpenRAG Workbench evaluation framework to the repository, providing a comprehensive solution for evaluating RAG (Retrieval-Augmented Generation) systems using the OpenRAG SDK.
Prerequisites
Requires the changes to the SDK from #1086
Changes Made
New Components Added
evaluation/openrag_workbench/)Key Files
Pipeline Implementations
pipelines/ingest.py- Document ingestion pipeline (265 lines)pipelines/inference.py- Inference pipeline for RAG queries (254 lines)Evaluation Entry Point
evaluate.py- Main evaluation scriptConfiguration & Utilities
boards/table_rich/board.yaml- Board configuration for results displaylogging_config.py- Logging configurationutils.py- Utility functionsTesting
tests/pipelines/test_ingest.py- Ingestion pipeline tests (158 lines)tests/pipelines/test_inference.py- Inference pipeline tests (232 lines)tests/conftest.py- Test fixtures and configurationProject Configuration
pyproject.toml- Project dependencies and configurationuv.lock- Locked dependencies (3121 lines).env.example- Environment variable template.gitignore- Git ignore patternsREADME.md- DocumentationStatistics
Additional Notes
This implementation provides a production-ready evaluation framework that integrates seamlessly with the OpenRAG SDK, enabling comprehensive testing and benchmarking of RAG systems.