[Proposal] We built a Ragas Eval provider for Llama Stack. We would like to contribute it to the Ragas community. #2339
Description
Hello Ragas community! 👋 Diego here from Red Hat's TrustyAI team.
We built a Ragas Eval provider for Llama Stack. Would this be of interest to the Ragas community?
What is this?
This provider allows you to run all of Ragas' powerful evaluation metrics via Llama Stack's standardized evaluation API. It comes in two flavors:
- Inline: Runs evaluations in the same process as your Llama Stack server
- Remote: Leverages Kubeflow Pipelines for distributed evaluation workloads
Why did we do it?
We are adopting Llama Stack as the opinionated organization of GenAI APIs for inference, guardrailing, evaluation, etc. This integration allows us to bring Ragas into it:
- Unified Interface: Access Ragas evaluations alongside other Llama Stack components (inference, embeddings, datasets)
- Scalability: The remote provider enables running Ragas evaluations in Kubernetes environments via Kubeflow Pipelines
Current state
The provider is functional and includes support for both local and remote / KFP execution. It wraps the inference API provided by Llama Stack so that generations and embeddings required by Ragas are routed to the Llama Stack server.
Proposal / Discussion
We'd love for this provider to become part of Ragas - perhaps under an integrations repo or within a subfolder? That way it can grow alongside Ragas. Perhaps we can separate the Llama Stack piece from the Kubernetes piece so folks can also use the Kubeflow pipeline by itself and make Llama Stack optional?
Would love to hear your thoughts and thank you for Ragas!
Current Repository: https://github.com/trustyai-explainability/llama-stack-provider-ragas
PyPI: https://pypi.org/project/llama-stack-provider-ragas/