Evaluate your LLM's response with Prometheus and GPT4 💯
-
Updated
Apr 25, 2025 - Python
#
llm-as-evaluator
Here are 14 public repositories matching this topic...
Deliver safe & effective language models
nlp artificial-intelligence benchmarks benchmark-framework model-assessment ai-safety mlops responsible-ai ml-safety trustworthy-ai ethics-in-ai ml-testing large-language-models llm ai-testing llm-test llm-evaluation-toolkit llm-as-evaluator llm-testing
-
Updated
Oct 25, 2025 - Python
[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
benchmark regex reliability evaluation dataset gpt phi large-language-models llm cc-by-nc-nd-4 open-compass chatglm qwen lm-evaluation llm-as-a-judge llm-as-evaluator xfinder reliable-evaluation key-answer-extraction judge-model
-
Updated
Feb 26, 2025 - Python
This is the repo for the survey of Bias and Fairness in IR with LLMs.
information-retrieval recommender-systems bias ir fairness large-language-models llm chatgpt llm4rec llm4rs llm-as-a-judge llm-as-evaluator llm4ir
-
Updated
Sep 4, 2025
Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)
temporal-reasoning sota-model llms rlhf rlaif llm-as-a-judge llm-as-evaluator self-critic-framework colm2024
-
Updated
Oct 23, 2024 - Python
Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"
nlp evaluation bias bias-detection llm llms llm-evaluation llms-benchmarking llm-as-judge llm-as-a-judge llm-as-evaluator
-
Updated
Feb 16, 2024 - Jupyter Notebook
LLM-as-judge evals as Semantic Kernel Plugins
-
Updated
Aug 22, 2025 - C#
Use groq for evaluations
-
Updated
Jul 11, 2024 - Python
PrometheusLLM is a unique transformer architecture inspired by dignity and recursion. This project aims to explore new frontiers in AI research and welcomes contributions from the community. 🐙🌟
deep-learning mcp evaluation pipelines tracing language-model self-organization cognitive-architecture hermeneutics philosophy-of-mind gpt4 llm llmops ollama litellm llm-as-evaluator autopoietic-systems prompt-logging
-
Updated
Oct 30, 2025 - Python
Evaluation system for computer-use agents that uses LLMs to assess agent performance on web browsing and interaction tasks. This judge system reads screenshots, agent trajectories, and final results to provide detailed scoring and feedback.
computer-vision evaluation evaluation-metrics performance-testing evaluation-framework web-agent llm-as-judge llm-as-a-judge llm-as-evaluator computer-use
-
Updated
Oct 28, 2025 - Python
A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.
-
Updated
Sep 29, 2025 - Python
Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)
-
Updated
Jun 13, 2024 - Python
Automated evaluation of llm generated responses on aws
-
Updated
Apr 11, 2025 - Python
project from UDACITY second module 'agentic workflow' as part of Agentic AI Nanodegree
-
Updated
Aug 16, 2025 - Python
Improve this page
Add a description, image, and links to the llm-as-evaluator topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-as-evaluator topic, visit your repo's landing page and select "manage topics."