Subject
10 entries
Evaluation
Bookmarks
Awesome LLM Judges: curated research on LLM evaluation systems
Curated collection of research on LLM-as-a-judge evaluation systems — covering ensemble methods, fine-tuned judge models, hallucination detection, generative reward models, and safety moderation. Maintained by Haize Labs alongside their Verdict library.
Braintrust Data: LLM Evaluation Platform
Braintrust is an AI evaluation platform for logging, scoring, and comparing LLM outputs across experiments and prompt versions. Addresses the core LLMOps problem: how do you know if your prompt change made things better or worse?
Comparing RAG Copilots: OpenAI, Anthropic, Perplexity, and More
Graphlit's comparison of RAG-powered copilots from OpenAI, Anthropic, Perplexity, and others against unstructured data retrieval tasks. Useful late-2023 benchmark of the RAG quality landscape across providers before the tooling matured.
LlamaIndex Retrieval and Chunk Evaluation Notebook
A LlamaIndex Google Colab notebook for evaluating retrieval quality and chunk size in RAG pipelines — demonstrating how to measure retrieval hit rate and MRR across different chunk sizes. Practical tooling for the underappreciated problem of RAG evaluation.
Reasoning or Reciting? LLM Capabilities via Counterfactual Tasks
An arxiv paper testing whether LLMs are reasoning or reciting — using counterfactual tasks (modified versions of standard benchmarks) to distinguish genuine reasoning from pattern-matching on training data. Finds LLMs show sharp performance drops on counterfactual variants, suggesting heavy reliance on memorization.
Beware of Unreliable Data in Model Evaluation
Cleanlab's case study showing that noisy test data leads to suboptimal prompt selection for LLMs — you can choose the wrong prompt because your evaluation data contains labeling errors. A practical warning about data quality in LLM evaluation pipelines.
Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
Chatbot Arena is LMSYS's crowdsourced LLM benchmark where users rate anonymous head-to-head model comparisons using Elo ratings — producing rankings that reflect real user preferences rather than task-specific metrics. The Elo format became the dominant live benchmark for tracking which frontier model is currently best.
Large Language Models are Human-Level [...] (Nov 2022)
A November 2022 paper claiming LLMs perform at human level on some capability, with a truncated filename that doesn't reveal the specific domain. Likely from the computational social science or NLP annotation literature; the save date coincides with ChatGPT's launch and peak interest in LLM capability claims.
An Open Source AutoML Benchmark
This paper presents an open-source, extensible benchmark for comparing AutoML systems across 39 classification datasets, finding that no single system consistently dominates a tuned random forest baseline. It establishes best practices for fair AutoML evaluation and provides a living framework that accepts community contributions.
The ROC Curve Explained
A visual explanation of the Receiver Operating Characteristic (ROC) curve and AUC for binary classifier evaluation. One of the clearest introductions to the concept for practitioners learning to assess model performance beyond accuracy.