Subject
5 entries
Benchmarks
Bookmarks
Claude Code Performance Tracker: daily SWE-Bench monitoring
MarginLab's Claude Code Performance Tracker monitors daily SWE-Bench-Pro scores with statistical significance testing to detect degradation. Answers whether Claude Code's coding performance is consistent over time.
Reasoning or Reciting? LLM Capabilities via Counterfactual Tasks
An arxiv paper testing whether LLMs are reasoning or reciting — using counterfactual tasks (modified versions of standard benchmarks) to distinguish genuine reasoning from pattern-matching on training data. Finds LLMs show sharp performance drops on counterfactual variants, suggesting heavy reliance on memorization.
Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
Chatbot Arena is LMSYS's crowdsourced LLM benchmark where users rate anonymous head-to-head model comparisons using Elo ratings — producing rankings that reflect real user preferences rather than task-specific metrics. The Elo format became the dominant live benchmark for tracking which frontier model is currently best.
On the Measure of Intelligence
François Chollet's foundational 2019 paper arguing that measuring AI by task skill conflates intelligence with prior knowledge, proposing instead a definition of intelligence as skill-acquisition efficiency — and introducing the ARC benchmark designed to test it. One of the most influential theoretical framings of what AGI evaluation should actually measure.
Semantic Textual Similarity — Papers with Code Benchmark
Papers with Code's benchmark page for Semantic Textual Similarity — a task measuring how similar two sentences are in meaning, scored against human judgments. The canonical reference for comparing embedding models on this fundamental NLP task.