Subject
3 entries
Benchmark
Bookmarks
Benchmarking Postgres vector search: pgvector vs Lantern
Tembo's benchmark comparing pgvector and Lantern for vector similarity search in PostgreSQL — tests query speed, indexing time, and recall across different dataset sizes. Practical data for choosing a Postgres vector extension.
On the Paradox of Learning to Reason from Data
Zhang, Li, Meng, Chang, and Van den Broeck (UCLA) show that BERT achieves near-perfect accuracy on in-distribution logical reasoning problems while completely failing to generalize to other distributions over the same problem space. The explanation: BERT learned statistical features of the logical reasoning distribution, not the underlying reasoning function — a fundamental distinction between benchmark performance and genuine reasoning.
An Open Source AutoML Benchmark
This paper presents an open-source, extensible benchmark for comparing AutoML systems across 39 classification datasets, finding that no single system consistently dominates a tuned random forest baseline. It establishes best practices for fair AutoML evaluation and provides a living framework that accepts community contributions.