---
title: "Which enterprise RAG benchmarks predict production quality"
description: "No single benchmark predicts production. A field guide to enterprise RAG benchmarks, their blind spots, and the evidence ladder to demand before you buy."
canonical: https://annamlabs.com/blog/enterprise-rag-benchmark-landscape-guide
---


# Which enterprise RAG benchmarks predict production quality

No single benchmark predicts production. A field guide to enterprise RAG benchmarks, their blind spots, and the evidence ladder to demand before you buy.

*Võ Ngọc Quang Minh · October 31, 2026 · 16 min read*







## Four kinds of evaluation, four kinds of failure







## Start from the production question












## The landscape at a glance



| EnterpriseRAG-Bench | 2026 | End-to-end enterprise QA over multi-tool corpora | ~500,000 synthetic docs (Slack, Gmail, Linear, Drive and more); 500 questions | Yes — questions, gold answers and facts are published | One simulated company; a public set invites tuning; self-selected submissions |
| --- | --- | --- | --- | --- | --- |
| HERB | 2025 | End-to-end enterprise QA with scattered, people-centric evidence | 39,190 artifacts; 1,514 questions (815 answerable, 699 not) | Code and evaluation repository released | Simulated organization; scores not comparable to other benchmarks |
| FinanceBench | 2023 | Numeric extraction from real financial filings | 10,231 defined triplets; 150-question public sample | Partial — full set licensed | Single domain; the 81% failure headline describes a 2023 model, not current ones |
| LegalBench-RAG | 2024 | Span-level retrieval precision for citations | 6,858 QA pairs over 79M+ characters | Yes | Retrieval only; says nothing about the generator |
| CRAG | 2024 | Freshness, long-tail facts, abstention | 4,409 QA pairs with mock web and knowledge-graph APIs | Yes | Public-web facts, not internal documents |
| RAGBench | 2024 | Evaluator quality via the TRACe framework | ~100,000 examples from 12 datasets | Yes | Measures judges, not retrievers or generators; heterogeneous sources |
| FRAMES | 2024 | Multi-hop chaining across many documents | 824 questions, 2-15 Wikipedia articles each | Yes | Wikipedia memorization inflates the no-retrieval baseline |
| MultiHop-RAG | 2024 | Multi-document inference and abstention | 2,556 queries over 609 news articles; ~88% answerable | Yes | Small, static news corpus |
| BEIR | 2021 | Zero-shot retrieval ranking | 18 datasets | Yes | Lexical bias inherited from how some datasets were built |
| MTEB / MMTEB | 2023-2025 | Embedding model shortlisting | 500+ tasks, 250+ languages | Yes | A weak predictor of enterprise retrieval; suite versions change the task mix |
| KARLBench | 2026 | Agent search policy | Six search regimes | Partial — some components not independently reproducible | Vector-tool-only agents measure policy, not retrieval stacks; vendor-built |
| BrowseComp-Plus | 2025 | Deep agentic search over a frozen open-web corpus | 830 questions; 100,195 documents | Yes | Open web: no permissions, no internal corpora |



## What every synthetic benchmark has in common








## The evidence ladder








## What no public benchmark measures









## Five questions before you trust a vendor score







---

Read the HTML version with visuals: https://annamlabs.com/blog/enterprise-rag-benchmark-landscape-guide
More articles: https://annamlabs.com/insights