The RAG reality check: New open-source framework lets enterprises scientifically measure AI performance
1 min read
Summary
Vectara, in conjunction with the University of Waterloo, has launched an open-source framework that assesses and benchmarks retrieval-augmented generation (RAG) systems.
RAG systems are increasingly popular in the enterprise sector as multi-step agentic systems, but they are difficult to evaluate objectively.
The Open RAG Eval framework uses automatic grading of “nuggets” of information — basic factual elements of responses — to allow organisations to test any RAG pipeline.
Four metrics are used to test the systems: hallucination detection, citation, auto nugget and UMBRELA ( Unifiied Method for Benchmarking Retrieval Evaluation with LLM Assessment)
TheOpen RAG Eval framework is available to download at https://github.