Open-source Python evaluation library for LLM and RAG applications. It runs LLM-based and deterministic metrics over datasets, generates synthetic test sets, and tracks results across experiments so changes can be compared. It is a local library with no hosted control plane, access control or audit trail.
open_source · generally available · Research snapshot 2026-09-06
Show me a Ragas experiment comparing two retrieval configurations on a generated test set, with per-metric scores and reasons.
Capabilities and evidence
Support labels reflect the supplied research. Documentation and vendor claims are not independent product tests. “Not established” means the researcher did not find support; it does not prove a capability is absent.
Documented by provider
Ragas documents an experiments-first workflow combining LLM-driven metrics, custom metrics defined with decorators, built-in dataset management and result tracking for LLM applications.
Limit: Docs do not state that spans, tool calls or multi-agent handoffs are captured; scope is dataset-level scoring.
The repository states Ragas automatically creates test datasets covering a range of scenarios and supports production-aligned test set generation, with objective LLM-based and traditional metrics.
Limit: No published validation of generated test-set representativeness.
The canonical repository resolves to the vibrantlabsai organization and the library is installed from PyPI or source; no enterprise tier or hosted deployment is documented.
Limit: License text and any commercial offering were not established from the pages fetched.