A lot of teams have a hard time measuring their RAG apps. LLMs and techniques for vector search have come a long way, but they still hallucinate, or generate incorrect information. And those out-of-the-box solution architectures still can’t address every pitfall of your specific use case. 

As a developer, it’s tough to figure out the best way to solve these problems for your specific needs. And there is no shortage of LinkedIn posts about the next revolutionary chunking strategy that your team must use or else, fall behind.

Thankfully, evaluating Retrieval Augmented Generation (RAG) has also come a long way. So you don’t have to go to production entirely on the anecdotal evidence from your dev and QA teams. Instead, you can adopt a metric-driven development approach. A metrics-driven approach is all about measuring, not guessing. When you measure performance, you improve it—no more wasting time on solutions that don’t make a difference or cause setbacks. 

We’ll cover how to get started by establishing a set of baseline metrics. We’ll also use the friendly and pragmatic RAG Assessment (Ragas) framework to reason more specifically about the performance of our GenAI apps. 

Let’s start with a simple RAG app.

Here’s a quick example of a simple RAG app using LangChain, Redis, and OpenAI to answer questions about financial documents. We’re using Nike’s 2023 10-K document as our contextual data, but feel free to tailor it to your own use case. The complete code example is available within our AI resources repo. 

Split and load the doc

Create vector embeddings for the chunks and store in Redis as the vector store

Define the LLM and PromptTemplate

Create RAG question and answer chain

Test it out

Sample Output

Let’s evaluate our RAG app

The Ragas framework consists of four primary metrics: faithfulness, answer relevancy, context precision, and context recall. Context precision and recall measure how well the app retrieves data from the vector store, while faithfulness and answer relevance quantify how accurately the system generates results from that data. Together, these metrics give you a complete view of how your app is really performing.

To calculate these metrics, we need to collect four pieces of information from our RAG interactions:

• The question that was asked
• The answer that was generated
• The context that was provided to the LLM to generate the answer
• And, depending on which metrics you’re interested in, a ground-truth answer determined either by a critic LLM or a human-in-the-loop process. In our case, it’s only context recall that uses the ground truth labels. 

First test

Question: Where is Nike headquartered and when was it founded? 

Ground truth: Nike is headquartered in Beaverton, Oregon and was founded in 1964.

Results of our test

Each metric shows a different flavor of quality

Let’s start with the metrics that look promising.

Answer relevancy

Answer relevancy is calculated under the hood by asking an LLM to generate hypothetical questions based on the answer returned, and then taking the average cosine similarity between those generated questions. 

A high score means there’s not much variation in how the answer could be determined. It makes sense intuitively for our example that this score is high since it’s fairly obvious what sort of questions lead to the answer, “Nike is headquartered in Beaverton, Oregon and was founded in 1967.” But a low score? That gives us an indication of a vague answer that isn’t necessarily related to what was asked.

Context precision

Next, the context precision for our question/answer pair was 1.0. Context precision measures how *good* the returned context was and is defined as:

A true positive is a document that is relevant and was returned in the result set and a false positive is a document that was not relevant and was returned in the result set. 

In this case, the evaluation showed that all the docs returned were relevant to the ground truth provided. This is good but does require a bit of faith in the LLM’s ability to determine what is relevant, and that’s a whole other topic on its own. I recommend reading the full paper for those interested in gaining more insight on this front.

Faithfulness

Moving to the metrics that were less promising, faithfulness is defined‌ as:

For our example, there are two claims that can be determined from the answer: “Nike is headquartered in Beaverton, Oregon and was founded in 1967.”

1. Nike is headquartered in Beaverton, Oregon.

2. Nike was founded in 1967.

The context doesn’t mention Nike being in Beaverton, Oregon, so that claim can’t be inferred from the text.

But the claim that Nike was founded in 1967 can be inferred from the context, since the doc specifically mentions Nike being incorporated in 1967. This result highlights an important point about faithfulness—it doesn’t measure accuracy. What’s interesting here is that the claim about Beaverton (Nike is located in Beaverton), though factually correct, couldn’t be pulled from the context.

On the flip side, the claim about Nike being founded in 1967 is incorrect but can be inferred from the text.

Faithfulness measures how true to the text an answer was. It doesn’t tell us if the answer was correct or not.

Context recall

Accuracy can be understood from context recall, which is defined as:

Context recall is the only metric of the four that utilizes the ground truth data.

The ground truth we provided for this example was `Nike is headquartered in Beaverton, Oregon and was founded in 1964` which can be broken down into two sentences/claims:

1. Nike is headquartered in Beaverton.

2. Nike was founded in 1964.

Neither of these claims can be inferred correctly from the context; therefore, context recall is 0/2 or 0.

The first example question provided here is intentionally general and meant to bring up an important point about RAG: RAG is an architecture designed to answer specific questions about a context. It is not necessarily ideal for answering general questions—that is what an LLM is for. 

The question “Where is Nike located and when was it founded?” is a general knowledge question that isn’t specific to the 10-K document we loaded into our context. When designing a test and educating users about how to best interact with a RAG app, it’s important to emphasize what type of questions are meant to be answered by the app. 

This is also why an agent layer can be essential to chat experience because general questions should be handled by a general language model, while specific contextual questions should be handled by RAG, and a layer to determine the difference can greatly improve performance.

Let’s ask a different question

Results

Our second analysis gets better results

For this test, we saw better Ragas scores, largely because the question is well-suited for our RAG app.

– The question directly connects to the context.

– It uses specific terms that help with matching in the vector space.

– The ground truth is similar to the doc content.

With RAG, the question format really matters, just like using the right keywords in a Google search. Since we’re using math to process natural language, we have to be mindful of interacting with the system in a way that lends itself to that paradigm. 

Coincidentally, this is why query rewriting in your apps can be really powerful.You’re making conversions that are obvious to humans but not to machines, and it can really improve performance. Plus, now you have the tools to test it yourself.

We’ll evaluate our RAG app using a test dataset

Now that we have an understanding of the metrics in play and a better idea of what they tell us about our app, the next question becomes: How do we go about creating a dataset to test our specific app? This is where the Ragas library really shines. 

Ragas is designed to be ‘reference-free’ and gives us a helper class for auto-generating a test set. In fact, the second example question was generated this way. It’s worth noting that generating a synthetic dataset is not a replacement for collecting user data or labeling your own set of test questions with ground truth; but, it can be a very effective baseline for getting an initial sense of app performance when a polished test set is not yet available or feasible.

In the initial paper proposing Ragas, a pairwise comparison between human annotators and the Ragas approach found that the two were in agreement 95%, 78%, and 70% of the time, respectively, for faithfulness, answer relevance, and contextual relevance. Note: This was research done on the WikiEval dataset, which is probably one of the easier datasets for LLMs. Even so, it shows that Ragas is a solid and reliable first step.

There’s no special trick to creating a test set. All you need is a set of questions labeled with ground truth answers either by you or your favorite model. An hour of thought and labeling effort is a valuable exercise and could even be used as an example to an LLM for the type of questions you expect and want your app to be tested with.

Code to generate a test set with the Ragas library:

Note: Depending on which model you use and your personal/company limits, it isn’t uncommon to hit rate limits when generating a test set. If this happens, don’t be afraid to try smaller models or generate questions in batches. 

Running the test set generation process will output something like this:

It’s important to go through each question and the ground-truth answers carefully. Although the LLM generally does a good job of coming up with questions and answering them, it can definitely miss the mark sometimes. If that happens, don’t worry. Just check your source data, try answering the question yourself, and update the value. The test set generator class helps us create a solid test set but it doesn’t have to be where we stop, and the more care you put into your test set the better your results will be. 

Let’s run it again with more questions this time

The above code was used to generate a test set with 15 questions to evaluate the basic RAG app. The results are shown in the table below.

The performance of our RAG in this case is mediocre. While there are no exact target ranges for these values, you should definitely be concerned to see numbers below 0.5 as a rule of thumb. 

On the other hand,if you’re seeing perfect scores across the board, it might be worth double-checking if your test set is challenging enough. Values between 0.75-0.95 are solid, but whether you need to optimize further depends on your app’s purpose. For example, having near perfect faithfulness might be great for fact retrieval, but it could make  for a chat experience that’s not as fluid, or conversational. 

What’s great  about this approach is that while writing this blog, I quickly ran the same tests with a few different chunk sizes to see how they compared and found that 2500 produced the best overall results. 

Without taking a metrics-driven approach, it would be really hard for me to gain any idea of how the changes were affecting my system. This little study also leads me to realize that optimizing chunk size alone doesn’t have a giant effect on my app performance overall. This is critical. One of the biggest challenges of every engineering team is knowing what to prioritize. A system of evaluation helps us figure out what’s important much quicker than going on a hunch. 

Output of alternative chunk sizes

Wrapping up

In this blog, we covered:

1. How to build a simple RAG app, generate a test set, and get started with the Ragas framework.
2. How metrics-driven development can help improve our apps by zeroing in on the optimizations that matter most.
3. How to use Ragas for offline evaluations to track regression between RAG app versions.
4. Design considerations around what type of questions work best for RAG.
5. Simple guidelines and rules of thumb to think about when making sense of your metric results.

For a full Ragas example plus more AI recipes from the team at Redis check out our AI resources repo.