How to evaluate AI applications

This month, NIST introduced its AI Risk Management Framework. One of the key activities highlighted in the framework is TEVV (Test, Evaluation, Verification, and Validation) — tasks that are performed throughout the AI lifecycle to measure and govern risk from non-deterministic AI systems. This is becoming increasingly relevant as many enterprises are looking to take their prototypes to production this year, while others with production AI applications are facing increased exposure to risk (eg: last week’s prompt injection attack on DPD’s support chatbot).

‍

NIST's AI Risk Management Framework, highlighting testing and evaluation (TEVV) across the entire application lifecycle.

‍

To mitigate risk and ensure applications are ready for production scale, testing and evaluation of model responses is required at every step of the application lifecycle, both during development and in production. Greg Brockman and others seem to agree. Well-thought evaluations can help you maximize LLM performance, validate the robustness of your application before deploying to production, and set up industrial-grade evals and alerts to monitor any issues in production.

‍

What to test for

‍

There is no right answer here – the specific traits that you choose to test in your evaluations are highly dependent on your use-case and what characteristics you are looking to test. We recommend developers to critically think about what their app is trying to achieve, in what manner, and where users are running into issues. It’s helpful to start by manually reviewing some user sessions to understand what issues users run into, accordingly set up some tests, and add more tests as you find more patterns of problematic behavior.

‍

Here are some common types of evaluators we’ve seen across our customers:

‍

Response Quality: Based on your use-case, these evaluators may involve looking at traits like conciseness or coherence, checking for specific keywords or phrases in model responses (eg: prevent using “As an AI language model”), etc.
Context Quality: RAG evaluators such as context relevance, correctness, and faithfulness are helpful to evaluate the quality of your retrieval pipeline and detect hallucinations or factual inaccuracy across model responses.
Safety: Safety evaluators allow you to red-team your model and validate the safety of your model generated responses. A few evaluators to consider include toxicity (model-graded or binary classifiers), PII detection (model-graded, NER-based, or using Python packages like piicatcher), assertions for specific keywords like swearing, etc.
Format Validation: Some use-cases such as data extraction or code generation require the model to output responses in structured formats like JSON or SQL queries. It’s important to be able to validate whether a model is correctly outputting the structured text. Libraries like json, jsonschema, and sqlglot can help you validate output formats easily.
Agent Trajectory or Reasoning: This is an emerging class of evaluators, typically used to evaluate the performance of agents on complex reasoning tasks. This often involves passing in the full model trace to an LLM like GPT-4 or LMM like LLaVa, along with relevant context from the user query to evaluate the agent’s trajectory and its effectiveness at completing the task at hand.

‍

Setting up an evaluation harness

‍

To evaluate the performance of your app, it’s important to be able to set up the right evaluators and datasets to test over, which is also referred to as an evaluation harness. This will allow you to repeatably evaluate performance over a dataset as you iterate and try different prompts, models, or app architectures (eg: KNN-based RAG vs multi-doc retrieval, etc.). Setting up a harness is similar to setting up an automated CI testing script in traditional software development, with the difference being constantly evolving datasets and fine-grained evaluators to probe and test the application’s robustness.

‍

It all starts from a “golden” evaluation dataset

‍

To be able to scalably evaluate your candidate models, you need to build a “golden dataset” of prompt/response pairs that are representative of real world usage from your application. You may choose to include adversarial test cases if you’re looking to test models on specific traits or characteristics (eg: reasoning ability, unsafe behavior, writing styles, etc.). If you log sufficient data and feedback from users in production, you can simply filter for responses users strongly prefer and strongly dislike to create a diverse dataset of test cases. At this stage, it’s important to have annotators validate and correct the outputs that are being used as ground truth since your evaluation system will ultimately rely on these examples as the gold standard.

‍

In case you don't have any logs to evaluate over, we recommend generating a synthetic dataset of inputs by using GPT-4 in a few-shot setting (i.e. by providing the model a few examples of expected inputs).

‍

Within HoneyHive, this process is straightforward — you can quickly filter your logged data by custom heuristics and create a test dataset within the platform. You can also upload your evaluation datasets via the UI or SDK.

‍

Adding ground truth: Next step in the process is adding ground truth labels, or corrections - i.e. what the expected response from the model should be. This allows you to correct any mistakes made by the model and track expected values from your model for continuous testing. In case you are unable to manually create ground truth, you can leverage GPT-4 or Claude 2.1 to generate a relative benchmark for your open source models and use the model’s outputs as ground truth for each datapoint. This approach, albeit a bit noisy and not as robust, is relatively fast, cheap, and easy to try out.

‍

HoneyHive makes this easy within the app — you can invite domain experts to add ground truth labels to a dataset curated in the platform.

‍

Lack of ground truth: In many cases, it may be difficult to collect ground truth outputs for your dataset. In the absence of ground truth labels, for many use-cases like simple RAG, writing assistants, etc, you could alternatively create a list of inputs to pass to your app and use GPT-4 as a judge to evaluate the quality of your responses.

‍

The key is selecting the right evaluators

‍

Evaluators allow you to scale your evaluation process and benchmark different models against a quantitative measure of quality and performance. Selecting the right evaluators is a critical step in being able to accurately evaluate your candidate models.

‍

Typically, we see most teams relying on humans to evaluate the quality of model outputs. Human evaluation can be a slow, highly manual, and cost-prohibitive process, which is why we recommend pairing it with programmatic approaches.

‍

Here are the different types of model evaluators you should consider using:

‍

Python Evaluators: Configuring unit tests like keyword assertions can help you evaluate the “brand voice” of your models, or any specific syntax you’re interested in (eg: ensure the model does not say “As a large language model..”). In many cases, we’ve seen several examples ranging from simple metrics like semantic similarity (between ground truth and model responses) to running code in containerized environments to check for code validity. The metric you choose here really depends on your use-case and its specific requirements.
Model-graded Evaluators: You can alternatively use LLMs like GPT-4 or emerging open-source alternatives like Prometheus to evaluate the quality of your model responses. These evaluators are typically used to evaluate more subjective traits like context relevance, answer faithfulness, coherence, etc. for use-cases with open-ended input space like RAG, writing assistants, etc. While this approach is quickly emerging as a scalable solution to model evaluation in the absence of manual review and labeling, it is important to be aware of certain biases in this approach such as positional bias or lack of consistency across models, which is why it’s important to critically think about the prompt, grading scale, etc. being used to mitigate bias as much as possible. We recommend using model-graded evaluators as a useful heuristic but still rely on subjective human judgment as much as possible. Model-graded evaluation is an area we’re actively studying and will be publishing research around in the near future.
Statistical NLP Evaluators: Some teams may choose to evaluate model performance using NLP metrics like ROUGE, BLEU, MAUVE, etc. for traditional tasks like extractive summarization. That said, it’s important to note that many of these metrics like ROUGE have proven to not correlate well with human feedback, i.e. a better ROUGE score may not necessarily indicate a better summary. We generally recommend teams to avoid this category of metrics for most abstract, generative tasks. They serve well as a sanity check at best.
Human Evaluators: While automated evaluation methods are increasingly getting better, manual evaluation still reigns supreme. If evaluating offline, depending on your budget and how frequently you wish to test, it might make sense to manually review the results from automated evaluations (using the above 3 types of evaluators) to validate performance and correlation with human feedback. If you’re looking to evaluate production logs, it’s generally a good idea to capture implicit and explicit feedback from users to further use for fine-tuning or improving your evaluator. We typically see 3 types of feedback - Actions, Ratings, and Issues (free form text or pre-selected categories). Actions are implicit in nature and generally depend on your app’s UI. Some common examples of positive feedback include saving or sharing a chat conversation. Ratings are explicitly provided by users in the form of thumbs-up/down or on a likert scale (1-5). Issues are a chance for users to provide explicit feedback in free form that can be helpful to debug specific sessions.

‍

When to set up evaluations

‍

Start early, ideally at the prototyping stage

‍

You should start evaluating your application as early as the prototyping stage in the Playground. This will allow you to get a better sense of where the model fails, and track problematic behavior for future iterations as you improve and try to prevent regressions. A systematic approach to testing from the beginning can help you iterate and get to a production-ready stage 10x faster. Human evaluation can serve as a good starting point, though it’s important to be aware that human evals won’t scale as you get to the POC or validation stage.

‍

Set up an offline eval harness before a beta rollout

‍

Once you’ve built an initial prototype that works and are ready to roll out the application to a broader group of beta users, it’s important to set up both an offline evaluation harness and a way to capture and analyze human feedback from users using your application. Human feedback from your users will allow you to quickly debug issues, while having a harness will allow you to quickly iterate based on your diagnosis and check for regressions before rolling out updates.

‍

Set up online evals and CI tests before a production rollout

‍

Once you’re past the beta stage and have a much better understanding of the kinds of issues you see users running into, it’s time to evolve your test suite and set up more robust online evaluators to monitor issues in production. We highly recommend setting up evaluators to detect toxicity, PII leakage, prompt injection attacks, and hallucinations (in case of RAG applications). Since cost can be an issue at production scale, we recommend using random sampling strategies to evaluate over a subset of your production logs. Depending on your application and latency requirements, these evals can be set up to run async within your data warehouse or live as a validation step within your app orchestration (read: a live guardrail).

‍

At this stage, we also recommend setting up robust CI tests to make sure you can quickly check for regressions as you iterate. If you’re using closed-source providers like OpenAI, they often tend to update their models on a continuous basis which can cause regressions – such issues can be mitigated by setting up an automated testing job using your eval harness that runs on a daily, weekly, or monthly cadence, along with tools to analyze the results and check for regressions.

‍

Evaluation is a continuous process

‍

Evaluation is ultimately a continuous process and your datasets and evaluators should evolve as you discover more problematic behavior from your users in production. It’s important to make sure you have the right tools and processes in place to analyze your production logs, continuously update your datasets, and add more evaluators as you iterate.

‍

HoneyHive for AI evaluation

‍

HoneyHive makes the process of setting up evaluations and CI tests easy and repeatable. We provide many common evaluators out-of-the-box, and allow users to define and configure their own Python or LLM evaluators that best suit their use-case, either within the UI or programmatically via the SDK. Once set up, you can send your inference data to us to automatically evaluate these criteria and return whether a particular test passed or failed. This can then be used with frameworks like Pytest and tools like Github Actions to set up automated CI tests that check for regressions in your application as you iterate.

‍

If interested, you can read more about setting up evaluators and guardrails with HoneyHive in our docs. If you’re curious to learn more about how to set up evaluators for your enterprise-specific guardrails and taxonomy, book a call with our team.

About the author:

Mohak Sharma

Co-Founder & CEO

How to evaluate AI applications

About the author:

Mohak Sharma

Join our monthly newsletter