Testing Large Language Models (LLMs) within real-world applications presents a unique set of challenges. In this post, I’ll share my experience navigating the complexities of testing an LLM-based API, the hurdles encountered, and how tools like Pytest and Trulens became instrumental in achieving reliable and meaningful test results.
Introduction
As I was tasked with testing LLM responses, I faced one of the biggest fears of any QA automation engineer: creating infrastructure and tests for an inconsistent and unpredictable system, which could lead to unreliable tests. How could I create reliable tests that would ensure our LLM output provides users with the accurate information they need to know?
The project involved developing an application where the LLM assists users by answering medical insurance policy-related questions. Users input specific medical services, and the LLM provides detailed information based on predefined policy data. Ensuring the accuracy and consistency of these responses was critical, as they directly influenced users’ understanding of their health coverage and potential out-of-pocket costs.
The consequences of providing incorrect information are clear—nobody wants to believe they are covered for a medical service, only to discover that the information is totally wrong when they are already at the doctor’s office.
Why Is Testing LLMs So Challenging?
Varied Responses: LLMs often generate varied responses even when presented with the same prompt, leading to inconsistent test results.
Hidden Hallucinations: Subtle errors or inaccuracies may seem minor but can significantly impact the user's understanding of their policy. For example, a small typo or incorrect amount can cause confusion or financial misunderstandings.
First Strategy: Testing with Trulens
Our initial strategy was to find a tool that allowed us to create ground truth, test the LLM response, and display results on a user-friendly dashboard. That’s when we discovered Trulens.
Features of Trulens:
- Provides similarity scores and metrics to evaluate LLM responses.
- Tests relevance, groundedness, and context alignment.
- Supports a variety of test scenarios.
Example Evaluation:
Ground Truth – Final Report for MRI:
In-Network:
Coverage: Covered with additional cost-sharing.
Requirements: Referral is required.
Liabilities: $300 copayment per MRI test; deductible does not apply.
Out-of-Network:
Coverage: Not covered.
Additional Information: Ensure the MRI is medically necessary as determined by your doctor.
First Test Results:
LLM Response: Nearly perfect match.
Trulens Score:1.0
Second Test Results:
LLM Response: Incorrect copayment of $30 instead of $300.
Trulens Score:0.6
Challenges with Trulens:
Ambiguous Scoring: High scores assigned to incorrect responses.
Interpretation Complexity: Nuanced scores required deeper interpretation.
Non-Binary Outcomes: Spectrum of similarity scores complicated automation.
To address these, we tried focusing on specific report parts but still faced similar issues.
Adopting a New Strategy: Direct Assertion Against Ground Truth
Given the limitations of Trulens, we adopted a direct assertion strategy, querying another LLM (e.g., GPT-4) about specific aspects of the report and asserting the responses against ground truth.
Process:
Generate the Final Report: Primary LLM generates the output.
Structured Testing: Questions about the report are posed to an external LLM.
Assertion: Responses are directly compared with the expected answers.
Example:
Question: "What is the cost of the payment?"
Expected Answer:$300
Test Tool: Pytest for direct value comparison.
Challenges and Refinements:
During this transition, I encountered a couple of unexpected hurdles:
Unexpected Response Formats
Sometimes, the external OpenAI LLM would respond with full sentences or phrases instead of just the expected single letters ("a", "b", etc.). For example:
Question: "Is a referral required for MRI? Choose one of the following: a. yes, b. no"
LLM Answer: "a. yes", "option a".
This caused the assertions in the tests to fail, even if the underlying answer was correct.
Misinterpretation of Report Structure
While sometimes the final report output came with structural differences, it led the external LLM to sometimes overlook or misinterpret the relevant information, thinking that some details weren't present in the report.
Refined Prompt Structure:
To address these challenges, I refined both the prompts sent to the external LLM and the questions themselves to ensure clarity and adherence to the expected response format.
Example:
Based on the following policy details for MRI:
- {{ Here we place the answer we would like to validate }}
Question: What is the copayment/liabilities amount for MRI? Choose one of the following:
- a. $100
- b. $200
- c. $300
- d. $400
- e. $500
Please respond with only the lowercase letter corresponding to the correct answer (e.g., 'a').
To ensure that the external LLM focuses on the correct section of the report and responds in the desired format, I made the questions more explicit. These refinements ensured that the external LLM understood exactly where to look in the report and how to format its response, thereby eliminating the issues of unexpected response formats and misinterpretation of the report's structure.
Benefits of the New Approach:
Clarity and Precision:
By structuring the prompt to require only the corresponding letter as a response, the likelihood of extraneous text is minimized, leading to more straightforward assertions.
Direct Comparison:
Eliminating the intermediary evaluation step allows for direct comparison between the LLM's response and the expected ground truth, ensuring unequivocal test results.
Reduced Ambiguity:
This method avoids the ambiguity introduced by similarity scores, focusing solely on whether the response matches the expected answer.
Simplified Testing Framework:
Streamlining the testing process by removing the reliance on Trulens simplifies the overall framework, making it easier to maintain and understand.
Early Error Detection:
By directly asserting responses, any changes or errors in the final report that affect crucial information are quickly identified, preventing misleading users.
Scaling the Test:
As the new strategy was found to be very useful, the potential to automate the test development process was promising.
Since the test procedure was built to suit all possible scenarios, we created a JSON file with the tests data: medical service, questions about the final report, and expected answers about it.
Sample JSON Structure:
{
"services": [
{
"query": "MRI",
"tests": [
{
"test_number": 1,
"additional_question": "What is the copayment amount for MRI? Choose one of the following: a. $100, b. $200, c. $300",
"expected_response": "c"
},
{
"test_number": 2,
"additional_question": "Is a referral required for MRI? Choose one of the following: a. yes, b. no",
"expected_response": "a"
}
]
}
// Additional services and tests...
]
}
This structured approach ensures that every crucial aspect of the policy is covered, and the expected responses are clearly defined. Also, it allows us to expand the test scale as we want - add new medical services and questions about the report as we want, as we just need to add the desired question to the JSON.
Moreover, to create the flow almost fully automated, we provided ChatGpt our verified answers of the final reports and example questions about some reports, and it created a full JSON file as above, with all the scenarios to be tested. In that way we can expand our tests with minimal work.
What next ?
As you can understand, this strategy can open many doors for testing other LLM applications in the future. All we need is to define our ground truth—the aspects we expect the LLM to provide 100% reliable information about—and simply ask another LLM questions about the output.
And remember, when your LLM tries to pull a fast one on you with a "slightly off" answer, a well-crafted prompt and a sharp eye can keep your tests running smoothly. After all, in the wild world of LLM testing, it's better to catch the $300 copayment errors before they turn into $30,000 misunderstandings!
SOCIAL SHARE CARD GENERATOR