AI Test Report - Summarize Failed Tests

The AI Test Reporter is a powerful tool that works with any test framework and leverages your existing accounts with AI language models like OpenAI GPT, Azure and Anthropic Claude to create concise, human-readable summaries of test results, highlighting likely causes and suggesting possible solutions to help your team resolve issues faster.

TL:DR — Check out the a follow and support our mission to enhance test reporting with AI. 🙌

Key Features

Immediate Insights: Print summaries directly to the stdout after running your tests.

Seamless Integration: Integrate with developer tools like GitHub, Slack and more for effortless collaboration.

Universal Compatibility: Works with all major testing frameworks via standardized CTRF reports.

AI Summary JSON Report: Add an AI summary to your CTRF JSON report, enabling further processing and integration with other tools and workflows.

Here is an example of the stdout:

Add the following to your workflow YAML file:

- name: Publish CTRF AI Test Summary Results

  run: npx github-actions-ctrf ai <path-to-ctrf-report>

  if: always()

You will have an AI-generated failed test summary, seamlessly integrated into your GitHub Actions workflow summary

Send AI test summary to your Slack Channel

Use the

More CI/CD integrations are being built, checkout the AI Test Reporter documentation for all integrations

How Does This Work?

CTRF (Common Test Report Format) is a standardized test report schema that serves as a bridge between various testing frameworks and developer tools. It includes a collection of tooling to integrate with platforms like GitHub Actions and more. Among its various properties, the CTRF schema supports an ai property for AI-generated summaries.

Here’s an example of what a CTRF report test object looks like with the AI property

{

  "name": "should display title",

  "status": "failed",

  "duration": 1200,

  "message": "error message",

  "trace": "error trace",

  "ai": "AI generated test summary"

}

In this schema:

name: The name of the test.

status: The result of the test (e.g., failed).

duration: The time the test took to run (in milliseconds).

message: The error message from the failed test.

trace: The stack trace of the error.

ai: The AI-generated test summary, provides a concise explanation of the likely cause of failure and possible solutions.

By using CTRF, you can easily integrate AI-generated test summaries into your workflow. The ai property is filled with the summary provided by AI models such as OpenAI or Anthropic, which can then be displayed in your GitHub Actions summary or added as a comment to a pull request.

You can do so much with CTRF, and the tooling is constantly evolving to better support your testing and development workflows.

If you found this useful, remember to follow the organization and give a star to the repositories you find useful.

🧡 Your support helps the library grow and is truly appreciated!