Build an AI code review assistant with v0.dev, litellm and Agenta

The code for this tutorial is available . Just provide the URL to a public PR and receive a review from our AI assistant.

.

Adding Observability: Instrumenting the code with playground.

LLM Evaluation: Using LLM-as-a-judge to evaluate prompts and select the best model.

Deployment: Deploying the app as an API and building a simple UI with provides a unified interface for working with various LLM providers—making it easy to experiment with different models later:

CODE Kopieren

prompt_system = """

You are an expert Python developer performing a file-by-file review of a pull request. You have access to the full diff of the file to understand the overall context and structure. However, focus on reviewing only the specific hunk provided.

"""



prompt_user = """

Here is the diff for the file:

{diff}



Please provide a critique of the changes made in this file.

"""



def generate_critique(pr_url: str):

    diff = get_pr_diff(pr_url)

    response = litellm.completion(

        model=config.model,

        messages=[

            {"content": config.system_prompt, "role": "system"},

            {"content": config.user_prompt.format(diff=diff), "role": "user"},

        ],

    )

    return response.choices[0].message.content

Adding observability

Observability is crucial for understanding and improving LLM applications. It helps you track inputs, outputs, and the information flow, making debugging easier. We'll use Agenta for this purpose.

Agenta is an open-source LLMOps platform that provides you with all the tools needed to build production-ready LLM-powered applications. It offers a centralized environment to manage prompts, instrument applications for tracking inputs and outputs, and run evaluations to assess result quality. You can signup for free for the .

) and optionally AGENTA_HOST if we're self-hosting.

We can now run the app and see the traces in Agenta.

Evaluating using LLM-as-a-judge

To evaluate the quality of our AI assistant's reviews and compare prompts and models, we need to set up evaluation.

First, we'll create a small test set with publicly available PRs.

Next, we'll set up an LLM-as-a-judge to evaluate the quality of the reviews.

To do this, navigate to the evaluation view, click on "Configure evaluators", then "Create new evaluator" and select "LLM-as-a-judge".

With our evaluator set up, we can run experiments and compare different prompts and models. In the playground, we can create multiple variants and run batch evaluations using the pr-review-quality LLM-as-a-judge.

This gives you an API endpoint ready to use in your application.

to quickly generate a UI. After providing our API endpoint and authentication requirements, we had a working UI in minutes. You can try it yourself: .

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen