Building a Serverless AI Model Evaluation Platform on AWS

The Problem

A media company needed to evaluate which AI model produces the best podcast-style summaries from news articles. They wanted to:

• Send an article to multiple AI models simultaneously


• Compare the outputs side by side


• Score each output automatically


• Generate a visual comparison report

Doing this manually, copying articles into different model playgrounds, reading outputs, judging quality, doesn't scale. They needed an automated evaluation pipeline that could run experiments on demand and produce consistent, comparable results.

What We Built

A fully serverless evaluation platform on AWS that accepts an article, runs it through multiple foundation models in parallel, scores each output using a separate AI judge, and produces an HTML comparison report. All triggered by a single API call.

The system handles the entire lifecycle:

1. 
   Prompt optimization — an AI agent refines the user's instructions into an effective prompt


2. 
   Parallel model invocation — multiple Bedrock models generate summaries simultaneously


3. 
   Automated scoring — a scoring agent evaluates each output against quality criteria


4. 
   Report generation — produces a formatted HTML comparison page

Architecture Overview

Step 1: Validate

def validate(event):

    """Read and validate the experiment definition from S3."""

    definition = s3.get_object(Bucket=BUCKET, Key=f"definitions/{experiment_id}/definition.json")

    # Validate required fields: article, models, prompt

    # Fail fast if inputs are malformed

    return validated_definition

Why a separate step? Fail-fast validation before incurring any Bedrock costs. If the definition is malformed, we stop here — no wasted model invocations.

Step 2: Invoke Models (Parallel)

This is where it gets interesting. We invoke multiple Bedrock models simultaneously using Python's ThreadPoolExecutor:

from concurrent.futures import ThreadPoolExecutor, as_completed



def invoke_models(definition):

    models = definition['models']  # e.g., ["meta.llama3-70b", "deepseek-r1", "amazon.nova-lite"]

    prompt = definition['prompt']

    article = definition['article']



    results = {}



    with ThreadPoolExecutor(max_workers=len(models)) as executor:

        futures = {

            executor.submit(invoke_bedrock, model_id, prompt, article): model_id

            for model_id in models

        }

        for future in as_completed(futures):

            model_id = futures[future]

            response = future.result()

            results[model_id] = {

                "output": response['output']['message']['content'][0]['text'],

                "usage": {

                    "input_tokens": response['usage']['inputTokens'],

                    "output_tokens": response['usage']['outputTokens']

                }

            }



    return results

Why ThreadPoolExecutor inside Lambda? Bedrock API calls are I/O-bound. Running them in parallel within a single Lambda invocation means we pay for one Lambda execution instead of three, and the total wall-clock time is roughly equal to the slowest model rather than the sum of all models.

Step 3: Store Outputs

Writes comparison.json to S3 — containing all model outputs but no scores yet. This creates a checkpoint: if scoring fails, we don't lose the generated content.

Step 4: Score (Parallel)

The scoring agent (Claude Haiku) evaluates each model's output against quality criteria. Again, parallel execution via ThreadPoolExecutor:

def score(outputs):

    scoring_prompt = """Rate this podcast summary on:

    - Accuracy (1-10): Does it faithfully represent the article?

    - Engagement (1-10): Would a listener find this compelling?

    - Structure (1-10): Is it well-organized for audio?

    Respond with JSON only."""



    with ThreadPoolExecutor(max_workers=len(outputs)) as executor:

        futures = {

            executor.submit(invoke_bedrock, SCORING_MODEL, scoring_prompt, output): model_id

            for model_id, output in outputs.items()

        }

        # ... collect scores

Why a separate scoring model? Using a different model (or at minimum, a separate invocation with a scoring-specific prompt) as the judge avoids self-evaluation bias. The scoring agent doesn't know which model produced which output.

Step 5: Store Scores

Updates comparison.json with the scores attached to each model's output.

Step 6: Generate HTML

Produces a formatted comparison.html report that displays all outputs side by side with their scores. This is the final deliverable the user downloads.

Why Amazon Bedrock's Converse API?

We use the


💼 LinkedIn: https://www.linkedin.com/company/storm-reply/posts/?feedView=all

Date: May 2026

The full system runs in eu-central-1 (Frankfurt), costs under $20/month excluding Bedrock usage, and handles the entire evaluation lifecycle in a single API call. Serverless means we pay nothing when nobody's running experiments, and scale automatically when they are.

If you're building something similar — any system where API calls trigger expensive downstream operations — lock down your API first, validate inputs aggressively, and always know what each request costs.

Built with AWS Lambda, Step Functions, and Amazon Bedrock.