Optimize an AI agent to sound human, judged by an AI detector

You can tell when an LLM wrote an email. The "I hope this email finds you well" opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn't do that, and "don't sound like an AI" turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold-email took more iterations than I'd guessed.

This is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that.

So I stopped hand-tuning and let LaunchDarkly . This tutorial is the how. If you don't have an account yet, . Clone it to follow along.

Prerequisites

• 
  
  


• Python 3.11+ and for the AI-likeness detector


• A LaunchDarkly SDK key and a REST API key, in a local .env
  

Install the project and its dependencies:

Terminal

uv sync

# .env holds LD_SDK_KEY, LD_API_KEY, LD_PROJECT_KEY,

#            ANTHROPIC_API_KEY, AI_LIKENESS_API_KEY

The repo aliases the short LD_* names to the LAUNCHDARKLY_* names the SDK expects, so the short names in .env are enough.

How agent optimization works

Agent optimization runs an iterative loop against an saved as an AgentControl config by its key.

Explore candidates. Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn't win. Even then, clearing the gate doesn't certify that a candidate is good enough to ship. A fuller eval decides that, later.

Commit the winner. The recommended variation shows up in LaunchDarkly, and with autoCommit it publishes back to the agent's . A model reads the history of prompts and their scores, then writes the next candidate to try. It searches over prompts, not model weights, so each candidate is cheap to run and nothing is ever trained.

How the sample is organized

The companion repo keeps the moving parts in small files, so each piece is easy to find and swap:

• 
  bootstrap.py: seeds the three LaunchDarkly objects, the email-agent config, the ai-likeness judge, and the email-agent-opt optimization. It's safe to re-run, and it prints links to the configs and the Results tab.


• 
  optimize_from_config.py: the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab's link at the end.


• 
  optimize.py: the two callbacks the run needs. handle_agent_call drafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. handle_judge_call scores AI-likeness with GPTZero and hands the optimizer the per-reply detector output.


• 
  detector.py: the GPTZero client. score_with_response returns the number the judge gates on and the full GPTZero JSON.


• 
  messages.py: the synthetic input messages.


• 
  gptzero_test.py: a standalone probe for scoring a draft by hand.


• 
  clients.py and env.py: the LaunchDarkly and Anthropic clients, built once each, and the .env loader.

The saved optimization holds what you're optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds how the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together.

Step 1: Bootstrap the agent, judge, and optimization

One command seeds everything this tutorial needs. bootstrap.py creates three objects in LaunchDarkly, and it's idempotent, so anything that already exists is left alone:

• 
  email-agent: the agent , an AI detector. The score is 1 - P(human): near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down.
  
  
  

  GPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge, its prompt stays a placeholder, and the real number comes from a small client you call yourself:

  
  
  

  detector.py
  
  

  
  
  
  
  

  CODE Kopieren

  def score_with_response(text: str):
  
      data = _api_call(text)                                    # POST the reply to GPTZero
  
      doc = (data.get("documents") or [{}])[0]
  
      return 1.0 - doc["class_probabilities"]["human"], data    # 1 - P(human), + full JSON
  
  

  
  
  
  
  
  
  
  

  score_with_response returns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set AI_LIKENESS_API_KEY to your GPTZero key.

  
  
  

  Two things here cost me time. GPTZero sits behind Cloudflare, and a plain urllib request comes back as a 403 with error code: 1010 before it reaches the API. That reads like a bad key, but it isn't. Sending any real User-Agent header clears it. The score also comes from 1 - P(human) rather than the average_generated_prob field, which is the fraction of sentences flagged AI and reports 1.0 even on text the detector still classifies as human. Gating on that field punishes replies that already passed.

  
  
  

  GPTZero is the only judge, so handle_judge_call doesn't branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages:

  
  
  

  optimize.py
  
  

  
  
  
  
  

  CODE Kopieren

  async def handle_judge_call(key, config, context, is_evaluation=True):
  
      text = _extract_candidate(context.user_input or "")
  
      replies = [r for r in (json.loads(text) or {}).get("replies", [])
  
                 if isinstance(r, str) and r.strip()]
  
      if not replies:                              # empty draft must FAIL the inverted gate
  
          return OptimizationResponse(output=json.dumps(
  
              {"score": 1.0, "rationale": "empty or degenerate candidate"}))
  
      results = []
  
      for reply in replies:                        # one GPTZero call per reply in the batch
  
          s, raw = detector.score_with_response(reply)        # 1 - P(human); + full JSON
  
          results.append({"reply": reply, "ai_likeness": s, "gptzero": raw})
  
      avg = round(sum(r["ai_likeness"] for r in results) / len(results), 4)
  
      # The gate is the AVERAGE; the full per-reply GPTZero JSON becomes the rationale.
  
      return OptimizationResponse(output=json.dumps(
  
          {"score": avg, "rationale": f"Average AI-likeness = {avg}. {json.dumps(results)}"}))
  
  

  
  
  
  
  
  
  
  

  Two choices in there are what make optimizing against a detector actually work.

  
  
  

  Score a batch and average. A single short reply's GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against.

  
  
  

  Forward the whole detector response. The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side.

  
  
  

  The empty-draft case is the one I got wrong first. An empty reply scores 0.0, which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns 1.0 and fails the gate.

  
  
  

  To build intuition before you run the loop, probe GPTZero on a draft by hand:

  
  
  

  Terminal
  
  

  
  
  
  
  

  CODE Kopieren

  uv run python gptzero_test.py "your draft reply"
  
  

  
  
  
  
  
  
  
  

  
  

  A detector is a black box, so gate accordingly

  
  
  

  A detection service gives you a defensible score immediately, with no model to train. It also has real limits. You can't tune it, it leans toward calling short LLM text AI, and it bills per call across every iteration. That confidence is the reason to average over a batch instead of gating on a single reply.

  
  

  
  
  

  
  
  
  Step 4: Run the optimization
  

  
  
  

  The saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization:

  
  
  

  bootstrap.py (optimization)
  
  

  
  
  
  
  

  CODE Kopieren

  {
  
      "key": "email-agent-opt",
  
      "aiConfigKey": "email-agent",
  
      "maxAttempts": 10,
  
      "judgeModel": "claude-haiku-4-5-20251001",       # required by the API; the GPTZero judge never calls it
  
      "modelChoices": ["claude-haiku-4-5-20251001"],   # the Claude model the agent drafts with
  
      "judges": [{"key": "ai-likeness", "threshold": 0.5}],   # generator gate, not a winner test
  
      "variableChoices": [   # interpolated into the instructions; the optimizer must use every one
  
          {"sender_type": "friend", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
  
          {"sender_type": "professional contact", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
  
      ],
  
      "userInputOptions": ["Draft the replies now."],  # trigger turn; messages come from {{messages}}
  
      "autoCommit": True,
  
  }
  
  

  
  
  
  
  
  
  
  

  MESSAGES_BLOCK is the message list from Step 2, formatted and fed in through {{messages}}. respondent_name and sender_type are the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps {{messages}} and the JSON envelope intact through every rewrite.

  
  
  

  The threshold is 0.5, and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above 0, so a gate near 0 never trips and the loop never finds anything to keep. At 0.5, this run passed at iteration 6 with 0.43, while the earlier iterations landed between 0.54 and 1.00. That gave the loop room to explore without rubber-stamping every candidate.

  
  
  

  Run it from the saved config:

  
  
  

  Terminal
  
  

  
  
  
  
  

  CODE Kopieren

  OPTIMIZATION_KEY=email-agent-opt uv run python optimize_from_config.py
  
  

  
  
  
  
  
  
  
  

  
  

  Settings live on the optimization

  
  
  

  Thresholds, inputs, models, and maxAttempts are baked into email-agent-opt at bootstrap time. Change them by editing the optimization in the UI, or by deleting it and re-running bootstrap.py with the matching environment variables set. Committing the winner back as a variation needs the REST API key.

  
  

  
  
  

  The command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate.

  
  
  

  
  
  
  Step 5: Read the winner
  

  
  
  

  Open the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends.

  
  
  
  
  

  Iteration 1, the baseline template: the JSON-and-{{messages}} instruction, the "Draft the replies now." trigger input, and the replies it produced. At 0.64 it didn't clear the gate.

  
  
  

  The optimization sets autoCommit, so on success the winner publishes back to email-agent as a new variation. Open the agent Variations tab to read what the optimizer wrote.

  
  
  

  
  
  
  What the optimizer changed
  

  
  
  

  The run committed a new variation, optimistic-coyote. The Variations tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the {{sender_type}} and {{messages}} variables, and built a full humanization spec around them:

  
  
  

  for real data, which the sections below cover.

  
  
  

  
  
  
  The UI, the saved config, and the SDK
  

  
  
  

  You can run all of this from the UI or from code. The New optimization form builds the same optimization by hand and streams to the same Results tab. This tutorial used optimize_from_config, which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, optimize_from_options takes the settings directly and accepts more than one judge, and optimize_from_ground_truth_options handles Expected Output mode when you have correct answers to match.

  
  
  

  This demo leaves several controls unused: token_optimization and latency_optimization for a cost-and-latency pass, token_limit for a spend cap, variation_key, output_key, context_choices, and the on_turn, on_passing_result, on_failing_result, and on_status_update callbacks.

  
  
  

  
  
  
  Optimization is not evaluation
  

  
  
  

  Offline evals and optimization sit next to each other in AgentControl and do opposite jobs. An , score it with for what "better" means, the way you defined human-sounding replies here.



• 
  Baseline with , so you know where the current agent stands and have a regression check to compare against. For a worked example, read and AI Insights, then feed what you learn back into the dataset and the next optimization. , then read the winning humanization spec off the Variations tab. The loop's job was to explore cheaply, and the trustworthy verdict comes from a proper .
  
  
  

  for what "better" means, optimize against them to surface candidates, then check those candidates with . What you learn in production feeds the next round. If you're getting started, and point the loop at your own prompts.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen