I built a scorecard that grades each AI agent's ROI — here's how it works

I was running 11 AI agents — sales outreach, customer support triage, document review, lead scoring, content generation. They were all "working." But I couldn't answer the question every manager asks about their team: "who's pulling their weight?"

I had cost dashboards. I could see total LLM spend. But no one could tell me: this agent made $5,000 in pipeline and cost $800. That one cost $400 and produced nothing measurable.

So I built server that lets Claude, Cursor, or any MCP-compatible client query agent performance data directly.

The key insight: agents themselves can use these tools. An agent can check its own ROI, compare its performance to other agents, and recommend optimization actions. This is the start of self-managing AI workforces.

*The 23 tools (all prefixed `metrx_`) cover 10 domains:

**

| Domain | Tools | What It Does |

|--------|-------|-------------|

| Agent Fleet Overview | 3 | Agent scorecards, performance summaries, detailed agent profiles |

| Optimization | 4 | Model routing, provider arbitrage, cost-per-quality recommendations |

| Budgets | 3 | Spend limits, enforcement modes, budget status |

| Alerts | 3 | Threshold monitoring, acknowledgment, failure prediction |

| Experiments | 3 | A/B model testing, results with statistical significance, winner promotion |

| Cost Leak Detection | 1 | Comprehensive 7-check waste audit |

| Revenue Attribution | 3 | Revenue linking, per-agent ROI calculation, multi-source attribution reports |

| Alert Configuration | 1 | Threshold tuning with automated actions |

| ROI Audit | 1 | Board-ready fleet performance reports |

| Upgrade Justification | 1 | Business case generation for tier upgrades |

*Integration: One Line Change

*

// Before

const openai = new OpenAI({

  apiKey: process.env.OPENAI_API_KEY,

});



// After — just change the base URL

const openai = new OpenAI({

  apiKey: process.env.OPENAI_API_KEY,

  baseURL: "https://gateway.metrxbot.com/v1",

  defaultHeaders: {

    "x-metrx-agent": "sales-outreach",

  },

});

That header is what enables agent-level attribution. Every call tagged with an agent identity flows into that agent's scorecard. Sub-5ms overhead.

*The Self-Optimizing Loop

*

Here's what gets me excited about the MCP approach. When agents have access to their own performance data, they can:

1. Self-assess: "My ROI dropped 20% this week — what changed?"


2. Self-optimize: "I'm using GPT-4o for classification that GPT-4o-mini handles at 1/10th the cost"


3. Self-report: "Generate a board-ready audit of my fleet's performance this quarter"


4. Self-experiment: "Run an A/B test — does switching to Claude Haiku for my routing layer maintain quality at lower cost?"

This is the difference between a cost dashboard (humans stare at charts) and a workforce management system (agents manage their own performance).

*Try It

*

• Dashboard: — MIT licensed


• npm: npx @metrxbot/mcp-server — try in 30 seconds with --demo flag


• Pricing: Free → Lite ($19/mo, 10 agents) → Pro ($49/mo, unlimited)

If you're running AI agents in production, I'd love to hear: how do you know which agents are worth keeping? Drop a comment or find me on X @metrxbot_.