TL;DR: GitHub gave Copilot code review better shared tools, but reused generic instructions — reviews got pricier and less accurate until they rewrote the instructions for how a reviewer actually works, cutting cost ~20% with no quality loss.
Shared tooling is supposed to be the easy win: less duplicated code, fewer things to maintain, improvements that carry automatically across products. GitHub's own account of an internal migration – moving Copilot code review onto its shared CLI toolset – makes the case for treating that assumption with at least a little suspicion.
Migrating to shared tools made Copilot's reviews pricier and less accurate
Copilot code review previously ran its own code-exploration tools — list directories, search files, search directories, read code — purpose-built for earlier, less capable models. GitHub's Copilot CLI, meanwhile, runs a broader Unix-style toolset — grep, glob, view — that several other Copilot products draw on too.
GitHub decided to migrate Copilot code review onto that shared CLI toolset — retiring its own tools in favour of the same grep, glob, and view already used elsewhere. The appeal was, essentially, less duplicated engineering effort, and a single toolset that could be improved once and inherited everywhere it was used.
In offline benchmarks, the opposite happened. Review cost went up and fewer useful issues got flagged. Napalys Klicius, software engineer at GitHub, notes that moving to the shared CLI toolset was expected to improve results by giving the agent more flexible code-exploration tools, that didn't hold up once they looked at what the agent was actually doing.
"The tools weren’t the problem, the instructions were," Klicius writes – meaning the prompt-level guidance that tells the agent when and how to use each tool.
"Once we rewrote them for the way a reviewer actually reads a pull request, the regression flipped into a win."
Cost per review fell by around a fifth, without the quality of the reviews slipping.
Klicius likens tool descriptions and system instructions to API documentation — when that documentation is muddled, a developer ends up making worse calls, not because the underlying tool is flawed, but because the guidance around it failed them.
“Unclear tool prompting can do the same for an LLM; a small wording change can affect cost, quality, and the shape of the investigation because it changes how the agent spends its attention,” Klicius writes.
Tool traces showed the agent exploring code instead of reviewing a diff
What made the benchmarks useful here wasn't the score itself — it was that GitHub could pull up exactly which tools the agent reached for, in what order, and how much came back each time. What that record showed was an agent acting less like a reviewer and more like someone poking around a codebase for the first time — casting a wide net, taking guesses at where relevant code might live, and pulling back far more than any single review question called for.
Before — a simplified illustration of the general-purpose behavior we observed: widening the search, guessing paths, and accumulating context. (GitHub)
None of that extra material got discarded — it sat in the agent's working memory for the rest of the review, driving up cost without necessarily helping the agent reach a better answer.
None of that was irrational — it's exactly how you'd want an assistant to behave if its job was to get oriented in a codebase before touching it. But reviewing a pull request is a different task. The goal isn't to build a broad understanding of the codebase, it's to gather just enough context to determine whether a specific change introduced a problem.
New guidance narrowed the agent's search and cut review cost by a fifth
Nothing changed about the tools themselves. What changed was the order the agent was told to reach for them — start from the diff, narrow candidates with grep and glob, and only call view once it actually knew which file or line range mattered. Even failure handling got more specific: a search that came back empty should be retried once with simpler terms, not treated as a cue to start guessing at neighbouring files.
After — a simplified illustration of the review-shaped behavior the prompt guided toward: stay anchored to the diff, narrow with grep and glob, then read focused ranges with view. (GitHub)
In production, that shift held: a roughly 20% drop in average review cost, with review quality unchanged. Worth flagging that this is GitHub's own reported figure from its own benchmarking, not an independently verified number.
The more interesting result came from testing the same fix somewhere it didn't help. GitHub tried applying the same review-shaped guidance inside Copilot CLI itself and saw no equivalent gain, because a CLI session has no single pull request anchoring it — a developer might redirect the whole task halfway through, so there's no diff to narrow around in the first place. The tool was never the variable that mattered. What mattered was whether the guidance around it matched the job the agent was actually being asked to do.
The benchmarks that proved the fix
None of this would have been visible without a way to test it. GitHub could only identify the regression — and prove the rewrite worked — because it had a benchmark suite that could replay the same reviews before and after, measuring both cost and quality. Without that evidence, the new tools would have made an easy scapegoat, and the actual cause — instructions that no longer matched the job — could have gone unnoticed indefinitely.
That's the same discipline behind Tessl's evals model: testing and measuring a skill's instructions before and after every change, treating them as something that needs continuous verification. GitHub built that evaluation infrastructure internally for Copilot code review. Teams managing skills across many agents and many tools need the same kind of repeatable evidence to separate a genuine improvement from a change that simply altered agent behaviour.
The wider lesson here is that any team consolidating tools, upgrading models, or standardising instructions across agents is making the same bet GitHub made: that shared components will behave the same way everywhere they're used. That bet doesn't announce itself when it fails — it just shows up as slightly worse output that nobody's measuring closely enough to catch, which is the argument for building that measurement in before a change ships.
SOCIAL SHARE CARD GENERATOR