I want to start with a failure, because it taught me more than the wins.
We built an AI code review bot.
It was wired directly into pull requests.
The pitch was simple.
AI reviews first, then humans just approve the good stuff.
It sounded like free velocity.
The first thing it did was add about half a day to PR cycle time.
Not because it broke CI.
Because authors read the suggestions line by line, trying to separate signal from noise.
By around day 60, engagement in the targeted group dropped into the single digits.
We didn't turn it off.
Developers stopped believing it was worth reading.
That was the lesson.
If automation isn't trusted, it doesn't save time.
It creates work.
I used to think that was a soft, cultural problem.
"We need better change management."
"People will get used to it."
"It's new. Give it time."
Now I think it's much more practical, and much more predictable.
If a tool is noisy, it doesn't just fail to help.
It changes behavior.
And the behavior change is usually the opposite of what you wanted.
People become cautious.
They slow down.
They pull in more reviewers.
They second-guess merges.
They spend time reading output they don't believe, because ignoring it feels risky.
That is not an AI problem.
That's a product problem.
And it shows up fast.
Why We Tried It Anyway
The success criteria looked reasonable on paper:
- Engagement: >80% meaningful interaction (open the review + leave a signal)
- Competitive: >25% more helpful than a Copilot-style review experience (Copilot stayed on, we inserted our tool in the same path, and compared thumbs up/down + which suggestions were accepted)
- Satisfaction: >70% across our language mix (Swift, Kotlin, BrightScript, TypeScript, C++, Java, Go, Python)
- Outcomes: reduce iteration from ~4-5 commits/PR to ~2 and PR cycle time from ~2.5 days to ~1 day
And we had a cultural motivation that never shows up in a slide deck:
We wanted to reduce the number of times a senior engineer had to be the "human lint rule".
If you can get to "AI-first pass, then one human pass," you don't just save time.
You reduce review fatigue.
You create more space for mentoring and architecture decisions.
That was the bet.
We also had a cost model.
We ran ~8,000 PRs/month. PR authors iterated ~4-5 commits.
Each commit-run through the pipeline (CI + tests + LLM steps) was about $6.
If AI review could save ~3 iteration commits per PR, that's ~24,000 fewer commit-runs/month, or roughly ~$144K/month.
(Commits don't always map 1:1 to CI runs. Treat that as an order-of-magnitude model.)
What Actually Happened
We ran the experiment on a fraction of traffic.
In ~90 days it processed ~1,500 PRs.
At first, the bot looked like it was "working".
It posted comments.
People reacted.
Some people even replied.
Then we did the one thing that matters.
We asked developers if it helped.
When we asked developers if it helped, 65%+ said no.
The quote that ended the debate was brutally specific:
"Maybe one useful comment out of 10, and some are wildly wrong."
And it wasn't a clean "works" vs "doesn't work" story.
It was uneven.
- Best: Java and Python
- Worst: Swift and BrightScript
In 2025, that cross-language gap was big enough to break trust.
What I didn't anticipate: noise is not neutral.
Noise taxes attention.
Noise adds friction.
Noise makes people defensive.
It also creates a weird new kind of social load.
If a human reviewer leaves a bad comment, you can talk to them.
If a bot leaves bad comments, people either ignore it (quietly) or they start debating whether the bot is "allowed" to be wrong.
Once a bot is in the workflow, the default expectation becomes: "someone is accountable for what the bot says."
That someone is usually the platform team.
Which is fine.
But it means you are not shipping a feature.
You are shipping a new source of obligations.
The most humbling part: it didn't just fail to help. Early on, it made things worse.
PR cycle time went up by ~0.5 days because authors read suggestions just to discard most of them.
This is the part I wish more leaders would internalize.
Bad automation doesn't look like failure.
It looks like work.
It looks like people doing their jobs.
It looks like activity.
It just doesn't look like impact.
If you are reading this and thinking, "but that's just tuning," I agree.
The problem is what happens to trust while you tune.
Once developers decide a tool isn't worth reading, you don't get a slow decline.
You get a cliff.
So we killed it.
And weirdly, that was the most strategic thing we did that quarter.
Not because killing is fun.
Because killing was clarity.
It was the moment we stopped pretending that "shipping" is the same thing as "posting a bot comment".
If you're wondering what "kill" looked like in practice:
- we wrote the closure report
- we summarized the evidence
- we made it explicit what we'd keep (telemetry, infra patterns, some of the prompt work)
- and we made it explicit what we'd stop paying for (comments that developers didn't trust)
That doesn't make the miss feel good.
But it prevents the miss from turning into a slow leak.
What I Trust Now (90-Day Bets)
After that, I stopped trusting slide-deck strategy.
I started trusting a simple 90-day bet system:
- pick a small number of bets
- name owners
- define ship criteria before build
- define kill criteria before build
- keep a decision log
The reason I like the 90-day framing is that it forces honesty.
You can hide behind a roadmap for a year.
You can't hide behind a bet for 90 days.
By day 30, you should know whether you have a signal.
By day 60, you should know whether the signal is improving or decaying.
By day 90, you should have the courage to call it.
The mistake I see teams make is dragging the decision out because the pilot is "almost good enough".
If you're in that place, you're not running a bet.
You're paying a tax.
In our case, engagement fell off a cliff around day 60.
That was the signal.
The remaining 30 days were for confirming the trend and documenting the why, not for hoping the trend reversed.
This isn't just for AI. It's platform strategy.
The Bets That Shipped
These were separate 90-day cycles across different years, not one big program.
2018: telemetry + anomaly detection + correlation
This one wasn't called "AI strategy" at the time.
It was called "stop paging the wrong person" and "stop guessing".
It reached ~20 DAU / ~150 MAU.
It contributed to ~$5M avoided (directional).
Here's what people miss.
If the workflow is "read-only dashboards," daily actives will look small.
But if those daily actives are the people on the critical path during incidents, the impact is not linear with headcount.
Also, this bet didn't succeed because we built a beautiful dashboard.
It succeeded because the workflow moved.
Alerts showed up in Teams and email.
On-call didn't have to hunt.
And when the signal was good enough, people stopped arguing about whether something was "real".
They acted.
2023: CI/CD runners on AWS
Linux on EC2, macOS on Dedicated Hosts. Outcome: ~$120K/month less in direct runner infrastructure spend and ~30% faster on the workflows that mattered.
That bet wasn't glamorous.
It was just a refusal to build on top of a delivery system that engineers had learned to ignore.
If you're trying to make any kind of AI program work, this is the boring prerequisite.
You can't build faster decision-making on top of slow, untrusted delivery signals.
2025: Backstage as an internal developer portal
Humans used it for onboarding, access requests (including AI model access), support tickets, and provisioning infrastructure. It hit ~30 DAU / ~500 MAU.
This bet taught me something uncomfortable.
Most "AI adoption" problems are actually platform problems.
If developers don't know where to go to onboard, request access, provision a service, or find ownership, AI doesn't fix it.
AI just becomes another tab.
Backstage worked for us because it made one path the default.
It made the "right" path the easy path.
It made the "what do I do now?" questions self-serve.
And the AI model access workflow mattered more than I expected.
When developers can request access to approved models and request new models through the same portal as any other access request, governance stops being a separate program.
It becomes a platform capability.
That shift is quiet.
But it's how you avoid shadow AI without turning governance into a brake.
If I Tried AI Code Review Again
The model landscape is better now than it was in 2025.
I would not generalize our result into "never do AI review".
But I would change the shape of the bet.
- I would segment by language from day 1 (different thresholds, different model choices)
- I would start with "summary only" and "ask-to-expand" rather than defaulting to full comment spam
- I would treat false positives as the primary enemy, not missing edge cases
- I would measure acceptance and downstream rework, not just thumbs up/down
Most importantly: I would assume trust is fragile.
You only get a small number of chances before people stop reading.
Where I Want Your Pushback
If you're leading platform or AI work right now:
- What are your kill criteria?
- Have you killed something popular?
- Have you had a pilot that made cycle time worse before it got better?
And the one I'm genuinely curious about:
If you could only run one 90-day bet next quarter, what would you pick and what would make you stop?
SOCIAL SHARE CARD GENERATOR