This is a submission for the is the project I came back to finish for this challenge.
It started as a small SEO crawler I built for myself because I wanted a tool I could understand end to end. The first version looked more complete than it really was: it printed scores, generated reports, and even had a few useful ideas like crawl graph analysis and snapshot diffs.
But I eventually realised I could not trust its output.
This rebuild added a lot of new capabilities, but the most important change was that I finally made the tool trustworthy. I turned that abandoned prototype into a TypeScript CLI that can crawl real sites, handle redirects and robots.txt, render JavaScript-heavy pages when needed, and report issues with context and suggested fixes instead of hiding everything behind one misleading score.
The Before
The old script was roughly 700 lines. It had classes, interfaces, config objects, sitemap parsing, structured output, an HTML report, and a snapshot diff system.
Running it on example.com looked like this:
Starting SEO audit of https://example.com
https://example.com
Score: 80/100
Grade: B
Title: Example Domain
Meta description: missing
Canonical: missing
Audit complete. Report saved to ./audit-output/
And this was the kind of report it produced:
Before screenshot placeholder: old report with a misleadingly good score
That output was the problem.
A page with no meta description, no canonical tag, and no robots.txt should not feel "basically fine". But the tool wrapped weak logic in clean output, so I trusted it.
Two parts of the first version were actually useful:
- a crawl graph that mapped internal links and orphan pages
- a snapshot diff system that compared audits over time
Those good parts are exactly why I missed the bad foundation for so long.
Where v1 was wrong
The bugs were not dramatic. The script did not crash. It just produced plausible-looking answers.
1. Redirects were treated like failures
const CONFIG = {
followRedirects: false,
};
if (res.status >= 300 && res.status < 400) {
return;
}
That meant normal 301/302 behavior could stop the crawl.
2. Meta description extraction accepted invalid patterns
const metaDescription =
$('meta[name="description"]').attr('content') ||
$('meta[property="description"]').attr('content') ||
$('meta[name="og:description"]').attr('content') ||
undefined;
Those fallback selectors are wrong. A page could fail the real check but still look fine in my report.
3. Scoring was just arbitrary deductions
let score = 100;
if (!page.title) score -= 15;
if (!page.metaDescription) score -= 10;
if (!page.h1) score -= 15;
No real severity model. No category breakdown. No evidence. Just a number that looked authoritative.
That was the worst part of the old tool: not that it was incomplete, but that it was confidently wrong.
Why I stopped working on it
I did not abandon the first version because I got bored with SEO. I abandoned it because I lost confidence in the code.
Every time I tried to improve it, I ran into another blocker:
- redirects broke assumptions in the crawl flow
- site restrictions and real-world variance made naive checks unreliable
- some ideas I wanted, like better JavaScript rendering, stronger rules, and more trustworthy scoring, felt hard or impossible inside that codebase
- fixing one weak part usually exposed two more
At some point the project stopped feeling like "one more weekend and it is done" and started feeling like a pile of compromises I no longer trusted.
That killed my motivation more than the size of the code ever did.
Finishing it meant rebuilding it
For this challenge, I did not "polish the old script". I kept the useful ideas, then rebuilt the project around one rule:
correctness before polish
Here is the real before/after:
| Area | Before | After |
|---|---|---|
| Crawl handling | Naive fetch() flow | rate limiting, retries, redirect-chain handling |
| Extraction | Fragile selectors | validated extractors and cross-checks |
| Link checking | false positives | better status handling and concurrency control |
| Scoring | one magic number | category-based scoring with severities |
| Output | score first | findings first, with fixes |
| Good ideas kept | crawl graph, snapshot diff | both retained and expanded |
What I actually accomplished in the rebuild
The rebuilt version became
Project:
Final thoughts
The lesson from this project was simple:
a polished tool that gives wrong answers is worse than a rough tool that tells the truth
I stopped working on the first crawler when every fix revealed another bad assumption. Finishing this project did not mean forcing that codebase a little farther. It meant admitting the foundation was wrong, keeping the useful ideas, and rebuilding the rest so the output could be trusted.
That is why this challenge fit so well. I did not just reopen an abandoned project. I finally finished the hard part: making it honest enough to use.
SOCIAL SHARE CARD GENERATOR