How I Built a Broken Link Scanner for Large Websites

A broken link checker sounds like a weekend project. Fetch a page, collect every <a href>, request each URL, and report anything that returns 404. That approach works surprisingly well—until you point it at a real website. Large websites introduce duplicate URLs, redirects, external domains, missing assets, rate limits, malformed HTML, CSS...