I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper", was a tiny slice of the whole thing. The years went into everything around it.

Here's what actually broke, more or less in the order it broke.


...