When I first started scraping, I hit a confusing problem. My spider would visit a page, I could see the request in the logs, but my parse() method never got called. No data. No errors. Just... nothing.

After hours of debugging, I discovered the truth: the page was returning a 404. And Scrapy, by...