A while back, I was working on a data project. Nothing crazy. I just needed to pull product prices from a handful of e-commerce sites every day and dump them into a spreadsheet. Simple enough, right?

Wrong.

Getting the first spider up and running took me way longer than it should have. And once I finally got it working, I quickly realized that...