Typically, tutorials focus on the technical aspects, on what you can replicate: "Start here, follow this path, and you'll end up here." This is great for learning a particular technology, but it's sometimes difficult to understand why the author decided to do things a certain way or what guides their development process.
One of our community members wrote this blog as a contribution to Crawlee Blog. If you want to contribute blogs like these to Crawlee Blog, please reach out to us on our
This is the official developer community of Apify and Crawlee. | 8987 members
discord.com
In this blog, I'll discuss the general rules and principles that guide me when I work on web scraping projects and allow me to achieve great results.
So, let's explore the mindset of a web scraping developer.
1. Choosing a data source for the project
When you start working on a project, you likely have a target site from which you need to extract specific data. Check what possibilities this site or application provides for data extraction. Here are some possible options:
Official API - the site may provide a free official API through which you can get all the necessary data. This is the best option for you. For example, you can consider this approach if you need to extract data from approach
If one data source fails, try accessing another available source.
For example, for Yelp, all three options are available, and if the Official API doesn't suit you for some reason, you can try the other two.
2. Check
I think everyone knows about robots.txt and sitemap one way or another, but I regularly see people simply forgetting about them. If you're hearing about these for the first time, here's a quick explanation:
robots is the established name for crawlers in SEO. Usually, this refers to crawlers of major search engines like Google and Bing, or services like Ahrefs and ChatGPT.
robots.txt is a file describing the allowed behavior for robots. It includes permitted crawler user-agents, wait time between page scans, patterns of pages forbidden for scanning, and more. These rules are typically based on which pages should be indexed by search engines and which should not.
sitemap describes the site structure to make it easier for robots to navigate. It also helps in scanning only the content that needs updating, without creating unnecessary load on the site
Since you're not for
By analyzing it, you'll easily discover that all the data can be obtained with a single request. You simply need to copy this data from your browser into a JSON file, and your task is complete.
. This will allow you to better understand how the site interacts with the backend, what framework it's built on, and what behavior can be expected from it.
5. Data doesn't appear out of thin air
This is obvious, but it's important to keep in mind while working on a project. If you see some data or request parameters, it means they were obtained somewhere earlier, possibly in another request, possibly they may have already been on the website page, possibly they were formed using JS from other parameters. But they are always somewhere.
If you don't understand where the data on the page comes from, or the data used in a request, follow these steps:
Sequentially, check all requests the site made before this point.
Examine their responses, headers, and cookies.
Use your intuition: Could this parameter be a timestamp? Could it be another parameter in a modified form?
Does it resemble any standard hashes or encodings?
Practice makes perfect here. As you become familiar with different technologies, various frameworks, and their expected behaviors, and as you encounter a wide range of technologies, you'll find it easier to understand how things work and how data is transferred. This accumulated knowledge will significantly improve your ability to trace and understand data flow in web applications.
6. Data is cached
You may notice that when opening the same page several times, the requests transmitted to the server differ: possibly something was cached and is already stored on your computer. Therefore, it's recommended to analyze the site in incognito mode, as well as switch browsers.
This situation is especially relevant for mobile applications, which may store some data in storage on the device. Therefore, when analyzing mobile applications, you may need to clear the cache and storage.
7. Learn more about the framework
If during the analysis you discover that the site uses a framework you haven't encountered before, take some time to learn about it and its features. For example, if you notice a site is built with Next.js, understanding how it handles routing and data fetching could be crucial for your scraping strategy.
You can learn about these frameworks through official documentation or by using LLMs like . These AI assistants are excellent at explaining framework-specific concepts. Here's an example of how you might query an LLM about Next.js:
CODE
I am in the process of optimizing my website using Next.js. Are there any files passed to the browser that describe all internal routing and how links are formed?
Restrictions:
- Accompany your answers with code samples
- Use this message as the main message for all subsequent responses
- Reference only those elements that are available on the client side, without access to the project code base
You can create similar queries for backend frameworks as well. For instance, with GraphQL, you might ask about available fields and query structures. These insights can help you understand how to better interact with the site's API and what data is potentially available.
For effective work with LLM, I recommend at least basically studying the basics of looks like
Let's consider this .
I especially recommend paying attention to and , and no, not because they allowed me to write in their blog, but because it shows excellent development dynamics and is aimed at making life easier for web crawler developers. It allows for faster crawler development by taking care of and hiding under the hood such critical aspects as session management, session rotation when blocked, managing concurrency of asynchronous tasks (if you write asynchronous code, you know what a pain this can be), and much more.
:::tip
If you like the blog so far, please consider on Discord.
You can also find me on the following platforms: , , Contra.
Thank you for your attention :)
Vollständiger Original-Bericht
Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR