How to scrape Google search results with Python

Scraping Google Search delivers essential SERP analysis, SEO optimization, and data collection capabilities. Modern scraping tools make this process faster and more reliable.

One of our community members wrote this blog as a contribution to the Crawlee Blog. If you would like to contribute blogs like these to Crawlee Blog, please reach out to us on our that can handle result ranking and pagination.

We'll create a scraper that:

• Extracts titles, URLs, and descriptions from search results


• Handles multiple search queries


• Tracks ranking positions


• Processes multiple result pages


• Saves data in a structured format

Prerequisites

• Python 3.7 or higher


• Basic understanding of HTML and CSS selectors


• Familiarity with web scraping concepts


• Crawlee for Python v0.4.2 or higher

Project setup

1. 
   

   Install Crawlee with required dependencies:
   
   

   
   
   CODE Kopieren

   pipx install crawlee[beautifulsoup,curl-impersonate]
   
   

   
   
   


2. 
   

   Create a new project using Crawlee CLI:
   
   

   
   
   CODE Kopieren

   pipx run crawlee create crawlee-google-search
   
   

   
   
   



3. When prompted, select Beautifulsoup as your template type.



4. 
   

   Navigate to the project directory and complete installation:
   
   

   
   
   CODE Kopieren

   cd crawlee-google-search
   
   poetry install
   
   

   
   
   

Development of the Google Search scraper in Python

1. Defining data for extraction

First, let's define our extraction scope. Google's search results now include maps, notable people, company details, videos, common questions, and many other elements. We'll focus on analyzing standard search results with rankings.

Here's what we'll be extracting:

Based on the data obtained from the page, all necessary information is present in the HTML code. Therefore, we can use as our http_client with preset headers and impersonate relevant to the to control scraping aggressiveness. This is crucial to avoid getting blocked by Google.

If you need to extract data more intensively, consider setting up

There's an obvious distinction between readable ID attributes and generated class names and other attributes. When creating selectors for data extraction, you should ignore any generated attributes. Even if you've read that Google has been using a particular generated tag for N years, you shouldn't rely on it - this reflects your experience in writing robust code.

Now that we understand the HTML structure, let's implement the extraction. As our crawler deals with only one type of page, we can use router.default_handler for processing it. Within the handler, we'll use BeautifulSoup to iterate through each search result, extracting data such as title, url, and text_widget while saving the results.

@crawler.router.default_handler

async def default_handler(context: BeautifulSoupCrawlingContext) -> None:

    """Default request handler."""

    context.log.info(f'Processing {context.request} ...')



    for item in context.soup.select("div#search div#rso div[data-hveid][lang]"):

        data = {

            'title': item.select_one("h3").get_text(),

            "url": item.select_one("a").get("href"),

            "text_widget": item.select_one("div[style*='line']").get_text(),

        }

        await context.push_data(data)

4. Handling pagination

Since Google results depend on the IP geolocation of the search request, we can't rely on link text for pagination. We need to create a more sophisticated CSS selector that works regardless of geolocation and language settings.

The max_crawl_depth parameter controls how many pages our crawler should scan. Once we have our robust selector, we simply need to get the next page link and add it to the crawler's queue.

To write more efficient selectors, learn the basics of syntax.

    await context.enqueue_links(selector="div[role='navigation'] td[role='heading']:last-of-type > a")

5. Exporting data to CSV format

Since we want to save all search result data in a convenient tabular format like CSV, we can simply add the export_data method call right after running the crawler:

await crawler.export_data_csv("google_search.csv")

6. Finalizing the Google Search scraper

While our core crawler logic works, you might have noticed that our results currently lack ranking position information. To complete our scraper, we need to implement proper ranking position tracking by passing data between requests using user_data in

The code repository is available on - you might need a ready-made solution.

Consider using , ,

What will you scrape?

In this blog, we've explored step-by-step how to create a Google Search crawler that collects ranking data. How you analyze this dataset is up to you!

As a reminder, you can find the full project code on GitHub.

I'd like to think that in 5 years I'll need to write an article on "How to extract data from the best search engine for LLMs", but I suspect that in 5 years this article will still be relevant.