Scrape Google Search Results Using Python
Google holds an immense volume of data for businesses and researchers. It performs over 8.5 billion daily searches and commands a 91% share of the global search engine market.
Since the debut of ChatGPT, Google data has been utilized not only for traditional purposes like rank tracking, competitor monitoring, and lead generation but also for developing advanced LLM models, training AI models, and enhancing the capabilities of Natural Language Processing (NLP) models.
Scraping Google, however, is not easy for everyone. It requires a team of professionals and a robust infrastructure to scrape at scale.
— To pull HTML data from the Google Search URL.
The filtered data would contain this information:
- Title
- Link
- Displayed Link
- Description
- Position of the result
Let us import our installed libraries first in the scraper.py file.
from bs4 import BeautifulSoup
import requests
Then, we will make a GET request on the target URL to fetch the raw HTML data from Google.
headers={'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361681276786'}
url='https://www.google.com/search?q=python+tutorials&gl=us'
response = requests.get(url,headers=headers)
print(response.status_code)
Passing headers is important to make the scraper look like a natural user who is just visiting the Google search page for some information.
The above code will help you in pulling the HTML data from the Google Search link. If you got the 200 status code, that means the request was successful. This completes the first part of creating a scraper for Google.
In the next part, we will use BeautifulSoup to get out the required data from HTML.
soup = BeautifulSoup(response.text, ‘html.parser’)
This will create a BS4 object to parse the HTML response and thus we will be able to easily navigate inside the HTML and find any element of choice and the content inside it.
To parse this HTML, we would need to first inspect the Google Search Page to check which common pattern can be found in the DOM location of the search results.
The displayed link or the cite link can be found inside the cite tag.
Wrapping all these data entities into a single block of code:
organic_results = []
i = 0
# Parse organic results with error handling
for el in soup.select(".g"):
try:
title = el.select_one("h3").text if el.select_one("h3") else "No title"
displayed_link = el.select_one(".byrV5b cite").text if el.select_one(".byrV5b cite") else "No displayed link"
link = el.select_one("a")["href"] if el.select_one("a") else "No link"
description = el.select_one(".VwiC3b").text if el.select_one(".VwiC3b") else "No description"
organic_results.append({
"title": title,
"displayed_link": displayed_link,
"link": link,
"description": description,
"rank": i + 1
})
i += 1
except Exception as e:
print(f"Error parsing element: {e}")
print(organic_results)
We declared an organic results array and then looped over all the elements with g class in the HTML and pushed the collected data inside the array.
Running this code will give you the desired results which you can use for various purposes including rank tracking, lead generation, and optimizing the SEO of the website.
[
{
"title": "Python Tutorial",
"displayed_link": "https://www.w3schools.com \u203a python",
"link": "https://www.w3schools.com/python/",
"description": "Learn Python. Python is a popular programming language. Python can be used on a server to create web applications. Start learning Python now.",
"rank": 1
},
{
"title": "The Python Tutorial \u2014 Python 3.13.1 documentation",
"displayed_link": "https://docs.python.org \u203a tutorial",
"link": "https://docs.python.org/3/tutorial/index.html",
"description": "This tutorial introduces the reader informally to the basic concepts and features of the Python language and system. It helps to have a Python interpreter handy\u00a0...",
"rank": 2
},
....
]
So, that’s how a basic Google Scraping script is created.
However, there is a CATCH. We still can’t completely rely on this method as this can result in a block of our IP by Google. If we want to scrape search results at scale, we need a vast network of premium and non-premium proxies and advanced techniques that can make this possible. That’s where the SERP APIs come into play!
Scraping Google Using ApiForSeo’s SERP API
Another method for scraping Google is using a dedicated SERP API. They are much more reliable and don’t let you get blocked in the scraping process.
The setup for this section would be the same, just we need to register on
After activating the account, you will be redirected to the dashboard where you will get your API Key.
You can also copy the code from the dashboard itself.
Setting Up our code for scraping search results
Then, we will create an API request on a random query to scrape data through ApiForSeo SERP API.
import requests
api_key = "APIKEY"
url = "https://api.apiforseo.com/google_search"
params = {
"api_key": api_key,
"q": "elon+musk",
"gl": "us",
}
response = requests.get(url, params=params)
if response.status_code == 200:
data = response.json()
print(data)
else:
print(f"Request failed with status code: {response.status_code}")
You can try any other query also. Don’t forget to put your API Key into the code otherwise, you will receive a 404 error.
Running this code in your terminal would immediately give you results.
"organic_results": [
{
"title": "Elon Musk - Wikipedia",
"displayed_link": "https://en.wikipedia.org › wiki › Elon_Musk",
"snippet": "Elon Reeve Musk is a businessman known for his key roles in the space company SpaceX and the automotive company Tesla, Inc. His other involvements include ...Musk family · Tesla Roadster · Tesla, SpaceX, and the Quest... · Maye Musk",
"link": "https://en.wikipedia.org/wiki/Elon_Musk",
"extended_sitelinks": [
{
"title": "Musk family",
"link": "https://en.wikipedia.org/wiki/Musk_family"
},
{
"title": "Tesla Roadster",
"link": "https://en.wikipedia.org/wiki/Elon_Musk%27s_Tesla_Roadster"
},
{
"title": "Tesla, SpaceX, and the Quest...",
"link": "https://en.wikipedia.org/wiki/Elon_Musk:_Tesla,_SpaceX,_and_the_Quest_for_a_Fantastic_Future"
},
{
"title": "Maye Musk",
"link": "https://en.wikipedia.org/wiki/Maye_Musk"
}
],
"rank": 1
},
{
"title": "Elon Musk - Forbes",
"displayed_link": "https://www.forbes.com › profile › elon-musk",
"snippet": "Real Time Net Worth · Elon Musk cofounded seven companies, including electric car maker Tesla, rocket producer SpaceX and artificial intelligence startup xAI.Will Elon Musk’s Silicon Valley... · Forbes Real Time Billionaires · Tesla · Peter Thiel",
"link": "https://www.forbes.com/profile/elon-musk/",
"extended_sitelinks": [
{
"title": "Will Elon Musk’s Silicon Valley...",
"link": "https://www.forbes.com/sites/gregorme/2024/12/11/will-elon-musks-silicon-valley-playbook-work-in-government/"
},
{
"title": "Forbes Real Time Billionaires",
"link": "https://www.forbes.com/real-time-billionaires/"
},
{
"title": "Tesla",
"link": "https://www.forbes.com/companies/tesla/"
},
{
"title": "Peter Thiel",
"link": "https://www.forbes.com/profile/peter-thiel/"
}
],
"rank": 2
},
.....
]
The above data contains various points, including titles, links, snippets, descriptions, and featured snippets like extended sitelinks. You will also get advanced feature snippets like People Also Ask For, Knowledge Graph, Answer Boxes, etc., from this API.
Conclusion
The nature of business is evolving at a rapid pace. If you don’t have access to data about ongoing trends and your competitors, you risk falling behind emerging businesses that make data-driven strategic decisions at every step. Therefore, it is crucial for a business to understand what is happening in its environment, and Google can be one of the best data sources for this purpose.
In this tutorial, we learned how to scrape Google search results using Python. If you found this blog helpful, please share it on social media and other platforms.
Thank you!
SOCIAL SHARE CARD GENERATOR