Automating SEO with Python and Scraping API

Software as a Service costs for SEO can hit hundreds of dollars every month. Most of these platforms just wrap search data with a basic UI. You can build the same logic using Python and a scraping API. This approach gives you total control over the data. You skip the marketing and just get the insights.

Table of Contents

• Build Instead of Buy


• Recursive Keyword Discovery


• Semantic Intent Mapping


• SERP Similarity Heatmaps


• AI Overview Visibility Tracking


• Conclusion

Code Snippets and Resources

Full scripts and detailed data outputs would make this post too long to read. I am sharing the core logic and functional snippets here to show the engineering behind the tools. If you want the complete ready to run scripts and deep dives into the results you can find the full guide on the HasData Blog.

Build Instead of Buy

Search engines require JavaScript rendering and complex proxy rotation. Writing a DIY scraper is a maintenance nightmare. A better way is just using a dedicated scraping API to handle the browser headers and CAPTCHAs. This lets you focus on the data logic. You get structured JSON instead of HTML. You pay for the data you use rather than a flat monthly fee for features you might not even need.

Recursive Keyword Discovery

Most keyword tools rely on old databases. You can find real time intent by scraping Google Autosuggest recursively. The logic uses a thread pool to append characters to a seed keyword. This forces the API to reveal long tail queries.

import requests

import string

from concurrent.futures import ThreadPoolExecutor



def fetch_suggestions(query, api_key):

    url = f"https://suggestqueries.google.com/complete/search?output=toolbar&q={query}"

    payload = {"url": url, "jsRendering": False}

    headers = {"x-api-key": api_key}

    response = requests.post("https://api.hasdata.com/scrape/web", headers=headers, json=payload)

    return response.content



# Recursively expand from 'coffee a' to 'coffee az'

This method collects thousands of unique keywords in seconds. You are getting the data directly from the source.

Semantic Intent Mapping

Search intent tells you if a user wants to buy something or just learn. You can automate this by analyzing the top ten URLs for any query. Using regex to scan for paths like /shop/ or /blog/ provides a breakdown of the SERP composition.

def classify_url(url):

    if "wikipedia.org" in url:

        return "Encyclopedic"

    if "/product/" in url or "amazon.com" in url:

        return "Transactional"

    return "Informational"

If sixty percent of the results are products you know you need a landing page. This removes the guesswork from content strategy.

SERP Similarity Heatmaps

Targeting the same topic with two pages causes cannibalization. You can solve this by comparing the Jaccard Index of search results. If two keywords share many of the same URLs they belong on the same page.

def calculate_jaccard(set1, set2):

    intersection = len(set1.intersection(set2))

    union = len(set1.union(set2))

    return intersection / union

Using Seaborn to plot these scores as a heatmap makes the clusters obvious. You see exactly which topics need their own articles and which should be merged.

AI Overview Visibility Tracking

Standard rank trackers often miss AI Overviews. These summaries push organic results down the page. You can monitor this by parsing the aiOverview object in your API response. This script checks if your domain appears in the AI citations.

def check_sge_visibility(serp_data, target_domain):

    references = serp_data.get("aiOverview", {}).get("references", [])

    return any(target_domain in ref.get("link") for ref in references)

Tracking this allows you to see if you are losing traffic to AI even when your organic rankings stay high.

Conclusion

The repository at HasData Python for SEO contains the full implementation of these features.