Note: Not an output of chatGPT/ LLM
Data scraping is process of collecting data from public web sources and it is mostly done using script in a automated way. Due to automation, often collected data have errors and need to filter out and clean for use. However, it will be better if scraped data can be validate during scraping.
Considering the data validation requirement, most of scraping framework like Scrapy have inbuilt pattern that can be used for data validation. However, many a time, during the data scraping process, we often just use general purpose modules like requests and beautifulsoup for scraping. In such case, it is hard to validate the collected data, so this blog post explain a simple approach for data scraping with validation using Pydantic.
Plan of scraping :
In this blog, we will scrap quotes from the quotes site.
We will use requests and beautifulsoup to get the data Will create a pydantic data class to validate each scraped data Save the filtered and validated data in a json file.
For better arrangement and understanding, each step is implemented as a python method that can be used under main section.
Basic import
import requests # for web request
from bs4 import BeautifulSoup # cleaning html content
# pydantic for validation
from pydantic import BaseModel, field_validator, ValidationError
import json
1. Target site and getting quotes
We are using (
Below method is a general script to get html content for a given url.
def get_html_content(page_url: str) -> str:
page_content =""
# Send a GET request to the website
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
page_content = response.content
else:
page_content = f'Failed to retrieve the webpage. Status code: {response.status_code}'
return page_content
2. Get the quote data from scraping
We will use requests and beautifulsoup to scraped the data from given urls. The process is broken into three parts: 1) Get the html content from the web 2) Extract the desired html tags for each targeted fields 3) Get the values from each tags
def get_tags(tags):
tags =[tag.get_text() for tag in tags.find_all('a')]
return tags
def get_quotes_div(html_content:str) -> str :
# Parse the page content with BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Find all the quotes on the page
quotes = soup.find_all('div', class_='quote')
return quotes
Below script get the data point from each quote's div.
# Loop through each quote and extract the text and author
for quote in quotes_div:
quote_text = quote.find('span', class_='text').get_text()
author = quote.find('small', class_='author').get_text()
tags = get_tags(quote.find('div', class_='tags'))
# yied data to a dictonary
quote_temp ={'quote_text': quote_text,
'author': author,
'tags':tags
}
3. Create Pydantic dataclass and Validate the data for each quote
As per each fields of the quote, create a pydantic class and use same class for data validation during data scraping.
The pydantic model Quote
Below is the Quote class that is extended from BaseModel having three fields like quote_text, author, and tags. Out of these three, quote_text and author are type of string (str) and tags is a list type.
We have two validator methods (with decorators):
1) tags_more_than_two () : Will check that it must have more than two tags. (it is just for example, you can have any rule here)
2.) check_quote_text(): This method will remove "" from quote and test for text.
class Quote(BaseModel):
quote_text:str
author:str
tags: list
@field_validator('tags')
@classmethod
def tags_more_than_two(cls, tags_list:list) -> list:
if len(tags_list) <=2:
raise ValueError("There should be more than two tags.")
return tags_list
@field_validator('quote_text')
@classmethod
def check_quote_text(cls, quote_text:str) -> str:
return quote_text.removeprefix('“').removesuffix('”')
Getting and validating data
Data validation is very easy with pydantic, for example, below code, pass scraped data to pydantic class Quote.
quote_data = Quote(**quote_temp)
def get_quotes_data(quotes_div: list) -> list:
quotes_data = []
# Loop through each quote and extract the text and author
for quote in quotes_div:
quote_text = quote.find('span', class_='text').get_text()
author = quote.find('small', class_='author').get_text()
tags = get_tags(quote.find('div', class_='tags'))
# yied data to a dictonary
quote_temp ={'quote_text': quote_text,
'author': author,
'tags':tags
}
# validate data with Pydantic model
try:
quote_data = Quote(**quote_temp)
quotes_data.append(quote_data.model_dump())
except ValidationError as e:
print(e.json())
return quotes_data
4. Store the data
Once data is validated that will be save to a json file. (A general purpose method is written that will convert Python dictionary to json file)
def list_of_dict_to_json(list_data_dict:list, filename:str)-> None:
"""This is a utiltiy method to write python dictonary data
to a json file.
Args:
list_data_dict (list): Python list of dict having the data
filename (str): Filename for the json file
example_data = [
{'name':'ajit', 'age':30},
{'name':'Raushan', 'age':30}
]
list_of_dict_to_json(example_data, 'names') # output: names.json
"""
# check given filename, if ends with .json or not
if not filename.endswith('.json'):
filename = filename+'.json'
# Write data to a JSON file
with open(filename, 'w') as json_file:
json.dump(list_data_dict, json_file, indent=4)
Putting all together
After understanding each piece of scraping, now , you can put all together and run the scraping for data collection.
if __name__ == '__main__':
# URL of the Quotes to Scrape website
url = 'http://quotes.toscrape.com/'
# 1. get the content
page_html = get_html_content(url)
# 2. get the div for each quote
quotes_div = get_quotes_div(page_html)
# 3. validate and get the data point
quotes_data = get_quotes_data(quotes_div)
# 4. Store the data to json
list_of_dict_to_json(quotes_data, 'quotes')
Note: A revision is planned, let me know your idea or suggestion to include in the revised version.
Links and resources:
SOCIAL SHARE CARD GENERATOR