Have you ever encountered a web page requiring actions like “clicking a button” to reveal more content? Such pages are called "dynamic webpages," as they load more content based on user interaction. In contrast, static webpages display all their content at once without requiring user actions.
Scraping content from dynamic pages can be daunting as it requires simulating user interactions, such as clicking a button to access additional hidden content. In this tutorial, you'll learn how to scrape data from a webpage with infinite scrolling via a "Load more" button.
Prerequisites
To follow along with this tutorial, you need:
: For parsing HTML
.
In addition, you’ll need to have a basic understanding of HTML, CSS, and JavaScript. You’ll also need a web browser like
Next, run the following command in the terminal to install the needed packages for this build.
$ npm install cheerio puppeteer
Create a new file inside your project folder in the code editor and name it dynamicScraper.js.
Excellent work, buddy!
Accessing the Content of the Page
Puppeteer is a powerful Node.js library that allows you to control headless Chrome browsers, making it ideal for interacting with webpages. With Puppeteer, you can target a webpage using the URL, access the contents, and easily extract data from that page.
In this section, you’ll learn how to open a page using a headless browser, access the content, and retrieve the HTML content of that page. You can find the target website for this tutorial (IIFE) method to make things much faster.
Define an
The browser loads up, Puppeteer fetches its entire HTML content, and Console logs the content to the terminal.
Here’s the output you should get in your terminal:
<!DOCTYPE html>
<html lang="en">
<head>
<title>Load More Button Challenge - ScrapingCourse.com</title>
</head>
<body>
<header>
<!-- Navigation Bar -->
<nav>
<a href="/">
<img src="logo.svg" alt="Logo">
<span>Scraping Course</span>
</a>
</nav>
</header>
<main>
<!-- Product Grid -->
<div id="product-grid">
<div class="product-item">
<a href="/ecommerce/product/chaz-kangeroo-hoodie">
<img src="mh01-gray_main.jpg" alt="Chaz Kangeroo Hoodie">
<span class="product-name">Chaz Kangeroo Hoodie</span>
<span class="product-price">$52</span>
</a>
</div>
<div class="product-item">
<a href="/ecommerce/product/teton-pullover-hoodie">
<img src="mh02-black_main.jpg" alt="Teton Pullover Hoodie">
<span class="product-name">Teton Pullover Hoodie</span>
<span class="product-price">$70</span>
</a>
</div>
<!-- Additional products (3-12) follow the same structure -->
</div>
<!-- Load More Button -->
<div id="load-more-container">
<button id="load-more-btn">Load more</button>
</div>
</main>
</body>
</html>
Note that the code structure above is what your output should look like.
Wow! You should be proud of yourself for getting this far. You’ve just completed your first attempt at scraping the contents of a webpage.
Simulate the LOad More Products Process
Here, you want to access more products, and to do that, you need to click on the “Load more” button multiple times until you’ve either exhausted the list of all products or gotten the desired number of products you want to access.
To access this button and click on it, you must first locate the element using any CSS selectors (the class, id, attribute of the element, or tag name).
This tutorial aims to get at least 48 products from the
Selecting the inspect option will open up developer tools just like the page below:
library to convert the parsed data into its corresponding CSV format.
Start by importing the required modules.
Node.js provides the file system (fs) module for file handling, such as writing data to a file. After importing the fs module, you should destructure the parse() method from the json2csv library.
const fs = require('fs');
const { parse } = require('json2csv');
CSV files usually require column headers; carefully write this in the same order as your parsed information. Here, the parsed data is the products array, where each element is an object with four keys (name, price, image, and link). You should use these object keys to name your column headers for proper mapping.
Define the fields (Column headers) for your CSV file:
// Define CSV fields
const fields = ['name', 'price', 'image', 'link'];
Now that you’ve defined your fields, the following line of action is to convert the current parsed information to a CSV format. The parse() method works in this format: parse(WHAT_YOU_WANT_TO_CONVERT, { YOUR_COLUMN_HEADERS }).
// Convert JSON to CSV
const csv = parse(products, { fields });
You now have to save this CSV information into a new file with the .csv file extension. When using Node.js, you can handle file creation using the writeFileSync() method on the fs module. This method takes two parameters: the file name and the data.
// Save CSV to a file
fs.writeFileSync('products.csv', csv);
Your complete code for this section should look like this:
const puppeteer = require('puppeteer');
const cheerio = require('cheerio');
const fs = require('fs');
const { parse } = require('json2csv');
(async () => {
const browser = await puppeteer.launch({ headless: false }); // Launch Puppeteer
const page = await browser.newPage(); // Open a new page
// Navigate to the website
await page.goto('https://www.scrapingcourse.com/button-click', {
waitUntil: 'networkidle2',
});
// Click "Load More" 3 times to load all products
for (let i = 0; i < 3; i++) {
try {
await page.waitForSelector('#load-more-btn', { visible: true });
await page.click('#load-more-btn');
await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for 2 seconds
} catch (error) {
console.log('No more "Load More" button or an error occurred:', error.message);
break;
}
}
// Get the final HTML content
const html = await page.content();
// Use Cheerio to parse the product data
const $ = cheerio.load(html);
const products = [];
$('.product-item').each((_, element) => {
const name = $(element).find('.product-name').text().trim();
const price = $(element).find('.product-price').text().trim();
const image = $(element).find('.product-image').attr('src');
const link = $(element).find('a').attr('href');
products.push({
name,
price,
image,
link,
});
});
console.log(`Total products parsed: ${products.length}`);
// Convert product information to CSV
try {
// Define CSV fields
const fields = ['name', 'price', 'image', 'link'];
// Convert JSON to CSV
const csv = parse(products, { fields });
// Save CSV to a file
fs.writeFileSync('products.csv', csv);
console.log('Product information exported to products.csv');
} catch (error) {
console.error('Error exporting to CSV:', error.message);
}
await browser.close(); // Close the browser
})();
You should see an automatic addition of a file named products.csv to your file structure once you save and run the script.
The output - products.csv:
Conclusion
This tutorial delved into the intricacies of scraping data from a page that requires simulation to access its hidden contents. You learned how to perform web scraping on dynamic pages using Node.js and some additional libraries, parse your scraped data into a more organized format, and unpack it into a CSV file.
SOCIAL SHARE CARD GENERATOR