Building a Web Crawler in Node.js to Discover AI-Powered JavaScript Repos on GitHub

for free access to a treasure trove of resources!

Step 3: Fetch and Parse the HTML

const fetchRepositories = async () => {

    try {

        // Fetch the search results page

        const { data } = await axios.get(SEARCH_URL);

        const $ = cheerio.load(data); // Load the HTML into cheerio



        // Extract repository details

        const repositories = [];

        $('.repo-list-item').each((_, element) => {

            const repoName = $(element).find('a').text().trim();

            const repoUrl = `https://github.com${$(element).find('a').attr('href')}`;

            const repoDescription = $(element).find('.mb-1').text().trim();



            repositories.push({

                name: repoName,

                url: repoUrl,

                description: repoDescription,

            });

        });



        return repositories;

    } catch (error) {

        console.error('Error fetching repositories:', error.message);

        return [];

    }

};

Here’s what’s happening:

• 
  Fetching HTML : The axios.get method retrieves the search results page.


• 
  Parsing with Cheerio : We use Cheerio to navigate the DOM, targeting elements with classes like .repo-list-item.


• 
  Extracting Details : For each repository, we extract the name, URL, and description.

Step 4: Display the Results

Finally, call the function and log the results:

(async () => {

    const repositories = await fetchRepositories();

    console.log('AI-Powered JavaScript Repositories Found:', repositories);

})();

Part 4: Running the Crawler

Save your script and run it with Node.js:

node crawler.js

You’ll see a list of AI-related JavaScript repositories, each with its name, URL, and description, neatly displayed in your terminal.

Part 5: Enhancing the Crawler

Want to take it further? Here are some ideas:

1. 
   Pagination : Add support for fetching multiple pages of search results by modifying the URL with &p=2, &p=3, etc.


2. 
   Filtering : Filter repositories by stars or forks to prioritize popular projects.


3. 
   Saving Data : Save the results to a file or database for further analysis.

Example for saving to a JSON file:

const fs = require('fs');

const saveToFile = (data) => {

    fs.writeFileSync('repositories.json', JSON.stringify(data, null, 2));

    console.log('Data saved to repositories.json');

};



// Call saveToFile after fetching repositories

(async () => {

    const repositories = await fetchRepositories();

    saveToFile(repositories);

})();

The Beauty of Automation

With this crawler, you’ve automated the tedious task of finding relevant repositories on GitHub. No more manual browsing or endless clicking—your script does the hard work, presenting the results in seconds.

For more tips on web development, check out DailySandbox and sign up for our free newsletter to stay ahead of the curve!