Introduction
I was recently made aware of a waitlist opening up for a new service in my local area. I thought it would be a nice exercise to automate the registration process and secure one of the first places on the list.
Registration System
The online form already existed as it was used to register for a number of different services. The provider announced that there would be a new option added to the form in the coming days.
My plan was to watch the site for any changes. As soon as the new checkbox option appeared, my script would fill out the rest of the form with my details, check the new input and submit.
Since the form was loaded with javascript, I would use a headless browser to monitor the page for changes using Puppeteer. I also decided to implement an SMS notification system using Twilio. This would notify me if there were any issues submitting the form.
Plan
- Load the target web page in a headless browser.
- Check if there are more checkboxes on the page than before.
- If not, repeat steps 1 and 2.
- If so, open a browser window, fill out and submit the form.
- Send an SMS notification.
Let's Begin
Step 1. Initialize Local Project
Create a new folder and then navigate to it and run the following commands from the terminal:
npm init
npm i puppeteer
touch index.js
Step 2. Create Browser Instance
Create a headless browser instance using Puppeteer to check the target web page.
const URL = <TargetURL>
const headlessBrowser = await puppeteer.launch({ headless: true })
const headlessPage = await headlessBrowser.newPage()
await headlessPage.goto(URL)
Step 3. Check for Updates
Develop a function to check the page for the expected condition. There are many different ways to compare pages. I checked if there was a new checkbox on the page using CSS selectors. The function returns true if the number of checkboxes on the page is greater than the number provided as an argument. You could also check the text of the page for diffs or use regex to search for matches.
async function hasAdditionalCheckbox(initialCount) {
await headlessPage.reload({
waitUntil: 'load',
})
const checkboxCount = await headlessPage.evaluate(() => {
return document.querySelectorAll("input[type='checkbox']").length
})
return checkboxCount > initialCount
}
By setting waitUntil to "load", we instruct Puppeteer to wait for the load event of the page to be fired before proceeding. This means Puppeteer will ensure that all resources, including scripts, stylesheets, and images, have been fully loaded before moving on to the next line of code where we select form elements.
If you don't need to access javascript elements it is much easier, as you can simply use Node's native
fetchmethod to grab the HTML of a page or to fetch a response from an API. In that case Puppeteer is overkill for this function.
Step 4. Create a function to autofill and submit the form.
This will fill out the form using a combination of Puppeteer's page interaction APIs and regular CSS selectors. The click and type functions are self-explanatory. The evaluate function takes a callback which allows you to run javascript inside of the page.
function autofillForm() {
const browser = await puppeteer.launch({ headless: false })
const page = await browser.newPage()
await page.goto(URL, {
waitUntil: 'load',
})
await page.type('#date-picker', '22/07/1989')
await page.click('#radio-button-2')
await page.type('#name', 'John Doe')
await page.type('#email', '[email protected]')
await page.evaluate(() => {
const checkboxes = document.querySelectorAll('input[type="checkbox"')
checkboxes.forEach((checkbox) => {
const label = checkbox.previousElementSibling
if (label.innerText.includes(<expectedText>)) {
checkbox.checked = true
}
})
})
await page.click('#submit-btn')
}
I created a new browser instance using the options argument headless: false in this function because I wanted to see the confirmation of my form submission on my screen. It would also have been fine to use the existing headless instance and then parse the result of the form submission in Node.
Step 5. Set up SMS notifications with Twilio.
You may want to be notified when your conditional is triggered. In my case, this was to check if the form had been submitted correctly. Another use-case would be if your form requires manual completion of a captcha or entry of sensitive billing information.
Go to https://www.twilio.com/ and create a new account. Once you have verified your phone number, go to the console and navigate to the messages section. You should see a heading titled "Send your First SMS" with some code snippets below. Click the Node.js tab and copy the code snippet provided.
This will include your API keys but it's fine since we'll be running the app locally. If you are hosting your script in a public repo or sharing it you should place these in an environmental variable.
In your project directory, run npm i twilio
Create a new async function called sendSMS and paste in the snippet from Twilio. It should look something like this:
async function sendSMS() {
const accountSid = process.env.ACCOUNT_SID
const authToken = process.env.AUTH_TOKEN
const client = twilio(accountSid, authToken)
const message = await client.messages.create({
body: 'Waitlist is open!',
from: '<yourTwilioPhoneNumber',
to: '<yourPhoneNumber>',
})
}
You can also trigger an email or a phone call if preferred.
Step 6. Putting it All Together
Now we can put everything in an interval:
const waitlistScraper = setInterval(async () => {
const isWaitlistOpen = await hasAdditionalCheckbox(3)
if (isWaitlistOpen) {
autoFillForm()
sendSMS()
clearInterval(waitlistScraper)
}
}, 5000) // 5 seconds
And we're done! Use the command node index.js when you are ready to run the application. You may need to adjust your machine's sleep settings to ensure that the script stays running. Mac users can use the command caffeinate to achieve this.
Conclusion
This project demonstrates the flexibility of automation through web scraping and browser control. Such techniques can be applied to various scenarios, making it a valuable skill in web development.
Whether you're booking tickets, monitoring product availability, or gathering data, these skills can greatly enhance your productivity. Plus, they're fun little projects, especially when they save you from staying up all night refreshing a page!
SOCIAL SHARE CARD GENERATOR