How to Build a Competitor Monitoring Pipeline in 30 Minutes
Stop checking competitor prices manually. Here's the exact stack I use to monitor 50+ competitor URLs in real-time, without getting blocked.
What We're Building
A system that:
- Checks competitor URLs every 6 hours
- Extracts prices, titles, and availability
- Alerts you only when something significant changes
- Doesn't get blocked by anti-bot protection
The Stack
XCrawl — residential proxies + scraping API
Airtable — free, simple data storage with nice UI
GitHub Actions — free cron scheduling
Telegram bot — instant alerts on your phone
Total cost: under $20/month for 50 URLs.
Step 1: Define What You're Monitoring
Create a simple Airtable table with columns:
url— competitor product page
name— competitor or product name
last_price— price from last check
last_checked— timestamp
change_alert— threshold % for alerts
Start with 10-20 URLs. You can always add more later.
Step 2: The Scraping Function
CODE
const { XCrawlScraper } = require('xcrawl-scraper');
const xcrawl = new XCrawlScraper({
apiKey: process.env.XCRAWL_API_KEY,
});
async function checkCompetitor(url) {
try {
const result = await xcrawl.scrapeMarkdown(url, {
render: true,
});
const markdown = result.data.markdown;
const priceMatch = markdown.match(/\$[\d,]+\.?\d*/);
const price = priceMatch
? parseFloat(priceMatch[0].replace(/[$,]/g, ''))
: null;
const titleMatch = markdown.match(/#\s+(.+)/);
const title = titleMatch ? titleMatch[1].trim() : 'Unknown';
return {
url,
price,
title,
checkedAt: new Date().toISOString(),
success: true,
};
} catch (err) {
return {
url,
price: null,
title: null,
checkedAt: new Date().toISOString(),
success: false,
error: err.message,
};
}
}
Step 3: Alert Logic
You don't want to be alerted every time a price changes by $0.01. You want to know when something significant happens:
CODE
function shouldAlert(lastPrice, currentPrice, threshold = 5) {
if (!lastPrice || !currentPrice) return false;
const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);
return change >= threshold;
}
function formatAlert(competitor, lastPrice, currentPrice) {
const direction = currentPrice < lastPrice ? 'DROPPED' : 'INCREASED';
const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);
return direction + ' ' + competitor.name + '
' +
'Price: $' + lastPrice + ' -> ' + currentPrice + ' (' + change.toFixed(1) + '% change)
' +
'URL: ' + competitor.url;
}
Step 4: GitHub Actions Scheduling
Create .github/workflows/competitor-check.yml:
CODE
name: Competitor Price Monitor
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
workflow_dispatch: # Manual trigger
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install xcrawl-scraper airtable
- name: Run competitor checks
env:
XCRAWL_API_KEY: ${{ secrets.XCRAWL_API_KEY }}
AIRTABLE_API_KEY: ${{ secrets.AIRTABLE_API_KEY }}
run: node competitor-check.js
- name: Send alert to Telegram
if: matrix.changed == true
env:
TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
run: |
curl -s -X POST "https://api.telegram.org/bot$TG_BOT_TOKEN/sendMessage" -d "chat_id=$TG_CHAT_ID" -d "text=${{ matrix.alertMessage }}"
Step 5: The Full Check Loop
CODE
async function runChecks() {
const competitors = await airtable.fetchCompetitors();
const alerts = [];
for (const competitor of competitors) {
const result = await checkCompetitor(competitor.url);
if (result.success) {
await airtable.updateRecord(competitor.id, {
last_price: result.price,
last_checked: result.checkedAt,
last_title: result.title,
});
if (shouldAlert(competitor.last_price, result.price)) {
alerts.push(formatAlert(competitor, competitor.last_price, result.price));
}
} else {
console.log('Failed: ' + competitor.url + ' - ' + result.error);
}
// Be respectful - don't hammer the servers
await sleep(2000 + Math.random() * 3000);
}
if (alerts.length > 0) {
await sendTelegramAlerts(alerts);
}
console.log('Checked ' + competitors.length + ' competitors. ' + alerts.length + ' alerts.');
}
What This Actually Looks Like in Practice
Here's the alert you'd get on your phone:
CODE
DROPPED Amazon Basics Mouse
Price: $29.99 -> $24.99 (16.7% change)
URL: amazon.com/dp/B07ZVPQR5M
That's it. No dashboards to check. No manual research. Just the signal you need to act.
Key Decisions That Made This Work
Residential proxies — datacenter IPs get blocked immediately on Amazon, Best Buy, etc.
6-hour check interval — fast enough to catch changes, slow enough to not look like a bot
Store everything in Airtable — the UI makes it easy to spot patterns over time
Telegram over email — faster, and your phone will buzz to wake you up if needed
The ROI
My client monitors 47 competitor URLs across 8 product categories. In the first month:
- Caught 3 competitor price drops within hours
- Adjusted their own pricing proactively
- Estimated incremental revenue: $2,400 that month
System cost: $49/month (XCrawl) + free (GitHub Actions, Airtable, Telegram).
Building something similar? I've open-sourced the monitoring template. Link in bio. Questions below.
Vollständiger Original-Bericht
Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR