In this full guide, you’ll learn:
- 📛 Why , and for AI agent automation without user’s assistance manually.
, , and declare you’ve built an autonomous agent.
await page.goto(url)
await page.click(selector)
await page.type(selector, text)
In reality, you’ve ONLY automated a browser. Commercial sites don’t gauge how intelligent your agent is. They judge whether they believe your browser is genuine.
Before a page even finishes loading, they inspect what your browser actually is: the , cookies, device characteristics, and even the rhythm of your connection. Dozens of signals are examined in the time it takes the page to start loading.
If those signals don’t look authentic, your agent rarely reaches the real application. Instead, it encounters CAPTCHA challenges, verification pages, that quietly return entirely different content.
I’ve always wanted to is meaningless.
Fortunately, solving this problem doesn’t require .
Instead, we’ll let
The entire project is built with plain JavaScript. No proprietary framework. No billion-dollar AI platform. Just Node.js, Playwright, Bright Data, and Gemini working together.
Here’s a quick overview of what we’re going to build:
🍋🟩Note: The complete source code is available on GitHub at the end of this article.
At a high level, the agent follows this workflow:
🪪 Prerequisite
- Basic JavaScript/TypeScript or Python. Even though I used javascript all over this project, you can follow through this article if you know any programming language. But familiarity with asynchronous programming (
async/await) is required, as both scraping and AI calls rely heavily on it. - Fundamental Node.js or Python Environment: Ability to install packages via
npmorpipand manage local environment variables (.env). - API Keys: Active accounts and API keys for both
Step 2: 🗝️ Get Gemini API Key
Go to:
Click on the Top right corner “Create API” and choose option labeled Browser API:
This is Because:
We are NOT launching:
CODEchromium.launch()
Instead:
CODEchromium.connectOverCDP()
Meaning: Your browser lives in the cloud.
To help you understand lets write a simple code block to browse CNN’s title from our local terminal using Playwright and Bright Data.
Step 4: 🦧 Connect Playwright to Bright Data
Inside your index.js file write that below to access CNN title from terminal:
CODEimport { chromium }
from "playwright";
const BRIGHT_WS_ENDPOINT = "your_endpoint"
const browser =
await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT
);
const page =
await browser.newPage();
await page.goto(
"https://cnn.com"
);
console.log(
await page.title()
);
If you run this on the terminal, you will see a real CNN website title has been shown. We successfully extracted information and displayed the result:
So
connectOverCDP()does NOT launch a browser locally.
Instead, this is what happened:
Step 2: Setup Playwright to connect to Bright Data’s WebSocket
As I have already shown you how to set up Bright Data with Playwright earlier, I will go straight to searching for e-commerce products.
Note: Unlike a Playwright’s normal local browser chromium.launch() we used chromium.connectOverCDP(). This browser runs inside Bright Data’s infrastructure. IP rotation, browser fingerprinting, and anti-bot protection are handled automatically. Your local machine only sends instructions.
To test our playwright and bright data working properly, let’s search for “gaming laptops” from amazon:
CODE//setup Playwright to connect to Bright Data's WebSocket endpoint and navigate to Amazon search results for "gaming laptop"
async function main(){
const browser = await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT);
const page = await browser.newPage();
const query = "gaming laptop";
const searchUrl = `https://www.amazon.com/s?k=${encodeURIComponent(query)}`;
console.log("Navigating Bright Data Playwright to:", searchUrl);
await page.goto(searchUrl, { waitUntil: "domcontentloaded" });
const title = await page.title();
console.log("Amazon search page title:", title);
console.log("Bright Data navigation complete for query:", query);
await browser.close();
}
main();
Here is our terminal output 👇:
At this point, SIKKI has successfully observed the environment.
Now we give it the ability to reason.
Step 4: Ask Gemini To Analyze The Market
Create a prompt:
CODEconst prompt = `You are an ecommerce analyst.
Analyze these products:
${JSON.stringify(products)}
Return:
1. Cheapest Product
2. Average Market Price
3. Best Rated Product
4. Competitor Comparison
5. Final Recommendation
`;
Generate:
CODEconst result = await model.generateContent(prompt);
const analysis =
result.response.text();
Terminal output:
Step 6: 📸Teach SIKKI To Capture Evidence
Now that SIKKI can scrape product metadata, the next job is proof.
Capture evidence by taking:
- one search-results summary screenshot
- a product card screenshot for the top result
- direct product page screenshots for the highest-ranked items
Here is how:
CODEconst searchScreenshot = path.join(screenshotDir, "search-results.png");
await page.screenshot({ path: searchScreenshot, fullPage: true });
const summaryPath = path.join(screenshotDir, "top-product-summary.png");
await cards[0].screenshot({ path: summaryPath });
Result:
after a successful analysis and saving a screenshot inside the “screenshot” folder:
Step 9: Generate A Dashboard Summary
Finally, create a human-readable summary file with the important metrics.
Compute:
- total products extracted
- cheapest offer
- average price
- best rated product
- top 5 product summary
Example:
CODEconst priced = enriched.filter((p) => !isNaN(p.priceValue));
const cheapestObj = priced.length ? priced.reduce((a, b) => (a.priceValue <= b.priceValue ? a : b)) : null;
const avgPriceVal = priced.length ? (priced.reduce((s, p) => s + p.priceValue, 0) / priced.length).toFixed(2) : “N/A”;
const bestRatedObj = enriched.slice().sort((a, b) => {
const ra = parseFloat((a.rating || “”).split(“ “)[0]) || 0;
const rb = parseFloat((b.rating || “”).split(“ “)[0]) || 0;
return rb — ra;
})[0] || null;
Then write:
CODEconst dashboard = [];
dashboard.push(“==================================”);
dashboard.push(“SIKKI MARKET DASHBOARD”);
dashboard.push(“==================================”);
dashboard.push(`Query: ${query}`);
dashboard.push(`Products Extracted: ${enriched.length}`);
dashboard.push(`Cheapest Product: ${cheapestObj ? `${cheapestObj.title} | ${cheapestObj.price}` : “N/A”}`);
dashboard.push(`Average Price: ${avgPriceVal === “N/A” ? “N/A” : `$${avgPriceVal}`}`);
dashboard.push(`Best Rated: ${bestRatedObj ? `${bestRatedObj.title} | ${bestRatedObj.rating || “N/A”}` : “N/A”}`);
dashboard.push(“\nTop 5 Products:”);
enriched.slice(0, 5).forEach((p, i) => {
dashboard.push(`${i + 1}. ${p.title} | ${p.price} | ${p.rating || “N/A”} | ${p.url || “N/A”}`);
});
fs.writeFileSync(dashboardPath, dashboard.join(“\n”), “utf8”);
Lets have a look at the result:
🏌️Where To Go From Here
This article intentionally focused on the foundations. But SIKKI can become significantly more powerful. You can extend it with:
- 🤹Multi-site product comparison
- 🔎Competitor tracking
- ⏰Scheduled market reports
- 📊Interactive dashboards
- 📈Price trend monitoring
- 🛍️Autonomous shopping assistants
- 🗃️Agent memory using databases
- 🍶Multi-agent workflows
You could even create an agent that monitors hundreds of e-commerce websites every day and sends market intelligence directly to your inbox.
The possibilities become surprisingly large once you stop thinking of browser automation as scraping and start thinking about it as an intelligent system.
. I am offering a 50% discount using the code “earlybird” during checkout ;only for the first 50 copies! Plus, enjoy a 30-day money-back guarantee. no risk, all reward.
🏁Conclusion
Many developers think AI Agents begin with LLMs. I think they begin with provides a production-ready browser infrastructure that handles these challenges behind the scenes that allows your AI agent to interact with websites more reliably while you focus on building application logic.
2. Do I still need Playwright if I’m using Bright Data?
Yes. Bright Data and Playwright solve different problems. provides a cloud browser environment designed to access modern websites reliably. Together they create a much more robust browser automation stack than using either tool alone.
3. What kinds of AI agents can I build with Bright Data?
Once you have reliable browser access, the possibilities extend far beyond web scraping. The same architecture can power shopping assistants, market research agents, competitor monitoring systems, travel assistants, browser testing agents, lead generation workflows, news monitoring, and autonomous research assistants. By combining an LLM for reasoning, Playwright for browser control, and Bright Data for browser infrastructure, you can build AI agents capable of operating on real websites with minimal changes to the overall architecture.
↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR