Websites can detect headless browsers through dozens of signals. Most scraper tutorials show you how to launch Playwright — none explain why it still gets blocked. Here's what sites are actually checking and how to evade each signal.
How headless detection works
Detection is probabilistic, not binary. Sites like Cloudflare, Akamai, and DataDome maintain a "bot score" based on many signals. Each anomaly adds to your score. Enough anomalies = block.
The most common signals:
navigator.webdriver = true- Missing browser plugins
- Specific headless Chrome flags
- Canvas/WebGL fingerprint anomalies
- Chrome DevTools Protocol (CDP) exposure
- User-agent / platform inconsistencies
- Missing media codecs
- Behavioral signals (mouse movement, timing)
Signal 1: navigator.webdriver
The most obvious signal. Chrome headless sets navigator.webdriver = true by default.
How to fix:
// Playwright
const context = await browser.newContext();
await context.addInitScript(() => {
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined,
configurable: true,
});
});
# Playwright Python
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context()
await context.add_init_script("""
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
""")
Why this isn't enough: Sites also check for chrome.runtime, window.chrome, and CDP artifacts.
Signal 2: Chrome DevTools Protocol (CDP) detection
Playwright communicates with Chrome via CDP. Some sites detect open CDP connections:
// Detection technique used by some sites
const ws = window.WebSocket;
window.WebSocket = function(...args) {
if (args[0].includes('devtools')) {
throw new Error('DevTools detected!');
}
return new ws(...args);
};
Fix: Use --remote-debugging-port=0 to disable CDP discovery, or better — use camoufox which patches CDP exposure:
from camoufox.async_api import AsyncCamoufox
async with AsyncCamoufox(headless=True) as browser:
page = await browser.new_page()
await page.goto("https://bot-test.com")
# CDP not detectable from page JavaScript
Signal 3: Missing Chrome plugins and extensions
Real Chrome has plugins (navigator.plugins). Headless Chrome has zero:
// Detection: checks for plugins
if (navigator.plugins.length === 0) {
// Likely headless
}
Fix: Spoof plugins array:
// Add fake plugins
Object.defineProperty(navigator, 'plugins', {
get: () => [
{ name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
{ name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
{ name: 'Native Client', filename: 'internal-nacl-plugin' },
]
});
Signal 4: Canvas fingerprinting
Sites draw to canvas and compare the output hash. Headless Chrome produces a different hash than normal Chrome due to rendering differences.
Detection:
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
ctx.fillText('fingerprint', 10, 50);
const hash = canvas.toDataURL();
// Compare hash to known-bot hashes
Fix: Use noise injection to randomize canvas output:
# Playwright: add canvas noise
await context.add_init_script("""
const originalFillText = CanvasRenderingContext2D.prototype.fillText;
CanvasRenderingContext2D.prototype.fillText = function(...args) {
const noise = Math.random() * 0.001;
this.globalAlpha = 1 - noise;
return originalFillText.apply(this, args);
};
""")
Or use playwright-stealth which handles this automatically:
from playwright_stealth import stealth_async
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await stealth_async(page) # Patches all known detection vectors
await page.goto("https://bot.sannysoft.com")
Signal 5: Headless-specific user agent
Chrome headless used to include "HeadlessChrome" in the UA string. Now it doesn't, but other UA anomalies remain:
# BAD: Default headless UA has inconsistencies
browser = await p.chromium.launch(headless=True)
# GOOD: Use a real Chrome UA that matches the browser version
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
)
Also match the platform:
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
# navigator.platform must match the UA
extra_http_headers={"sec-ch-ua-platform": '"Windows"'}
)
Signal 6: Timing and behavioral signals
Bots move the mouse in straight lines, click instantly, and scroll at constant speeds. Sites use mouse heuristics to distinguish humans from bots.
Fix: Add realistic mouse movement:
import asyncio
import random
async def human_mouse_move(page, target_x, target_y, steps=10):
"""Move mouse in a curved path"""
box = await page.query_selector("body")
start = await page.evaluate("() => ({x: window.innerWidth/2, y: window.innerHeight/2})")
for i in range(steps):
# Bezier curve interpolation
t = i / steps
x = start['x'] + (target_x - start['x']) * t + random.gauss(0, 5)
y = start['y'] + (target_y - start['y']) * t + random.gauss(0, 5)
await page.mouse.move(x, y)
await asyncio.sleep(random.uniform(0.01, 0.05))
async def human_click(page, selector):
element = await page.query_selector(selector)
box = await element.bounding_box()
# Click slightly off-center (humans rarely click exact center)
x = box['x'] + box['width'] * random.uniform(0.3, 0.7)
y = box['y'] + box['height'] * random.uniform(0.3, 0.7)
await human_mouse_move(page, x, y)
await asyncio.sleep(random.uniform(0.1, 0.3)) # Pause before click
await page.mouse.click(x, y)
Signal 7: Chrome launch flags
Some --disable-* flags that make scraping easier also make you more detectable:
# Flags that increase detection risk:
args = [
"--disable-blink-features=AutomationControlled", # This is fine
"--disable-dev-shm-usage", # Low risk
"--no-sandbox", # INCREASES detection - only use in Docker
"--disable-setuid-sandbox", # Same
"--disable-web-security", # High detection risk - don't use
]
# Recommended minimal flags:
safe_args = [
"--disable-blink-features=AutomationControlled",
"--disable-infobars",
"--window-size=1280,800",
]
The complete stealth setup for Playwright Python
from playwright.async_api import async_playwright
import asyncio
import random
STEALTH_JS = """
// 1. Remove webdriver
Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
// 2. Add Chrome object
window.chrome = { runtime: {} };
// 3. Add fake plugins
Object.defineProperty(navigator, 'plugins', {
get: () => [
{ name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
{ name: 'Chromium PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
]
});
// 4. Add fake languages
Object.defineProperty(navigator, 'languages', {
get: () => ['en-US', 'en']
});
// 5. Fix permissions API
const originalQuery = window.navigator.permissions.query;
window.navigator.permissions.query = (parameters) => (
parameters.name === 'notifications'
? Promise.resolve({ state: Notification.permission })
: originalQuery(parameters)
);
"""
async def create_stealth_browser():
p = await async_playwright().start()
browser = await p.chromium.launch(
headless=True,
args=[
"--disable-blink-features=AutomationControlled",
"--disable-infobars",
"--window-size=1366,768",
]
)
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
viewport={"width": 1366, "height": 768},
locale="en-US",
timezone_id="America/New_York",
extra_http_headers={
"Accept-Language": "en-US,en;q=0.9",
"sec-ch-ua": '"Chromium";v="122", "Not(A:Brand";v="24"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
}
)
await context.add_init_script(STEALTH_JS)
return context
# Test against bot detection
async def test_stealth():
context = await create_stealth_browser()
page = await context.new_page()
await page.goto("https://bot.sannysoft.com")
await page.screenshot(path="stealth-test.png")
print("Check stealth-test.png for results")
asyncio.run(test_stealth())
camoufox: the nuclear option
camoufox is a Firefox fork specifically built for stealth scraping. It modifies Firefox at the binary level to eliminate detection signals:
from camoufox.sync_api import Camoufox
with Camoufox(headless=True, os="windows") as browser:
page = browser.new_page()
page.goto("https://bot.sannysoft.com")
# Passes most detection tests
Install: pip install camoufox && python -m camoufox fetch
camoufox passes Cloudflare's bot checks in headless mode — something no Playwright configuration can do reliably.
What still doesn't work
Even with perfect browser fingerprinting, some sites detect bots via:
IP reputation: residential proxies required
Account behavior: new accounts with no history get restricted
JavaScript execution patterns: some sites analyze CPU timing
Network timing: consistent request intervals are a bot signal
For the hardest sites (LinkedIn, Google, major e-commerce), managed actors handle all of this:
Apify Scrapers Bundle ($29) — pre-built actors with stealth, proxy rotation, and behavioral randomization built in.
Quick reference: detection evasion checklist
- [ ]
navigator.webdriver = undefined
- [ ] Add
window.chrome.runtime
- [ ] Spoof
navigator.plugins(≥2 plugins) - [ ] Set realistic
navigator.languages
- [ ] Match UA string to actual Chrome version
- [ ] Set
timezone_idto match locale - [ ] Use
--disable-blink-features=AutomationControlled
- [ ] Avoid
--no-sandboxin non-Docker environments - [ ] Add canvas noise injection
- [ ] Use human-like mouse movement for click actions
- [ ] Use residential proxies for IP reputation
- [ ] Or use
playwright-stealth/camoufoxfor automated patching
Related Tools
Pre-built actors for this use case: