Clutch.co is one of the richest public directories of B2B service providers — agencies for development, marketing, design, and more, each with ratings, verified reviews, hourly rates, team size, and location. For lead generation, competitive research, or market sizing, it is a goldmine.
It is also one of the more annoying sites to scrape. If you have tried, you already know why.
Why most Clutch scrapers fail
Send a plain HTTP request to a Clutch listing page and you get back a 403 with a Cloudflare challenge — not the data. This happens on almost every request, even for robots.txt. Tools built on simple HTTP clients (requests, axios, cheerio-only actors) hit this wall and either return nothing or break intermittently.
That is the single reason most Clutch scrapers on the market are unreliable: they never actually get past the front door. You will see them "succeed" with an empty result set, which is worse than failing loudly — you think you have data, and you don't.
What actually works
The fix is not a smarter HTTP header. It is using a real browser. A headless Chromium instance (via Playwright) solves Cloudflare's managed challenge automatically within a few seconds, exactly like a normal visitor's browser does — then the full, rendered page is available.
Two things make it reliable at scale:
A real browser, not raw HTTP. Playwright/Chromium passes the challenge; a bare request never will.
Residential proxies. Datacenter IPs get flagged quickly. Rotating residential IPs keep the success rate high across many pages.
Once you are in, the data is clean and structured — about 60 companies per listing page, each with name, profile URL, review count, rating, minimum project size, hourly rate, description, locations, and the agency's own website.
The extraction, step by step
- Open the listing URL (e.g.
https://clutch.co/agencies/digital-marketing) in a headless browser with a residential proxy. - Wait for the company cards to render (the Cloudflare challenge clears on its own).
- Read each card's fields from the page.
- Paginate with
?page=N, stopping when a page returns no new companies (Clutch clamps out-of-range pages to the last valid one, so naive pagination loops on duplicates — de-dupe by the stable provider id). - Fail loudly if a run returns zero results, so you never pay for an empty dataset.
That last point matters more than it sounds. The most common complaint about existing Clutch scrapers is "it ran, said success, returned nothing." A scraper you can trust treats an empty run as an error, not a success.
A ready-made option
If you would rather not build and maintain the browser + proxy + pagination logic yourself, I published a Clutch.co scraper on the Apify Store that does exactly the above:
→ Clutch.co Agency Directory Scraper
Give it a category or location URL and it returns clean, structured company data — name, rating, reviews, min project size, hourly rate, team size, location, and website. It uses a real browser with residential proxies to pass Cloudflare, paginates automatically, de-dupes, and fails loudly on an empty run. Pricing is pay-per-result ($3.00 per 1,000 companies), so you only pay for data you actually get.
When to use it
Lead generation — build a targeted list of agencies by service and location.
Competitive research — track how competitors are rated and priced.
Market analysis — size a category by number of providers, ratings distribution, and typical rates.
Whether you build your own or use the actor, the takeaway is the same: on Clutch, the wall is Cloudflare, the key is a real browser plus residential proxies, and the difference between a scraper you can trust and one you can't is whether it fails loudly when it gets nothing.
SOCIAL SHARE CARD GENERATOR