DEV.to Scraper: pull articles by tag, author, or feed into clean JSON

Quick answer: DEV.to (built on the — the open-source publishing engine that also powers CodeNewbie and several smaller communities. Launched in 2016, DEV.to hosts millions of articles across tags like python, webdev, typescript, beginners, and ai, written by everyone from student bloggers to senior engineers.

What makes DEV.to useful as a data source:

• Every article carries structured engagement metrics: positive reactions, comments, and reading time


• Articles are tagged with community-maintained taxonomy (lowercase tags like javascript, devops, aws)


• The body of every article is available as raw Markdown — ready to embed in a vector store without stripping HTML


• Authorship is consistent: every article has a username and display_name, making per-author analysis straightforward

What the platform does not give you: a bulk export, a search-by-keyword endpoint, or a way to get all articles in a tag older than the most recent thousand.

Does DEV.to have an API? 🔌

Yes — but it has meaningful limits. DEV.to's .

Open the Apify Console and click Start, or run it programmatically with the , Pinecone, or Weaviate. The url and author_username fields give you citation metadata for free.

Trending tag dashboards. Schedule a daily run on mode=tag, tag=ai with maxResults=50. Diff today's positive_reactions_count against yesterday's — any article that gained more than 50 reactions in 24 hours is trending. Wire it into a Slack webhook and you have a free daily briefing.

Author monitoring and portfolio analysis. Pull mode=username for a specific author to mirror their full catalogue. Useful for DevRel teams tracking competitor advocates, recruiters benchmarking engineering blog authors, or writers building a personal analytics dashboard outside DEV.to's own stats page.

Newsletter assembly. Pull mode=top or mode=latest with maxResults=10, sort by positive_reactions_count, and render the top 5 to Markdown for a weekly digest. The reading_time_minutes field tells readers upfront what they're committing to.

Engagement benchmarking. Pull 500 articles in a tag, group by author_username, and compute average reactions per post — a simple "who are the most impactful writers in this niche?" query for sponsorship research, guest-post pitching, or a contributor leaderboard.

Pricing — exact numbers 💰

Pay-per-event. You pay for articles written to your dataset, nothing for the ones you don't get.

• 
  $0.005 per run (covers the Actor warm-up)


• 
  $0.002 per article written to the dataset

Pull	Cost
30 articles (default)	$0.07
100 articles	$0.21
1,000 articles	$2.01
5,000 articles	$10.01
10,000 articles	$20.01

Apify's $5 free trial credit covers your first ~2,490 articles with no credit card required. No subscription, no minimum, no charge for runs that return zero results.

The technically interesting bit

DEV.to's API officially cuts off article listings at 1,000 per tag — but the per-article GET /articles/:id endpoint has no such limit. So a full corpus is achievable by combining the listing endpoint (for IDs) with the detail endpoint (for bodies): even when the listing only goes 34 pages deep, you can supplement IDs from the username endpoint, the latest feed, or a prior run's dataset. The Actor exposes this as a design choice — mode=tag is the fast lane for recent articles; mode=latest is the slow lane for full-history accumulation over scheduled runs. Both paths produce identical row shapes, so your downstream pipeline never needs to know which mode fed it.

Limitations 🚧

• 
  Tag endpoint hard cap at ~1,000 items. The DEV.to v1 API does not paginate beyond this for the tag feed. Full-history pulls require either the username mode (per-author) or multiple scheduled latest-mode runs.


• 
  Body Markdown is the API's version. If an author used DEV.to's rich editor with embedded Liquid tags (custom video/link cards), those render as raw Liquid syntax in the Markdown — not HTML. Post-processing is on you.


• 
  No comment bodies. comments_count is in the metadata, but fetching individual comment threads would multiply the request count significantly. Not in scope for v1.


• 
  No read-time filtering at the API level. You can filter post-scrape, but the API doesn't accept a min_reading_time param. Download the dataset and filter in Pandas.


• 
  Private/draft articles are inaccessible. The public API only surfaces published, non-hidden articles.

FAQ

Is scraping DEV.to legal?

This Actor calls DEV.to's own published public API (https://developers.forem.com/api/v1) — no authentication bypassed, no HTML scraped, no undocumented endpoint hit. The API is designed for programmatic access. Standard advice: read for direct warehouse ingestion.

Does DEV.to have an official bulk-export API?

No. The .

Free $5 trial credit, no credit card. Run it on tag=python with maxResults=30 and you'll have a full typed dataset in under a minute — Markdown bodies included if you leave includeBody: true. Need a field that isn't there (comment threads, co-authors, series metadata)? Drop a note in the comments. We read every one.

Built by Devil Scrapes — the devil's in the data, and we keep it clean. 😈

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen