What happens when you refuse to accept "the source is down" as an answer.

The problem

GitHub's trending page is where millions of developers find new projects every day. But GitHub has never shipped an official API for it — if you want that data programmatically, you have to scrape the HTML.

Every existing scraper I found had the same design: one source, one point of failure. Small HTML change from GitHub → your service is 503 for a week. Rate limited from Cloudflare → same result.

I wanted a trending API that stays up when GitHub's page is down. So I built Hydra, and put it live at hydra9.dev.

The architecture (in one picture)

┌───────────────┐

Request ───▶ │ Router │

└───────┬───────┘

│ tries in order, breaker-guarded

┌───────────────┼────────────────┬─────────────────┐

▼ ▼ ▼ ▼

┌────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐

│Track A │ │ Track B │ │ Track C │ │ Track D │

│HTML │ │Community │ │Search API │ │gitstar- │

│scrape │ │mirror │ │(created:>7d) │ │ranking.com │

└────────┘ └──────────┘ └──────────────┘ └──────────────┘

Any track dead? Circuit breaker opens, next request routes past it.

All tracks dead? Serve most recent Postgres snapshot + fire Prometheus alert.

Four independent sources. A track that fails 3 times in a rolling window gets its breaker opened for 60 seconds — the router jumps to the next track without adding latency to the user's request.

Why 4 sources and not 2

Two sources feels like enough right up until they share a failure mode. The four I picked are architecturally independent:

Track A (github.com/trending HTML): freshest data, but the format changes without warning

Track B (ghapi.huchen.dev community mirror): a nice person maintains this, but it can go quiet for hours

Track C (GitHub Search API created:>7d sort:stars): the most stable, but it's reconstructed trending — not what appears on the actual page

Track D (gitstar-ranking.com): different data model entirely (lifetime stars), useful only as a last-resort cross-check

If A breaks because of a GitHub HTML change, B doesn't (it has its own parser). If B breaks because the maintainer's server goes down, C is a completely different infrastructure at GitHub. If C rate-limits you, D is a totally different site.

Historical data — the thing GitHub won't give you

The official trending page is a snapshot in time. If you refresh, yesterday's list is gone forever.

Hydra runs a Celery beat task every 15 minutes that stores a fresh snapshot to Postgres. That unlocks four analytics endpoints the official page can't offer:

Repos that showed up today but weren't there yesterday

curl

Ranked by stars/hour instead of cumulative fame

curl

velocity in particular has been my favorite — it surfaces projects that are exploding right now, not projects that were famous 6 months ago and still coasting.

The stack (deliberately boring)

API: FastAPI (Python 3.11+)

Task queue: Celery + Redis (broker on db 1, results on db 2)

Storage: PostgreSQL 16 + Alembic

HTTP client: httpx (async everywhere)

HTML parser: selectolax (faster than BeautifulSoup)

Observability: Prometheus + Grafana + 7 alert rules

Deploy: Docker Compose (single file)

The whole thing is 52 Python files and 47 tests that run in 6 seconds.

One lesson: @lru_cache and Pydantic don't mix

The single bug that took me from "green in dev" to "500 in prod" was this innocent-looking function:

@lru_cache

def track_singletons(settings: Settings) -> dict[TrackId, Track]:

return {

TrackId.A: TrackA(settings),

TrackId.B: TrackB(settings),

TrackId.C: TrackC(settings),

TrackId.D: TrackD(settings),

}

Pydantic's BaseSettings is mutable and doesn't implement __hash_. lru_cache needs hashable arguments. Every real request threw:

TypeError: unhashable type: 'Settings'

Fix: since get_settings() is itself an @lru_cache'd singleton, the parameter was redundant. Call it inside:

@lru_cache

def _track_singletons() -> dict[TrackId, Track]:

settings = get_settings()

return {...}

Silly bug. The tests didn't catch it because they patched Settings with a mock that was hashable. Lesson: your test doubles should mirror the real thing's flaws, not paper over them.

Try it

Live:

Source (MIT):

cd Hydra && cp .env.example .env

docker compose -f infra/docker-compose.prod.yml up -d --build

curl http://localhost:8000/api/v1/trending

If it's useful, a ⭐ on the repo goes a long way. If you have a use case where you'd want more analytics (release velocity? topic clustering? language-specific momentum?), open an issue.

Hydra runs on a single 6GB VPS shared with another project. If the public instance gets hammered, self-hosting is a Docker Compose away.