Crawled 3,000,000 URLs and broke Vercel

• End-to-end keyword -> content -> visibility journeys


• Sitemap ingestion decision trees


• AI prompt simulation flows


• Worker-task separation for scaling


• Monorepo package boundaries

On

The platform is deployed in production with:

• Frontend: Next.js (App Router) deployed on Vercel


• Database & Auth: Supabase


• Workers: Node-based task processing (scaling toward GCP Tasks)


• CI/CD: GitHub Actions + automated branch workflows

What I Learned

1. Hard Limits Are Lazy Architecture

500 URLs felt safe.

It wasn’t. Adaptive systems scale. Static caps break.

2. AI Is a Force Multiplier; Not a Shortcut

Gemini was strongest when:

• The constraints were clear.


• The architecture goal was defined.


• The problem was decomposable.

It struggled when:

• Prompts were vague.


• Infra assumptions weren’t validated.


• Context wasn’t precise.

The biggest insight:

AI doesn’t reduce thinking.

It rewards better thinking.

3. Define Constraints Early

Scoring normalization should’ve been designed earlier.

Worker scaling should’ve been planned earlier.

I underestimated deployment complexity. I won’t again.

Google Gemini Feedback

To be perfectly candid

What Worked Exceptionally Well

• Long-context structured reasoning.


• Breaking down architectural trade-offs.


• JSON schema drafting.


• System decomposition.


• Scoring model design.


• Refactoring conceptual pipelines.

Gemini was incredibly strong at turning messy thoughts into structured systems.

Where I Hit Friction

• It assumes defaults unless corrected.


• Infra advice still needs documentation verification.


• Vague prompts produce vague output.


• Workspace-specific debugging needs exact error logs.

It’s powerful. But it’s not autonomous. And that’s fine.

The human still defines reality.

What’s Next

The near future includes the following:

• RAG-based citation validation


• Real-time AI visibility scoring


• GCP task queue scaling for worker orchestration


• Multi-tenant architecture hardening


• Platform publishing integrations


• Backlink automation engine

Soon, we'll get closer to the Phase 2 of our project, incorporating GEO.

SEO optimizes for search engines.

AEO finds out whether or not any given AI platform is citing us.

GEO figures out how to make the AI cite us in exactly the way we need.