Why Computer Use Agents (CUAs) Fail
When we set out to build AI tools for healthcare, we had to integrate with a maze of legacy systems that either had no public APIs or charged absurd fees to access their internal ones. Multiple customers were ready to pay us, as long as our AI could work with their EHRs, CRMs, and internal portals. Out of desperation, we turned to Computer Use Agents — the tech everyone promises will “solve browser automation.”
Here’s what we actually found:
Too slow — they reload entire pages just to extract a single field
Too expensive — you’re burning tokens just to click a button
Break constantly — especially when confronting date pickers 📅🤦 (I once looked through the reasoning logs and the LLM appeared to have briefly contemplated its own existence.)
CUAs promise generalizability, but modern web apps aren’t built for systems that take discrete screenshots and spend seconds reasoning before each click. They’re designed for a human’s continuous perception, attention, and reaction time. By the time a CUA decides what to do, the DOM has shifted, the element has re-rendered, and the button it planned to click has already disappeared.
Our Insight: Web Apps Already Reveal Their Own APIs
The breakthrough for us was realizing that every modern web app must communicate with backend services — and that communication is inherently structured. The browser already knows how to authenticate, fetch data, submit forms, paginate results, and trigger workflows. And if the browser knows it, we can know it too.
By observing the full browser state, we can decode the hidden contract between the frontend and backend:
HTML structure and DOM mutations
JavaScript bundles and the logic they execute
Cookies and authentication tokens
Local and session storage
All network traffic: XHR, fetch, GraphQL, REST, streaming
Runtime signals from the JS process itself
Once you can see all of this, the structure of the app’s communication reveals itself.
And once you understand that structure, you can replicate it without scraping pixels or driving a synthetic user.
Introducing Vectorly: Turn Web Apps into APIs
Vectorly doesn’t try to “be a human in the browser.” Instead, it instruments the browser and turns what it sees into reusable automation recipes.
Step 1 — Interact with any website in the Vectorly Console
You browse the site naturally—click, search, scroll, and explore as you would. While you interact, Vectorly captures everything the browser sees: network traffic, cookies, storage, HTML, and JS.
Step 2 — Tell the agent what you want
Once you’re done with the interaction, you simply describe what you need:
- The data you want extracted (e.g. “all upcoming bookings for this account”)
- The API behavior you want replicated (e.g. “search flights with these parameters”)
Our AI agent then reviews all the captured assets and infers the underlying structure of the app’s internal APIs and flows.
From that analysis, Vectorly's AI Agent will generate a routine: a structured automation recipe made of:
- Concrete browser steps (navigate, click, input, wait, scroll…)
- Direct internal API calls with the right parameters, headers, cookies, and auth
Step 3 — Call the Routine Via API or MCP
Vectorly then exposes this routine as:
A REST API endpoint you can call from your code
An MCP tool your LLM agents can use as a first-class capability
Define once. Reuse everywhere. You get the real behavior of the web app — without driving a fake user or scraping pixels.
📚 Resources
If you want to explore Vectorly, here are all the key links in one place:
🔗 Official Links
Developer Console: https://console.vectorly.app
Website: https://vectorly.app
Docs: https://www.vectorly.app/docs
Discord Community: https://www.vectorly.app/discord-invite
🧩 Open Source
Web Hacker (Reverse Engineering Framework):
https://github.com/vectorlyapp/web-hacker
🎥 Videos & Launch
Demo Video: https://youtu.be/xX4ycRAU5dA
YouTube Channel: https://www.youtube.com/@VectorlyAI
If you try Vectorly, we’d love to hear what workflows, websites, or integrations you want to unlock next!