Harness Tells Your Agent What to Do. GUI Agents Let It Actually Do It.

The Rise of Harness Engineering

Harness Engineering has become the defining conversation in AI agent development this quarter. Anthropic published "Effective Harnesses for Long-Running Agents." OpenAI released their own take on constraining agent behavior through software engineering practices. The thesis is straightforward: wrap your AI agent in a structured control layer—task routing, approval gates, verification loops, and retrospectives—so it behaves reliably over extended sessions.

The pattern makes intuitive sense. An unconstrained agent is a liability. A harnessed agent is a tool. The community has responded: open-source harness frameworks are emerging, giving teams reusable scaffolding for decision-level reliability.

But here's the question no one is asking loudly enough: after the harness decides what to do, how does the agent actually do it?

What Harness Solves

A harness framework operates at the decision layer. It answers:

• 
  What should the agent do next?


• 
  In what order should tasks execute?


• 
  When should it pause for human review?


• 
  How do we verify the outcome before moving on?

Think of it as the prefrontal cortex of your agent system—planning, sequencing, gating. Frameworks like cow-harness already provide open-source implementations of these patterns: task decomposition, approval workflows, retry logic, and audit trails.

This is genuinely valuable. Without a harness, agents hallucinate plans, skip steps, and compound errors. With one, they become predictable and auditable.

But predictable planning is not the same as reliable execution.

The Execution Gap

Consider a real scenario. Your harnessed agent determines the next action: "Open the CRM, navigate to the customer record for Acme Corp, and update the contract renewal date to June 15."

The harness has done its job. The decision is correct. The approval gate passed. Now... how does the agent physically perform this action?

Current execution methods each carry fundamental limitations:

CLI tools — Powerful but narrow. Only works for systems that expose command-line interfaces. Most enterprise software does not.

API calls — The gold standard when available. But many critical business systems—legacy ERPs, proprietary desktop apps, government portals—simply have no API. Or the API covers 20% of what the GUI exposes.

DOM manipulation — Works for web apps, breaks on desktop. Requires knowledge of the target app's internal structure. One frontend update can invalidate your selectors.

RPA scripts — The enterprise workaround. Record a macro, replay it. Brittle by nature: a single UI change—a moved button, a renamed field, a new modal dialog—breaks the entire flow. Maintenance cost scales linearly with the number of automations.

The common thread: all of these methods require a pre-existing technical interface to the target system. They assume the system was designed to be automated, or that someone has reverse-engineered a way in.

In enterprise reality, the most critical systems are often GUI-only black boxes. No API. No CLI. No stable DOM. Just a screen that a human clicks through.

This is the execution gap. Harness frameworks have nothing to say about it.

Vision-Based GUI Agents as the Execution Layer

What if the agent could interact with software the same way a human does—by looking at the screen and clicking?

That's exactly what vision-based GUI agents do:

1. 
   Input: A screenshot of the current screen state


2. 
   Understanding: A vision-language model identifies UI elements—buttons, text fields, menus, labels—and comprehends their spatial relationships and semantic meaning


3. 
   Output: Precise mouse coordinates and keyboard actions to accomplish the intended task

The key property: zero dependency on target system internals. The agent doesn't need an API, a DOM tree, or accessibility hooks. It sees pixels and acts on them. This works across:

• Web applications


• Native desktop software


• Remote desktop sessions


• Terminal UIs


• Even systems running in virtual machines

If a human can operate it by looking at a monitor, a vision-based GUI agent can too.

Putting It Together: Harness + GUI Agent

This is where the architecture becomes complete. The harness provides the brain—deciding what to do, when to pause, how to verify. The GUI agent provides the hands—executing actions on any visual interface.

Mano-P is an open-source GUI agent built for exactly this role. Developed by Mininglamp Technology under the Apache 2.0 license, Mano-P implements a Vision-Language-Action (VLA) architecture designed to serve as the execution layer in agentic systems.

The name encodes the philosophy: "Mano" is Spanish for "hand"—the part that acts. "P" stands for Private—your data never leaves the device.

Architecture: Think-Act-Verify

Mano-P operates through an inference loop that mirrors how a careful human operator works:

1. 
   Think — Observe the current screen state, reason about what UI elements are present, and determine the next action


2. 
   Act — Execute the precise mouse/keyboard operation


3. 
   Verify — Capture the resulting screen state and confirm the action had the intended effect

This loop provides built-in error detection. If a click lands on the wrong element or a form doesn't submit, the verify step catches it immediately—enabling retry or escalation back to the harness layer.

On-Device Performance

Mano-P is designed for local execution. The quantized 4B model runs on consumer hardware:

• 
  Minimum: Apple M4 chip + 32GB RAM (Mac mini or MacBook)


• 
  Performance on M5 Pro: ~80 tokens/s decode speed

The Cider SDK provides W8A8 activation quantization, delivering approximately 12.7% prefill acceleration compared to the W8A16 baseline, and 1.4x–2.2x prefill speedup versus MLX native W4A16. This means real-time interaction with GUIs—no cloud round-trip, no latency spikes.

Benchmark Results

On the OSWorld benchmark—the standard evaluation for GUI agent capabilities across real operating system tasks—Mano-P 1.0-72B achieved a 58.2% success rate, ranking #1 among specialized GUI agent models.

For web navigation specifically, the WebRetriever Protocol I achieved a 41.7 NavEval score, demonstrating reliable multi-step web interaction.

Mano-AFK: The Full Automation Loop

To demonstrate how harness-level planning connects to GUI-level execution, Mininglamp Technology built Mano-AFK—an end-to-end autonomous development pipeline:

Natural language requirement → PRD generation → Architecture design → Code generation → Deployment → E2E testing (Mano-P's visual model drives the browser to test the deployed app) → Bug detection → Fix → Retest

This is the harness + GUI agent pattern in its most complete form. The planning layer decomposes a vague requirement into structured development phases. The GUI agent handles the parts that require visual interaction—browser testing, UI verification, visual bug detection—without any test framework dependencies.

Privacy by Design

In local execution mode, all processing happens on-device. Screenshots are captured and analyzed locally. Model inference runs locally. No data transits to external servers. For organizations handling sensitive information—financial records, medical data, classified documents—this is not a feature. It's a requirement.

Honest Limitations

No technology is universally optimal. Vision-based GUI agents have real tradeoffs:

Overhead on simple web tasks — For well-structured web applications with clean APIs or stable DOM trees, direct API calls or DOM manipulation will always be faster than screenshot-based interaction. If you have a good API, use it.

Accuracy ceiling on complex UIs — The 4B on-device model handles standard interfaces well but can struggle with extremely dense or unconventional UI layouts. The 72B model pushes accuracy significantly higher but requires more compute.

Best suited for specific scenarios:

• Legacy enterprise systems with no API


• Cross-platform automation spanning web and desktop


• Data-sensitive workflows requiring strictly local execution


• Systems where UI changes frequently (vision adapts; scripts break)


• Remote desktop environments where DOM access is impossible

The right architecture uses the right tool for each target. API calls where APIs exist. DOM methods for stable web apps. And vision-based GUI agents for everything else—which, in most enterprises, is a surprisingly large surface.

Conclusion

The AI agent stack is crystallizing into two distinct layers:

The Brain — Harness frameworks that constrain, route, verify, and audit agent decisions. This is a solved problem with active open-source development.

The Hands — Execution layers that translate decisions into physical actions on real systems. For GUI-bound systems, vision-based agents are the only approach that scales without per-system integration work.

Harness tells the agent what to do. GUI agents let it actually do it. Together, they close the automation loop.

Mano-P is Apache 2.0 licensed and available on GitHub: https://github.com/Mininglamp-AI/Mano-P

Feedback and contributions welcome.⭐