Day 15. The agent switched between two apps, copied data from one, and pasted it into another.
For 14 days, my AI agent could do one thing well: operate within a single app. Open WhatsApp. Find a contact. Send a message. All within one ecosystem.
Today, that changed.
The Milestone
I gave the agent a command: "Copy my account balance from my banking app and send it to Mom on WhatsApp."
It opened the banking app. It read the balance from the screen using OCR (since banking apps have zero accessibility labels—F score, remember?). It copied the number. It switched to WhatsApp. It found Mom. It typed the balance. It hit send.
Two apps. One task. Completed autonomously.
What Made This Possible
The key was building a task memory system. The agent now stores values extracted from one app in a simple key-value store (a Python dictionary that persists for the duration of the task). When it switches to the next app, it retrieves that value and uses it.
The flow works like this:
- Parse the command into a plan with Gemma 4.
- Execute Step 1: Open banking app. Read the balance. Store it in memory as
account_balance. - Execute Step 2: Open WhatsApp. Find the contact. Retrieve
account_balancefrom memory. Type it. Send.
The memory system is simple—just a dictionary that gets passed between steps. But it's the bridge between single-app automation and true multi-app workflows.
Today's Progress
| Task | Status |
|---|---|
| Built task memory system (key-value store) | ✅ Done |
| Tested "Copy balance → send to Mom" | ✅ Success |
| Tested "Copy address → paste into Maps" | ✅ Success |
Updated agent.py with multi-app support | ✅ Done |
| Added task memory to README | ✅ Done |
What Still Breaks
The banking app is slow to load on my phone. Sometimes the agent times out waiting for the balance screen to appear. I added a retry loop, but it's not perfect.
App switching is clunky. The agent uses adb shell am start to launch apps, but if the app was already open in the background, it sometimes opens to the wrong screen. I'm working on a "reset to home screen" step between app switches.
And the OCR on banking app text is unreliable. The font is small. The contrast is low. The agent sometimes misreads the balance by a digit or two. For a task like "send my balance to Mom," that's embarrassing but not catastrophic. For a task like "transfer money," that's dangerous. I need higher accuracy before I can trust this with anything financial.
The Bigger Picture
Multi-app workflows are the difference between a cool prototype and a useful tool. An agent that can only operate in one app is a novelty. An agent that can move data between apps is an assistant.
The next step is chaining more complex workflows: "Find my last three transactions, summarise them, and email them to my accountant." That's three apps. That's the goal.
What's Next (Day 16)
- Add a "reset to home screen" step between app switches
- Improve OCR accuracy on banking apps
- Test a 3-app workflow
The Repo
👉 github.com/Dexter2344/phone-agent
agent.py now includes a task memory system. Multi-app workflows are supported. README updated with Day 15 status.
This is Day 15. The agent is no longer trapped in one app.
SOCIAL SHARE CARD GENERATOR