I typed "Clean the kitchen" into a chat box. A robot turned toward the right room, drove over, swept the floor in a tidy back-and-forth pattern, and came back home, while describing what it was doing the whole way.
No buttons, no coordinates, no code to write each time. Just a sentence.
I built this over the last few weeks. It's called NavMind, and it's the first project I'm writing up as I learn robotics in public. Here's what it does, the one idea that makes it work, and the things that broke along the way.
What it is, in one breath
NavMind is a robot that lives in a simulated house. You tell it what to do in plain English, and it figures out which room you mean, drives there on its own, cleans the whole floor, and returns to base. There are six named rooms: kitchen, bedroom, study, and so on, and it handles commands like "vacuum the bedroom" or "sweep the guest room."
It runs entirely on my own laptop. No cloud, no paid API, no physical robot, just a simulator and a language model running locally.
What happens when you type a command
Let's follow a single instruction all the way through.
You type "clean the kitchen." A small chat app (built with Streamlit) takes your sentence.
The language model reads it. A local model, Llama 3.2, running through Ollama on my machine, turns your loose sentence into a structured command: room = kitchen, action = clean.
That command is sent to the robot's "brain." A navigation node receives it. It already knows where each room sits on a map of the house, so it converts "kitchen" into an actual location to drive to.
The robot navigates there. This is handled by Nav2, the standard ROS2 navigation system. It works out where the robot currently is, plans a path to the kitchen, and follows it while avoiding furniture.
It cleans the room. Once it arrives, a coverage routine walks the robot back and forth across the floor in a lawnmower pattern, a robot vacuum uses, so that the whole room gets covered, not just one spot.
It narrates and returns. The whole time, the language model is writing plain-English commentary back to the chat, and when the room is done, the robot drives home.
That's the full loop: sentence → meaning → location → drive → clean → home.
The parts that fought back
Nothing about this was smooth, and the messy parts taught me the most. A few, in case they save you an evening:
- It takes seven terminals to run. The simulator, the bridge, the navigation system, the UI, and a few helpers each run separately, in order. Forget one and the robot just sits there.
- The language model couldn't reach the robot. I run everything in WSL2 on Windows, with Ollama on the Windows side, and those two don't share a network by default. The robot literally couldn't talk to the model until I pointed it at the right internal IP address.
- The navigation system wouldn't "wake up." Nav2 starts in a sleeping state and has to be switched on in a specific order. Until I learned that, every command was met with a perfectly content, perfectly motionless robot.
None of these were glamorous. All of them were where the real learning happened.
What's next
A few things I want to try: chaining commands together ("clean the kitchen, then the study"), smarter coverage that adapts to room shape, and a bigger local model to see if it parses instructions more reliably.
The full code and setup guide are on GitHub: github.com/sudeekshach/NavMind (MIT-licensed. Take it apart however you like).
This is my first write-up, so I'd genuinely love feedback on the project or the post. And if you build something on top of it, tell me what you made. 🤖
SOCIAL SHARE CARD GENERATOR