also lists two capabilities that turned out to be directly relevant here: native pointing (the model can return spatial coordinates for objects within an image, not just describe them) and interleaved multimodal input (audio and visual content freely mixed in a single prompt). Both are described under image understanding in the model card, leveraging this gave us more reliable responses.

The pipeline:

• 
  Scout: Classifies the request - direct answer or spatial analysis needed - and routes accordingly.


• 
  Mapper: Interprets the question, localizes relevant objects, and calls the TIPSv2 spatial tool stack to build a grounded scene state with distance and bearing per object.

User Question : Can you tell me how to get to the grill in my backyard?

Gemma Guide : The grill is 2.342 meters away, straight ahead. There are a few things in the way. You should first step slightly right to clear the table, which is 0.985 meters away about 25 degrees to your left. Then, the chair at 1.345 meters straight ahead will be in your path. Steer gently to your right to avoid it and step forward.

Key Findings

• Distance alone is not enough. Early versions reported metric depth but users had no sense of direction. Adding horizontal bearing from the object's position in the frame turned a distance reading into actionable spatial guidance.




• Whole-scene depth reasoning was too unreliable. Asking the model to reason over a full depth map produced ambiguous results. The reliable path was having Gemma localize the object first, then feeding that into the measurement pipeline - leveraging what the model is actually built for: reasoning, localization, and tool calling.




• Separation of concerns made outputs consistent. Combining scene description, tool orchestration, and navigation reasoning in one agent made outputs inconsistent. Splitting into Scout, Mapper, and Navigator fixed this.




• vLLM was the right local inference choice for now. Ollama lacks audio input entirely, ruling it out for a voice-driven pipeline, and its Gemma 4 tool-calling parser has had numerous bugs where vLLM's proved robust. With quantization, reduced max sequence length, and fewer image patches per call, the full stack - Gemma 4 E4B plus TIPSv2 - fits on a single 16GB GPU.




• Latency is the dominant UX constraint. Tool calls run in parallel, but the Mapper and Navigator still bottleneck on model reasoning itself, and that cost grows as conversation history lengthens. Complex scenes can push end-to-end response time past 20 seconds.




• Reliability is the remaining work. The navigation instructions are still sometimes off, and fine-tuning the grounding stack on task-specific data is where the remaining work sits.

Toward on-device deployment:

I explored Google AI Edge Gallery as a path to partial on-device deployment (Gemma locally, TIPS stack remote), but the current blocker is that the image Gemma sees in chat is not forwarded into the skill execution context, which breaks grounded measurement. A standalone mobile app with tighter camera and voice control is the stronger long-term path, and both models are compact enough to make fully offline deployment on edge hardware plausible.