Hundreds of enterprise leaders and technical experts packed the main ballroom of the luxurious Hotel Nia in Menlo Park this week for VB Transform 2026, the year's preeminent conference on using generative AI agents to drive business outcomes.
Rachad Alao, vice president of product engineering at the rising Canadian enterprise AI startup Cohere, joined VentureBeat CEO and editor-in-chief Matt Marshall for a fireside chat about building agentic systems without surrendering sensitive data, infrastructure control, or the ability to change vendors.
Alao, who previously led responsible AI and trust and safety engineering teams at Google and Meta, argued that AI sovereignty means more than downloading an open model or running an application behind a corporate firewall.
Asked how Cohere defines sovereignty, Alao pointed to organizations operating mission-critical systems, including banks, hospitals and governments.
“It is important to have very tight control on where the data resides, have tight control on the AI,” he said, adding that AI operations should take place in jurisdictions an organization understands or directly controls.
That extends from GPUs and private-cloud infrastructure through governance systems that route requests among models, as well as the connectors, search tools and agent frameworks acting on enterprise data.
“You want to have control on the entire stack,” Alao said.
Agent workloads could outrun falling token prices
Marshall challenged one of the central economic arguments for smaller, locally deployed models: Inference prices continue to fall rapidly, potentially weakening the case for optimizing every token.
Alao countered that total consumption is climbing even faster as enterprises move from relatively simple chatbots to agents that reason through problems, call tools, search internal systems and take multiple steps before returning an answer.
“Your token utilization is going exponentially up, because you’re dealing with more and more complex agentic use cases,” he said. Those workflows require “a lot of processing, thinking, tools interaction” to complete their objectives, he added.
Alao also drew a contrast between providers that bill customers according to token consumption and Cohere’s approach.
“If your whole way of charging customers is for token utilization, you want to maximize token utilization,” he said. “We do not sell our models and our platform that way.”
Instead, Alao said Cohere tries to help enterprises solve their hardest problems privately and securely while reducing unnecessary model usage. His prescription was straightforward: “Use the right model for the task at hand.”
Rather than sending every request to the largest available frontier model, enterprises should route work according to the intelligence required and the sensitivity or regulatory burden attached to the task.
Alao cited an unnamed Canadian bank that uses Cohere’s on-premises models for highly regulated workloads, while sending less sensitive tasks requiring greater intelligence through Cohere’s North platform to larger frontier models.
“So model routing can become super useful,” he said.
Smaller models for most enterprise work
Asked by an audience member how Cohere’s open-source North Mini Code, released last month, could compete against proprietary coding models, Alao acknowledged that larger frontier models may perform somewhat better on the hardest tasks.
But that advantage may not justify using them indiscriminately.
“For 80% of the use cases that they needed, this was a lot more effective, a lot cheaper,” Alao said of developers adopting the model.
Cohere’s North Mini Code runs on a single Nvidia H100 GPU and targets agentic software engineering, including terminal work, code review and tool use.
The company has also released Command A+, a 218-billion-parameter mixture-of-experts model with only 25 billion parameters active during each generation step.
Its compressed four-bit version reduces the hardware required for private deployment, while its Apache 2.0 license gives enterprises broad freedom to operate and modify it.
Search becomes part of the agent
Asked about Cohere’s longstanding work on embeddings and enterprise search, Alao said the field is moving beyond retrieving text and inserting it into a model’s context window.
“Today, the state of the art is around multimodal search,” he said. “It’s beyond just the text modality.”
Search across documents, images and other forms of information is becoming “an integral component of your agentic workflow,” Alao added, with the model deciding when and how to use retrieval like any other tool.
Asked what would persuade enterprises to move beyond bundled AI services from existing cloud providers, Alao returned to data control and portability.
“If you’re interested in sovereignty, you want to have more control on your data,” he said. Cohere’s governance layer, he added, lets customers route traffic to appropriate models, “breaking that vendor lock-in concern that a lot of our customers have.”
SOCIAL SHARE CARD GENERATOR