How companies are racing to solve the AI token problem

Because generative AI (genAI) tools and services have become so ubiquitous (and popular), the costs of using them are going through the roof — leading to an insatiable appetite for tokens.

Tokens represent a . 

There are a number of ways companies can rein in the price of AI at the model, infrastructure, silicon, and business levels. Here’s a look at how some of those savings might actually be achieved.

Switch to lower-cost models

One way of potentially saving money is by re-routing AI work to a cheaper model, Pichai said. At Google that would , CEO of , head of data science and AI solutions at ManpowerGroup. 

“They’re using fewer tokens and they’re simply more efficient,” he said. “And that in large part has to do with your ability to prompt efficiently.”

Go local

New AI hardware that generates free tokens at home could ease some of the cost crisis.

At .

Some companies are looking to reduce cloud AI costs by putting their own hardware in data centers, with vendors such as HPE and Dell providing servers installed in independent facilities. (On-premise AI is gaining ground amid sovereign AI and geopolitical concerns, including the recent conflict in the Middle East, where large data centers were struck with missiles.)

“There are local, region-specific and multiple vendor AI solutions. All of those things can help mitigate the risk. But they’re not going to eliminate it,” said Max Goss, senior director analyst at Gartner.

Use forward-deployed engineers

Reducing token costs is something that may fall to , managing director of AWS’s Generative AI Innovation Center.

“I expect these teams to be able to architect systems that have those cost requirements in mind, whether it’s use a different model or a different use case that doesn’t increase the per-token cost,” Rashid said.

Companies may spend heavily on token consumption, “but if you’re generating revenue, as long as the economics work out, then you’re at peace,” Rashid said.

The use of FDEs is gaining ground as IT decision-makers look to both rollout successful AI deployments while also keeping an eye on costs.

Change the measure of success from tokens to outcomes

Even with the current emphasis on reducing token use to save money, the metrics used to measure AI success are likely to shift, Gartner’s Seth said. At some point, token-based pricing will move more toward an outcome-based model, where the unit of value is outcomes, not fragments of words.

“Some companies are moving towards outcome-based pricing,” Seth said. “When people start realizing the real cost of tokens, then companies will start looking at token efficiency.”