Token Town

Many of the sessions from Tuesday, especially on the main stage, revolved around the idea of software factories. Coordinating agents with other agents that check even more agents' work, with humans in an overseer position. Currently, many organizations’ systems are not quite factory-ready, with agents doing work but only as directed and subsequently verified by humans. A critical part of the software factory paradigm is its ability to provide trust by being able to review itself. Those larger orchestration and evaluation tasks may need to use more powerful models in order to provide autonomous output while building the necessary trust with the humans in and above the loop. But do all of the agents in the factory have the same requirements?

Sarah Sachs from Notion gave an incredible talk about just that, and if you missed it, make a note to go watch the recording. The main stage was livestreamed; you could watch it right now. Her main point was this: Most of us don't work for a frontier AI lab. A large fraction of the things we do, both in our own work and in our products, don't require the biggest, hottest, most token-hungry models. If we lock ourselves into one AI vendor or one hard-coded AI model, we're doing ourselves, our shareholders, and our customers a disservice.

At this point, even the models that aren't bleeding edge from any vendor are pretty decent and good enough to do simple to medium tasks like summarization. Using Opus to summarize a Slack thread is a bad choice. You likely don't need any model to convert a file from one type to another. As AI is getting better, it's also getting more expensive. Price is only going up, as are other social and environmental costs, and "simply throwing more tokens at it" is not a viable move for most companies.

Sarah's advice was to build smartly and modularly enough to avoid vendor and model lock-in. Switch models and vendors on a dime when the billing shifts to make the right choice for your customers. Try out open-weight models, especially for those midrange tasks where "good enough" is good enough. Balance the needs of all the different agents in your “factory,” make the "tokenomics" work for you, and it will help keep the costs a little more balanced for everybody else, too.