The frontier for economic value from AI agents is non-gullibility

The usual measures of AI progress have not suited my lived experience for some time now:

• One measure is “maximum length of human task that AI can complete.” The ideal goal here seems to be AI developing ever larger software systems, like browsers.




• Another measure is “key reasoning breakthroughs,” like proving some math theorem or finding zero-days. The ideal goal here seems to be the , many people are still stuck in manual code review and approval fatigue because they feel obliged to prevent unlikely but devastating consequences.

  
  
  

  
  
  
  The interface between the agent harness and the model lacks a granular trust model
  

  
  
  

  Non-gullibility is not a property that can be achieved by the foundation model alone. It also requires correct construction of the agent harness, because only the harness knows the data's true source and trustworthiness, and the harness assembles trusted and untrusted data into the model’s input.

  
  
  

  Here, it seems to me that non-gullibility could be supported much better. LLM APIs today distinguish system, user, assistant, and tool messages in principle, but practical systems often mix authority levels for two reasons:

  
  
  
  
  
  1. 
     

     APIs and harnesses restrict which message types are permitted in which positions. System messages are often only available at the beginning of the conversation. Tool messages are only allowed in response to tool calls.

     
     
     

     A workaround in practical harnesses like Claude Code and opencode is to include a textual fragment like <system-reminder> wherever the information needs to go. A user or tool message may include such a reminder to tell the model that another file changed, or that the user interrupted the model mid-task.

     
     
     

     This is convenient, but it muddies the security boundary. A model trained to perform well in such a harness may learn to treat <system-reminder> as carrying special authority. How should the harness defend against prompt injection containing <system-reminder> tags? The obvious answer is escaping. But it does not seem customary to XML-escape all untrusted input, and escaping everything would also increase token usage.

     
     
  
  
  2. 
     

     Message history is a sequence, while authority is naturally a hierarchy.

     
     
     

     Consider the output of a hypothetical grep tool with line numbers. Different parts of the output have different authority levels:

     
     
     

     Higher-authority information: the fact that the trusted harness ran the built-in grep tool and found matches at certain paths and line numbers.

     
     
     

     Lower-authority information: the path strings and file contents themselves. These may come from an untrustworthy source. If a file contains a tag, the model should not treat it as special.

     
     
  
  
  
  
  

  
  
  
  A proposal for nested trust indicator delimiters
  

  
  
  

  To supplement fixed roles like system, user, assistant, and tool, imagine an API that also supports a pair of special tokens for opening and closing a lower-privilege scope within a message. The harness developer could then express the authority of conversation fragments precisely.

  
  
  

  If we visualize the lower-privilege opening token as {, and the corresponding closing token as }, then the grep tool could put quoted file contents inside {} to make clear that they are untrusted data. Here, { is a special token, not a regular character that could occur in text.

  
  
  

  A conversation history might look like this:
  
  

  
  
  
  
  

  CODE Kopieren

  System: You are a helpful assistant.
  
  User: {Where does the term "foo" occur in this repository?}
  
  Assistant: {<use-tool git-grep term={foo}>}
  
  Tool: { {blubb.txt}:3:{*foo* bar baz} }
  
  

  
  
  
  
  
  
  
  

  The system message is not wrapped in a trust limiter because it is already the highest-authority context.

  
  
  

  The user message is wrapped in a trust limiter because the user must not be able to override the system message.

  
  
  

  For the tool message, the important part is that the delimiters can be nested. The tool should not be able to assume the authority of the user or system, so the entire tool output is wrapped in a trust limiter. At the same time, the tool output contains parts that are themselves arbitrary data, so the tool wraps those in another, nested pair of delimiters.

  
  
  

  This proposal would have to be part of the API between harness and model. In particular, the LLM server would reject improperly balanced delimiters in the input, and would enforce that model responses are balanced as well. Literal { and } characters in user data would not matter, because they would not be the special delimiter tokens.

  
  
  

  
  
  
  Why a special token to indicate trust level is justified
  

  
  
  

  One could argue that if we used escaping correctly, the LLM should be able to reason about the effective trustworthiness of data in context, and introducing a new set of delimiters would run against the spirit of having general-purpose models. I see three counterarguments to this:

  
  
  
  
  

  1. The best open-source LLMs already use special tokens to indicate roles. The present proposal is not less generic, or less bitter-lesson-pilled, than what is already done.

  
  

  2. We already allow LLMs to use code interpreters for efficiency, even though LLMs should in principle be able to reason about the outcome of code execution. It is just more efficient and reliable to interpret code before the result reaches the LLM. Likewise, it may be more reliable to deterministically encode quoting and privilege boundaries before the token stream reaches the LLM.

  
  

  3. LLM input commonly has a very ad hoc nested structure. A Markdown code block may contain XML, which contains JSON, or any nesting thereof. It is not necessarily invalid input if a Markdown code block contains broken XML or broken JSON. Perhaps broken XML is exactly what the code block is supposed to show. Explaining to the model which parts of the input are to be expected to be properly escaped and which are not without ambiguity is tricky. The relevant question is not whether the model can reason about trust, but rather whether the harness has a precise way to express trust to the model.

  
  
  
  
  

  
  
  
  Conclusion
  

  
  
  

  For economically useful agents, I expect non-gullibility to matter more than another jump in task length or theorem-proving. The bottleneck is whether we can safely let the agent use the capability it already has.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen