A controlled experiment exploring how Claude and Qwen resolve conflicting instructions across system prompts, user messages, and tool descriptions.

Cross-posting from Medium:

The first thing I noticed was how familiar these numbers looked.

In Part 1, placing the instruction in the user message produced 64% compliance, while the system prompt managed 8% and the tool description 2%.

Now, under direct competition, the user message wins 60% of the time, the system prompt wins 2%, and the tool description never wins at all.

Although the experiments ask different questions, they tell a remarkably consistent story.

The slot that was strongest in isolation is also the slot that dominates when every instruction competes.

The conflict condition also exposed behavior that Part 1 could never reveal.

Nearly a third of the runs ended without any expected marker.

Another 6% produced multiple competing markers in the same response.

Instead of consistently selecting one instruction, the model sometimes failed to produce a single clear winner.

A note about tool execution

One implementation detail is important when interpreting these results.

Unlike the Claude models, Qwen never successfully executed the tool loop.

Rather than producing structured tool calls, it emitted tool-call JSON as plain text and completed every run in a single turn.

This means the tool description was never exercised as part of an actual tool invocation.

It existed only as text inside the context window.

That limitation is consistent with the results from Part 1, where the tool description also had almost no observable influence for Qwen.

Results: Claude Haiku 4.5 (Anthropic API)

Outcome	Frequency
User [FINISHED]	100%
System [DONE]	0%
Tool [COMPLETE]	0%
None	0%
Conflict in output	0%

Every run produced exactly the same outcome.

The model completed the tool loop correctly, used three turns, and always finished with [FINISHED].

This is where the experiment becomes interesting.

Part 1 suggested that every prompt slot was equally effective because each placement achieved 100% compliance.

Part 2 reveals a more nuanced picture.

When every slot contains the same instruction, every slot can successfully deliver that instruction.

Once those instructions conflict, however, the model consistently resolves the disagreement in favor of the user message.

The placement experiment and the conflict experiment are measuring different properties of the model.

Results: Claude Sonnet 4.6 (Anthropic API)

Outcome	Frequency
User [FINISHED]	100%
System [DONE]	0%
Tool [COMPLETE]	0%
None	0%
Conflict in output	0%

Claude Sonnet was tested across 12 runs, stopped early once the pattern was clearly established—that is, the user instruction determined the final formatting of the response.

Summary

Model	Type	System	User	Tool	None	Conflict
qwen2.5-coder:3b	Small local (Ollama)	2%	60%	0%	32%	6%
claude-haiku-4.5	Small frontier (Anthropic)	0%	100%	0%	0%	0%
claude-sonnet-4.6	Large frontier (Anthropic)	0%	100%	0%	0%	0%

Three observations stand out:

• The tool description never won: across all runs and all three models, [COMPLETE] never emerged as the surviving instruction.


• The system prompt rarely won: it appeared once for Qwen and never for either Claude model.


• Both Claude models behaved identically despite their difference in size. Haiku, Anthropic's smallest model, resolved the conflict exactly the same way as Sonnet.

Looking at Both Experiments Together

Although both experiments involve prompt placement, they answer different questions.

Part 1

Can this prompt slot successfully deliver an instruction?

Part 2

When multiple instructions compete, which one determines the final output?

For Qwen:

The user message was the strongest placement in isolation, and it remained the dominant placement under direct competition.

For the Claude models:

Part 1 showed that all three prompt slots could successfully deliver an instruction when no competing instruction existed.

Part 2 showed that once conflict was introduced, the user message consistently determined the final formatting in this experiment.

Together, the two experiments show that instruction visibility and instruction priority are different characteristics of an LLM.

A model may reliably process instructions from every prompt slot while still preferring one slot whenever those instructions disagree.

What This Means in Practice

If you're building agents with smaller open-weight models, prompt placement is more than a stylistic choice.

Across both experiments, the user message was consistently the most reliable place for formatting instructions.

System prompts and tool descriptions were substantially less effective, particularly when competing instructions existed.

For the Claude models tested here, the practical takeaway is different.

They successfully followed instructions regardless of placement when no conflict existed.

However, in this experiment, conflicting formatting instructions were consistently resolved in favor of the user message.

It's important to keep the scope of that finding in mind.

This experiment only examined formatting instructions within a controlled agent loop.

It does not imply that user prompts override safety policies or other system-level behaviors, which are governed by different mechanisms and would require a different experimental design.

Caveats

The markers [DONE], [FINISHED], and [COMPLETE] are different strings.

They differ in length and may differ in how frequently similar tokens appeared during model training.

Rotating the markers across prompt slots would control for that effect, but it would also triple the size of the experiment and was not done here.

The sample sizes also differ across models:

• 
  50 runs for Qwen


• 
  30 for Claude Haiku


• 
  12 for Claude Sonnet

The Anthropic models exhibited highly consistent behavior, allowing the experiments to stop once the dominant pattern was established.

Finally, these results are model- and task-specific.

Different architectures, quantization levels, or tasks may produce different behaviors.

The goal of this experiment is not to establish a universal prompt hierarchy, but to measure how these particular models behave under controlled conditions.

Statistical confidence intervals were calculated during analysis but are omitted here because the dominant winner was unambiguous.

Final Thoughts

The most interesting result wasn't that the user message won.

It was that two experiments, built to measure different properties, kept arriving at the same answer.

For one model, the strongest placement in isolation was also the strongest placement under conflict.

For the others, perfect placement compliance concealed a deterministic preference that only became visible once the prompts disagreed.

Sometimes the most interesting model behavior doesn't appear when there's only one correct instruction.

It appears when every prompt slot asks for something different, and the model has to decide which one deserves the final word.

Follow me on LinkedIn: Raj Kundalia

Related

• Where You Put the Instruction Matters More Than What It Says


• Why 95 Reviews Beats 20 Reviews—Even When Both Score 95%

Vollständiger Original-Bericht

Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

↗ Original-Artikel auf dev.to lesen