This is post #9 of the that researchers demonstrated how HITL confirmation dialogs are trivially bypassable because humans trust the AI's summary of what it wants to do.
2. Helpful Assistant Trojan
A compromised coding assistant suggests a "slick one-line fix." The developer pastes it. The command runs a malicious script that exfiltrates code and installs a backdoor.
an incident where an engineer acted on "inaccurate advice that an AI agent inferred from an outdated internal wiki," causing a production outage. The company stated the root cause was not faulty code, but a human executing an AI recommendation without verifying it. The audit trail showed the human performed the action. The agent's bad advice was invisible.
5. Governance Drift (The Slow Poison)
This is the sneakiest one. After weeks of accurate recommendations, humans start bulk-approving without reading. The agent's accuracy has earned it trust. Then one poisoned recommendation slips in among 50 legitimate ones.
OWASP calls this "governance drift cascade" and it's the natural human response to repetitive approval tasks that never fail.
Real incident: The in 2026. After three weeks of confirming 50 flawless AI recommendations, humans stop reviewing. Approval becomes a reflex while liability stays with the human approver. This is governance drift in action.
Why "Human-in-the-Loop" Isn't the Fix You Think It Is
Let me be blunt about this. "We have human-in-the-loop" is not a security control. It's a checkbox.
Here's what actually happens:
Week 1: Humans carefully review every agent recommendation
Week 4: Humans skim the summaries
Week 8: Humans click "approve" reflexively
Week 12: Someone asks "why do we have this approval step again?"
Automation bias is real. Authority bias is real. And agents trigger both simultaneously because they sound confident and knowledgeable.
The fix isn't "add a human." The fix is designing the human interaction so that rubber-stamping is structurally impossible.
Mitigating Trust Exploitation on AWS
1. Amazon A2I: Structured Human Review Workflows
lets you build approval gates that expire. If nobody reviews within the timeout, the action is denied by default, not approved.
{
"Comment": "High-risk action approval gate: approve -> execute; reject or timeout -> deny",
"StartAt": "ApprovalGate",
"States": {
"ApprovalGate": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish.waitForTaskToken",
"Parameters": {
"TopicArn": "arn:aws:sns:us-east-1:123456789012:high-risk-agent-approval",
"Message": {
"action_type": "financial_transfer",
"amount.$": "$.amount",
"recipient.$": "$.recipient",
"agent_rationale.$": "$.rationale",
"raw_source_data.$": "$.source_invoice",
"taskToken.$": "$$.Task.Token"
}
},
"TimeoutSeconds": 3600,
"Next": "ExecuteAction",
"Catch": [
{ "ErrorEquals": ["States.Timeout"], "Next": "DenyByDefault" },
{ "ErrorEquals": ["States.ALL"], "Next": "Rejected" }
]
},
"ExecuteAction": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:execute-financial-transfer",
"End": true
},
"Rejected": {
"Type": "Pass",
"Result": {"decision": "DENIED", "reason": "Human reviewer rejected the action"},
"End": true
},
"DenyByDefault": {
"Type": "Pass",
"Result": {"decision": "DENIED", "reason": "Approval timeout - no human reviewed within 1 hour"},
"End": true
}
}
}
Critical design choice: timeout = deny, not timeout = auto-approve. If the human doesn't actively engage, nothing happens. This inverts the typical pattern where inaction leads to approval.
3. Bedrock Guardrails: Output Validation
Use dashboard that gives reviewers context they wouldn't otherwise have:
- How many actions has this agent proposed in the last hour? (Spike = suspicious)
- What's the agent's recent approval rate? (100% = nobody's actually reviewing)
- Has the agent's behavior pattern changed? (New tool calls, new recipients)
- Is this action type unusual for this time of day?
cloudwatch = boto3.client("cloudwatch")
# unchanged - this was already correct: record how long each approval took
cloudwatch.put_metric_data(
Namespace="AgentTrust",
MetricData=[{
"MetricName": "ApprovalDecisionTimeSeconds",
"Value": time_to_decision,
"Unit": "Seconds",
"Dimensions": [
{"Name": "ReviewerId", "Value": reviewer_id},
{"Name": "ActionType", "Value": action_type},
],
}],
)
# Fire if ANY reviewer averages under 5s over the last hour.
# Metrics Insights reads the metric ACROSS its dimensions, so it actually sees the
# data the old dimensionless alarm never matched. GROUP BY + ORDER BY ASC + LIMIT 1
# reduces the query to one series: the fastest-clicking reviewer.
cloudwatch.put_metric_alarm(
AlarmName="rubber-stamp-detection",
AlarmDescription="A reviewer averaging under 5s per decision is not reading.",
Metrics=[{
"Id": "fastest_reviewer",
"Expression": (
'SELECT AVG(ApprovalDecisionTimeSeconds) FROM "AgentTrust" '
'GROUP BY ReviewerId ORDER BY AVG() ASC LIMIT 1'
),
"Period": 3600,
"ReturnData": True,
}],
EvaluationPeriods=1,
Threshold=5,
ComparisonOperator="LessThanThreshold",
TreatMissingData="notBreaching",
AlarmActions=["arn:aws:sns:us-east-1:123456789012:security-team"],
)
If the average decision time is under 5 seconds, someone's clicking "approve" without reading. That's a security signal worth acting on.
5. Provenance and Source Attribution
Never show humans only the agent's interpretation. Show the source data alongside it. On AWS, this means:
- Store original source documents in data events
- Tag agent outputs with confidence scores and source citations
- Clearly label what's "agent-generated rationale" vs "verified source data"
If the reviewer can see both the agent's recommendation AND the raw invoice/document/data it was based on, they can spot discrepancies. If they only see the agent's summary, they can't.
Key Takeaway
Don't trust the human to catch the agent's mistakes. Design the review workflow so that rubber-stamping is structurally impossible. Timeouts that default to deny. Mandatory verification questions. Decision-time monitoring. Raw source data alongside agent summaries. And alerts when the humans stop actually reviewing.
The bug isn't in the agent. It's in the assumption that a busy human will reliably catch a confident-sounding AI that's been right 49 times in a row and is wrong on the 50th.
Up Next
or Twitter, or drop them below. If you've built human review workflows for agentic systems, I'd love to hear what worked and what didn't.
Onward!!
SOCIAL SHARE CARD GENERATOR