Local LLM inference has an expensive habit:
It recomputes prefixes it has already seen.
A system prompt.
A reused RAG document.
A few-shot block.
A long static context.
If the prefix is identical, why pay the prefill cost again?
That's the problem I explored with EdgeSync-LLM.
The idea
The mechanism is simple:
Prompt = shared prefix + new suffix
On the first request, EdgeSync prefills the prefix and captures its KV cache state.
On the next request sharing that exact prefix, it restores the state and decodes only the new suffix.
No llama.cpp fork.
No patch.
The current validated path uses the public:
llama_state_seq_get_data
and
llama_state_seq_set_data
APIs.
Measured on a real Android ARM64 phone
Model:
Qwen2.5-0.5B-Instruct Q4_K_M
Shared prefix:
123 tokens
40 requests. 4 threads. Release build.
Path
Mean TTFT
p50
p95
Cold
4828 ms
4752 ms
5297 ms
KV state reuse
486 ms
476 ms
569 ms
9.9× lower TTFT on cache hits.
The warm path was approximately:
363 ms to decode the 10-token suffix
123 ms to restore the state blob
Fragment size: 1.64 MB
I also measured the same mechanism on x86-64.
Cold mean TTFT: 1395 ms
Warm mean TTFT: 185 ms
That's 7.5× on cache hits.
But I almost published a fake 8.8× speedup
This was the most important part of the project.
My first implementation directly copied raw K/V tensors.
It was fast.
Very fast.
The benchmark reported an 8.8× speedup.
There was one problem.
It was wrong.
llama.cpp tracks more than the K/V tensor values. Cache cells also have position and sequence metadata used to construct the attention mask.
Copying tensor values without restoring that bookkeeping produced an inert fragment.
The model skipped prefix computation...
...but attention could not actually see the restored prefix.
14 of 24 cache hits reproduced, token for token, the output of a generation with no prefix at all.
The “speedup” was dropped context.
So I discarded it.
Timing is not enough
A broken cache can be fast.
That's why EdgeSync now runs two correctness checks.
First:
Warm output must match cold output for the same prompt.
Results:
ARM64: 28/28
x86-64: 24/24
Second:
The warm output is compared with a generation from the suffix alone, without the prefix.
If both outputs match, the injected fragment may be doing nothing.
Results:
ARM64: 0/28 matches
x86-64: 0/24 matches
This second control is what caught the broken raw-tensor implementation.
A latency benchmark alone would not have caught it.
What I learned
There is a broader lesson here for LLM infrastructure benchmarking:
Skipping computation is not proof that state reuse works.
If a cache implementation silently drops context, it can look extremely fast.
Correctness must be part of the benchmark.
Not a separate test.
Not an assumption.
Part of the measurement itself.
What EdgeSync does not claim
I deliberately marked several parts of the project as unvalidated or unsound.
Semantic approximate prefix reuse is not proven.
Reusing KV state between merely “similar” prefixes is incorrect with the current mechanism.
Cross-engine reuse does not work with opaque llama.cpp sequence-state blobs.
Per-layer striding cannot preserve output.
The Android JNI bridge and several cache management components still need end-to-end validation.
The on-device ARM64 sampling is also thin and needs to be rerun with more timed repeats.
I would rather publish a smaller result that can be falsified than a larger claim that cannot survive reproduction.
What's next
The next steps are:
Rerun ARM64 with multiple timed repeats.
Compare complete generated strings instead of one token.
Benchmark HNSW against an exact radix-tree lookup.
Measure state restoration cost as prefix length increases.
Remove or redefine the current PARTIAL-hit mechanism.
The code, benchmark methodology and measured manifests are public.
EdgeSync-LLM:
https://github.com/bossandboss/EdgeSync-LLM
If you work on llama.cpp, local LLM inference, KV cache management or edge AI, I would especially value attempts to reproduce or break the benchmark.
Finding a flaw is more useful to me than a GitHub star.
Intelligence View
⚡ tsecurity.de Intelligence
I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State
Local LLM inference has an expensive habit: It recomputes prefixes it has already seen. A system prompt. A reused RAG document. A few-shot block. A long static context. If the prefix is identical, why pay the prefill cost again? That's the…
Reagiere als Erste:r — dein Feedback zählt!
1. Sofort-Triage & Abwehrmaßnahmen
SOC Incident Playbook: Vulnerability Remediation & Verification
Syntax validiert (0 Fehler)
title: Detect Exploitation - I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State
id: 3615d6d6-ed3b-4e4c-9ea8-2a15caab3d67
status: experimental
description: Automatisch generierte SIEM-Erkennungsregel basierend auf CTI Intelligence
references:
- https://tsecurity.de/
author: iShareStuff CTI Automated Detection Engine
date: 2026-09-27
logsource:
category: network_connection
product: any
detection:
selection:
CommandLine|contains:
- 'exploit'
condition: selection
falsepositives:
- Legitime administrative Zugriffe oder Penetrationstests
level: high
tags:
- attack.initial_accessSyntax validiert (0 Fehler)
rule CTI_Threat_Indicator {
meta:
author = "iShareStuff CTI Automated Detection Engine"
date = "2026-09-27"
description = "YARA Signature for "
strings:
$str = "I Got 9.9 Lower TTFT on a Real" ascii wide
condition:
any of them
}Syntax validiert (0 Fehler)
index=security sourcetype IN ("cisco:asa", "pan:traffic", "zeek_conn", "suricata", "WinEventLog:Security")
("I Got 99 Lower TTFT on a Real Android Ph")
| stats count earliest(_time) as first_seen latest(_time) as last_seen by src_ip, dest_ip, dest_host, signature
| eval first_seen=strftime(first_seen, "%Y-%m-%d %H:%M:%S"), last_seen=strftime(last_seen, "%Y-%m-%d %H:%M:%S")
| sort - countSyntax validiert (0 Fehler)
message: "*I Got 99 Lower TTFT on a Real Android Ph*"Syntax validiert (0 Fehler)
CommonSecurityLog
| where Message has "I Got 99 Lower TTFT on a Real Android Ph"
| summarize EventCount = count(), FirstSeen = min(TimeGenerated), LastSeen = max(TimeGenerated) by SourceIP, DestinationIP, DestinationPort, Activity
| extend DetectionRule = "iShareStuff-CTI-Compiled"
| sort by EventCount desc2. Cyber Threat Intelligence & Forensik
🎯
MITRE ATT&CK Matrix Navigator 14 Taktiken
Reconnaissance
-
Resource Development
-
Initial Access
Execution
Persistence
-
Privilege Escalation
Defense Evasion
Credential Access
-
Discovery
-
Lateral Movement
-
Collection
-
Command and Control
Exfiltration
-
Impact
tsecurity.de Cognitive Threat RAG
Fokus-Vektor:
Analyse für identifizierte Bedrohung auf Basis von Live-CTI (ENISA EUVD): CVSS 0.0 · EPSS 0.0% · CISA KEV: nein. Handlungsableitung aus den verlinkten Hersteller-Quellen.
🛡️ Angriffsfläche & Exposure
Netzwerk/Remote-Zugriff ohne Vorauthentifizierung möglich.
⚡ Empfohlene Sofortmaßnahmen
- 1. Perimeter-Inspektion: Relevante Portfreigaben und exponierte Endpunkte unverzüglich scannen.
- 2. Patch-Applikation: Hersteller-Hotfix einspielen oder betroffene Daemons in isolierte DMZ-Segmente überführen.
- 3. Telemetrie & EDR-Alerts: Prozessaufrufe und Child-Processes auf anomale Shell-Spawns überwachen.
Synthetische RAG-Antwort