The pool that got slower the more we shared it

How a 15-minute chat with an LLM found a hidden ArrayPool<byte>.Shared

contention point that no static analyzer would ever flag — and why the fix

had nothing to do with adding a lock.

The ticket

"CPU on market-data-gateway climbed ~20% after we added a broadcast fan-out

path last sprint. No new allocations, no GC pressure, nothing obviously

wrong in the diff. Can someone take a look before we scale out more pods?"

Nobody had touched anything that looked like a synchronization primitive.

The diff that shipped the week before only touched a buffer-pooling helper —

swapping a bespoke byte-array cache for the "obviously correct" BCL default,

ArrayPool<byte>.Shared. That refactor was reviewed and merged specifically

because it removed custom code in favor of a well-known, heavily optimized

framework type. If anything, it should have been a free win.

This is the kind of regression static analysis is structurally blind to: no

analyzer rule says "using ArrayPool<T>.Shared across threads is suspicious."

It's the recommended way to pool buffers in .NET. The problem isn't in the

code you'd read in a diff — it's in a runtime behavior that only shows up

under a specific, easy-to-miss usage pattern.

The investigation, as it actually happened

This is the real value proposition of ,

is for: static navigation and on-demand decompilation of compiled assemblies,

including the BCL itself.

assistant> Let's not guess. I'll look up the actual pool implementation

in the runtime binary you're running.

→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^Return$")

{

  "summary": "5 match(es) for /^Return$/.",

  "matches": [

    { "typeFullName": "System.Buffers.SharedArrayPool`1", "methodName": "Return",

      "signature": "void System.Buffers.SharedArrayPool`1.Return(!0[], bool)",

      "handle": "m:e8c78f6b-...:0x060038A7" }

  ]

}

assistant> Decompiling the real Return method for this exact runtime build.

→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038A7")

public override void Return(T[] array, bool clearArray = false)

{

    // ...

    SharedArrayPoolThreadLocalArray[] array2 = t_tlsBuckets ?? InitializeTlsBucketsAndTrimming();

    // ... same-thread fast-path slot swap ...

    if (array3 != null)

    {

        flag2 = (_buckets[num] ?? CreatePerCorePartitions(num)).TryPush(array3);

    }

    // ...

}

assistant> And the fallback path that pushes to a per-core partition:

→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^TryPush$")

→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038B3")

[MethodImpl(MethodImplOptions.AggressiveInlining)]

public bool TryPush(Array array)

{

    bool result = false;

    Monitor.Enter(this);

    // ... push onto a small fixed-size array-backed stack ...

    Monitor.Exit(this);

    return result;

}

There it is — Monitor.Enter(this), in the flesh, in the exact binary

this process is running: the actual internal type backing

ArrayPool<byte>.Shared in this runtime build is SharedArrayPool<T>, and

its per-core fallback is guarded by a plain CLR monitor. This is why the

call-tree pointed straight at Monitor.Enter_Slowpath: it's a real, ordinary

monitor, contended by every thread whose Return misses the per-thread

cache.

To be clear: dotnet/runtime's source is public on GitHub — nothing here was

hidden. The point isn't that the source was unreachable, it's the ergonomics

of the query. Getting the same answer the "normal" way means: know (or guess)

which internal type backs ArrayPool<T>.Shared today, find the right tag in

dotnet/runtime that matches the exact SDK/runtime patch this process is

actually running, navigate to the right file, and read enough surrounding

code to trust you're looking at the live behavior and not a since-changed

version. decompile_method(moduleVersionId, metadataToken) collapses all of

that into one call, scoped to the exact assembly loaded in the exact process

under investigation — no version-matching, no guessing a type name, no

context-switch out of the chat.

Checking the application code (only now, after both tools already

pointed at the exact mechanism, not before):

// One dedicated broadcaster thread — the ONLY thread that ever Rents.

while (running)

{

    var buffer = ArrayPool<byte>.Shared.Rent(FrameSize);

    FillFromMarketFeed(buffer);

    channel.Writer.TryWrite(buffer);

}



// N per-connection consumer tasks — each may Return a buffer it never Rented.

await foreach (var buffer in channel.Reader.ReadAllAsync())

{

    await connection.WriteAsync(buffer);

    ArrayPool<byte>.Shared.Return(buffer); // <-- always a different thread than Rent

}

Exactly the pattern the tool output implied: one thread rents, many other

threads return.

Why this is invisible without live evidence

ArrayPool<T>.Shared isn't a naive shared bag — its current .NET 10

implementation, SharedArrayPool<T>, keeps a single-slot cache stored

per-thread ([ThreadStatic] buckets): if the same thread that rented a

buffer also returns it, the operation is a plain field swap, no

synchronization at all. The moment a different thread returns it, that fast

path can't apply (the slot belongs to another thread), so the buffer falls

through to a small set of per-core partitions, each guarded by a plain

Monitor. With a handful of cores and dozens of consumer threads all

returning concurrently, those few partitions become a genuine contention

point.

None of this is a bug in ArrayPool<T>. It's working exactly as designed —

optimized for the common case (rent-use-return on one thread, e.g. inside a

single request handler), and gracefully degrading, but not silently free,

for the cross-thread fan-out case. There is no compiler warning, no analyzer

rule, and no code-review heuristic for "you are about to violate an

implicit performance assumption of a BCL type you didn't write." The only way

to know, for certain, what this process, running this runtime build, is

actually doing is to watch where the CPU goes and read the exact binary it's

running.

Fix and verification

The fix isn't "add a lock" (there's nothing to protect) — it's swap the pool

implementation to one designed for the actual ownership pattern: many threads

returning items nobody-in-particular rented, i.e., a plain lock-free queue.

public sealed class CrossThreadBufferPool

{

    private readonly ConcurrentQueue<byte[]> _pool = new();

    public byte[] Rent(int size) => _pool.TryDequeue(out var buf) ? buf : new byte[size];

    public void Return(byte[] buffer) => _pool.Enqueue(buffer);

}

Isolated cost, before/after (BenchmarkDotNet, 20,000 cross-thread

Return calls, buffers pre-rented on a background thread outside the timed

region so only Return itself is measured):

The last row is the ceiling: the one scenario with genuinely zero

synchronization, because the per-thread cache slot is always hit. Comparing

it to the ConcurrentQueue fix shows exactly how much overhead our own fix

still carries: ~126 µs vs. ~115 µs — roughly 10% above the theoretical

best, for a data structure built to be used safely by an arbitrary number

of unrelated threads. That's the honest cost of "general-purpose

thread-safe," and it's a rounding error next to the 34x we recovered.

A single cross-thread Return call on the unfixed pool costs more than an

entire same-thread Rent+Return round trip. That's how disproportionate the

fast-path miss is.

Live re-check after the fix — same 2-thread (1 producer / 1 consumer)

topology, same CPU sampler, before touching anything else:

assistant> With the lock-free pool under the same load:

→ collect_sample(kind="cpu", processId=41213, durationSeconds=8, topN=20)

{

  "summary": "Hottest self-time method: System.Threading.Thread.Sleep(int32)

              (3972 exclusive, 40.6% of samples). No Monitor frames in the

              top 20 hotspots.",

  ...

}

No Monitor.Enter_Slowpath anywhere in the profile. The contention didn't

move — it disappeared, because the pool that produced it is gone.

The obvious follow-up question: is pooling even worth it here?

ArrayPool<T>.Shared itself exists as an optimization over the naive

baseline — just calling new byte[size] and letting the GC reclaim it. A

fair objection to everything above: if the shared pool is this contended

under cross-thread Return, maybe the honest fix is to drop pooling

entirely, not swap in a different pool. Mean latency might even look

competitive, since a plain allocation has zero synchronization of any kind —

but the expectation is that Gen0/Gen1 GC pressure would show up in the tail

(P99), not the mean.

We tested it — same 2-thread producer/consumer topology, only the pool

implementation swapped for one that allocates fresh on every Rent and does

nothing on Return:

No pooling isn't a tail-latency-only regression — it's ~18x slower on raw

throughput, worse than the contended ArrayPool.Shared baseline we started

from. The reason shows up immediately in a live GC pause capture over the

same window:

assistant> Let's check what removing the pool entirely does to GC.

→ collect_events(kind="gc", processId=41213, durationSeconds=10)

{

  "summary": "134 collection(s), max pause 69.9ms, total pause 5118.1ms."

}

assistant> Breaking that down:

→ query_snapshot(handle="...", view="pauseHistogram")

{

  "totalCollections": 134,

  "maxPauseTime": "00:00:00.0698743",

  "buckets": [

    { "label": "<1ms",      "count": 0 },

    { "label": "1-10ms",    "count": 0 },

    { "label": "10-100ms",  "count": 134 },

    { "label": "100-1000ms","count": 0 },

    { "label": ">=1s",      "count": 0 }

  ]

}

assistant> And by generation:

→ query_snapshot(handle="...", view="byGeneration")

{

  "generations": [

    { "bucket": "gen1", "count": 134, "totalPause": "00:00:05.1180759",

      "meanPause": "00:00:00.0381945", "maxPause": "00:00:00.0698743" }

  ]

}

134 gen1 collections in a 10-second window — 13.4 per second, every single

one landing in the 10–100ms bucket — for 5.12 seconds of accumulated pause

time inside those 10 seconds. Over half the wall clock is GC pause, not

application throughput. That's not a tail-latency footnote; it dominates the

mean too, once allocation volume is high enough to keep pushing 4KB buffers

into gen0 fast enough to promote and collect on a ~130ms cadence.

So the honest ranking, worst to best, for this specific cross-thread

fan-out pattern is: no pooling (worst, by both mean and tail) → shared pool

with cross-thread Return (contended but bounded) → a pool built for the

actual ownership shape (best on every axis measured). ArrayPool<T>.Shared

being contended here doesn't mean pooling was the wrong call — it means the

specific pool's ownership assumption (same-thread Rent/Return) didn't match

this workload's shape. The fix was never "go back to allocating"; it was

"pool it the way this workload actually uses it."

The takeaway

Static analysis, linters, and code review are excellent at catching shapes

of bugs — a missing await, a lock around the wrong object, an O(n²)

loop. They cannot catch a case where the code is a textbook-correct call into

a well-designed BCL type, and the only thing wrong is a runtime ownership

pattern that BCL type happens to be sensitive to. You either know this

specific gotcha already, or you find it by watching the process — there is no

third option.

dotnet-diagnostics-mcp exists so that "watching the process" is a normal

step in an LLM-driven investigation, not a specialized skill gated behind

whoever on the team happens to know dotnet-trace by heart. Every dynamic

step above — inspect_process, collect_sample, query_snapshot — is a

single MCP tool call an assistant can make on its own, against a real running

.NET process, with no code changes and no redeploy. And when the trail leads

into a BCL type nobody on the team has source for, its companion server,

,

a public repo, commit .

It ships both an MCP server (for your LLM client of choice) and a standalone

CLI (dotnet-diagnostics-cli) for scripting the same investigations by hand.