We recently migrated a .NET Core application from .NET 8 to .NET 10 and moved its hosting model from AWS Lambda to ECS. The service acts as a wrapper around a downstream API that reads and updates client “fact find” data.
Post-migration, we started seeing critical issues:
Users were seeing other users’ data
Updates were being applied to the wrong clients
Users could access data they shouldn’t have permission to see
This was a data isolation failure. The root cause turned out to be a misuse of a singleton combined with differences in execution models between Lambda and ECS.
Architecture Context
The service:
Accepts user requests
Adds headers (auth/user context)
Calls a downstream API
Returns processed responses
A shared configuration object (ConnectionOptions) was introduced to manage headers for outbound calls.
This object:
Was registered as a singleton
Was mutated per request to inject headers
Why It “Worked” on Lambda
AWS Lambda has a different execution model:
Each invocation typically runs in isolation
Even with warm starts, concurrency per instance is limited
No true parallel execution within a single instance (by default)
So:
Mutating a singleton per request appears safe
Concurrency issues are masked
The flawed design went unnoticed.
What Changed in ECS
ECS runs your app as a long-lived service:
Multiple requests are processed concurrently
Threads share the same memory space
Singleton services are shared across all requests
Example
public class ConnectionOptions
{
public Dictionary<string, string> Headers { get; set; }
}
services.AddSingleton<ConnectionOptions>();
connectionOptions.Headers["Authorization"] = userToken;
The Problem
Two requests arrive at the same time:
| Request A | Request B |
|---|---|
| Sets header to User A | Sets header to User B |
| Calls downstream API | Calls downstream API |
Because both share the same singleton:
Headers overwrite each other
Requests leak state
Downstream API gets incorrect user context
This leads to:
Cross-user data exposure
Incorrect updates
Non-deterministic failures
Root Cause
Mutable shared state in a singleton in a concurrent environment.
More precisely:
ConnectionOptionsheld request-specific dataIt was registered as a singleton
It was mutated per request
This violates a core rule:
Singletons must be stateless or immutable.
Why This Is Hard to Catch
This issue didn’t show up in:
Local development (low concurrency)
Lambda (isolated execution)
Unit tests (typically single-threaded)
It only surfaced under:
Concurrent load
Multi-threaded execution (ECS)
Fixes
Option 1: Use Scoped Lifetime
services.AddScoped<ConnectionOptions>();
Each request gets its own instance.
Option 2: Make It Immutable
public class ConnectionOptions
{
public IReadOnlyDictionary<string, string> Headers { get; }
public ConnectionOptions(Dictionary<string, string> headers)
{
Headers = headers;
}
}
Create a new instance per request instead of mutating.
Option 3: Avoid Shared State (Preferred)
Pass headers explicitly:
await client.CallAsync(headers);
Or:
var request = new HttpRequestMessage();
request.Headers.Add("Authorization", token);
This removes shared mutable state entirely.
Key Takeaways
1. Execution Model Matters
Lambda and ECS behave differently:
Lambda hides concurrency issues
ECS exposes them
2. Singleton ≠ Safe
Singletons are:
Shared globally
Unsafe if mutable
Use only for:
Stateless services
Immutable configuration
3. Don’t Store Request Data in Singletons
If data changes per request:
- It should not live in a singleton
4. Concurrency Bugs Are Non-Deterministic
They:
Are hard to reproduce
Appear random
Often show up only in production
Final Thought
This wasn’t a .NET 10 issue or an ECS issue.
It was a design flaw that only became visible when the runtime stopped masking it.
When migrating from Lambda to ECS (or any long-running service model), review:
Singleton usage
Shared mutable state
Request-scoped data handling
That’s where subtle, high-impact bugs tend to hide.
SOCIAL SHARE CARD GENERATOR