API layer pattern (HTTP)
1) Contract
- Require
Idempotency-Keyon POST that creates resources (optional on GET, safe methods). - For PUT/PATCH, prefer natural keys (resource URI) + concurrency (ETag) over custom keys.
- Return
Idempotency-Replayed: trueon replays.
2) What to persist
(TenantId, IdempotencyKey)— unique.- Request fingerprint (stable hash of method + path + canonicalized body).
- First status code, headers (whitelisted), response body (or a pointer), timestamps, and a short TTL.
SQL table (Azure SQL) example
CREATE TABLE Idempotency (
TenantId NVARCHAR(64) NOT NULL,
IdempotencyKey NVARCHAR(128) NOT NULL,
RequestHash VARBINARY(32) NOT NULL, -- SHA-256
StatusCode INT NULL,
ResponseBody VARBINARY(MAX) NULL, -- or NVARCHAR(MAX) if JSON
CreatedAtUtc DATETIME2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
CompletedAtUtc DATETIME2(3) NULL,
CONSTRAINT PK_Idem PRIMARY KEY (TenantId, IdempotencyKey)
);
CREATE INDEX IX_Idem_Created ON Idempotency (CreatedAtUtc);
(Cosmos DB works too: define a unique key on /tenantId/idempotencyKey and set TTL on the container.)
3) Minimal middleware/filter in ASP.NET Core
public class IdempotencyMiddleware : IMiddleware
{
private readonly IIdemStore _store; // wraps Azure SQL or Cosmos
public IdempotencyMiddleware(IIdemStore store) => _store = store;
public async Task InvokeAsync(HttpContext ctx, RequestDelegate next)
{
var tenantId = ctx.User.FindFirst("tenant_id")?.Value ?? "public";
var key = ctx.Request.Headers["Idempotency-Key"].ToString();
if (string.IsNullOrWhiteSpace(key)) { await next(ctx); return; }
var hash = await HashRequestAsync(ctx.Request); // method+path+normalized body
var existing = await _store.TryGetAsync(tenantId, key);
if (existing is { Completed: true } && existing.RequestHash.SequenceEqual(hash))
{
ctx.Response.Headers["Idempotency-Replayed"] = "true";
ctx.Response.StatusCode = existing.StatusCode;
if (existing.ResponseBody is { Length: > 0 })
{
ctx.Response.ContentType = "application/json";
await ctx.Response.Body.WriteAsync(existing.ResponseBody);
}
return;
}
// Reserve the key up-front (prevents thundering herds)
var reserved = await _store.ReserveAsync(tenantId, key, hash); // insert if not exists
if (!reserved) // concurrent duplicate while first is in-flight
{
// Poll or short 409 telling client to retry shortly; or wait on a small backoff and re-check
ctx.Response.StatusCode = StatusCodes.Status409Conflict;
await ctx.Response.WriteAsync("{\"error\":\"Request in progress\"}");
return;
}
// Capture response
var originalBody = ctx.Response.Body;
await using var mem = new MemoryStream();
ctx.Response.Body = mem;
try
{
await next(ctx);
await _store.CompleteAsync(tenantId, key, ctx.Response.StatusCode, mem.ToArray());
}
finally
{
mem.Position = 0;
await mem.CopyToAsync(originalBody);
ctx.Response.Body = originalBody;
}
}
static async Task<byte[]> HashRequestAsync(HttpRequest req)
{
req.EnableBuffering();
using var sha = System.Security.Cryptography.SHA256.Create();
var body = "";
if (req.ContentLength > 0)
{
using var reader = new StreamReader(req.Body, leaveOpen:true);
body = await reader.ReadToEndAsync();
req.Body.Position = 0;
}
var canonical = $"{req.Method}\n{req.Path}\n{NormalizeJson(body)}";
return sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(canonical));
}
static string NormalizeJson(string json) => string.IsNullOrWhiteSpace(json)
? ""
: System.Text.Json.JsonDocument.Parse(json).RootElement.GetRawText(); // canonical spacing/order for objects mostly stable in client contracts
}
Notes
ReserveAsyncdoes an INSERT guarded by the unique key; if it fails, another request holds the key.- Set a TTL/cleanup job (e.g., 24–72h) to trim completed entries.
PUT semantics with ETags (Concurrency)
- PUT should replace the full resource representation (idempotent by nature).
- Use ETag +
If-Matchto prevent lost updates.
Cosmos DB / Azure Table / Azure SQL
- Cosmos & Tables expose ETags natively; for SQL, emulate with a
rowversioncolumn.
Controller snippet
[HttpPut("{id}")]
public async Task<IActionResult> Put(string id, [FromBody] Widget dto, [FromHeader(Name="If-Match")] string etag)
{
// Load current, compare ETag/rowversion; if mismatch => 412 Precondition Failed
var ok = await _repo.ReplaceAsync(id, dto, etag);
return ok ? NoContent() : StatusCode(StatusCodes.Status412PreconditionFailed);
}
Messaging/Queue consumers (de-dup & exactly-once illusions)
Azure Service Bus (recommended over Storage Queues for de-dup)
- Enable Duplicate Detection on the queue/topic (
DuplicateDetectionHistoryTimeWindow, e.g., 10 minutes–7 days). - Set
MessageIdto a business idempotency key (e.g., the order ID or the HTTP Idempotency-Key). - Still make the handler idempotent (at-least-once delivery can still surface via re-enqueue/abandon).
Producer
var client = new ServiceBusClient(connStr);
var sender = client.CreateSender("orders");
var msg = new ServiceBusMessage(BinaryData.FromObjectAsJson(order))
{
MessageId = order.Id, // business key
Subject = "OrderCreated"
};
await sender.SendMessageAsync(msg);
Function/Worker handler (inbox pattern)
public class OrderHandler
{
private readonly IInboxStore _inbox; // Azure SQL/Cosmos unique on (MessageId)
public async Task HandleAsync(ServiceBusReceivedMessage m, CancellationToken ct)
{
// Fast de-dup check
if (!await _inbox.ReserveAsync(m.MessageId)) return; // already processed
try
{
var order = m.Body.ToObjectFromJson<OrderCreated>();
await ProcessAsync(order, ct); // make this idempotent
await _inbox.CompleteAsync(m.MessageId);
}
catch
{
await _inbox.ReleaseAsync(m.MessageId); // allow retry
throw;
}
}
}
Azure Storage Queues / Event Hubs
- No built-in de-dup for Storage Queues; always use an inbox table keyed by a message/business id.
- Event Hubs is stream-oriented; treat processing as idempotent, or checkpoint and maintain an inbox keyed by event idempotency key.
Outbox (publish exactly-once to the bus)
- In the same DB transaction as state change, write Outbox rows (events to publish).
- A background dispatcher reads un-sent rows and publishes to Service Bus; upon success, marks them sent.
- Libraries: NServiceBus/MassTransit have outbox patterns built-in; or roll your own with EF Core.
EF Core Outbox table
CREATE TABLE Outbox (
Id UNIQUEIDENTIFIER PRIMARY KEY,
OccurredUtc DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
Type NVARCHAR(200) NOT NULL,
Payload NVARCHAR(MAX) NOT NULL,
Sent BIT NOT NULL DEFAULT 0
);
Hashing strategy (practical)
- Canonicalize:
METHOD + PATH + canonical(JSON body) + critical headers (e.g., Content-Type, Version) + TenantId. - Use SHA-256; store 32-byte hash.
- Hash matters for replay safety: if key reused with different payload, reject with
422 Unprocessable Entity(or409 Conflict) to avoid accidental cross-use.
Where to put what (Azure choices)
Idempotency store:
Cosmos DB: low latency, unique key on(tenantId, idemKey), TTL per item.
Azure SQL: great if you already use SQL; unique composite key,rowversionfor concurrency.
Redis: fast reservation locks viaSETNX+ TTL—but still persist the final response in durable storage.
Queue: Azure Service Bus with duplicate detection + sessions if you need ordered processing per aggregate.
Error handling & UX
- If a duplicate arrives before the first completes: return 409 (“in progress”) or 202 with a status URL the client can poll.
- If a duplicate arrives after completion with the same hash: return the original status/body with
Idempotency-Replayed: true. - If same key, different hash: 422/409 to force client to pick a new key.
Observability
- Log the idempotency key, request hash, and replay status.
- Add metrics: “idem_hits”, “idem_inflight”, “idem_conflicts”, and “handler_dedup_hits”.
Summary: Implementation Checklist (.NET + Azure)
HTTP: requireIdempotency-Keyon POST creates; middleware reserves key → executes → stores response → replays on duplicates.
PUT: design as full replace + ETag (If-Match) to make retries safe and prevent lost updates.
Queues: use Service Bus withDuplicateDetection+MessageId; consumers maintain an inbox table and business-idempotent handlers.
Outbox: publish messages from a DB outbox to get “exactly-once” effect across boundaries.
Persistence: Cosmos (unique key + TTL) or SQL (unique index).
Hashing: canonical SHA-256 over method/path/body/tenant; reject key reuse with different payloads.
SOCIAL SHARE CARD GENERATOR