Idempotency keys are not optional

Every payment API gets called twice. Not occasionally, but routinely, and for reasons that have nothing to do with anyone doing something wrong.

A mobile client times out at 30 seconds while your gateway takes 32. The user taps the button again because nothing appeared to happen. A load balancer retries an upstream request it believes failed. A provider redelivers a webhook because your acknowledgement arrived after their timeout. A background job restarts mid-batch and replays from its last checkpoint.

In each case the request is genuinely delivered twice, both deliveries are legitimate, and exactly one of them should result in money moving. That is the entire problem.

The approach that looks correct and is not

The instinct is to check before acting:

// Do not do this.
var existing = await db.Payments
    .FirstOrDefaultAsync(p => p.OrderId == request.OrderId);

if (existing is not null)
    return Ok(existing);

var payment = await gateway.ChargeAsync(request);
db.Payments.Add(payment);
await db.SaveChangesAsync();
return Ok(payment);

This is a check-then-act race, and the window between the two is exactly where duplicate requests live. Two concurrent calls both run the FirstOrDefaultAsync, both get null, and both charge the card. The failure is timing-dependent, which means it will pass every test you write and surface in production under load.

Worse, it is silent. Nobody gets an error. There are simply two charges, and you find out when the customer does.

What an idempotency key actually is

An idempotency key is a client-generated identifier for an intent. Not for a request body and not for a resource, but for the specific thing the caller is trying to make happen, once.

The client generates it before the first attempt and reuses the same value across every retry of that same intent:

POST /v1/payments
Idempotency-Key: 7f3c1e8a-4b21-4c9e-9f0d-2a6b5c8e1d34
Content-Type: application/json

{ "orderId": "ORD-99213", "amountMinor": 4999, "currency": "EUR" }

The critical detail is that the key comes from the client. A server-generated key cannot work, because the server has no way to know that this request is a retry of the previous one. That knowledge only exists on the caller’s side.

Store the response, not a flag

A common half-implementation records that a key has been seen, then returns a bare 200 on replay. This is not enough. The caller retried because it never received the original response, so it still needs that response: the payment id, the status, the provider reference.

Store the full response:

CREATE TABLE idempotency_records (
    key                 VARCHAR(255)  NOT NULL,
    endpoint            VARCHAR(120)  NOT NULL,
    request_fingerprint CHAR(64)      NOT NULL,
    state               VARCHAR(20)   NOT NULL,  -- in_progress | completed
    response_status     INT           NULL,
    response_body       NVARCHAR(MAX) NULL,
    created_at          DATETIME2     NOT NULL,
    expires_at          DATETIME2     NOT NULL,
    CONSTRAINT pk_idempotency PRIMARY KEY (key, endpoint)
);

Scoping the key by endpoint matters. The same key arriving at /payments and at /refunds describes two different intents, and collapsing them would be wrong.

The in-flight case, which is the hard part

Most implementations handle the completed case, where the key is seen, the response is stored and it can be replayed, then quietly ignore the case where the duplicate arrives while the original is still running. That is the common case in practice, because the duplicate is usually triggered by the original taking too long.

The fix is to make the database enforce the ordering rather than your application logic. Insert the record first, and let the primary key constraint decide who wins:

public async Task<IActionResult> CreatePayment(
    [FromHeader(Name = "Idempotency-Key")] string key,
    PaymentRequest request)
{
    var fingerprint = Sha256(Canonicalise(request));

    try
    {
        await db.IdempotencyRecords.AddAsync(new IdempotencyRecord {
            Key = key,
            Endpoint = "POST /v1/payments",
            RequestFingerprint = fingerprint,
            State = "in_progress",
            CreatedAt = clock.UtcNow,
            ExpiresAt = clock.UtcNow.AddHours(24),
        });
        await db.SaveChangesAsync();   // wins or throws
    }
    catch (DbUpdateException e) when (e.IsUniqueViolation())
    {
        var record = await LoadRecord(key, "POST /v1/payments");

        // Same key, different body: the client has a bug. Say so loudly.
        if (record.RequestFingerprint != fingerprint)
            return UnprocessableEntity(new {
                error = "idempotency_key_reuse",
                message = "This key was used with a different request body."
            });

        // Original still running. Tell the caller to wait rather than guessing.
        if (record.State == "in_progress")
            return StatusCode(409, new {
                error = "request_in_progress",
                message = "The original request is still being processed."
            });

        return StatusCode(record.ResponseStatus!.Value, record.ResponseBody);
    }

    var result = await gateway.ChargeAsync(request);
    await CompleteRecord(key, 201, result);
    return StatusCode(201, result);
}

Three behaviours here are worth naming explicitly, because they are the ones usually missing.

The unique constraint is the lock. Not a distributed lock, not a Redis mutex, not an application-level check. The database already provides exactly the atomic test-and-set primitive this problem needs, and it survives process restarts and network partitions in a way an in-memory lock does not.

Fingerprint mismatch is an error, not a replay. If the same key arrives with a different body, something is genuinely wrong on the caller’s side, usually a key generated once per session rather than once per intent. Returning the original response would hide a real bug and silently drop a payment the client believes it made. Return 422 and let them find it.

In-flight returns 409, not a wait. Blocking the second request until the first completes seems friendlier, but it ties up a connection for the duration of an operation that is already slow, and it turns one hung request into two. A 409 with a Retry-After gives the client something actionable.

Canonicalising the fingerprint

Hashing the raw request body will produce false mismatches, because JSON serialisers reorder keys, vary whitespace, and format decimals inconsistently across client versions. Canonicalise before hashing: sort keys, normalise numbers to a fixed representation, drop fields that do not affect the outcome.

Then hash only the fields that determine what happens. A clientTimestamp or a traceId that legitimately differs between retries must not be part of the fingerprint, or every retry will be rejected as a mismatch.

Expiry

Idempotency records cannot live forever, but they must outlive any plausible retry window. Twenty-four hours is a reasonable default: long enough to cover a client with exponential backoff and an overnight outage, short enough that the table stays manageable.

After expiry, the same key is treated as new. This is a real, if unlikely, correctness gap: a retry arriving 25 hours later would charge twice. It is the accepted trade-off, and it is the reason the window should be measured in hours rather than minutes.

Money is never a float

Adjacent to idempotency but worth stating in the same breath, because both failures are silent: represent amounts as integer minor units with an explicit currency. 4999 and "EUR", never 49.99.

Binary floating point cannot represent most decimal fractions exactly. The error is invisible in a single calculation and accumulates across a settlement run. By the time it is large enough to notice, it has been wrong across every transaction in between, and reconstructing the correct figures is considerably more expensive than getting the type right at the start.

What this buys you

Idempotency done properly means a client can retry any request, any number of times, without reasoning about whether it is safe. That is a significant simplification. Retry logic becomes a transport concern rather than a business one, and the entire class of duplicate-charge incidents disappears.

It costs one table and roughly forty lines of middleware. It is close to the highest correctness-per-line-of-code available in a payment system, and it is much cheaper to build in at the start than to retrofit onto a ledger that already contains duplicates.