Idempotency: Make Retries Safe

engineering architecture #csharp #dotnet #idempotency #background-processing

An operation is idempotent when repeating it has the same intended effect as doing it once. That property is called idempotency: the code can run more than once without applying the same business change again.

For example, a background job grants a customer 100 reward points. The update succeeds, but the acknowledgement never reaches the caller. The caller sees a timeout and submits the job again. If the worker treats the second delivery as a new grant, the customer gets 200 points for one purchase.

After a timeout, the caller doesn't know whether the operation failed before changing anything, completed successfully, or is still running. It only knows that no result arrived in time. The same gap appears when an API connection breaks, a worker stops before acknowledging a message, or a scheduled task is submitted twice.

Retries help recover from temporary failures, but they can also repeat a business change. Before adding a retry policy, decide how the application will handle that repetition. A longer timeout may make the problem less frequent, but it cannot establish whether an earlier attempt already succeeded.

The example uses a C# background worker, but the same retry problem applies to APIs and message consumers.

Start with the effect, not the retry mechanism

Setting an account's preferred language to English is naturally repeat-safe. Repeating that assignment leaves the preference set to English. Adding 100 points to a balance is different: each addition changes the balance again. To make a grant repeat-safe, the system needs to recognize that several attempts represent one grant.

HTTP contracts use the same distinction. RFC 9110 defines idempotency in terms of the intended server effect. It identifies PUT, DELETE, and safe methods as idempotent, but does not require every repeated request to return an identical response.[1] A repeated deletion can find that the resource is already gone while preserving the intended outcome.

The reward operation in this example also saves its result as a receipt and returns it on retry. That lets the caller recover the result it missed. Returning the receipt is part of this application's contract; idempotency alone only requires the intended effect to stay the same.

A log entry for every attempt does not necessarily break idempotency; awarding points for every attempt does.

A queue can deliver duplicate work too. Amazon SQS documents that standard queues can deliver a message copy again and recommends idempotent processing.[2] Check your queue's contract: delivery rules vary, and a local queue does not automatically provide recovery.

Give one operation one identity

Use the original business operation's key for its retries. With a fresh key for every attempt, the receiver sees several different grants instead of several deliveries of one grant.

For a purchase reward, the order identifier can be part of that identity if the business rule permits exactly one such grant per order. If an order can generate multiple reward operations, an order identifier alone is insufficient. The key must distinguish those operations without changing when one of them is retried.

The example scopes a key to an account and to the grant-points operation. The same textual key on another account is a separate operation.

For multiple tenants or operation types, include those boundaries in the key's scope too. One global key space risks collisions between unrelated callers.

Reusing a key also means keeping the request the same. A caller cannot use the same account and key to grant 100 points, then change the amount to 200 points and still call it a retry. The receiver must reject that mismatch rather than silently return an unrelated result or apply the changed request.

Matching payloads do not prove that two requests are for the same operation. A customer can earn 100 points from each of two purchases. Deduplicating only by account and amount would discard the second purchase's reward.

The operation key expresses intent; comparing the payload prevents that key from being reused for different work.

The smallest background-processing example

The C# solution has one .NET 10 console project with a points grant, an in-memory store, and a background consumer. Running it demonstrates the retry and checks the expected behavior.

From the code-samples repository root:

dotnet run --project idempotency-make-retries-safe/IdempotencyDemo

In GrantPoints.cs, the operation is just data:

public sealed record GrantPoints(string Key, string AccountId, int Points);

A delivery is an attempt to process that data. It can fail to report success even though the grant itself succeeded.

PointWorker.cs passes deliveries through a queue inside the process, using Channel<T>. Producers write to it, and an asynchronous consumer reads from it. Channels are part of the .NET shared framework. With a bounded channel and Wait, asynchronous writes wait for space instead of dropping work.[3]

private readonly Channel<Delivery> inbox = Channel.CreateBounded<Delivery>(
    new BoundedChannelOptions(16) { SingleReader = true, FullMode = BoundedChannelFullMode.Wait });

The capacity is for this demo, not a throughput recommendation. Producers can submit concurrently, but there is one consumer. It runs in the same process, not as a separate service or necessarily on a dedicated thread.

This channel is not a durable broker and does not redeliver a lost message after a process crash. The demo caller submits the retry itself, so you can see exactly when it happens.

Protect the grant and save its result

The store keeps account balances and completed-grant receipts. A receipt records the account, operation key, granted points, and resulting balance. GrantResult.cs defines that saved result.

The complete Apply method in PointStore.cs handles both new grants and retries:

public GrantResult Apply(GrantPoints grant)
{
    ArgumentNullException.ThrowIfNull(grant);
    ArgumentException.ThrowIfNullOrWhiteSpace(grant.Key);
    ArgumentException.ThrowIfNullOrWhiteSpace(grant.AccountId);
    ArgumentOutOfRangeException.ThrowIfNegativeOrZero(grant.Points);

    lock (gate)
    {
        var key = (grant.AccountId, grant.Key);
        if (completed.TryGetValue(key, out var previous))
        {
            if (previous.Points != grant.Points)
                throw new InvalidOperationException("The same key cannot describe a different grant.");

            return previous;
        }

        var balance = checked(balances.GetValueOrDefault(grant.AccountId) + grant.Points);
        var result = new GrantResult(grant.Key, grant.AccountId, grant.Points, balance);
        balances[grant.AccountId] = balance;
        completed.Add(key, result);
        return result;
    }
}

The method first checks the input. A missing account, blank key, or nonpositive amount is not a temporary delivery failure. Retrying the same invalid input cannot fix it. Arithmetic uses checked, so an overflow fails before the method changes the balance or records the operation as completed.

Inside the lock, the store looks up the account/key pair. If a receipt exists, the store checks that the points match and returns it without adding points. Otherwise, it calculates the new balance, updates the account, and saves the result.

The lock covers the whole decision, not just the dictionary access. Two callers must not both observe a missing key and each apply the grant. A thread-safe map alone would not make a separate balance update and receipt insertion one protected operation.

All callers need to share one store instance. A new store per delivery has its own balances, receipts, and lock. Separate application processes also have separate stores; a local lock cannot coordinate them.

The lock coordinates callers, but it cannot roll back a partial update or recover data after a crash or restart. For example, if updating the balance succeeds but saving the receipt fails, the two dictionaries disagree. The lock is not a database transaction.

For a database-backed version, see the transaction and uniqueness rules in the restart section below.

Lose the acknowledgement after success

The worker simulates a lost acknowledgement after store.Apply returns: the points have been added, but the caller receives a timeout instead of the result. The consumer loop in PointWorker.cs places the failure at that exact point:

private async Task ConsumeAsync()
{
    await foreach (var delivery in inbox.Reader.ReadAllAsync())
    {
        try
        {
            var result = store.Apply(delivery.Grant);
            if (delivery.LoseAcknowledgement)
                throw new TimeoutException("Simulated acknowledgement loss after the grant succeeded.");

            delivery.Completion.SetResult(result);
        }
        catch (Exception error) when (error is ArgumentException or InvalidOperationException or OverflowException or TimeoutException)
        {
            delivery.Completion.SetException(error);
        }
    }
}

Each delivery has a completion task so the submitting caller can observe its result or exception. The worker reports expected processing failures to that caller and continues with later deliveries. It does not catch a conflict and pretend the work succeeded.

There is no network connection, timer race, or broker acknowledgement in this project. Throwing at a fixed point makes the simulated failure repeatable: the grant succeeds before the caller sees the timeout.

The caller retries the original operation. This block is copied from Program.cs, including the checks that throw if the guarantee fails:

var backgroundStore = new PointStore();
await using (var worker = new PointWorker(backgroundStore))
{
    var work = new GrantPoints("order-1042", "account-7", 100);
    await ExpectAsync<TimeoutException>(() => worker.ProcessAsync(work, loseAcknowledgement: true),
        "the first delivery loses its acknowledgement after applying the grant");
    Check(backgroundStore.GetBalance("account-7") == 100,
        "the grant succeeded despite the missing acknowledgement");
    var retry = await worker.ProcessAsync(work);
    Check(retry.Balance == 100 && backgroundStore.GetBalance("account-7") == 100,
        "retrying through the worker adds no extra points");
}

By the time the first attempt reports a timeout, the balance is already 100. The retry returns the saved balance of 100 without changing the account again. The worker handled two deliveries of the same grant.

The program also checks that changed payloads are rejected, keys are scoped to accounts, and new grants add points. Other checks cover concurrent store callers, invalid input, overflow, continued processing after a rejected request, and rejection of new deliveries after shutdown. If any check fails, dotnet run exits with an error.

A saved receipt is not the current account state

Suppose a later purchase grants another 25 points. The account now has 125. Retrying the original 100-point grant should return its saved receipt, whose resulting balance was 100, while leaving the current balance at 125.

The receipt describes the earlier grant, so recalculating it from the current balance would change its meaning. Provide a separate read operation for callers who need the latest balance.

The sample checks this behavior, then submits the grant with a fresh key and confirms that it adds points again. The store treats that key as a new operation. Keeping the original key through every retry is the caller's responsibility; the store cannot infer that a new key was meant to be a retry.

What changes when the process can restart

When the process exits, it loses the balances, queued deliveries, and receipts. The sample shows how duplicate handling works while the process is running. It cannot recognize an earlier grant after a restart.

Persisting only the completed keys is not enough. If you record the key before applying the effect, a crash can leave it marked complete even though no points were granted.

If you apply the points first and record the key afterward, a crash can preserve the points without the receipt. A retry can then add them again.

For business state and receipts in one database, make those writes part of one transaction. Enforce uniqueness for the operation's scope in that shared store, so different workers cannot commit the same operation independently.

Return or acknowledge success only after commit. A worker that loses its acknowledgement after commit can then read the committed receipt on retry.

If two attempts conflict on the uniqueness constraint, the one that cannot commit needs to roll back its changes and retrieve the committed result. It should not apply the grant again.

For lengthy work in an external service, decide separately which worker owns the operation and how another worker can recover it. Holding a local lock does not make a remote call transactional.

The console project leaves out the database so you can run it without setup and follow the duplicate-handling code directly. Its shared store coordinates callers only while the process is running. Extending that guarantee across restarts or processes requires durable storage and transactions.

Retention and external effects define the boundary

If you delete completed-operation records, an old key may look new again. Keep the records long enough to cover the retries you need to support, including delayed jobs, recovery, and manual reprocessing. Choosing when to delete them changes which retries are safe; it isn't just a way to save storage.

Provider rules differ. Stripe's API v1 documentation describes saving the first response after endpoint execution begins, rejecting changed parameters, and allowing keys to be removed once they are at least 24 hours old. The same documentation describes different retry behavior for API v2.[4] Do not copy one provider's retention period or error replay rules into every application.

The demo retains all receipts for its finite run. It has no expiration policy or cleanup process.

In a real service, define what a duplicate receives while the first attempt is running and which failures remain retryable. Also decide whether durable business data can identify an operation after its receipt expires.

Authorization must still happen on every attempt. An operation key is an identifier, not permission to read another account's receipt. Derive tenant and account scope from the authenticated caller where appropriate, and do not let guessed keys expose saved results.

An email, payment-provider call, or message sent to another service sits outside a local database transaction. A record saying that points were granted does not prove that an email was sent once. Use a stable operation identity at the receiving boundary when it supports one.

An outbox can preserve the intent to send alongside a committed state change, but delivery can still repeat. The downstream effect still needs its own protection or recovery policy.

Retry only failures that may recover, and limit the attempts. Changed payloads and invalid input need correction. A temporary failure may justify retrying with the same identity. If the outcome remains unknown, check the operation's actual state rather than retrying indefinitely.