Distributed Locks in Real Applications: Preventing Duplicate Jobs, Token Refreshes, and Concurrent Updates

Your billing job ran every night at midnight and worked for a year. Then you scaled from one container to three, because one container is a single point of failure. That night all three woke up, all three ran the job, and your customers got the same report email three times.

You did not write a bug. The code is the same as it was yesterday. The only thing that changed is the number of copies running it.

This post shows you how to stop that with a distributed lock, and then shows you the case a lock cannot stop. Every code block below was run against Redis and PostgreSQL in Docker, and every log is real output from those runs.

Quick Answer

A distributed lock lets many servers agree that only one of them may do a piece of work at a time, by storing the claim somewhere they all read. In Redis that is a single SET command with NX and PX; in PostgreSQL it is an advisory lock. But every distributed lock is a lease with an expiry, not a guarantee: if your process stalls for longer than the lease, a second server takes the lock while the first still believes it holds it. So you use a lock to prevent wasted work, and you use the data layer, meaning unique constraints, version columns and fencing tokens, to prevent damage.

What Is a Distributed Lock?

On one server a lock is easy. The server keeps a note in its own memory that says busy, do not touch.

Run a second copy of your app and that note stops working. Each copy has its own memory, so copy A cannot see copy B's note. Both act at the same time.

A distributed lock moves the note outside all your servers, into one store they all read and write. Now there is one note, and everyone checks the same one.

It is a lease, not a lock

You do not hold a distributed lock until you finish. You hold it for a fixed number of seconds.

That is deliberate. If a server crashes while holding the lock, nothing is left to release it, and without an expiry that lock blocks the job forever. The expiry is what stops a dead server from freezing your system.

You pay for it like this: if your work runs longer than the lease, the lock ends while you are still working, and nothing tells you. The fencing token step below shows that happening.

The Problem

The same race turns up in three places. Here is each one, running and failing.

Duplicate jobs

Three containers share one schedule, and all three wake at the same moment.

async function sendDailyReport(worker) {
  log(worker, 'cron fired -> building daily report');
  await sleep(120);
  emailsSent++;
  log(worker, 'report email SENT');
}

await Promise.all(['worker-1', 'worker-2', 'worker-3'].map(sendDailyReport));

[07:22:46.687] worker-1  report email SENT
[07:22:46.687] worker-2  report email SENT
[07:22:46.687] worker-3  report email SENT

EMAILS SENT: 3   (should be 1)

The token refresh stampede

Your app holds an access token from an outside service. The token expires while 50 requests are in flight. All 50 notice, and all 50 ask for a new one. The provider issues a fresh token each time and kills the previous one, so only the last token still works.

async function handleRequest() {
  const token = await provider.refresh();   // every request refreshes
  await sleep(100);                         // the request now uses that token
  try { await provider.callApi(token); ok++; }
  catch { failed++; }
}

REFRESH CALLS TO PROVIDER: 50   (should be 1)
REQUESTS OK:    1
REQUESTS FAILED (401): 49

Concurrent updates

Two admins open the same product. Stock is 10, one sells 3 and the other sells 2, so the answer is 5.

const { rows } = await client.query('select stock from products where id = 1');
const stock = rows[0].stock;
await sleep(thinkTime);                     // the admin fills in the form
await client.query('update products set stock = $1 where id = 1', [stock - qty]);

[07:22:47.944] admin-B  read stock = 10
[07:22:48.132] admin-A  sold 3 -> wrote stock = 7
[07:22:48.348] admin-B  sold 2 -> wrote stock = 8

FINAL STOCK: 8   (should be 5)

Admin A's sale disappeared. Notice that the transaction did not help you here. Both transactions committed, both were valid, and Postgres did exactly what you told it to do.

All three failures are the same one: two servers acted on the same stale value.

The Solution

Work in two layers. The lock stops the waste. The database stops the damage.

Step 1: build a correct Redis lock

Three common versions are broken, and all three are easy to write by accident.

No expiry. await redis.setnx('lock:report', 1) leaves a key with a TTL of -1, so a crashed worker blocks the job forever.

Expiry as a second command. Calling setnx and then expire is two steps, and a crash between them leaves you with the first problem again.

One command, blind release. This one is the dangerous one:

await redis.set('lock:report', 1, 'NX', 'PX', 30000);
// ... later ...
await redis.del('lock:report');

If your work overruns the lease, the lock expires, worker 2 takes it, and then worker 1 finishes and calls DEL, deleting worker 2's lock instead of its own:

worker-1 holds the lock, value = worker-1
worker-2 acquires it, value = worker-2 (set returned OK)
worker-1 calls DEL. lock now = null  <- worker-2 lost its lock
worker-3 can now acquire it too: OK -> worker-2 and worker-3 both think they hold it

Give each holder a unique token, and release with a Lua script so the check and the delete cannot be split apart:

import Redis from 'ioredis';
import { randomUUID } from 'crypto';

const redis = new Redis({ host: '127.0.0.1', port: 6399 });

const RELEASE = `
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end`;

export async function acquire(key, ttlMs) {
  const token = randomUUID();
  const ok = await redis.set(key, token, 'NX', 'PX', ttlMs);
  return ok === 'OK' ? token : null;
}

export function release(key, token) {
  return redis.eval(RELEASE, 1, key, token);
}

Step 2: wrap the job

async function sendDailyReport(worker) {
  const key = 'lock:daily-report:2026-09-26';
  const token = await acquire(key, 30000);
  if (!token) return log(worker, 'lock is taken -> skipping');

  try {
    await sleep(120);
    log(worker, 'report email SENT');
  } finally {
    await release(key, token);
  }
}

[07:22:46.800] worker-3  lock is taken -> skipping
[07:22:46.921] worker-1  report email SENT
[07:22:46.922] worker-1  lock released

EMAILS SENT: 1   (should be 1)

Put the date in the key, so tomorrow's run is not blocked by today's lock. Let the losers skip rather than wait, because a scheduled job that waits just runs twice, late. Release in a finally, so a thrown error still returns the lock.

Step 3: collapse the token stampede

A lock on its own is the wrong fix here. If 49 requests queue for the lock and then each refresh in turn, you still make 50 calls, only slower. You need three parts: one refresh per server, one refresh across all servers, and a shared cache so the losers read the winner's token instead of fetching their own.

async getToken() {
  const cached = await redis.get('oauth:token');
  if (cached) return cached;

  if (this.inflight) return this.inflight;                     // one per server
  this.inflight = this.refresh().finally(() => { this.inflight = null; });
  return this.inflight;
}

async refresh() {
  const lock = await acquire('lock:oauth:refresh', 10000);     // one across all servers
  if (!lock) {
    for (let i = 0; i < 100; i++) {                            // wait, then read
      await sleep(20);
      const t = await redis.get('oauth:token');
      if (t) return t;
    }
    throw new Error('timed out waiting for the refresh');
  }
  try {
    const fresh = await provider.refresh();
    await redis.set('oauth:token', fresh, 'PX', 60000);        // share it
    return fresh;
  } finally {
    await release('lock:oauth:refresh', lock);
  }
}

[07:22:47.379] server-C  another server refreshed -> reusing its token

REFRESH CALLS TO PROVIDER: 1   (should be 1)
REQUESTS OK:    50
REQUESTS FAILED (401): 0

Refresh at about 80 percent of the token's life rather than waiting for a 401, so the work happens while the old token still works.

Step 4: let the database stop lost updates

You do not need a lock service for this one. If the new value depends only on the old one, never read it first:

update products set stock = stock - 3 where id = 1 and stock >= 3;

When you must read first, read a version number with the row and write only if it has not moved:

const { rows } = await pool.query('select stock, version from products where id = 1');
const { stock, version } = rows[0];

const res = await pool.query(
  `update products
      set stock = $1, version = version + 1
    where id = 1 and version = $2`,
  [stock - qty, version]
);

if (res.rowCount === 0) {
  // someone changed the row first -> read again and retry
}

[07:22:48.964] admin-B  version changed under me -> retry 1
[07:22:49.007] admin-B  read stock = 7 (version 2)
[07:22:49.089] admin-B  sold 2 -> wrote stock = 5

FINAL STOCK: 5   (should be 5)

When conflicts are frequent enough that retrying costs more than waiting, lock the row instead with SELECT FOR UPDATE.

Step 5: add a fencing token

Here is a lock doing exactly what it promises and still failing. Worker 1 takes a one second lease, then its process stalls for 2.5 seconds. Garbage collection, a slow disk or a paused container will all do that to you.

[07:22:51.356] worker-2  got the lock, starting work
[07:22:51.657] worker-2  finished work after 300ms and is about to write
[07:22:52.656] worker-1  finished work after 2500ms and is about to write

WORKERS RUNNING AT THE SAME TIME: 2   (the lock said 1)

No TTL setting fixes this. A longer lease only means a crashed worker blocks you for longer.

Hand out a number with every lock grant, and make the storage refuse anything lower than what it has already accepted:

async function acquireWithFence(key, ttl) {
  const token = await acquire(key, ttl);
  if (!token) return null;
  const fence = await redis.incr('fence:report');   // 1, 2, 3, never repeats
  return { token, fence };
}

async function write(who, content, fence) {
  const res = await pool.query(
    'update reports set content = $1, fence = $2 where id = 1 and fence < $2',
    [content, fence]
  );
  return res.rowCount === 1;                        // false means it was stale
}

[07:22:54.379] worker-2  got the lock with fence 2
[07:22:54.721] worker-2  write ACCEPTED (fence 2)
[07:22:55.665] worker-1  write REJECTED (fence 1 is stale)

FINAL ROW: content="written by worker-2"  fence=2

Worker 1 still woke up and still tried to write. The row refused it.

Trade-offs

A distributed lock is the wrong tool more often than you would expect, and it costs you things that are easy to miss.

It is not a correctness guarantee. Everything above still leaves the window in step 5. If a duplicate run moves money, the lock is not what protects you.

One key serialises everything. A single lock key turns your fleet into a queue of one. Put the entity id in the key, so unrelated work still runs in parallel.

Redis failover can hand out the same lock twice. Redis replication is asynchronous, so a lock the primary acknowledged can vanish when a replica is promoted.

Fencing only works if the target can check. A Postgres row can compare a number. A plain file upload cannot, and will accept whatever arrives last.

Redlock does not close the gap. Martin Kleppmann argued in How to do distributed locking that a process pause outlives any lease no matter how many nodes agree, and Salvatore Sanfilippo replied in Is Redlock safe? that the critique measures it against a promise it never made. Both are right about different things. If a duplicate run is only wasteful, one Redis is enough. If it is damaging, no Redis setup is the answer.

Before you reach for a lock at all, check whether one of these already covers you:

Conclusion

On Monday, find the jobs that run on a schedule and count how many copies of your app are running. If the answer is more than one and nothing is coordinating them, add the lock from step 1, put the entity id and the date in the key, and release it in a finally. Then ask the harder question about each job: if it ran twice, would you lose money or data? Where the answer is yes, a lock is not enough on its own, and the version column from step 4 or the fencing token from step 5 is what actually saves you.

Need a hand auditing this in your own stack? Contact us.