Designing Reliable Webhook Processing Systems

A few years back I spent most of a Saturday tracing down a bug where customers were getting charged twice for the same order. Same order ID, two successful charges, two confirmation emails, one very unhappy customer on the phone. We assumed it was the payment provider's fault at first. That's always everyone's first guess. It wasn't. Our server had received the same webhook twice, seconds apart, and processed both of them as if they were separate events.

Nothing about that bug showed up in code review. It only showed up in production, under load, when our endpoint was a little slower than usual and the provider's retry logic kicked in before our first request had finished. That's the thing about webhooks: they look almost insultingly simple in the docs. "We'll POST a JSON payload to your URL when something happens." Five minutes to wire up, and it works great in your local testing. Then it goes to production and starts misbehaving in ways that never showed up in the demo.

This post is what I've learned building and fixing webhook consumers across a few different systems: payments, e-commerce platform sync, notification pipelines. Designing reliable webhook processing isn't a separate discipline from normal backend work, it's just backend work applied honestly to an endpoint that gets hit by forces outside your control. Not a checklist copied from a vendor's docs page, but the actual failure modes I've hit and the patterns that held up.

Quick answer

If you only fix one thing: acknowledge the webhook fast (verify the signature, write the raw event, return a 2xx) and make every event idempotent before you do anything else with it. That single combination, a fast ack plus a database-enforced idempotency check, prevents the two failure modes that actually cause incidents in practice: duplicate processing from provider retries, and a slow request timing out and triggering a retry storm on top of itself. Everything else here (ordering, partitioning, dead-letter queues, reconciliation) is real, but it's defense in depth sitting on top of that one foundation.

What "reliable" actually means here

A webhook is just an HTTP callback: a provider sends a POST request with a JSON payload to a URL you control whenever something happens on their end. That part takes five minutes to wire up. Reliable webhook processing is everything around it: making sure that callback still does the right thing when it arrives twice, arrives late, arrives out of order, or doesn't arrive at all. It isn't a separate technology you install. It's a set of design decisions applied to an endpoint that would otherwise be a single route handler.

If you're integrating with a payment provider, an e-commerce platform, a CI system, or basically anything that pushes events at you instead of waiting for you to ask, this is the layer that decides whether an outage on their end, or a slow moment on yours, turns into a data problem for you six months later.

The mental model that gets you in trouble

Most people build their first webhook handler assuming three things that are all false:

• "It'll only arrive once." It won't. Almost every provider worth using is at-least-once delivery, not exactly-once. Stripe says this outright in their docs, and so does GitHub. If your endpoint doesn't respond fast enough, times out, or returns anything other than a clean 2xx, the provider assumes it failed and sends it again. Sometimes more than once.
• "They'll arrive in order." Also not guaranteed. Two events for the same resource can be sent in close succession, hit different edge nodes, retry independently, and land on your server in the wrong order.
• "If my endpoint returns 200, I'm done." Returning 200 only tells the provider you received it. It doesn't mean you finished processing it safely. Those are two very different guarantees, and conflating them is where most of the fragile implementations come from.

Once you actually internalize that webhooks are unreliable-by-design (not a bug, a deliberate trade-off the provider made), the rest of this is just a set of defenses against that reality.

What you need before wiring this up

Nothing exotic. Everything in this guide assumes:

• A provider that sends webhooks with some form of request signing. HMAC is the common case, and Stripe, GitHub, Shopify, and most serious providers all do it.
• A durable place to write the raw event before you process it. A Postgres table is enough for this; it doesn't need to be fancy.
• Some kind of background job runner or queue (BullMQ, SQS, Sidekiq, whatever your stack already has), so processing doesn't happen inline in the HTTP request.
• Basic familiarity with HMAC/crypto verification and whatever query layer or ORM you're already using.

You don't need a message broker cluster or a full event-sourcing architecture to do this properly. Most of it comes down to one table and one queue, used deliberately instead of skipped.

Verify before you trust anything

Before your code does anything with a payload, it needs to know the request actually came from the provider and not from someone who found your endpoint URL. Almost every serious webhook provider signs the payload with a shared secret using HMAC, and almost every homegrown implementation I've seen skips or half-implements this check.

const crypto = require('crypto');

function isValidSignature(rawBody, signatureHeader, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody, 'utf8')
    .digest('hex');

  const provided = Buffer.from(signatureHeader, 'utf8');
  const computed = Buffer.from(expected, 'utf8');

  if (provided.length !== computed.length) return false;

  // timing-safe comparison: a regular === leaks timing info
  // that can be used to guess the signature byte by byte
  return crypto.timingSafeEqual(provided, computed);
}

Two details people skip, and both matter:

1. Compare against the raw request body, not the parsed JSON. If you let Express (or whatever framework) parse the body first and then re-serialize it to check the signature, whitespace and key ordering differences will break verification on payloads that are perfectly legitimate. Capture the raw bytes before any JSON parsing happens.
2. Reject old timestamps. Most providers include a timestamp in the signed payload or headers specifically so you can reject replayed requests. Someone who captured a valid signed request a week ago shouldn't be able to replay it against you today. A five-minute tolerance window is a reasonable default.

Acknowledge fast, do the real work somewhere else

This is the single biggest structural mistake I see: handling the webhook synchronously, in the same request, doing database writes, calling three other internal services, sending an email, all before returning a response. It works fine in a demo. It falls over the first time one of those downstream calls gets slow.

Here's why it matters: providers set a timeout on your endpoint, usually somewhere between 5 and 30 seconds. If you don't respond in time, they treat the delivery as failed and retry it, which means your slow request and the retry are now both running, both trying to do the same work, and now you have a duplicate-processing race on top of a latency problem.

The fix is boring and it works: verify the signature, write the raw event somewhere durable, return 2xx, and let a separate worker do the actual processing.

Provider  →  HTTPS endpoint
                 │
         verify signature
                 │
        insert raw event row
       (idempotent, see below)
                 │
           return 200 OK  ← provider is happy, connection closes
                 │
            enqueue job
                 │
      ┌──────────┴──────────┐
      │   background worker  │
      │  does the real work  │
      └──────────┴──────────┘
                 │
        on repeated failure
                 ↓
          dead-letter queue

app.post('/webhooks/orders', express.raw({ type: 'application/json' }), async (req, res) => {
  const signature = req.get('X-Signature');

  if (!isValidSignature(req.body, signature, process.env.WEBHOOK_SECRET)) {
    return res.status(401).send('invalid signature');
  }

  const event = JSON.parse(req.body);

  // durable write happens before we ack; if this fails, the
  // provider retries and we haven't lost anything
  const wasNew = await storeRawEvent(event);

  if (wasNew) {
    await queue.add('process-order-event', { eventId: event.id });
  }

  // ack even if it was a duplicate; that's expected, not an error
  res.status(200).send('ok');
});

Notice the response is 200 in both branches. A duplicate delivery isn't a failure case from the provider's point of view. It's normal behavior, so there's no reason to make it retry again.

Every event needs to be safe to process twice

Once you accept that duplicates are inevitable, the actual work is making sure processing an event twice has the same effect as processing it once. That's what idempotency means here: not "we won't get duplicates," but "duplicates are harmless."

The cleanest way I've found is a small table that records every event ID you've seen, with a unique constraint doing the actual enforcement instead of an application-level check-then-insert (which has its own race condition if two workers pick up the same retried delivery at once).

CREATE TABLE processed_webhook_events (
  event_id   TEXT PRIMARY KEY,
  source     TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  status     TEXT NOT NULL DEFAULT 'pending'
);

async function storeRawEvent(event) {
  const result = await db.query(
    `INSERT INTO processed_webhook_events (event_id, source)
     VALUES ($1, $2)
     ON CONFLICT (event_id) DO NOTHING
     RETURNING event_id`,
    [event.id, event.source]
  );

  return result.rowCount > 0; // true only if this row is genuinely new
}

ON CONFLICT DO NOTHING plus checking rowCount is doing the whole job here. The database, not your application code, is the source of truth on whether this is the first time you've seen this event, and it's atomic, so two concurrent workers processing the same retried delivery can't both think they're first.

If the provider doesn't give you a stable event ID (some smaller or homegrown systems genuinely don't), hash the payload plus a coarse timestamp bucket instead. It's not as clean, but it's better than nothing.

Don't trust delivery order, trust the resource's own state

This one bit me on an order-sync system. We had order.created and order.updated events, and our handler assumed created always arrived first. Reasonable assumption, wrong in practice. Under retries, an updated event for a brand-new order occasionally beat the created event to our server by a few hundred milliseconds.

The fix wasn't to enforce ordering. You can't, not without the provider guaranteeing it, and most don't. The fix was to stop caring about arrival order and start comparing against the resource's own version:

async function applyOrderEvent(event) {
  const existing = await db.getOrder(event.orderId);

  if (existing && existing.updatedAt >= event.payload.updatedAt) {
    // we already have a version of this order that's the same age or newer
    // this event is stale, safe to skip, not an error
    return;
  }

  await db.upsertOrder(event.payload);
}

If the provider includes a version number or timestamp on the resource itself, use that as your conflict resolution, not the order things showed up in your inbox.

Scaling out without losing per-resource ordering

Once volume grows past what one worker can chew through, the obvious move is to add more workers. The less obvious problem: if you just throw every event at a shared pool of workers pulling from one queue, you reintroduce the ordering problem you just solved, except now it's happening because of your own infrastructure instead of the provider's retries. Worker A can pick up order_42.updated at the same moment Worker B picks up an earlier order_42.created retry, and there's no guarantee which one finishes writing to the database first.

The pattern that's worked for me is partitioning by resource ID rather than scaling a single shared queue. Every event for a given order, customer, or subscription gets routed to the same partition, and only one worker processes a given partition at a time. Across partitions, everything runs in parallel; within a partition, order is preserved because there's no concurrency to race against.

// route by a stable hash of the resource ID, not round-robin:
// this guarantees every event for the same order lands on the same partition
const PARTITION_COUNT = 8;

function partitionFor(orderId) {
  const hash = crypto.createHash('sha1').update(orderId).digest();
  return hash.readUInt32BE(0) % PARTITION_COUNT;
}

await queue.add(`process-order-event-p${partitionFor(event.orderId)}`, {
  eventId: event.id,
});

If you're using something like BullMQ, this often just means one queue per partition, each with a concurrency of 1. If you're using SQS with FIFO queues, the MessageGroupId does exactly this for you natively. It's worth reaching for a FIFO queue specifically because this problem is what it was built to solve, rather than reinventing partitioning yourself on top of a standard queue.

The number of partitions is a throughput knob, not a correctness knob: more partitions means more parallelism, but the ordering guarantee only holds within a single order's events, never across different orders, which is exactly what you want.

Isolate failures so one bad payload doesn't take down the queue

At some point a payload will show up that your code doesn't handle well: a null where you expected a string, a currency you haven't onboarded yet, a schema field the provider quietly added last month. If your worker throws on that message and the queue retries the same message indefinitely, it can back up everything behind it.

Two things fix this:

• Retry with backoff, but cap it. A handful of retries with increasing delay handles transient failures: a database blip, a downstream service restarting. It should not run forever.
• After the cap, move it to a dead-letter queue and alert, don't drop it silently. You want a human to look at it, not lose it.

worker.process('process-order-event', async (job) => {
  try {
    await handleOrderEvent(job.data);
  } catch (err) {
    if (job.attemptsMade >= 5) {
      await moveToDeadLetter(job, err);
      logger.error('event moved to dead-letter after max retries', {
        eventId: job.data.eventId,
        error: err.message,
      });
      return; // don't rethrow, we've handled the failure terminally
    }
    throw err; // let the queue's backoff retry it
  }
});

The dead-letter queue isn't a graveyard. It's a worklist. Someone should be looking at what lands there, at least until you understand why it's happening.

Keep the raw payload, always

Before I started doing this, we had an incident where a provider silently changed a field name in their payload, our parser quietly ignored the field it no longer recognized, and we had no record of what the original request actually contained. We couldn't tell if it was their bug or ours, because we'd already thrown away the evidence.

Now every raw payload gets written to storage before anything else touches it, independent of whether processing later succeeds. It's cheap (a JSONB column or a blob in object storage) and it turns "we have no idea what happened" into "let me pull up the exact request from 2:14am and look at it."

That same raw store is also what makes replay possible. Being able to say "take event evt_88213, run it through the processor again" and have that just work, because your processing is already idempotent, turns a lot of 2am incidents into a five-minute fix instead of a manual data-repair job.

Security doesn't stop at the signature check

Signature verification handles authenticity: proving the payload wasn't tampered with and genuinely came from someone holding the shared secret. It doesn't handle everything else that goes wrong at an internet-facing endpoint, and it's easy to stop there and assume you're covered.

A few things worth adding on top:

• Rate limit the endpoint independently of your normal API rate limits. A webhook URL is a public POST endpoint by definition. If it leaks (pasted into a support ticket, logged somewhere it shouldn't be) it becomes a target, signature or not. A sane rate limit means a flood of junk requests gets throttled instead of falling straight through to your signature check on every single one.
• IP allowlisting where the provider publishes a stable IP range. Not all of them do, and it shouldn't be your only defense, but as a second layer it costs almost nothing to add and blocks a lot of noise before it reaches application code.
• Rotate the shared secret occasionally, and support two valid secrets during rotation. If you only ever support one active secret, rotating it means a window where either old deliveries fail or you have to time the rotation perfectly on both sides. Accepting the old and new secret for a short overlap period removes that pressure.
• Treat the endpoint URL itself as a secret you don't want indexed or logged in plaintext. I've seen webhook URLs end up in error-tracking tools because someone logged req.originalUrl on an unrelated 500 error. It's not the end of the world if your signature check is solid, but it's one less thing to worry about if it just doesn't leak.

None of this replaces the signature check. It's what you add once you stop assuming the signature check is the whole security story.

Testing this before it's on fire, not after

The naive way to test a webhook handler is to trigger the real event in a sandbox account and see what happens. That works for the happy path and basically nothing else, because you can't easily make a provider send you a duplicate, an out-of-order pair, or a payload with a field you don't recognize on demand.

What's worked better for me:

• Capture real payloads once, replay them forever. The first time a real event comes through in a sandbox or staging environment, save the raw body. That becomes a fixture you can replay against your handler in tests without needing the provider involved at all.
• Write the duplicate case as an actual test, not a mental note. Send the same captured payload through your handler twice in the same test and assert the side effect (a charge, a row insert, an email send) happened exactly once. This is the test that would have caught the double-charge bug that started this whole post.
• Test the out-of-order case explicitly. Feed the updated event through before the created event and confirm the final state is still correct once both have been processed, regardless of order.
• Kill the worker mid-job on purpose. Whatever your queue library is, most of them let you simulate a crash or a forced restart mid-processing in a test. If your idempotency and retry logic are solid, the job just gets picked up again cleanly. If they're not, this is where you find out in a test run instead of during an actual deploy.
• Use a local tunnel for manual testing against a real provider. Tools like ngrok or a similar tunnel let a provider's sandbox hit your local machine directly, which is genuinely useful for the exploratory phase, but don't mistake "it worked when I clicked the button once in ngrok" for "it's handled correctly under retries," because those are testing two different things.

Monitoring: know it's broken before your customers tell you

A webhook consumer can fail silently in a way most other parts of your system can't. If an API call from your frontend fails, a user sees an error immediately. If a webhook fails, the provider might retry a few times and give up, and nothing about that shows up anywhere unless you're explicitly watching for it.

The handful of signals that have actually caught real incidents for me, not just theoretical ones:

• Processing failure rate, as a percentage, not a raw count. A raw count of failures spikes naturally with traffic; a percentage tells you whether something's actually degraded.
• Dead-letter queue size, trending, not just a threshold alert. A DLQ that's slowly climbing over hours is a different problem than one that jumps from zero to two hundred in a minute, and both matter, but they point at different root causes.
• Time between receipt and successful processing. If this creeps up, your worker is falling behind real-time, which usually means either a downstream dependency slowed down or you need more processing capacity.
• Signature verification failure count, tracked separately from processing failures. A sudden spike here is either a misconfigured secret after a deploy or someone probing your endpoint. Worth knowing which, and worth knowing immediately rather than during the next scheduled review.

None of this needs to be elaborate. A dashboard with these four numbers and an alert on the DLQ size and the failure percentage covers most of what actually goes wrong in practice.

Webhooks are a notification, not a queue you can fully trust

Even with all of the above, there's a failure mode you can't defend against purely from the receiving side: the event never gets sent at all, because your endpoint was down for longer than the provider's retry window, or because of a bug on their end. This happens rarely, but "rarely" across enough volume is still "regularly."

The systems I've seen hold up best treat webhooks as a fast path, not the only path. A periodic reconciliation job (pull the last N hours of orders/transactions/whatever directly from the provider's API and diff against what you have) catches the gaps the fast path misses. It doesn't need to run often. It just needs to exist.

Build vs. buy: when this is worth outsourcing

Everything above is a well-understood set of problems, which also means it's a well-understood set of problems other people have already built solutions for. Services like Svix or Hookdeck exist specifically to sit between a provider and your endpoint, handling signature verification, retries, ordering, and replay tooling as a managed layer, and there's a real argument for reaching for one instead of building this yourself.

Where I'd actually make that call:

• If you're a provider sending webhooks out to other people's endpoints, a managed layer handles the retry-and-backoff logic, delivery logs, and per-customer secrets for you. That's a genuinely different (and arguably harder) problem than consuming webhooks, and it's the case these tools are usually built around first.
• If you're consuming from a handful of providers and the volume is modest, the patterns in this post are a day or two of work and give you full control over the failure modes, which matters if webhook processing touches something sensitive like payments.
• If you're consuming from a large number of different providers with inconsistent signing schemes and retry behavior, a managed layer that normalizes all of that into one consistent interface starts looking a lot more appealing than writing and maintaining N slightly different verification implementations.

There's no universally right answer here, it's the same build-vs-buy tradeoff as anything else in infrastructure. I'd just push back on reaching for a managed service as a way to skip understanding these failure modes, because you still need to understand idempotency and ordering to use one correctly; it removes the plumbing, not the thinking.

Common problems and errors

These are the ones that come up often enough to be worth naming directly, separate from the personal war stories later in this post.

"We're getting charged or created twice for the same event"

Cause: No idempotency check, or one that isn't atomic: a check-then-insert instead of a unique constraint. Two retried deliveries race each other and both pass the check before either has actually inserted its row.

Fix: Use a unique constraint plus ON CONFLICT DO NOTHING, as shown earlier, and let the database resolve the race instead of application code trying to coordinate it.

"The endpoint returns 200 but the event never actually gets processed"

Cause: The raw-event write succeeds but the enqueue call afterward fails silently, or the worker process simply isn't running in that environment.

Fix: Where possible, make the raw-event write and the enqueue happen in the same transaction, and alert on a growing gap between rows in your events table and completed job counts. That gap is your canary.

"Signature verification fails randomly, not consistently"

Cause: Almost always a body-parsing problem. Something upstream (a proxy, a framework's default JSON parser) has already touched the raw bytes before your signature check runs, so what you're hashing doesn't match what the provider actually signed.

Fix: Confirm the route uses the raw, unparsed body specifically for signature verification, never a re-serialized version of the parsed JSON.

"Events are missing and nobody noticed for weeks"

Cause: No reconciliation job, and no alerting on the webhook endpoint's uptime specifically. A deploy, an expired certificate, or a firewall change silently broke inbound delivery and nothing told anyone.

Fix: A periodic reconciliation pull against the provider's API, even a daily one, plus uptime monitoring on the webhook URL itself rather than just your general app health check.

"The queue backs up and everything gets slow"

Cause: Usually one of two things: a poison message retrying forever with no cap, or every event landing in a single queue with no partitioning, so nothing runs in parallel.

Fix: Cap retries and dead-letter after the cap. Partition by resource ID if a single queue turns out to be the actual bottleneck.

If you only remember a handful of rules

• Verify the signature against the raw body, with a timing-safe comparison, before touching anything else.
• Acknowledge fast (write the raw event, enqueue it, return 2xx) and do the real work in a separate worker.
• Make every event idempotent with a database-level unique constraint, not an in-memory set or a check-then-insert.
• Resolve conflicts using the resource's own version or timestamp, never the order things happened to arrive in.
• Cap retries and route failures to a dead-letter queue instead of retrying forever.
• Keep the raw payload regardless of whether processing succeeds. It's your only evidence when something looks wrong later.
• Run a periodic reconciliation job as a backstop, because "the provider will eventually deliver it" is a strong likelihood, not a guarantee.

Webhooks vs. polling vs. an event stream

Before assuming webhooks are the right shape for a given integration, it's worth being honest about what they're actually good at compared to the alternatives.

Webhooks win when a third party controls the data and pushing is cheaper than you polling them constantly. Polling is still the right call when the provider doesn't offer webhooks at all, or when you need a periodic reconciliation pass anyway, and as covered above, you'll want one of those even if you do use webhooks. A full event stream is a different tool for a different problem: high-volume, ordered, internal event distribution, not receiving notifications from an external provider you don't control.

When should you actually reach for webhooks?

Webhooks make sense when:

• The provider supports them and signs their payloads.
• You need near-real-time updates, and polling would mean either too much latency or wasted API calls.
• You're willing to build, or adopt, the fast-ack, idempotency, and dead-letter handling this guide covers. A webhook consumer that skips all of it is a liability, not a shortcut.

Reach for polling or a direct API pull instead when:

• The provider doesn't offer webhooks, or their implementation is unreliable or undocumented.
• The integration is low-frequency enough that a scheduled pull is simpler and just as timely in practice.
• You need a full, authoritative snapshot on a regular basis anyway, in which case a webhook is really just an optimization on top of a pull you're doing either way.

Trade-offs

Every architecture choice trades one set of headaches for another. Adding an async queue, a deduplication store, and resource partitioning prevents double charges and timeout cascades, but it introduces real complexity:

• Infrastructure overhead. You now maintain and monitor a raw events table, a message broker (such as SQS or BullMQ), separate worker processes, and dead-letter queue alerts. For simple internal notifications where an occasional duplicate or dropped message carries no consequence, this infrastructure is unnecessary work.
• Eventual consistency latency. Returning 200 OK in under 50ms means the caller disconnects before processing finishes. If your customer completes checkout and immediately redirects to their dashboard, their order might still be in the queue for another 200ms, creating a brief window where user-facing state appears stale.
• Unbounded storage growth. Storing every raw payload indefinitely will bloat your database. You need automated retention policies, such as archiving or deleting payloads older than 30 to 90 days.
• Partition bottlenecks under burst traffic. Hashing by resource ID preserves ordering per entity, but uneven traffic creates hot partitions. A flood of events for a single active merchant or account will queue behind one worker while other partition workers sit idle.

When this approach is the wrong one

Do not build this architecture if your webhook contract requires immediate synchronous round-trip feedback to the caller (such as an auth challenge where the provider expects an allow or deny response in the HTTP payload). In those cases, an async queue breaks the protocol contract. Similarly, if your volume is trivial and events trigger read-only or low-impact side effects (like posting an informational link into an internal team chat), a single inline route handler without an asynchronous queue is simpler, cheaper, and sufficient.

Questions I get asked about this a lot

Do I really need a queue, or can I just process the webhook inline?

You can, right up until one downstream call gets slow. Inline processing is fine for low-stakes, low-volume webhooks. For anything touching money, inventory, or user-facing state, the fast-ack-plus-queue pattern is worth the extra hour of setup.

What if the provider doesn't sign their webhooks?

Some smaller or older systems genuinely don't. Treat the payload as untrusted input, the same way you'd treat any public POST endpoint: rate limit it heavily, validate the shape strictly, and consider an IP allowlist if the provider publishes one. It's a weaker guarantee than signature verification, so keep the blast radius of whatever that endpoint can trigger as small as possible.

How long should I keep raw webhook payloads?

Long enough to debug and replay a real incident, which in practice means weeks, not days. Storage is cheap relative to the cost of not having the evidence when you actually need it. A lot of teams land on 30–90 days, with older records archived to cold storage instead of deleted outright.

Is idempotency the same thing as deduplication?

Close, but not identical. Deduplication means "don't process this event twice." Idempotency means "processing it twice is harmless even if dedup somehow fails." You want both, but idempotency is the deeper guarantee. It's what saves you when your dedup check has a bug or a gap you haven't found yet.

Do I need any of this if I'm only handling a handful of webhooks a day?

The failure modes don't care about your volume: a duplicate delivery is just as possible at ten events a day as at ten thousand. What scales with volume is how often you'll actually hit the edge case in practice. Low volume buys you time to notice problems before they compound, not immunity from them.

A few things I got wrong before I got this right

• I trusted a provider's docs that said events "would generally arrive in order." Generally isn't always, and the exception always finds you in production.
• I thought our endpoint was "fast enough" to process inline safely. It was, until one downstream call had a bad day, and then every webhook in flight backed up behind it.
• I didn't store raw payloads because it felt like unnecessary overhead. The first time I needed one and it wasn't there, that stopped being a debate.
• I used an in-app Set to track processed event IDs to avoid duplicates. It worked in one process. It fell apart the moment we scaled to more than one worker.
• I let the queue retry a bad payload with no upper bound, on the assumption that "it'll sort itself out." It didn't. It sat there retrying every few seconds for two days, quietly eating worker capacity, until someone finally noticed the DLQ we didn't have yet would have caught it immediately.
• I logged the parsed event instead of the raw request body, on the theory that the parsed version had everything useful in it. It didn't have the one field I needed six weeks later, because our parser had silently dropped it as unrecognized.

None of these were exotic mistakes. They're the default way most people build a webhook handler the first time, because the naive version genuinely works right up until it doesn't.

Further reading

These are the primary sources I keep coming back to, not summaries of them. The actual docs:

• Stripe: Webhooks overview
• Stripe: Verifying webhook signatures
• GitHub: Validating webhook deliveries
• AWS: Amazon SQS dead-letter queues
• PostgreSQL: Constraints (unique constraints, used for idempotency)
• IETF RFC 2104 (HMAC: Keyed-Hashing for Message Authentication)

Conclusion

None of this is exotic engineering. Signature checks, a fast ack, an idempotency table, a dead-letter queue, and keeping the raw payload: that's the whole list. What makes webhook systems fragile in practice isn't that these ideas are hard, it's that the naive version works fine right up until the moment it's handling real traffic, and by then it's a production incident instead of a design review comment.

If you're building a webhook consumer right now and it doesn't do at least the fast-ack-plus-idempotency part yet, that's the one to fix first. It's the one that turns "provider retried and now we double-charged someone" from a recurring incident into a non-event.

Contact us