Universal Third-Party Integration Framework: Adapters, OAuth, Webhooks, Retries, Rate Limits, and Sync

You start with one integration. Stripe, two days, done. Then sales asks for HubSpot, support asks for Zendesk, and a customer asks for Shopify. Each one is written by a different person in a different week, and each one invents its own token refresh, its own retry loop and its own webhook handler.

Six months later you are paged at 2 a.m. The Shopify sync is stuck behind a rate limit, the GitHub job is getting 403s it reads as auth failures, and a refund went out twice because a retry fired after a dropped connection. Nothing is broken in one place. It is broken in nine places that each look slightly different.

This post builds the layer that stops that: one call path, one set of rules, and a small adapter per provider. Every code block below runs, and every log in this post is real output from running it.

Quick Answer

A universal integration framework is one shared call path that every third-party provider goes through, plus a small adapter per provider that answers only the questions where providers differ. The shared part owns the things that are the same everywhere: store a token and refresh it once, wait for a rate limit token before each attempt, retry with capped backoff and jitter under a retry budget, verify and deduplicate webhooks before doing any work, and keep a sync cursor you can resume from. The adapter owns the things that are never the same: the base URL, how this provider counts its limit, how it signs a webhook, and what its change feed looks like. You do not get a framework that makes providers identical. You get one that stops nine different bugs from being nine different fixes.

What Is an Adapter Boundary?

An adapter is a small object that translates one provider into the shape your code already understands.

Your code says "list the changed contacts". The adapter knows that means GET /crm/changes?cursor=... for one provider and a GraphQL query for another. Your code never learns that difference.

The boundary is the line between the two. Everything above it is written once. Everything below it is written per provider, and kept as small as you can make it.

The Problem

Here is what happens with no boundary. Three providers, called the obvious way: fan out and hope.

const results = await Promise.all(
  Array.from({ length: 8 }, () => fetch(`${BASE}/github/repos`).then(r => r.status))
);

[08:37:45.347] github    statuses: 200 200 200 200 200 403 403 403
[08:37:45.348] github    failed: 3 of 8
[08:37:45.356] stripe    statuses: 200 200 200 429 429 429
[08:37:45.356] stripe    failed: 3 of 6

Two providers, two different status codes for the same problem. GitHub answers a secondary rate limit with 403 or 429. Stripe answers with 429 and a Stripe-Rate-Limited-Reason header. If you wrote if (status === 429) backOff(), the GitHub 403 falls through to your auth error path and your on-call gets told the token expired.

The limits themselves are not even the same unit:

Sources: GitHub rate limits, Stripe rate limits, Shopify GraphQL Admin rate limits.

Now the expensive one. A write times out, so you retry it:

[08:37:45.362] refund    attempt 1 -> connection dropped, retrying
[08:37:45.364] refund    attempt 2 -> re_0002
[08:37:45.365] refund    REFUNDS ACTUALLY CREATED: 2   (should be 1)

The first refund went through. You just never heard back. The retry made a second one.

The Solution

Six steps. Each one moves a decision out of the integration and into the shared path.

Step 1: One adapter, five questions

Every adapter answers the same five things, and nothing else:

export const github = {
  name: 'github',
  limiter: new Bucket({ capacity: 5, refillPerSecond: 5 }),
  call: () => fetch(`${BASE}/github/repos`),
  cost: () => 1,
  // GitHub tells you where you stand on every response. Listen to it.
  readLimits(res) {
    const remaining = Number(res.headers.get('x-ratelimit-remaining'));
    const reset = Number(res.headers.get('x-ratelimit-reset'));
    if (!Number.isFinite(remaining) || !Number.isFinite(reset)) return null;
    const secondsLeft = Math.max(1, reset - Math.floor(Date.now() / 1000));
    return { remaining, refillPerSecond: remaining / secondsLeft };
  },
};

A Stripe adapter is the same shape, but readLimits returns null, because Stripe sends no remaining-quota header. The difference is four lines in one file, not an if statement scattered through your app.

Keep business logic out. The moment an adapter knows what an invoice means to you, you have two places to change when invoices change.

Step 2: One token, refreshed once

A token expires while 20 requests are in flight. All 20 notice. All 20 refresh. The provider rotates the token on every refresh, so 19 of them are already dead by the time they get used:

[08:40:18.166] naive     refresh calls: 20   distinct tokens: 20

Collapse them. One refresh per process, and refresh before the token dies rather than after:

async function getToken(id) {
  const hit = cache.get(id);
  if (hit && hit.expiresAt - Date.now() > 300_000) return hit.accessToken;  // refresh 5 min early
  if (inflight.has(id)) return inflight.get(id);                            // one per process

  const p = (async () => {
    const t = await refreshFromProvider(id);        // one per fleet: take a lock here
    cache.set(id, { accessToken: t.accessToken, expiresAt: Date.now() + t.expiresInMs });
    return t.accessToken;
  })().finally(() => inflight.delete(id));

  inflight.set(id, p);
  return p;
}

[08:40:18.221] single    refresh calls: 1   distinct tokens: 1

That inflight map covers one process. Run three containers and you are back to three refreshes, so the refresh itself needs a lock shared across the fleet and the new token written somewhere all of them read.

Two rules from RFC 9700, the OAuth 2.0 security best current practice, that are easy to get wrong and expensive to fix later. Public clients MUST use PKCE. Redirect URIs MUST be compared by exact string match, not by prefix.

Step 3: A limiter per provider, tuned by the provider

Give each adapter its own token bucket, set a little below the published limit, because your clock and theirs do not agree.

export class Bucket {
  tryTake(cost = 1) { ... }           // non-blocking: used as a retry budget
  async take(cost = 1) { ... }        // blocking: used as a rate limiter
  retune({ capacity, refillPerSecond }) { ... }   // the provider told us the real numbers
}

One shape, two jobs. Blocking, it is a rate limiter. Non-blocking, it is a retry budget. You use both in the next step.

Then let the response correct your guess. GitHub sends x-ratelimit-remaining and x-ratelimit-reset on every call, so you can work out how fast you may spend what is left. Shopify sends extensions.cost.throttleStatus with currentlyAvailable and restoreRate in the body. Stripe sends nothing, so your guess stands until a 429 says otherwise.

Step 4: Retry once, with jitter and a budget

Backoff without jitter is not backoff. Every client that failed in the same second wakes up in the same millisecond:

[08:38:21.709] backoff   no jitter, 5 clients, attempt 3 -> 1600ms 1600ms 1600ms 1600ms 1600ms
[08:38:21.710] backoff   full jitter, 5 clients, attempt 3 -> 1514ms 485ms 442ms 315ms 615ms

export function backoffMs(attempt, { baseMs = 200, capMs = 20_000 } = {}) {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);   // full jitter
}

Three rules around it.

Honour `Retry-After` over your own maths. When a provider tells you how long to wait, it is not guessing.

Retry in one layer only. The AWS Builders' Library measures what happens when you do not: five layers each retrying three times puts 243 times the load on the thing that is already struggling.

Put a budget on it. A bucket that does not refill fast stops a dead provider from being hammered for an hour:

[08:38:23.064] budget    budget exhausted — giving up
[08:38:23.064] budget    calls made before giving up: 4   (attempts allowed: 20)

And the retries must go through the limiter too, or the retry storm becomes the next rate limit breach:

const attempt = async () => {
  await adapter.limiter.take(adapter.cost(opts));   // every attempt waits for a token
  return adapter.call(opts);                        // including the retries
};
const res = await withRetry(attempt, { budget: opts.budget });

Same eight calls, same six refunds, through the limiter:

[08:39:12.143] github    statuses: 200 200 200 200 200 200 200 200
[08:39:12.144] github    429/403 responses: 0   took 2888ms
[08:39:14.145] stripe    429 responses: 0   took 2001ms

Slower, and completely boring. That is the trade you want.

The retry is still not safe on its own. Send an idempotency key so the provider can recognise a repeat:

[08:39:14.954] refund    attempt 1 -> re_0001 (replayed: true)
[08:39:14.954] refund    REFUNDS ACTUALLY CREATED: 1   (should be 1)

The connection still dropped. The framework still retried. The key made the second attempt a replay instead of a second refund.

Step 5: Webhooks — verify, deduplicate, return 200

Order matters here, and most handlers get it wrong by doing the work before the 200.

function verify(headers, rawBody) {
  const id = headers['webhook-id'];
  const ts = Number(headers['webhook-timestamp']);
  const got = headers['webhook-signature'] || '';
  if (!id || !Number.isFinite(ts)) return 'missing headers';
  if (Math.abs(Date.now() - ts * 1000) > TOLERANCE_MS) return 'timestamp outside tolerance';

  const signed = `${id}.${ts}.${rawBody}`;      // id . timestamp . raw body
  const expected = 'v1,' + crypto.createHmac('sha256', SECRET).update(signed).digest('base64');
  // check every signature in the header, so a rotating secret has no downtime
  return got.split(' ').some(c =>
    c.length === expected.length &&
    crypto.timingSafeEqual(Buffer.from(c), Buffer.from(expected))) ? null : 'bad signature';
}

[08:39:36.202] receiver  accepted msg_1 -> queued, returning 200
[08:39:36.210] receiver  duplicate msg_1 -> already processed, 200 anyway
[08:39:36.212] receiver  REJECTED: bad signature
[08:39:36.213] receiver  REJECTED: timestamp outside tolerance
[08:39:36.214] receiver  queued for processing: 1   (4 requests arrived)

Four requests arrived. One job was queued.

The pieces that make that work:

• Sign the raw body. Not the parsed object. Any middleware that reads and re-serialises the JSON changes a space somewhere and breaks every signature. Stripe's docs call this out as the most common verification failure.
• Check the timestamp. Stripe's libraries default to a five-minute tolerance, and the docs say never set it to zero.
• Compare in constant time. === on a signature leaks timing.
• Deduplicate by the event id. The Standard Webhooks spec exists for this: webhook-id, webhook-timestamp, webhook-signature, signed as id.timestamp.payload, with the id used as the idempotency key.
• Return 200, then work. Stripe retries a failed delivery for up to three days. A handler that does the work first turns one slow database write into a retry storm.

Step 6: Sync — the cursor is the truth

A webhook tells you something changed. It does not reliably tell you what, or in what order. Stripe states plainly that events are not delivered in the order they were generated, and that you should not use the created timestamp to order them or to detect duplicates.

So treat the webhook as a nudge, and keep a cursor that you can always resume from:

async function sync(startCursor) {
  let cursor = startCursor;
  for (;;) {
    const res = await fetch(`${BASE}/crm/changes?cursor=${cursor}`);
    if (res.status === 410) {                  // the provider lost our place
      return sync('0');                        // throw the cursor away, start again
    }
    const page = await res.json();
    for (const c of page.changes) apply(c, { checkVersion: true });
    if (page.nextCursor) { cursor = page.nextCursor; continue; }
    return page.newCursor;                     // caught up, save this
  }
}

Two cursors, not one. One moves while you page through a backlog, the other marks "caught up".

Microsoft Graph calls them @odata.nextLink and @odata.deltaLink. Google Drive calls them nextPageToken and newStartPageToken. Same idea, different names. Save the second one, never the first.

Apply with a version check, because the same record will arrive twice and sometimes backwards:

[08:39:54.415] store     wrote c_2 = Grace H (v2)
[08:39:54.415] store     ignored c_2 v1 (already at v2)
[08:39:54.415] store     c_2 with no version check: {"name":"Grace","version":1}   (wrong)

And handle the resync, because it is not an edge case. Graph delta tokens are valid for seven days for directory objects, and a 410 Gone with an empty token means start over:

[08:39:54.417] sync      410 Gone -> cursor is dead, starting a full resync from 0
[08:39:54.420] sync      caught up, saving cursor 4

Your incremental sync is a full sync every so often, whether you planned for it or not. Size your job for that day, not the quiet ones.

Trade-offs

This layer costs you things, and some of them are not obvious until you are already committed.

The abstraction leaks at the limiter. Requests per hour, requests per second and cost points per second do not reduce to one number. Your Bucket grows a cost() hook and a readLimits() hook, and that is where every provider's weirdness ends up living.

A shared limiter across tenants punishes the quiet ones. One customer running a bulk import fills the bucket and everyone else waits. The fix is a limiter key per connection, not per provider, which multiplies your state.

Dedupe storage is a correctness setting disguised as a cache. Standard Webhooks suggests holding ids for five minutes. Stripe retries for three days. If your window is shorter than the retry window, a late retry is processed twice.

Generic retries will corrupt data. The framework cannot tell a safe call from an unsafe one. Without an idempotency key, a retried write is a second write. Mark unsafe calls and let them fail instead.

It is the wrong call for one integration. Writing this for a single Stripe webhook is three days spent on a problem you do not have. Build it at provider three, when the duplication is visible.

Some things belong to the data layer, not here. A framework cannot make a double charge safe. Reach for these first:

Conclusion

On Monday, open the two integrations you trust least and find four things: where the token is refreshed and whether two requests can do it at once, where the retry lives and whether a second retry sits above it, which status codes trigger the backoff and whether 403 falls through to the auth path, and what happens when the sync cursor is rejected. Those four answers tell you which of the six steps you actually need. Start with the limiter and the retry budget, because they are what turn a bad afternoon into a bad week.

Want a second pair of eyes on your integration layer? Contact us.