Works, stops for hours, works again? You’re being rate limited.
A feature that depends on an outside service works, stops for a stretch, then works again — with nothing deployed in between, no error page, and no alert. The affected records simply have blanks where the integration should have filled them in. That is a rate limit answering 429 while your code reads it as “no results.” Here is how to make it visible and how to stop losing data to it.
It doesn’t crash. It quietly does incomplete work.
Outside services cap how many calls you may make in a window and answer 429 once you cross it. Code that treats any non-success response as nothing-to-do carries on as though the call returned empty. The feature does not crash, it produces incomplete work, and it recovers on its own when the window resets — which is exactly what makes it look intermittent rather than broken.
The self-healing is the trap. By the time anyone investigates, the integration is working perfectly and the evidence has expired, so the incident gets filed as a glitch. Meanwhile the records written during the window stay wrong forever, because nothing ever comes back to finish them. A visible outage would have cost you an hour; this costs you a slice of your data.
Six ways a rate limit stays invisible
Any non-success response is treated as “nothing to do”
Symptom: Records save successfully with the integration's fields blank, and no error appears anywhere.
Fix: Branch on the status code. A 429 is not an empty result — it is a refusal to answer. Until those two cases take different paths, this failure cannot be seen from inside your own app.
Status codes aren't logged at all
Symptom: You suspect a rate limit but there is nothing in the logs to confirm it either way.
Fix: Log the status code and any retry hint from every outbound call before anything else. This costs an hour and turns an invisible failure into an obvious one; every other fix here depends on it.
The code bursts one call per record
Symptom: Failures cluster at the start of a batch job or immediately after an import, then clear.
Fix: Pace the work. Batch records into runs with a deliberate gap between calls instead of firing as fast as the loop allows, which is what pushes you over the limit in the first place.
Retries are immediate, so they make it worse
Symptom: Once the failures start, they last much longer than the provider's stated window.
Fix: Honour the wait the response asks for, then back off with an increasing gap and a little jitter. Retrying instantly spends the quota you are waiting to recover and can extend the block.
Staging and production share one quota
Symptom: Production breaks while someone is testing, or a seed script takes the integration down.
Fix: Use separate API keys per environment where the provider allows it, so a test run cannot consume the quota your customers depend on.
Exhausted retries still write a “complete” record
Symptom: After the outage the data is still wrong, and nothing reprocesses it.
Fix: Fail loudly. Mark the record incomplete, alert, and make it re-runnable — a partial result that looks finished is worse than a visible failure, because nothing will ever come back for it.
Paste this into your incident notes
Third-party rate-limit (429) triage
[ ] Make it visible first. Log, for every outbound call:
- the status code
- Retry-After, and the provider's remaining / reset headers
- the record or job the call belonged to
Until 429 is distinguishable from "no results", this is invisible.
[ ] Confirm the pattern:
- do failures cluster at the start of a batch or import?
- do they end after a fixed interval with no deploy?
- does a single manual call succeed while the job fails?
[ ] Fix the calling behaviour:
- treat 429 as retryable, never as final
- wait at least the interval the response asks for
- back off with an increasing gap + jitter, capped attempts
- never retry immediately in a tight loop
- pace batches instead of one call per record as fast as possible
- cache responses that rarely change
[ ] Fix the data damage:
- mark records the integration could not fill as INCOMPLETE
- alert when retries are exhausted; do not write a partial result
that looks finished
- re-run the affected range once quota is back
[ ] Prevent the repeat:
- separate API keys per environment so staging cannot eat prod quota
- a check that asserts the response CONTENT, not just a 200Assert the content, not the status code
An uptime check cannot see this. Your app is up, fast, and returning 200 the entire time the integration is being refused — the response is simply missing the data it should contain. Nightlamp’s api_canary exercises the integration path on a schedule and asserts the response actually contains what it should, so a throttled call fails the check instead of passing as an empty success.
A heartbeat on the job that consumes the API catches runs that die partway through, and http_status trends show the error rate climbing before the feature stops entirely. Together they give you the one thing a self-healing failure normally denies you: a record that it happened, taken while it was happening, so the investigation starts from evidence rather than from a memory of something that looked wrong on Tuesday.
Catch the failure that answers 200
Start a trial and point a check at the endpoint your integration feeds, with an assertion on the field it is supposed to populate. Nightlamp runs it on a schedule and tells you when the data stops arriving — even though nothing is technically down.
Start 14-day trial · no cardNewsletter
One real no-code incident + the fix, monthly
A post-mortem from a real no-code outage — what broke, how it was found, and the fix — once a month. Confirming also gets you the no-code incident runbook pack.
Double opt-in. One-click unsubscribe. No spam, ever.