Heartbeat monitors

What is a heartbeat monitor?

Every other check type is active — Nightlamp makes a request and evaluates the response. A heartbeat is the inverse: the monitored job tells Nightlamp it succeeded, and the absence of that signal is the alert. You configure one number — expected_interval_seconds, how often the job should check in — and Nightlamp watches for a gap.

• expected_interval_seconds — how often the job is expected to ping (30–604800s).
• grace_seconds — extra slack before a missed ping alerts. Optional; defaults to half the interval, so an incident opens at roughly 1.5× the interval.
• last_ping_at — set by each ping; read-only.
• heartbeat_ping_url — the unique URL to ping; generated when you create the monitor.

How do I create one and where is the ping URL?

Create it like any monitor — pick Heartbeat — dead-man's switch as the type and set the expected interval. The ping URL is generated on creation and shown on the monitor page (and on the edit form). The unguessable token in the path is the credential, so treat the URL as a secret; rotate it by recreating the monitor.

From automation, POST to the monitors API:

The response includes heartbeat_ping_url. The monitor's check-type is fixed after creation, but the interval and grace are editable.

How do I ping from a cron job?

POST or GET the URL after the work succeeds. GET is accepted so plain curl / wget one-liners work. Ping only on success — putting the ping at the end of the script, after the real work, means a crash or non-zero exit withholds it.

For a self-converging control plane, gate the ping behind a post-run health check so a rolled-back or no-op convergence withholds it — see the control-plane self-convergence guide.

When does Nightlamp open an incident?

On each evaluation, Nightlamp compares the current time against the last ping. When now − last_ping_at > expected_interval_seconds + grace_seconds, the monitor goes down and an incident opens through the normal alert pipeline. Before the first ping ever arrives, the clock runs from the monitor's creation time — so a job that never starts pinging is caught too, not just one that stops. When a ping arrives after a miss, the next evaluation resolves the incident automatically.

The ingest endpoint and its payload

Pings go to https://api.nightlamp.app/heartbeat/{monitor_id}/{token} over POST or GET. There is no session or API-key auth — the unguessable token in the path is the credential. Every accepted ping returns {"ok": true, "received_at": <unix>}; any miss (wrong id, wrong token, deleted monitor) returns a uniform 404 so the endpoint never reveals which ids or tokens exist.

A POST may carry a JSON body of numeric fields. Nightlamp stores it as last_ping_metrics on the monitor and evaluates it against any payload_thresholds you've set (below). GET pings carry no body and leave the previously-stored metrics untouched, so a plain cron one-liner still keeps the dead-man's switch alive.

Alerting on a value, not just a missed ping

A heartbeat can also carry numeric payload thresholds. On each evaluation, once Nightlamp confirms a ping arrived on time, it reads the most recent last_ping_metrics and checks every rule. If any rule is breached the monitor goes down with a detail like backlog=83.0 > 50.0. This lets a job self-report not just that it's alive but that its work is actually keeping up — e.g. a forwarder whose queue is draining and whose oldest item is fresh.

• field — a key in the posted JSON. Dotted paths like queue.backlog resolve into nested objects.
• op — one of gt, gte, lt, lte, eq, ne.
• value — the numeric threshold to compare against.

A field absent from the latest ping is skipped, never treated as a breach, so partial payloads don't false-alarm. A missed ping always takes precedence over a threshold breach — a stale heartbeat is the louder signal. Up to 20 rules per monitor.

How do I pause alerts during maintenance?

Pause the monitor for the window — the Pause control on the monitor page, or PATCH /monitors/{id} with a future paused_until timestamp. A paused monitor is skipped by the evaluator, so a deliberately stopped job won't page on-call, and it resumes automatically when the window passes.