Your control plane stopped converging — silently? Catch the reconcile loop that dies, no-ops, or rolls back.

A reconcile loop fails as an absence, not an error.

The whole point of a self-converging control plane is that it keeps the desired state applied without anyone watching. So when the convergence loop stops, the symptom is… nothing. The box stays up, the API keeps serving the last-applied config, and drift accumulates silently until the next deploy fails in a confusing way. You cannot monitor that with an uptime check — the endpoint is fine. You monitor it the way you monitor any guaranteed-to-happen event: a dead-man’s switch that expects a heartbeat on success and raises an incident when it doesn’t arrive, plus proof the last reconcile actually applied.

Five ways a convergence loop breaks in silence

Ping on a health-gated success, not on every run

The key idea: POST the heartbeat URL only after the post-converge health gate passes — both api.steada.dev/healthz and api.keynest.dev/healthz returning 200. A failed or rolled-back converge withholds the ping, and Nightlamp opens an incident at roughly 1.5x the expected interval. Pinging on every timer fire would hide the exact rollback you want to catch.

# after ansible-pull applies desired state:
if curl -fsS https://api.steada.dev/healthz  >/dev/null \
&& curl -fsS https://api.keynest.dev/healthz >/dev/null; then
  # both vhosts healthy → record success + ping Nightlamp
  write_converge_status ok=true last_success="$(date -Is)" config_rev="$(git rev-parse HEAD)"
  curl -fsS -X POST "$NIGHTLAMP_HEARTBEAT_URL" >/dev/null
else
  write_converge_status ok=false last_success="$LAST_SUCCESS"
  # no ping → Nightlamp opens an incident at ~1.5x the interval
fi

Three Nightlamp checks, one for each failure shape

Map the recipe onto Nightlamp’s real check schema: a heartbeat for “it stopped succeeding,” an http_keyword on the status artifact for “it ran but did nothing,” and a plain status check for external liveness independent of the converge loop.

# 1 — dead-man's switch: success heartbeat (job pings after the health gate)
check_type: heartbeat
expected_interval_seconds: 600     # job cadence: every 10 min
# grace defaults to ~half the interval → incident if no ping within ~900s

# 2 — "ran but did nothing": converge-status freshness
check_type: http_keyword
url: https://api.steada.dev/converge-status
keyword: '"ok":true'               # true only when last_success is recent
interval_seconds: 300              # evaluate every 5 min

# 3 — external liveness, independent of the converge loop
check_type: http_status
url: https://api.steada.dev/healthz
expected_status_codes: [200]
interval_seconds: 60

Pause the switch during a deliberate stop

When you intentionally stop convergence for a deploy or migration, pause the heartbeat so it doesn’t page: set paused_until to the end of the window via PATCH /monitors/{id} (or the Pause control on the monitor page). The evaluator skips paused monitors and resumes automatically when the window passes — the same mechanism the recipe’s suppress_when: scheduled-maintenance intent maps onto today.