api.limitguard.ai returned HTTP 503 from 2026-06-19T13:57Z until 2026-09-05T17:16Z. The process was alive the whole time. Its event loop was blocked in one synchronous call that had no timeout, and nothing in the system was positioned to notice.
This page covers what happened, why it took 78 days to see, what was fixed, and what is structurally different now. It does not cover customer data.
Timeline
All times UTC.Root cause
/health called the tracing client’s auth_check() synchronously, on the asyncio event-loop thread, on every probe. The client’s synchronous HTTP call has no timeout by default. The tracing host stalled once, the call blocked, and with the loop blocked the API could serve nothing at all.
Three things had to be true for a single stalled call to take the service down for 78 days:
- Blocking network I/O was allowed on the event-loop thread.
- That I/O had no timeout, so one stall was permanent.
- The blocked process still counted as running, so the platform never restarted it.
Detection gap
Nobody was paged because nothing was wired to page them. No monitor, internal or external, was connected to a route that reached a person. Inside the platform, the healthcheck did fail, but a failed healthcheck on a process that has not exited does not trigger Docker’s restart policy, and there was no in-process watchdog to turn “the loop has not moved in 30 seconds” into an exit. The service was unhealthy and the system agreed, and that agreement went nowhere. The lesson is not “add a dashboard”. Every signal that would have shown the outage existed already. The gap was that none of them was wired to an action.Fixes
On the host, a cron job restarts the container if Docker reports it unhealthy, and host-side scripts post to Slack. The in-process watchdog is the first line; the host cron is the second.
What is different now
- Nothing blocking runs on the event loop. Probes that can stall run in a bounded pool with a timeout, and a slow dependency degrades one probe rather than the service.
- A hung process becomes an exited process. The 30 s loop watchdog turns the failure mode that lasted 78 days into one that lasts under a minute, because an exited container is a state the platform knows how to recover from.
- Unhealthy reaches a human. Host-side signals post to Slack, and the host restarts an unhealthy container on its own within five minutes.
- Recovery is designed for, not just hoped for. Because the watchdog can hard-exit mid-operation, the settlement path was made safe against exactly that, rather than assuming clean shutdowns.