> ## Documentation Index > Fetch the complete documentation index at: https://docs.limitguard.ai/llms.txt > Use this file to discover all available pages before exploring further. # Outage post-mortem, 2026-06-19 to 2026-09-05 > 78 days of 503s from one synchronous call with no timeout on the event loop, why nothing noticed, and what is structurally different now `api.limitguard.ai` returned HTTP 503 from 2026-06-19T13:57Z until 2026-09-05T17:16Z. The process was alive the whole time. Its event loop was blocked in one synchronous call that had no timeout, and nothing in the system was positioned to notice. This page covers what happened, why it took 78 days to see, what was fixed, and what is structurally different now. It does not cover customer data. ## Timeline All times UTC. | When | What | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-06-19 13:56 | `/health` latency climbs from about 270 ms to 1.4 s over one minute. | | 2026-06-19 13:57:56 | Last log line. The event-loop thread enters a blocking `read()` on a socket to Langfuse, our tracing backend, and never returns. Every request queued behind it hangs. The reverse proxy reports no upstreams and serves 503. | | 2026-06-19 to 2026-09-05 | The container never exits, so the `restart: unless-stopped` policy never fires. The Docker healthcheck keeps executing into the hung process every 15 s. Each of those execs leaks descriptors that are never reclaimed, and by September the container's shim has reached its file-descriptor hard limit (524,287), at which point `docker exec` stops working too. | | 2026-09-05 | The outage is found during unrelated work. Read-only diagnosis before any restart: the main thread is in `read()` on fd 13, a blocking socket to the tracing host; the shim is at its descriptor limit. | | 2026-09-05 17:16 | The fix from PR #167 is deployed from its branch. The wedged container cannot be stopped normally and is killed and recreated. Service restored. | | 2026-09-05 20:03 | PR #167 merged after CI. PR #168 adds Slack alerting the same evening. | | 2026-09-06 | PR #171 (dedicated thread pool for blocking probes) and PR #172 (settlement reconciliation after a hard exit) merged. | ## Root cause `/health` called the tracing client's `auth_check()` synchronously, on the asyncio event-loop thread, on every probe. The client's synchronous HTTP call has no timeout by default. The tracing host stalled once, the call blocked, and with the loop blocked the API could serve nothing at all. Three things had to be true for a single stalled call to take the service down for 78 days: 1. Blocking network I/O was allowed on the event-loop thread. 2. That I/O had no timeout, so one stall was permanent. 3. The blocked process still counted as running, so the platform never restarted it. The descriptor exhaustion, which is what made the container impossible to inspect or stop cleanly by September, was a consequence of the hang, not its cause. ## Detection gap Nobody was paged because nothing was wired to page them. No monitor, internal or external, was connected to a route that reached a person. Inside the platform, the healthcheck did fail, but a failed healthcheck on a process that has not exited does not trigger Docker's restart policy, and there was no in-process watchdog to turn "the loop has not moved in 30 seconds" into an exit. The service was unhealthy and the system agreed, and that agreement went nowhere. The lesson is not "add a dashboard". Every signal that would have shown the outage existed already. The gap was that none of them was wired to an action. ## Fixes | PR | Change | | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | #167 | The tracing probe runs in a worker thread with a 2 s cap and a 60 s cache; the tracing client gets a 5 s timeout; DNS and TLS probes are offloaded with a hard cap. A loop watchdog exits the process if the event loop is unresponsive for 30 s, so the platform restarts it. | | #168 | CI and host security signals post to Slack. | | #171 | Blocking probes get their own thread pool, sized from the container's CPU limit rather than the host's, with its saturation state exposed. Without this, slow resolvers could silently degrade results while `/health` stayed green. | | #172 | Settlement reconciliation for rows stranded by a hard exit between crediting a ledger and submitting the on-chain leg. The watchdog from #167 introduced hard exits as a recovery path; this closes the consistency gap that path could open. | On the host, a cron job restarts the container if Docker reports it unhealthy, and host-side scripts post to Slack. The in-process watchdog is the first line; the host cron is the second. ## What is different now * **Nothing blocking runs on the event loop.** Probes that can stall run in a bounded pool with a timeout, and a slow dependency degrades one probe rather than the service. * **A hung process becomes an exited process.** The 30 s loop watchdog turns the failure mode that lasted 78 days into one that lasts under a minute, because an exited container is a state the platform knows how to recover from. * **Unhealthy reaches a human.** Host-side signals post to Slack, and the host restarts an unhealthy container on its own within five minutes. * **Recovery is designed for, not just hoped for.** Because the watchdog can hard-exit mid-operation, the settlement path was made safe against exactly that, rather than assuming clean shutdowns. Live status: [status.limitguard.ai](https://status.limitguard.ai).