Field debrief for the runtime outage on 2026.05.13 — 03:18Z to 06:41Z. Eleven slides. No friendly icons. Read top to bottom.
A regional DNS provider returned stale records for 3h 23m. Halcyon's resolver pinned to one of three upstream providers; the failover threshold was set too high. Tier-3 (research) clients with aggressive retry policies amplified failure into customer-visible errors. Customers on tier-1 (transactional) saw degradation but no failure.
Currently we wait until the upstream provider misses six hundred milliseconds of probes before failing over to provider 02. The new threshold ratifies a single missed probe at 180 ms.
The pin to provider 04 was a vestige from the 2025 cost review. We move to a three-way Anycast resolver, weighted equally, with provider failure quarantined for 30 minutes after a missed probe.
Research-agent clients amplified failure 4.6× by retrying inside the failure window. We expose a budget — N retries per 60s — and refuse beyond it with an explicit, customer-readable error.
The 4,840 € refund cycle was hand-cranked by two engineers between 04:30 and 09:00. We codify a refund pipeline keyed to tier × failure-class × duration, with an audit log and a postmortem hook.
During the window, the status page showed eight green pills and one yellow chevron. The customer's experience was "everything is on fire." We replace the dashboard with a one-paragraph human summary, updated every 10 minutes.
After: resolver is unpinned and weighted across providers 02 / 04 / 07. Failover threshold drops to 180 ms. Tier-3 retry budget caps amplification at ×1.4. The bottom row of this diagram never gets drawn again.
A three-letter root cause for an eleven-slide debrief — set in Archivo Black at clamp(140px, 22vw, 360px), tracking −0.06em, leading 0.82. The hazard period is the only part of this slide that is not phosphor white.
If we delay the resolver redesign past Friday close, we re-enter the failure window with the same posture we left it in. The new policy is one ticket. The redesign is a fourteen-day commitment from Q.ALB & H.NAI. This deck is the ratification artefact.
Sign-off lines below. Anything not signed by 16.05.2026 17:00Z is escalated to the founders' weekly.