Postmortem: Cluster connectivity outage, September 24, 2026

Duration: 21:43 to 23:15 UTC (about 92 minutes) Status page: Incident report

Summary

On September 24, 2026, clusters connected to Signadot lost their connection to the Signadot control plane for about 92 minutes, and sandbox creation failed during that window. A control plane release that day included an upgrade to Go 1.27. A behavior change in Go 1.27's HTTP client caused an internal health check to leak connections until the service that maintains cluster connections ran out of resources. We restored service by rolling back the release, and have since shipped a fix.

We're sorry for the disruption this caused.

What happened

For over a year, an internal health check had opened a connection on each run and closed it without reading the full response. Under earlier Go versions, that closed the connection. Go 1.27 instead keeps the connection open for reuse, and since the health check never reused it, every run leaked one.

The leak began when the release rolled out and grew steadily until the affected service ran out of resources and clusters lost connectivity.

Timeline (UTC, September 24)

Time Event 20:03 Control plane release rolls out. The leak begins. 21:43 Clusters lose connectivity. 23:04 Release rolled back. 23:09 Root cause identified. 23:15 Connectivity restored.

A fix was merged on September 25 and shipped in a follow-up release.

What we found

• A toolchain upgrade changed the behavior of unchanged code. Our pre-release testing didn't run at production scale, so the leak grew too slowly there to notice.
• A missing resource limit turned a leak into an outage. With a limit in place, the leak would have caused a quick restart instead.
• We needed better health signals. Alerting on resource usage and cluster connections would have flagged the leak well before it became an outage.
• The same pattern existed elsewhere, including in software that runs in customer clusters. None of it had shipped with Go 1.27, so it was never affected, and it is now fixed.

What we've done

• Fixed the health check, with a regression test that fails on Go 1.27 without the fix.
• Found and fixed the same pattern across our other components before any affected release shipped.
• Added a resource limit so a similar leak is contained to a quick restart.
• Re-released the control plane with the fix and monitored it closely.

For a more detailed postmortem, please contact us.