We determined the problem to be an unexpectedly large amount of traffic near our London region. It defeated our rate limiting by overwhelming our load balancers. In the future we will aim to have far more load balancer capacity than is necessary during normal conditions, to ensure high traffic from one customer doesn't take us down.
Posted Oct 06, 2020 - 19:00 UTC
Service has stabilized. Some accounts may be getting rate limited.
Posted Oct 06, 2020 - 12:49 UTC
We are experiencing severe connectivity problems right now, possibly due to high load. Originally this started in the London region. Failover to other regions was initiated, but this caused SF to fail. We are still investigating the cause.
Posted Oct 06, 2020 - 08:10 UTC
This incident affected: POP - London and POP - San Francisco.