• We are introducing Instantaneous PowerLoss Storm, a new testing paradigm within the meta-infrastructure to manage and mitigate instantaneous or unforeseen power outages in our data centers.
• We share: How we built the readiness to tolerate immediate failures in our existing systems using deep defense strategies; what compromises were made during implementation and how we confirmed our willingness.

---

Disaster preparedness is not an option. Hurricanes, wildfires, power and network disruptions, and countless other disaster scenarios pose risks to our data center (DC) infrastructure.

Early warning systems and proven mitigation strategies already serve us well in situations where advance warning takes a few hours or longer. While these strategies have matured over time as we have expanded our DC footprint, the ever-increasing size and diversity of our infrastructure requires a greater level of preparation for unpredictable disasters (those that occur without warning), のような: instantaneous loss of powerと minimal impact on overall fleet availability.

Instantaneous PowerLoss Storm is a new testing paradigm within the long-established meta-method Disaster preparedness (DR)」Storm” program that provides the last line of defense and ultimate safety net to manage and mitigate immediate or unforeseen power outages due to known, emerging and unknown risks.

How we built the readiness to tolerate immediate failures in our existing systems using defense-in-depth strategies.

The ability to handle an instantaneous power outage had to be built into our DC stack from the ground up, from mechanical and electrical to server racks, from storage to compute and core cord Container orchestrator. Fortunately, each of these architectures has already been designed with power failure tolerance as an integral component.

Provides the ability to retain in-memory data when racks run out of power using batteries and rechargeable batteries Siren in case of power failure (PLS) is one such capability. Another advantage is having a robust DC region-wide asynchronous signaling mechanism for Twine services in the form of Unavailability Events (UE). (A DC region – hereinafter referred to as a “地域” – is a region in which several DC buildings are located next to each other and share a common network and power connection.)

While these capabilities were battle-tested and hardened in individual fault domains within individual DCs, we identified salient vulnerabilities in scenarios that spanned an entire region. さらに, when testing a region, we not only had to deal with issues of scaling (a typical region is typically 50-60x the size of the typical fault domains) and replica placement, but also issues of autonomous bootstrapping.

Bootstrapping refers to kickstarting a powered off region and requires millions of services to be started at once and discover each other autonomously. Below we describe two of the problems we encountered while bootstrapping that required the introduction of a Belt-and-suspenders approach to cover all possible eventualities and contingencies.

A prominent problem that has dogged us from the beginning is that of dependencies and especially the dreaded dependencies circular dependency, “ouroboros“Risk! Our Twine orchestrator has a number of control plane services – plannerallocator, broker, Zelos (coordinator) 等. – without which we cannot operate or start other services in the region. While the risk of circular dependencies is low in regular operations, the risk and impact is far higher when bootstrapping an entire region. It’s a real chicken and egg problem.

We solved this problem through identification critical startup dependencies among the control plane services, and we continually detect these early and often と Belljar tests in our CI/CD pipelines. These helped uncover and eliminate most, if not all, dependency risks before deploying them into production. Given the rapid development of our infrastructure and as a belt-and-suspenders solution, we needed this too Capability To Break all circular dependencies this may have happened unexpectedly. A specially designed Twine recovery kit provides this “jump start” capability to restore the Twine services that operate Twine itself. Together with Belljar and Twrko, we have managed to successfully end the specter of circular dependencies.

We are also on a “Boomerang” Problem in the same environment – の Generator of a critical signal influenced by the same signal. The UEs used to orchestrate service shutdown and recovery ended up shutting down the Orchestrator control plane services themselves, resulting in orphaned services that could no longer run.harvested」 (because they never received a UE.) While this problem could have been solved with complicated solutions such as excluding a preset set of services from the UE dispatch list, we opted for a simpler and more sustainable approach by allowing control plane services to simply “ignore” shutdown signals associated with power-related UEs.

Tradeoffs in finding the right balance between reliability and speed of growth.

While it is feasible to build a watertight tolerance to immediate losses, doing so may come with an opportunity cost in infrastructure or the risk of over-engineering our systems. The latter even carries the risk that false alarms will affect regular operations. したがって, we had to make certain compromises to find the right balance between reliability and technology.

We started by drawing the line at which impacts must be avoided. Data loss from storage and database systems, permanent damage to DC facilities (mechanical/electrical), or lasting impacts beyond a single region are some that we have clearly stated as essential requirements. Transient service failures, rack failures (within a predefined threshold), and limited staleness in service routing tables or region unavailability detection (this is an issue). difficult problem for asynchronous systems) were viewed as tolerable risks. In general, only issues that fell outside the limit of tolerable impacts that cannot be mitigated by post-incident remedial action and within a reasonable mean response time (MTTR).

How we confirmed our readiness through the Instantaneous PowerLoss Storm exercise and how this allows us to push the boundaries even further.

Validating the above expectations and preparing by shutting down a large production region involves significant risks with several known and unknown unknowns. To solve this chicken-and-egg problem of taking risks to address risks, we developed an incremental approach where we validated self-contained issues such as dependencies as new regions/pre-production regions emerged and ran tests in “shadow regions” that replicate production regions. We were then able to successfully test in our newest (and therefore smallest) production regions with a limited explosion radius. Finally, we shut down large production regions that house critical storage, AI, and data warehouse workloads. At this point we have named these assault drills Immediate storms with power outages.

From an altitude of 10,000 feet, the storm consists of an injected power failure resulting in an immediate shutdown of the entire region and after a short MTTR, remedial action is taken to seal off the affected region from global controllers/planners. We also wanted to avoid taking preventive measures before testing that could actually constitute an unexpected power outage. The MTTR selected for testing reflected the typical MTTR observed in real incident scenarios.

Each of these exercises helped iteratively train our infrastructure and engineers toward the long-term goal of managing the loss of a region as seamlessly as the loss of a subregional fault domain.

Springboard into the future: Slow is gentle. Smooth is fast

Despite all the precautions, this was not a completely smooth journey, but a journey with numerous opportunities for learning and improvement, which not only improved our testing capabilities, but was also reflected in our entire infrastructure and brought about several architectural improvements to our existing systems.

Ours in tandem Infra has rapidly evolved to cover countless use cases of capacity and AI. Fast action is only possible if we have a strong foundation. reliability そして speed are two facets of the same coin. You can’t have one without the other. The ability to restore a region after a sudden outage has laid a strong foundation that has allowed us to innovate and validate DC designs, build reliability in lockstep with rapid capacity deployments, and continue to push the limits of risk we can tolerate.

While previous storms primarily validated storage and database backends, we follow the same incremental strategy to validate regions with live client traffic against immediate failures. (More on this in an upcoming post!) We are also constantly reviewing and reworking trade-offs in light of new challenges that arise in this phase of growth.