Businesses look to the public cloud for flexibility, rapid scalability and dependability for many application types -- but public clouds are not perfect. Every major provider has experienced an outage with internal systems or storage, as well as external resources such as network connectivity. Outages can be devastating for any business owner, but cloud outages can affect hundreds of users.
All this underscores a common reality of public cloud computing: Users need a disaster recovery plan just like they do with local data centers. The plan put in place, and the actions taken when a cloud outage strikes, can either mitigate or exacerbate the impact on an enterprise. Let's consider six important steps to survive a public cloud outage.
The first step to survive any cloud outage is to create and implement a disaster recovery (DR) plan and have it in place long before a disaster ever occurs. While cloud providers offer a wealth of services and resources, it's up to the user to create, deploy, configure and monitor those services and resources for each workload.
The actual DR strategy can vary radically depending on the needs of the workload and its importance to the enterprise. Everyday applications may be well-suited for conventional data backups and VM snapshots to secondary locations -- such as other provider regions, another cloud provider or even to local storage resources.
An advanced DR plan can employ standby instances that are deployed but idle in another region and ready to take over when the primary instance is disrupted. Even more comprehensive DR strategies can include distributed clustering that runs duplicate workload instances across multiple cloud regions or availability zones. For example, such a strategy can include the use of load balancers to distribute traffic across multiple instances and redirect traffic when a cloud outage occurs in that region.
An extreme permutation of these duplication efforts is a multi-cloud DR strategy in which the workload is operated redundantly across two or more clouds -- such as AWS and Microsoft Azure, or Azure and Google Cloud -- to guard against the possibility of an entire cloud disruption.
Find out what's happening in the cloud when things go sideways. Cloud providers have traditionally been opaque about service disruptions and outages, but this is changing as businesses entrust more valuable workloads to the public cloud. Businesses demand more cloud transparency, and providers are improving their communication with users, offering more timely insights about the nature of an outage and its current status.
For example, AWS provides a Service Health Dashboard that offers the current status of all services, while Azure provides a similar Azure Status page. DR decision-making can hinge upon your understanding of the disaster and its severity, the provider's estimates about its duration -- all of which can improve with greater cloud transparency.
But don't stop there. The business and user base depends on the affected workloads, so it's equally important to communicate details of the outage to internal users or customers. Inform them about the outage, its effect on the workloads and the steps being taken to resolve it.
Determine what needs to be done to implement the DR plan. Some plans are automatic. For example, vital workloads are often protected through some type of clustering, and a cluster should continue to function even when a node -- or instance -- fails. However, a DR strategy for less important workloads can require manual intervention or discrete steps, such as restoring and relaunching a snapshot or switching over to a backup instance.
When manual intervention is required, consider the effort and expense involved in the recovery process and determine the business value of initiating the recovery. Ask if recovering a workload will take longer and cost more than simply waiting for the cloud provider to resolve the outage. Communications from the cloud provider will influence this determination heavily.
In many cases, mission-critical DR plans may be fully automated and administrators may not need to take any deliberate action. For example, a cluster spanning AWS availability zones, or Azure regions, may continue to function even when one node becomes unavailable during a cloud outage.
However, less critical workloads might require deliberate action. Turn to prepared scripts, templates or other resources to orchestrate the appropriate DR response. Admins must take immediate action when the business decides to initiate a DR plan that requires manual intervention. This might include restarting from a snapshot or redirecting traffic to a standby instance for the duration of the cloud outage.
DR plans require periodic testing. Perform testing drills to ensure proper procedures and resources needed to drive workload recoveries are in place. Testing also verifies and validates the configuration of associated resources, such as IP addresses and related drivers and dependencies. If the recovery functions properly during regular tests, it's likely to function properly in actual DR situations.
Regardless of the level of deliberate work or automation involved in implementing a DR strategy, it's still important to verify that recovered workloads are functioning normally. Administrators should compare the performance of workloads operating in a DR state against the performance of those same workloads operating under normal conditions.
Application monitoring tools, such as Amazon CloudWatch and Google Stackdriver, look at workload health. These tools also collect logs, metrics and events that relay operational data about the recovered workloads. Additionally, they continue to monitor the workload's performance and availability throughout the cloud outage.
Cloud outages may be painful, but they don't last forever. As the cloud provider resolves its outage and normal workload operation is restored, organizations need to perform a postmortem of the event and evaluate their DR responses.
Consider how well the DR plan did, or did not, work and adjust the plan as needed. This may include changing the level of DR protection allocated to an application, fine-tuning the process used to implement a DR procedure or other changes that can potentially mitigate the effect of future cloud outages.
