A production deployment checklist for enterprise apps

1. Use automation tools

Developers use a diverse set of tools to create, test and control software builds. Deployment automation offers the same benefits of other forms of IT automation: It hastens the implementation of approved policies and best practices within the deployment environment, while ensuring security and consistency -- no steps are skipped or erroneous. The production deployment checklist begins here.

There are many application deployment automation tools available. Simple scripts, such as Windows PowerShell, are sufficient in many cases. But IT teams can add more sophisticated and full-featured tools, such as SolarWinds Patch Manager, Octopus Deploy, Jenkins, Bamboo, Maven and Team Foundation Server, to the development toolchain, as well. Evaluate tools based on their ability to integrate with the existing toolchain, including with repositories like Git. In addition, evaluate these tools based on security features, support for updates and patches, and post-deployment monitoring capabilities.

2. Coordinate development and IT

Modern tools and the DevOps paradigm have given software development teams unprecedented autonomy in the production environment, including the ability to provision IT resources, apply patches and deploy new software to production. Still, IT operations retains primary responsibility for the infrastructure and production environment, so it is vital that developers collaborate and coordinate releases and patches with the operations team.

Timely coordination ensures that IT resources -- server VMs, network subnets and storage volumes or logical unit numbers (LUNs), for example -- are available. If a new release requires specific resources, such as a new physical server or additional network bandwidth, IT can implement and prepare those resources -- or make any other necessary infrastructure changes prior to release. Typically, operations staff has immediate access to infrastructure error logs and alerts that are invaluable for monitoring and troubleshooting the deployed applications.

3. Understand KPIs

Deployment is not the end of a software cycle. Teams must monitor modern applications to oversee health and performance. Monitoring involves a broad category of tools, known as application performance monitoring (APM). But before they can monitor applications, developers and stakeholders must establish the key performance indicators (KPIs), or other metrics, on which they will assess software behavior. New software releases might demand new KPIs, while patches and updates call for additions or tweaks to existing KPIs.

KPIs include simple metrics, such as CPU and memory use, webpage load times, database access or query response times and basic application health. Complex enterprise applications rely on more sophisticated KPIs, calculated or derived from software performance parameters. These might include orders or transactions per second; queue depth, or how many tasks are waiting for response; and response times for critical tasks.

4. Have a deployment and rollback plan

Developers must establish a clear understanding of, and approach to, any given deployment. For example, a patch or update might overwrite the prior build, which effectively applies the patch or update. However, new applications, or builds with substantially new or changed feature sets, might deploy in parallel with the current build in an A/B, or blue/green, deployment. A parallel approach enables IT to test new features and functionality in production across a systematically widened user base without jeopardizing the established build. If the new build, B, is then adopted, all users can be cut over to the B environment and the prior A environment is decommissioned to free IT resources.

Regardless of the deployment approach, it must include a comprehensive rollback or restoration plan to invoke when monitoring data indicates critical production problems. This can be as simple as restoring a prior, known-working build. IT might need to deactivate the alternative A/B deployment and redirect all traffic to the prior known-working build. Rollback plans must account for dependencies, such as database files, to ensure no critical business data is lost in the rollback.

5. Back up and protect the current build and dependencies

Backups, in their various forms, should be part of any production deployment checklist. Before applying a patch or application update, capture a complete copy of the working application and its dependencies. Maintain and restore to this copy if the patch or update encounters problems. The actual backup process can vary from traditional copies of application and data files to up-to-the-second VM snapshots.

Modern applications are complex, and demand a sophisticated operational infrastructure.

The emergence of immutable infrastructure has affected the nature of the backup as well. The concept of immutable infrastructure is that of no change. When a new build, such as a patch or update, is available, the build replaces the former, similar to the A/B deployment above. If there is a problem with the new B deployment, it is destroyed and the previous working deployment, A, is redeployed. This approach relies less on traditional copies, but rather on data file protection and build and release repository management to retain and control previous application builds.

6. Provision and deploy

Modern applications are complex, and demand a sophisticated operational infrastructure. Typically, developers start a deployment with automated tools to provision a suitable operational environment. A simple application might need only a VM with a valid IP address and access to a LUN. But more complex applications might require numerous VMs, redundant and synchronized data files, connections to dependencies like databases, and connections to load balancers for high-availability or high-throughput operation.

The provisioning and deployment process brings developers and operations staff into close contact. Developers must coordinate their efforts clearly with operations to ensure the availability of necessary resources. For example, provisioning additional VMs on a full application server will cause deployment problems and delays -- and potentially affect the performance of applications already running in the environment.

Use software tools to provision and manage resources and services from common pools. This enables the software-defined environment to decide where to place VMs, LUNs and other components.

Once the operational environment is provisioned, use automated tools to install the application build into its appropriate resources -- such as loading build images into VMs or containers. Open the application for initial testing to ensure it works. If the deployment checks out, open it up to users; if not, roll back and adjust.

7. Communicate to stakeholders

All applications have stakeholders, which include employees, business leaders, business partners and the broad spectrum of business customers. Stakeholder communication is often overlooked in application development and deployment. However, development teams should inform stakeholders of key application events.

These events include the schedule for patches and updates that might incur downtime and result in flaws. Stakeholders must know when to expect disruptions, when it's clear to use the application and how to report errors or request help. Communication related to new or substantially upgraded applications might also include educational content, such as detailed change logs, new documentation or videos and opportunities for hands-on training.

8. Connect application monitoring

Application deployment into the production environment usually includes monitoring tools, such as APM and log management software, to oversee application availability, health and performance. IT teams typically deploy monitoring separately, as an agent, within the app's environment. Include monitoring tools in any new deployment process; verify those tools remain connected and functional after any patches or upgrades.

Configure monitoring tools to use suitable thresholds, check or calculate desired KPIs, generate desired reports or log content and alert appropriate IT staff to errors or alerts.

9. Monitor the deployment environment

The final item on this production deployment checklist is to use the monitoring software. Ongoing review of monitoring data reveals performance and behavioral trends that influence the application's support needs and further development. Developers, for example, will want to watch the KPIs and rate of incident help desk ticket submissions for signs of application problems. Compare KPIs before and after a patch to identify any performance issues with an update. An uptick in incident rates might suggest a flaw in the update -- or signal users' confusion or dissatisfaction with a corresponding feature or function.

Dependencies can affect application performance, and often require additional data monitoring. For example, trouble with a database or web server can cause performance problems in the application; granular visibility into dependencies can aid with application issue isolation and remediation. Similarly, more granular components of the deployment infrastructure, such as server use, network use, storage use and load balancer levels, influence application performance. While infrastructure monitoring is usually an IT operations activity, share data with developers to help identify possible application problems.