Why production environment testing is better than a lab setting

Production environment testing of management tools

Reading about tools and techniques is great, but I'm a huge fan of hands-on learning. To truly understand a product, you must use it. When learning how to use tools, production environment testing is a good option, where a mistake has no impact on production. Usually, I set up a test lab and deploy a new product into that lab to get an understanding of what it does and how it might affect the production environment. Modern management applications learn about your environment and deliver value by analyzing behavior over time. The applications observe the impact of daily, weekly and monthly cycles in your environment; as they see more cycles, they make better suggestions. To assess new products, you need an environment with normal cycles.

When you have dozens of hosts and hundreds of VMs, there is no way a human can manage all the pieces, which means automated monitoring and alerting tools are essential to administrators.

So, when it comes to getting hands-on time with a management product in the lab, there are a few challenges. First, many lab environments are heavily overloaded with far more than would be allowed in a production system. This is to keep the cost of the lab environment down. Most management products will have a lot of suggestions and recommendations about buying new hardware due to the overload.

The other challenge is that labs do not have the same weekly and monthly cycles that we see in production systems. Lab environments are often loaded in surges with little cyclic behavior. Any management tool that looks for patterns will likely be confused by the workload changes on lab environments. These two obstacles make it hard to use a lab to get a feel for what a management tool is like in a production environment.

Labs are great for understanding the install and integration process for management products. Labs are also a good place to learn the user interface of the product. Your evaluation should start in the lab, as this is the first stage of establishing value and trust in the product. But, a lab is unlikely to give you a good idea of what the product will want to do to your production environment. To really understand how a management product works in your production environment, you will need to deploy it there.

Testing for real in production

Deploying a new product into production to complete an evaluation isn't usually a recommended practice. The good thing is that these management products all have a "report-only" mode, where the product makes recommendations, but lets the administrator choose whether to apply the recommendation. First, make sure the install process is not going to cause a problem in your lab. Next, deploy the product in production, but set to report only. Then, the product can start learning your cycles and you can be sure it won't apply recommendations that don't suit your organization. This is the second phase of evaluating the product.

The third stage is actually seeing the insights from the management product driving a positive change in production. A few days or weeks of monitoring and reporting on your production environment should be enough time. This will give you a solid understanding of the product's ability to understand your environment and recommend optimizations. If you choose, you could even apply just some of the recommendations to ensure that they do yield benefits. This stage should build a lot of trust in the product and drive a purchase decision, if that hasn't already been made.

The final stage is to operationalize the product in your environment. This is where I have seen the most waste. A great management product is selected and purchased, but is not always fully implemented. Sometimes, there is a pretty dashboard and a set of reports. If the product isn't used to make the business better, then all the time and money is wasted.

The management product should deliver insights and administrators should act on those suggestions to improve the production environment. This is where products that are able to act automatically deliver huge value -- especially if a VM needs to be migrated or resized, then minimize the manual intervention. Code in your change management workflow and your maintenance times. Let the expensive and complex tool do its job, and then you can work on understanding the real business requirements.

Once a basic level of understanding and trust is established, there is no better place for production environment testing than in an actual production environment due to the report-only feature. The goal of any management product is to use computers to do complex, yet predictable tasks and free humans from higher-level thinking.