Real system administrators don't use visual tools. Real sys admins use scripts and command-line interfaces, like a dedicated car enthusiast getting dirty and greasy with wrenches and screwdrivers under the hood of a classic muscle car.
There is a certain power to a script. It enables greater innovation and control than a prescriptive point-and-click interface provides. But there is a downside to so much raw power. Users, even sys admins, make mistakes around script management: A set of commands that has been input 99 times correctly by the sys admin may be slightly off the 100th time. Without the safety blanket of automation intelligence between the sys admin and the script or Command-line interface (CLI), all hell can break loose. Most sys admins will create batch files that contain the script commands to avoid such problems, but when a one-off or a rarely used action requires a series of commands, it's often just back to raising that hood and getting the toolkit out.
What could go wrong?
The raw power of scripting was on display when a sys admin issued a clean-up script as a maintenance job at U.K. hosting company 123 Reg. Unfortunately, a command to delete certain files actually deleted everything from the root folder down, wiping out a large number of users' websites. The plug was pulled before it spread across the whole server estate, but the damage to the servers and 123 Reg's brand was done.
Automating a software build raises the same problem. Just because a scripted build sequence worked well in the development environment and passed all the requirements doesn't necessarily mean it will implement correctly in the test or production IT environments. Figuring out the problems unintentionally introduced by human hands takes time and effort that is better spent elsewhere.
Script automation is handy: It ensures repeated and consistent application of proven scripts. This kind of script management also tests that the commands are contextually aware and able to deal with different environments without manual intervention. Good script automation tools also include rollback, so it is not the end of the world should anything go wrong.
Rollback is important as scripts become more complex. Even a script for provisioning a new application becomes complicated quickly. For example, the task may require the app launch on a remote server with the code built into a provisionable container or VM. The sys admin identifies and prepares resources in the production environment for the code, setting up a virtual environment with the right server, storage and networking resources. Then the code is installed on the platform, and the sys admin implements all the settings in a contextually correct manner, from making sure the right IP addresses are in place to double-checking the logical unit numbers for storage, the ports for networking and other resource instructions. Locking in the proper settings is tedious and painful at times.
Applications encounter many issues during rollout to production, such as retaining a hard-coded link to a test database that would cause the application to not work at all. In this example scenario, all data written goes to the test database; all data read comes from the live database. While the app fails to function because of this scripting error, it only requires one change to make writes go to the live database and not test, resurrecting the app.
Another common issue is when an application needs a functional patch. This small item of additional code might be a high-priority requirement that wasn't fully logged through the change management system. Along comes a dumb script with a brute-force update, and that extra code is blown out the door. The IT operations group can spend ages trying to find out what happened while the end users are up in arms.
Better script management
Most OSes come with a CLI or with scripting capability. Microsoft's PowerShell enables the creation and operation of scripts, and bash is a de facto shell environment for the CLI instantiation and method of running commands and script files within Linux. While sys admins need scripts, they also need script management and automation tools that help prevent common human errors.
Capistrano is a basic but effective system that can be built into typical DevOps and remote IT administration environments, setting up code and scripts to provision out to multiple servers, either physical or logical, in the IT platform. Written in Ruby, Capistrano is easily integrated with other systems and also has many extensions to help provide additional functionality, but the software is somewhat wanting in the area of rollback capabilities.
Beyond basic script management and automation, systems such as SaltStack, ElectricFlow, IBM UrbanCode and Serena Deployment Automation provide enhanced functionality. These platforms support graphically-based deployment and advanced script management. A graphical user interface is a useful starting point before digging into deeper levels of control and granularity. Full management platforms ensure that a valid and error-free script is created as a basis from which technical staff members can then make changes without introducing errors. Other systems that are catching the highly technical scripters' eyes include Red Hat Ansible and Evolven's analytics product.
The other benefit of script management tool sets is that the basic script is automatically documented -- something that your office's coding wizard may not take an interest in doing. Should a sys admin leave the company, no one has to untangle his reams of scripts to understand how the IT environment functions. It is far better to have a curated and controlled library of scripts that are fully documented.
In parting, I'll leave you with an rm -r * <enter>
Why use a configuration management system
The dangers of IT automation
How SRE automates rote ops tasks