My experience rejuvenating CI/CD infrastructure with AI

The challenges of legacy CI/CD migration

There can be many issues with legacy CI/CD migration. Sometimes we, the administrators, are not fully aware of the digital landmines scattered about the estate. Specific issues might include:

• Plugin dependency rot
• Undocumented groovy scripts
• Brittle pipelines
• Lack of observability
• Undocumented and unowned pipelines.
• Jumping through hoops to get pipelines to work in newer infrastructure.
• Hidden risks, such as hundreds of ad hoc one-off unmaintained jobs.
• Understanding what's in the estate to begin with.

AI can turn that headache -- mapping the current environment, creating a migration plan and managing the inherent risk of migration -- into a set of achievable goals. It can gather large amounts of spread-out data into a centralized trove. Presented correctly, this helps with understanding, planning and migration to a modern installation.

Properly scoped plans that include multiple steps that collect data and baseline data can highlight interesting statistics that would be onerous to collect manually at scale. An example is the percentage of jobs that have never actually been run or completed successfully.

The secret to any properly designed plan is not to use AI as if it were a search engine, but to create a thought-out plan with an investigation phase, a reporting phase and a build-the-tooling phase. Virtually all AI vendors provide this ability. Doing it this way means the administrator already has a plan of execution -- not just "ask, ask, ask," as this also drives up token costs.

That saves the migration team from having to move those scripts -- which likely won't be touched again, let alone used. The beauty of Jenkins is that all these jobs are contained in their own files, so they can be manually imported if really needed. However, stored secrets may pose a slight issue.

It's also important to keep in mind that AI is just a tool, a co-pilot that will do a lot of the legwork. It is the administrator's responsibility to understand what the AI infrastructure is telling them and to stay on top of the planning.

How to configure AI for legacy pipeline modernization

Configuration splits into two discrete parts. One is the technical side. The second is planning and setting up the AI project. These projects require access to the Jenkins API and the ability to execute API calls to gather the relevant project data. Jenkins has a CLI tool that can be downloaded from the existing Jenkins server. Some data extraction works better with CLI tools over API calls. The installation process can be found in the Jenkins documentation here.

1. Set up the AI environment

Coding environments can be highly personal, but I recommend Visual Code with the AI plugin Claude.ai enabled. This helps AI to do most of its own data collection. In my case, it helped to tell the AI that it could use jenkins-cli.jar to run commands if needed, since some data types are easier to obtain through the CLI than the API.

2. Tell the AI assistant what you need and how you need it

Writing the prompt properly is essential. Using a multi-phase plan provides a framework that you can tweak and run as needed to get the results you need. Abraham Lincoln said, "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." The same logic applies here, getting the discovery prompt right.

You can ask the AI to help you write the phases as well. This is night and day compared to the piecemeal approach of "ask questions about the data."

Here's an example of telling the AI tool what you need for a migration plan:

• Executive Summary: Problem, goal, scope, risks, recommendation.
• Phase 1 - Discovery & Inventory: Enumerate everything (jobs, plugins, credentials, last-run).
• Phase 2 - Risk Assessment & Classification: Active/dormant/dead, owners, prod/secret flags, scoring.
• Phase 3 - Target Architecture & Standards: Pipeline conventions, JCasC, credentials, disaster recovery.

Interestingly, some of the files that it extracted caused the default context window to be exceeded, thereby causing errors and failures. The fix here was to tell the AI to create a Python script to extract only the information it needs, rather than trying to extract the entire half-million lines of data. The administrator might need to prompt it and ask further questions, but the great thing is you can ask it to interpret information or play "what-if."

Benefits of CI/CD modernization with AI: Executing the project

Within two hours, I had a detailed multipoint migration plan. The tool also framed the document in terms of potential processes, such as:

1. A deep analysis of the legacy environment in general and issues around versions and out-of-date plugins.
2. A detailed breakdown of the jobs and the plugins the developers needed, and whether they are maintained and secure.
3. Repeatable scripts to assist in the migration.

Once the project run results are created, the user can begin asking questions about the data. Examples include questions such as "How many jobs are in a failed or never run state?" or "Of the jobs, what is needed to address any dependency issues?"

Once the tool fully understands the data, it can start to create magic. Either separately, or as one of the phases, the user can instruct the AI to create several sets of documents -- one for the technical stakeholders, and one for management and other less technical but important stakeholders.

Limitations of CI/CD modernization with AI

Using AI to understand the data contained within the environment is a great use of AI infrastructure. That said, the data must be reviewed by a human who understands the estate. As we all know, it can hallucinate, and the administrator needs to be able to deal with that.

During my experimentation with the migration project, it became obvious that to get the most out of AI -- including using it to build the supporting tooling -- the human in charge must supply the right guidance and ask the right questions. This can include getting AI to help develop baseline tooling.

For example, when not explicitly prompted to do them, the AI tool can overlook obvious tasks, such as forgetting to create a remediation plan for OS upgrades.

Closing thoughts

This approach of extensive data gathering, followed by shaping the process and tooling with AI, isn't limited to just CI/CD pipelines. It excels at deep diving, data collection and making the overload of data more understandable. Asking it to apply its "knowledge" of services, such as Jenkins, means it can offer useful suggestions on how to turn the mass of data into a migration plan. 

Stuart Burns is a DevOps engineer at a leading shipping and logistics provider.