GitHub Universe: Large IT orgs share AI automation gains

Nordstrom NordAssist's evolution

Migrating each of Nordstrom's code repositories from GitLab to GitHub required a fundamental restructuring because of differences in how the two software development platforms handle configuration files.

"In GitLab, a developer's pipeline is generated at runtime," said co-presenter Ella Ordona, senior software engineer at Nordstrom, during the session. "We use the include keyword, which is GitLab's flavor of reusability, and local variables to construct one large, monolithic file assembled behind the scenes. GitHub Actions works completely differently: it favors reusable workflows and Actions which are smaller, composable files that teams can call directly."

Moving existing repos from one platform to the other required extracting configurations from GitLab's files and converting them to GitHub Actions workflows. That's where NordAssist came in, for developers using Nordstrom's Standard Pipeline internal developer platform.

Agents aren't perfect. … Know where you need 100% accuracy and always keep a human in the loop before merging.

NordAssist's AI agents are built using Microsoft's Azure OpenAI service, OpenAI vector stores and the AutoGen AI agent framework, according to Watt. Initially, NordAssist took a single-agent approach to automating the conversion between GitLab CI files and GitHub workflow files, where one agent handled the entire process of extracting variables, generating templates, populating values and creating the final GitHub pull request for developer review.

This resulted in some pain, Watt said.

"We quickly learned one agent doing everything doesn't scale well," Watt said. "Variables weren't always mapped correctly. Values were missing or misinterpreted. Jobs were skipped or incorrectly generated, and most importantly, the agent didn't follow rules 100% of the time and took a long time to generate the final output."

With the single NordAssist agent, only 54% of the configuration code was generated by AI and only 28% of pull requests were merged without manual commits. The NordAssist team went back to the drawing board, replacing the first iteration with a multi-agent system that also incorporated more traditional deterministic automation code to improve the results.

In this second system, variable extraction, mapping and filtering from GitLab files is handled using deterministic logic. It's handed off to two specialized agents, a template generator agent that creates the new GitHub workflow files and a populator agent that fills in variables.

"Template generation requires pattern recognition, adaptability and contextual awareness. This agent doesn't just copy and paste; it constructs," Watt said. "It lists valid job options to guide engineers and inserts smart defaults based on known patterns and at least placeholders where values are unknown, signaling where human [intervention] and further automation is needed. By focusing solely on structure, this agent ensures every workflow is well-formed, logically organized and ready for the next phase."

Similarly, the populator agent has its own sense of the context of variables and their scopes, and can apply them intelligently and format them correctly across production and non-production environments, Watt said. With the newer multi-agent system, 70% of code during the migration is generated by NordAssist and 40% is merged without manual commits.

"These stats are still growing as of today, and as we're early in our migration, we're hopeful that these stats will continue to grow," Watt said.

Among her takeaways for the audience was that "agents aren't perfect. They can miss constraints, run over time or produce unexpected results. Know where you need 100% accuracy and always keep a human in the loop before merging."

Cathay Pacific applies AI to DevSecOps

As with many enterprise IT presenters at GitHub Universe this year, Cathay Pacific, an airline based in Hong Kong, has recently adopted platform engineering and tool consolidation to enhance its software development practices. The company transitioned from more than 50 separate toolchains to the GitHub platform in 2024, which included GitHub Copilot, Advanced Security, Code Security and Autofix. It began applying these tools to improve DevSecOps this year.

Before this consolidation and adoption of AI to automatically scan code for security vulnerabilities, automatically fixing some and suggesting fixes for others, it was common for projects to run into late-stage security fixes that took up between 10% and 25% of sprint times, according to Cathay Pacific presenters. Now, more than 600 of the company's 1,000 developers use Copilot, 95% of whom are active users, and 89% of whom use Copilot agent mode.

So far, the results include 15% velocity gains and up to a 30% code completion acceptance rate. More dramatic improvements in application security feature a 40% improvement in security fixes compared to last year, a 16% higher vulnerability remediation rate, and a 63% drop in developers' mean time to remediate vulnerabilities, according to Naveen Jaisankar, DevSecOps practice lead at Cathay Pacific, during his breakout session presentation.

"We started by shifting left," Jaisankar said. "We made sure that all those tools are embedded in the development environment itself, and all the vulnerabilities that were there, we were able to bring those in context back to the IDE [integrated development environment] for developers to fix."

As with Nordstrom, it took some massaging of AI-based systems to improve their accuracy before the company started seeing these returns.

"What we have seen is a bit more of a low adoption rate at the initial stage, and then come the LLM [large language model] hallucinations," he said. "Incomplete context and older models did not help us … which caused confusion among the developers and contributed to productivity slowdowns instead of productivity gains."

Moving to Copilot agent mode and tying in data from other corporate systems, such as Atlassian's collaboration software suite using Model Context Protocol, boosted the accuracy of AI responses, Jaisankar said.

Cathay Pacific's story was also similar to Nordstrom's in that the airline's platform teams created and maintain homegrown tools to improve processes for developers. In Cathay's case, GitHub Advanced Security presents vulnerabilities in a way that enables risk-based vulnerability management. However, the company has added its own internal system to deduplicate and prioritize alerts among GitHub and the multiple types of third-party security scanning tools it's required to use for regulatory compliance.

"We reduced the noise so that people can focus on addressing the real issues with guardrails … that push people to address critical or high [severity] vulnerabilities," Jaisankar said.

AI automation has helped developers address more security vulnerabilities, but the security of AI itself has also been a major consideration for Cathay Pacific, Jaisankar said, especially the potential for shadow AI.

"Keeping humans in the loop for all the critical approvals and following software development best practices is much more critical than ever," he said. "This required frequent training [for humans] and having governance built into the whole workflow. … While we are focusing on ensuring that velocity goes up, we need to make sure that we aren't growing the risk."

Beth Pariseau, a senior news writer for Informa TechTarget, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.