Flavijus Piliponis â stock.ado

Unpatched flaw in Anyscale's Ray AI framework under attack

Oligo Security researchers say thousands of Ray servers have been compromised through the flaw, but Anyscale said it has received no reports of exploitation.

A critical vulnerability in a popular AI framework that was disclosed in November but never patched is now under widespread exploitation, according to new research from Oligo Cyber Security.

In November, Anyscale disclosed five vulnerabilities that affect the open source AI framework Ray. Anyscale addressed four of the flaws with the release of Ray 2.8.1. However, Anyscale disputed one of the vulnerabilities, tracked as CVE-2023-48022, stating it was a feature, not a vulnerability, and therefore did not require a fix.

Now Oligo said the unpatched vulnerability, which it dubbed "ShadowRay," is being widely exploited and could allow remote code execution on servers running the AI framework. Oligo researchers Avi Lumelsky, Guy Kaplan and Gal Elbaz detailed an ongoing attack campaign in a blog post on Tuesday that warned the vulnerability could let attackers access organizations' highly sensitive data as well as hijack their computing power.

CVE-2023-48022, which received a 9.8 CVSS score, involves a lack of authentication for the framework's Jobs API, which allows unauthorized users to access Ray's dashboard. Oligo said Ray, which organizations use to scale AI and Python applications, has access to data sets and models that are typically connected to company databases and knowledge graphs. The researchers stated another potential risk is that attackers could gain access to third-party tokens.

"Thousands of companies and servers running AI infrastructure are exposed to the attack through a critical vulnerability that is under dispute and thus has no patch. This flaw has been under active exploitation for the last 7 months, affecting sectors like education, cryptocurrency, biopharma and more," Lumelsky, Kaplan and Elbaz wrote in the blog post.

The researchers emphasized that "thousands of publicly exposed Ray servers" globally have already been compromised throughout those seven months. Additionally, the blog post stressed that because Anyscale did not address ShadowRay, many development teams may not be aware of the risks the feature poses.

However, Anyscale pushed back on Oligo's findings. "We have notified all Anyscale customers of the vulnerability and that they are not affected. To our knowledge, no Anyscale customers have reported malicious activity that could be related to this vulnerability," a company spokesperson told TechTarget Editorial. "Our team is currently working on a fix that will make it easy for users to verify their configuration and avoid accidental exposure."

The Anyscale spokesperson said the company will have more information within the next 24 hours.

Update: In a blog post published Wednesday evening, Anyscale released a new authentication tool for Ray users. "In light of reports of malicious activity, we have moved quickly to provide tooling to allow users to verify proper configuration of their clusters to avoid accidental exposure," the company said.

The tools include a client-side script and server-side code, which are available along with instructions on GitHub. Anyscale said the authentication capabilities will be included in version 2.11 of Ray, which is expected to launch in April.

An ongoing dispute

It's unclear how many organizations might be affected by ShadowRay exploitation. While Oligo researchers say they discovered thousands of compromised Ray servers, the vendor said it's difficult to estimate the actual number of end-user organizations. A single cluster IP address can represent anywhere from one to millions of users.

"We observed around 500 IPs that were exposed, which could equate to hundreds of organizations, spread across multiple cloud vendors. Regarding the scale of exploitation, what we can say is that there are many vulnerable machines. We have noticed the same attack techniques and threat actors across multiple different environments," Oligo's research team said in a statement to TechTarget Editorial.

CVE-2023-48022 was discovered and reported to Anyscale by offensive cybersecurity company Bishop Fox. In a November blog post, Berenice Flores Garcia, senior security consultant at Bishop Fox, warned that the flaw poses significant risk to organizations running the framework on cloud platforms such as Amazon EC2.

"This lack of authentication mechanisms allows unauthorized actors to freely submit jobs, delete existing jobs, retrieve sensitive information, and achieve remote command execution," she wrote. "The vulnerability could be exploited to obtain operating system access to all nodes in the Ray cluster or attempt to retrieve Ray EC2 instance credentials."

Flores Garcia also noted that at the time, Ray's GitHub repository had been forked more than 4,900 times while Ray's Docker image was pulled approximately 5 million times.

Oligo researchers also expressed concern because they observed compromised systems that included command history. That could make it easier for attackers to learn and leak secrets from productions that were used in previous commands, Flores Garcia warned.

Oligo said Anyscale listed high-profile organizations that use Ray in production, including Uber, Amazon and OpenAI. "All organizations using Ray are advised to review their environments to ensure they are not exposed and to analyze any suspicious activity," the blog read.

In a November blog post disclosing the five reported vulnerabilities, however, Anyscale disputed the findings and said it does not consider CVE-2023-48022 to be a vulnerability or even a bug. The company said the lack of authentication in the framework was based on "a long-standing design decision" in accordance with Ray's security boundaries and deployment best practices.

Anyscale emphasized that Ray is designed to provide arbitrary remote code execution as a service. Because it's a distributed execution framework, the company said it's incumbent upon Ray users to avoid running it on exposed networks and to prevent access to the clusters from untrusted systems.

"We have considered very seriously whether or not something like that would be a good idea, and to date have not implemented it for fear that our users would put too much trust into a mechanism that might end up providing the façade of security without properly securing their clusters in the way they imagined," Anyscale wrote in the blog post.

The company did say in the November post that it planned to introduce authentication in a future version of Ray, though no specific timeline was given.

A possible zero-day?

Oligo researchers said they discovered a cryptomining campaign that exploited ShadowRay to hijack victims' computing resources. "The first crypto-miner we noticed was installed on Feb. 21, 2024," the researchers wrote in the blog post. "Using public web intelligence tools, we discovered that the IP has been accepting connections to the target port since Sept. 5, 2023, indicating the breach might have started before the vulnerability was disclosed."

Oligo's research team confirmed to TechTarget Editorial that the attack included exploitation evidence for CVE-2023-48022. However, the team said as of now, ShadowRay is not considered a zero-day vulnerability.

Oligo said it has notified numerous companies about the attacks. The company also warned that based on the scale of the attacks and malicious activity observed, the researchers believe "the threat actors are probably part of a well-established hacking group."

To defend against ShadowRay attacks, Oligo urged organizations to deploy Ray clusters in only secure and trusted environments that aren't exposed to the public internet. The company also recommended implementing firewall rules or security groups to restrict access to Ray clusters and adding authorization on top of the Ray dashboard port, which is Port 8265 by default.

This article was updated on 3/28/24.

Arielle Waldman is a news writer for TechTarget Editorial covering enterprise security.

Rob Wright is a longtime reporter and senior news director for TechTarget Editorial's security team. He drives breaking infosec news and trends coverage. Have a tip? Email him.

Dig Deeper on Threats and vulnerabilities

Enterprise Desktop
Cloud Computing