Unpatched flaw in Anyscale's Ray AI framework under attack

An ongoing dispute

It's unclear how many organizations might be affected by ShadowRay exploitation. While Oligo researchers say they discovered thousands of compromised Ray servers, the vendor said it's difficult to estimate the actual number of end-user organizations. A single cluster IP address can represent anywhere from one to millions of users.

"We observed around 500 IPs that were exposed, which could equate to hundreds of organizations, spread across multiple cloud vendors. Regarding the scale of exploitation, what we can say is that there are many vulnerable machines. We have noticed the same attack techniques and threat actors across multiple different environments," Oligo's research team said in a statement to TechTarget Editorial.

CVE-2023-48022 was discovered and reported to Anyscale by offensive cybersecurity company Bishop Fox. In a November blog post, Berenice Flores Garcia, senior security consultant at Bishop Fox, warned that the flaw poses significant risk to organizations running the framework on cloud platforms such as Amazon EC2.

"This lack of authentication mechanisms allows unauthorized actors to freely submit jobs, delete existing jobs, retrieve sensitive information, and achieve remote command execution," she wrote. "The vulnerability could be exploited to obtain operating system access to all nodes in the Ray cluster or attempt to retrieve Ray EC2 instance credentials."

Flores Garcia also noted that at the time, Ray's GitHub repository had been forked more than 4,900 times while Ray's Docker image was pulled approximately 5 million times.

Oligo researchers also expressed concern because they observed compromised systems that included command history. That could make it easier for attackers to learn and leak secrets from productions that were used in previous commands, Flores Garcia warned.

Oligo said Anyscale listed high-profile organizations that use Ray in production, including Uber, Amazon and OpenAI. "All organizations using Ray are advised to review their environments to ensure they are not exposed and to analyze any suspicious activity," the blog read.

In a November blog post disclosing the five reported vulnerabilities, however, Anyscale disputed the findings and said it does not consider CVE-2023-48022 to be a vulnerability or even a bug. The company said the lack of authentication in the framework was based on "a long-standing design decision" in accordance with Ray's security boundaries and deployment best practices.

Anyscale emphasized that Ray is designed to provide arbitrary remote code execution as a service. Because it's a distributed execution framework, the company said it's incumbent upon Ray users to avoid running it on exposed networks and to prevent access to the clusters from untrusted systems.

"We have considered very seriously whether or not something like that would be a good idea, and to date have not implemented it for fear that our users would put too much trust into a mechanism that might end up providing the façade of security without properly securing their clusters in the way they imagined," Anyscale wrote in the blog post.

The company did say in the November post that it planned to introduce authentication in a future version of Ray, though no specific timeline was given.

A possible zero-day?

Oligo researchers said they discovered a cryptomining campaign that exploited ShadowRay to hijack victims' computing resources. "The first crypto-miner we noticed was installed on Feb. 21, 2024," the researchers wrote in the blog post. "Using public web intelligence tools, we discovered that the IP has been accepting connections to the target port since Sept. 5, 2023, indicating the breach might have started before the vulnerability was disclosed."

Oligo's research team confirmed to TechTarget Editorial that the attack included exploitation evidence for CVE-2023-48022. However, the team said as of now, ShadowRay is not considered a zero-day vulnerability.

Oligo said it has notified numerous companies about the attacks. The company also warned that based on the scale of the attacks and malicious activity observed, the researchers believe "the threat actors are probably part of a well-established hacking group."

To defend against ShadowRay attacks, Oligo urged organizations to deploy Ray clusters in only secure and trusted environments that aren't exposed to the public internet. The company also recommended implementing firewall rules or security groups to restrict access to Ray clusters and adding authorization on top of the Ray dashboard port, which is Port 8265 by default.

This article was updated on 3/28/24.

Arielle Waldman is a news writer for TechTarget Editorial covering enterprise security.

Rob Wright is a longtime reporter and senior news director for TechTarget Editorial's security team. He drives breaking infosec news and trends coverage. Have a tip? Email him.