Supply chain attack sends shockwaves through open-source community

Developers and security experts all over the world have been sent reeling over the past week by a narrowly avoided catastrophe in a software utility used in popular versions of the open-source operating system Linux. 

A week ago, a Microsoft developer was debugging a discrepancy in a networking protocol when he appears to have stumbled onto one of the most sophisticated supply chain attacks ever discovered. 

Beginning in February, a shadowy developer known as Jia Tan began to stealthily insert a backdoor into a piece of software known as XZ Utils, which is a compression utility present on most if not all versions of Linux — a piece of software the provides one of the basic building blocks of the internet as we know it. 

Had XZ Utils been inserted into stable — as opposed to experimental — versions of Linux, Jia Tan and his (potential) collaborators would, in theory, have been able to break into Linux servers using the utility and run arbitrary code. 

The particulars of this incident are even more alarming. Jia Tan was able to get himself designated as a maintainer on XZ Utils by taking advantage of the exhausted lone developer that had been maintaining the project. 

The near catastrophe has all the hallmarks of a highly patient espionage operation carried out by a sophisticated intelligence agency, but exactly who is behind it remains a mystery. 

XZ Utils had been kept up to date by a single maintainer working for free in his spare time. It is used throughout the world, from small projects to Fortune 500 companies, making the utility a prime target.

“It’s not a technology problem; it’s a people problem. And that’s what makes it worse,” said Omkhar Arasaratnam, general manager at the Open Source Security Foundation, a part of the Linux Foundation. “This kind of erosion of trust wasn’t because the computer was broken. It’s because somebody tricked a human.”

The vulnerability — CVE-2024-3094 — could have impacted a significant portion of the world’s servers, but even if the supply chain attack was ultimately unsuccessful, the brazen nature and close call of this incident has served as an alarm to the security community.  There was no security protocol or technology that discovered and stopped this attack.

“The good news is that we found it early,” Arasaratnam said.

The Cybersecurity and Infrastructure Security Agency sent an alert warning about the package and pointed to a warning from Red Hat, an enterprise open-source software company, about the backdoor.

The incident appears to have its origins in October 2021, when an individual calling themselves “Jia Tan” sent what was to become the first of many “fixes” to the mailing list for the data compression library.

A few months later in March, two more personas enter the scene using the monikers “Jigar Kumar” and “Dennis Ens.” They begin a pressure campaign targeting the project’s maintainer, Lasse Collin, criticizing him for his lack of updates with the apparent goal of getting Jia Tan on board as a new maintainer, according to a timeline put together by Russ Cox, a programmer at Google.

At one point, Jigar asks, “Jia I see you have recent commits. Why can’t you commit this yourself?” A “commit” is a term for adding code to a project that is only available for those who have specific access to that repository, and Jigar’s message appears aimed at convincing Collin to give Jia greater authority over XZ Utils as a maintainer.

At the time, Collin was suffering from personal and mental health issues. He eventually acquiesced and made Jia Tan a maintainer on the project nearly a year after Tan sent the first fix. 

Having acquired the authority he sought, Jia Tan then slowly began adding malicious code, bit by bit, until the backdoor was added to a XZ version. 

Then, Tan began to pressure different Linux distributions to add the malicious version to their operating systems.

The backdoor only works for a few Linux distributions, such as Debian and Fedora, but they are among the largest and most widely used. There are also signs that Tan rushed the supply chain attack in the final months, as another program was set to implement a change that would have rendered the attack useless, according to the researcher Kevin Beaumont.

Jia Tan would have gotten away with it, too, if it wasn’t for a curious software engineer named Andres Freund. Freund, who works at Microsoft, stumbled upon the backdoor while trying to debug performance issues on SSH, a network protocol that is a secure way to communicate between computers and is often used to login to a remote desktop or server. The discovery itself was almost pure luck. Freund said uncovering the backdoor required “a lot of coincidences.”

Freund then alerted the open-source community about what he found, setting off a frenzy of official alerts, investigations, the creation of free scanning tools, and dozens of blog posts about a historic caper that could have been disastrous. 

The reaction best illustrates the power of open-source projects: Within a few days, analysts graphed GitHub commits to timelines, malware researchers took apart the code, IRC chats were logged and researchers picked apart what had happened.

For defenders of open source, the incident is something of a vindication of the community’s premise: that openly available code can be scrutinized to find vulnerabilities. 

But that assumes that all of Jia Tan’s malicious code has been discovered.

Jia Tan appears to have contributed to other open-source projects, such as the widely used compression library libarchive, and now the hunt is on for whether his contributions to these tools sought to undermine them.  

According to the cybersecurity firm NetRise, contributions from Tan in libarchive found their way into at least 180 instances of the firmware of operational technology, Internet of Things devices and network devices. And while it’s not clear whether there is any malicious code — particularly as the contributions may have been a part of building the persona — the risk remains.

The complicated nature of the case, the years spent working on the utility, the complex code and multiple personas apparently working together have led many security experts to conclude that the operation targeting XZ Utils was carried out by a nation state. 

Whoever Jia Tan and his apparent compatriots Dennis and Jigar worked for, they appeared to have good operational security, as none of their emails have been seen elsewhere on the internet, including in data leaks, according to journalist Brian Krebs.

A timeline of Tan’s commits to GitHub show what appears to be someone based in China, as does his name. However, analysis by researchers Rhea Karty and Simon Henniger suggest that this might be a misdirection. Based on inconsistencies in the timezone in the commits metadata and a few times when they worked during Chinese national holidays, they hypothesize that Tan is actually based somewhere in Eastern Europe.

Security concerns around open-source software often are centered around unintentional mistakes in code that can introduce a vulnerability in a widely used software. And while concerns of a malicious hacker abusing open-source packages to open a pathway for future attacks are not new, many of the publicly known cases are financially motivated, such as cryptocurrency miners that rely on an unknowing user installing a malicious open-source package.

In December, the python package distributor PyPI temporarily shut off new registration due to the “volume of malicious users and malicious projects.”

In many open-source projects, there is a certain amount of trust in the maintainer, explained Arasaratnam. The modern economy depends on and largely exists because of a cadre of volunteers who work, often for free as a side project or hobby, on programs that underpin nearly all aspects of digital life. Maintainers are often the first and last line of defense in quality of code, feature requests and, ultimately, risks.

There likely is not going to be a “silver bullet” that can protect against nation-state operations like the XZ case, Arasaratnam said.

“The problem is this notion of trust,” Arasaratnam said. “A trusted maintainer is going to find a different way to manipulate that trust if they’re a bad actor in the system. That’s the part where I think the community doesn’t have consensus yet as to how to address that. And it’s going to be a long journey for us.”