Crime Story behind xz and the need to improve

Sustainable Open Source

May 19, 2024

Hey, I am Klaus Haeuptle! Welcome to this edition of the Engineering Ecosystem newsletter in which I write about a variety of software engineering and architecture topics like clean code, test automation, decision-making, technical debt, large scale refactoring, culture, sustainability, cost and performance, generative AI and more.

The story of the xz backdoor is a fascinating tale of intrigue, espionage, and betrayal. A program called xz Utils, which provides data compression for most Linux distributions, was found to have a backdoor that affected sshd. The attack involved manipulative contributions and patches by suspicious accounts over several years, ultimately compromising the tool’s security. The incident highlights the need to consider not only technical aspects but also the human costs of open source development. The attacker strategically exploited norms within the open source community, emphasizing the importance of vigilance and collective responsibility.

As Ars Technica reported, “Anyone in possession of a predetermined encryption key could stash any code of their choice in an SSH login certificate, upload it, and execute it on the backdoored device.” There are no reports of the backdoor being used, but the story behind how the backdoor landed is something straight out of a spy novel. A scary part that it required so little resources that there could probably 100s of these attacks going on right now across the entire supply chain. It also shows the importance of a secure supply chain and e.g. taking care every part has also a sustainable financial setup.

Sustainable Financial Model for Small Open Source Projects

The blog from Tidelift on Paying maintainers: the HOWTO describes the importance that companies take ownership and contribute to creating a sustainable financial model for small Open Source projects including the ask to follow secure supply chain and software engineering practices. The blog also shows how this can look like. Their approach includes involves paying maintainers to implement secure software development practices, validate the practices they follow, and contractually commit to continuing these practices into the future so that organizations can confidently make long-term investments in the packages they use.

Sovereign Tech Fund - Government Support for Open Source Projects

The Sovereign Tech Fund, financed by the German Federal Ministry for Economic Affairs, is a program that supports the development, improvement, and maintenance of open digital infrastructure. It provides funding to variuous open sources organizations to strengthen infrastructure and security. Its mission revolves around promoting digital sovereignty—the self-determined use of digital technologies and systems by individuals, industry, and governments. The Sovereign Tech Fund is a great example of how governments can support the open-source ecosystem and help ensure that critical projects are funded and maintained. The fund is a step in the right direction towards creating a more sustainable financial model for small open-source projects, but it is not sufficient. Companies and organizations that rely on open-source software should also contribute to the funding of these projects to ensure their long-term sustainability.

Aligned Funding Initiatives - Collective Responsibility in the Open Source Ecosystem

A central body which takes the responsibility of securing the Open Source would be a great improvement. All private, public entities interested in using OSS should become paid member in such central body. This body would then distribute the funds to the projects based on different factors like criticality, need, privacy / security concerns and adoption. This would ensure that the funds are distributed in a fair and transparent way and that the projects that need the most help get the most funding. This would also ensure that the projects that are most critical to the ecosystem get the funding they need to continue to thrive.

Resources

Below you can find some more resources with more details on the xz attack and the research for similar incidents.

The blog Everyhing I know behind the xz backdoor attack documents the different events behind the social engineering attack and analyzes the discovery of the backdoor.
Open Source Security (OpenSSF) and OpenJS Foundations Issue Alert for Social Engineering Takeovers of Open Source Projects: The OpenJS and OpenSSF Foundations explored shares some warning signs to look out for and some good practices to follow for maintainers of OpenSource projects.
Open Source Funding: With an estimated 97% of codebases containing open source, both the private and public sectors depend on the maintenance of open source software (OSS). Open Source is the foundation for many projects, at the same time it is often underfunded. Nicolas Zakas founder of ESlint wrote a interesting series on the topic of open source funding:

Conclusion

The xz backdoor story is a narrative that stresses the importance of security, good software engineering, the need for constant vigilance, and the value of community support. As the open-source world grapples with these challenges, one thing is clear: the need for a fortified line of defense against the ever-evolving threats and necessary improvements around various aspects like a sustainable financial setup for Small Open Source projects.

Mark as not spam: : When you subscribe to the newsletter please do not forget to check your spam / junk folder. Make sure to "mark as not spam" in your email client and move it to your Inbox. Add the publication's Substack email address to your contact list. All posts will be sent from this address: ecosystem4engineering@substack.com.

❤️ Share it — The engineering ecosystem newsletter lives thanks to word of mouth. Share the article with someone to whom it might be useful! By forwarding the email or sharing it on social media.

Software Engineering Ecosystem

Discussion about this post