Who is Lasse Collin?
WMMM #067 - This week I share another edition in the Master Useless Information series.
Jeff Keplar Newsletter June 15, 2024 7 min read
Mastering “Useless Information” is an antiphrasis for “being interesting” as a sales skill, in addition to “being interested.”
Living on the Brink of Disaster
Did you know the world narrowly avoided a cyberattack of epic proportions two months ago?
The targets were some of our most important computer systems:
Computers used by banks that play a vital role in our monetary system
Airline systems whose purpose is to ensure safe commercial air travel
Military computers that protect countries around the world.
What these computers had in common was that they all relied on open-source software.
A strange fact about modern life is that most of the computers responsible for it are running open-source software. That is, software that was primarily written by unpaid, sometimes even anonymous, volunteers.
Some crucial open-source programs are managed by just a single overworked programmer.
And as the world learned last month, these programs can become attractive targets for hackers.
In this case, the hackers had infiltrated a popular open-source program called XZ.
Slowly, over two years, they transformed XZ into a secret backdoor.
And if they hadn't been caught, they could have taken control of large swaths of the internet.
In this week's edition of Win More, Make More, I share the story behind the XZ hack, what made it possible, and how the hackers took advantage of the strange way we make modern software.
And how that impacts the sale of software to enterprises.
h/t: NPR's Planet Money, "< The hack that almost broke the internet," May 17, 2024
Linux
"The first clue was this message that appeared in his inbox on February 26th."
Richard Jones is a senior engineer at Red Hat.
He helped make the Linux operating system, used pervasively worldwide, including Fortune 500 companies, hospitals, banks, and the U.S. military.
Linux is composed of different pieces of software that individuals are developing for free.
There was nothing unusual about the email.
Richard often emails with strangers on the internet.
"I don't know who half the people I talk with… are… I've never met any of them. Instead, we work on reputation."
Richard's email was from a new-ish person who had built a solid reputation.
The name was Jia Tan.
Jia Tan
For about a year, Jia had been a volunteer in charge of a popular software program called XZ.
XZ is compression software.
It isn't the fastest, but has the highest compression rates and is very useful to Red Hat.
In the email, Jia sounds enthusiastic.
"I just made a cool update to XZ. Hope you guys can put it in your operating system."
The email looked very innocent, with smiley emojis and a friendly tone.
So Richard puts the updated version of XZ into the staging process to be included in Linux.
He then starts receiving bug reports.
The messages were strange but not entirely unusual.
The updated XZ appeared to be messing with crucial components of the server's memory.
But bugs in software aren't unusual.
So Richard emailed Jia and asked him to look at the problem.
Jia responded within a few days and apologized.
He explained that they had just released a new version that fixed the bug for Red Hat.
Jia asked Richard to upgrade to this new version.
Richard does, and everything seems fine until it doesn't.
Someone discovers that this new version of XZ is not what it appears to be.
And, Jia Tan is not who he seems to be.
Jia Tan was a group of hackers.
And they were trying to pull off one of the most audacious cybersecurity attacks in history.
Over two years, these hackers had infiltrated XZ, one of the most popular programs in the world.
And if they had not been caught, they would have had a secret back door to some of the most critical systems of our society.
The Open-Source Software Movement
If you peek under the hood of the internet, you'll find that most of the computers powering it are running free, open-source software.
Small teams write a lot of that software.
Sometimes, these teams are only one person, making them vulnerable and easy to infiltrate.
So these trillion-dollar corporations work side-by-side with unpaid, sometimes anonymous volunteers to write the software that powers the internet.
The XZ hack could not have happened if it hadn't been for this weird way that most software is developed.
Doesn't that sound a bit unhinged?
Welcome to how "modern" software is developed.
How did we get here?
Bruce Perens, one of the founders of the open-source movement, was a young programmer at Pixar.
He wrote software that helped make movies like "Toy Story 2."
Bruce wrote a piece of software that would monitor other software and alert him when it would overwrite things that it shouldn't.
This solved a common problem for him.
He called the software "Electric Fence" because it would zap the offending software like an electric fence does when you touch it.
Electric Fence" proved helpful, so Bruce began sharing it with colleagues.
And Bruce's colleagues began sharing handy routines that they had written.
They began posting code on an online bulletin board.
Soon, their software was being used all over the world.
Then, something unintended happened.
Some of the programmers Bruce had never met began adding improvements to "Electric Fence."
Not only could giving your code away help others, but they could help make your code better.
And if they all collaborated, they could write better software much faster.
This was the beginning of the powerful idea behind open source.
Individuals could come together to produce software in an open, crowdsourced way, enabling them to compete with the goliaths of tech.
The economics of making software could be transformed.
As the internet came of age, startups like Google and Facebook weren't purchasing commercial off-the-shelf (COTS) software to build their internet companies.
They went with do-it-yourself (DIY.)
They started with Linux, a free, open-source operating system, and ran open-source on top of that.
Soon, every startup began adopting this approach.
Open source is now the default way to make modern software.
A library of building blocks.
However, there is also a weakness to this open-source model.
And this is not merely the opinion of those who value the COTS approach to building and maintaining software.
This weakness became painfully obvious when the XZ hack went down.
Anatomy of the Attack
There is a famous cartoon by webcomic xkcd about how the internet works.
There is a drawing of a giant Jenga tower.
All these blocks are stacked on top of one another, balancing on one tiny, skinny little block.
Open-source software is this huge decentralized community of people building software on top of other software on top of other software.
While an incredibly efficient way of building software, it can also lead to weak spots.
The leading creator of XZ is Lasse Collin.
His location is unknown, but his website is hosted in Finland.
Lasse first published XZ in 2009.
XZ achieved an order of magnitude better compression than any other software.
It was a breakthrough technology.
Everyone began using it, and XZ became one of the most widely distributed programs in the world.
There is a good chance that XZ is on your phone and TV.
This is how the Jenga tower problem starts.
The whole world unknowingly becomes dependent on a single, random person.
In open-source, this scenario is not uncommon.
But software isn't a thing you write once.
You've got to maintain it.
This makes the Jenga tower problem a recurring one.
Computers change, and new processors are released.
Operating systems change, and new kinds of computers are invented.
We must keep our software up to date, or it will rot.
Doing so isn't glamorous work.
Most open-source volunteers want to contribute to the shiny new projects.
They don't want to spend their time with older legacy software.
So, after many years, maintaining XZ falls to Lasse and Lasse alone.
The hackers, calling themselves Jia Tan, entered the scene in 2021.
Posing as an individual, Jia Tan begins by suggesting improvements to XZ.
It seemed innocent at the time.
A few months later, Lasse started receiving emails from XZ users.
They complain that Lasse has been falling behind with maintaining XZ.
One of them is somewhat nasty and accuses Lasse of no longer caring about XZ.
So, Lasse is doing this for free and receiving rude emails.
Lasse apologizes to these users and confesses to being distracted by personal issues.
That still does not appease them.
One suggests that he step down and let someone else manage the updates to XZ.
Soon after, that is what Lasse does.
He passes the baton to a new volunteer.
Their name was Jia Tan.
We now know that Jia Tan was an invented personality.'
We also suspect that the users harassing Lasse were also invented.
These hackers executed a social engineering attack.
They basically ran a long con and tricked Lasse into doing things and giving them permissions they should not have received.
Over the next few years, Jia makes all these little changes to XZ.
The changes appear innocent, but they make XZ into a Trojan horse.
OpenSSH is the garage door opener to the internet.
It lets you remote-control other computers
Pretty much every web server is running it.
It is a critical piece of software.
Everyone has their eye on Open SSH.
Why is this relevant?
OpenSSH depends on XZ
XZ is nowhere near as well-known as Open SSH.
There are few eyes on XZ.
By infiltrating Open SSH with XZ, Jia Tan would give themselves access to every critical computer on the internet.
This plan was incredibly well-orchestrated.
Earlier this year, Jia Tan began pressuring the major open-source operating systems to use the new sabotaged version of XZ.
And the compromised version of XZ begins spreading across the internet.
Discovered by Accident
A programmer at Microsoft working on open-source saw that Open SSH was acting kind of slow.
He started picking the code apart and found a thread to pull.
He pulled and found the hack.
Andres Freund sent an email alerting the compromise he found, and the attack was thwarted.
Luckily, the sabotaged version of XZ was caught before widespread distribution.
It was mostly running on staging servers used to update production software.
The Jia Tan hack has made people reconsider the open-source economic model.
Impact on Selling Enterprise Software
Large companies in all industries have safeguards to protect themselves from the scenario described in the Jia Tan hack.
These safeguards require months of testing, including running a test system and its production counterpart in parallel in a pseudo A/B test.
They include multiple approvals and signoffs such as compliance, security, risk, and data administration.
Regulated industries like financial services and telecommunications have even more stringent processes a third party must navigate before software can be purchased and implemented for use by their enterprise.
Every software component that makes up your software must be identified and vetted.
As a salesperson, this means work.
We must identify not only the decision process but the buying process.
It will involve multiple stakeholders and require multiple approvals.
We will encounter "prove its" from different areas and must pass each test.
This requires time, patience, and the buyer's trust in you and your employer.
Thank you for reading.
Jeff
When you think “sales leader,” I hope you think of me.
If you like what you read, please share this with a friend.
I offer my help to sales leaders and their teams.
I possess the skills identified in this article and share them as part of my service.
In my weekly newsletter, Win More, Make More, I provide tips, techniques, best practices, and real-life stories to help you improve your craft.
