Tag: I do this because I love you

I have told you to Do Your Updates, twice. A good example of why is the recent news about supply chain attacks in popular npm packages, which may mean nothing to you, and I figured I’d break it down.

Firstly, most folks understand that a supply chain is… a chain… of supplies. Tautology aside, it specifically means the chain of manufacturers, people, places, and companies through which various stuff flows through to an endpoint. Let’s take my fake coffee shop, Bobbucks, as an example. Bobbucks sells fancy coffee and (of course) pastries. Bobbucks does not want to have to have individual bakeries in every city/county/country that it owns, because Bobbucks’ primary focus offering is *coffee*, not pastries. Therefore, Bobbucks contracts with local corporate bakeries across the world.

Those bakeries make pastries according to Bobbucks standards, but key ingredients are fairly universal: for example, flour. All of those bakeries need to get flour, and they probably don’t all get it from the same place across the world, but there’s a good bet they get it from the same place in a geographic region. We’ll take that part of the chain. Now we have Bobbucks, which contracts with Starbakers for pastries, which in turn contracts with Queen Guenevere Flour company. Queen Guenevere Flour company in turn gets the wheat from Alan’s Wheat Farm.

Those products don’t magically flow, though, so for this supply chain we need trucks, and trucking companies. The trucking companies that are used in each part of the chain are contracted between the two links, e.g., Bobbucks and Starbakers have one trucking company (probably more, but we’ll say one to make it easy) between the two of them; Starbakers and Queen Guenevere Flour may have a different one.

If someone wanted to attack this supply chain, they could do it at different spots, with different results. For example: if someone were to put some laxatives in the pastries at Starbakers, then Bobbucks is unknowingly buying laxative danishes and selling them to people, who will then get sick. Bobbucks will need to do some investigating to figure out where it’s coming from, would probably quickly find the culprit in the danishes, and push back to Starbakers. Now Starbakers has to figure out if it’s one of their staff, or one of their ingredients.

Maybe it *wasn’t* some gremlin at Starbakers, maybe it was a gremlin at Queen Guenevere Flour company putting laxatives in the flour. Or maybe one of the trucking companies. Each company has to spend time and money to figure out where it happened, to rectify it. In the meantime, people need to be notified to get their pastries elsewhere and to take Imodium.

Specious examples aside, you also see this not so much in supply chain *attacks* but general “oopsie” like when a farm has questionable fertilization practice and ships a bunch of lettuce with ecoli– which then gets washed and chopped up in a processing plant (but maybe not washed enough) — which then gets packaged up with authentic Pirate Frank’s packaging for all the Pirate Frank stores — which then ends up in your cart. How many food recalls have you seen lately?

“But Bobbie”, you say. “Bobbie, that is concrete hard things that move from place to place. How do you attack a software supply chain?”

By poisoning a package. Or several.

As we’ve discussed previously, it is not efficient for you, the developer, to create a formula every time you want to say, convert Celcius to Fahrenheit. Someone else has done it and they’ve put it available for others to use, up in a registry. If you, a developer, need to create a shiny new website for your Ancient History Studies college courses, you would go searching for a package that already exists on the registry that, say, converts Julian dates to Gregorian dates (or vice-versa). You wouldn’t hand-code it yourself because you value your time and also your sanity.

That registry is visible and more importantly, open source. That means that if Person A has built that Julian to Gregorian date converter, and Person B has a Mayan Calendar conversion they want to add, they can publicly add to that package to make it more useful for them and others. That add is visible, and can be checked both by the registry and subsequent editors/adders/changers. There are all kinds of places and ways the content can get scrutinized.

For each fine cat, a fine rat. A particularly fine set of rats have gone to the very most popular packages – packages that handle string pattern matching, or prettifying things, or cleaning up things, or converting things – and put some poison in them. Sometimes the poison is to capture credentials (e.g., your logins or suchlike). Sometimes the poison is to silently watch what you do on your machine for ages to see if you go to any crypto sites (so it can grab your wallet) or banking or whatever. The little code injection captures what it needs and sends it faithfully off to the architect of this chaos, and sometimes you find out right away and sometimes you don’t.

The thing about supply chain attacks is that it isn’t just you, or a handful of yous. Much like with our flour analogy, those packages get used by Company A to build a thing which Company B buys, and uses in their thing that they in turn sell to Company C. Each of those companies have customers who use their products and it’s possible a customer is a customer of all 3 and so tracing back to “where did this come from and what is it doing” can take an appalling amount of time. Also, it’s not just one package. They use more than just one package. They may use dozens, or even hundreds, throughout a large product offering. And sometimes it’s a combinate poison: part 1 of the poison is in package Foo, but part 2 of the poison is in package Bar, and engineers tend to use both Foo and Bar packages.

Once the real origin is figured out though, time is still of the essence. Companies and developers have to update to the last known good or the newest known good version of those packages, push those updates out to *their* customers, and *also* have to sanitize all their stuff, change their passwords, their 2FA/multi FA, etc. It’s not enough to take Imodium, you’ll also want a probiotic and lots of Gatorade. And you may stop getting pastries from Bobbucks.

So do your updates.

PS – “how were attackers able to poison the packages in the first place?” – Phishing. They sent official looking (down to the return address) scary mails to package owners telling them they had to update their 2FA credentials and used that data to gain access to multiple packages and locations. They sent the same kind of official mail, with lots of urgency in it, to lots of package owners, and lots of package owners fell for it.

DO NOT click links in official sounding scary emails. All of those that purport to come from your bank, or important places like this, have actual websites you can actually go to directly without clicking on specious links. Same thing goes for phone calls from “the bank”, “social security”, “the IRS”, etc. Thank them for calling, tell them you will hang up and call them back. Then call back on the phone number from the *website*, not the number they called you from. (The IRS doesn’t call – they don’t have anywhere near the human capacity for that).