Kubernetes is in Hospice

Kubernetes and containers have been making the headlines lately but not in a good way.

The site, https://k8s.af has been passed around on a few sites. It is a collection of horror stories on how companies have utterly failed with kubernetes, wasted ridiculous amounts of money and shot themselves in the foot. What's really disconcerting about some of the stories is the fact that there was something like a half dozen different talks at the recent European KubeCon on "failing with k8s". This is not something to applaud and "learn from" - this is something to be avoided.

When's the last time you told your boss that you caused a half million dollars worth of damage and went to a sunny european beach town to tell people that at least you learned something?

On twitter we're starting to see more comments like this:

or

 "kubernetes is a problem factory"

or

"Should we use containers at all is the first question. Why people skip over this one ? :)"

and of course the one I like to use:

 "kubernetes is the mongodb of devops"

(That's not a compliment.)

We've been handing out these pictures recently and yes we'll be at OSCon in Portland next week if you want to grab one!

It's even gotten to the point that the #1 container maven put a comment out on the twitters stating this:

Whether she meant that in jest or was serious doesn't seem to matter - quite a few people including other container celebrities responded in the affirmative.

People like myself have long been aware of the problems containers bring to the table and we've always just kinda ignored it. I'm mostly a live and let live guy. However, it's now gotten so bad that everyone from the military to hospitals are starting to deploy this crap and frankly that's bad news for everyone.

Let me be clear - it's completely irresponsible.

One of the things that always bugged a lot of engineers about the "cloud native" world was that it seemed mostly marketing passing for engineering.

No other industry is like this. You can't pretend to be a doctor and go do open heart surgery. You can't pretend to be a civil engineer and build a bridge that is not going to fall down. Peoples lives are at stake. Billions of dollars are at stake. However in the software realm it's seemingly completely ok to advertise this - not just advertise it but pretend that it's the second coming of Jesus.

That infuriates a lot of engineers - they are just too nice to publicly rant on it and they have better things to do.

The Birth

Here's the deal - k8s was just an attempt by Google to get people on to Google Cloud. You see Google should be the number one cloud provider by all rights yet the Google culture prevents them from being so. They are not even #2 on the list because their culture is so antithetical to this thing called "customer service". It doesn't matter how many awesome technical innovations they bring to the table (TPUs are awesome!) until their culture is changed they will never win. You could nuke us-east-1 and Jeff will still win.

The fact that storage && networking - primitives for just getting the most basic things done -- like i don't know, run a website -- have spawned complete cottage industries is insane. "I just want to run a database" entails buying something from a vendor or your database magically flying away on a given day - and oh well that devops person - they quit and joined another company cause the market is so in demand for that talent. Can anyone say "liability"?

Coming Out of the WoodWork

We are also starting to see developers start to crawl out of the woodwork at Google because they are sick and tired of hearing the marketing bullshit that k8s is somehow a replacement for Borg, the internal scheduler, that is infinitely more scaleable and that Google is running this as their main scheduler internally.

Full stop - that is a straight up lie that keeps getting told. Someone even hacked google cloud and took a screenshot proving this false for crying out loud:

(taken from https://opnsec.com/2018/07/into-the-borg-ssrf-inside-google-production-network/ )

Maybe that wasn't super wise of them to do that but there's the picture proof.

And you have to wonder why Facebook has it's own scheduler as well rather than use Kubernetes?

From The Next Platform (a favorite website of mine):

"In terms of scale, Tang says that Kubernetes cannot hold a candle to the scale of Borg/Omega and certainly not to Tupperware."

You see - these companies actually have scaling problems. They're behemoths. They would never use k8s internally. They can't.

Here's the rub though - your average 20 person startup or small/medium sized business has no need for this type of scheduling/scaling either.

The thing is that neither of these schedulers have a place outside of their respective organizations since they are both tied into a myriad of internal projects that would just not make any sense to decouple. Borg also happens to be written in c++ which is not an 'open sourceable' language. Oh? You thought the open source was just because they were nice people? There's a reason why the most popular projects on Github are written in javascript - not because of technical merit - it's just more accessible to your average developer. Most estimates plug the number of developers in the world at 20M+. I think it's probably more like 30M or more and only rising.

The vast majority code in javascript. There's a million damn repos of it on github:

The other thing that has been pissing off engineers is the generalization of the term "containers". Google's "containers" are most definitely not "docker containers". At the end of the day we really are just talking about various parts of Linux here.

I think it's the intense marketing that are straight up lies that piss some of us off the most. I'm from Missouri - the "show me" state. So call it a "white lie" or "massaging the truth" - a lie is a lie. Some of the container marketing people are just straight up professional pinocchios.

Complexity

Another recurring conversation that's been happening lately is that a lot of people have been saying that k8s is complicated. Some advocates reframe this as "you are incapable of understanding/using k8s". That might be the case for some but I'd imagine for the majority of people what they are really saying is that "There is too much complexity for something that should be simple".

Security

Then there is security. Oh boy. Container security has been a complete and total failure and shows a deep lack of leadership from those that pushed containers so hard. Containers have arguably made malware like ransomware and cryptojacking way easier to deliver.

While VMWare did scoop up Heptio for $550M the next largest ecosystem acquisition was Twistlock recently selling to Palo Alto for a cool $400M because so many security teams have severely grave concerns with the use of containers, however, since they don't necessarily have control over the devops spectrum cause they don't actually provision the software themselves a market need has been created for container security and it's the leader in the various container software segments.

It's also why we are starting to see other "container like" projects come out on the big clouds like gVisor from Google and Firecracker from AWS. They realize that there is just too much pushback for provisioning containers as is today under the current security climate.

If you're a heavy docker/k8s user I apologize for this article's tone however this ecosystem is going up in smoke. It's time to divest.