skamille/gist:78fcc0576a0db3e72c25ec0cf456f619 Secret

Last active May 6, 2021 00:47

Star 45 You must be signed in to star a gist
Fork 2 You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/skamille/78fcc0576a0db3e72c25ec0cf456f619.js"></script>
Save skamille/78fcc0576a0db3e72c25ec0cf456f619 to your computer and use it in GitHub Desktop.

Download ZIP

Questions on the future of Open Source

Raw

gistfile1.md

Regarding this tweet:

https://twitter.com/skamille/status/1106955368901550082

Which links to this article:

https://redmonk.com/sogrady/2019/03/15/cloud-open-source-powder-keg/

Which led to some thoughts and questions in my mind about the future of open source systems projects.

Here are some assertions about a class of great open source produced over the past 10 or so years:

It seems to come out largely from developers in big but not usually the biggest companies that have large technology teams (specifically: not necessarily "pure" tech companies). See: Kafka from LinkedIn, Envoy from Lyft, Airflow from airbnb, Mesos from Twitter, and a ton of stuff from Netflix
It was built to meet a specific internal need that developers in the company were facing. It was not built with the intention of being turned into a company, or to sell to customers.
Open sourcing seems initially aimed to: drive brand awareness for hiring at the company in question, give back to the community, reward the developers themselves who worked on it with a degree of industry recognition. Open sourcing is most specifically not done to get more customers for the company itself. (Some will claim that the open sourcing of some of the technologies was done as "trojan horse" exercises to distract other companies with a product that was never actually used internally. I don't believe anyone ever intentionally did something like this, although there are definitely products that caused that outcome for some of their early adopters).

For a lot of these products even to this day, the only way the developers ever got compensated for their innovative work was through the salaries and equity they got from their employers, and the name recognition that made them more desirable to be hired. But a few of the major projects (Kafka and the Hadoop ecosystem being easy examples) decided to go the venture capital route, creating startups (and eventually public companies) built around these technologies. Ideally, this allows the products to be developed more reliably, funds an ecosystem of experts, consultants, trainers who can teach other developers how to effectively use and run these projects, and most importantly provides major rewards for the creators of the projects that are tied to the success of the project itself in a core way, and not just the success of the business that, as a side effect, produced the project.

Along comes the cloud. Now, fewer and fewer people are actually taking this open source software and running it themselves. Sure, the cloud providers employ a lot of engineers, but overall they shrink the demand for widespread knowledge about the operational details of these projects. Furthermore, they capture the lion's share of the value that otherwise might have gone to the creators of the project if they were able to successfully spin the project into a company.

As the article linked above says, the cloud providers are incentivized to hire the people who created these successful projects in the first place, or the major contributors. And they do pay well, perhaps on average a better outcome for many contributors to these projects than any startup-driven outcome might be. Economically, this might be overall good for our open source creators.

But where will the next generation of open source systems come from, in this world? The open source software being created from the ground up by these cloud providers and big company consortiums frequently has complexity commensurate with the scale and resources of its creators. The beauty of many of the successful projects is that they were purpose built to a pretty specific use case, one that just happens to be widely shared in the industry. They started with clear focus and a small customer base they needed to please. We're seeing a ton of projects being built the opposite way, trying to address a perceived need in the industry (or, in my mind, worse: trying to take a piece of a successful ecosystem from a very large and extremely scaled tech company and make it useful for external developers outside of that ecosystem, thus generally violating Gall's Law).

If most companies are completely reliant on the cloud for all of the software that requires operations in some way, I wonder whether we will really see much new open source creation in this space at all. Sure, we'll continue to see new OSS front-end frameworks, and libraries, and languages, and all of that is extremely valuable, but we're going to lose the stuff that made being a distributed systems engineer in the wider world fun over the past 15 years. And I believe that we will lose out in innovation and usability, the kind of work that comes from people who are trying to solve a specific need for a small group of people.

This feels especially dangerous to me as all of the big-3 cloud providers (Google, Amazon, Microsoft) were not initially built on a bedrock of open source contribution. They have all done their fair share of contribution, but for their size, they've been slower than many smaller companies to earnestly give back important pieces of technology, perhaps because their core technologies created a competitive moat that was not the case for smaller companies. I am skeptical of companies who don't really have open source in their founding DNA to do the best thing for the community in the long-run, and I worry that the cloud is just moving us back to a world of proprietary software. Hell, many of these providers are just providing open source API compatibility with custom-built backends! What happens when no new open source comes out of the smaller companies, and the big-3 decide they don't really need or want to play nice anymore?

avidas commented Mar 17, 2019

I disagree that this does not come from the biggest companies, e.g. Kubernetes/Tensorflow from Google, React from Facebook.

noahgibbs commented Mar 17, 2019

@avidas She doesn't say nothing comes from them - Cassandra was from Facebook as well, for instance. But one thing these have in common is being well away from the companies' primary interests (Cassandra) or done much less or later to try to head off OSS competition (Kubernetes.) These are not companies that contribute a lot, proportional to their size and activity. I am ignoring Google's whitepapers, which have been excellent and have started many of these OSS projects. But that's different from them contributing working software (e.g. for Chubby, BigTable, etc.)

ahrkrak commented Mar 18, 2019

Some great thoughts and questions. I would love to read a part 2 (&3 & ...) for a more complete picture of your views and thoughts as to how we should proceed as an industry — and why the traditional angel/VC-funded startup model isn’t right for generating innovation in open source, even though as you say we see a “ton” of projects from that route (admittedly most will fail of course, but that’s the model).

sandrom commented Mar 18, 2019

Some really good questions I am wondering about myself. I think it is an important discussion we need to raise in the industry

johnmark commented Mar 18, 2019

Thank you for writing this. It's very well thought out. What you described is why I believe that a.) Open source has helped distribute wealth upwards and b.) the only way out if this cul de sac is government mandates and regulation. We just happened to come of age at a time in the software industry when it was economically advantageous to participate in open source ecosystems. With the cloud, as you noted, the incentive structure has changed.

I, too, worry about the long-term viability of open source ecosystems. On the other hand, I haven't seen any appreciable slowdown in useful software being produced. But that's not to say we shouldn't anticipate negative consequences from current trends.

keleffew commented Mar 18, 2019 •

edited

You probably already know about these all definitions, but for the other people who visit this gist as well, there are only a few distinct business models that work to subsidize the development of open source software.

For example, there is the often cited Red Hat model where open source projects subsidize development with professional services (like consulting, integration, and help-desk). These are often billed through recurring support subscriptions - which can lead to misaligned incentives causing developers to create components that need to be ‘supported’ over long periods of time.

Years ago, the dominant open source monetization strategy consisted of bundling software with OEMs. This used to be the norm - but as hardware standardized throughout the 1980s, it became more difficult for manufacturers to derive profit, as the underlying operating system would run on any manufacturer's computer that shared the same architecture.

More common today is the open core model. This model involves having a dual-licensing scheme with a publicly available codebase, and a forked private enterprise version which has more enterprise-focused features. Usually, this means additional security, tooling, and monitoring. This model is widely deployed across the open source community and has been widely implemented by successful players in the enterprise space. However, in many ways, this model does not fully play into the underlying spirit of free and open source software.

Finally, there is always the route of monetization through advertising. However, if data is not carefully secured and anonymized, this model can quickly devolve into abusive privacy violations for end users.

Way back in 2002, Joel Spolsky (founder of Stack Overflow) wrote a great blog post around a business pattern where successful technology companies try to ‘commoditize their products compliments’, in order to increase the demand of their own products.

In this article, Joel cites, Netscape’s early model of ‘giving away’ the browser so they could make money selling servers. Microsoft’s had similar success in the 80s/90s by commoditizing the PC market (to increase demand for MS-DOS and Windows).

This is essentially the classic Razor-Razor-Blade model taught in business school.

This model plays out today in Public Cloud, where software is commoditized in order to drive cloud consumption. In the case of OSS, it is literally given away for free in order to drive margin on hardware usage and its associated margin, which has increased steadily year over year due to flat prices and decreasing hardware costs.

As Ben Golub (CEO of Storj Labs, former CEO of Docker, Gluster) states in this Linux blog post, “the most common way that open source is used in the cloud is as a loss-leader to sell infrastructure.”

This model is obviously not sustainable in the long-run, and will inhibit OSS innovation and experimentation if left unchecked

afolarin commented Mar 18, 2019

Then what's your take on https://aws.amazon.com/blogs/opensource/keeping-open-source-open-open-distro-for-elasticsearch/. Though I do think there's a problem with some "open core" proponents at times pressure from VC can mean these end up restricting anything competing with their business model.

keleffew commented Mar 19, 2019

I think it’s great when major Cloud Providers support Open Source, but in reality, their messaging can be somewhat misleading.

The real underlying business case for OSS with cloud providers is to drive cloud consumption (they make all their revenue off the margin between hardware and cloud consumption) free software = more cloud consumption. It seems a little asymetrical, especially when OSS generates all the value that gets captured by the cloud providers.

The case of Elasticsearch is difficult, as, on the one-hand Elastic created the product, and should have some basic right to the intellectual property. On the other hand, I believe the world would be better off (from an innovation and security perspective) if all software was open source.

At the end of the day, it’s best to examine the business incentives, AWS wants elasticsearch open because it drives cloud consumption (hence profits) for Amazon. EC wants a piece of this pie, which is fair, as they built and marketed the initial software. A good middle ground might be if OSS creators could profit off the margin of cloud consumption in the same way that Big Cloud does.

The marriage between Decentralized Cloud swarms and Open Source might be a good pathway towards a better deal for OSS.