Skip to Content [alt-c]

October 2, 2016

Systemd is not Magic Security Dust

Systemd maintainer David Strauss has published a response to my blog post about systemd. The first part of his post is replete with ad hominem fallacies, strawmen, and factual errors. Ironically, in the same breath that he attacks me for not understanding the issues around threads and umasks, he betrays an ignorance of how the very project which he works on uses threads and umasks. This doesn't deserve a response beyond what I've called out on Twitter.

In the second part of his blog post, Strauss argues that systemd improves security by making it easy to apply hardening techniques to the network services which he calls the "keepers of data attackers want." According to Strauss, I'm "fighting one of the most powerful tools we have to harden the front lines against the real attacks we see every day." Although systemd does make it easy to restrict the privileges of services, Strauss vastly overstates the value of these features.

The best systemd can offer is whole application sandboxing. You can start a daemon as a non-root user, in a restricted filesystem namespace, with mandatory access control. Sandboxing an entire application is an effective way to run potentially malicious code, since it protects other applications from the malicious one. This makes sandboxing useful on smartphones, which need to run many different untrustworthy, single-user applications. However, since sandboxing a whole application cannot protect one part of the application from a compromise of a different part, it is ineffective at securing benign-but-insecure software, which is the problem faced on servers. Server applications need to service requests from many different users. If one user is malicious and exploits a vulnerability in the application, whole application sandboxing doesn't protect the other users of the service.

For concrete examples, let's consider Apache and Samba, two daemons which Strauss says would benefit from systemd's features.

First Apache. You can start Apache as a non-root user provided someone else binds to ports 443 and 80. You can further sandbox it by preventing it from accessing parts of the filesystem it doesn't need to access. However, no matter how much you try to sandbox Apache, a typical setup is going to need a broad amount of access to do its job, including read permission to your entire website (including password-protected parts) and access to any credential (database password, API key, etc.) used by your CGI, PHP, or similar webapps.

Even under systemd's most restrictive sandboxing, an attacker who gains remote code execution in Apache would be able to read your entire website, alter responses to your visitors, steal your HTTPS private keys, and gain access to your database and any API consumed by your webapps. For most people, this would be the worst possible compromise, and systemd can do nothing to stop it. Systemd's sandboxing would prevent the attacker from gaining access to the rest of your system (absent a vulnerability in the kernel or systemd), but in today's world of single-purpose VMs and containers, that protection is increasingly irrelevant. The attacker probably only wants your database anyways.

To provide a meaningful improvement to security without rewriting in a memory-safe language, Apache would need to implement proper privilege separation. Privilege separation means using multiple processes internally, each running with different privileges and responsible for different tasks, so that a compromise while performing one task can't lead to the compromise of the rest of the application. For instance, the process that accepts HTTP connections could pass the request to a sandboxed process for parsing, and then pass the parsed request along to yet another process which is responsible for serving files and executing webapps. Privilege separation has been used effectively by OpenSSH, Postfix, qmail, Dovecot, and over a dozen daemons in OpenBSD. (Plus a couple of my own: titus and rdiscd.) However, privilege separation requires careful design to determine where to draw the privilege boundaries and how to interface between them. It's not something which an external tool such as systemd can provide. (Note: Apache already implements privilege separation that allows it to process requests as a non-root user, but it is too coarse-grained to stop the attacks described here.)

Next Samba, which is a curious choice of example by Strauss. Having configured Samba and professionally administered Windows networks, I know that Samba cannot run without full root privilege. The reason why Samba needs privilege is not because it binds to privileged ports, but because, as a file server, it needs the ability to assume the identity of any user so it can read and write that user's files. One could imagine a different design of Samba in which all files are owned by the same unprivileged user, and Samba maintains a database to track the real ownership of each file. This would allow Samba to run without privilege, but it wouldn't necessarily be more secure than the current design, since it would mean that a post-authentication vulnerability would yield access to everyone's files, not just those of the authenticated user. (Note: I'm not sure if Samba is able to contain a post-authentication vulnerability, but it theoretically could. It absolutely could not if it ran as a single user under systemd's sandboxing.)

Other daemons are similar. A mail server needs access to all users' mailboxes. If the mail server is written in C, and doesn't use privilege separation, sandboxing it with systemd won't stop an attacker with remote code execution from reading every user's mailbox. I could continue with other daemons, but I think I've made my point: systemd is not magic pixie dust that can be sprinkled on insecure server applications to make them secure. For protecting the "data attackers want," systemd is far from a "powerful" tool. I wouldn't be opposed to using a library or standalone tool to sandbox daemons as a last line of defense, but the amount of security it provides is not worth the baggage of running systemd as PID 1.

Achieving meaningful improvement in software security won't be as easy as adding a few lines to a systemd config file. It will require new approaches, new tools, new languages. Jon Evans sums it up eloquently:

... as an industry, let's at least set a trajectory. Let's move towards writing system code in better languages, first of all -- this should improve security and speed. Let's move towards formal specifications and verification of mission-critical code.

Systemd is not part of this trajectory. Systemd is more of the same old, same old, but with vastly more code and complexity, an illusion of security features, and, most troubling, lock-in. (Strauss dismisses my lock-in concerns by dishonestly claiming that applications aren't encouraged to use their non-standard DBUS API for DNS resolution. Systemd's own documentation says "Usage of this API is generally recommended to clients." And while systemd doesn't preclude alternative implementations, systemd's specifications are not developed through a vendor-neutral process like the IETF, so there is no guarantee that other implementers would have an equal seat at the table.) I have faith that the Linux ecosystem can correct its trajectory. Let's start now, and stop following systemd down the primrose path.

Comments

Reader nextime on 2016-10-02 at 21:28:

More: Even assuming sandboxing is a panacea ( it isn't, but it's a good thing anyway ), the assumption that systemd make it easy and without systemd it isn't easy it's false.

There are plenty of easy to use tools, let me say for example firejail but many others works too, that implement sandboxing in easy and accessible way, and probably thanks to a better awareness, even better than what is usually done with systemd.

Reply

Reader JD on 2016-10-02 at 22:30:

This (and your previous post) has to be one of the best and most well thought-out posts about why we should not be using systemd. Most people seem to hate systemd for religious reasons but these are excellent scientific reasons. Bravo!

Reply

Reader David Strauss on 2016-10-02 at 22:55:

Here you go: https://medium.com/@davidtstrauss/ayer-vs-systemd-part-4-832e0cc374c1

Reply

Anonymous on 2016-10-03 at 08:41:

Why do this read like a mailing list argument in blog form?

Reply

Anonymous on 2016-10-03 at 08:17:

Damn, Ayer got wrecked by David's response https://medium.com/@davidtstrauss/ayer-vs-systemd-part-4-832e0cc374c1

Kid doesn't have a clue.

Reply

Anonymous on 2016-10-03 at 12:50:

It is more like Strauss doesn't have a clue at all. He resorts to ad-hominem again (like you just did with the "kid" attack), picks sentences out of context, answers things that weren't even asked, does a typical systemd apologist post - when in doubt, non-sequitur, ad-hominem, fud.

Reply

Anonymous on 2016-10-04 at 12:07:

He also makes mention that "OpenSSH on Fedora ships with systemd" and infers that this is OK. Wait, you work for Red Hat, they pay you and also ship Fedora? Bring me back to where this matters???

Reply

Andrew Ayer on 2016-10-03 at 15:18:

LOL

Reply

Reader DV on 2016-10-04 at 14:01:

He, he. Strauss simply doesn't understand, or doesn't want to see, that sandboxing is one thing. However, privilege separation and design in an application is another. The former does not protect an application from getting completely owned.

The fact that you've got a systemd developer talking like this is exceptionally worrying. He also seems to think a non-privileged user being able to do a denial of service on the system is a 'trivial security problem'. Is it me or do systemd's developers not get the concept of a multi-user system or anything Unix systems have had for decades?

Reply

Anonymous on 2016-10-05 at 12:14:

He is likely inspired by the latest out of academia, unikernels.

Effectively DOS in a VM...

Reply

Anonymous on 2016-10-03 at 14:42:

I am not well-versed in security matters, but I want to make my best. So I read both your post and the latest answer of Strauss.

The result is that I understand your position and arguments, while the the post of Strauss leaves me utterly confused.

Which can only lead me to conclude that you are right, since clarity of design seems to be the top quality in terms of security.

So thanks for the warning.

Reply

Reader JCC1 on 2016-10-03 at 18:37:

Glad to see you mention qmail, and his response to your comment was fairly hilarious. Privilege separation via exec chaining (aka Berstein Chaining) is a key concept that the systemd developers don't get. If they DID get it, they'd understand that that's precisely why you don't run all of this in the same process. Small, simple utilities that do what they need to do, drop privileges, and execute something else.

Supervise > tcpserver > drop privs > app authentication > remainder of the app. This is the antithesis of the systemd model. If they truly understood it, they'd understand the importance of a tiny PID 1, which serves as the basis for everything else: small, secure tools which do precisely and only what they need to do, remove their privileges, and chain to the next step.

Reply

Reader DV on 2016-10-04 at 15:13:

There's a couple of possibilities. Either they don't understand this, which I don't think they do anyway, or they deliberately don't want to do this.

If they were to turn PID 1 into something simple that basically spawned processes then there is the possibility that a replacement could be made, replacements for components could be made or alternative init systems can keep getting used. By controlling PID 1 from the kernel all the way down and software then starting to rely on systemd you are hard pushed to get this thing dropped or replaced.

If PID 1 is the way it is for political reasons then it's the worst possible reason, because they have completely sacrificed any sort of priorities on security for politics. Once it becomes more widespread in various Linux systems, embedded systems and network devices, this is going to blow up in all our faces.

Reply

Reader DV on 2016-10-04 at 15:30:

Andrew says systemd has got to where it is by marketing, but I don't think that's true. It's politics, and one of the most political piece of software we've ever seen in the open source world, and that's saying something.

Reply

Anonymous on 2016-10-06 at 11:27:

Marketing and politics overlap to a increasing degree.

Frankly what is going on is that systemd devs envision only two use cases, desktop and containers (the latter being a way for them to gain some appearance of credibility). Anything else is either archaic or "wrong".

And all of this is wrapped in a massive amount of paternalism that originates with Gnome and has spread across the larger Linux userspace via Freedesktop.

On top of this, the core devs of systemd is backed by Red Hat's "deep" pockets. And something shifted within RH some years ago and now they seem to have a unstated goal of controlling the Linux ecosystem by generating "code churn".

Meaning that they get to say the direction of the ecosystem with tactic not that dissimilar from Microsoft's Embrace Extend Extinguish.

They have key personnel embedded within various projects, and keep introducing various changes between releases that they coordinate via company channels. Thus third parties find themselves either chasing the RH dust, or turn the keys over to RH.

Reply

Reader Yuhong Bao on 2017-01-25 at 04:33:

This Jeff Johnson bug that led to a RPM fork is probably worth mentioning: https://bugzilla.redhat.com/show_bug.cgi?id=119185

Reply

Reader Anonymous on 2016-10-03 at 21:46:

It is puzzling that Mr Strauss doesn't appreciate the points you are making - the argument goes back many years. Maybe a picture might help ? The RAID conference proceedings of 2001 has one on page 38...

Reply

Reader Wrzi on 2016-11-10 at 08:57:

Save for Slackware, every mainstream and/or popular distribution accepted systemd. Linus Torvalds himself is only able to - or willing to - grumble about the binary logs and ignores both the users' screams and the rest of the catastrophes systemd creates.

For now, save for RedHat stuff (RHEL, CentOS, Fedora) we can get rid of systemd manually, but for how long? What if the softwares will begin to require systemd? Because of stupidity, or because if they don't support systemd, then their software won't run properly or won't run at all. GNOME3, one of the most used desktop environments already depends on systemd and the trend will continue. (And to make it worse, the package maintainers link systemd as dependency even for programs/libraries actually not depending on systemd, but by this they prevent you to install the package without installing systemd.) Like with PulseAudio. Tons of Linux software depends on that unnecessary and bloated layer over OSS/ALSA. And when most of the Linux software will require systemd, then the entire Linux ecosystem will depend on systemd and we will not have the choice to ditch systemd, but to accept it and let it taking over our systems or moving to some BSD or Solaris (or whatever you can find).

Whatever Poettering creates is fundamentally broken and yet, whatever he creates, will be accepted as a quasi-standard in the Linux world. I do not want to create CT-s here, i am merely stating some facts. But i would not be surprised, when the events i mentioned above shall happen, then RH will close the sources of systemd. And the users will have the illusion, that they have an open source OS and they running open source programs, but in reality, they have a blackbox between the two, what can do anything, what it's creators want.

It seems to me, that whoever who try to oppose systemd is waging a losing war. Linux is lost. It's much less painful, to slowly switch to another UNIX step by step now, than do it instantly when systemd ultimately conquered Linux.

Sorry for pouring this much bullshit here.

Reply

Anonymous on 2016-11-27 at 10:50:

Free/OpenBSD is a great alternative and still blessedly systemd-free.

Reply

Reader DES on 2016-11-27 at 17:09:

To provide a meaningful improvement to security without rewriting in a memory-safe language,

If you sincerely believe that rewriting Apache or any other piece of software in a memory-safe language would make it inherently more secure, I'll have to agree with David's assessment of your inexperience.

Reply

Anonymous on 2016-12-28 at 09:22:

If you sincerely believe that rewriting [...] software in a memory-safe language would make it inherently more secure [...]

(emphasis mine)

Ooh, that's a nice strawman you've got there.

The author wrote:

To provide a meaningful improvement to security without rewriting in a memory-safe language, Apache would need to [...]

Which doesn't state that rewriting is the only thing needed to make it more secure. You're misrepresenting his "rewriting is one possible necessary condition, but I do not state whether it is a sufficient one" to "rewriting is one possible necessary and sufficient condition to making the software safer".

Not sure whether to attribute this to malice or accidental (or emotion-influenced) lack of reading comprehension.

Given the history of tactics employed by pro-systemd (and to be fair, anti-systemd) folks, it's getting harder to give the benefit of doubt. But I (and others, I hope) should strive to keep the discussion free of fame wars and intellectual dishonesty, right?

Reply

Reader Nick on 2017-03-02 at 07:16:

Only another ignorant systemd hater.

Pro tip: More valid arguments, less hate.

Reply

Anonymous on 2017-03-08 at 00:34:

Yup, look at this firefox 52 requires libpulse (pulseaudio) that requires systemd. F4k this shit.

Reply

Anonymous on 2017-03-15 at 21:57:

systemd sucks, only supported by company. microsoft-way not going to work.

Reply

Reader infosec on 2017-06-03 at 14:36:

Uhm... about Apache:

1. "Broad" access? No. A site exists under a directory. Of course the httpd will have access to files it is, uhm... serving.

2. Credentials used by PHP? Nope. Only if you run PHP DSO which you shouldn't. The web server can have zero access to PHP files, because properly configured fastcgi daemon (eg. php-fpm) will get a fastcgi REQUEST sent to by httpd. It is then the PHP itself that finds and opens a file for reading, parsing and execution.

3. Attacker gaining remote execution of httpd will be able to do all those things regardless of an init system. HOWEVER, systemd ALLOWS very simple and easy, all within the one unit file, configuration of seccomp filters, capability dropping, binding to <1000 ports for daemons not running as root, etc... I don't think any other init system does this? Oh, sure, it's not the domain of an init system to do all this? Think again, it exactly is, because init is managing processes.

It's hard to take the rest of your post seriously when you make such huge mistakes.

Reply

Reader mark on 2017-12-04 at 00:19:

This is so typical of the pro-systemd camp.

Not only go they the potty-mouth ad hominem attacks but they also shine with stupidity and lack of understanding of core *nix concepts.

Poettering is especially incompetent here.

The main actor is Red Hat though. They have been paying for systemd after all. I am still surprised how the debian people switched to become dependent on Red Hat.

Reply

Post a Comment

Your comment will be public. To contact me privately, email me. Please keep your comment polite, on-topic, and comprehensible. Your comment may be held for moderation before being published.

(Optional; will be published)

(Optional; will not be published)

(Optional; will be published)

  • Blank lines separate paragraphs.
  • Lines starting with > are indented as block quotes.
  • Lines starting with two spaces are reproduced verbatim (good for code).
  • Text surrounded by *asterisks* is italicized.
  • Text surrounded by `back ticks` is monospaced.
  • URLs are turned into links.
  • Use the Preview button to check your formatting.