Systemd vs. Docker
Please consider subscribing to LWN Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net. |
There were many different presentations at DevConf.cz, the developer conference sponsored by Red Hat in Brno, Czech Republic this year, but containers were the biggest theme of the conference. Most of the presentations were practical, either tutorials showing how to use various container technologies like Kubernetes and Atomic.app, or guided tours of new products like Cockpit.
However, the presentation about containers that was unquestionably the most entertaining was given by Dan Walsh, Red Hat's head of container engineering. He presented on one of the core conflicts in the Linux container world: systemd versus the Docker daemon. This is far from a new issue; it has been brewing since Ubuntu adopted systemd, and CoreOS introduced Rocket, a container system built around systemd.
Systemd vs. Docker
"This is Lennart Poettering," said Walsh, showing a picture. "This is Solomon Hykes", showing another. "Neither one of them is willing to compromise much. And I get to be in the middle between them."
Since Walsh was tasked with getting systemd to work with Docker, he detailed a history of code, personal, and operational conflicts between the two systems. In many ways, it was also a history of patch conflicts between Red Hat and Docker Inc. Poettering is the primary author of systemd and works for Red Hat, while Hykes is a founder and CTO of Docker, Inc.
According to Walsh's presentation, the root cause of the conflict is that the Docker daemon is designed to take over a lot of the functions that systemd also performs for Linux. These include initialization, service activation, security, and logging. "In a lot of ways Docker wants to be systemd," he claimed. "It dreams of being systemd."
The first conflict he detailed was about service initialization and restart. In the systemd model, all of this is controlled by systemd; in the Docker world, it is all controlled by the Docker daemon. For example, services can be defined in systemd unit files as "docker run
" statements to run them as containers, or they can be defined as "autorestart" containers in the Docker daemon. Either approach can work, but mixing them doesn't. The Docker documentation recommends Docker autorestart, except when mixing containerized services with services not in a container; there it recommends systemd or Upstart.
Where this breaks down, however, is when services running as containers
depend on other containerized services. For regular services, systemd has
a feature called sd_notify
that passes messages about when services are ready, so that services that
depend on them can then be started. However, Docker has a client-server
architecture. docker run
and other commands are called in
the client for each user session, but the containers are started and
managed in the Docker daemon (the "server" in this relationship). The client can't send sd_notify status messages because it doesn't actually manage the container service and doesn't know when the services are up, and the daemon can't send them because it wasn't called by the systemd unit file. This resulted in Walsh's team attempting an elaborate workaround to enable sd_notify:
- systemd requests sd_notify from the Docker client
- That client sends an sd_notify message to the Docker daemon
- The daemon sets up a container to do sd_notify
- The daemon gets an sd_notify from the container
- The daemon sends an sd_notify message to the client
- The client sends an sd_notify message to tell systemd that the Docker container is ready
Walsh was unsurprised when the patches to enable this byzantine system were not accepted by the Docker project. sd_notify does work for the Docker daemon itself, so systemd services can depend on the daemon running. But there is still no way to do sd_notify for individual containerized services, so the Docker project still has no reliable way to manage containerized service dependency startup order.
Systemd has a feature called "socket activation", where services start automatically upon receiving a request to a particular network socket. This lets servers support "occasionally needed" services without running them all the time. There used to be support for socket activation of the Docker daemon itself, but the feature was disabled because it interfered with Docker autorestart.
Walsh's team was more interested in socket activation of individual containers. This would have the benefit of eliminating the overhead of "always on" containers. However, the developers realized that they'd have to do something similar to the sd_notify workaround, only they'd be passing around a socket instead of just a message. They didn't even try to implement it.
Linux control groups, or cgroups, let
you define system resource allocations per service, such as CPU, memory,
and I/O limits. Systemd allows defining cgroup limits in the
initialization files, so that you can define resource profiles for services
when they start. With Docker, though, this runs afoul of the client-server
model again. The systemd cgroup settings affect only the client; they do
not affect the daemon process, where the container is actually running.
Instead, each one inherits the cgroup settings of the Docker daemon. Users
can pass cgroup limits by passing
flags to the docker run
statement instead, which
works but does not integrate with the overall administrative policies for
the system.
The only success story Walsh had to relate was regarding logging. Docker logs also didn't work with systemd's journald. Logging of container output was local to each container, which would cause all logs to be automatically erased whenever a container was deleted. This was a major failing in the eyes of security auditors. Docker 1.9 now supports the --log‑driver=journald switch, which logs to journald instead. However, using journald is not the default for Docker containers, so the switch needs to be passed each time.
Systemd inside containers
Walsh also wanted to get systemd working in Fedora, Red Hat Enterprise
Linux (RHEL), and CentOS container base images, partly because many
packages require
the systemctl
utility in order to install correctly. His first
effort was something called "fakesystemd"
that replaced systemctl with a service that satisfied the
systemctl requirement for packages and did nothing else. This turned out to cause problems for users and he soon abandoned it, but not soon enough to prevent it from being released in RHEL 7.0.
In RHEL 7.1, the team added something called "systemd-container", that was a substantially reduced version of systemd. This still caused problems for users who needed full systemd for their software, and Poettering pressured the container team to change it. As of RHEL 7.2, containers have real systemd with decreased dependencies installed so that it can be a little smaller. Walsh's team is working on reducing these dependencies further.
The biggest problem with not having systemd in the container, according to Walsh, is that it goes "back to the days before init scripts." Each image author creates his or her own crazy startup script for the application inside the container, instead of using the startup scripts crafted by the packagers. He showed how easily service initialization is done inside a container that has systemd available, by showing the three-line Dockerfile that is all that is required to create a container running the Apache httpd server:
FROM fedora RUN yum -y install httpd; yum clean all; systemctl enable httpd; CMD [ "/sbin/init" ]
There is a major roadblock to making systemd inside Docker work, though: running a container with systemd inside requires running it with the --privileged flag, which makes it insecure. This is because the Docker daemon requires the "service" application run by the container to always be PID 1. In a container with it, systemd is PID 1 and the application has some other PID, which causes Docker to think the container has failed and shut it down.
Poettering says that PID 1 has special requirements. One of these is killing "zombie" processes that have been abandoned by their calling session. This is a real problem for Docker since the application runs as PID 1 and does not handle the zombie processes. For example, containers running the Oracle database can end up with thousands of zombie processes. Another requirement is writing to syslog, which goes to /dev/null unless you've configured the container to log to journald.
Walsh tried several approaches to make systemd work in non-privileged containers, submitting four different pull requests (7685, 10994, 13525, and 13526) to the Docker project. Each of these pull requests (PRs) was rejected by the Docker maintainers. Arguments around these changes peaked when Jessie Frazelle, a Docker committer, came to DockerCon.EU 2015 with the phrase "I say no to systemd specific PRs" printed on her badge (seen at right).
The future of systemd and containers
The Red Hat container team has also been heavily involved in developing the runC tool of the Open Container Project. That project is the practical output of the Open Container Initiative (OCI), the non-profit council established through the Linux Foundation in 2015 in order to set industry standards for container APIs. The OCI also maintains libcontainer, the library that Docker uses to launch containers. According to Walsh, Docker will eventually need to adopt runC as part of its stack in order to be able to operate on other platforms, particularly Windows.
Using work from runC, Red Hat staff have created a patch set called "oci-hooks" that
adds a lot of the systemd-supporting functionality to Docker. It
makes use of a "hook" that can activate any executables found in a specific
directory between the time the container starts up and when the application is running. Among the things executed by this method is the RegisterMachine hook, which notifies systemd's machinectl
on the host that the container is running. This lets users see all Docker containers, as well as runC containers, using the machinectl
command:
# machinectl MACHINE CLASS SERVICE 9a65036e4a6dc769d0e40fa80871f95a container docker fd493b71a79c2b7913be54a1c9c77f1c container runc 2 machines listed.
The hooks also allow running systemd in non-privileged containers. This PR (17021) was also rejected by the Docker project. Nevertheless, it is being included in the Docker packages that are shipped by Red Hat. So part of the future of Docker and systemd may involve forking Docker.
Walsh also pointed out that cgroups, sd_notify, and socket activation all work out-of-the-box with runC. This is because runC does not use Docker's client-server model; it is just an executable. He does not see the breach between Docker Inc. and Red Hat over systemd healing over in the future. Walsh predicted that Red Hat would probably be moving more toward runC and away from the Docker daemon. According to him, Docker is working on "containerd", its new alternative to systemd, which will take over the functions of the init system.
Given the rapid changes in the Linux container ecosystem in the short time since the Docker project was launched, though, it is almost impossible to predict what the relationship between systemd, Docker, and runC will look like a year from now. Undoubtedly there will be plenty more changes and conflicts to report.
[ Josh Berkus works for Red Hat. ]Index entries for this article | |
---|---|
GuestArticles | Berkus, Josh |
Conference | DevConf.cz/2016 |
(Log in to post comments)
Systemd vs. Docker
Posted Feb 24, 2016 19:57 UTC (Wed) by cyperpunks (subscriber, #39406) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 21:34 UTC (Wed) by job (guest, #670) [Link]
How do you, for example, clean zombie processes then?
Systemd vs. Docker
Posted Feb 24, 2016 22:32 UTC (Wed) by smoogen (subscriber, #97) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 22:46 UTC (Wed) by smurf (subscriber, #17840) [Link]
Thus, please tell us what the craftsman should do differently in order to fix this.
Systemd vs. Docker
Posted Feb 25, 2016 6:58 UTC (Thu) by oranges (guest, #100823) [Link]
You don't patch a service in place, you spin up a new docker instance with your new patched version of the app. Do a green blue failover and then tear the buggy/old container down completely.
That's why the docker system has all the associated autoconfiguration/autodetection/autoregistration tools, so you can effectively manage this ever changing fleet of containers.
The only thing that ever stays permanent in a docker ecosystem is the database.
Systemd vs. Docker
Posted Feb 25, 2016 7:37 UTC (Thu) by job (guest, #670) [Link]
The only people who patches running software in production are very highly available services such as phone switches where you have this large underpinning in place to guarantee no dropped connections. But, again, Docker doesn't remotely touch those use cases.
But that's not the issue here. The issue here is that if you don't run an init in a container as pid 1, who does the work that init normally does? Including, but not limited to, the process maintenance required so you don't end up with zombies. (Google it if you need to fresh up your knowledge about process lifecycles.) It's a bit behind the times if every application re-invents that particular wheel, poorly.
Unless you are advocating the 80s DOS experience of rebooting applications every morning just-in-case, of course. But we've tried that, and quality suffered. Let's not go there again.
Systemd vs. Docker
Posted Feb 25, 2016 17:15 UTC (Thu) by mm7323 (subscriber, #87386) [Link]
PID 1 is the reaper of last resort, and only comes in to tidy zombies when some process with unreaped zombie children exits itself.
I think this 'issue' might be being overstated.
Systemd vs. Docker
Posted Feb 25, 2016 18:58 UTC (Thu) by nybble41 (subscriber, #55106) [Link]
Their *direct* children, yes. Not all the indirect children (descendents) which would normally be reparented to PID 1 when the child process exits. Aside from the resource issues, this could break (admittedly fragile) programs which were not designed to handle child processes they didn't start themselves.
For example, on a normal system if you fork() twice and then call wait() twice, you will wait for both child processes you started—if they fork() in turn, those processes are handled by PID 1. If your program runs as PID 1, however, and the first child to exit left behind a zombie process, then the second wait() will return that zombie and your program does not end up waiting for the second direct child to exit.
Systemd vs. Docker
Posted Feb 27, 2016 14:01 UTC (Sat) by fw (subscriber, #26023) [Link]
(I really don't get why you can't use run a minimal wrapper which runs the actual application as PID 2.)
Systemd vs. Docker
Posted Mar 3, 2016 0:25 UTC (Thu) by helsleym (subscriber, #92730) [Link]
The signal situation is far worse for LD_PRELOAD libraries because basically neither the apps nor the libs were written with the idea you'd patch in a signal handler. Your library has to know the specific (even down to bug-matching) way both the app and the lib(s) coordinate signal usage.
Systemd vs. Docker
Posted Feb 24, 2016 23:09 UTC (Wed) by job (guest, #670) [Link]
How are you supposed to kill the container if you don't catch the signals? Is every application developer required to re-implement init?
Systemd vs. Docker
Posted Feb 25, 2016 16:48 UTC (Thu) by fandingo (guest, #67019) [Link]
Systemd vs. Docker
Posted Feb 26, 2016 20:41 UTC (Fri) by wahern (subscriber, #37304) [Link]
What it does is iteratively read a /proc file which lists all the PIDs in a cgroup. It then walks the list and kill(2)s each PID. This is racy because the PID might have been recycled to a process outside the cgroup between when systemd read the proc file and when it calls kill(2). This is the exact same race you see with traditional PID files, although in this case because systemd is PID 1 the window for a race is smaller the race, and can only happen if the parent process is ignoring SIGCHLD. But if this does happen, things get worse: systemd skips PIDs that it has already killed, so if a PID is recycled into the same cgroup, then cg_kill will loop endlessly.
I don't think Poettering ever claimed that cgroups allowed systemd to fix all PID races. But people assumed this because Poettering _did_ claim that systemd resolved PID-file races. (Important distinction.) But implicit in that characterization was that it only applied if processes were cooperative. systemd does not solve the problem of uncooperative daemons.
cgroups doesn't provide the necessary magic. Specifically, cgroups does not provide a way to atomically broadcast a signal to all members in a cgroup. That's what you would need to fix this problem, and that functionality doesn't exist.
Systemd vs. Docker
Posted Feb 26, 2016 21:09 UTC (Fri) by smurf (subscriber, #17840) [Link]
Another way to fix this race would be to first stop all members of the cgroup in question, then check each PID whether it's still in there. If so, really kill the process, else send SIGCONT. But I agree that a kernel patch would be the best approach to this problem. In fact I wonder why that feature doesn't exist yet.
Systemd vs. Docker
Posted Feb 27, 2016 0:04 UTC (Sat) by wahern (subscriber, #37304) [Link]
The problem with process groups is that forking daemons usually create a new process group, and systemd is making a superficial attempt to handle those uncooperative daemons. I'd guess the idea was that most existing forking daemons don't know anything about cgroups so aren't going to be changing their membership.
As for the kernel maintainers, I never followed the LWN coverage of the systemd debates very closely, but it was my understanding that they were resistant to patches to cgroups which reinvented the wheel of interfaces like process groups. Also, the cgroups data structures are supposedly awkward and inefficient (in part, I assume, because of the nesting semantics) and they were reluctant to allow them to become more deeply embedded in the fundamentals of process management. But maybe I totally misunderstood things.
One possible hack would be to use a seccomp filter to silently ignore attempts to create a new process group. Another might be PID namespaces, though I'm not very familiar with them.
Although, to be clear, IMO all of these options are trying to put lipstick on a pig. I'm not sure systemd _should_ be fixed. I'd rather see all the poorly written software fixed. Teach poorly written daemons to optionally run in the foreground so systemd or any other service manager doesn't have to use hacks to track it. And for daemons that fork new processes, work to improve their correctness, and to not needlessly create new process groups. Basically, _subtract_ code, rather than add hundreds of thousands of lines of new code to the pile.
But things like systemd, Docker, etc, are all the rage these days. Apparently people prefer fixing problems with "full stack" solutions rather than submitting a 5-line diff. Whatever... just keep it all off my lawn :)
Systemd vs. Docker
Posted Feb 27, 2016 1:57 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
Process handles are another way to solve this.
Systemd vs. Docker
Posted Feb 27, 2016 7:00 UTC (Sat) by wahern (subscriber, #37304) [Link]
Process handles are useful for passing off the management of a process. For example, traditional shell-based init.d scripts could acquire a process handle and pass it on to a service manager, in effect obviating the need for a PID file. The service manager wouldn't even need to be PID 1, yet the PID file race issue would be solved just as well (or just as incompletely, depending on your perspective).
It does look like PID namespaces solves this problem. The manual page says that when the PID 1 "init" process in the namespace terminates, all the processes in that namespace are SIGKILL'd. So you could just kill that process without having to bother enumerating the PIDs, perhaps simply by closing its process handle. The problem with that solution--and all the others--is that it's not backwards compatible. Neither the software nor administrators may be expecting the process(es) to be running in a different PID namespace.
Systemd vs. Docker
Posted Feb 27, 2016 7:16 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
Systemd vs. Docker
Posted Feb 27, 2016 8:31 UTC (Sat) by smurf (subscriber, #17840) [Link]
Systemd vs. Docker
Posted Mar 7, 2016 1:41 UTC (Mon) by cg909 (guest, #95647) [Link]
The main problem is that process groups always belong to a session. So every service that spawns user sessions would also need to break out of the process group.
If you use seccomp filters to ignore setpgrp() and setsid() sshd would fail in spectacular ways as all processes in the process group will share the same controlling terminal and so every process spawned by sshd will receive SIGHUP when a session is closed. Also anything spawning a shell might run into problems as shells use process groups to separate tasks.
You'd need "super process groups" which may span multiple sessions and contain multiple process groups.
And this is what cgroups provide.
Systemd vs. Docker
Posted Jun 14, 2016 17:32 UTC (Tue) by davidlee (guest, #109327) [Link]
The "solution" I am using is call supervisord. If I need something controlled from inside the Docker, I do it myself. I note some of the previous comments made derisive comments about the kinds of init scripts folks like me might come up with. So what? I don't need their permission, nor do I need their acceptance. With four decades of script writing, I think I can write one that will do the job.
Yes, I could use systemd for those applications that install scripts it will use. I like the three-line example for Apache. But it was quite unique in that Apache installs SO MUCH STUFF that the example works. Perhaps Nagios might also work. Or Splunk. Or a myriad of other major applications which are mature enough to do so.
One of my recent docker containers was an interface with Dropbox. Nope, no three-liner there. It required too much to configure and set up. Actually, virtually every Docker container I have designed has required setup and configuration -- and, thank you very much, a carefully crafted startup script.
If I have a docker which needs to manage internal processes, I'll stick with solutions like supervisord (there are more options, but this is the one I have settled on). I'd rather use that on the few docker containers I need it than have the weight of systemd in every single docker container I generate. Imagine a busybox container with systemd running...
Systemd vs. Docker
Posted Feb 27, 2016 12:20 UTC (Sat) by paulj (subscriber, #341) [Link]
/me very interested to hear of rebuttals.
Systemd vs. Docker
Posted Feb 27, 2016 14:14 UTC (Sat) by smurf (subscriber, #17840) [Link]
However, it is still *way* better than anything any other init system is doing. Also, this method is able to identify and kill all processes which belong to the current task (again, assuming they're not maliciously forking around), which again is way better than anything else out there.
Systemd vs. Docker
Posted Feb 27, 2016 18:52 UTC (Sat) by cortana (subscriber, #24596) [Link]
Systemd vs. Docker
Posted Feb 27, 2016 23:42 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
Forkbombs are more interesting - you CAN cause PID starvation by launching a forkbomb in an unconfined cgroup. https://www.kernel.org/doc/Documentation/cgroup-v1/pids.txt can help against it.
Another problematic case are PID races. SIGSTOP+SIGKILL does the job reliably, SIGSTOP can't be ignored and also forces the process to stick around.
Systemd vs. Docker
Posted Feb 28, 2016 3:43 UTC (Sun) by mchapman (subscriber, #66589) [Link]
It might be possible for two or more cooperating processes to circumvent this by continually SIGCONTing each other, forking new processes along the way. cortana's suggestion of using the freezer controller seems like a better approach.
Systemd vs. Docker
Posted Feb 28, 2016 6:44 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]
It appears that process handles or PID namespaces is the only reliable way.
Systemd vs. Docker
Posted Feb 28, 2016 20:21 UTC (Sun) by justincormack (subscriber, #70439) [Link]
Systemd vs. Docker
Posted Mar 1, 2016 12:14 UTC (Tue) by nix (subscriber, #2304) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 10:16 UTC (Thu) by federico3 (guest, #101963) [Link]
Please tell us this was a joke.
> However that is the failure of the craftsman not the tool.
A tool that works only under very unrealistic conditions (running only short-lived idempotent microservices) might be flawed.
Systemd vs. Docker
Posted Feb 25, 2016 11:25 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
Why? That's a correct way to design a service.
If you're building fault-tolerant systems then you should be prepared to survive failures of individual nodes - hardware is not perfect, even if your software is. And if you have a failover mechanism in place, then why not just use it?
Obviously, you need to keep persistent state somewhere, but that can often be moved to a few dedicated and specialized nodes/containers. It's possible to use Docker with stateful services, just not as easy as with stateless ones.
Systemd vs. Docker
Posted Feb 25, 2016 11:33 UTC (Thu) by smurf (subscriber, #17840) [Link]
Running a process not designed for running as PID-1 in that role does not work. Yet, Docker does that. Therefore Docker has a (serious IMHO) design bug.
This is not a question of keeping persistent state.
Systemd vs. Docker
Posted Feb 25, 2016 11:54 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
What would help, is systemd acquiring a notion of externally-managed services. It'll help to declare dependencies between systemd units and Docker instances.
Systemd vs. Docker
Posted Feb 25, 2016 17:45 UTC (Thu) by rahvin (guest, #16953) [Link]
What would help, is systemd acquiring a notion of externally-managed services.According to the article systemd is already capable of that through it's messaging and socket activation protocols. I see no reason you couldn't run systemd inside the container, in fact I believe that's the significant difference in opinion between Docker and Redhat that's driving this article.
Though docker has the marketshare at this point there are alternatives developing that appear to take security and other things a lot more seriously than the docker folks. Time will tell but I believe ultimately Docker is going to lose this war.
Systemd vs. Docker
Posted Feb 26, 2016 7:23 UTC (Fri) by ibukanov (subscriber, #3942) [Link]
Systemd vs. Docker
Posted Mar 4, 2016 12:16 UTC (Fri) by ofr (guest, #107486) [Link]
http://engineeringblog.yelp.com/2016/01/dumb-init-an-init...
Systemd vs. Docker
Posted Mar 12, 2016 20:08 UTC (Sat) by ploxiln (subscriber, #58395) [Link]
Some services do spawn child processes, which spawn children themselves and then sometimes die without cleaning them up. This does result in zombies. But you can and should avoid such services. If you really need to run one, you can use a 20-some line c utility which will just run the service and reap zombies (linked to in another reply).
Whether systemd has too much functionality for init on a normal desktop is debatable. But for a container? Yes, definitely.
Systemd vs. Docker
Posted Mar 12, 2016 22:17 UTC (Sat) by flussence (subscriber, #85566) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 22:07 UTC (Wed) by Tet (subscriber, #5433) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 2:36 UTC (Thu) by b7j0c (subscriber, #27559) [Link]
if you deploy large, hairy php/python/ruby/perl projects (i.e., lots of source files, lots of libraries that include shared objects, runtime version dependencies)....it can be very helpful to hardwire the configuration in a Dockerfile and keep some of the nastiness from leaking out
if you are a sysadmin, it gives you one interface to deployment
that said, Docker containers are also absurdly bloaty and produce kinda-opaque environments
Systemd vs. Docker
Posted Feb 25, 2016 9:39 UTC (Thu) by daniels (subscriber, #16193) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 20:00 UTC (Wed) by SEJeff (guest, #51588) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 20:44 UTC (Wed) by raven667 (subscriber, #5198) [Link]
> PR (17021) was also rejected by the Docker project.
> Jessie Frazelle, a Docker committer, came to DockerCon.EU 2015 with a the phrase "I say no to systemd specific PRs" printed on her badge
> So part of the future of Docker and systemd may involve forking Docker.
This behavior I think comes from a fundamental mis-alignment between the needs of Docker, Inc. and any other vendor who ships infrastructure and not an edge or leaf application. Docker, Inc. is trying to make their implementation of containers a de-facto standard by writing as many features as quickly as possible (which invariably leads to low quality due to the triple-constraint) to gain marketshare so that they can extract monetary value from becoming a gatekeeper for Docker in particular and containers in general. This is a tried-and-true strategy in the tech sector and what attracts VC money to start-ups, all the talk about first-mover advantage and whatnot is about becoming so big so fast as to crush your competition before they get started. In the Linux world though this means that they can't accept changes which would undermine their position as a gatekeeper, so that means not helping Redhat or CoreOS with systemd integration as they are fundamentally infrastructure competitors who could take the focus of development away from Docker, Inc. The inability of these entities to work together should not be a surprise as they are not neutral vendors. Maybe there will be standards when the dust settles but there will be a lot of in-fighting between now and then, possibly so much that some other vendor comes in and sweeps all these away, like MS/WinNT did to {Sun,HP,IBM,SGI,SCO,etc.}/Unix back in the day.
Systemd vs. Docker
Posted Feb 25, 2016 16:36 UTC (Thu) by sciurus (guest, #58832) [Link]
http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-u...
"""This and other issues have been brought up to Docker developers by network vendors, and are usually closed as "working as intended", even though they make non-Docker third-party systems more difficult to integrate with. Throughout this investigation Docker has made it clear that they’re not very open to ideas that deviate from their current course or that delegate control. This is very worrisome to us, since Kubernetes complements Docker and adds so much functionality, but exists outside of Docker itself."""
Systemd vs. Docker
Posted Feb 25, 2016 17:00 UTC (Thu) by shykes (guest, #107289) [Link]
You can scroll down to see my rebuke of Dan's article: http://lwn.net/Articles/677221/
I didn't muster up the courage to rebuke Google's article but it's just as misleading.
Systemd vs. Docker
Posted Feb 25, 2016 18:48 UTC (Thu) by drag (guest, #31333) [Link]
Redhat, Kubernetes, et al.. wants to use Docker as a component to build off of. They want to use Docker as a major component the OS. It's something to build on top of.. it's not a holistic solution in itself.
Probably what needs to happen at this point is for systemd more fully integrate rkt and make containers a first class 'service type', and then work on compatibility with docker containers and not so much docker itself. That way it becomes just another commodity-generic service the OS offers natively.
This really seems to be what they want. Docker's goals seem to be different.
Rebuke
Posted Mar 2, 2016 9:18 UTC (Wed) by Cato (guest, #7643) [Link]
Rebuke is often used to mean sharp criticism of a person or act.
Rebuke
Posted Mar 3, 2016 16:12 UTC (Thu) by shykes (guest, #107289) [Link]
Unfortunately I can't edit my comments.
Systemd vs. Docker
Posted Feb 24, 2016 21:13 UTC (Wed) by jejb (subscriber, #6654) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 21:31 UTC (Wed) by drag (guest, #31333) [Link]
It's pretty much the 'PHP' of infra.
It's Redhat's job to provide the software and support people want. Sometimes that results in less then ideal solutions from a engineering standpoint.
Systemd vs. Docker
Posted Feb 24, 2016 22:38 UTC (Wed) by sionescu (subscriber, #59410) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 14:49 UTC (Thu) by drag (guest, #31333) [Link]
But please take the time to understand WHY docker is popular. If you are of the opinion 'docker got here first, therefore it's more popular'... you are going to miss a hell of a lot.
Systemd vs. Docker
Posted Feb 24, 2016 21:17 UTC (Wed) by mezcalero (subscriber, #45103) [Link]
But yeah, a pity all that. I'd actually be happy if they'd actually talk to us from time to time. I mean, they are a major consumer of our APIs (or at least should be), but they never contacted us in any way. No question on the ML, no filed bug, no nothing. Just silence, and apparently badges like the above.
Lennart
Systemd vs. Docker
Posted Feb 24, 2016 21:36 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]
How about those patches for musl libc support?
supporting musl libc
Posted Feb 24, 2016 23:09 UTC (Wed) by smurf (subscriber, #17840) [Link]
Disclaimer: Tell me if I missed anything.
supporting musl libc
Posted Mar 4, 2016 16:27 UTC (Fri) by nwmcsween (guest, #62367) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 7:55 UTC (Thu) by nhippi (guest, #34640) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 11:40 UTC (Thu) by smurf (subscriber, #17840) [Link]
http://searchservervirtualization.techtarget.com/feature/...
"Docker was released as an open source project by dotCloud, a platform as a service company, in 2013."
So how should that be the systemd people's responsibility? They were here first.
It is, or seems to be, the Docker people who refuse patches which improve co-existence with systemd.
Systemd vs. Docker
Posted Feb 25, 2016 12:42 UTC (Thu) by mezcalero (subscriber, #45103) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 16:45 UTC (Thu) by shykes (guest, #107289) [Link]
This entire article is about requests to modify docker so that systemd can better manage it from the host, and better run inside its ontainers. So it's exactly the opposite of what you're saying.
Systemd vs. Docker
Posted Feb 26, 2016 6:22 UTC (Fri) by edomaur (subscriber, #14520) [Link]
Me, I am somewhat of the mind that Docker must seemlessly integrate with the OS it is running on. You seriously needs to take some time to support not only systemd, but any init infrastructure from every OS you plan to run Docker on. It's a bit of common sense. Otherwise, it's no different than a rootkit.
Systemd vs. Docker
Posted Mar 3, 2016 16:07 UTC (Thu) by shykes (guest, #107289) [Link]
The first problem is when you want to expand the role of systemd beyond the traditional init role, and insist on wrapping every call to Docker with systemd, even for things like dynamic deployment of applications. That requires integrating with Docker in a way that is not natural because it was not designed to be wrapped in that way. The correct answer is simply to not try anything unnatural.
The second problem is running systemd *inside* a container. Because systemd does things in a very unique way, it does not fit easily inside a container. There is a disagreement on whether docker should be patched with unnatural special cases only needed for systemd. Docker maintainers argue that it's already possible to package systemd into a container with minor configuration work, and that it's not sustainable to modify the runtime to fit every quirk of every application it runs. It would soon become a maintenance nightmare. I think this last part of the argument is hard to understand for the systemd community because they are not at all used to systemd being just another process in your application container. Usually it's the other way around: systemd dictates the rules and other applications adapt.
Systemd vs. Docker
Posted Mar 3, 2016 21:35 UTC (Thu) by Tet (subscriber, #5433) [Link]
Many people use Docker on systemd-based machines and it works perfectly fine.I'm afraid this is simply not true in my experience. I gave up on using Docker specifically because I have better things to do with my life than to try and beat it into shape to do what I want to do when it seems to go out of my way to make that difficult. Amongst other things, I want containers to be started by the native init system (which in my case is systemd), like any other service on my machine. With full dependencies between other things that happen to be running on that machine. If that works "perfectly fine" with Docker, then I'm either blind and stupid or the means of doing so is undocumented, undiscoverable and unintuitive.
Systemd vs. Docker
Posted Mar 4, 2016 2:00 UTC (Fri) by raven667 (subscriber, #5198) [Link]
The fact that you perceive this from an authoritarian perspective, where one part of the system dictates to subordinate parts and you perceive systemd in a container as a priority inversion, is part of the disconnect, you can't see that different parts of the system, different groups of developers, work together to a common goal without a master/slave, dictator/subject relationship.
Systemd vs. Docker
Posted Mar 4, 2016 14:18 UTC (Fri) by evinco (guest, #89291) [Link]
This is a strange claim; the systemd developers run it in a container all the time (via `systemd-nspawn --boot`) to test it, and it works perfectly well. rkt works similarly.
(By "container" here I'm referring to the most common technical conception, of Linux chroot+cgroups+namespaces. Maybe you have a different definition.)
Systemd vs. Docker
Posted Mar 6, 2016 10:01 UTC (Sun) by hitmark (guest, #34609) [Link]
Also, last time i checked, rkt is specifically built on top of systemd.
Systemd vs. Docker
Posted Mar 10, 2016 9:59 UTC (Thu) by evinco (guest, #89291) [Link]
And yeah, the default rkt stage1 implementation uses `systemd-nspawn` internally.
Systemd vs. Docker
Posted Feb 25, 2016 9:13 UTC (Thu) by shykes (guest, #107289) [Link]
Happy hacking
Systemd vs. Docker
Posted Feb 25, 2016 12:34 UTC (Thu) by johannbg (guest, #65743) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 15:44 UTC (Thu) by shykes (guest, #107289) [Link]
Systemd vs. Docker
Posted Feb 26, 2016 9:20 UTC (Fri) by mildred593 (guest, #107325) [Link]
Systemd vs. Docker
Posted Mar 3, 2016 16:10 UTC (Thu) by shykes (guest, #107289) [Link]
Systemd vs. Docker
Posted Mar 4, 2016 2:03 UTC (Fri) by raven667 (subscriber, #5198) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 22:19 UTC (Wed) by dberkholz (guest, #23346) [Link]
Systemd vs. Docker
Posted Feb 24, 2016 22:20 UTC (Wed) by jberkus (guest, #55561) [Link]
Oh, sorry! We tagged my employer in the prior Devconf.cz post. I work for Red Hat.
Systemd vs. Docker
Posted Feb 25, 2016 4:31 UTC (Thu) by pabs (subscriber, #43278) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 7:46 UTC (Thu) by jcc1 (guest, #107291) [Link]
Walsh also wanted to get systemd working in Fedora, Red Hat Enterprise Linux (RHEL), and CentOS container base images, partly because many packages require the systemctl utility in order to install correctly. ... This still caused problems for users who needed full systemd for their software.This strikes me as the root of the problem. It seems like Docker does (or tries to do) all the management necessary for the container. So why is systemd necessary, within this specific context? And if the answer is "because systemd is required in an RPM" or "because applications have become dependent on systemd to function and are no longer portable", then it seems like that validates the many, many concerns that have been had with the project.
The biggest problem with not having systemd in the container, according to Walsh, is that it goes "back to the days before init scripts." ... He showed how easily service initialization is done inside a container that has systemd available, by showing the three-line Dockerfile that is all that is required to create a container running the Apache httpd server:How exactly is this a feature, above and beyond the equivalent non-systemd setup of "chkconfig httpd on"? The "days before init scripts" were 30 years ago... Just use an init script here. Heck, it even requires fewer characters to type.FROM fedora RUN yum -y install httpd; yum clean all; systemctl enable httpd; CMD [ "/sbin/init" ]
Systemd vs. Docker
Posted Feb 26, 2016 7:32 UTC (Fri) by ibukanov (subscriber, #3942) [Link]
Systemd vs. Docker
Posted Feb 26, 2016 22:30 UTC (Fri) by flussence (subscriber, #85566) [Link]
And once you're aware of that background, it seems all the more bizarre to try and transplant the workaround (with all its inherent downsides) into a systemd *or* docker environment, where neither side of the original problem it was supposed to solve exists!
Note that there is nothing wrong with sysvinit itself, in fact it's a perfectly serviceable PID 1 in a controlled environment like a container, where it's not buried away under a quarter-century of accumulated hacks.
Systemd vs. Docker
Posted Feb 25, 2016 8:54 UTC (Thu) by shykes (guest, #107289) [Link]
> "This is Lennart Poettering," said Walsh, showing a picture. "This is Solomon Hykes", showing another. "Neither one of them is willing to compromise much. And I get to be in the middle between them."
I can't speak for Lennart, but this has nothing to do with me personally. I haven't been involved in the day-to-day of the project for over a year. There is a fantastic group of maintainers, from all over the industry, who have done a great job shephearding the project.
> In many ways, it was also a history of patch conflicts between Red Hat and Docker Inc.
This is misleading. There is a history of Red Hat sending patches, and the community of Docker maintainers (many of whom aren't Docker employees) rejecting some of them for legitimate technical and usability reasons.
> [Dan Walsh]According to Walsh's presentation, the root cause of the conflict is that the Docker daemon is designed to take over a lot of the functions that systemd also performs for Linux. These include initialization, service activation, security, and logging. "In a lot of ways Docker wants to be systemd," he claimed. "It dreams of being systemd."
Let's set aside the condescending tone. This is incorrect. Docker is designed to run applications encapsulated in containers. My understanding is that systemd is designed to boot and configure machines, and spawn system services.
Docker definitely cannot configure and boot a machine. I guess you *could* use it to spawn system services (and some people do), but it's definitely not the main use case, and we don't particulary recommend it.
> [... several paragraph explaining that the client-server model of Docker makes it hard for systemd to wrap it ...]
That is correct. Docker is a daemon, and was not designed to be wrapped by another daemon like systemd. Wrapping the docker client is indeed a hack, and we don't recommend it.
I don't understand why this is controversial. What design principle does Docker break by being a daemon and using a client-server architecture?
> Walsh tried several approaches to make systemd work in non-privileged containers, submitting four different pull requests (7685, 10994, 13525, and 13526) to the Docker project. Each of these pull requests (PRs) was rejected by the Docker maintainers.
Here are the links to those 4 pull requests. Please take a moment to read through the discussions.
https://github.com/docker/docker/pull/7685
https://github.com/docker/docker/pull/10994
https://github.com/docker/docker/pull/13525
https://github.com/docker/docker/pull/13526
As you can see, the maintainers gave each pull request serious attention, asked relevant and detailed questions, entertained detailed discussion with the wider community, and took the time to explain why they were rejecting the patches. They remained polite, respectful and professional the whole time.
> Arguments around these changes peaked when Jessie Frazelle, a Docker committer, came to DockerCon.EU 2015 with a the phrase "I say no to systemd specific PRs" printed on her badge (seen at right).
It's true that Red Hat contributions have become a recurring joke among Docker maintainers, due to their reputation of ignoring feedback and showing contempt for the principles of the project, especially its focus on distro-independence. If you build a Docker image, it's important that it works the same way on any distro. Since the business of Red Hat is based on end-to-end platform lock-in, most of their patches focus on breaking that portability in some way. Invariably those patches get refused with clear and respectful feedback. Invariably the feedback is ignored and a similar patch re-opened the next day. It's a weird war of attrition, and I can't blame Jess for being tired of it.
> The Red Hat container team has also been heavily involved in developing the runC tool of the Open Container Project. That project is the practical output of the Open Container Initiative (OCI), the non-profit council established through the Linux Foundation in 2015 in order to set industry standards for container APIs. The OCI also maintains libcontainer, the library that Docker uses to launch containers. According to Walsh, Docker will eventually need to adopt runC as part of its stack in order to be able to operate on other platforms, particularly Windows.
This is really misleading. Dan omits the following facts:
- runC and libcontainer were both created by Docker, specifically Michael Crosby
- the Open Container Project was created by Docker, which donated runC in the process
- runC was created specifically to address Dan's concern with Docker's client-server architecture. Now instead of wrapping the docker client (which as discussed earlier is a hack), systemd can wrap runC
- Of course Docker is going to adopt runC! That's the reason it was developed in the first place. The switch to runC is scheduled for the next Docker release.
> Walsh predicted that Red Hat would probably be moving more toward runC and away from the Docker daemon
Yes please! That's our recommendation and the main reason we developed runC in the first place! So, you're welcome I guess?
> According to him, Docker is working on "containerd", its new alternative to systemd, which will take over the functions of the init system.
This is plain wrong. Containerd is not an init system! It's a lightweight daemon which does one thing only: run OCI containers with runc. It's the next step in making the Docker daemon more modular. This is stated plainly on https://containerd.tools
Systemd vs. Docker
Posted Feb 25, 2016 10:48 UTC (Thu) by pabs (subscriber, #43278) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 12:52 UTC (Thu) by zuki (subscriber, #41808) [Link]
The first one shows the attitude that was described in the article quite nicely:
> https://github.com/docker/docker/pull/7685
Dan comes in with a small patch to add *one standard environment variable*, there is a request to add documentation, there is a request to add tests, comments are positive, and then comes the killer counter-arguments: it's a systemd interface, so it's important no to support it, so maybe instead of $container_uuid it should be $container_id, or just $cid, or maybe just override the hostname, let's implement a fully generic mechanism, etc.
All those counter-ideas are based on the NIH rejection of something that's been standarized elsewhere and it's important for the consumers that the interface is supported as-is, not with a different name, not in slightly incompatible format.
Systemd vs. Docker
Posted Feb 25, 2016 14:20 UTC (Thu) by cpuguy83 (guest, #107303) [Link]
Disclaimer, I'm a maintainer on the Docker engine product and engineer at Docker Inc.
Systemd vs. Docker
Posted Feb 25, 2016 15:40 UTC (Thu) by zuki (subscriber, #41808) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 16:05 UTC (Thu) by shykes (guest, #107289) [Link]
You are entitled to your opinion. The maintainers had a different opinion, which is that not all docker interface consumers are systemd, not all machines running docker are running systemd, and not all containers run systemd. Consequently docker has to take into account all the other consumers of its interface who have no desire of pulling in the systemd universe as a dependency.
In any case at least we can agree this was a civilized debate over design and engineering, with the people in charge making the best possible call.
Systemd vs. Docker
Posted Feb 25, 2016 16:15 UTC (Thu) by zuki (subscriber, #41808) [Link]
>In any case at least we can agree this was a civilized debate over design and engineering,
> with the people in charge making the best possible call.
I agree that the debate was civilized, that's good. But I don't think that the decision was "the best possible call" in any way. I'm arguing, and I think it's pretty transparent if you read the bug thread, that there's little technical disagreement, and the main reason to reject the patches is that they'd make systemd easier to use in docker containers.
Systemd vs. Docker
Posted Feb 25, 2016 16:38 UTC (Thu) by shykes (guest, #107289) [Link]
Here's the last comment by a maintainer: 'Although the PR initially started about "exposing" container ID, the requirements have changed along the way to supporting the systemd journal. We believe with this new information we should take a step back and think about the requirements to support the systemd journal with Docker containers.
We're closing the PR as it won't be merged as it is, but discussions can continue here.'
Seems very reasonable to me.
Before the scope was expanded, there were plenty suggestions to improve the patch in a way that would allow merging. To my knkwledge none of the suggestions were picked up by Dan.
Systemd vs. Docker
Posted Feb 25, 2016 18:00 UTC (Thu) by rahvin (guest, #16953) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 18:40 UTC (Thu) by shykes (guest, #107289) [Link]
And they have done exactly that. Now docker supports systemd journaling, which as it turns out was the initial requirement.
> Exposing a UUID should not be controversial.
Are you sure you read the technical discussion? It provides very specific feedback on how the patch could be modified to be accepted, and why. If you want to carry the suggested modifications and re-submit the patch, it will get merged. The only controversy is the one created here.
Systemd vs. Docker
Posted Feb 26, 2016 2:13 UTC (Fri) by drag (guest, #31333) [Link]
You read it wrong then.
There were people rejecting the idea simply because they disliked the idea of having special support for systemd, but the bug report evolved to a discussion about supporting journald properly and thus was outside the scope of the original patch. Keeping the bug report alive after that didn't make any sense.
Systemd vs. Docker
Posted Mar 3, 2016 16:31 UTC (Thu) by mmulvaney (guest, #107463) [Link]
https://github.com/docker/docker/pull/7685#issuecomment-6...
Now the PR isn't just about adding an environment variable, it is about setting up /var/log/journal and whatever else all that code is doing.
Systemd vs. Docker
Posted Feb 25, 2016 16:20 UTC (Thu) by jberkus (guest, #55561) [Link]
Hey, thanks for responding! Appreciate getting your perspective on this.
Systemd vs. Docker
Posted Feb 25, 2016 17:44 UTC (Thu) by niner (subscriber, #26151) [Link]
This reasoning with which pull request 17021 [1] was closed sounds has nothing to do with technical reasons and seems to be mostly about Docker Inc.'s marketing requirements. That in itself is still ok I guess, companies need to live after all. But please at least be honest about it. Otherwise you're just showing contempt for contributors.
Systemd vs. Docker
Posted Feb 25, 2016 17:51 UTC (Thu) by shykes (guest, #107289) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 18:01 UTC (Thu) by niner (subscriber, #26151) [Link]
Systemd vs. Docker
Posted Feb 25, 2016 18:26 UTC (Thu) by shykes (guest, #107289) [Link]
> It's not about the user, because a feature that can be misused is still better than not having the feature at all
That's not how it works. Just because you disagree with someone's opinion of what's best for the user doesn't make it "not about the user".
You are talking about someone who has reviewed literally THOUSANDS of patches and worked very hard to earn the respect lf Docker users and contributors. You are entitled to disagreeing with her. But who are you, exactly, to disrespect her motivations and speak on behalf of the user with more authority than she does?
Systemd vs. Docker
Posted Feb 25, 2016 18:51 UTC (Thu) by niner (subscriber, #26151) [Link]
Please note that I at least tried to write about "what it sounds like" in my original post, even if that obviously became victim to poor editing of the sentence. I used definitive expressions for the sake of clarity after you asked me to explain. As English is not my native language, that seemed like a safer way to get the reasoning across.
Maybe she can chime in and enlighten us on the real meaning of her words?
Systemd vs. Docker
Posted Feb 25, 2016 18:34 UTC (Thu) by fandingo (guest, #67019) [Link]
It breaks the UNIX principle that *something* needs to reap unowned children and send SIGCHLD to living parents. Due to the way process hierarchies work and Docker's running the app/entrypoint as PID 1 in the namespace, it's broken, unless that app/entrypoint brings its own reaper functionality, which is outside the purview of applications. Really, really stupid oversight by Docker, although typical of their mindset of only contemplating the simple and easy.
> It's true that Red Hat contributions have become a recurring joke among Docker maintainers, due to their reputation of ignoring feedback and showing contempt for the principles of the project,
I think that has far more to do with RH actually understanding operating systems and Docker only caring about its niche. Docker plays fast and loose with everything, and it's no wonder that there's a lot of friction when people who have well-designed systems try to work with something crafted from soda crackers and spaghetti.
> https://github.com/docker/docker/pull/7685
> https://github.com/docker/docker/pull/10994
> https://github.com/docker/docker/pull/13525
> https://github.com/docker/docker/pull/13526
Yes, some of the PRs are a little wacky, but my conclusion after reading through the entire discussions was that the Docker devs are obstinate, and don't understand the issues they pontificate about, especially @jfrazelle. I get the feeling that Docker devs feel like every other tool, even for unrelated tasks, is an attack on Docker and somehow adding any sort of integration to foundational tools on modern Linux is heretical.
> Since the business of Red Hat is based on end-to-end platform lock-in, most of their patches focus on breaking that portability in some way. Invariably those patches get refused with clear and respectful feedback. Invariably the feedback is ignored and a similar patch re-opened the next day. It's a weird war of attrition, and I can't blame Jess for being tired of it.
Wow, just wow. It's difficult to take anyone in the Docker project seriously when they spout nonsense like this, although I suppose the quality of their product already says plenty. It's Docker that tries to lock people in and goes out of its way not to cooperate with other tools and projects.
>> According to him, Docker is working on "containerd", its new alternative to systemd, which will take over the functions of the init system.
> This is plain wrong.
Nope, it's just your reading comprehension. "Functions" in this sense doesn't mean all that the init system can do. Instead, it means the limited functionality that it's being used in containers.
Systemd vs. Docker
Posted Feb 25, 2016 19:38 UTC (Thu) by shykes (guest, #107289) [Link]
> > What design principle does Docker break by being a daemon and using a client-server architecture?
> It breaks the UNIX principle that *something* needs to reap unowned children and send SIGCHLD to living parents. Due to the way process hierarchies work and Docker's running the app/entrypoint as PID 1 in the namespace, it's broken, unless that app/entrypoint brings its own reaper functionality, which is outside the purview of applications.
This is completely unrelated to client-server architecture, so you're not answering my question, but I'll address your comment anyway. Requiring that the image provide the pid1 command was a conscious tradeoff.
On the one hand, docker could bundle its own lightweight init to reap orphans as you describe. That would work perfectly for 90% of applications because, as you point out, app developers are typically not concerned with this functionality at all. But this would make life very, very difficult for the remaining 10% which want to bundle their own init process in the image. At the top of that list, of course, is the systemd community which is adamant that the only good way to run any container is with systemd inside (see the article above). Although I personally disagree with that opinion, the whole point of Docker was to allow running anything inside a container, including a custom init.
So the other option was to run the image-specified command as pid1. This solves the problem of bringing your own custom init (although as we later found out, it takes a LOT more to make systemd happy inside a container...). It does introduce a few issues for the 90% of applications which don't require a custom init: first, an application which forks huge numbers of short-lived processes may eventually cause trouble on the system by leaving too may unreaped orphans. second, most applications only handle a minimal set of signals, so you can't communicate with them with the full spectrum of signals typically handled by an init process. The most typical example of this is that containers running a shell script as an entrypoint can only be terminated with SIGKILL, which means that 'docker stop' will hang for a few seconds of grace period. In practice this has proven to be a very minor issue: either users aren't affected at all, or they care enough and simply add an init process to their images, which solves the problem.
We could change the behavior now to insert a lightweight pid1 by default, and allow a custom field to request a custom pid1. But that would break many existing users who rely on the current behavior. Keep in mind that Docker was adopted incredibly quickly, so interface stability became an issue before we had much time to work out these issues.
I hope this helps understand our technical choices better.
> Really, really stupid oversight by Docker, although typical of their mindset of only contemplating the simple and easy.
A strong choice of words. If you have evidence to back your claim that we "only contemplate the simple and easy", feel free to share it, it would give me an opportunity to rebuke. As it stands there is really nothing more I can answer.
> I think that has far more to do with RH actually understanding operating systems and Docker only caring about its niche. Docker plays fast and loose with everything, and it's no wonder that there's a lot of friction when people who have well-designed systems try to work with something crafted from soda crackers and spaghetti.
Again, a lot of strong opinions, but no evidence to back it up.
> > Since the business of Red Hat is based on end-to-end platform lock-in, most of their patches focus on breaking that portability in some way. Invariably those patches get refused with clear and respectful feedback. Invariably the feedback is ignored and a similar patch re-opened the next day. It's a weird war of attrition, and I can't blame Jess for being tired of it.
> Wow, just wow. It's difficult to take anyone in the Docker project seriously when they spout nonsense like this
I suppose what you meant to say was "I disagree"? Anyway, here is an article which describes the strategy in more detail https://gigaom.com/2014/05/13/is-red-hat-the-new-oracle/
Selected quote: "Red Hat — which makes its money selling support and maintenance for its open-source products — [will] refuse to support of users of Red Hat Enterprise Linux who also run non-Red Hat versions of OpenStack. [...] that could ding OpenStack’s promise of no-vendor-lock-in."
> although I suppose the quality of their product already says plenty.
More strong opinions without evidence to back it up.
> It's Docker that tries to lock people in and goes out of its way not to cooperate with other tools and projects.
I'm not sure you understand what "lock-in" means... Do you have any evidence whatsoever that Docker - an open-source tool whose primary feature is portability and independence of the operating system and hardware - is contributing to lock-in?
Same question on going out of our way not to cooperate. What is your evidence? I wish you at least made the effort to articulate a basic argument? Otherwise it's very difficult to have a constructive conversation.
>>> According to him, Docker is working on "containerd", its new alternative to systemd, which will take over the functions of the init system.
>> This is plain wrong.
> Nope, it's just your reading comprehension. "Functions" in this sense doesn't mean all that the init system can do. Instead, it means the limited functionality that it's being used in containers.
The same article says "they dream of being systemd". In that context I think my interpretation is reasonable.
Systemd vs. Docker
Posted Feb 25, 2016 20:06 UTC (Thu) by roblucid (guest, #48964) [Link]
Suggestions, that software inside containers ought to be modified for the environment after initially saying they shouldn't be able to detect that they're running under Docker, seems especially unfortunate.
I don't get the problem with bundling, you can offer source for a minimal init to copy into containers or have user deploy another (or none). Presumably the customers can actually choose what they run in the containters, right? So why not on the environment setup?
Similarly things like logging, seem like a basic requirement, it needs solving and if you choose to deploy on modern Linux, systemd needs supporting. Saying a flag like "--init==systemd" is compromising the product is ludicrous.
Remember, Dan's start point is stuff deployable for users, and that things like systemctl DO get used. To reinvent another wheel, will be an act of value subtraction, for Docker & the general Linux community. There's enough splintered ways of doing things already.
As it stands, as a sysadmin, we can use a system which simply runs unmodified application services, or ONE market leader that requires a load of special patches and support plus doesn't try to integrate into the system; I KNOW which one I'd choose to try out first.
Systemd vs. Docker
Posted Feb 25, 2016 21:33 UTC (Thu) by fandingo (guest, #67019) [Link]
> At the top of that list, of course, is the systemd community which is adamant that the only good way to run any container is with systemd inside (see the article above).
WOW! You guys are unbelievable. Everything you say is filled with insinuations and disapproving value judgments. You're not casting aspersions; you're hurling them.
> On the one hand, docker could bundle its own lightweight init to reap orphans as you describe.
Why this wasn't considered and completed years ago is beyond me. Again, Docker is interested in the big, shiny, and invariably simple things to implement. When the going gets tough Docker tends to offer nothing, and even worse, rebukes people who are trying to improve things.
> So the other option was to run the image-specified command as pid1. [...]
I pretty much entirely disagree with this entire assessment. You act like making a change to support some sort of init system flag couldn't be accompanied by a few other changes to work around the difficulties. For instance, the kill behavior can be changed and made more cooperative when init flags are in use. But, no, since one change doesn't solve all the problems and create world peace, it's unreasonable.
> A strong choice of words. If you have evidence to back your claim that we "only contemplate the simple and easy", feel free to share it, it would give me an opportunity to rebuke. As it stands there is really nothing more I can answer.
Not implementing reaper functionality. Next question.
> Selected quote: "Red Hat — which makes its money selling support and maintenance for its open-source products — [will] refuse to support of users of Red Hat Enterprise Linux who also run non-Red Hat versions of OpenStack. [...] that could ding OpenStack’s promise of no-vendor-lock-in."
Will Docker Inc support my customized version of Docker under the standard support agreement. This is a silly point. It also ignores how RH's OpenStack is the most upstream friendly.
> I'm not sure you understand what "lock-in" means... Do you have any evidence whatsoever that Docker - an open-source tool whose primary feature is portability and independence of the operating system and hardware - is contributing to lock-in?
I'd say the same to you about Red Hat. Most open source friendly company in the world. For someone constantly complaining about lack of evidence and links, this shows how worthless grabbing the first match off Google is. This article says nothing.
>The same article says "they dream of being systemd". In that context I think my interpretation is reasonable.
What does that have to do with anything?
Systemd vs. Docker
Posted Feb 26, 2016 23:12 UTC (Fri) by flussence (subscriber, #85566) [Link]
"Wow" indeed! If he's that bad, what position does that leave *you* in with your outright attacks on his person?
Oh wait, I think I know: *plonk*
Systemd vs. Docker
Posted Mar 16, 2016 13:56 UTC (Wed) by nye (guest, #51576) [Link]
>Oh wait, I think I know: *plonk*
Wait, what? I think you might have got the two people in this discussion mixed up?
Systemd vs. Docker
Posted Feb 26, 2016 0:57 UTC (Fri) by rahvin (guest, #16953) [Link]
> I suppose what you meant to say was "I disagree"? Anyway, here is an article which describes the strategy in more detail https://gigaom.com/2014/05/13/is-red-hat-the-new-oracle/ Selected quote: "Red Hat — which makes its money selling support and maintenance for its open-source products — [will] refuse to support of users of Red Hat Enterprise Linux who also run non-Red Hat versions of OpenStack. [...] that could ding OpenStack’s promise of no-vendor-lock-in."
Seriously? A speculative news article analysis of a wall street news article about Redhat support.(especially in the new era where these type of attack articles can be purchased) That's your evidence?
I personally don't know very many companies that provide support for software they didn't supply. Why on earth would Redhat supply support for Openstack products that someone downloaded from some unknown source? They couldn't even begin to troubleshoot the issue because they couldn't even duplicate the install conditions and have no idea of the compilation options and the hundred other variables that could be modified or even code patches that could be applied. I haven't used Redhat support in a long time but I don't think they offer support on random software they didn't provide. It is not at all unreasonable that Redhat would require that you use the packages they supply to get support on that software. After all, Redhat doesn't offer support for Windows or Oracle either, you going to ding them for that too?
Systemd vs. Docker
Posted Feb 29, 2016 22:40 UTC (Mon) by shykes (guest, #107289) [Link]
There is no speculation involved: the article is discussing a verifiable fact.
Systemd vs. Docker
Posted Mar 1, 2016 1:04 UTC (Tue) by vonbrand (guest, #4458) [Link]
Once you install anything non-Red Hat, it isn't the Red Hat supplied system anymore. It is reasonable for them to restrict support for the resulting chimaera, particularly if what is being replaced is some fundamental piece. I.e., if I install e.g. git
from source, they won't mind (but steer clear from my repositories); if I compile my own glibc
, they'll balk with good reason. Docker is even more intrusive than the later.
Systemd vs. Docker
Posted Mar 1, 2016 1:07 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
Why? Docker is just another application that happens to utilize cgroups.
Systemd vs. Docker
Posted Mar 1, 2016 10:09 UTC (Tue) by vonbrand (guest, #4458) [Link]
Come on, it provides an environment in which software runs. More comprehensive than glibc
for sure
Systemd vs. Docker
Posted Mar 1, 2016 10:23 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
However, simply running Docker _inside_ RHEL is definitely covered. It's not fundamentally different from running DOS games inside a DOSBox emulator on RHEL.
Systemd vs. Docker
Posted Mar 2, 2016 2:43 UTC (Wed) by fatherlinux (guest, #93873) [Link]
No need to make a fancy distinction...
Systemd vs. Docker
Posted Mar 3, 2016 16:15 UTC (Thu) by shykes (guest, #107289) [Link]
Systemd vs. Docker
Posted Feb 26, 2016 8:37 UTC (Fri) by ibukanov (subscriber, #3942) [Link]
It us unfortunate that Lennart decided to spread FUD with such statements. For logging Docker advocated that application should just log to stdout/stderr and it is the job of the container manager to deal with that, exactly the same what systemd also suggests. For the cases when the application insists on using syslog, one can trivially expose syslog into the container via passing -v /dev/log to docker create/run.
As for reaping zombies, I suspect that most containers just do not need to deal with this as they do not manage arbitrary child processes. When they do, that can be done with a trivial wrapper process of 20 or so lines of C or even with a shell script.
Systemd vs. Docker
Posted Feb 29, 2016 21:06 UTC (Mon) by davidstrauss (guest, #85867) [Link]
You're claiming it's FUD, but you're leaving out a key trait Lennart said any PID 1 should support: a range of signal handling. Do you dispute that the signal-handling expectations for a PID 1 are higher than a standard daemon would manage? And, if so, would you expect daemons to, themselves, implement these signal handlers that service managers like systemd already provide?
When you namespace PIDs and therefore spawn a PID 1 in a container, other tools will expect that PID 1 to behave in a way consistent with PID 1 expectations, including signal handling.
> For logging Docker advocated that application should just log to stdout/stderr and it is the job of the container manager to deal with that, exactly the same what systemd also suggests. For the cases when the application insists on using syslog, one can trivially expose syslog into the container via passing -v /dev/log to docker create/run.
While log capture is something systemd handles for services, it is not a traditional PID 1 requirement.
> As for reaping zombies
Same for this.
Systemd vs. Docker
Posted Mar 1, 2016 9:10 UTC (Tue) by ibukanov (subscriber, #3942) [Link]
I suppose Red Hat guys are perfectly aware of the triviality of the issue. If anything, the problem is that Docker should better document it, pick one of those simple C programs and ensure that it is available in their popular images. Ideally they should also work with kernel guys to have an option to delegate PID-1 responsibilities to another process outside the PID namespace to make the issue the thing of the past.
Systemd vs. Docker
Posted Mar 1, 2016 18:10 UTC (Tue) by jcc1 (guest, #107291) [Link]
A typical use case for Docker is a container running a single process or perhaps the main process and few workers. For this use cases the extra requirements imposed on PID-1 does not matter at all. For more exotic cases signal handling can be done using a simple C program like https://github.com/Yelp/dumb-init/blob/master/dumb-init.c .
I suppose Red Hat guys are perfectly aware of the triviality of the issue.
They're perfectly aware (and, to be fair, it's more the "Fedora guys" and systemd "cabal" than Red Hat per se), but allowing that would reduce the impact of the Embrace, Extend, Extinguish "take no prisoners" strategy with regard to init systems.
A traditional 20-line C program as "/sbin/init" is verboten. Not just an option, explicitly disallowed and unpackaged. Traditional initscripts used for starting on other services can now no longer be packaged in Fedora -- not even as optional packages, not packaged at all. This rapid split is a source of continual frustration for those of us using versions 5 or 6 of RHEL or any of its rebuilds (CentOS or Scientific Linux) and the EPEL packages.
The systemd cabal are being poor winners and trying to eliminate any vestige of anything not meeting their aesthetics. I wish more people were willing to seriously reconsider Fedora's place as the top of the Red Hat ecosystem. IMO, they no longer deserve it... and haven't since the F15 era.
Systemd vs. Docker
Posted Mar 2, 2016 16:09 UTC (Wed) by johannbg (guest, #65743) [Link]
Red Hat owns the Fedora trademark thus Red Hat polices Fedora as in
Red Hat owns and manages the entire project infrastructure from web sites to build system to bug trackers and so on.
Any changes to it will have to be approved by Red Hat and it's employee and more often than not implemented by them as well.
If these are shared infrastructure resources between Fedora and RHEL ( as in not Fedora only like bugzilla ) those changes get rejected.
Red Hat dictates and decides which direction the project is allowed to take both directly and indirectly through it's employee who view and approve any changes that are requested to be made with their RHEL maintenance hat on encase they are involved with that..
Red Hat decides which applications and products that resides within Fedora are "recommended" and considered "default".
etc. etc. etc.
In the end of the day Red Hat controls ( not just sponsors ) Fedora from a to z with the end result and the fact that Fedora is neither free and available to all ( for example it's bound u.s. export administration regulations ) nor a community project, there exist only the sold out lie and the illusion of community participation since for as long as your contribution aligns with Red Hat goals or does not interfere with it and their early product implementation and adoption you are a good community member but if it does not, as in competes with it's solution, be it application,application stack or product you work will not stand on equal term with theirs and they will yank it's corporate slave chain that community members have around their nack so they fall back into their place as a second class Fedora citizen member or even go so far as to threaten to out them from the project so claiming that Fedora is not Red Hat or is just wrong and there exist no "Fedora Guys' capable of changing anything let alone the direction of the project.
And FYI there exist no "Embrace, Extend, Extinguish take no prisoners" strategy within the systemd cabal or the systemd community in general and even if it did, nobody from within the systemd community has forced downstreams to adopt systemd including Fedora.
It was quite foreseeable when systemd became the init system in Fedora that eventually it would not ship any more legacy sysv initscript since it was driven by a community member not Red Hat employee but the plan was of the one that handle the various integration/migration bits of systemd in Fedora ( including the migration for legacy sysv initscript ) to have every component of Fedora migrated before that happened but as luck had it Red Hat employees in FESCo thought they knew better than the guy that had been working on it for four years and when that happened that itch to scratch turned into bruise, with those Red Hat employees that thought that they "knew better" failing miserably in completely that work, which in turned forced them to drop all existing un-migrated component, since those component where starting to becoming hindrance to other ( deprecation ) work to be completed.
That should not come as a surprise since it just followed Red Hat patterns of drive by featuring and half implementing things into the distribution just like Casey Dahlin an Red Hat employee responsible for and ran the feature of switching the init system to upstart where no initscript got migrated to native upstart format.
Systemd vs. Docker
Posted Mar 3, 2016 18:33 UTC (Thu) by jcc1 (guest, #107291) [Link]
There was a huge amount of discussion on the Fedora-devel list over this, but the initial presentation and feature set was "a replacement for init (upstart) that does better parallelization". That alone was controversial enough, which led to it being rejected as a F14 feature (along "if not broke, don't fix it" and KISS lines). It got even more heated once the actual problems in the paradigm started impacting people as F15 started taking shape.
Many of the most active contributors on fedora-devel during that time were NOT Red Hat employees. They were pushing systemd for various reasons, but there had always been (and still remains) a desire there to be on the bleeding edge, and value new and shiny things over reliability. That's the problem. At some point, Fedora's leaders and most of its user base stopped being sysadmins and started being developers
The folks that care about reliability and consistency haven't used Fedora as a server platform for years (unless it's being managed hands-off by the end-user via Plesk or some other control panel) -- they've been using EL or one of its rebuilds. They (we) assumed that Fedora was being reasonable in their choices, not realizing that Fedora had caught the "developer" disease of trying to endlessly fix everything.
In short, there was a mass exodus around that time, caused not only by systemd, but also other needlessly disruptive dumb ideas, like mandated TmpfsOnTmp, and the UsrMove. If "Red Hat" corporate were in control like you say, they would have listened to their revenue-generating customers who'd have said that the churn wasn't anywhere near the breakage. Instead, the Freedesktop.org crowd (i.e., the "systemd cabal") prioritized features they wanted on their laptop over any problems for server users.
I have no idea why you're dismissing the concept of an agenda by LP and others -- it's completely salient in the presentations, discussions, and white papers they've been making. It WASN'T *completely* salient in the Fedora 14/15 days, which is why many people are upset about this all. If they wanted to remake the Linux ecosystem, fine... They should have done what everyone else has had to do and build a distribution that fits their needs, then let it rise or fall according to market acceptance of their ideas. Instead, they snuck it in under the guise of a small improvement to one specific area that no one thought would cause too much trouble -- especially in light of the fact that the transition from "traditional" sysvinit to upstart was mostly invisible and painless. (Primarily because, as you said, Fedora/RHEL was *using* upstart, but wasn't really making use of any of its advanced features. Many thought systemd would be similarly handled -- present if you wanted to use the new hotness as a service manager, but otherwise operationally invisible.)
Furthermore, even agreeing with what you'd said about everything else, LP is an employee of Red Hat too. If RH Corporate controls things, then he's quite part of that.
Systemd vs. Docker
Posted Mar 3, 2016 19:44 UTC (Thu) by pizza (subscriber, #46) [Link]
Red Hat's revenue-generating customers don't use Fedora, they use RHEL.
And I may be wasting my breath, but I'll also point out that systemd is far more useful on servers than laptops.
(And, incidentally, RHEL7, with systemd, has now been out for about a year. RH's revenue-generating customers have yet to leave en masse)
Systemd vs. Docker
Posted Mar 3, 2016 20:04 UTC (Thu) by jcc1 (guest, #107291) [Link]
Red Hat's revenue-generating customers don't use Fedora, they use RHEL.Exactly. Had RHEL (or broader EL community) users known what systemd was going to be allowed to turn into, and how deeply it'd be integrated into the base OS, and had a say, we'd have knifed it before it started.
And I may be wasting my breath, but I'll also point out that systemd is far more useful on servers than laptops.Hardly. Anyone who had a need for a service manager in EL-land had already solved that problem using one of the existing ones out there (daemontools, runit, xinetd, etc.), had scripted their own solution, was using script management for verification, or -- "mu" -- didn't want one running because they relied on the monitoring system to catch something being dead and they wanted to go in an investigate. For servers, 90% of systemd-as-an-init-system solves problems that servers don't have. 90% of systemd-as-everything-else (autofs, crond, ntpd) solves problems that had already been solved. Basically the only thing that systemd does well that was something that needed to be done better is cgroup management. But those who were using it heavily had often already decided on their own solutions. The churn of dealing with systemd wasn't worth that benefit.
(And, incidentally, RHEL7, with systemd, has now been out for about a year. RH's revenue-generating customers have yet to leave en masse)Leave? No. Migrate to RHEL7? Anecdotally, there's a LOT of concern. I know a lot of shops that are staying with EL6 for the time being, and it's not just for the normal reason of "letting things shake out for a year". There are serious concerns with systemd's reliability and the philosophy behind it. The only places that are all-in with systemd from an EL perspective are those that are fully on board with containers. Even then, those who were most in need of stateless solutions (high performance compute clusters) already had solutions for running EL in stateless modes, and deploying / managing their images.
Systemd vs. Docker
Posted Mar 3, 2016 20:50 UTC (Thu) by bronson (subscriber, #4806) [Link]
Not true. You're writing in an omniscient voice but you certainly aren't speaking for me. My team moved to RHEL 7 just for systemd. We're sick of process monitors and their bugs. (it's especially great when you discover your process monitor has a memory leak... good thing there are zillions to choose from.)
So far systemd has been better than I'd hoped. For deploying web services, it's easy to use, mature, and rock solid. Everyone on the team has gotten to know its config -- no need to learn how to work with some guys favorite tool.
Reliable as all hell. No way we'd go back.
> I know a lot of shops that are staying with EL6 for the time being
And I know a lot of shops that are still on 5. Has nothing to do with systemd.
If you were to tone it down a bit, maybe I could understand what you're getting at... Right now, though, it seems like many of your statements are demonstrably false.
Systemd vs. Docker
Posted Mar 3, 2016 21:30 UTC (Thu) by pizza (subscriber, #46) [Link]
Speaking for my employer, they're sticking with EL6 because the vendors of their business-criticial EDA tools haven't certified said tools for use with EL7 yet. Indeed, they are still using EL5 on some systems for similar reasons.
(We're talking about per-seat licensing costs in the upper five figures here. The actual RHEL license price is a rounding error, as is the hardware cost for all but the beefiest of boxes)
Systemd vs. Docker
Posted Mar 4, 2016 2:13 UTC (Fri) by raven667 (subscriber, #5198) [Link]
I still have plenty of EL5 systems as well, for most purposes its totally fine, feature complete. We will migrate in the future because our developers are getting tired of PERL 5.8.8 and would like to be able to use current libraries without trouble, other than that I can't think of any base OS improvement that is substantial, except for systemd, which I like because I'm tired of crappy init scripts or people just running things out of rc.local.
Systemd vs. Docker
Posted Mar 3, 2016 20:14 UTC (Thu) by davidstrauss (guest, #85867) [Link]
This view seems made up to me, and I can speak to why personally. My company runs runs hundreds of servers on Fedora, and I am a systemd committer. Both have been true since about Fedora 18. I have also worked with the primary systemd developers on key issues affecting server infrastructure, including being invited to present on remaining issues at systemd's latest conference [1]. And, in case you're wondering, those remaining issues relate to optimizations at scale, not fundamental design issues with what systemd does.
Moreover, there has been substantial use on the server side by plenty of other people. Some early adopters were shoehorning it into CentOS 6 to use systemd's tools for their own services; one of those companies posted publicly to the Debian mailing list during the init discussions to publicly support moving to systemd. I've also been asked to speak at Facebook on best practices for running services within systemd, which I'm doing later this month [2].
> In short, there was a mass exodus around that time, caused not only by systemd, but also other needlessly disruptive dumb ideas, like mandated TmpfsOnTmp, and the UsrMove.
Mostly false, at least for the cause/effect you're suggesting. Fedora lost substantial users over the F15-F20 period, but it has more than recovered them since (F21 and F22) [3] without reversing those decisions. If those were the decisions rationally driving users away, they would have stayed away. Also, other distributions adopting systemd have not seen any "mass exodus" at the point of their decision, either. So, your claim fails doubly: (1) reversal of effect without alteration of the supposed cause and (2) lack of effect when applying the supposed cause elsewhere.
> Instead, the Freedesktop.org crowd (i.e., the "systemd cabal") prioritized features they wanted on their laptop over any problems for server users.
I'm not even sure how to disprove that systemd developers are in some kind of FreeDesktop.org crowd/cabal.
> Many thought systemd would be similarly handled -- present if you wanted to use the new hotness as a service manager, but otherwise operationally invisible.
Likewise here. You're not providing testable criteria, just other statements that are "not even wrong." [4] How could I falsify your view here? What tests could show that systemd is "operationally invisible" to most users running distributions with it?
[1] https://systemd.events/systemdconf-2015/sessions/challeng...
[2] http://www.meetup.com/South-Bay-Site-Reliability-Engineer...
[3] https://youtu.be/qJ3CozFrEvg?t=29m9s
[4] https://en.wikipedia.org/wiki/Not_even_wrong
Systemd vs. Docker
Posted Mar 3, 2016 23:04 UTC (Thu) by jcc1 (guest, #107291) [Link]
This view seems made up to me, and I can speak to why personally.It's quite possible that we run in different circles, but the view is definitely not "made up". Every EL shop I know of is being more cautious with EL7 than with previous transitions.
Mostly false, at least for the cause/effect you're suggesting. Fedora lost substantial users over the F15-F20 period, but it has more than recovered them since (F21 and F22) [3] without reversing those decisions. If those were the decisions rationally driving users away, they would have stayed away.Although numbers have recovered, that doesn't really speak to whether the then-users stayed away or not. As focus changes, so does audience. I'd suggest that Fedora users who didn't agree with direction things were going went with EL, or perhaps Debian. e.g., from this wonderful thread: https://lists.fedoraproject.org/pipermail/devel/2011-June/152636.html
What tests could show that systemd is "operationally invisible" to most users running distributions with it?Well, perhaps we're equivocating on "user" here, a not-uncommon issue. For some distributions, "users" are people relying on the distribution to control apps. For others, "users" are the system administrators who are deeply involved in reliability engineering and may or may not have specific ways of setting up systems for their company's needs.
I'd submit that neither the systemd-as-init nor the systemd-as-collection-of-tools aspects have been anywhere near as "operationally invisible" to sysadmins as the move to upstart was. Depending on the needs of non-sysadmin users who don't know what the /etc/fstab file is, it may have been more "operationally invisible", except for the opportunity cost of all the other people having to deal with systemd bugs over the years.
Systemd vs. Docker
Posted Mar 4, 2016 21:17 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
Not sure this is a fair comparison. The switch to upstart didn't make anything worse, but I also never noticed anything getting better with it either. With systemd, on the other hand, I've seen bugs, but also *vast* improvements in various places, across the board. Certainly, for me, worth the issues I've hit.
Systemd vs. Docker
Posted Mar 4, 2016 22:30 UTC (Fri) by johannbg (guest, #65743) [Link]
If they would have, it most certainly would never have been ""operationally invisible".
Systemd vs. Docker
Posted Mar 10, 2016 2:17 UTC (Thu) by mathstuf (subscriber, #69389) [Link]
Systemd vs. Docker
Posted Mar 5, 2016 6:48 UTC (Sat) by seyman (subscriber, #1172) [Link]
Did anyone actually move to upstart? Even Ubuntu never used native upstart job files for the vast majority of their services in their distribution, preferring to run them using upstart's compatibility mode.
Systemd vs. Docker
Posted Mar 5, 2016 20:47 UTC (Sat) by raven667 (subscriber, #5198) [Link]
I don't think so, there were some high hopes that upstart would be the next gen init system but clearly the native job files arent sufficient in practice and fixing the deficiencies was not possible without Canonical dropping the copyright assignment requirement, for-profit companies often don't like doing free consulting work for other for-profit companies, not even the original author was allowed to submit patches after taking a job working on ChromeOS.
Systemd vs. Docker
Posted Mar 6, 2016 0:30 UTC (Sun) by anselm (subscriber, #2796) [Link]
The original author of Upstart (who is now working on different projects at Google) is on record as saying that it needed a massive redesign in order to cope with some nasty architectural deficiencies that were discovered in practice. At that point Upstart was basically dead in the water since nobody seemed eager to take on the work, and the Ubuntu people sensibly decided to go along with most other Linux distributions and adopt systemd.
(The fact that the Debian project had just decided to opt for systemd as the default init system on Linux also contributed to this, since if Debian had gone with Upstart then the Ubuntu people would have been able to rely on Debian developers doing much of the work adapting existing packages to native Upstart, instead of having to do all of the work by themselves.)
Systemd vs. Docker
Posted Mar 6, 2016 3:43 UTC (Sun) by raven667 (subscriber, #5198) [Link]
Right, I was just pointing out my opinion on what the largest factor which made it difficult to pick up the maintainership of Upstart was Canonical's insistence on copyright ownership. Many strong developers (or their employers) insist on owning their own copyrights, with few exceptions like for the FSF. If a strong maintainer had stepped forward and had been effective in getting the restructuring done there never would be a systemd. Even if systemd were written in a world with a non-broken Upstart, it wouldn't gain any traction because it wouldn't be enough better to justify the cost of transition.
Systemd vs. Docker
Posted Mar 6, 2016 3:51 UTC (Sun) by vonbrand (guest, #4458) [Link]
Check out the systemd documents on why it's development started. You'll see that the upstart problems were severe enough to require a complete rearchitecture, not just some bug fixes. The whole basic model (define some "events", and trigger them) is fundamentally impossible to scale, you can't decouple things enough. That is a good part of the reason why no native upstart configurations ever materialized. The "refurbished upstart" might just look an awfully lot like systemd. In a sense, systemd is upstart's better replacement.
Systemd vs. Docker
Posted Mar 6, 2016 7:04 UTC (Sun) by smurf (subscriber, #17840) [Link]
Systemd vs. Docker
Posted Mar 6, 2016 15:17 UTC (Sun) by vonbrand (guest, #4458) [Link]
Reachitecture and incrementally. The mind boggles.
Systemd vs. Docker
Posted Mar 6, 2016 15:34 UTC (Sun) by reedstrm (guest, #8467) [Link]
There almost always is a way to do it, if you're willing to spend the resources (time as much as money). Unfortunately, actual rearchitecting (as opposed to new feature development) requires someone with a deep understanding of both the old and new systems, to develop the path forward. And patience to complete, as well as put up with any less than perfect intermediate states along the way. Usually a hard sell in today's go go go faster faster world.
Systemd vs. Docker
Posted Mar 6, 2016 17:13 UTC (Sun) by vonbrand (guest, #4458) [Link]
In this case, the path forward was from SysVinit to something new. Upstart never got any significant hold except as a SysVinit lookalike. And systemd also offers SysVinit compatibility, but the switchover to native systemd was surprisingly fast and smooth, something I believe shows they got it right. Many distribution's users fought it nail and tooth in the ubiquitous flamefests, while their developers just moved over silently.
Systemd vs. Docker
Posted Mar 6, 2016 22:50 UTC (Sun) by smurf (subscriber, #17840) [Link]
In this case, about half of Upstart's events translate reasonably cleanly to dependencies (i.e. instead of job B's start being triggered by "A has started", let B simply depend on A). Deprecate the other events aggressively. Now you can hook dependent jobs to states instead of state transitions without changing anything. Next step, add stuff you couldn't even think about in Upstart ("let D run as soon as both B and C are up", B and C being independent of each other). At last you're in a position to add targets. QED.
That being said, I'd guesstimate that doing this requires at least three times the amount of work on the core system, compared to starting from scratch, and a work-in-progress that's definitely worse than Upstart. Thus, one might reasonably conclude that Upstart's copyright assignment policy was a Good Thing in this case because it prevented the systemd people from getting mired in a prolonged Upstart refactoring that may or may not have succeeded.
Systemd vs. Docker
Posted Mar 7, 2016 2:22 UTC (Mon) by vonbrand (guest, #4458) [Link]
Upstart style events need to be quite a bit more abstract than "A started" to be useful in case the functionality offered by A can be obtained in different ways... and there it breaks down.
Systemd vs. Docker
Posted Mar 8, 2016 5:58 UTC (Tue) by dberkholz (guest, #23346) [Link]
Systemd vs. Docker
Posted Mar 8, 2016 10:23 UTC (Tue) by vonbrand (guest, #4458) [Link]
Fork and scrap was the way forward here, why fork? Besides, upstart in practice was only used as a SysVinit drop-in replacement.
Systemd vs. Docker
Posted Mar 3, 2016 21:38 UTC (Thu) by johannbg (guest, #65743) [Link]
I do believe I'm not misremembering anything but then again this was 5 years ago but as I recall it, the reason systemd got rejected in F14 was lack of proper documentation both upstream ( man pages ) and downstream, though I think I, Rahul Sundaram, Lennart and Matthias Clasen manage to take care of most that before F14 got released, lack of clear packaging guidelines and related upgrade path and hand full of bugs somewhere between 5 - 10 which I think Lennart and Kay had fixed before F14 got released but due to a short timeframe and to much of grey area surrounding this FESCo decided to take the safer route ( defer systemd to the f15 release ) to have those things properly settled out, despite the media hype around that feature ( which the feature process basically is ) that had taken place.
"Many of the most active contributors on fedora-devel during that time were NOT Red Hat employees" I'm not sure what you mean by that but the components that make up the RHEL release has always been owned and maintained by Red Hat employees most of which are smart and thus stay away from getting involved in distribution politics so you wont notice them on -devel. ( It's not like it matters decisions that affect those components are made elsewhere than in Fedora ) and the number of active "core" package maintainers as always remained the same despite significant growth in components in the distribution and since stats ( in no relation to what actually matters ) where thrown at devconf this year I'm pretty sure in twenty sixteen we will see alot of hype surrounding much about nothing regarding those stats from the PLL.
"The folks that care about reliability and consistency haven't used Fedora as a server platform for years (unless it's being managed hands-off by the end-user via Plesk or some other control panel) -- they've been using EL or one of its rebuilds. They (we) assumed that Fedora was being reasonable in their choices, not realizing that Fedora had caught the "developer" disease of trying to endlessly fix everything."
Oh really I know alot of people that have used and still use Fedora as server platform in business and mission critical infrastructure so I'm not getting what you are referring to and the core/baseOS has been pretty well maintained.
( We are not talking about Anaconda/Gnome here which get rewritten every cycle with let's throw it over the wall and see what leaks mentality ).
There was no "mass exodus around that time" no more than was happening with other distribution and happens every cycle for all of them ( distribution hoppers jump between distribution which is the feature most or the most hype is about ) and you certainly cannot directly relate that to systemd, Gnome Shell, X, rewrite and or whatever else was supposed to be going on and if you are claiming that by all means show the stats to back that up.
Systemd vs. Docker
Posted Mar 4, 2016 7:41 UTC (Fri) by smurf (subscriber, #17840) [Link]
I seriously doubt that it's just my experience which shows that, once adapted to, systemd is *way* better at reliability, consistency and whatnot than any alternate system one could name.
Systemd vs. Docker
Posted Mar 4, 2016 11:12 UTC (Fri) by vonbrand (guest, #4458) [Link]
I must second that. Not often, but enough to be annoying, I saw SysVinit (and it's upstart emulation) messing up. To be able to start and stop anything anytime, with no regard to dependencies, meant one had to have a map of running services for all the random machines with specialized tasks in the head, or figure it out on the fly. Not funny.
Systemd vs. Docker
Posted Mar 6, 2016 10:04 UTC (Sun) by hitmark (guest, #34609) [Link]
Systemd vs. Docker
Posted Mar 6, 2016 10:06 UTC (Sun) by hitmark (guest, #34609) [Link]
Oh so very very much this.
Systemd vs. Docker
Posted Mar 6, 2016 12:21 UTC (Sun) by anselm (subscriber, #2796) [Link]
There seems to be this weird notion around that the systemd developers were somehow able to pressurise everybody into adopting systemd against their better judgement. In actual fact, the mainstream Linux distributions came on board exactly because their developers looked at systemd and decided – in some cases after extensive and painful deliberation – that it was an idea worth pursuing, over the available alternatives. Sounds like “market acceptance of their ideas” to me.
Would it have been possible to beef up SysV init or Upstart to do what systemd does? Perhaps, but in spite of a lot of talk about doing this, nobody volunteered to put in the effort – and in the free-software world, working code beats hypothetical yet-to-be-written code any day.
Systemd vs. Docker
Posted Mar 6, 2016 15:16 UTC (Sun) by hummassa (guest, #307) [Link]
Systemd vs. Docker
Posted Feb 28, 2016 11:20 UTC (Sun) by misc (subscriber, #73730) [Link]
My interpretation is that rocket was made precisely to avoid the docker centralized model with a deamon (or at least, that's how I read the announce ( https://coreos.com/blog/rocket/ )). And since even Docker maintainer acknowledge that this is the right way forward (cf the discussion about runc in those comments, runc being the answer from docker to the rocket announce, given the timeline), and since it provides interesting security properties (hardware isolation, signature of the image in a proper decentralized way, etc), I think people should start to play with it.
While a tool like mesos or kubernetes is needed when it come to using docker in production for a large architecture (or just to support stuff like secrets, monitoring, etc that are not well supported out of the box by docker), there is space for using a lighter and more integrated model, and reusing systemd would fullfill the need. Watching the presentation of Viacom on using Kubernetes/Coreos ( https://www.youtube.com/watch?v=OLwzSHBtxhI&list=PLlh... ), the presenter do mention the cultural shock of moving to containers, and being able to use continers outside of the high levels tools seems to be a good way to make people more familliar with containers without making a big overhaul.
Systemd vs. Docker
Posted Mar 8, 2016 11:36 UTC (Tue) by ibukanov (subscriber, #3942) [Link]
Could you clarify this? It is very trivial to deal with passing secrets to Docker with a minimal shell script as the tool provides all the necessary hooks. As for monitoring, support for logging to syslog or journald with custom tags in Docker 1.8 resolved all the monitoring issues for me.
Systemd vs. Docker
Posted Jul 17, 2016 21:55 UTC (Sun) by MarkRiggins (guest, #109800) [Link]
If the goal is to be able to run services, and clean up zombie processes, then maybe this tool that I'm developing could be useful. It's called "dockerfy" and is a simple binary that when used as the main ENTRYPOINT of a docker image, can start services, reap zombies, pre-run initialization commands (before the primary command starts), edit configuration files, and even inject secrets into containers. For example:RUN wget https://github.com/markriggins/dockerfy/releases/download/0.2.4/dockerfy-linux-amd64-0.2.4.tar.gz; \ tar -C /usr/local/bin -xvzf dockerfy-linux-amd64-*tar.gz; \ rm dockerfy-linux-amd64-*tar.gz; ENTRYPOINT dockerfy COMMAND dockerfy \ --secrets-files /secrets/secrets.env \ --template /app/nginx.conf.tmpl:/etc/nginx/nginx.conf \ --wait 'tcp://{{ .Env.MYSQLSERVER }}:{{ .Env.MYSQLPORT }}' --timeout 60s \ --run '/app/bin/migrate_lock' --server='{{ .Env.MYSQLSERVER }}:{{ .Env.MYSQLPORT }}' --password='{{.Secret.MYSQLPASSWORD}}' -- \ --start /app/bin/cache-cleaner-daemon -- \ --stderr /var/log/nginx/error.log \ --reap \ --user nobody -- \ nginx "-g daemon off;"Would do the following: - load secrets from the file /secrets/secrets.env, - create an nginx.conf file from a template, - wait up to 60 seconds for a mysql database to start accepting connections - run a database migration against the mysql database, using a secret password BEFORE starting nginx - start a service named 'cache-cleaner-daemon' - start a go routine to reap zombies - run nginx as user "nobody" - tail the /var/log/nginx/error.log to stderr If the database migration fails, then the container will exit without starting nginx. While nginx is running, if the cache-cleaner-daemon dies, the entire container will shut down so the cloud platform can start up another instance. The nginx.conf.tmpl template can use the full features of Go languages templates to inject environment variables and secrets into the nginx.conf file. A template like this:
server { location / { proxy_pass {{ .Env.PROXY_PASS_URL }}; proxy_set_header Authorization "Basic {{ .Secret.PROXY_PASSWORD }}"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect {{ .Env.PROXY_PASS_URL }} $host; } }could generate an nginx.conf file like this:
server { location / { proxy_pass http://myserver.com; proxy_set_header Authorization "Basic a2luZzppc25ha2Vk"; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect http://myserver.com $host; } }Hope this helps -- You can build from source or use a pre-built binary from my latest github release
Container history
Posted Mar 8, 2020 15:00 UTC (Sun) by JvB (guest, #137643) [Link]