The Emacs dumper dispute
Benefits for LWN subscribers The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today! |
The Emacs editor is, at its core, a C program, but much of the editor's functionality is actually implemented in its special "Elisp" dialect of Lisp. Starting the editor requires loading a great deal of Elisp code and initializing its state, a process that can take a long time. To avoid making users wait for this process, Emacs has long used a scheme whereby the Elisp code is loaded once and a memory image is written to disk; starting Emacs becomes a matter of reading the memory image back in, which is a much faster process. Supporting this "dumping" functionality (also known as "unexec") has never been easy; beyond the technical challenges, it now appears that it may lead to a significant split within the Emacs community.
As covered here in January, the Emacs dumping (and "undumping") mechanism has long depended on some low-level hooks in the GNU C Library's memory allocation subsystem. The Glibc developers would like to modernize and improve this code, improving the library overall but removing the hooks that Emacs depends upon. At the end of the January discussions, the Emacs developers had decided to move to a workaround implementation that allowed the dumper to continue to work in the absence of Glibc support.
"Unstable" is the sort of behavior that users of text editors normally go well out of their way to avoid; it's also the sort of thing that could give vi a definitive advantage in the interminable editor wars. So something clearly needs to be done to make the Emacs dumping facility more stable and, preferably, more maintainable going forward. What that "something" would be is unclear, and the posting of a possible solution appears to have simply muddied the waters further.
That solution comes in the form of the "portable dumper" patch from Daniel Colascione. This patch is not small; it adds over 4,500 lines of code to Emacs and it is not yet complete. Rather than try to capture the state of the C library's memory-allocation subsystem, it simply marshals and saves the set of Elisp objects known to the editor. The file format is designed for performance and, in some settings at least, Emacs can start by simply mapping the file into memory and initializing a set of pointers.
Colascione describes the result this way:
It also, he says, matches the startup performance of the current "unexec" system to within 100ms, and he has not yet had the time to collect a bunch of low-hanging optimization fruit. In other words, it seems like an interesting solution to the problem, but a patch of this size is always going to generate some discussion.
Some of that discussion focused on how this dumper works when address-space layout randomization (ASLR) is in use. Current Emacs binaries must disable ASLR entirely, thus losing the security benefits that ASLR is meant to provide. The new dumper does not require disabling ASLR, but it does contain an optimization that can be applied if the dump file can be successfully mapped at a specific address: most of the data therein can be used directly from the mapped image, without the need to allocate storage for and copy it. That should speed the startup process considerably, at the cost of always mapping the dump image at the same location.
Paul Eggert worried about the potential security implications of losing ASLR protection for the bulk of the editor's data. Colascione responded that, since no part of the data image is marked executable, there is little risk of attackers running code from there. But, as Eggert pointed out, that view overlooks an important detail: that memory is full of Elisp bytecode that is executed in the editor itself, and which can do just about anything an attacker might want. So, if this approach is adopted, the fixed-location mapping might have to be turned off, at least by default.
There is, however, a bigger disagreement involving Zaretskii, who described this work as "a wrong
direction.
" His objection, in short, is that this patch adds a lot
of low-level complexity, implemented in C, that will be a maintenance
burden going forward. That is, he said, a
threat to the future of the project:
It makes sense to put thought into the maintainability of the code base and how it can be evolved to attract more developers. It is not entirely clear, though, that C programmers are actually a dying breed — or that the long-term supply of Elisp developers is more certain. In any case, the Emacs community needs to fix the startup problem; those who oppose the portable dumper solution presumably have something else in mind.
Zaretskii's preferred solution would be to make the Elisp loader faster, to the point that it can be used to read Elisp code directly at startup time. That is a solution that others might like to see as well, but it has one significant shortcoming: no code toward that goal exists, and there are no signs that anybody is working in that area. Colascione's solution, instead, does exist and has an interested developer behind it. In almost any development project, working code and ongoing maintenance carries a lot of weight.
Zaretskii feels strongly enough about this issue that he has threatened to resign as co-maintainer if the portable dumper is adopted. He appears to be nearly alone in this stance, though. Colascione has said repeatedly that he sees no other way to get the required performance. Richard Stallman is guardedly favorable to this solution, noting that it will be far easier to maintain than the current unexec code. John Wiegley, the other Emacs co-maintainer, also favors going with the portable dumper code.
The wind thus appears to be blowing in the direction of adopting the
portable dumper patch. Nobody seems to want to see Zaretskii relinquish
the co-maintainer role (a role he only accepted last July), so, if the portable
dumper is merged, the community can only hope that he will change his
mind. Any large development project will occasionally make decisions that
are opposed by some of its developers, even when those developers are
maintainers. But the venerable Emacs editor will still be there, and will
still have no end of other problems to solve.
(Log in to post comments)
The Emacs dumper dispute: ASLR
Posted Nov 30, 2016 16:51 UTC (Wed) by smurf (subscriber, #17840) [Link]
Oh come on …
The Emacs dumper dispute
Posted Nov 30, 2016 17:25 UTC (Wed) by nix (subscriber, #2304) [Link]
This is just Eli being Eli. One hopes he will change his mind in time. (But... a portable dumper! At last!)
The Emacs dumper dispute
Posted Nov 30, 2016 17:40 UTC (Wed) by josh (subscriber, #17465) [Link]
The Emacs dumper dispute
Posted Nov 30, 2016 17:52 UTC (Wed) by felixfix (subscriber, #242) [Link]
The Emacs dumper dispute
Posted Nov 30, 2016 18:56 UTC (Wed) by simcop2387 (subscriber, #101710) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 7:07 UTC (Thu) by felixfix (subscriber, #242) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 22:52 UTC (Fri) by kleptog (subscriber, #1183) [Link]
The Emacs dumper dispute
Posted Dec 3, 2016 8:58 UTC (Sat) by smurf (subscriber, #17840) [Link]
The main problem with RPython is that you don't have a dumper any more. Emacs would need to use a lot more on-demand loading, instead of packing everything and its kitchen sink into a dump image, to get initial performance up to acceptable levels.
The Emacs dumper dispute
Posted Dec 8, 2016 17:48 UTC (Thu) by nix (subscriber, #2304) [Link]
The problem is disentangling that such that the interpreter can be replaced, without making Emacs into even more of a maintenance nightmare than it already is.
The Emacs dumper dispute
Posted Nov 30, 2016 19:04 UTC (Wed) by jlargentaye (subscriber, #75206) [Link]
https://www.facebook.com/notes/daniel-colascione/buttery-... (Notes are FB's blog format)
It's an entertaining and interesting dive into the insane layers of compatibility Emacs has accumulated over the decades.
The Emacs dumper dispute
Posted Nov 30, 2016 22:16 UTC (Wed) by louie (guest, #3285) [Link]
Emacs double buffering
Posted Nov 30, 2016 23:14 UTC (Wed) by corbet (editor, #1) [Link]
Indeed worth a read. There are often times when I'm amazed that anything we use actually works; this is one of them.Wish I'd seen it when it was fresh, it would have been great QOTW material.
Emacs double buffering
Posted Dec 1, 2016 6:03 UTC (Thu) by AdamW (subscriber, #48457) [Link]
This is how I feel literally all the time.
The Emacs dumper dispute
Posted Dec 1, 2016 4:55 UTC (Thu) by zblaxell (subscriber, #26385) [Link]
I am completely mystified about why the second half of the article isn't"...so we spawn a second thread in main() which emulates exactly the terminal IO model Emacs was designed for. Every 16.7ms, this thread takes a snapshot of what Emacs has written on its frames so far, renders the text in a backbuffer, and swaps buffers with the display frontbuffer (unless the frames haven't changed and we don't have any Expose events to deal with). Emacs updates its frames whenever the hell it wants and nobody bothers Emacs to do anything it doesn't want to. Then we walked through all the #ifdef five-toolkits mess and just deleted it. You're welcome."
In this century we have smartphones that can render a screen full of text faster than the monitor refresh rate while digitally emulating in real time the electron-beam distortions that text would encounter as it passes through the analog electronics of a vintage 1980's CRT. It sounds like this Emacs hack is solving the problem the Emacs way instead of the sane way.
I suspect the solution to the mystery is going to be another level of mystery (I think the first level of that might be "OK so threads and unexec don't get along well, but why did anyone think unexec was the best of several potential alternative solutions, even at the time?"), recursively descending until one of us shouts "you people are crazy" and storms out of the room.
the sort of thing that could give vi a definitive advantage in the interminable editor warsThis is a serious misunderstanding of the editor wars. The alternative is not the other editor. The alternative is downgrading glibc.
The Emacs dumper dispute
Posted Dec 1, 2016 7:17 UTC (Thu) by smurf (subscriber, #17840) [Link]
Suppose you do horizontal scrolling. You don't want the top part of your buffer to be to the right of the bottom part. Not even for 1/60th of a second.
Yes, power users care about that sort of thing. Yes, it can be noticeable, even at that speed.
The Emacs dumper dispute
Posted Dec 2, 2016 3:48 UTC (Fri) by zblaxell (subscriber, #26385) [Link]
Seriously, though, in the world outside of Emacs it would be much easier to have the elisp thread ping the render thread to say "ok now is/is not a good time to take your screenshot away for rendering" than to try to get a single thread to run elisp, push updates at the display refresh rate, and avoid burning resources on unnecessary rendering. The original Emacs design assumed those things would be done on a dedicated external processor, not embedded in the REP loop.
Apart from anything else it's a waste of CPU cores to only use one. The rendering task takes a few milliseconds, and waiting for rendering will burn time Emacs could be spending on deciding what to display next.
The Emacs dumper dispute
Posted Dec 2, 2016 2:22 UTC (Fri) by da4089 (subscriber, #1195) [Link]
The Emacs dumper dispute
Posted Nov 30, 2016 19:08 UTC (Wed) by madscientist (subscriber, #16861) [Link]
So, in Jan 2016 Emacs devs switched to a different implementation for dumper. Presumably this different implementation (is that one of the ones described in this article? Or something different?) works well with both old and new glibc implementations... doesn't it?
Then it was (as I understand it) discovered that if you had an older Emacs without the different implementation but installed a newer glibc without the old hooks, then a older, slower, unstable interface "ralloc" (whatever that is) would be used.
So the solution is to... create more new interfaces, which will be released in even newer versions of Emacs, which still won't help anyone running an older Emacs without these new interfaces? Why is that an improvement?
And, since Emacs 25.1 was released just over two months ago, why is that version also negatively impacted, if it had the new implementation decided on in January?
Sorry for being dense but I think I'm missing something fundamental here.
The Emacs dumper dispute
Posted Nov 30, 2016 23:58 UTC (Wed) by nix (subscriber, #2304) [Link]
I'll have to catch up with the emacs dev list, clearly -- I had no idea about this latest ralloc mess, though it was probably predictable because next to nothing has used ralloc for years so it has probably bitrotted catastrophically.
The Emacs dumper dispute
Posted Dec 3, 2016 0:35 UTC (Sat) by giraffedata (guest, #1954) [Link]
I don't know what any of that has to do with the question, which was a very good one.But I combed the article again, and see the answer:
The problem being solved by the portable dumper work is not that old Emacs won't run on new Glibc. The article starts off describing that problem, but only as an example of the effect of the larger problem:
So something clearly needs to be done to make the Emacs dumping facility more stable and, preferably, more maintainable going forward.
The article doesn't mention any solution for the Emacs 25.1-on-glibc-2.24 problem; maybe there isn't one.
The Emacs dumper dispute
Posted Dec 8, 2016 17:53 UTC (Thu) by nix (subscriber, #2304) [Link]
Er, the entire comment directly relates to what happens when a newly-compiled Emacs starts on a new glibc (it uses its own malloc()) and/or what an Emac compiled against an older glibc does (Emacs binds to the malloc_set_state() compat symbol, the implementation of which detects the old glibc's heap format etc etc).
The Emacs dumper dispute
Posted Nov 30, 2016 19:15 UTC (Wed) by jlargentaye (subscriber, #75206) [Link]
Speaking of which, how's Guile-Emacs doing?...
The Emacs dumper dispute
Posted Nov 30, 2016 19:43 UTC (Wed) by vadim (subscriber, #35271) [Link]
I haven't worked on the Emacs code, but if I was going to be turned off by something, it would be by some weird black magic that relies on the obscure innards of the implementation of malloc(), and not on a mechanism that serializes data. Walking a data structure and dumping stuff to disk is standard C fare. It might be a complicated instance of it, but unlike something that digs around in the innards of malloc it would be something I could expect to reasonably understand given a normal amount of time and effort. Not so with the unexec stuff.
Besides, we're talking about what amounts to an optimization. If I was going to start working on Emacs now, I don't see myself getting scared off by what is effectively a completely optional piece of code.
The Emacs dumper dispute
Posted Dec 1, 2016 0:03 UTC (Thu) by nix (subscriber, #2304) [Link]
Also, anything that makes Emacs more portable is good in my book, and unexec() was a major impediment to Emacs porting. Congratulations, Daniel!
The Emacs dumper dispute
Posted Nov 30, 2016 21:28 UTC (Wed) by kfogel (subscriber, #20531) [Link]
https://lists.gnu.org/archive/html/emacs-devel/2016-11/ms...
From: Daniel Colascione
Subject: Re: Preview: portable dumper
To: Emacs Devel
Date: Wed, 30 Nov 2016 12:18:21 -0800
Message-ID: <r0251sxsoh6a.fsf@dancol.org>
The Emacs dumper dispute
Posted Dec 1, 2016 4:36 UTC (Thu) by karim (subscriber, #114) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 4:47 UTC (Thu) by quotemstr (subscriber, #45331) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 4:52 UTC (Thu) by karim (subscriber, #114) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 4:55 UTC (Thu) by quotemstr (subscriber, #45331) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 6:12 UTC (Thu) by drag (guest, #31333) [Link]
The Emacs dumper dispute
Posted Dec 1, 2016 10:17 UTC (Thu) by SEMW (guest, #52697) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 15:56 UTC (Fri) by tome (subscriber, #3171) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 16:07 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
And there are alternative projects to improve Vim. Personally, I'm donating to https://neovim.io/ - it's a very nice project. I'm planning to sponsor the development of next version of MidnightCommander based on it.
The Emacs dumper dispute
Posted Dec 6, 2016 5:49 UTC (Tue) by lambda (subscriber, #40735) [Link]
As weird as it feels to suggest it, have you tried Visual Studio Code? It's free software, and it's actually a fairly decent lightweight IDE scriptable in JavaScript.
The Emacs dumper dispute
Posted Dec 1, 2016 6:50 UTC (Thu) by rsidd (subscriber, #2582) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 12:23 UTC (Fri) by nix (subscriber, #2304) [Link]
(Needless to say, Emacs should not be providing its own not-terribly-good malloc() implementation. It should rely on the documented malloc() interface, sans ridiculous dumping hooks, and serialize for dumping. That Eli doesn't understand that this is obviously the only even vaguely maintainable approach is baffling to me.)
The Emacs dumper dispute
Posted Dec 2, 2016 14:09 UTC (Fri) by rsidd (subscriber, #2582) [Link]
The Emacs dumper dispute
Posted Dec 4, 2016 9:25 UTC (Sun) by eliz (guest, #94829) [Link]
This is factually incorrect: Emacs only uses its own malloc implementation during dumping. The dumped emacs executable uses the native malloc implementation on all the platforms mentioned in the previous post.
> It should rely on the documented malloc() interface, sans ridiculous dumping hooks, and serialize for dumping. That Eli doesn't understand that this is obviously the only even vaguely maintainable approach is baffling to me.
In fact, Eli understands this simple fact very well. There's never been any question that unexec should go. The issue at hand is what is the best alternative to unexec, with all its problems.
The Emacs dumper dispute
Posted Dec 8, 2016 17:55 UTC (Thu) by nix (subscriber, #2304) [Link]
Really? Last time I tried to trace through the bugger it spotted the lack of malloc_set_state() and fell back to gmalloc for everything from dumping onwards, with fairly nasty consequences for memory usage. If the nasty consequences only relate to the relatively small amount of state stored during dumping, I'm fairly mystified as to how multi-hundred-MiB variances could be so apparent -- but Emacs is such a dynamic system that maybe I was being misled by perfectly normal run-to-run variability.
The Emacs dumper dispute
Posted Dec 8, 2016 14:54 UTC (Thu) by welinder (guest, #4699) [Link]
regular basis. (And tried to hire him. Maybe it's time to try again.) If you think you
have found some basic programming thing he doesn't understand, chances are
good you overlooked something.
*My* preferred solution to the dumping issue would be to collect all the elc files
that are currently predumped and concatenate them all into one giant startup
file to-be-loaded, basically, by temacs. That's not currently fast enough, but
lexing and parsing elc files should be possible at hundreds of megabytes per
second.
The Emacs dumper dispute
Posted Dec 1, 2016 22:06 UTC (Thu) by eduard.munteanu (guest, #66641) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 12:25 UTC (Fri) by nix (subscriber, #2304) [Link]
The Emacs dumper dispute
Posted Dec 2, 2016 16:36 UTC (Fri) by quotemstr (subscriber, #45331) [Link]
You could argue that dumping is also papering over performance problems, but at least dumping does so in a way that doesn't have negative global performance implications. I wish kernel-level checkpoint and restart were widely available: if the system managed the dumping instead of individual processes like Emacs having to do this work, we'd be better off.
The Emacs dumper dispute
Posted Dec 4, 2016 17:19 UTC (Sun) by mstone_ (subscriber, #66309) [Link]
The Emacs dumper dispute
Posted Dec 4, 2016 19:21 UTC (Sun) by viro (subscriber, #7872) [Link]
1) that data structures won't take a lot more than the code would
2) that *all* developers can tell their arse from elbow and memory leak from decent behaviour.
Note that relatively few participants can introduce leaks faster than everybody else manages to fix them...
The Emacs dumper dispute
Posted Dec 4, 2016 18:09 UTC (Sun) by drag (guest, #31333) [Link]
There are a lot of benefits to this approach besides speeding up start-up.
Gnome-terminal, tmux, emacs... these are a few of the things that have this model of having a 'server'. If you want to expand the definition somewhat this is not much far off of what Chrome and Firefox browsers are following.
Seems to be a nice way to go.
The Emacs dumper dispute
Posted Dec 4, 2016 22:12 UTC (Sun) by eduard.munteanu (guest, #66641) [Link]