Synergy vs xmodmap: FIGHT!

Why you can't use xmodmap to change how Synergy handles modifier keys

As previously mentioned I am doing a major overhaul of my workstation setup, the biggest since 1999. This has turned out to be a huge can of worms.

I will try to explain the tentacular complications caused by some software called Synergy...

I use an Apple trackpad and keyboard, and `1`Password

I made a less-big change several years ago - big in terms of the physical devices I use, but not the software configuration: I fell in love with Apple, especially the Magic Trackpad. On the downside, getting an Apple trackpad to talk directly to a FreeBSD or Debian box is (sadly) a laughable proposition, and because of that my setup at work is a bit weird.

(Digression 1: I also like Apple keyboards. I previously used a Happy Hacking Keyboard (lite 2), which I liked a lot, but I like the low profile and short travel of the Apple keyboards more. Both my HHKB and Apple keyboards use rubber membranes, so despite appearances I have not in fact given up on proper keyswitches - I just never used them much.)

(Digression 2: More recently I realised that I could not continue without a password manager, so I started using 1Password, which for my purposes basically requires having a Mac. If you aren't using a password manager, get one ASAP. It will massively reduce your password security worries.)

At work I have an old-ish cast-off Mac, which is responsible for input devices, 1Password, web browser shit, and disractions like unread mail notifications, iCalendar, IRC, Twitter. The idea (futile and unsuccessful though it frequently is) was that I could place the Mac to one side and do Real Work on the (other) Unix workstation.

Synergy is magic

The key software that ties my workstations together is Synergy.

Synergy allows you to have one computer which owns your input devices (in my case, my Mac) which can seamlessly control your other computers. Scoot your mouse cursor from one screen to another and the keyboard input focus follows seamlessly. It looks like you have multiple monitors plugged into one computer, with VMs running different operating systems displayed on different monitors. But they aren't VMs, they are different bare metal computers, and Synergy is forwarding the input events from one to the others.

Synergy is great in many ways. It is also a bit too magical.

How I want my keyboard to work

My main criterion is that the Meta key should be easy to press.

For reference look at this picture of an Apple keyboard on Wikimedia Commons.

(Digression 3: Actually, Control is more important than Meta, and Control absolutely has to be the key to the left of A. I also sometimes use the key labelled Control as part of special key chord operations, for which I move my fingers down from the home row. In X11 terminology I need ctrl:nocaps not ctrl:swapcaps. If I need caps it's easier for me to type with my little finger holding Shift than to waste a key on Caps Lock confusion . Making Caps Lock into another Control is so common that it's a simple configuration feature in Mac OS.)

I use Emacs, which is why the Meta key is also important. Meta is a somewhat abstract key modifier which might be produced by various different keys (Alt, Windows, Option, Command, Escape, Ctrl+[, ...) depending on the user's preferences and the vagaries of the software they use.

I press most modifier keys (Ctrl, Shift, Fn) with my left little finger; the exception is Meta, for which I use my thumb. For comfort I do not want to have to curl my thumb under my palm very much. So Meta has to come from the Command key.

For the same reasons, on a PC keyboard Meta has to come from the Alt key. Note that on a Mac keyboard, the Option key is also labelled Alt.

So if you have a keyboard mapping designed for a PC keyboard, you have Meta <- Alt <- non-curly thumb, but when applied to a Mac keyboard you get Meta <- Alt <- Option <- too-curly thumb.

This is an awkward disagreement which I have to work around.

X11 keyboard modifiers

OK, sorry, we aren't quite done with the tedious background material yet.

The X11 keyboard model (at least the simpler pre-XKB model) basically has four layers:

keycodes: numbers that represent physical keys, e.g. 64
keysyms: symbols that represent key labels, e.g. Alt_L
modifiers: a few kesyms can be configured as one of 8 modifiers (shift, ctrl, etc.)
keymap: how keysyms plus modifiers translate to characters

I can reset all these tables so my keyboard has a reasonably sane layout with:

    $ setxkbmap -layout us -option ctrl:nocaps

After I do that I get a fairly enormous variety of modifier-ish keysyms:

    $ xmodmap -pke | egrep 'Shift|Control|Alt|Meta|Super|Hyper|switch'
    keycode  37 = Control_L NoSymbol Control_L
    keycode  50 = Shift_L NoSymbol Shift_L
    keycode  62 = Shift_R NoSymbol Shift_R
    keycode  64 = Alt_L Meta_L Alt_L Meta_L
    keycode  66 = Control_L Control_L Control_L Control_L
    keycode  92 = ISO_Level3_Shift NoSymbol ISO_Level3_Shift
    keycode 105 = Control_R NoSymbol Control_R
    keycode 108 = Alt_R Meta_R Alt_R Meta_R
    keycode 133 = Super_L NoSymbol Super_L
    keycode 134 = Super_R NoSymbol Super_R
    keycode 203 = Mode_switch NoSymbol Mode_switch
    keycode 204 = NoSymbol Alt_L NoSymbol Alt_L
    keycode 205 = NoSymbol Meta_L NoSymbol Meta_L
    keycode 206 = NoSymbol Super_L NoSymbol Super_L
    keycode 207 = NoSymbol Hyper_L NoSymbol Hyper_L

These map to modifiers as follows. (The higher modifers have unhelpfully vague names.)

    $ xmodmap -pm
    xmodmap:  up to 4 keys per modifier, (keycodes in parentheses):

    shift       Shift_L (0x32),  Shift_R (0x3e)
    lock      
    control     Control_L (0x25),  Control_L (0x42),  Control_R (0x69)
    mod1        Alt_L (0x40),  Alt_R (0x6c),  Meta_L (0xcd)
    mod2        Num_Lock (0x4d)
    mod3      
    mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
    mod5        ISO_Level3_Shift (0x5c),  Mode_switch (0xcb)

How Mac OS -> Synergy -> X11 works by default

If I don't change things, I get

    Command -> 133 -> Super_L -> Mod4 -> good for controlling window manager

    Option -> 64 -> Alt_L -> Mod1 -> meta in emacs

which is not unexpected, given the differences between PC and Mac layouts, but I want to swap the effects of Command and Option

Note that I get (roughly) the same effect when I plug the Apple keyboard directly into the PC and when I use it via Synergy. I want the swap to work in both cases.

xmodmap to the rescue! not!

Eliding some lengthy and frustrating debugging, the important insight came when I found I could reset the keymap using setxkbmap (as I mentioned above) and test the effect of xmodmap on top of that in a reproducible way. (xmodmap doesn't have a reset-to-default option.)

What I want should be relatively simple:

    Command -> Alt -> Mod1 -> Meta in emacs

    Option -> Super -> Mod4 -> good for controlling window manager

So in xmodmap I should be able to just swap the mappings of keycodes 64 and 133.

The following xmodmap script is a very thorough version of this idea. First it completely strips the higher-order modifier keys, then it rebuilds just the config I want.

    clear Mod1
    clear Mod2
    clear Mod3
    clear Mod4
    clear Mod5

    keycode  64 = NoSymbol
    keycode  92 = NoSymbol
    keycode 108 = NoSymbol
    keycode 133 = NoSymbol
    keycode 134 = NoSymbol
    keycode 203 = NoSymbol
    keycode 204 = NoSymbol
    keycode 205 = NoSymbol
    keycode 206 = NoSymbol
    keycode 207 = NoSymbol

    ! command -> alt
    keycode 133 = Alt_L
    keycode 134 = Alt_R
    add Mod1 = Alt_L Alt_R

    ! option -> super
    keycode 64 = Super_L
    keycode 108 = Super_R
    add Mod4 = Super_L Super_R

WAT?! That does not work

The other ingredient of the debugging was to look carefully at the output of xev and Synergy's debug logs.

When I fed that script into xmodmap, I saw,

    Command -> 64 -> Super_L -> Mod4 -> good for controlling window manager

    Option -> 133 -> Alt_L -> Mod1 -> Meta in emacs

So Command was STILL being Super, and Option was STILL being Alt.

I had not swapped the effect of the keys! But I had swapped their key codes!

An explanation for Synergy's modifier key handling

Synergy's debug logs revealed that, given a keypress on the Mac, Synergy thought it should have a particular effect; it looked at the X11 key maps to work out what key codes it should generate to produce that effect.

So, when I pressed Command, Synergy thought, OK, I need to make a Mod4 on X11, so I have to artificially press a keycode 113 (or 64) to have this effect.

This also explains some other weird effects.

Synergy produces one keycode for both left and right Command, and one for both left and right Option. Its logic squashes a keypress to a desired modifier, which it then has to map back to a keysym - and it loses the left/right distinction in the process.
If I use my scorched-earth technique but tell xmodmap to map keysyms to modifiers that Synergy isn't expecting, the Command or Option keys will mysteriously have no effect at all. Synergy's log complains that it can't find a mapping for them.
Synergy has its own modifier key mapping feature. If you tell it to map a key to Meta, and the X11 target has a default keymap, it will try to create Meta from a crazy combination of Shift+Alt. The reason why is clear if you work backwards from these lines of xmodmap output:
```
keycode  64 = Alt_L Meta_L Alt_L Meta_L
mod1        Alt_L (0x40),  Alt_R (0x6c),  Meta_L (0xcd)
```

My erroneous mental model

This took a long time to debug because I thought Synergy was mapping keypresses on the Mac to keycodes or keysyms on X11. If that had been the case then xmodmap would have swapped the keys as I expected. I would also have seen different keysyms in xev for left and right. And I would not have seen the mysterious disappearing keypresses nor the crazy multiple keypresses.

It took me a lot of fumbling to find some reproducible behaviour from which I could work out a plausible explanation :-(

The right way to swap modifier keys with Synergy

The right way to swap modifier keys with Synergy is to tell the Synergy server (which runs on the computer to which the keyboard is attached) how they keyboard modifiers should map to modifiers on each client computer. For example, I run synergys on a Mac called white with a synergyc on a PC called grey:

    section: screens
        grey:
            super = alt
            alt = super
        white:
            # no options
    end

You can do more complicated mapping, but a simple swap avoids craziness.

I also swap the keys with xmodmap. This has no effect for Synergy, because it spots the swap and unswaps them, but it means that if I plug a keyboard directly into the PC, it has the same layout as it does when used via Synergy.

Coda

I'm feeling a bit more sad about Synergy than I was before. It isn't just because of this crazy behaviour.

The developers have been turning Synergy into a business. Great! Good for them! Their business model is paid support for open source software, which is fine and good.

However, they have been hiding away their documentation, so I can't find anything in the current packages or on the developer website which explains this key mapping behaviour. Code without documentation doesn't make me want to give them any money.