UCW keyboard layout on XWayland @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ :slug: ucw-keymap-on-wayland :date: 2023-12-31 18:23 :tags: xkb, linux, xwayland :category: random :keywords: xkb, xorg, wayland, hack :lang: en :translation: false :status: draft This is a story/writeup of how I debugged UCW layout on XWayland. You may learn here how keyboard layouts work on Linux. Background: The UCW layout ========================== The UCW keyboard layout is one of the interesting methods of typing Czech letters on a rather American keyboard. The main idea is having a classic US (QWERTY) layout, but without the CapsLock, which serves as a key to add diacritics, so that e.g. Caps+s creates "š". It is naturally possible to combine Caps with Shift to create uppercase letters, and because not all letters can have diacritics in Czech (e.g. there is no "ǩ"), it also manages to cover all the German and Slovak letters. (Sometimes, there are multiple diacritics for a single letter, then the additional diacritics are nearby: Caps+e is "é", Caps+w is "ě".) The CapsLock can still be pressed by using Alt+Caps. The layout has some nice features like avoiding deficiencies with the Czech layout (diacritics on number row are too far, both round parentheses are on the same key, other parentheses only with AltGr at random places, no asterisk, for some reason we have "§" though, …). Another nice feature is that it is rather interoperable: I am able to type on any computer with the "most standard" layout and when writing in Czech I can just not press the CapsLock key and only lose my diacritics, computers can have this layout system-wide even when foreign people use it,… I would not say many people use the layout but at least several my friends do and I have come across several random people on the Internet who do also. More on this below :-) The important part is how the layout is set up. Fortunately, it is contained in the xkeyboard-config database, so the following command just enables it in Xorg:: setxkbmap us,cz -variant ,ucw -option grp:caps_switch Technically, this sets *two* layouts (or *groups*), which are switched by pressing the CapsLock key. This has some disadvantages (I have managed a few times to have the order accidentally swapped and layout switchers (or their users) are sometimes confused), but actual issues are rare. Under X11, that is… Corollary 1: I want UCW to work differently. My custom keyboard layout? -------------------------- I didn't like fact that UCW layout is in fact just an overlay and not a "proper" level 3, so I started studying how to create a custom keyboard layout. Few notable resources: `XKB page on ArchWiki `__, manpages for `xkeyboard-config(7) `__, `setxkbmap(1) `__ and `xkbcomp(1) `__ and looking into ``/usr/share/X11/xkb/symbols/cz`` (found by grepping ``/usr/share/X11`` for "ucw"). Another reason for creating my own layouts is various tweaks esp. on laptop keyboards. For example, the laptop I am typing this sentence on has broken the up arrow key, so I would like to map it somewhere. I use ``xmodmap`` for that, but if this could be contained in a single layout, it would make stuff simpler for me. (And as I will later learn, ``xmodmap`` `does not work well `__ with xkb layouts…) Also, I have another tweak on my keyboards: Compose key. This really calls for the custom layout! And when at it, I should create one for the ttys as well… But this had quite a low priority (the ``kbd.sh`` script in *all* my homes does the job well enough), so I didn't get to it in time. Luckily, maybe – if I were quicker I would not end up in this rabbit hole. At this point, I knew the basic stuff and was reasonably sure that I could hack it together, at least somehow; what was stopping me was lack of time and not being sure how I want to manage machine-specific tweaks in two different layout syntaxes (xkb for X11 and kbd for ttys). Aside: A few notes about xkb layouts for completeness ----------------------------------------------------- This all has been described elsewhere in more detail, but it is useful to know when debugging layouts, so I will mention it here again. There are two forms xkb layouts may be specified. The low-level one is called KcCGST (keycodes, compat, geometry, symbols, types) and describes very much everything that should happen on any keypress. These actions are described in subdirectories of ``/usr/share/X11/xkb/`` of the same name. The high-level descriptions are called RMLVO (rules, model, layout, variant, option) and are what you usually configure, either with ``setxkbmap`` or using GUI tools. Files in ``/usr/share/X11/xkb/rules/`` describe the translation of RMLVO to KcCGST (the XML files add human-readable names for the GUIs and the like). Maybe confusingly, the RMLVO names of options and layouts are very similar to the names of KcCGST compat and symbols, but are generally a different thing AFAIK. KcCGST also come in two forms: the readable one and the complete one. The readable one does not determine everything in one file, instead including others. The complete description is, well, complete. It is kind of similar to the C preprocessing. The readable KcCGST are what is stored in the filesystem as well as what you get from ``setxkbmap -print`` and may serve as a good starting point for tweaks. To get the complete ones one can use ``xkbcomp $DISPLAY -``. The processing of these description is `different `__ in Xorg and in Wayland. If I were to bet where the bug is before digging into it, this sounds like a very likely culprit. A wild friend has appeared ========================== … and he had a problem and asked me if I would have a look into it. Apparently, `group switching didn't work in XWayland `__. At first, I wanted to work it around by finally creating the Unified UCW keymap™, but I wanted to learn more about XKB (from TODO GUIDES), which took me quite long, again. (Studying, working, helping with camps *and* fixing layouts takes some time…) Also, since the issue is apparently bigger, solving all group switches would be a better and more useful solution than just hacking up the Unified UCW keymap (even though I want my tweaks to eventually be xkb-based anyway). I had very little experience with Wayland until then, mostly because X11 worked for me rather well. I tried running Sway few times before, but usually quickly reverted to using Xorg for various reasons (Nvidia GPU in laptop, very laggy mouse, maybe *this exact* issue with UCW layout) which I didn't feel like solving at that moment. I wanted to help my friend, though, so I started looking more into using Sway on another laptop and reading about internals of input system on `Peter Hutterer's blog `__ TODO NAME SPELLING? Occasionally I would randomly google (`with DuckDuckGo `__ :-D) for the issue with group switching, just in case… The nerd snipe issue ==================== `TODO SOMEONE `__ on GitHub found out an interesting workaround: when they *disabled* the group switch, the UCW layout *started working*. Complete "huh?" moment, I knew I wanted to know more about why that worked. The `xkeyboard-common(7) manpage `__ says about more group switching options, so I randomly tried switching by Menu key in XWayland (using ``setxkbmap`` even though it complained and was supposed to do nothing), and ended up with *both* CapsLock and Menu adding diacritics. After a bit of mathematic thinking (more in my `comment of the main issue `__) I concluded that it was actually working fine, just that the ``SetGroup`` action was evaluated *twice*, which would overflow back to the first group when two layouts are set (probably the most common case). Mathematic term for this issue is "arithmetic in the Z_2 group" :-) Corollary 2: A simple workaround is adding the UCW overlay twice, e.g. ``setxkbmap us,cz,cz -variant ,ucw,ucw`` (and thus changing the algebraic group to be Z_3). I think that another workaround would be to change the ``compat`` rules for XWayland, but it feels nasty to have such a quirk in xkeyboard-config database. Changing them for everyone (in ``compat/evdev`` directly) might break other systems, so that also does not seem to be a good way. Corollary 3: We are fixing XWayland (or the way it processes events from Wayland). Learning what happens ===================== I continued reading, this time mostly `who-t's post series about custom layouts `__, `core Wayland protocol `__ as well as the source code of various tools (``xev``, ``wev``, ``xkbcli interactive-wayland`` &c.) Given that I knew very little about the inner workings of the stack, I wanted to find some code that would be run and "enhance" it with so many debug prints that I would understand what was the state of various parts of the Xwayland's "stacked xkb". I kind of knew that one part of this is *somewhere* in the XWayland server, whose code felt intimidating, so I wanted to determine where to start nudging it from what APIs the clients use. .. tip:: Downloading various packages in Linux is rather simple, as well as their rebuilding. In Arch, I can get the package's PKGBUILD just with ``yay -G package``, build it with ``makepkg`` and install it using ``yay -U package-….pkg.tar``. Makepkg also has some options which allow me to tweak the source code after downloading it and before building it. Other distros are probably similar, e.g. for Debian-based distros one can use ``apt source``, ``debuild`` and ``apt install`` to do the same. One interesting observation is that contrary to my expectation, XWayland does not seem to use libxkbcommon (according to the ``/proc/PID/maps`` file). This can have several reasons, but its source code also contains a slightly tweaked version of (Xorg-style) xkb, which might mean I will be dealing with `the ugly code `__ :-/ (Actually, XWayland might not process key events itself, instead just passing them to clients, but this seems inconsistent with the issue – what else would be introducing the second group switch?) So now I need to understand a (hopefully small) parts of two protocols: X11 to understand what the X11 clients receive from XWayland, and Wayland to understand what XWayland gets from my compositor. Understanding X11 ----------------- This part is maybe the easier one, because ``xev`` pretty much dumps the data. I should check in the server code as well, but for the Czech chords ``xev`` seems to receive ``KeyEvents`` with state ``0x2000`` under Xorg (i3wm) and with ``0x4000`` under XWayland when three groups are set up (for two groups the state is ``0x0``, i.e. the same as for English keypresses). State bits this high encode the active group and are described in `the relevant part `__ of `the XKB extension documentation `__. We can see that if we set the ``GroupsWrap`` control of XKB (I don't know where/how yet) to ``ClampIntoRange`` it should also help work around the problem. But our goal is to align behaviour of XWayland and JustWayland, i.e. prevent XWayland seeing two group shifts, not to hack our way in either protocol, so we do not stop here. .. Read also xkbproto: Server Internal Modifiers and Ignore Locks Behavior (might explain sth?) At this point, I had an approximate idea of how XKB is supposed to do, hopefully. I have not yet read the implementation in XWayland though… Understanding Wayland --------------------- This part is supposed to also be simple: the core Wayland protocol will send us `wl_keyboard::keymap `__, `wl_keyboard::key `__ and `wl_keyboard::modifiers `__ events containing the required data. The client does not seem to have a say in what kind of keymap it will get, it is up to the compositor. This means that even XWayland cannot say it wants raw data, but ``xkb_v1``-style keycodes are probably close. While exploring what happens I found a bug in ``wev`` (as of version 1.0.0-13 in Arch): for wl_keyboard::modifiers the group is reported as the serial number:: [14: wl_keyboard] key: serial: 238464; time: 554528472; key: 66; state: 1 (pressed) sym: Mode_switch (65406), utf8: '' [14: wl_keyboard] modifiers: serial: 1; group: 0 depressed: 00000000 latched: 00000000 locked: 00000000 Apparently I am `fixing `__ more than one project :-) Note: Currently, SourceHut is down (TODO: remove this when sr.ht is up), so I cannot link you to my patch, so here it is instead:: --- wev.c.orig 2024-01-13 22:38:03.739239211 +0100 +++ wev.c 2024-01-13 22:38:08.012465335 +0100 @@ -368,7 +368,7 @@ uint32_t mods_locked, uint32_t group) { struct wev_state *state = data; int n = proxy_log(state, (struct wl_proxy *)wl_keyboard, "modifiers", - "serial: %d; group: %d\n", group); + "serial: %d; group: %d\n", serial, group); if (n != 0) { printf(SPACER "depressed: %08X", mods_depressed); print_modifiers(state, mods_depressed); But I am also interested in what is exchanged between the compositor and XWayland server. And there does not seem to be any kind of sniffer yet, so `I started writing one `__ .. TO-MENTION: - nested compositor issues - the IM resolves compose? (I could bisect it, but too lazy) - copy my issue comment in case link rots. - list of all the packages I had downloaded and compiled .. TODO: rename XWayland to Xwayland…