UCW keymap (wip)

No branches, I don't care :-)

content/ucw-keymap-on-xwayland.rst

@@ -0,0 +1,210 @@
+UCW keyboard layout on XWayland
+@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+
+:slug: ucw-keymap-on-wayland
+:date: 2023-12-31 18:23
+:tags: xkb, linux, xwayland
+:category: random
+:keywords: xkb, xorg, wayland, hack
+:lang: en
+:translation: false
+:status: draft
+
+This is a story/writeup of how I debugged UCW layout on XWayland. You may learn
+here how keyboard layouts work on Linux.
+
+Background: The UCW layout
+==========================
+
+The UCW keyboard layout is one of the interesting methods of typing Czech
+letters on a rather American keyboard. The main idea is having a classic US
+(QWERTY) layout, but without the CapsLock, which serves as a key to add
+diacritics, so that e.g. Caps+s creates "š". It is naturally possible to
+combine Caps with Shift to create uppercase letters, and because not all
+letters can have diacritics in Czech (e.g. there is no "ǩ"), it also manages to
+cover all the German and Slovak letters. (Sometimes, there are multiple
+diacritics for a single letter, then the additional diacritics are nearby:
+Caps+e is "é", Caps+w is "ě".) The CapsLock can still be pressed by using
+Alt+Caps.
+
+The layout has some nice features like avoiding deficiencies with the Czech
+layout (diacritics on number row are too far, both round parentheses are on the
+same key, other parentheses only with AltGr at random places, no asterisk, for
+some reason we have "§" though, …). Another nice feature is that it is rather
+interoperable: I am able to type on any computer with the "most standard"
+layout and when writing in Czech I can just not press the CapsLock key and only
+lose my diacritics, computers can have this layout system-wide even when
+foreign people use it,…
+
+I would not say many people use the layout but at least several my friends do
+and I have come across several random people on the Internet who do also. More
+on this below :-)
+
+The important part is how the layout is set up. Fortunately, it is contained in
+the xkeyboard-config database, so the following command just enables it in Xorg::
+
+    setxkbmap us,cz -variant ,ucw -option grp:caps_switch
+
+Technically, this sets *two* layouts (or *groups*), which are switched by
+pressing the CapsLock key. This has some disadvantages (I have managed a few
+times to have the order accidentally swapped and layout switchers (or their
+users) are sometimes confused), but actual issues are rare. Under X11, that is…
+
+Corollary 1: I want UCW to work differently.
+
+My custom keyboard layout?
+--------------------------
+
+I didn't like fact that UCW layout is in fact just an overlay and not a
+"proper" level 3, so I started studying how to create a custom keyboard layout.
+Few notable resources: `XKB page on ArchWiki`__, manpages for
+`xkeyboard-config(7)`__, `setxkbmap(1)`__ and `xkbcomp(1)`__ and looking into
+``/usr/share/X11/xkb/symbols/cz`` (found by grepping ``/usr/share/X11`` for
+"ucw").
+
+Another reason for creating my own layouts is various tweaks esp. on laptop
+keyboards. For example, the laptop I am typing this sentence on has broken the
+up arrow key, so I would like to map it somewhere. I use ``xmodmap`` for that,
+but if this could be contained in a single layout, it would make stuff simpler
+for me. (And as I will later learn, ``xmodmap`` `does not work well`__ with xkb
+layouts…)
+
+Also, I have another tweak on my keyboards: Compose key. This really calls for
+the custom layout! And when at it, I should create one for the ttys as well…
+
+But this had quite a low priority (the ``kbd.sh`` script in *all* my homes does
+the job well enough), so I didn't get to it in time. Luckily, maybe – if I were
+quicker I would not end up in this rabbit hole. At this point, I knew the basic
+stuff and was reasonably sure that I could hack it together, at least somehow;
+what was stopping me was lack of time and not being sure how I want to manage
+machine-specific tweaks in two different layout syntaxes (xkb for X11 and kbd
+for ttys).
+
+Aside: A few notes about xkb layouts for completeness
+-----------------------------------------------------
+
+This all has been described elsewhere in more detail, but it is useful to
+know when debugging layouts, so I will mention it here again.
+
+There are two forms xkb layouts may be specified. The low-level one is called
+KcCGST (keycodes, compat, geometry, symbols, types) and describes very much
+everything that should happen on any keypress. These actions are described in
+subdirectories of ``/usr/share/X11/xkb/`` of the same name.
+
+The high-level descriptions are called RMLVO (rules, model, layout, variant,
+option) and are what you usually configure, either with ``setxkbmap`` or using
+GUI tools. Files in ``/usr/share/X11/xkb/rules/`` describe the translation of
+RMLVO to KcCGST (the XML files add human-readable names for the GUIs and the
+like). Maybe confusingly, the RMLVO names of options and layouts are very
+similar to the names of KcCGST compat and symbols, but are generally a
+different thing AFAIK.
+
+KcCGST also come in two forms: the readable one and the complete one. The
+readable one does not determine everything in one file, instead including
+others. The complete description is, well, complete. It is kind of similar to
+the C preprocessing.
+
+The readable KcCGST are what is stored in the filesystem as well as what you
+get from ``setxkbmap -print`` and may serve as a good starting point for
+tweaks. To get the complete ones one can use ``xkbcomp $DISPLAY -``.
+
+The processing of these description is `different`__ in Xorg and in Wayland. If
+I were to bet where the bug is before digging into it, this sounds like a very
+likely culprit.
+
+A wild friend has appeared
+==========================
+
+… and he had a problem and asked me if I would have a look into it. Apparently,
+`group switching didn't work in XWayland`__. At first, I wanted to work it around
+by finally creating the Unified UCW keymap™, but I wanted to learn more about
+XKB (from TODO GUIDES), which took me quite long, again. (Studying, working,
+helping with camps *and* fixing layouts takes some time…)
+
+Also, since the issue is apparently bigger, solving all group switches would be
+a better and more useful solution than just hacking up the Unified UCW keymap
+(even though I want my tweaks to eventually be xkb-based anyway).
+
+I had very little experience with Wayland until then, mostly because X11 worked
+for me rather well. I tried running Sway few times before, but usually quickly
+reverted to using Xorg for various reasons (Nvidia GPU in laptop, very laggy
+mouse, maybe *this exact* issue with UCW layout) which I didn't feel like
+solving at that moment.
+
+I wanted to help my friend, though, so I started looking more into using Sway
+on another laptop and reading about internals of input system on `Peter
+Hutterer's blog`__ TODO NAME SPELLING? Occasionally I would randomly google
+(`with DuckDuckGo`__ :-D) for the issue with group switching, just in case…
+
+The nerd snipe issue
+====================
+
+`TODO SOMEONE`__ on GitHub found out an interesting workaround: when they
+*disabled* the group switch, the UCW layout *started working*. Complete "huh?"
+moment, I knew I wanted to know more about why that worked. The
+`xkeyboard-common(7) manpage`__ says about more group switching options, so I
+randomly tried switching by Menu key in XWayland (using ``setxkbmap`` even
+though it complained and was supposed to do nothing), and ended up with *both*
+CapsLock and Menu adding diacritics.
+
+After a bit of mathematic thinking (more in my `comment of the main issue`__) I
+concluded that it was actually working fine, just that the ``SetGroup`` action
+was evaluated *twice*, which would overflow back to the first group when two
+layouts are set (probably the most common case). Mathematic term for this issue
+is "arithmetic in the Z_2 group" :-)
+
+Corollary 2: A simple workaround is adding the UCW overlay twice, e.g.
+``setxkbmap us,cz,cz -variant ,ucw,ucw`` (and thus changing the algebraic group
+to be Z_3).
+
+I think that another workaround would be to change the ``compat`` rules for
+XWayland, but it feels nasty to have such a quirk in xkeyboard-config database.
+Changing them for everyone (in ``compat/evdev`` directly) might break other
+systems, so that also does not seem to be a good way.
+
+Corollary 3: We are fixing XWayland (or the way it processes events from Wayland).
+
+Learning what happens
+=====================
+
+I continued reading, this time mostly `who-t's post series about custom
+layouts`__, `core Wayland protocol`__ as well as the source code of various
+tools (``xev``, ``wev``, ``xkbcli interactive-wayland`` &c.) Given that I knew
+very little about the inner workings of the stack, I wanted to find some code
+that would be run and "enhance" it with so many debug prints that I would
+understand what was the state of various parts of the Xwayland's "stacked xkb".
+I kind of knew that one part of this is *somewhere* in the XWayland server,
+whose code felt intimidating, so I wanted to determine where to start nudging
+it from what APIs the clients use.
+
+.. tip::
+
+   Downloading various packages in Linux is rather simple, as well as their
+   rebuilding. In Arch, I can get the package's PKGBUILD just with ``yay -G
+   package``, build it with ``makepkg`` and install it using ``yay -U
+   package-….pkg.tar``. Makepkg also has some options which allow me to tweak
+   the source code after downloading it and before building it.
+
+   Other distros are probably similar, e.g. for Debian-based distros one can
+   use ``apt source``, ``debuild`` and ``apt install`` to do the same.
+
+One interesting observation is that contrary to my expectation, XWayland does
+not seem to use libxkbcommon (according to the ``/proc/PID/maps`` file). This
+can have several reasons, but its source code also contains a slightly tweaked
+version of (Xorg-style) xkb, which might mean I will be dealing with `the ugly
+code`__ :-/ (Actually, XWayland might not process key events itself, instead
+just passing them to clients, but this seems inconsistent with the issue – what
+else would be introducing the second group switch?)
+
+
+.. understanding xev, XKeyEvents &c.
+.. Note: neither xev is very barebones and does very little postprocessing of
+   events. (naturally.)
+.. X11: state: 0x2000, sway: 0x4000 or 0x0. These state bits are *not*
+   mentioned in libX11 docs… (bits 13 and 14)
+
+.. TO-MENTION:
+  - nested compositor issues
+  - the IM resolves compose? (I could bisect it, but too lazy)
+  - copy my issue comment in case link rots.
+  - list of all the packages I had downloaded and compiled