My gripes with systemd @@@@@@@@@@@@@@@@@@@@@@ :slug: systemd-gripes :date: 2023-11-13 21:50:36 :tags: systemd, linux :category: programming :keywords: systemd, cgroup, programming, bug :lang: en :translation: false :status: draft This is mainly me complaining about systemd, in very concrete instances. And I will probably extend this in future, when some noteworthy bug appears again. I think I could love systemd… ============================= Now, don't get me wrong. The *idea* behind systemd is a nice one, it is awesome that we can track processes with cgroups and have fine-grained control over them. In fact, as of now, my server uses this extensively to limit resources of various services and allow me to always SSH to it, even when services try to use all CPU and RAM. The idea with everything using sockets, starting asynchronously and waiting on sockets for services that did not yet start also feels like a neat hack (I do not yet know the downsides, it feels there are some, but whatever). If this worked well, I would probably be happy. This is also the feeling I get from the uselessd project and I am sad it died. … but I don't ============= The actual systemd implementation, however, has quite many issues and a rather steady influx of new ones. (They are being fixed, but they are still included in official releases, making a painful experience for bleeding-edge users.) Many of my gripes feel like a lack of quality control: either the devs forgot to consider some use cases, or did not test thoroughly enough. In a few cases, the changes seem to be merged without too much thought. That would be great for a toy project, but that IMHO should not have become a main implementation of a critical system part I have also the feeling that the project is too big for devs' capabilities, so seeing the impact of changes is hard. This can be rephrased as "the devs are too incompetent to handle a project this big/tangled".[#reach]_ Also: there is a lot written from other people: - TODO - TODO - TODO I do not necessarily agree with everything on the pages linked above, but I am definitely not alone. However, this page serves to list my own views and experiences, so I will try to not repeat what has already been said. Here come the concrete cases: Disabling the user instance --------------------------- Applies to version: 247.3-7+deb11u4 A common systemd deployment spawns the user instance automatically when logging in. There does not seem to be a documented way of disabling it, but masking ``default.target`` in the user instance seems to do the job. (Another way is masking the ``user@XXX.service`` in the system instance, if you have the rights.) Firstly, this is undocumented (and might break in the future). Secondly, if you try to find out the current default target, it throws a lovely bullshit error:: $ systemctl --user get-default Failed to get default target: Operation not possible due to RF-kill You know what 'wat' means? The message was transferred through D-bus and I am pretty sure that kernel did not return ERFKILL, this is some internal systemd's confusion. OOM-kill kills the whole session -------------------------------- Applies to version: TODO! 252? (fixed in 252.4 iirc) I was so mad at this one. Out of memory is a quite common condition for me, because I happen to use a few programs that leak memory. I am well aware of the risk that kernel's OOM killer may kill some vital process, but also know that pretty much all the time the leaking program gets shot, so it's not a big deal. Then new systemd landed. It started monitoring OOMs, so that it can terminate and fail the unit which ran OOM. The idea is probably not to leave processes running without other processes of the same service. That seems reasonable. For services. The session is *also* a unit. No special handling. Yes, *any* runaway process would kill the *entire* session. Or rather systemd would, proactively. This had quite a few strong words in the issue report, and systemd devs said such language was unwarranted. I am siding with the reporter, sending SIGKILL to everything in a session is not a sane behaviour imho. .. TODO: link to github Hardcoded user.slice in logind's PAM module ------------------------------------------- Applies to version: TODO The PAM module, which takes care about creating the cgroup hierarchy, is hardcoded to put everything under ``user.slice``. .. TODO: link source code This effectively prevents having service/monitoring access over SSH be in correct place in the hierarchy, i.e. somewhere under ``system.slice``. (Well, you could skip the PAM module, but then everything is under ``sshd.service``). In my use case, I wanted to allocate a tiny amount of resources to the monitoring (which logs in as a system account over SSH) and have the rest available to users, so that I can monitor the machine even when the users use all available memory or forkbomb it. Can't do. Generators ---------- Applies to version: TODO Systemd wants almost everything to be a unit. Unfortunately, not everything natively is, so systemd needs generators to create the units ad-hoc. However, generators run very early, so many facilities of systemd are not available, making for a very hacky feel. Namely: there is no logging, so generators need to log to kernel's ring buffer, and they *cannot* be disabled in any way. Quite naturally, generators form another layer in the stack and as such, they can introduce bugs. For me, the pain point was when the fstab generator (which creates ``.mount`` units) did not parse ``noauto`` or ``_netdev`` flag correctly, thus hanging every single boot of my machine for 90 seconds waiting for a drive that I expected not to be available. When I learned the nature of the problem, I pulled systemd out of the system. (I was already aware of many other deficiencies of systemd at that time, this sounded like fun and surprisingly the first random attempts at removing systemd worked well enough, so I did not revert that state.) Spooky action at a distance w.r.t. logging from user instance ------------------------------------------------------------- Applies to version: TODO Services have logs, and logs are to be read. User services have also logs, but systemd *sometimes* fails at the reading part. Namely, this assert and broken login D-bus and weird interactions various quality of modules generator and fstab Can systemd fix this? ===================== I think it might be possible, since most of these issues stem from not caring about various usecases very much. So an end-to-end test suite of the different usecases could help avoid many of these problems. I have no idea how many people work on systemd and what the community looks like, so I don't know whether the test suite or other remediations would be viable. Do I use systemd? ================= Tough question. I use several machines with different stage of systemd-eviction: - As said, my server uses systemd quite extensively. Each service has its own user, whose user instance runs the service, the system service manages resource limits on the user instances (``user-XXXX.slice`` from the system instance POV). A few services for which I wanted to have logs accessible, run under s6. - My desktop is complete opposite: system is pretty stable, so it is only started with a trivial shell script (without supervision or service management). User's services run under s6-rc.[#stability]_ There is no systemd anywhere. - My laptops and other computers usually run system-wide systemd, but often have very tweaked user-instance – either masked altogether, or with non-default setup of the slices and services, or at least with some services masked. And I am very much not afraid to kill user-systemd on the slightest issue, because I have debugged the s6-rc solution. The main reason for using systemd for me is that I want to use the pre-built Arch's packages (esp. on low-end hardware) and have them work without rebuilding. (I am considering creating a build-server to build packages for my other machines, but it is a rather low priority and thus has not happened yet.) And systemd somehow manages to only suck sometimes :-) ------- .. [#reach] This is a rather bold statement. If you disagree even after reading *the whole article*, please reach me at ``__, I would like to know your POV. .. [#stability] The requirement for service manager is Linux's sound system and its many quirks. I found out that when Pipewire (or its companions) crashes, it sometimes brings down other services (MPD), which is unpleasant. On the other hand, OpenSSH daemon does not crash like ever, and IP configuration is oneshot, so there is no need for supervision on the system part.