blog/content/linux-default-app-mess.rst

Linux setting of default application is a mess
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

:date: 2023-02-19 12:00

.. TODO: frontmatter
    Date: 2023-02-19

… so I made it `worse <https://xkcd.com/927/>`_ by trying to solve it once again.

This is a sad tale of a rather simple sysadmin's problem, distribution
differences, standards and various ways of their abuse.

The problem
===========

We have a too-conventional setup with desktops shared by many users. This means
we have many desktop environments and many different programs and now we need
to make sure that even a new user with no defaults set in their profile can use
the desktop at least somewhat.

Thus, I need to make sure that the default default applications are reasonable
and do e.g. open programs with user-friendly interface. And I want a reasonable
way of specifying those defaults.

Our environment is Debian based, but I found no reasonable method, so I wanted
to check whether there is one in another distro. Oh boy that was a mistake, the
state of affairs made me quite sad.

The standards
=============

There seem to be two major standards, both regarding mapping of MIME types to
programs. The older standard is Mailcap. Somewhat specified in `RFC 1524
<https://www.rfc-editor.org/rfc/rfc1524>`_ (which was never declared to be a
standard), it describes what applications should be used with what actions and
under what conditions. One can use ``run-mailcap`` to use this way of opening
files. This was in 1993.

Then the "Linux desktop hurr durr people" (aka XDG or freedesktop.org) came
along and saw that the Mailcap file is not enough INI. (And probably too
useful, since the conditions could be arbitrary shell commands.) Comming from
corporate environments, several standards (albeit quite short) have emerged:

- `Desktop files with MimeType support directives
  <https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html#recognized-keys>`_
- |dfl-mime|_

  .. |dfl-mime| replace:: Default application spec (``mimeapps.list``)
  .. _dfl-mime: https://specifications.freedesktop.org/mime-apps-spec/mime-apps-spec-latest.html
  .. What a lovely syntax this is.
     Ref: https://docutils.sourceforge.io/FAQ.html#is-nested-inline-markup-possible
- `MIME type subclassing
  <https://specifications.freedesktop.org/shared-mime-info-spec/shared-mime-info-spec-latest.html#subclassing>`_
  (not to be confused or compared with subtypes from `original MIME type
  RFC 2026 <https://www.rfc-editor.org/rfc/rfc2046>`_)

Also, the same group created ``xdg-utils``, supposedly reference implementation
of the standard, containing the scripts like ``xdg-open`` and ``xdg-mime``.

.. TODO: find the history of the stds to understand mimeinfo.cache,
   defaults.list and so on.

The XDG standard has several deficiences, like not being able to specify broad
rules like for ``video/*``. This leads to the ``mimeapps.list`` file being
quite long and hard for administrators to actually generate it (e.g. if one
would want to use, say, VLC for all video and audio files).

With Mailcap, this really is quite similar, but on Debian a ``mailcap.order``
can be set up, which provides a simple way to prioritize some programs. More on
that later.

Also, Mailcap had support for different programs for viewing and creating the
files. AFAIK that is not possible with XDG, since ``mimeapps.list`` has no way
of distinguishing the use-cases. IMHO that is a shame, but apparently we have
moved on. (XDG's deskop files usually describe human-readable program names and
descriptions, which is a useful thing for desktop environments, so I guess that
it is then simple to just embrace the XDG ecosystem.)

Because the XDG standard is quite old, one would suppose that everyone uses
``mimeapps.list`` these days, and also that ``xdg-mime`` is bug-free. Well,
neither of these is true.

The abuse – what distributions do
=================================

So I went to see how the popular desktop basic-user targetting distributions
determine the default applications. I usually asked ``xdg-mime`` what program
would be used for a PDF (or whatever got determined to have ``application/pdf``
type), and then ``strace``-ed it to determine which file was used to give the
definitive answer.

Since this is a bit of reverse engineering, I might have made some mistake and
get wrong results. I just needed to skim through the distros, so I did not
verify my findings. The command used: ``strace -ff -e %file,read,write -z
xdg-mime query default application/pdf 2>&1 | less``. (Also, not everything
would use ``xdg-mime``'s implementation to find the default app, but I need to
go with something…)

The following table shows my findings. I used the LiveCDs extensively (to
quickly get a vanilla working setup), so I used the ISO filename as distro
name (this also helps reproducing my results). Explanations follow the table.

I was interested in both what file determines the defaults and how it got
created.

.. list-table:: Distribution default application mechanisms
   :header-rows: 1

   * - Distro name (ISO)
     - Defaults file used
     - Package
     - Method of creation
     - Mailcap?
     - Other notes
   * - ``openSUSE-Leap-15.4-XFCE-Live-x86_64-Build31.98-Media.iso``
     - ``xfce-mimeapps.list``
     - ``xfce4-session-branding-openSUSE``
     - Generated from ``/etc/xfce_defaults.conf`` by ``suse-update-mime-defaults``
     - No
     - Actually usable for an admin, their config is short, readable and understandable.
   * - ``debian-live-11.6.0-amd64-xfce.iso``
     - ``mimeinfo.cache``
     - ``desktop-file-utils``
     - ``update-desktop-database`` reads all the .desktop files and dumps the cache.
     - No
     - No way to prioritize, always uses lexicographically first?
   * - ``ubuntu-22.10-desktop-amd64.iso``
     - ``defaults.list`` (``mimeinfo.cache`` as fallback)
     - ``desktop-file-utils``
     - Unknown (shipped); ``update-desktop-database``
     - Generated by ``update-mime``
     - Several issues, see below
   * - ``Fedora-Workstation-Live-x86_64-37-1.7.iso``
     - ``gnome-mimeapps.list`` (``mimeapps.list`` also present)
     - ``gnome-desktop3`` (``shared-mime-info``)
     - Unknown (shipped in both packages)
     - Minimal, delegating everything to ``xdg-open``
     -

Of course, interpretation of the files and the order is up to the implementation…

``xdg-mime``: The horror
------------------------

The spec for ``mimeapps.list`` describes the order for where to look for the
file (I am omiting the environment variables and DE-specific lists for
simplicity):

#. ``~/.config/``
#. ``/etc/xdg/``
#. ``~/.local/share/applications/`` (deprecated)
#. ``/usr/local/share/applications/`` and ``/usr/share/applications/``

This seems to be somewhat honored by ``xdg-mime``, as long as there is nothing
else. When looking at what files the ``xdg-mime`` process (and children) opens,
we see several other interesting paths:

- ``~/.local/share/applnk/`` and ``/usr/share/applnk``. I have no idea where these come from.
- ``~/.local/share/applications/defaults.list`` and ``mimeinfo.cache`` and the
  same files in ``/usr/share/applications/`` The `Debian wiki
  <https://wiki.debian.org/MIME>`_ tells us that the former is a historic name
  for ``mimeapps.list`` and the latter is just a mapping from MIME types to any
  desktop files which claim support for said type.

  Strangely, these two files seem to be checked at the same time, even though
  the ``mimeinfo.cache`` is much worse source for finding reasonable apps.
  (Also: at least on Debian ``mimeinfo.cache`` has wrong section header, but
  nobody seems to care.)
- As the absolutely last resort, it seems to just read all the ``.desktop``
  files and find any that could open the given MIME type.

More bugs
`````````

Apart from weird orders, there are several more bugs and issues:

- While the shared-mime-info spec defines subclassing of MIME types, neither
  ``xdg-utils`` nor its dependencies care for the subclasses.
- I am almost sure that the ``cut`` command in the ``check_mimeapps_list``
  function (`source
  <https://cgit.freedesktop.org/xdg/xdg-utils/tree/scripts/xdg-mime.in?id=8ae02631a9806da11b34cd6b274af02d28aee5da#n346>`_)
  prevents any fallbacks for a default app. Either the first desktop file in
  the list is present, or the whole ``mimeapps.list`` file is just skipped.
- It seems that it sometimes does not check whether the given desktop file
  exists. Esp. on the live Ubuntu the best match for ``audio/flac`` is
  ``rhythmbox.desktop`` (from ``defaults.list``), but only
  ``org.gnome.Rhythmbox3.desktop`` exists. (``xdg-open`` can find the correct
  desktop file, I have no idea how.)

Overall by skimming through the source code of ``xdg-mime`` I have the uneasy
feeling of it being very inconsistent (e.g. sometimes a desktop file is parsed
using ``awk``, and sometimes using ``grep`` and ``cut``). I did not dig too
deep into the code nor the specifications, so I cannot say whether this is
required by something or not. Does not feel like a dependable software though…

Few notes on Mailcap
--------------------

Unfortunately the venerable Mailcap feels quite irrelevant today, so I didn't
even bother looking much into it yet. Weirdly though, various applications in
Arch, Debian and Ubuntu do depend on or recommend Mailcap package, which seems
that sometimes it might still be used, but AFAIK the XDG spec is prevalent,
meaning the presence of mailcap might actually worsen the situation by having
multiple configurations of default apps (except on Fedora, where it defers to
``xdg-open``).

It should in theory be possible to create a default desktop app that would
conversely defer handling of all known MIME types to Mailcap, but it is hard to
make sure that the Mailcap file contains all programs. Since packages ship the
desktop files, either the Mailcap generation would rely on the shipped file
(which is not better than just using XDG), or it would require immense amount
of work to keep everything in sync.

Also, Mailcap itself does not in fact solve the administration problem of
bulk-setting default apps. With vanilla Mailcap the order of directives in the
file has to be maintained manually, and wildcard support for subtypes is prone
to matching even unsupported files. This would create the opposite issue which
would again lead to enumerating valid mime types for all apps.

Debian has a mechanism for solving this: ``mailcap.order``. This file
represents *MIME packages*, which take precedence in the resulting
``/etc/mailcap``. The package itself is just a file in ``/lib/mime/packages``
containing Mailcap entries. However, these files somewhat duplicate what
desktop files already do (but probably didn't do when the mechanism emerged),
and while they have more expressive power, only a handful of packages seem to
provide these descriptions, rendering the idea useless. (Again, one could put
immense work into creating all the packages, but at what cost. Also, this seems
to be Debian-specific at the moment.)

The solution – how and why I reimplemented openSUSE's solution
==============================================================

At this point, I am quite sure that the only reasonable way forward is
generating ``mimeapps.list`` from the desktop files. The easy way forward would
just be using openSUSE's ``suse-update-mime-defaults``. Unfortunately it has
several deficiencies:

.. TODO: link the openSUSE sources

- It cannot declare any kind of subset of types. Either an association is made
  for single (proper) MIME type, or for all supported types. This clashes with
  *web* browsers wanting to open everything including PDFs. This either means
  enumerating the allowed types, or elaborate setting of precedences in order
  to achieve the required order; neither option seems to be pleasant for an
  administrator. Similarly, one could end up with MPV displaying images, since
  it could do that.
- It seems to be intended as internal tool for openSUSE, so I don't know how
  much would upstream be cooperative.
- According to the recent changes, it needs to be updated for every DE
  supported. While it makes sense to tailor the application set, I don't think
  code changes are really needed.
- It is mostly written in awk, which is unpleasant for me to read, so I want to
  avoid maintaining any forks of it. Also, I have no idea how and why it
  interacts with gio.

My approach is to create a few simple utilities, which can be used as a
building blocks for creating any required combinations:

- The *defaulter* processes regex rules of default apps and produces a
  ``mimeapps.list`` file conforming to these rules and their order.
- The *combiner* applies several ``mimeapps-list``-style files after one
  another, allowing reordering preferences and merging files.
- The *expander* (not yet implemented) duplicates rules for subclasses, thus
  working around ``xdg-mime``'s failure to resolve superclasses when specific
  match is not found.
- Possibly an *realizer* could be written, which would filter the list to
  contain only executable applications on current system, thus working around
  the failing-fallbacks bug.

I chose to use regexes to achieve a format similar to the resulting
``mimeapps.list`` while being as general as possible. Particularly,
wildcard subtypes, general rules and proper types are all quite simple (even
though one needs to write e.g. ``image/.*`` instead of the canonical
``image/*``), and e.g. matching video codecs is possible with
``video/.*theora.*`` (albeit probably with some traps of matching wrong types).

The scripts do not impose any kind of flow. You can pipeline them, ``make`` a
workflow that creates DE-tailored lists, create a hook to update the defaults
whenever a new package is installed, … I am not thinking of providing any such
infrastructure first hand, though there might be some examples (possibly
related to my use-case).

Also, it is written in Python, which might provide better readability than awk.
(Or it might not, given the shortcuts I sometimes use; you decide :-)) I do not
aim for this to be a basic system tool, so I can afford to use non-POSIX
interpreter.

Enough talking, here is the repository: https://gitea.ledoian.cz/LEdoian/mimeapps-list-tools.

Other things learned
====================

- Finding package sources for openSUSE is not straight forward. I have not
  found any kind of source download from Open Build Service, so I ended up with
  whatever ``zypper`` can do. There is ``zypper source-install``, which
  downloads the sources *somewhere* into ``/usr/src`` (the spec file ended up
  in different directory than the files for some reason).

  `This answer on Unix.SE <https://unix.stackexchange.com/a/301446>`_
  describes a way to get the tarball. (One apparently cannot query
  ``osc`` without having an account.)

.. If the current state of "Linux on the desktop" is this horrible, I only want
   to run Linux on a server with keyboards, mice and monitors attached.