|
|
|
|
@ -0,0 +1,246 @@
|
|
|
|
|
<!doctype html>
|
|
|
|
|
<html>
|
|
|
|
|
<head>
|
|
|
|
|
<meta charset=utf-8>
|
|
|
|
|
<meta name=generator content="Pelican 4.11.0">
|
|
|
|
|
<meta name=author content="LEdoian">
|
|
|
|
|
<meta name=description content="My personal webpage">
|
|
|
|
|
<meta name=referrer content=no-referrer>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<link rel=stylesheet href="../theme/css/theme.css">
|
|
|
|
|
|
|
|
|
|
<title>Recovering non-booting PostmarketOS – LEdoian's Blog</title>
|
|
|
|
|
</head>
|
|
|
|
|
<body>
|
|
|
|
|
|
|
|
|
|
<header>
|
|
|
|
|
<h1>LEdoian's Blog</h1>
|
|
|
|
|
</header>
|
|
|
|
|
|
|
|
|
|
<div id=main>
|
|
|
|
|
<nav>
|
|
|
|
|
<div>
|
|
|
|
|
<!-- Main navigation -->
|
|
|
|
|
<!-- TODO! -->
|
|
|
|
|
</div>
|
|
|
|
|
<div>
|
|
|
|
|
<h2>Categories</h2>
|
|
|
|
|
<ul>
|
|
|
|
|
<li><a href="../category/networking.html">networking</a></li>
|
|
|
|
|
<li><a href="../category/programming.html">programming</a></li>
|
|
|
|
|
<li><a href="../category/queer.html">queer</a></li>
|
|
|
|
|
<li><a href="../category/talks.html">talks</a></li>
|
|
|
|
|
<li><a href="../category/technology.html">technology</a></li>
|
|
|
|
|
<li><a href="../category/til.html">til</a></li>
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<h2>Tags</h2>
|
|
|
|
|
<ul>
|
|
|
|
|
<li><a href="../tag/gleam.html">gleam</a></li>
|
|
|
|
|
<li><a href="../tag/software.html">software</a></li>
|
|
|
|
|
<li><a href="../tag/web.html">web</a></li>
|
|
|
|
|
<li><a href="../tag/gender.html">gender</a></li>
|
|
|
|
|
<li><a href="../tag/identity.html">identity</a></li>
|
|
|
|
|
<li><a href="../tag/linux.html">linux</a></li>
|
|
|
|
|
<li><a href="../tag/lifehack.html">lifehack</a></li>
|
|
|
|
|
<li><a href="../tag/relationships.html">relationships</a></li>
|
|
|
|
|
<li><a href="../tag/print.html">print</a></li>
|
|
|
|
|
<li><a href="../tag/comics.html">comics</a></li>
|
|
|
|
|
<li><a href="../tag/ipv6-only.html">ipv6-only</a></li>
|
|
|
|
|
<li><a href="../tag/dns.html">dns</a></li>
|
|
|
|
|
<li><a href="../tag/meta.html">meta</a></li>
|
|
|
|
|
<li><a href="../tag/infrastructure.html">infrastructure</a></li>
|
|
|
|
|
<li><a href="../tag/smrst.html">smršť</a></li>
|
|
|
|
|
<li><a href="../tag/trains.html">trains</a></li>
|
|
|
|
|
<li><a href="../tag/software-engineering.html">software-engineering</a></li>
|
|
|
|
|
</ul>
|
|
|
|
|
</div>
|
|
|
|
|
<div>
|
|
|
|
|
<h2>Stalk me also at</h2>
|
|
|
|
|
TODO!
|
|
|
|
|
|
|
|
|
|
<h2>I stalk</h2>
|
|
|
|
|
TODO!
|
|
|
|
|
</nav>
|
|
|
|
|
|
|
|
|
|
<main>
|
|
|
|
|
<div>
|
|
|
|
|
<div class="details"><time datetime="2025-02-05T19:52:00+01:00">2025-02-05 19:52</time></div>
|
|
|
|
|
<h1>Recovering non-booting PostmarketOS</h1>
|
|
|
|
|
<div class="section" id="the-problem">
|
|
|
|
|
<h2>The problem</h2>
|
|
|
|
|
<p>I rebooted my PmOS on FOSDEM, and it didn't boot up. I got stuck with the penguins. basically brick, but a soft one. Probably early kernel panic, but I can only tell it was stuck (no logs whatsoever)</p>
|
|
|
|
|
<p>This is a story of how I tried to restore my working stuff w/o internet (this is my only internet-capable device)</p>
|
|
|
|
|
<!-- TODO: what happened and what I think has happened -->
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="the-tools">
|
|
|
|
|
<h2>The tools</h2>
|
|
|
|
|
<p>Obvs, Kr (1+6). That has a bootloader and maybe some quirks we will see later</p>
|
|
|
|
|
<p>Several Linux (x86_64, Arch linux) machines that have a rather sizeable amount of tools (file, binwalk, poke, adb, fastboot, …), but <em>almost no internet</em></p>
|
|
|
|
|
<p>The second day of FOSDEM (with free WiFi), 16 hours on the train home and more time at home :-D</p>
|
|
|
|
|
<p>Some experience with flashing firmware to phones including the 1+6</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="the-idea-goal">
|
|
|
|
|
<h2>The idea/goal</h2>
|
|
|
|
|
<p>Rebuild initramfs and whatnot from ad-hoc obtained postmarketos install, reboot and fix stuff</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="the-process">
|
|
|
|
|
<h2>The process</h2>
|
|
|
|
|
<div class="section" id="realization-we-can-fastboot">
|
|
|
|
|
<h3>1. Realization we can fastboot</h3>
|
|
|
|
|
<p>This was a rather obvious, given I flashed PmOS there in the 1st place. However, <tt class="docutils literal">fastboot <span class="pre">--help</span></tt> does not show too many useful options…</p>
|
|
|
|
|
<p>… apart from one: switching slots. I know I had LineageOS (or something similar, I don't remember).</p>
|
|
|
|
|
<p>However, the second slot did not work (I got bootloop instead). So, in slot A there is broken PostmarketOS, in B there is absolutely broken Lineage.</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="recovery">
|
|
|
|
|
<h3>2. Recovery</h3>
|
|
|
|
|
<p>I know about recoveries, but I was not able to boot into one (it just rebooted)</p>
|
|
|
|
|
<p>However, after a sleep (and losing access to internet), I realised that the recoveries might be also slotted. And they were, so I got access to a basic LineageOS recovery in slot B. And that means ADB shell</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="backup">
|
|
|
|
|
<h3>3. Backup</h3>
|
|
|
|
|
<p>Basically like this:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
adb shell cat /dev/block/by-name/system_a | pv > system-a
|
|
|
|
|
chmod a-w * # to make sure I don't break my own backups
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Took quite some time, I think mostly because I only have USB 2.x cables. The good thing is that partitions are named here</p>
|
|
|
|
|
<p>Also, the block devices <tt class="docutils literal">sda</tt> to <tt class="docutils literal">sde</tt> can be copied as whole. I don't know whether that does include the recovery…</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="partitions">
|
|
|
|
|
<h3>4. Partitions</h3>
|
|
|
|
|
<p>fdisk tells me that all of the partitions have 512B sectors and 2TB protective MBR. Really dummy stuff.</p>
|
|
|
|
|
<p>After some experimenting, I realised that the sectors are 4096B in fact. <tt class="docutils literal">dd if=sda bs=4096 skip=1 | file -</tt> however tells me that while this is a GPT data structure (in a nonstandard way, since that is at LBA 0), <tt class="docutils literal">fdisk <span class="pre">-l</span></tt> was not willing to tell me the details. Poking it would give me the data (GPT pickle seems to be part of the standard set), but that was quite unpleasant (I have little experience with Poke and wasn't able to convince it to show me the labels as strings and not byte values). <tt class="docutils literal">hexdump</tt> is a friend, though :-)</p>
|
|
|
|
|
<p>As for the partitions themselves, I don't understand much, but:
|
|
|
|
|
- <tt class="docutils literal">boot_a</tt> seems to be the thing that the bootloader can load. According to <tt class="docutils literal">file</tt> it is <tt class="docutils literal">Android bootimg, kernel (0x8000), ramdisk (0x1000000), page size: 4096, cmdline (console=ttyMSM0,115200 <span class="pre">pmos_boot_uuid=4724a893-210e-41b4-b89e-464252ae295f</span> <span class="pre">pmos_root_uuid=d1c201af-1ece-46f6-8e21-eb89acb494ff)</span></tt>, which sounds good
|
|
|
|
|
- <tt class="docutils literal">userdata</tt> contains another DOS partition table (with wrong sector size again) referencing partitions labeled <tt class="docutils literal">pmOS_boot</tt> and a LUKS one. This is well known from the pmOS userspace
|
|
|
|
|
- <tt class="docutils literal">system_a</tt> does not seem of interest, it is an ext2 filesystem with Android-style directory structure. (Maybe recoveries live in here? But I did not try to dis/prove this)</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="initramfs-from-pmos">
|
|
|
|
|
<h3>5. Initramfs from pmos</h3>
|
|
|
|
|
<p>The recovery's shell is minimal, but it has <tt class="docutils literal">cpio</tt> and <tt class="docutils literal">chroot</tt>. (I was hoping for kexec, but to no avail). So we can get at least the initramfs and get hopefully a better environment…</p>
|
|
|
|
|
<p>The initramfs is however compressed with zstd and lives on a nested partitions with wrong partition tables, so find the file with a bit of <tt class="docutils literal">dd</tt>-ing, <tt class="docutils literal">testdisk</tt>-ing and get the cpio archive with <tt class="docutils literal">zstdcat</tt> or whatever, then <tt class="docutils literal">adb push</tt> it to <tt class="docutils literal">/tmp</tt> on the phone; there we can do <tt class="docutils literal">cpio <span class="pre">-i</span> < <span class="pre">../initramfs</span></tt> and chroot into a random directory under <tt class="docutils literal">/tmp</tt>. The tmpfs's are quite big (4G), so it is not an issue.</p>
|
|
|
|
|
<p>Then the ordinary: fix <tt class="docutils literal">$PATH</tt>, install <tt class="docutils literal">busybox</tt>'s utils and try running init:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
PATH=/bin:/sbin
|
|
|
|
|
busybox --install /bin
|
|
|
|
|
/init # and pray
|
|
|
|
|
</pre>
|
|
|
|
|
<p>… didn't work. Somehow, <tt class="docutils literal">/init</tt> has too advanced syntax for the busybox, so that crashes on setting up log :-/</p>
|
|
|
|
|
<p>Also: we can run <tt class="docutils literal">mdev <span class="pre">-s</span></tt> to populate <tt class="docutils literal">/dev</tt>, but we don't get partition names. Of course, <tt class="docutils literal"><span class="pre">/dev/block/by-name/*</span></tt> in the recovery fs is just a bunch of symlinks, so we can learn that (in my case) <tt class="docutils literal">userdata</tt> is just <tt class="docutils literal">/dev/sda17</tt></p>
|
|
|
|
|
<p>However. there is <tt class="docutils literal">cryptsetup</tt> in <tt class="docutils literal">/sbin</tt>, so we should be able to do something. The question is: how to open and mount that.</p>
|
|
|
|
|
<p>The biggest issue is still with the partition tables: while we <em>do</em> have <tt class="docutils literal">losetup</tt> (actually, two implementations of it, since the recovery also has one), even with partition auto-find, but the tables are wrong, so the <tt class="docutils literal">loop2p*</tt> devices would point to bad places. And the working devices I have are Arch, not Gentoo, so I don't have the source for that.</p>
|
|
|
|
|
<p>Writing this paragraph (above) helped me realize one thing: I can do two things: Either I can fix the partition tables (if there is a bit for the size, and that would be written in the Pickles), and I have a PinePhone (since now I am already at home) that is Aarch64, too, so I can copy binaries from that! (It does not have a working modem, so still no internet, lol)</p>
|
|
|
|
|
<p>The latter is a nice thing, but I don't think it is much useful because the configuration is still located at the encrypted partition which is still inaccessible and I am not sure whether I can fit all the required stuff into the tmpfs in order to build the kernel without decrypting (once I can chroot into the decrypted partition, I think I have won, so then using other tools would not be an advantage).</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="fixing-some-of-the-partition-tables">
|
|
|
|
|
<h3>6. Fixing (some of) the partition tables</h3>
|
|
|
|
|
<p>We have <tt class="docutils literal">xxd</tt>, that can be used as a makeshift hex editor: dump what you need, <tt class="docutils literal">sed</tt> or <tt class="docutils literal">vi`</tt> the fix (or just copy that into the PC and use more comfortable tools) and use <tt class="docutils literal">xxd <span class="pre">-r</span></tt> to edit. Hopefully. (Using hex-editor and piping the result into <tt class="docutils literal">adb shell cat '>' <span class="pre">/dev/block/by-name/userdata</span></tt> might also be an option, but transfering the whole partition sounds painful)</p>
|
|
|
|
|
<p>Is this safe? Hopefully, because we are only modifying the inner table in <tt class="docutils literal">userdata</tt>, so it should not matter to the bootloader and should not affect the recovery.</p>
|
|
|
|
|
<p>Reading into the pickle (<tt class="docutils literal">/usr/share/poke/pickles/mbr.pk</tt> on my machine), we see that there is no field for sector size, everything just is 512B big :-/ And also CHS, not LBA. But DOS partition table means we can just extract the MBR (<tt class="docutils literal">dd if=userdata of=userdata.mbr bs=512 count=1</tt>) and hack on that without needing the rest. And since there is nothing of interest in the header, we only need to change the Partition Table Entries (PTEs)</p>
|
|
|
|
|
<p>So let's poke the partition table:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
shell$ poke userdata.mbr
|
|
|
|
|
(poke) .set obase 16
|
|
|
|
|
(poke) load mbr
|
|
|
|
|
(poke) var ptes = (MBR @ 0#B).pte
|
|
|
|
|
(poke) ptes[0].lba *= 8
|
|
|
|
|
(poke) ptes[0].sector_count *= 8
|
|
|
|
|
(poke) ptes[1].lba *= 8
|
|
|
|
|
(poke) ptes[1].sector_count *= 8
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Poke applies that right away, so we can check with <tt class="docutils literal">fdisk <span class="pre">-x</span> userdata.mbr</tt> the final sizes and do sth like <tt class="docutils literal">dd if=userdata bs=512 skip=START count=SECTORS | file -</tt> to check that the rest is OK. We will silently hope that the CHS data is just not used anyway, so we will not touch that.</p>
|
|
|
|
|
<p>Also note how we changed just the extracted MBR but checked against the original read-only backup.</p>
|
|
|
|
|
<p>Push that to the device and use <em>recovery's</em> <tt class="docutils literal">dd</tt> to patch the partition table:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
pc$ adb push userdata.mbr /tmp
|
|
|
|
|
recovery# dd if=/tmp/userdata.mbr of=/dev/block/by-name/userdata
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Fun fact: initramfs's <tt class="docutils literal">fdisk</tt> does assume 4k sectors correctly, but <tt class="docutils literal">losetup</tt> somehow still uses 512B ones, leading to mismatches… Tip: for ext* filesystem (like <tt class="docutils literal">pmOS_boot</tt>), there is a block of 0x400 zeroes at the start and the label is at offset 0x479 from the start of the filesystem, LUKS2 has <tt class="docutils literal">LUKS</tt> magic at the start and a JSON config at offset 0x1000</p>
|
|
|
|
|
<p>Set up the loop device, run <tt class="docutils literal">mdev</tt> to detect it and verify the partitions got detected correctly:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
initramfs# losetup -P -f /dev/sda17
|
|
|
|
|
initramfs# mdev -s
|
|
|
|
|
initramfs# losetup -a
|
|
|
|
|
/dev/loop4: 0 /dev/sda17
|
|
|
|
|
initramfs# xxd /dev/loop4p1 | less
|
|
|
|
|
initramfs# xxd /dev/loop4p2 | less
|
|
|
|
|
</pre>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="final-touches-w-w-generating-the-correct-er-initramfs">
|
|
|
|
|
<h3>7. Final touches^W^W Generating the correct-er initramfs</h3>
|
|
|
|
|
<p>Whoo! But this is the part I actually got most paranoid: the postmarketOS v24.12 uses kernel 6.12 (ish, it differs a bit across devices, the Oneplus 6 actually uses a -rc kernel), but the recovery has 4.9.337. That means that pmOS's userspace might be using too new system calls and random stuff might start failing at random stages. (So far we have been mostly treating the partitions like data, so everything was using the 4.9 calls and was more-or-less compatible)</p>
|
|
|
|
|
<p>Anyway:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# cryptsetup open /dev/loop4p2 cry
|
|
|
|
|
Enter passphrase for /dev/sda17:
|
|
|
|
|
# mdev -s
|
|
|
|
|
# mkdir /mnt
|
|
|
|
|
# mount -t ext4 /dev/mapper/cry /mnt
|
|
|
|
|
mount: mounting /dev/mapper/cry on /mnt failed: Invalid argument
|
|
|
|
|
</pre>
|
|
|
|
|
<p>… fuck.</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# dmesg
|
|
|
|
|
[100726.519997] EXT4-fs (dm-0): couldn't mount RDWR because of unsupported optional features (10000)
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Oh, here come the incompatibilities. OK, then:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# mount -t ext4 -o ro /dev/mapper/cry /mnt
|
|
|
|
|
</pre>
|
|
|
|
|
<p>There we go! And the usual:</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# for x in proc sys dev; do mount --rbind /$x /mnt/$x; done
|
|
|
|
|
# chroot /mnt /bin/bash
|
|
|
|
|
localhost:/#
|
|
|
|
|
</pre>
|
|
|
|
|
<p>We're in!</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# mount -a
|
|
|
|
|
mount: /boot: /dev/loop1 already mounted or mount point busy.
|
|
|
|
|
dmesg(1) may have more information after failed mount system call.
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Uh, the single partition I need… Oh, that was a previous attempt that lead nowhere, so I can just umount that and retry successfully.</p>
|
|
|
|
|
<pre class="literal-block">
|
|
|
|
|
# export PATH=/usr/sbin:/sbin:/usr/bin:/bin:/usr/local/bin
|
|
|
|
|
# mount -t tmpfs none /tmp
|
|
|
|
|
# export TMPDIR=/tmp
|
|
|
|
|
# mkinitfs
|
|
|
|
|
[…]
|
|
|
|
|
==> Not flashing boot in chroot
|
|
|
|
|
</pre>
|
|
|
|
|
<p>Ugh, so, did this work or not? I think the <tt class="docutils literal">pmOS_boot</tt> partition is correct, but I cannot tell whether it failed before or after loading this partition from <tt class="docutils literal">boot_a</tt>. But anyway, we can check that the <tt class="docutils literal">boot_a</tt> partition and <tt class="docutils literal">/boot/boot.img</tt> are in fact the same thing, so we can flash it with dd from the recovery later. (And it is likely that the actual kernel will not like our changes to the partition table anyway, so we might be doing a second round of the process…)</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="let-s-boot-take-1">
|
|
|
|
|
<h3>8. Let's boot, take 1</h3>
|
|
|
|
|
<p>A quick <tt class="docutils literal">sync</tt> and <tt class="docutils literal">reboot</tt>, followed by the slot switching trickery, and…</p>
|
|
|
|
|
<p>… the situation is still the same, just penguins and nothing more.</p>
|
|
|
|
|
<p>Let me just quickly flash the boot partition and revert the partition table (both from recovery shell using <tt class="docutils literal">dd</tt> and some of the magic from above)</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="let-s-boot-take-2">
|
|
|
|
|
<h3>9. Let's boot, take 2</h3>
|
|
|
|
|
<p>And… wait for it… it worked!</p>
|
|
|
|
|
<p>Now, naturally, I regenerate the initramfs again just in case the ad-hoc environment had some issues, and be done.</p>
|
|
|
|
|
</div>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="alternative-and-faster-solution">
|
|
|
|
|
<h2>Alternative and faster solution</h2>
|
|
|
|
|
<p>Of course I could just find someplace with internet connection and re-flash the device. But I chose not to: I am not sure whether I would be able to re-flash just part of the <tt class="docutils literal">userdata</tt> partition (I want to keep my data) and this seemed to be more of an adventure.</p>
|
|
|
|
|
</div>
|
|
|
|
|
<div class="section" id="other-thoughts-observations">
|
|
|
|
|
<h2>Other thoughts / observations</h2>
|
|
|
|
|
<p>So far, I have no idea how “Android bootimgs” are created, though I think, there was some note about <tt class="docutils literal">avdtool</tt> in the image.</p>
|
|
|
|
|
</div>
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
</main>
|
|
|
|
|
</div> <!-- #main -->
|
|
|
|
|
|
|
|
|
|
<footer>
|
|
|
|
|
<hr>
|
|
|
|
|
Written using Pelican 4.11.0 by LEdoian.
|
|
|
|
|
</footer>
|
|
|
|
|
|
|
|
|
|
</body>
|
|
|
|
|
</html>
|
