Migrating encrypted NVMe to different SoM (imx8mq to a311d)

I created my system using reform-setup-encrypted-nvme with rootfs and swap on my NVMe SSD. Today I upgraded my reform from the imx8mq to the a311d SoM and I wrote down what steps I took to migrate my system. These instructions will probably also work with only small changes for migrations to different SoMs. These instructions require an internet connection because I’m using apt install --reinstall to re-generate /boot/vmlinuz. If you are unable to get an internet connection during this process, make sure to load the contents of your working /boot on an external medium like a usb stick and use those to restore the contents of /boot. Then, replace the call to apt install --reinstall below with update-initramfs -u.

If you are switching to a311d, remember to make sure that your system is at least at linux 6.5. This means that you are either on Debian unstable or on Debian stable from reform.debian.net using the bookworm-backports kernel. Make sure that your system is not missing any updates and that sudo reform-check does not output anything fishy.

In my case, I’m on Debian stable (using the repos from reform.debian.net) and kernel 6.5.10 from the bookworm-backports repo on reform.debian.net. I am thus also using an SD-card image from reform.debian.net for a311d. If your setup looks different, make sure that you download a rescue system that best matches your system to avoid any funny surprises.

Flash your rescue system of choice onto an SD-card, reboot and boot into the rescue system. Once there, you need to unmount the /boot of your SD-card. This will make your rescue system unbootable but a311d requires /boot on SD-card as using eMMC might accidentally soft-brick your system if something goes wrong. This also means, that if any of these steps go wrong, your rescue system might not boot anymore and you might need to re-flash the SD-card with a working copy.

$ sudo umount /boot

Next, unlock your encrypted NVMe and mount it to /mnt:

$ sudo cryptsetup luksOpen /dev/nvme0n1 reform_crypt
$ sudo mount /dev/mapper/reformvg-root /mnt

Make sure that the /etc/fstab inside /mnt looks good. Essentially, make sure that the entries for the rootfs and /boot are correct. Mine looks like this:

/dev/reformvg/root / auto errors=remount-ro 0 1
/dev/mmcblk0p1 /boot auto errors=remount-ro 0 1
UUID=56fdfac2-44b8-4616-834e-2b61d1c1ab0a none swap sw 0 0

On imx8mq, /dev/mmcblk0p1 referred to the first partition on eMMC. On a311d, /dev/mmcblk0p1 does not refer to eMMC but the SD-card. Since I want to tell it to use the first partition on my SD-card as /boot I actually didn’t need to change anything here.

Next, mount all the things:

$ sudo mount /dev/mmcblk0p1 /mnt/boot/
$ sudo mount -o bind /dev /mnt/dev/
$ sudo mount -t sysfs sys /mnt/sys/
$ sudo mount -t proc proc /mnt/proc/

Copying over /etc/resolv.conf will give your chroot environment access to internet. If instead of using apt install --reinstall you have a backup of your old /boot, then this is the point at which you copy it over instead.

$ sudo cp /etc/resolv.conf /mnt/etc/resolv.conf

On imx8mq, we tell flash-kernel whether we want single display or dual display setup by writing the correct value into /etc/flash-kernel/machine. The contents of that file overwrite the information in /proc/device-tree/model. If this file does not get deleted, flash-kernel would copy over the dtb for imx8mq even on a311d. So we delete it and trust on /proc/device-tree/model instead which we mounted into the chroot environment earlier.

$ sudo rm /mnt/etc/flash-kernel/machine

The next step re-installs the kernel. Doing so will write out the right vmlinuz image into /boot, create boot.scr, trigger flash-kernel to copy over the correct dtb, create all symlinks and create extlinux.conf if u-boot-menu is installed. The only tricky bit here is to know which kernel package to re-install. If unsure, there is little harm in just re-installing all of the ones you have installed. In my case, I only re-installed the one that I am booting by default anyways:

$ sudo chroot /mnt apt-get install --reinstall linux-image-6.5.0-0.deb12.4-reform2-arm64

One way to find the right kernel package name is to look at the dependency relationship of the linux-image-arm64 meta-package:

$ sudo chroot /mnt apt-cache show --no-all-versions linux-image-arm64 | grep Depends:
Depends: linux-image-6.5.0-0.deb12.4-reform2-arm64 (= 6.5.10-1~bpo12+1+reform20231224T144225Z1)

This should’ve done the trick. You can make sure that everything is set up as expected by inspecting a few things, for example:

$ file /mnt/boot/dtb
/mnt/boot/dtb: symbolic link to dtbs/6.5.0-0.deb12.4-reform2-arm64/amlogic/meson-g12b-bananapi-cm4-mnt-reform2.dtb

My /boot now looks like this:

$ ls -lha /mnt/boot/
total 62M
drwxr-xr-x  5 root root 1.0K Dec 28 19:05 .
drwxr-xr-x 19 root root 4.0K Dec 28 19:03 ..
-rw-r--r--  1 root root 4.9K Dec 28 19:05 boot.scr
-rw-r--r--  1 root root 4.9K Dec 28 19:04 boot.scr.bak
-rw-r--r--  1 root root 297K Dec 24 15:42 config-6.5.0-0.deb12.4-reform2-arm64
lrwxrwxrwx  1 root root   82 Dec 28 19:05 dtb -> dtbs/6.5.0-0.deb12.4-reform2-arm64/amlogic/meson-g12b-bananapi-cm4-mnt-reform2.dtb
lrwxrwxrwx  1 root root   82 Dec 28 19:05 dtb-6.5.0-0.deb12.4-reform2-arm64 -> dtbs/6.5.0-0.deb12.4-reform2-arm64/amlogic/meson-g12b-bananapi-cm4-mnt-reform2.dtb
drwxr-xr-x  3 root root 1.0K Oct 23 13:50 dtbs
drwxr-xr-x  2 root root 1.0K Dec 28 19:05 extlinux
-rw-r--r--  1 root root 1.5M Oct 23 13:50 flash.bin
-rw-r--r--  1 root root  32M Dec 28 19:05 initrd.img-6.5.0-0.deb12.4-reform2-arm64
drwx------  2 root root  16K Oct 23 13:50 lost+found
-rw-r--r--  1 root root   83 Dec 24 15:42 System.map-6.5.0-0.deb12.4-reform2-arm64
-rw-r--r--  1 root root  29M Dec 24 15:42 vmlinuz-6.5.0-0.deb12.4-reform2-arm64

Now unmount everything, deactivate the LVM volume group and close the luks crypt:

$ sudo umount /mnt/proc
$ sudo umount /mnt/sys
$ sudo umount /mnt/dev
$ sudo umount /mnt/boot
$ sudo umount /mnt
$ sudo vgchange -an reformvg
$ sudo cryptsetup luksClose reform_crypt

Reboot and have fun! :partying_face:

EDIT

There might be a few more things to adjust in userspace to successfully convert from imx8mq to a311d. Like this:

  • remove/comment hwdec=auto from ~/.config/mpv/mpv.conf
  • install firmware-realtek from non-free-firmware (installed by default with sysimage-v4)
  • disable wifi powersaving (needs more investigation)

Here is a bigger threat about a311d issues: Tracking known issues and solutions with the BPi A311D upgrade

5 Likes