Encrypted NVME boot from EMMC confusions

Hey all :wave: DIY kit 157 checking in.

I received my reform mid week last week and got the chance to play with it over the weekend.

I’ve downloaded the reform v3 system image and flashed it to my SD card. I booted from the SD card and ran the reform-migrate encrypted Nvme command.

Booting from the Nvme with the SD card works well. However I want to use EMMC and no SD card. I flipped the switch on the SOM to off and now I boot into the recovery image.

From there I ran the reform-boot-config —EMMC Nvme. That worked fine, but on reboot I keep getting brought back to the recovery system image.

Due to the fact the Nvme is encrypted with luks, it can’t find the Nvme drive. The boot scripts are looking in the unencrypted Nvme drive locations when they should be looking for the encrypted drive.

I’ve looked at the reform-init script and tried playin around with setting the BOOTPART and other variables to get it to work how I’d like but have been unsuccessful.

From reading other threads it sounds like I shouldn’t have ran the reform-boot-config script?

I’d also like to mention that although my SD and Nvme are flashed with the v3 image I think my EMMC is running the v2 recovery image.

I’ve attached an image of the error I get upon booting. So this post is asking for help with how to get past this step. I need the EMMC to boot my encrypted Nvme, when it treats it like an unencrypted Nvme. :thinking:

So far I’m loving my reform and can’t wait to contribute to the community!

The only downside I’ve experienced is that my display arrived broken. I think the screws were tightened too much during assembly prior to me receiving it. I’m lucky it’s usable for now. I’m sure support will help me out when they see my email :slight_smile:

All in all the year long wait was well worth it! Such a beautiful device and cool community :sunglasses:

At this point I’m thinking I need to flash the EMMC with the v3 recovery image?

If I do that, and reflash the Nvme with the v3 image using the reform-migrate command it sounds like I’d be in good shape?

I’m not sure how to flash the EMMC with the new v3 recovery image. I’d imagine I need to boot from the SD, then flash EMMC.

In following this line of thought, I reflashed my SD card so I could try to flash the EMMC & NVME but now I can’t boot from SD card. I can only boot into recovery mode via the EMMC…

Perhaps if I flash the reform v3 image to a usb, then I set reform-init with the usb flag, I can boot from that and reset everything? Not sure why reform-init with the sd card arg doesn’t actually force it to boot from the sd card, it still tries to boot from the Nvme…

I guess if I wanted it to boot from the sd card temporarily, I’d have to open the system back up and flip the switch on the SOM temporarily, then boot from the sd, flash the EMMC with the new image and then flip the SOM switch back?

Trying to avoid opening the reform back up if I have to. Not sure which route is the best here…

Well. I modified the reform-init poorly and can’t exit the script now :sweat_smile: .

So now I’m looking at how to reset the EMMC entirely by reflashing the recovery image.

Rereading the handbook to see where it mentions this now. Also opening the reform up and switching the SOM to ON so it hopefully auto boots from SD card.

The boot process could be better, from my perspective. I have the same setup as you, and I have to keep my boot partition unmounted because if I don’t, apt upgrade can result in my laptop not being able to boot. Last time I upgraded my kernel, I had to pull the reform-system-image and reform-boundary-uboot repositories, change some of the hard-coded paths, and compile uboot and the boot image myself. Took me quite a few cycles of what you’re doing now to get that worked out.

Here’s a post summarizing what I did: MNT Reform System Image V3 Beta - #16 by lykso

Here’s the output of git diff in my reform-system-image repo as it now stands:

diff --git a/reform2-imx8mq/mkimage.sh b/reform2-imx8mq/mkimage.sh
index 5d8fa18..19bfd6f 100755
--- a/reform2-imx8mq/mkimage.sh
+++ b/reform2-imx8mq/mkimage.sh
@@ -5,7 +5,7 @@ set -e
 
 # make sure build tools are installed
 # FIXME: replace this by a check that everything is installed
-#sudo apt-get -y install parted multistrap udisks2 gcc-aarch64-linux-gnu make device-tree-compiler qemu-user-static binfmt-support build-essential bison flex libssl-dev mmdebstrap
+sudo apt-get -y install parted multistrap udisks2 gcc-aarch64-linux-gnu make device-tree-compiler qemu-user-static binfmt-support build-essential bison flex libssl-dev mmdebstrap
 
 # if we are in a git repository and if SOURCE_DATE_EPOCH is not set, use the
 # timestamp of the latest git commit
@@ -20,70 +20,70 @@ export SOURCE_DATE_EPOCH
 export TMPDIR="$(pwd)"
 
 # download u-boot from CI
-/usr/lib/apt/apt-helper download-file https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/master/raw/flash.bin\?job\=build flash.bin
+#/usr/lib/apt/apt-helper download-file https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/master/raw/flash.bin\?job\=build flash.bin
 
 # build the debian userland and configure it
 ./mkuserland.sh
 
 # Rescue System ---------------------------------------------------------
 
-./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland.tar \
-	| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
-dd if=boot.img of=reform-rescue-system.img seek=1 bs=4194304
-rm boot.img
-./tarfilter --path-exclude='/boot/*' < target-userland.tar \
-	| genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball - root.img
-dd if=root.img of=reform-rescue-system.img seek=65 bs=4194304
-rm root.img
-dd if=/dev/zero bs=512 count=1 >> reform-rescue-system.img
-/sbin/parted -s reform-rescue-system.img "mklabel msdos"
-# reproducible disk signature
-printf mntr | dd of=reform-rescue-system.img seek=440 bs=1 conv=notrunc
-/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 4MiB 260MiB"
-/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 260MiB 2308MiB"
-/sbin/parted -s reform-rescue-system.img print
-
-# install u-boot for i.MX8MQ
-dd if=./flash.bin of=reform-rescue-system.img conv=notrunc bs=1k seek=33
-
-echo Reform Rescue System Image created: reform-rescue-system.img
-
-# Full System -----------------------------------------------------------
-
-# chroot into the userland and add extra applications
-./mkuserland3.sh
-
-if [ $(id -u) -eq 0 ]; then
-	# genext2fs is very slow so if we run this script as root, we give up
-	# on reproducibility in favor of creating the image faster
-	mkdir target-userland
-	tar --directory target-userland --xattrs --xattrs-include='*' --extract --file target-userland-full.tar
-	/sbin/mke2fs -v -L 'MNTREFORMBOOT' -N 0 -E offset=4194304 -d target-userland/boot -t ext2 reform-system.img 256M
-	rm -rf target-userland/boot/*
-	/sbin/mke2fs -v -L 'MNTREFORMROOT' -N 0 -O 64bit -E offset=272629760 -d target-userland -m 5 -r 1 -t ext4 reform-system.img 9000M
-	rm -rf target-userland
-else
-	# if we don't run as root, use the slow (but bit-by-bit reproducible)
-	# genext2fs instead
-	./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland-full.tar \
-		| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
-	dd if=boot.img of=reform-system.img seek=1 bs=4194304
-	rm boot.img
-	./tarfilter --path-exclude='/boot/*' < target-userland-full.tar \
-		| genext2fs --block-size 1024 --size-in-blocks 9216000 --bytes-per-inode 16384 --tarball - root.img
-	dd if=root.img of=reform-system.img seek=65 bs=4194304
-	rm root.img
-fi
-dd if=/dev/zero bs=512 count=1 >> reform-system.img
-/sbin/parted -s reform-system.img "mklabel msdos"
-# reproducible disk signature
-printf mntr | dd of=reform-system.img seek=440 bs=1 conv=notrunc
-/sbin/parted -s reform-system.img "mkpart primary ext4 4MiB 260MiB"
-/sbin/parted -s reform-system.img "mkpart primary ext4 260MiB 9260MiB"
-/sbin/parted -s reform-system.img print
-
-# install u-boot for i.MX8MQ
-dd if=./flash.bin of=reform-system.img conv=notrunc bs=1k seek=33
-rm flash.bin
-
-echo Reform Full System Image created: reform-system.img
+#./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland.tar \
+#	| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
+#dd if=boot.img of=reform-rescue-system.img seek=1 bs=4194304
+#rm boot.img
+#./tarfilter --path-exclude='/boot/*' < target-userland.tar \
+#	| genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball - root.img
+#dd if=root.img of=reform-rescue-system.img seek=65 bs=4194304
+#rm root.img
+#dd if=/dev/zero bs=512 count=1 >> reform-rescue-system.img
+#/sbin/parted -s reform-rescue-system.img "mklabel msdos"
+## reproducible disk signature
+#printf mntr | dd of=reform-rescue-system.img seek=440 bs=1 conv=notrunc
+#/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 4MiB 260MiB"
+#/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 260MiB 2308MiB"
+#/sbin/parted -s reform-rescue-system.img print
+#
+## install u-boot for i.MX8MQ
+#dd if=./flash.bin of=reform-rescue-system.img conv=notrunc bs=1k seek=33
+#
+#echo Reform Rescue System Image created: reform-rescue-system.img
+#
+## Full System -----------------------------------------------------------
+#
+## chroot into the userland and add extra applications
+#./mkuserland3.sh
+#
+#if [ $(id -u) -eq 0 ]; then
+#	# genext2fs is very slow so if we run this script as root, we give up
+#	# on reproducibility in favor of creating the image faster
+#	mkdir target-userland
+#	tar --directory target-userland --xattrs --xattrs-include='*' --extract --file target-userland-full.tar
+#	/sbin/mke2fs -v -L 'MNTREFORMBOOT' -N 0 -E offset=4194304 -d target-userland/boot -t ext2 reform-system.img 256M
+#	rm -rf target-userland/boot/*
+#	/sbin/mke2fs -v -L 'MNTREFORMROOT' -N 0 -O 64bit -E offset=272629760 -d target-userland -m 5 -r 1 -t ext4 reform-system.img 9000M
+#	rm -rf target-userland
+#else
+#	# if we don't run as root, use the slow (but bit-by-bit reproducible)
+#	# genext2fs instead
+#	./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland-full.tar \
+#		| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
+#	dd if=boot.img of=reform-system.img seek=1 bs=4194304
+#	rm boot.img
+#	./tarfilter --path-exclude='/boot/*' < target-userland-full.tar \
+#		| genext2fs --block-size 1024 --size-in-blocks 9216000 --bytes-per-inode 16384 --tarball - root.img
+#	dd if=root.img of=reform-system.img seek=65 bs=4194304
+#	rm root.img
+#fi
+#dd if=/dev/zero bs=512 count=1 >> reform-system.img
+#/sbin/parted -s reform-system.img "mklabel msdos"
+## reproducible disk signature
+#printf mntr | dd of=reform-system.img seek=440 bs=1 conv=notrunc
+#/sbin/parted -s reform-system.img "mkpart primary ext4 4MiB 260MiB"
+#/sbin/parted -s reform-system.img "mkpart primary ext4 260MiB 9260MiB"
+#/sbin/parted -s reform-system.img print
+#
+## install u-boot for i.MX8MQ
+#dd if=./flash.bin of=reform-system.img conv=notrunc bs=1k seek=33
+#rm flash.bin
+#
+#echo Reform Full System Image created: reform-system.img
diff --git a/reform2-imx8mq/mkuserland.sh b/reform2-imx8mq/mkuserland.sh
index c920736..2bbb915 100755
--- a/reform2-imx8mq/mkuserland.sh
+++ b/reform2-imx8mq/mkuserland.sh
@@ -25,8 +25,8 @@ mmdebstrap \
 	--essential-hook='mkdir -p "$1"/etc/flash-kernel/ubootenv.d' \
 	--essential-hook='mkdir -p "$1"/etc/flash-kernel/preboot.d' \
 	--essential-hook='echo "MNT Reform 2" > "$1"/etc/flash-kernel/machine' \
-	--essential-hook='{ echo /dev/mmcblk1p2 / auto errors=remount-ro 0 1; echo /dev/mmcblk1p1 /boot auto errors=remount-ro 0 1; } > "$1"/etc/fstab' \
-	--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc1,115200 console=tty1 root=/dev/mmcblk1p2\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi\"; } > "$1"/etc/default/flash-kernel' \
+	--essential-hook='{ echo /dev/mmcblk0p2 / auto errors=remount-ro 0 1; echo /dev/mmcblk0p1 /boot auto errors=remount-ro 0 1; } > "$1"/etc/fstab' \
+	--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc1,115200 console=tty1 root=/dev/mmcblk0p2\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi\"; } > "$1"/etc/default/flash-kernel' \
 	${comment#select timezone} \
 	--essential-hook='echo tzdata tzdata/Areas select Europe | chroot "$1" debconf-set-selections' \
 	--essential-hook='echo tzdata tzdata/Zones/Europe select Berlin | chroot "$1" debconf-set-selections' \

I keep meaning to play around with writing an alternative “emergency” image that loads from the eMMC, decrypts LUKS partitions, and allows kexecing into whatever bootable partitions it finds (like Petitboot, basically) but I haven’t got around to it yet. I think I just keep hoping that the eMMC+NVMe story will get improved in an update, though I feel a bit unreasonable about that, as it’s such a small team maintaining this and I haven’t even tried to help with that problem yet myself.

Anyway, I hope this was helpful.

Edit: My kernel command line argument changes don’t seem to be in that diff. I think I might have somehow changed them in uboot before compiling it, but I don’t have the diff for that at hand.

2 Likes

Thank you Lykso! I will read through this and report back when I get some time :slight_smile:

At the moment I have the ssd & Nvme working with the sd card. Still trying to figure out how to flash the recovery image to the EMMC. I’ve tried the echo command that makes it writable, and I’ve used dd to flash the image, but nothing changes. Still fuzzy on this because after reading other posts it seems like Lukas inferred that using the encrypted Nvme and booting without the sd card was possible.

Perhaps after I take the time to digest your comment it will make more sense. I’ve also been scared to run apt update even after flashing to the v3 image lol. Don’t want to set everything up again :wink:

If you unmount /boot and remove or comment out the mountpoint from your /etc/fstab file, you should be safe to upgrade everything. It won’t upgrade your kernel or any part of your boot partition, because it won’t be mounted.

E.g.:

sudo umount /boot
sudo sed -i 's|\(.\+\)\(/boot\)\(.\+\)|#\1\2\3|' /etc/fstab

It looks like Petitboot does already have ARM support (well, I see some people have it running on ODROID SBCs anyway), so maybe I’ll be able to just compile it and flash it to my eMMC. I’m a bit hesitant to do so just because this is my daily driver and so having it out of commission puts the brakes on everything else for a bit, but I’ll see what starting down that road looks like and report back once I’ve hit a good stopping point.

1 Like

That is expected. Your eMMC does not contain a v3 system and thus reform-boot-config will not do the right thing. Try booting from a sysimage-v3 SD-Card and then run reform-boot-config --emmc which will modify /boot on eMMC instead of /boot on the SD-Card.

2 Likes

Thank you Josch! That is likely it. Finally found some spare cycles to hack on the reform again. Going to spend the rest of today trying to get that working and will report back / close this thread. Certainly sounds the most promising plan of attack.

Also wanted to mention that mnt was awesome and sent me a new screen. I plan to swap that out tonight / maybe tomorrow. Was super grateful that Lukas sent one out.