Encrypted NVME boot from EMMC confusions

megasquid · April 3, 2022, 9:23pm

Hey all DIY kit 157 checking in.

I received my reform mid week last week and got the chance to play with it over the weekend.

I’ve downloaded the reform v3 system image and flashed it to my SD card. I booted from the SD card and ran the reform-migrate encrypted Nvme command.

Booting from the Nvme with the SD card works well. However I want to use EMMC and no SD card. I flipped the switch on the SOM to off and now I boot into the recovery image.

From there I ran the reform-boot-config —EMMC Nvme. That worked fine, but on reboot I keep getting brought back to the recovery system image.

Due to the fact the Nvme is encrypted with luks, it can’t find the Nvme drive. The boot scripts are looking in the unencrypted Nvme drive locations when they should be looking for the encrypted drive.

I’ve looked at the reform-init script and tried playin around with setting the BOOTPART and other variables to get it to work how I’d like but have been unsuccessful.

From reading other threads it sounds like I shouldn’t have ran the reform-boot-config script?

I’d also like to mention that although my SD and Nvme are flashed with the v3 image I think my EMMC is running the v2 recovery image.

I’ve attached an image of the error I get upon booting. So this post is asking for help with how to get past this step. I need the EMMC to boot my encrypted Nvme, when it treats it like an unencrypted Nvme.

So far I’m loving my reform and can’t wait to contribute to the community!

The only downside I’ve experienced is that my display arrived broken. I think the screws were tightened too much during assembly prior to me receiving it. I’m lucky it’s usable for now. I’m sure support will help me out when they see my email

All in all the year long wait was well worth it! Such a beautiful device and cool community

megasquid · April 3, 2022, 9:32pm

At this point I’m thinking I need to flash the EMMC with the v3 recovery image?

If I do that, and reflash the Nvme with the v3 image using the reform-migrate command it sounds like I’d be in good shape?

I’m not sure how to flash the EMMC with the new v3 recovery image. I’d imagine I need to boot from the SD, then flash EMMC.

In following this line of thought, I reflashed my SD card so I could try to flash the EMMC & NVME but now I can’t boot from SD card. I can only boot into recovery mode via the EMMC…

Perhaps if I flash the reform v3 image to a usb, then I set reform-init with the usb flag, I can boot from that and reset everything? Not sure why reform-init with the sd card arg doesn’t actually force it to boot from the sd card, it still tries to boot from the Nvme…

I guess if I wanted it to boot from the sd card temporarily, I’d have to open the system back up and flip the switch on the SOM temporarily, then boot from the sd, flash the EMMC with the new image and then flip the SOM switch back?

Trying to avoid opening the reform back up if I have to. Not sure which route is the best here…

megasquid · April 3, 2022, 11:03pm

Well. I modified the reform-init poorly and can’t exit the script now .

So now I’m looking at how to reset the EMMC entirely by reflashing the recovery image.

Rereading the handbook to see where it mentions this now. Also opening the reform up and switching the SOM to ON so it hopefully auto boots from SD card.

lykso · April 5, 2022, 12:09am

The boot process could be better, from my perspective. I have the same setup as you, and I have to keep my boot partition unmounted because if I don’t, apt upgrade can result in my laptop not being able to boot. Last time I upgraded my kernel, I had to pull the reform-system-image and reform-boundary-uboot repositories, change some of the hard-coded paths, and compile uboot and the boot image myself. Took me quite a few cycles of what you’re doing now to get that worked out.

Here’s a post summarizing what I did: MNT Reform System Image V3 Beta - #16 by lykso

Here’s the output of git diff in my reform-system-image repo as it now stands:

diff --git a/reform2-imx8mq/mkimage.sh b/reform2-imx8mq/mkimage.sh
index 5d8fa18..19bfd6f 100755
--- a/reform2-imx8mq/mkimage.sh
+++ b/reform2-imx8mq/mkimage.sh
@@ -5,7 +5,7 @@ set -e
 
 # make sure build tools are installed
 # FIXME: replace this by a check that everything is installed
-#sudo apt-get -y install parted multistrap udisks2 gcc-aarch64-linux-gnu make device-tree-compiler qemu-user-static binfmt-support build-essential bison flex libssl-dev mmdebstrap
+sudo apt-get -y install parted multistrap udisks2 gcc-aarch64-linux-gnu make device-tree-compiler qemu-user-static binfmt-support build-essential bison flex libssl-dev mmdebstrap
 
 # if we are in a git repository and if SOURCE_DATE_EPOCH is not set, use the
 # timestamp of the latest git commit
@@ -20,70 +20,70 @@ export SOURCE_DATE_EPOCH
 export TMPDIR="$(pwd)"
 
 # download u-boot from CI
-/usr/lib/apt/apt-helper download-file https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/master/raw/flash.bin\?job\=build flash.bin
+#/usr/lib/apt/apt-helper download-file https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/master/raw/flash.bin\?job\=build flash.bin
 
 # build the debian userland and configure it
 ./mkuserland.sh
 
 # Rescue System ---------------------------------------------------------
 
-./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland.tar \
-	| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
-dd if=boot.img of=reform-rescue-system.img seek=1 bs=4194304
-rm boot.img
-./tarfilter --path-exclude='/boot/*' < target-userland.tar \
-	| genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball - root.img
-dd if=root.img of=reform-rescue-system.img seek=65 bs=4194304
-rm root.img
-dd if=/dev/zero bs=512 count=1 >> reform-rescue-system.img
-/sbin/parted -s reform-rescue-system.img "mklabel msdos"
-# reproducible disk signature
-printf mntr | dd of=reform-rescue-system.img seek=440 bs=1 conv=notrunc
-/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 4MiB 260MiB"
-/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 260MiB 2308MiB"
-/sbin/parted -s reform-rescue-system.img print
-
-# install u-boot for i.MX8MQ
-dd if=./flash.bin of=reform-rescue-system.img conv=notrunc bs=1k seek=33
-
-echo Reform Rescue System Image created: reform-rescue-system.img
-
-# Full System -----------------------------------------------------------
-
-# chroot into the userland and add extra applications
-./mkuserland3.sh
-
-if [ $(id -u) -eq 0 ]; then
-	# genext2fs is very slow so if we run this script as root, we give up
-	# on reproducibility in favor of creating the image faster
-	mkdir target-userland
-	tar --directory target-userland --xattrs --xattrs-include='*' --extract --file target-userland-full.tar
-	/sbin/mke2fs -v -L 'MNTREFORMBOOT' -N 0 -E offset=4194304 -d target-userland/boot -t ext2 reform-system.img 256M
-	rm -rf target-userland/boot/*
-	/sbin/mke2fs -v -L 'MNTREFORMROOT' -N 0 -O 64bit -E offset=272629760 -d target-userland -m 5 -r 1 -t ext4 reform-system.img 9000M
-	rm -rf target-userland
-else
-	# if we don't run as root, use the slow (but bit-by-bit reproducible)
-	# genext2fs instead
-	./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland-full.tar \
-		| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
-	dd if=boot.img of=reform-system.img seek=1 bs=4194304
-	rm boot.img
-	./tarfilter --path-exclude='/boot/*' < target-userland-full.tar \
-		| genext2fs --block-size 1024 --size-in-blocks 9216000 --bytes-per-inode 16384 --tarball - root.img
-	dd if=root.img of=reform-system.img seek=65 bs=4194304
-	rm root.img
-fi
-dd if=/dev/zero bs=512 count=1 >> reform-system.img
-/sbin/parted -s reform-system.img "mklabel msdos"
-# reproducible disk signature
-printf mntr | dd of=reform-system.img seek=440 bs=1 conv=notrunc
-/sbin/parted -s reform-system.img "mkpart primary ext4 4MiB 260MiB"
-/sbin/parted -s reform-system.img "mkpart primary ext4 260MiB 9260MiB"
-/sbin/parted -s reform-system.img print
-
-# install u-boot for i.MX8MQ
-dd if=./flash.bin of=reform-system.img conv=notrunc bs=1k seek=33
-rm flash.bin
-
-echo Reform Full System Image created: reform-system.img
+#./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland.tar \
+#	| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
+#dd if=boot.img of=reform-rescue-system.img seek=1 bs=4194304
+#rm boot.img
+#./tarfilter --path-exclude='/boot/*' < target-userland.tar \
+#	| genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball - root.img
+#dd if=root.img of=reform-rescue-system.img seek=65 bs=4194304
+#rm root.img
+#dd if=/dev/zero bs=512 count=1 >> reform-rescue-system.img
+#/sbin/parted -s reform-rescue-system.img "mklabel msdos"
+## reproducible disk signature
+#printf mntr | dd of=reform-rescue-system.img seek=440 bs=1 conv=notrunc
+#/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 4MiB 260MiB"
+#/sbin/parted -s reform-rescue-system.img "mkpart primary ext4 260MiB 2308MiB"
+#/sbin/parted -s reform-rescue-system.img print
+#
+## install u-boot for i.MX8MQ
+#dd if=./flash.bin of=reform-rescue-system.img conv=notrunc bs=1k seek=33
+#
+#echo Reform Rescue System Image created: reform-rescue-system.img
+#
+## Full System -----------------------------------------------------------
+#
+## chroot into the userland and add extra applications
+#./mkuserland3.sh
+#
+#if [ $(id -u) -eq 0 ]; then
+#	# genext2fs is very slow so if we run this script as root, we give up
+#	# on reproducibility in favor of creating the image faster
+#	mkdir target-userland
+#	tar --directory target-userland --xattrs --xattrs-include='*' --extract --file target-userland-full.tar
+#	/sbin/mke2fs -v -L 'MNTREFORMBOOT' -N 0 -E offset=4194304 -d target-userland/boot -t ext2 reform-system.img 256M
+#	rm -rf target-userland/boot/*
+#	/sbin/mke2fs -v -L 'MNTREFORMROOT' -N 0 -O 64bit -E offset=272629760 -d target-userland -m 5 -r 1 -t ext4 reform-system.img 9000M
+#	rm -rf target-userland
+#else
+#	# if we don't run as root, use the slow (but bit-by-bit reproducible)
+#	# genext2fs instead
+#	./tarfilter --path-exclude='*' --path-include=/boot --path-include='/boot/*' --strip-components=2 < target-userland-full.tar \
+#		| genext2fs --block-size 1024 --size-in-blocks 262144 --bytes-per-inode 16384 --tarball - boot.img
+#	dd if=boot.img of=reform-system.img seek=1 bs=4194304
+#	rm boot.img
+#	./tarfilter --path-exclude='/boot/*' < target-userland-full.tar \
+#		| genext2fs --block-size 1024 --size-in-blocks 9216000 --bytes-per-inode 16384 --tarball - root.img
+#	dd if=root.img of=reform-system.img seek=65 bs=4194304
+#	rm root.img
+#fi
+#dd if=/dev/zero bs=512 count=1 >> reform-system.img
+#/sbin/parted -s reform-system.img "mklabel msdos"
+## reproducible disk signature
+#printf mntr | dd of=reform-system.img seek=440 bs=1 conv=notrunc
+#/sbin/parted -s reform-system.img "mkpart primary ext4 4MiB 260MiB"
+#/sbin/parted -s reform-system.img "mkpart primary ext4 260MiB 9260MiB"
+#/sbin/parted -s reform-system.img print
+#
+## install u-boot for i.MX8MQ
+#dd if=./flash.bin of=reform-system.img conv=notrunc bs=1k seek=33
+#rm flash.bin
+#
+#echo Reform Full System Image created: reform-system.img
diff --git a/reform2-imx8mq/mkuserland.sh b/reform2-imx8mq/mkuserland.sh
index c920736..2bbb915 100755
--- a/reform2-imx8mq/mkuserland.sh
+++ b/reform2-imx8mq/mkuserland.sh
@@ -25,8 +25,8 @@ mmdebstrap \
 	--essential-hook='mkdir -p "$1"/etc/flash-kernel/ubootenv.d' \
 	--essential-hook='mkdir -p "$1"/etc/flash-kernel/preboot.d' \
 	--essential-hook='echo "MNT Reform 2" > "$1"/etc/flash-kernel/machine' \
-	--essential-hook='{ echo /dev/mmcblk1p2 / auto errors=remount-ro 0 1; echo /dev/mmcblk1p1 /boot auto errors=remount-ro 0 1; } > "$1"/etc/fstab' \
-	--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc1,115200 console=tty1 root=/dev/mmcblk1p2\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi\"; } > "$1"/etc/default/flash-kernel' \
+	--essential-hook='{ echo /dev/mmcblk0p2 / auto errors=remount-ro 0 1; echo /dev/mmcblk0p1 /boot auto errors=remount-ro 0 1; } > "$1"/etc/fstab' \
+	--essential-hook='{ echo LINUX_KERNEL_CMDLINE=\"console=ttymxc1,115200 console=tty1 root=/dev/mmcblk0p2\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi\"; } > "$1"/etc/default/flash-kernel' \
 	${comment#select timezone} \
 	--essential-hook='echo tzdata tzdata/Areas select Europe | chroot "$1" debconf-set-selections' \
 	--essential-hook='echo tzdata tzdata/Zones/Europe select Berlin | chroot "$1" debconf-set-selections' \

I keep meaning to play around with writing an alternative “emergency” image that loads from the eMMC, decrypts LUKS partitions, and allows kexecing into whatever bootable partitions it finds (like Petitboot, basically) but I haven’t got around to it yet. I think I just keep hoping that the eMMC+NVMe story will get improved in an update, though I feel a bit unreasonable about that, as it’s such a small team maintaining this and I haven’t even tried to help with that problem yet myself.

Anyway, I hope this was helpful.

Edit: My kernel command line argument changes don’t seem to be in that diff. I think I might have somehow changed them in uboot before compiling it, but I don’t have the diff for that at hand.

megasquid · April 5, 2022, 5:26pm

Thank you Lykso! I will read through this and report back when I get some time

At the moment I have the ssd & Nvme working with the sd card. Still trying to figure out how to flash the recovery image to the EMMC. I’ve tried the echo command that makes it writable, and I’ve used dd to flash the image, but nothing changes. Still fuzzy on this because after reading other posts it seems like Lukas inferred that using the encrypted Nvme and booting without the sd card was possible.

Perhaps after I take the time to digest your comment it will make more sense. I’ve also been scared to run apt update even after flashing to the v3 image lol. Don’t want to set everything up again

lykso · April 5, 2022, 8:47pm

If you unmount /boot and remove or comment out the mountpoint from your /etc/fstab file, you should be safe to upgrade everything. It won’t upgrade your kernel or any part of your boot partition, because it won’t be mounted.

E.g.:

sudo umount /boot
sudo sed -i 's|\(.\+\)\(/boot\)\(.\+\)|#\1\2\3|' /etc/fstab

It looks like Petitboot does already have ARM support (well, I see some people have it running on ODROID SBCs anyway), so maybe I’ll be able to just compile it and flash it to my eMMC. I’m a bit hesitant to do so just because this is my daily driver and so having it out of commission puts the brakes on everything else for a bit, but I’ll see what starting down that road looks like and report back once I’ve hit a good stopping point.

josch · April 7, 2022, 3:01pm

That is expected. Your eMMC does not contain a v3 system and thus reform-boot-config will not do the right thing. Try booting from a sysimage-v3 SD-Card and then run reform-boot-config --emmc which will modify /boot on eMMC instead of /boot on the SD-Card.

megasquid · May 22, 2022, 11:12pm

Thank you Josch! That is likely it. Finally found some spare cycles to hack on the reform again. Going to spend the rest of today trying to get that working and will report back / close this thread. Certainly sounds the most promising plan of attack.

Also wanted to mention that mnt was awesome and sent me a new screen. I plan to swap that out tonight / maybe tomorrow. Was super grateful that Lukas sent one out.

megasquid · June 15, 2022, 6:28am

Okay so I’m still confused on how to use the tools in /sbin and can’t seem to get encrypted nvme to boot from emmc.

Here is the process I’ve followed thus far, where am I going down the wrong path? I don’t mind wiping everything.

Download reform-system-image from gitlab.
Flash reform-system-image to sd card.
Boot from sd card.
Login as root.
Run reform-setup-encrypted-nvme
Run reform-boot-config --emmc nvme

After step 6 this is the output I get:

This script selects your preferred boot medium. It writes your choice to the file in /etc/fstab

[ 443.192891] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Quota mode: none.
mount: /mnt/boot: mount point does not exist.
dmesg(1) may have more information after failed mount system call. 
umount: /mnt/boot: no mount point specified.
umount: /mnt/dev: no mount point specified.
umount: /mnt/sys: no mount point specified.
umount: /mnt/proc: no mount point specified.

I’m thinking that I didn’t follow the output from step 5 properly. It suggests that I create a partition on the nvme drive… I logged in as a newly created user, went into sway, launched gnome-disks and formatted the nvme as an EXT4 partition for the full size of the nvme and then named it NMVE. I rebooted and followed step 6.

Once that didn’t work, I tried to follow the instructions that suggested running parted /dev/nvme0n1 "mklabel msdos", parted /dev/nvme0n1 "mkpart primary ext4 4MiB -1", and mkfs.ext4 /dev/nvme0n1p1. All of which work fine, but I’d think I don’t need to do that because the reform-setup-nvme script supposedly takes care of this already.

I end up with the same errors when I try to run reform-boot-config --emmc nvme.

I can boot into the emmc, and use no sd card, but I can’t boot into the nvme with no sd card. Which makes sense, because I can’t mount the drive…?

When I check lsblk I do see nvme0n1 with the partition nvme0n1p1 that was created from parted.

I know that the reform supports this because we have the reform-setup-encrypted-nvme, but I can’t for the life of me figure out what I’m not following.

Happy to try any other suggestions, quite lost and have been spinning my wheels trying this out over the past couple of months during free time. I need to try something else because re-running these scripts in different variations clearly is not working

Worth noting I am doing all of this with the switch flipped. So maybe I should remove the heatsink and flip the switch back? Should I be having the switch flipped one way for certain steps, and then the other way for other steps?

I’ve read through the handbook and many other threads in this forum but am not able to make heads or tales of this.

If someone is able to help me figure this out I would be happy to open a PR to the handbook / write up some documentation that lists out each and every step to make this work. Seeing as the code has been written script wise for this I’m sure it’s just something obvious I’m not seeing…

josch · June 15, 2022, 11:06pm

Hi @megasquid, I’m sorry to hear that you’ve made such bad experiences with setting up an encrypted nvme rootfs that boots from emmc. I am quite certain that this is due to my lack of documentation of the tools you used which I am apologizing for. I’ve made some attempts of documenting things in this issue: document boot options (#2) · Issues · Reform / reform-handbook · GitLab

This is not really proper documentation but more a collection of notes that somebody has to turn into proper legible docs as part of the official handbook. I’m bad at writing documentation so I hope this can be done by somebody who has that skill.

That being said, I want to say a few bits to the stuff you wrote about:

reform-setup-encrypted-nvme sets up a rootfs located at /dev/reformvg/root but reform-boot-config nvme expects the rootfs at /dev/mmcblk0p2 which is wrong in your case – you have to run this (very confusing) command instead: ROOTPART=/dev/reformvg/root reform-boot-config --emmc sd. This is not obvious and only sorta documented further down under the Examples section and I’ve yet to file this as a bug but so far this is thus left as a FIXME in my link.
reform-boot-config fails to run because it cannot mount /boot – the error message should be improved to give you a better idea of what is actually happening
The instructions involving running parted /dev/nvme0n1 "mkpart primary ext4 4MiB -1 do set up one large partition on nvme but the result will not be encrypted
You saying you are “doing all of this with the switch flipped” – is your dip switch flipped to ON or to OFF? As written in the link, you first want it set to ON (to load u-boot from sd-card) and only after you confirmed everything working, set it to OFF to load u-boot from emmc.
the handbook will not be of much help to you because it doesn’t (yet) document how sysimage-v3 works
i’d be super happy if you can contribute more docs or just things that need to be documented – thank you!!
and sorry for the mess

megasquid · June 16, 2022, 4:32am

@josch Thank you for taking the time to write and share these scripts, as well as help me troubleshoot!

The documentation write up looks awesome and I’m excited to follow suggested steps .

Happy to help review and battle test the steps and offer updates where I experience bumps along the way!

I will attempt to setup encrypted nvme with emmc again and report back .

megasquid · June 18, 2022, 7:48am

Wow. This is amazing. Able to boot into encrypted nvme without sd card

Thank you @josch!

It was because I didn’t have the dip switch set to ON when I was setting the boot config. After doing that, as well as using the ROOTPART env var as suggested in the docs it worked.

Once I get my reform fully setup and have some time I’ll attempt to make a PR for the FIXME you have in the docs.

I’m thinking it makes sense to add another step to the reform-setup-encrypted-nvme script that asks if you want to boot from SD or EMMC and then warns to say ensure the DIP switch is set to ON.

Any thoughts about that? Don’t want to attempt something that doesn’t make sense for the way this is designed.

josch · June 18, 2022, 7:57pm

Awesome that it worked for you!

I wouldn’t call that issue “docs” but I moved my FIXME comment to a proper issue for reform-tools and also added your suggestion to tell the user that to boot from emmc, the dip switch has to be set accordingly:

I’ll be on holidays for 10 days, starting 2022-06-22, so maybe I only get to fix those things afterwards. In any case, naturally anybody is welcome to comment on these issues or propose merge requests. I’d be happy to review them if you @ mention me. Thanks!

Jt0 · September 17, 2022, 11:27pm

This worked for me, but I want to make sure I understand it right (and possibly clear up confusion for others who come across this thread): if I understand the reform-boot-config script correctly, the “sd” here is only needed to prevent the script from printing usage information and exiting without doing anything, but doesn’t actually tell the script to configure the system to expect the root filesystem to be on an SD card since the ROOTPART variable is already set. The choice of “sd,” “nvme,” etc. only sets ROOTPART if the variable is not defined or empty, do I have that right?

josch · September 18, 2022, 10:39am

This was correct at the point when I wrote the message you quoted in June. In July I fixed this problem with reform-tools 1.8 and now you can just run this instead:

reform-boot-config --emmc /dev/reformvg/root