Booting from SD, encrypted / on NVME

The title states what I would like to do, but which I have not managed to do so far. I have run reform-setup-encrypted-nvme, agreed to run reform-migrate at the end of the script, then answered the question that I would like to keep /boot on the SD card. After rebooting, / is still taken from the SD card.

I have tried to run “reform-boot-config nvme”, but it complains about /dev/nvme0n1p1 not being a block device. Indeed the partition does not exist: The reform-setup-encrypted-nvme script apparently encrypts the complete disk and then does some LVM magic (with which I am not familiar).

Alternatively “reform-boot-config /dev/nvme0n1” also does nothing (and I am anyway afraid it will not handle decryption and lvm correctly and leave me with an unbootable system).

What would be the next step? My impression is that the two scripts reform-setup-encrypted-nvme and reform-migrate are somewhat out of sync.

An option would be to create a simple encrypted partition on the NVME without lvm, as apparently is recommended by the Reform handbook. But how to set up automatic decryption then?

Or would it be better or at least simpler to keep everything on the SD card and move only /home to an encrypted partition on the NVME? (If I understand correctly, this means creating the encrypted partition once and for all, then editing /etc/crypttab and /etc/fstab to automate decryption and mounting.) This also looks like the safest option to recover from when something goes wrong, like the SD card breaking down, which I suppose is bound to happen after a while.

When reform-setup-encrypted-nvme finished, does it print the following message?

/dev/mmcblk1p1 is still in use

If /dev/nvme0n1p1 is not a block device, then you probably never ran reform-flash-rescue to update your rescue system on eMMC to sysimage-v3? You can only enable your eMMC as the boot device with sysimage-v3 on nvme if eMMC has a compatible partition layout and reform-flash-rescue will do the required setup.

I think the problem is reform-boot-config but to make sure, please first confirm my initial question above.

You set up automatic decryption by writing the correct entry into /etc/crypttab and /etc/fstab and then regenerating your initramfs. You can read more about that in the reform-setup-encrypted-nvme script. I can also tell you more about the manual route, but maybe lets first see which bug you encountered and fix that instead, so that others don’t run into the same problem.

Your NVMe is probably supporting way more I/O than your SD-card. I think it makes sense to move all you can to NVMe. If you want to keep using your SD-Card for parts of your system, you can keep having /boot on it. The contents of /boot are only rarely updated. Personally, I have even my /boot on eMMC and my SD-Card only has u-boot on it. Another reason to have everything on eMMC is, that by running the default layout it will become easier to debug any potential problems in the future.

Maybe this is an interesting resource for you?

2 Likes

When reform-setup-encrypted-nvme finished, does it print the following message? /dev/mmcblk1p1 is still in use

Something like this. When I now run “reform-boot-config sd” (which should change nothing), it prints /dev/mmbcl1p2 is still in use, which is the / taken from the SD card.

If /dev/nvme0n1p1 is not a block device, then you probably never ran reform-flash-rescue to update your rescue system on eMMC to sysimage-v3? You can only enable your eMMC as the boot device with sysimage-v3 on nvme if eMMC has a compatible partition layout and reform-flash-rescue will do the required setup.

I do not want to boot from eMMC, but keep booting from SD (my first action with the Reform was to “apt dist-upgrade” to a kernel with problems that would not boot; I was happy that I could just dd a working image to the SD card and start afresh, and as someone said in another recent thread, I am afraid of bricking my system otherwise). I just wanted to have /boot on the SD and an encrypted / on NVME, so running reform-flash-rescue should not make a difference. I just did so nevertheless, and ended up with a frightening message that my system would now boot from eMMC. Luckily (maybe because I did not touch the DIP switch) it still booted from SD card.

Maybe this is an interesting resource for you?

Indeed this clarifies a lot of things, thank you for writing it up! It also explains that my fear mentioned in the previous paragraph seems to be unjustified; if now I understand things correctly, the documentation of reform-boot-config is quite misleading: It does not “select your preferred boot medium”, but only the location of the / partition.

Your (almost) last example explains that running reform-setup-encrypted-nvme and reform-migrate should achieve what I wanted to do: “Your system is now configured such that it will load u-boot, kernel and initrd from your sd-card and the rootfs from your encrypted nvme ssd.” So indeed there seems to be a bug in one of the scripts.

I might still go for the route of having only /home on the encrypted NVME (as well as installing the Guix package manager with /gnu on the NVME, so that keeping the Debian software on the SD card would not make much of a difference), but I will be happy to test any changes you make to the scripts.

I don’t understand. What was the exact message? Reading the code, there is only this message:

If the SoM dip switch is turned off and no SD-Card is present, your
system will now boot from eMMC and load the rootfs from there as well.

Which documentation are you referring to here?

That bug should be fixed now. Could you either apt-get upgrade or flash the latest system image and try running reform-setup-encrypted-nvme again?

I do not remember any more. It could also have been the message from reform-boot-config --emmc , run from within reform-flash-rescue, that says

Your boot partition is on emmc

I tend to look for help output by running commands as reform-boot-config --help; this has the same effect here as not giving a command line switch, which still does output a help text, which starts like this:

This script selects your preferred boot medium. It writes your choice to the file /etc/fstab.

When you come from x86 BIOS, this sounds as if it fixes the boot order; I deduced that if I ran reform-boot-config emmc and somehow broke things on emmc, the machine would try to boot from emmc, fail, and then not boot at all any more; and without being able to run reform-boot-config sd, I did not see how to switch the boot order back to prefer the SD card. But the script is much less dangerous, from your detailed explanation I understood that even --emmc does a bit less, and continues to “boot from SD” in the sense that it takes uboot there unless the DIP switch is flipped. It would be nice to merge your text into the official handbook (and replace some /sbin/reform-* by /usr/sbin/reform-* in the process).

Thanks, will do!

When reporting problems, please be precise and exactly copy&paste error messages. Otherwise it becomes very hard or impossible to help.

That is good practice. Unfortunately, the scripts do not yet produce good --help output. This has to be fixed.

In fact, you cannot change the “boot order” at all. :slight_smile: What you can change is whether u-boot is loaded from eMMC or from your SD-Card. And then u-boot has a strict boot order, trying out all of the devices it can boot from until it finds one where it succeeds. The former is changed via the DIP-switch on the SoM. The latter is changed by recompiling u-boot. What reform-boot-config selects is where your / will come from and it stores that information in the initramfs in the /boot of your choice.

Yes, that’s why it’s filed as an issue against reform-handbook. The handbook has to be updated in several places to be consistent with the current state of the software. But this is a big effort.

Let us know what happend! :slight_smile:

I suppose nothing: I wanted to try the apt update; apt upgrade route, but reform-tools is not among the packages that would be upgraded. In my /etc/apt/sources.list, I have

deb [trusted=yes] Index of /reform-debian-repo/ reform main

Would I need something else?

Well, these are just shell scripts; I suppose I can just copy reform-setup-encrypted-nvme and reform-boot-config from git, which are the only scripts that were changed recently.

Ah you are right. The packages are rebuilt weekly. The last rebuild was 3 days ago but I only fixed the problem afterwards. So the repo still has the old version. I just manually scheduled a rebuild. The package should show up in the repos in 30 minutes to 1 hour.

Yes, that would work too if you do not want to wait.

I used the scripts from git, and everything worked very well. Thanks a lot for your quick fix! This is very comfortable now.

Nice!!! Thanks for confirming that my fix indeed did what I expected it to do in your case. :slight_smile:

If you find more problems with the image, please report them as well, either here in the form (you can @mention me), directly on https://source.mnt.re or via private mail to josch@debian.org (public disclosure is preferred though) – thanks!

1 Like

Sure, although to be honest I do not expect to find more problems (also because once everything is set up, I should not need to use the reform tools any more).