Upgrading from System Image Version 2 to Version 3 on NVME

So, I want to upgrade from my current system (which I believe uses System 2) to System Image V3 [1] which was released a while back.

The trouble is, all of the upgrade instructions assume a fresh install to an SD Card, which I really would rather avoid. I have a few things set up, including GNU Guix, in a way that would take a long time to rebuild should I need to start from scratch.

Is there a way to do this upgrade in a safe way without a full reinstall? Something akin to Debian’s apt’s dist-upgrade?

Being able to use “apt upgrade” is exactly what sysimage-v3 allows. Doing that without breaking the installation was not possible before. So there is no “apt upgrade” or any script that automatically converts you to sysimage-v3. It would be theoretically possible to write such a script but the number of possible configurations is very large which makes any such endeavor not only very time consuming but also very easy to fail if one forgets about any corner-case. Also, the number of installations is also much too small to get any amount of sensible testing of such a script and if something goes wrong, important data might be lost. Thus, to this day, no automated conversion script exists.

But it is possible to convert without doing a full reinstallation. But to be able to do so, one needs to have some knowledge about how all the parts are working together. Here are some notes about how to do a conversion:

But if you are worried about guix, are not all important bits contained in your $HOME directory and would you thus not just have to copy that?

Hey @josch, thanks for taking the time to respond to my question.

There’s no mutable state that would be lost outside of my /home directory, but I really don’t want to wait for everything under /gnu/store to rebuild if possible. I was under the impression that if I were to perform this upgrade, I would need to:

  1. Revert to booting entirely from the SDCARD, including flipping the switch on the motherboard to do so.
  2. Flash the new image over the old one using dd.
  3. Boot the new image and perform a new reform-migration to boot to the NVME drive again, which would necessitate removing all of the old files from the partition/reformatting the root partition so no mutable state remains from the old files, similar to how one one do so on a traditional debian install.
  4. Flip the switch back to boot from the NVME drive.

I’m assuming this because of my prior experience setting this up, but I did most of the non-guix work over a year ago, so I may be mistaken: Is it less involved than the above? Is there a more in-depth guide somewhere concerning the different ways to boot the MNT Reform? I am not familiar with uboot outside of my experience with this one machine, but I’m happy to learn!

I then assume you have the DIP switch on your SoM set up such that it reads u-boot from eMMC instead of from SD-Card? Depending on the version of u-boot you have on your eMMC, you don’t have to make any changes to that switch because the u-boot shipped in the MNT boundary-uboot repository currently (and I don’t know if that was always the case) prefers booting from SD-Card (if one exists) over eMMC. So before flipping that switch you might want to try just inserting an SD-Card and see if it automatically boots from it.

You mean flash it to the SD-Card using dd? If so, then yes. If you are talking about updating your eMMC, then there is now the reform-flash-rescue script which (if you have internet access) will do that for you because updating the eMMC (including u-boot) is a bit more involved.

A “traditional” Debian installation would be done with debian-installer which I tried but which still has some problems that are really tricky to fix because some of the things that are necessary for the Reform still need patches that are not in upstream Linux and others are non-free firmware blobs that cannot be shipped by Debian (not even in its non-free repository).

Unfortunately with sysimage-v3 the instructions in the handbook are quite outdated and sometimes wrong so I tried to collect some helpful hints here:

@sbates also added some instructions that might be helpful. If you have more to add, maybe it could be collected in that issue so that at some point in the future the handbook could be made up-to-date.

Hi!

I’m following the instructions in the linked doc to upgrade from 2 to 3.
I flashed the SD card, and flipped the DIP to boot from it.
Then I ran reform-flash-rescue, but I’m getting the following:

update-intramfs: Generating /boot/initrd.img-6.1.0-reform2-arm64
W: No zstd in /usr/bin:/sbin:/bin, using gzip
W: missing /lib/modules/6.1.0-reform2-arm64
W: Ensure all necessary drivers are built into the linux image!
depmod: ERROR: could not open directory /lib/modules/6.1.0-reform2-arm64: No such file ore directory
depmod: FATAL: could not search modules: No such file or directory

There are several variations of “No such file or directory” after that.

/lib/modules/6.1.0-reform2-arm64 exists, so I’m not sure what the problem is. Is this a known issue?

I should add that I got the SD card image I while ago (can’t remember when), but I can’t get the latest one because the artifact seems to have expired from Gitlab.

@violet recently managed to do the conversion manually. Maybe there is something useful for you in the IRC log:

https://mntre.com/reform-irc-logs/2023-07-06.log.html

So you flashed an SD-Card with the most recent sysimage-v3, booted from it, ran reform-flash-rescue and got that error message? I haven’t experienced this before.

Here is the last reform-system-image build from June:

Is that the one you tried it with? If yes, I will attempt to reproduce the issue.

Thanks for taking a look!

I did it with whatever you get when you run wget -O reform-system.img.gz https://source.mnt.re/reform/reform-system-image/-/jobs/artifacts/main/raw/reform2-imx8mq/reform-system.img.gz\?job\=build

I decided to try running reform-setup-encrypted-nvme instead, and it failed because /dev/mmcblk1p1 is still mounted somewhere.

I’m currently trying it again after a umount.

OK, reform-setup-encrypted-nvme worked after a umount.

I left boot on the SD card and rebooted. Root is on /dev/mapper/reformvg-root.

I ran reform-boot-config --emmc /dev/reformvg/root, and it complained that /boot was mounted, so I unmounted it and ran it again, and got the same depmod: ERROR: could not open directory /lib/modules/6.1.0-reform2-arm64: No such file ore directory errors as above.

EDIT: It’s really late where I am so I’m going to get some rest and try again in the morning.

Hey quick update. Typing this from my fully-upgraded MNT Reform!

Thank You for those links; they were crucial to my successful migration. Biggest hurdle (hinted at in the IRC logs) was the stateful reform-flash-* scripts on the system image provided; The Checksum mismatch is apparently due to an updated reform-check et al that is not on that image, along with a few other changes. Running a full apt update && apt upgrade on the system image, not on the prior installation, was required for me to be able to use the reform-tools.

Unfortunately, having taken a page from violet’s experience and decided to flash the system image directly to my emmc when it was not working, I ran into another barrier: There is not enough free space on the emmc at that point to allow the apt upgrade to take place. I had to work around this by mounting my nvme /home partition on /var/cache/apt/ to give it enough space to download everything.

In summation, the steps I took were:

  1. Run apt update and apt upgrade on my prior OS.
  2. Run apt install linux-image-arm64 explicitly, as for some reason it wasn’t installed. reform-tools was, though.
  3. Run the commands in the script to remove the unneeded parts from the prior installation.
  4. Run reform-check a bunch, addressing the issues mentioned. Specifically, I needed to install a bunch of packages and fix my fstab a bit (though this was overwritten later).
  5. Configure my MNT Reform to boot to the emmc using reform-boot-config.
  6. Reboot.
  7. Mount one of my nvme partitions on /var/cache/apt to give apt more space.
  8. Connect to my internet.
  9. Run apt update && apt upgrade in the emmc enviroment.
  10. Make sure my /etc/fstab in the emmc environment has a /boot/ entry.
  11. Run the commands You mentioned in the IRC log:
    • { echo LINUX_KERNEL_CMDLINE=\"console=ttymxc0,115200 console=tty1\"; echo LINUX_KERNEL_CMDLINE_DEFAULTS=\"ro no_console_suspend cma=512M pci=nomsi\"; } > /etc/default/flash-kernel
    • printf "pwm_imx27\nnwl-dsi\nti-sn65dsi86\nimx-dcss\npanel-edp\nmux-mmio\nmxsfb\nusbhid\nimx8mq-interconnect\n" > /etc/initramfs-tools/modules
    • echo "MNT Reform 2" > /etc/flash-kernel/machine
  12. Run reform-check a bunch, addressing the issues mentioned. Specific issues here included needing to update uboot on everything (reform-flash-uboot; reform-flash-uboot emmc; reform-flash-uboot sd) and having to clean up some errant files.
  13. Perform a fresh reform-migrate to the first partition on my NVME drive, allowing old files to be overwritten with new ones.
  14. Run reform-boot-config nvme to boot back to the right place.
  15. Reboot.
  16. Remove the SD Card I ended up not really using, as after updating my emmc uboot it will always be the boot target from emmc, as You mentioned.
  17. Reboot again.
  18. Finally in my nvme OS again, but a few stateful things aren’t working on the debian side, and /etc/fstab has been completely overwritten.
  19. Run apt --fix-broken install to fix the conflicts introduced by copying one debian over another.
  20. Run systemctl list-units --failed, mine had 3 issues:
    • I needed to run apt reinstall openssh-server avahi-daemon exim4 rtkit to let Debian do its thing.
    • I needed to delete the old /var/log/exim4/* files, as they were not accessible by the new exim4 user.
    • I needed to ignore the failed rtkit service, as it no longer exists and disappeared after a reboot.
  21. I had to reset my hostname using hostnamectl.
  22. I had to recreate my user, because (obviously) they were not in the new system yet, despite their homer directory existing (do this before You mount /home so that You don’t need to move Your files!).
  23. I had to add my user to the dialout group, as that’s the only supplemental group that’s missing from the defaults.
  24. /etc/fstab needed to be fully recreated; This prevented Guix from working at all up until this point, as the store is on a seperate partition. Once /etc/fstab had everything, I ran systemctl daemon-reload and mount -a.
  25. I then noticed my time was way off, so I ran timedatectl set-timezone to set my local time zone.
  26. I also needed to re-setup a bunch of other Guix things; After downloading the binary install script again, I reran a few of its sections:

NOTE: If You run Guix, this edit bypasses a bunch of the safety checks and stuff. Might be worth rerunning the script as a whole first and seeing if it works for You.

_msg()
{ # Default message to stdout.
    printf "[%s]: %s\n" "$(date +%s.%3N)" "$1"
}

_debug()
{
    if [ "${DEBUG}" = '1' ]; then
        printf "[%s]: %s\n" "$(date +%s.%3N)" "$1"
    fi
}

sys_create_build_user()
{ # Create the group and user accounts for build users.

    _debug "--- [ ${FUNCNAME[0]} ] ---"

    if getent group guixbuild > /dev/null; then
        _msg "${INF}group guixbuild exists"
    else
        groupadd --system guixbuild
        _msg "${PAS}group <guixbuild> created"
    fi

    if getent group kvm > /dev/null; then
        _msg "${INF}group kvm exists and build users will be added to it"
        local KVMGROUP=,kvm
    fi

    for i in $(seq -w 1 10); do
        if id "guixbuilder${i}" &>/dev/null; then
            _msg "${INF}user is already in the system, reset"
            usermod -g guixbuild -G guixbuild${KVMGROUP}     \
                    -d /var/empty -s "$(which nologin)" \
                    -c "Guix build user $i"             \
                    "guixbuilder${i}";
        else
            useradd -g guixbuild -G guixbuild${KVMGROUP}     \
                    -d /var/empty -s "$(which nologin)" \
                    -c "Guix build user $i" --system    \
                    "guixbuilder${i}";
            _msg "${PAS}user added <guixbuilder${i}>"
        fi
    done
}

sys_create_shell_completion()
{ # Symlink supported shell completions system-wide

    var_guix=/var/guix/profiles/per-user/root/current-guix
    bash_completion=/etc/bash_completion.d
    zsh_completion=/usr/share/zsh/site-functions
    fish_completion=/usr/share/fish/vendor_completions.d

    { # Just in case
        for dir_shell in $bash_completion $zsh_completion $fish_completion; do
            [ -d "$dir_shell" ] || mkdir -p $dir_shell
        done;

        ln -sf ${var_guix}/etc/bash_completion.d/* "$bash_completion";
        ln -sf ${var_guix}/share/zsh/site-functions/* "$zsh_completion";
        ln -sf ${var_guix}/share/fish/vendor_completions.d/* "$fish_completion"; } &&
        _msg "${PAS}installed shell completion"
}

sys_maybe_setup_selinux()
{
    if [ -f /sys/fs/selinux/policy ]
    then
	prompt_yes_no "Install SELinux policy required to run guix-daemon?" \
	    || return

	local var_guix=/var/guix/profiles/per-user/root/current-guix
	semodule -i "${var_guix}/share/selinux/guix-daemon.cil"
	restorecon -R /gnu /var/guix
    fi
}

sys_create_build_usr
sys_create_shell_completion
sys_maybe_setup_selinux

After all of that, I have a fully working MNT Reform on System Version 3 that boots from emmc to an NVME drive, without having to start completely from scratch.

Upon reflection, would probably have been easier to just fully start over with a fresh root partition, but I’m very stubborn, haha. I wanted to share my full experience in case it’s useful to anyone else trying to do the same thing.

I think my main barrier was that I have limited experience with Uboot, the ARM architecture in general, and the reform community in general (for news about updates and features; My machine had gotten very out of date). Here’s a brief list of things I learned about along the way that I couldn’t find clear documentation for before taking these last few days; Might try to write a guide to it somewhere eventually:

  • How Uboot actually works, including /boot.scr and the way the emmc uses it.
  • The differences between /dev/mmcblk0boot1, /dev/mmcblk0p1, /dev/mmcblk1p1, and how the system uses them.
  • What reform-boot-config, reform-check, reform-flash-*, and friends do.

Thanks for Your help and links! I learned a lot, and am happy to be a part of this community!

2 Likes