A couple of days ago my Reform stopped booting. The last thing that happened to it was that it went into suspend, and attempting to wake it got the screen backlight to turn on, but didn’t recover the system. I used Circle + 0 to shut down and Circle + 1 to restart, and the screen backlight would turn on, but there was never any output to the screen.
#!/bin/bash
# Create the Debian environment
debootstrap unstable debian-chroot http://deb.debian.org/debian/
# Probably unnecessary (I think debootstrap handles
# this) but ensure chroot has network access
cp /etc/resolv.conf debian-chroot/etc/
# Run chroot stuff
cat <<EOF | chroot debian-chroot /bin/bash
# Get dependencies
apt install -y mmdebstrap \
genext2fs \
e2fsprogs \
binfmt-support \
git \
mount \
arch-test \
qemu-user-static \
parted
# Set locale to handle UTF-8
export LANG="C.UTF-8" && \
export LC_ALL="C.UTF-8"
# Get the code
git clone https://source.mnt.re/reform/reform-system-image.git
# cd and build
cd reform-system-image/reform2-imx8mq
./mkimage.sh
EOF
This successfully built reform-system.img and reform-rescue-system.img, but trying to boot from either of these results in a failure because u-boot can’t find a flattened device tree. I’ve mounted both of these images on my desktop and the device tree files seem to be there, and I’ve flipped the switch on the SoM to ensure that u-boot will be loaded from the SD card, so I’m not sure what the problem could be.
I’m a bit confused. You said that after a failure from resume (note that suspend/resume used to work but broke somewhere around linux 5.18) switching the system on just left the display black? Do you have a USB UART adapter that you can use to figure out what is happening?
Then you built your own system image (which looks like you did it correctly) and booting that gives you the display (was your old system using u-boot without display support) but also some error message. Do you have a more precise error message?
You do not necessarily need to flip the switch on the SoM as that will only choose where u-boot is loaded from but then u-boot will search through the first partitions of USB, SD-Card and eMMC (in that order) to find an extlinux.conf or boot.scr (in that order) and use that partition to boot if it finds it.
connect with a serial adapter to figure out what is going on
if you cannot get a serial console, take a photo of the screen so that we can see the exact error output (typing all of that is of course also possible but might be slower and prone to typos)
Right, the display remains black. I don’t have a USB-to-UART adapter on hand, unfortunately.
The error itself was “Did not find a cmdline Flattened Device Tree”. I don’t have a more complete message to reference at the moment because I downloaded the Gitlab CI artifact and it worked. There must be something wrong with the image I built; it was ~200 MB smaller than the CI image.
On the working CI image, I decrypted the nvme and ran fsck on my root filesystem, then rebooted. With the SD card inserted, it just mounts the SD card’s root. With the SD card removed, it stays on a black screen (this is after flipping the switch back to the original OFF position I started in, documented in document boot options (#2) · Issues · Reform / reform-handbook · GitLab).
I guess at this point, the next step is to overwrite u-boot on the emmc?
If you are interested, you could upload the image you built yourself somewhere and I could investigate and find out what went wrong with it.
I think it’s unlikely that something is wrong with your u-boot on emmc. I think it’s more likely that something is wrong with your /boot on emmc. I think what you could try would be to do this:
decrypt your nvme
mount your decrypted root on nvme to /mnt
mount your /boot on emmc to /mnt/boot
bind-mound /sys, /proc and /dev into /mnt as well
chroot into /mnt
run update-initramfs -u as well as flash-kernel
leave the chroot
unmount /boot, /sys, /proc and /dev from /mnt
unmount /mnt
close your luks
This will refresh the contents of your /boot on emmc with a newly generated initramfs as well as device tree as created by flash-kernel.
Another thing you can try is to insert a sd-card with u-boot on it but nothing else (no partitions) with the DIP switch set to ON. This will load a more recent u-boot with graphical support from your sd-card which will find your boot.scr on your emmc which will in turn boot from nvme. If there is an error during that process, you will see it on your screen thanks to more recent u-boot. Alternatively, you can also flash current u-boot to emmc of course.
I’ll work on getting that image uploaded and send you the link. I’ll try those steps to refresh /boot and update this post when done.
Update: flash-kernel fails:
Using DTB: freescale/imx8mq-mnt-reform2.dtb
Couldn't find DTB imx8mq-mnt-reform2.dtb in /usr/lib/linux-image- or /etc/flash-kernel/dtbs
Installing into /boot/dtbs//freescale/imx8mq-mnt-reform2.dtb
cp: cannot stat '': No such file or directory
ls /usr/lib/linux-image-*/freescale/imx8mq-mnt-reform2.dtb
/usr/lib/linux-image-6.1.0-8-reform2-arm64/freescale/imx8mq-mnt-reform2.dtb /usr/lib/linux-image-6.1.0-9-reform2-arm64/freescale/imx8mq-mnt-reform2.dtb /usr/lib/linux-image-6.1.0-reform2-arm64/freescale/imx8mq-mnt-reform2.dtb
ls -d /usr/lib/linux-image-*/
/usr/lib/linux-image-6.1.0-8-reform2-arm64/ /usr/lib/linux-image-6.1.0-9-reform2-arm64/ /usr/lib/linux-image-6.1.0-reform2-arm64/
Not sure about the formatting on this last one, I’m copying by hand:
dpkg -l | grep linux-image-
rc linux-image-5.18.0-reform2-arm64 5.18.5-1+reform1 arm64 Linux 5.18 for 64-bit ARMv8 machines
rc linux-image-5.19.0-reform2-arm64 5.19.11-1+reform20220925T130929Z1 arm64 Linux 5.19 for 64-bit ARMv8 machines
rc linux-image-6.0.0-reform2-arm64 6.0.12-1+reform20230104T204057Z1 arm64 Linux 6.0 for 64-bit ARMv8 machines
rc linux-image-6.1.0-7-reform2-arm64 6.1.20-2+reform20230419T235848Z1 arm64 Linux 6.1 for 64-bit ARMv8 machines
ii linux-image-6.1.0-8-reform2-arm64 6.1.25-1+reform20230428T084918Z1 arm64 Linux 6.1 for 64-bit ARMv8 machines
ii linux-image-6.1.0-9-reform2-arm64 6.1.27-1+reform20230522T044718Z1 arm64 Linux 6.1 for 64-bit ARMv8 machines
ii linux-image-6.1.0-reform2-arm64 6.1.12-1+reform20230215T190829Z1 arm64 Linux 6.1 for 64-bit ARMv8 machines
ii linux-image-arm64 6.1.27-1+reform20230522T044718Z1 arm64 Linux for 64-bit ARMv8 machines (meta-package)
reform-check says flash-kernel isn’t installed, then fails to download a file because the system isn’t connected to the internet:
reform-check
I: not installed: flash-kernel libreoffice libreoffice-gtk3 man-db plasma-discover plasma-nm plasma-workspace-wayland powerdevil swayidle swaylock systemsettings telnet
E: Failed to fetch https://source.mnt.re/reform/reform-boundary-uboot/-/jobs/artifacts/2023-01-25/raw/flash.bin?job=build Could not resolve 'source.mnt.re'
E: Download Failed
Update: connected to the internet and got some more useful output from reform-check. It was missing everything in /boot because I had mounted the root filesystem that was on emmc rather than the boot partition. Correcting this actually gives me output from update-initramfs -u (before it was apparently failing silently) and flash-kernel.
After running update-initramfs -u and flash-kernel, then powering off and removing the SD card, I’m able to get output on the screen on boot. Unfortunately, the Reform still fails to mount the root filesystem:
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
.
.
.
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for suspend/resume device
done.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/reformvg/root does not exist. Dropping to a shell!
BusyBox v1.35.0 (Debian 1:1.35.0-4+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
remove the kernel package linux-image-6.1.0-reform2-arm64. That one is an old kernel package that still suffers from a bug that I fixed in the other kernel packages (the -7, -8 and -9 packages) that you have installed.
if reform-check says that you are missing flash-kernel, please install it. Without it, you have to manually make sure that the dtb is in its place
if the initramfs cannot find /dev/reformvg/root then that hints that you generated that initramfs either on a system that was not your encrypted system or that you somehow removed important packages from that system or that you change the luks configuration in /etc of that system
Alright, got it working again! Thanks so much for your help.
I started by trying to install the packages that reform-check said were missing. apt said I needed to run dpkg-reconfigure -a, and running that finished the setup step for a bunch of package installations. This may have been the root cause of the failure to boot - I may have had an upgrade running that I forgot about when the system went into suspend. Once that finished, I was only missing swayidle, swaylock, and telnet, so I installed those. I also removed the old kernel.
I’m now able to boot my Reform normally without the SD card inserted.
Well that was “fun”. It seems that to download from that website, you first download something via javascript to a local cache which then gets decrypted and only then do you copy that to your disk. This killed firefox as it ended using up all my memory. I succeeded downloading that with the machine of my partner.
The reason that your image doesn’t work is that you are missing /boot/boot.scr or plain boot.scr in the first partition. The reason for that could be this bug:
No, I just used the script in the OP. I don’t remember if when I was originally putting that together I had already manually mounted /sys, /dev, etc. in debian-chroot, but for this time I just ran the script.