Linux 7.3 on imx8m+ Pocket Reform is now fixed and boots again

Hi,

I wanted to write a short post (whoops, seems it’s gotten 1.7k words long…) about the recent problem discovered by @kensanata in No boot after sudo apt update && sudo apt full-upgrade - #10 by kensanata

If you are on imx8m+ and have not experienced any issues, then that’s great. If you upgrade today you should not run into the same problem. This post is for those who would like to get some insights in some of the things that kept me awake the past few nights. :smiley:

What happened? Most SoMs for the Reform can read the bootloader (like u-boot) from SD-card if they cannot find one on eMMC (for example after writing zeroes to where the bootloader should be). This means that it’s relatively safe to test new versions of u-boot on an SD-card and only flash them to eMMC after they were found to work. imx8m+ is different as it exclusively loads u-boot from eMMC. If you flash a broken u-boot, your system won’t boot anymore and thus you cannot easily flash a working bootloader. Recovery is possible but not for the faint of heart, so reform-check as well as reform-flash-bootloader print a big warning if a u-boot upgrade on im8xm+ is attempted.

This is why many (most?) people with imx8m+ are still on an ancient u-boot version and we have to live with a whole lot of baggage which we keep carrying forward to support that old u-boot version. For example, ancient imx8m+ uboot does not set the ${fdtfile} environment variable and thus has no idea which device tree to load. This is why we still have a /boot/dtb-* symlink which is updated by flash-kernel to link to the correct device-specific device tree. Why is this a problem? Because it makes the system image platform specific (compressed images for all platforms are more than 30 GiB these days). With modern u-boot one image (we call it reform-system-any) can boot on all platforms. It would be nice to be able to drop these kind of workarounds but we cannot because it would remove support for older platforms.

Thanks to @kensanata we just discovered a new problem of imx8m+ u-boot which we now have to work around and carry with us forever going forward because we cannot require that everybody upgrade their bootloader. The memory locations for kernel, device tree and initramfs on imx8m+ u-boot are hardcoded like this:

  • ${kernel_addr_r} = 0x40480000
  • ${fdt_addr_r} = 0x43000000
  • ${ramdisk_addr_r} = 0x43800000

This means that if the kernel grows beyond 45613056 bytes (43.5 MiB), it will clash with location for the device tree in memory.

With kernel 7.3 the kernel grew to 44481024 bytes (42.42 MiB) and imx8m+ devices no longer booted with this on serial:

Found U-Boot script /boot.scr
5643 bytes read in 4 ms (1.3 MiB/s)
## Executing script at 40480000
44481024 bytes read in 1789 ms (23.7 MiB/s)
56355 bytes read in 7 ms (7.7 MiB/s)
41788646 bytes read in 1732 ms (23 MiB/s)
Booting Debian 7.3.3-mnt-reform-arm64 from mmc 0:1...
Moving Image from 0x40480000 to 0x40600000, end=43160000
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
script exited: continuing...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@30b40000.blk...
Scanning disk mmc@30b60000.blk...
Found 6 disks
No EFI system partition
fail to find output device
ERROR: invalid device tree
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
5486 bytes read in 2 ms (2.6 MiB/s)
## Executing script at 40480000
script exited: continuing...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
ERROR: invalid device tree

Why was there a problem even though the kernel did not exceed 43.5 MiB? Because u-boot shifts the kernel location to the next 2 MiB boundary and that then lets it clash with the device tree location.

The most obvious fix is to just change the memory addresses in u-boot and give more room to the kernel. This is what I did here:

But that would require all imx8m+ users who want the more recent kernel to upgrade their u-boot. Because many (most?) don’t read the forum before they upgrade (why would they, it’s supposed to just work) we’d have lots of users whose systems suddenly became unbootable and to fix that, they’d have to flash a system image to sd-card and upgrade their u-boot through that. Very much not ideal.

Fortunately, our users still have flash-kernel installed. We want to get rid of it as even its maintainer @vagrantc keeps pointing out (and I agree). Alas we cannot because we still need to support ancient u-boot versions and in this case, the fact that boot.scr is written in a turing complete mini-shell language comes to the rescue.

u-boot will first try booting using /boot/extlinux/extlinux.conf. This fails because the load addresses hardcoded in u-boot let the kernel overwrite the device tree location when the kernel becomes too large:

Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
U-Boot menu
1:	Debian GNU/Linux forky/sid 7.3.3-mnt-reform-arm64
2:	Debian GNU/Linux forky/sid 7.3.3-mnt-reform-arm64 (rescue target)
Enter choice: 1:	Debian GNU/Linux forky/sid 7.3.3-mnt-reform-arm64
Retrieving file: /initrd.img-7.3.3-mnt-reform-arm64
Retrieving file: /vmlinuz-7.3.3-mnt-reform-arm64
append: ro no_console_suspend cryptomgr.notests plymouth.ignore-serial-consoles pcie_aspm=off ro no_console_suspend pci=pcie_bus_perf nvme_core.default_ps_max_latency_us=0 console=ttymxc1,115200 console=tty1 cma=256MB console=tty1
Retrieving file: /dtbs/7.3.3-mnt-reform-arm64/freescale/imx8mp-mnt-reform2.dtb
Moving Image from 0x40480000 to 0x40600000, end=43160000
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
2:	Debian GNU/Linux forky/sid 7.3.3-mnt-reform-arm64 (rescue target)
Retrieving file: /initrd.img-7.3.3-mnt-reform-arm64
Retrieving file: /vmlinuz-7.3.3-mnt-reform-arm64
append: ro no_console_suspend cryptomgr.notests plymouth.ignore-serial-consoles pcie_aspm=off ro no_console_suspend cryptomgr.notests plymouth.ignore-serial-consoles pcie_aspm=off ro no_console_suspend pci=pcie_bus_perf nvme_core.default_ps_max_latency_us=0 console=ttymxc1,115200 console=tty1 cma=256MB console=tty1 console=tty1 single
Retrieving file: /dtbs/7.3.3-mnt-reform-arm64/freescale/imx8mp-mnt-reform2.dtb
Moving Image from 0x40480000 to 0x40600000, end=43160000
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
SCRIPT FAILED: continuing...

u-boot will then try the next method: /boot/boot.scr and that file is generated by flash-kernel and can include our own script snippets. To work around this problem without requiring everybody to upgrade their u-boot I patched /etc/flash-kernel/ubootenv.d/00reform2_ubootenv such that if the currently booting system is imx8m+, the memory addresses for kernel, device tree and initramfs would be different and allow for a larger kernel. This is implemented in this MR:

To quickly deploy that fix, we added a hotfix patched to reform-tools in the MNT Debian repository via the reform-debian-packages repository:

While I was at it, I also adjusted the location of the kernel to be aligned with a 2 MiB offset to avoid the extra time spent on each boot to move the kernel around in RAM.

This worked insofar that even if you have old u-boot, the kernel, device-tree and initramfs would be successfully loaded and start booting even on old u-boot because the offsets are adjusted in boot.scr. But a problem remains for those who have u-boot-menu installed. The package is installed by default because we’d prefer to use extlinux.conf for booting. It is preferred by u-boot and in contrast to boot.scr it is not turing complete but a simple machine-readable declarative configuration file. In contrast to boot.scr you can also just edit it by hand to try things out or repair things. You cannot just edit boot.scr because it contains a tiny binary checksum at the start of the file and if you attempt to edit it, u-boot will refuse to load it. Unfortunately, extlinux.conf has one disadvantage: it requires that the env variables set by u-boot are correct. Ancient imx8m+ u-boot does not set ${fdtfile} and now we learned that the hardcoded memory offsets are also unfit for the ever growing Linux kernel.

So what is the problem? As we saw above, using extlinux.conf will fail and u-boot falls back to using boot.scr. The problem is, that before using extlinux.conf failed, u-boot tried to load all entries of extlinux.conf. All entries fail because the kernel became too large. Unfortunately the last entry in extlinux.conf is the one for the single-user rescue system and it will add the argument single to ${bootargs} which forms the kernel cmdline. And then for whatever reason, when boot.scr loads, it keeps that last cmdline and does not initialize it anew. So the single argument will remain in the kernel cmdline.

We knew about this issue already and reform-check will emit a warning about it since version 1.83:

W: Your kernel cmdline has 'single' in it and will boot into rescue mode on each boot. Maybe booting via extlinux.conf failed?

So once we are in boot.scr we’d like to remove the single argument from the cmdline or otherwise on every boot, the boot process will halt and ask the user to press Ctrl+D for an emergency console or Enter to continue. Theoretically, the limited shell language of boot.scr allows for some string manipulation and even regex matching but ancient u-boot does not have these options turned on so we cannot rely on that feature. I also didn’t find a good way to do string manipulation. This is not POSIX shell. What I ended up doing is to check whether we booted with an old u-boot by inspecting the value of ${kernel_addr_r}. If it’s the old value, it must be old u-boot and then I simply reset the value of ${bootargs} to a known good default.

As you are reading this, all of this has been deployed and I think you can now upgrade without having to worry about this issue.

But the kernel will keep growing because the Debian kernel tries to be a universal kernel which runs on all kind of hardware. We don’t want to ship a specialized kernel (even though there are good reasons for doing so) because it would be nice if one day the MNT Debian repository wasn’t even needed anymore because all patches are upstreamed. So instead of removing from the Debian kernel, we add on top in a way which can hopefully be upstreamed one day.

So how do we stop this from happening in the future? I investigated all SoMs for the MNT Reform I had at home and created a table with the memory offsets for kernel, device tree and initramfs for each platform (imx8mq, imx8m+, a311d, ls1028a, rk3588) to figure out what our limits actually are. You can find the data I collected in this MR:

In summary:

  • max kernel: 46.0 MiB (imx8m+)
  • max initrd: 192.0 MiB (imx8mq)
  • max dtb: 1.5 MiB (rk3588)

Technically, maximum dtb size is platform dependent, but even the largest of our dtbs does not exceed 0.18 MiB and the largest of all dtbs shipped by Debian is 0.218 MiB we just make sure that all of them are below 1.5 MiB.

We cannot check initrd size here but its current size (40 MiB) is far below the lowest maximum of 192.0 MiB.

If a kernel image should start exceeding 46.0 MiB, the reform-debian-packages gitlab CI pipeline will fail and the resulting image will never appear in the MNT Debian repository.

If you find bugs in anything I described here, please send them over!

10 Likes

2 posts were split to a new topic: Imx8m+ with extlinux.conf boots kernels older than 7.3 in rescue mode