A311D I broke and am in the process of recovering

[Savasten]

So I ran into a bug after upgrading to kernel 6.18.3 where my NVMe and thus my encrypted volume was not found.

System: Pocket reform, system controller FW 20251118, Keyboard custom map build of 2025118, OS Debian Trixie migrated to sid with reform repos as well, Running sway and hyperland via tuigreet. emmc boot of encrypted NVME.

I play around and break my system often.

I tried to recover my booting system by booting my SD card and using chroot.

from earlier posts:

@josch

$ cryptsetup luksOpen /dev/nvme0n1 reform_crypt
$ vgchange -ay reformvg
$ mount /dev/reformvg/root /mnt
$ mount /dev/mmcblk1p1 /mnt/boot
$ mount -o bind /dev /mnt/dev
$ mount -t sysfs sys /mnt/sys
$ mount -t proc proc /mnt/proc
$ cp /etc/resolv.conf /mnt/etc
$ chroot /mnt bash

I tried updating the system and other misc things with no luck not understanding the kernel issue at the time. I ended up running reform-check and saw the messages about u-boot not being up to date.

I ran the reform-flash-uboot sd and reform-flash-uboot emmc . That was a mistake. I rebooted to a system that would power on but not boot from emmc or sd card. Just keyboard lights and dark screen. I was able to get a UART log attached below. (I did struggle with the pin numbering on the board. I finally figured it out)

https://pastebin.com/ArJMRgxD

After that I figured that I broke the EMMC boot process. I ordered a CM4 IO board and figured out the steps to recover SD card booting.

Power up the board in USB boot, flash android 9, then flash android again but this time disconnect the USB power at the end of the 7% formatting stage. From there I could boot the Debian Banana Pi image and dd zero boot0, boot1, and the main partition of the emmc.

I have reassembled the system and it now will boot to the SD card again.

I am partially back to the beginning of the issues I caused.

I would like to rebuild/image the emmc back to the standard layout.

Current layout:

/dev/mmcblk1boot0 4 Mib

/dev/mmcblk1boot1 4 Mib

/dev/mmcblk1p1 14.6G boot=set type=Linux 83 ext4 - This currently is set as the boot directory for my NVME os

My goal is to revert my wrongs and get my old system booting. If not I am fine reloading the system but that seems less fun.

Help needed:

• Correct emmc layout and data including u-boot
• How to fix u-boot to point to the encrypted nvme as the root partition.
• I am not sure where the default swap drive was located.

Thank you everyone for all you help and sorry about the long message.

Keith

[josch]

@Savasten

I ran the reform-flash-uboot sd and reform-flash-uboot emmc . That was a mistake.

If you are unsure about u-boot on a311d, zero-out emmc (you can use reform-flash-uboot --zero emmc for that) and flash u-boot to sd-card. A311D will only read u-boot on sd-card if it cannot find anything on emmc. So flashing to both sd and emmc does not make sense.

When I tested the most recent u-boot for a311d I indeed tested it only on classic reform. I did not test a311d pocket reform so it is possible that you are encountering an a311d-pocket specific issue.

@Savasten

From there I could boot the Debian Banana Pi image and dd zero boot0, boot1, and the main partition of the emmc.

You went through a lot of effort. I hope it wasn’t too painful for you. But what you did to fix it sounds good.

@Savasten

My goal is to revert my wrongs and get my old system booting. If not I am fine reloading the system but that seems less fun.

It could be that the u-boot you flashed to emmc is borked for a311d-pocket. Instead of getting yourself into this boot loop again, you could try flashing the same u-boot version to sd-card and not to emmc. If you get into the same boot loop then the u-boot version is indeed borked and we should immediately pull the bad version from the repo. As long as you leave emmc boot0/boot1 zeroed out, you can experiment with u-boot without soft-bricking your system.

@Savasten

Correct emmc layout and data

You flashed other stuff to emmc so if you like you can reset your emmc content to something which will boot your encrypted rootfs on nvme. The tool reform-emmc-bootstrap is written to do exactly that. You can run the tool from a system you booted from an sd-card and it will repartition and set up the contents of your emmc with a /boot partition that can boot your encrypted system on nvme.

@Savasten

including u-boot

The u-boot topic is orthogonal to filling emmc with the correct /boot partition. I would suggest you experiment with different u-boot versions on sd-card. This means that to boot your pocket you have to have an sd-card inserted. Make sure that this sd-card does not come with a boot.scr or extlinux.conf or otherwise u-boot will prefer booting the system on sd-card and not from nvme.

@Savasten

How to fix u-boot to point to the encrypted nvme as the root partition

u-boot knows nothing about your encrypted rootfs. That information is stored in your initramfs and that sits in the /boot partition on your emmc.

@Savasten

I am not sure where the default swap drive was located.

If you used reform-setup-encrypted-disk then your swap is inside the encrypted luks partition.

So i think your next steps are:

1. boot a system from sd-card and run reform-emmc-bootstrap
2. move/remove boot.scr and extlinux on your sd-card so that u-boot on your sd-card will not load boot.scr or extlinux.conf from the sd-card but from emmc
3. boot into your encrypted system and make sure that this works

Once you are at that point you can start with experimenting with u-boot.

On rk3588 there was a problem with the most recent u-boot release where Linux would sometimes (not always) not find the nvme anymore. Reverting to the u-boot version before that fixed this issue. We need to investigate why this happened but maybe you are facing a similar problem on a311d and maybe using the previous u-boot version fixes things?

But maybe the nvme troubles are not at all (or not only) u-boot specific but also rely on your kernel version. In the other thread i read that even without touching u-boot, linux 6.18 on a311d does not see the nvme and that downgrading to 6.17 fixes this. I was yet unable to confirm this and have to spend more time on that.

I really appreciate the detailed work and investigation you are doing and the detail in which you are writing down your findings. Especially during the last week, the issues which popped up with linux 6.18, u-boot and of course the gdm issue ate multiple hours of my time each day which is really hard to manage next to my regular day job and family. So I really appreciate getting help by others in figuring out the source of these problems.

Let me know if there is anything else I can help you with.

Thank you!

[Savasten]

Thank you so much @josch, I will try all those steps and post a follow-up.
Unfortunately I do not have the exact message but, i remember something about a checksum mismatch after the reform-flash-uboot emmc

I will test on sd card only first.

[Savasten]

Thank you so much @josch, I will try all those steps and post a follow-up.
Unfortunately I do not have the exact message but, i remember something about a checksum mismatch after the reform-flash-uboot emmc

I will test on sd card only first.

I was able to get everything back up and running. Thank you so much.

My last hang-up was getting u-boot updated. I kept getting a sha1 hash mismatch and u-boot would be broken.

I found that the issue occurred when running reform-flash-uboot from a chroot environment.

@josch it may be worth adding a warning to reform-tools / reform check not to run the flash while in chroot.

Thank you again,
Keith

[josch]

@Savasten

My last hang-up was getting u-boot updated. I kept getting a sha1 hash mismatch and u-boot would be broken.

Should be fixed by this MR: reform-tools/debian/patches/series: don't discard... (!163) · Merge requests · Reform / reform-debian-packages · GitLab

@Savasten

I found that the issue occurred when running reform-flash-uboot from a chroot environment.

What was the error? Just the sha1 mismatch? That might just be because of different reform-tools versions on the inside and the outside.

@Savasten

it may be worth adding a warning to reform-tools / reform check not to run the flash while in chroot.

There should be no problem with running reform-flash-uboot from inside a chroot environment if /dev was mounted.

I’d advice against upgrading u-boot very often unless you want to help fix issues with more recent u-boot versions. There is a reason for the big warning which reform-flash-uboot prints before you flash u-boot to emmc. If you indeed do want to help with testing then I’m very grateful!

[Savasten]

I have done some more tests and on after a full update with updated reform-tools I got the same sha1 error. I tried a manual dd of the flash file to the SD card.

Looks like the reform-flash-uboot is one record short 3109 vs 3110

I n
Will dig deeper.

[josch]

@Savasten

I have done some more tests and on after a full update with updated reform-tools I got the same sha1 error

Yes, the problem is known and fixed in above MR.

The problem is that the if/else condition is the wrong way round. Read the code of reform-flash-uboot to verify this for yourself. This is fixed in git but not in the MNT repositories because we cannot deploy fixes in reform-debian-packages because mesa ftbfs because libclc is uninstallable because llvm-toolchain ftbfs (#1125733) because cmake 4 was uploaded to unstable a few days ago (#1113237).

EDIT:

You can run reform-flash-uboot with --verbose to see the exact dd invocations.