Boot Failure A311D Restoring Connection to Encrypted NVME

MNT Reform
1

I need advice on setting the latest SD image to use an existing encrypted NVME drive.

After a couple of weeks being off, Reform #200 didn’t boot. Used a 90W charger to reach full charge at 3.4V on all cells. Still failed to boot.

System boots after flashing SD card with latest version – without asking for passphrase to decrypt the NVME drive (MNT installed it on a mini-PCIE board at the factory). Need to check how to change drive setting for operating system?

Considering restoring original SD card image and testing again. But, the boot partition for the older image has a UUID for the volume name rather than reformsdboot.

2

The LPC status readout shows 1800MAh when at 100% (3.4 V per cell) which lead me to seek out higher charging capacity. (Cells on board: Enerpower 3.2V2000MAh)

3

I restored the original SD image and corrected two missing settings compared to the latest SD image. The 500 MB partition in the original SD image didn’t have a label or the bootable flag.

Yet, adding a label and bootable flag have no effect on booting into the system on the NVME.

4

Next stop: apply fix for missing LUKS password prompt

reform-tools 1.31

5

Still unable to boot, even using vkoskiv’s alternative boot.scr and original SD card image. My hypothesis: the issue is like the problem that blocked ephase last autumn (Reform won’t finish booting - #4 by ephase).

Reform #200 stays blank screen with keyboard lit when LPC screen turns off.

Maybe the next step involves using the latest SD image again and working on the settings to reach the encrypted NVME. I wonder if I “soft-bricked” the unit inadvertently during one of the updates in Mid August 2024, before I had to set it down 2 weeks.

6

Hi @ve7det and welcome to the community!

I must admit I’m quite a bit confused by your messages and am not quite sure what your current status is and how to help you.

Can you pick one specific problem you have and then we can try to resolve it?

7

Sorry, I have been logging my progress in the thread. I’m stuck and think the first post describes what I should have done. I could edit the opening post once I understand the problem.

This is all the same problem: Powering up appears to stop with a blank screen while the LPC logo fades away.

First try at fix: charge the batteries more completely

  • Get a 90W power supply
  • Still “stops” at blank screen

Second try at fix: replace the SD card with updated files

  • Boots to system :partying_face:
  • Skips prompt to decrypt the NVME :disappointed:
  • Only uses the SD card

Third try at fix: re-use the original SD card files with modifications

  • change the boot.scr based on what seemed to be a similar problem described in forums
  • fix? label of boot partition on SD card (from UUID to reformsdboot)
  • Still “stops” at blank screen

Next fix: check I’m using the correct reform-tools

  • I don’t know how to do this when I can’t use the system
8

I’ve read those but I have questions to nearly all of your messages. Let me ask them and you answer those that you find relevant to the problem. But maybe I’m just confused about what’s going on, sorry.

From other messages I understood that you are hit by the problem from this thread: Reform won’t finish booting but that is from October last year and that’s a bit more than a few weeks. What is Reform #200?

The big Reform will happily operate even when the cells are dead as long as the power cable is connected. So anything charge related should not be your problem. Especially since you can boot from a fresh sd-card just fine.

An SD-card with a vanilla factory image will not boot your system on NVMe by design. This is so that in case you make a mistake with your NVMe setup, you can still boot into a working rescue system instead of booting into a broken NVME installation. The SD-card image is supposed to give you a working system independent on any possible mis-configuration of your main system.

The use of disk labels was introduced in September 2023. This means that your other SD-card contains an image older than that?

What settings did you correct compared to the original SD image?

If your old system booted fine without those, then that’s not something you need. The partition labels were introduced to make it possible to use the same /etc/fstab independent on the underlying device which will number, for example, emmc devices differently.

Yes, what allows you booting from NVMe is the initramfs which needs to be configured to do so.

But as I understood your other messages, your problem is not a missing luks password prompt but your problem is that your screen stays black. Which one is it?

It would be useful to know when you last booted up your reform successfully.

If you updated it in August 2024, then your problem cannot be the one in this thread Reform won’t finish booting because that got fixed in October last year. Also, if you can successfully boot from a vanilla SD-card, then it will not be hard to restore things.

If it boots from sd-card, then this is very likely not power related.

This is very good and means we can fix this more easily.

This is what trips me up. The problem that was fixed with a boot.scr was resolved last year October. If you have not updated your Reform since then, then maybe it is that but I doubt it because your screen stays black and the problem there was that the LUKS passphrase prompt did not show up. The label of the boot partition has nothing to do with this either.

We can try out some things but first we need to figure out what the problem could be. It would be helpful to know when you last upgraded it.

Your problem with “screen stays black” sounds more like a bug in binutils which was found by IRC user pandora at around the end of June:

https://mntre.com/reform-irc-logs/2024-06-21.log.html#t09:22:03

Gory details here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1074111

Or reform community thread here: Sd boot img? (can no longer boot)

Could that be your issue? Does the time of the issue about fit with the last time you upgraded your system?

Also, do you happen to have a USB UART/serial adapter?

1 Like
9

What is Reform #200?

I am working with Reform bearing serial number 200.

But as I understand your other messages, your other problem is not a missing luks password prompt but your problem is that your screen stays black. Which one is it?

I started with screen staying black and am 80% sure the way forward is to refresh the SD card with the latest system. So, I was thinking that the way forward is the luks prompt.
From your wisdom, I am updating my hypothesis.

This is what trips me up…

I was grasping for potential fixes. I have updated several times since October 2023.

Does the time of the issue about fit with the last time you upgraded your system?

Yes. I’m 80% sure I updated in early August. I intended to update again when my Reform stopped working normally.

I don’t think I have a USB UART(serial) adapter. Is that a Prolific USB to DB9 serial adapter?

10

Okay, then in that case it cannot be that problem unless you updated very selectively and not your full system.

Maybe we can start with figuring out the truth here. After booting the rescue system, run the following commands on the terminal, preferably after you connected to your local ethernet or wifi to get internet with the rescue system:

$ cryptsetup luksOpen /dev/nvme0n1 reform_crypt
$ vgchange -ay reformvg
$ mount /dev/reformvg/root /mnt
$ mount /dev/mmcblk1p1 /mnt/boot
$ mount -o bind /dev /mnt/dev
$ mount -t sysfs sys /mnt/sys
$ mount -t proc proc /mnt/proc
$ cp /etc/resolv.conf /mnt/etc
$ chroot /mnt bash

At this point you will have a shell open inside your system on nvme and you can start poking around and investigating. For example, it would be useful to know the full output of running:

sudo reform-check

That command should include your reform-tools version and your kernel version. If it does not, then you can retrieve those by also giving me the output of these commands:

$ dpkg -l | grep reform-tools
$ dpkg -l | grep linux-image-

Lastly, to check the hypothesis with the binutils bug, could you please also run the following:

$ ls -lha /boot
$ lsinitramfs /boot/initrd.img-* | grep /lib/modules | wc -l

You can leave the chroot environment and get your terminal back by running these commands:

$ exit
$ umount --recursive /mnt
$ vgchange -an reformvg
$ cryptsetup luksClose reform_crypt

Looking forward to your findings. :smiley:

No, DB9 RS232 serial is quite different (last but not least voltage-wise) from UART serial.

1 Like
11

Super.
At mount /dev/mmcblk1p1 /mnt/boot, Reform gave an error. Output from lsblk below. Will try mmcblk1boot0.

NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
mmcblk0           179:0    0  28.3G  0 disk  
├─mmcblk0p1       179:1    0   488M  0 part  /boot
└─mmcblk0p2       179:2    0  27.8G  0 part  /
nvme0n1           259:0    0 931.5G  0 disk  
└─reform_crypt    254:0    0 931.5G  0 crypt 
  ├─reformvg-swap 254:1    0     4G  0 lvm   
  └─reformvg-root 254:2    0 927.5G  0 lvm   /mnt
mmcblk1           179:256  0  14.6G  0 disk  
mmcblk1boot0      179:512  0     4M  1 disk  
mmcblk1boot1      179:768  0     4M  1 disk
12

Bad choice. Gave an error, so using mmcblk1boot1.

Same error:

mount: /mnt/boot: wrong fs type, bad option, bad superblock on /dev/mmcblk1boot1, missing codepage or helper program or other error.
demsg(1) may have more information after failed mount system call.

13

After mounting the other units, I have the first reports (using the dpkg method).

reform-tools = 1.49

linux-images

rc    linux-image-6.6.13-reform2-arm64    6.6.13-1+reform20240121T174844Z1
ii    linux-image-6.6.9-reform2-arm64     6.6.9-1+20231225T111019Z1
rc    linux-image-arm64                  6.6.13-1+reform20240121T174844Z1

reform-check gave an error:

sudo: unable to allocate pty: No such device

When restoring, I gave this Reform the host name reform200. This may not have caused the pty assignment error, yet seems like I should have left the hostname alone.

Tried reform-check again. Transcribed results:

Platform Name: MNT Reform 2 with BPI-CM4 Module
Running Kernel Version: 6.10.7-mnt-reform-arm64
Installed reform-tools version: 1.49
Not installed: blueman firmware-misc-nonfree greetd linux-headers-mnt-reform-arm64 linux-image-mnt-reform-arm64 pocket-reform-handbook tuigreet wayvnc wf-recorder
/boot has nothing mounted on it – fix your /etc/fstab
/boot/flash.bin doesn’t exist
You can download the latest version by running as root: reform-flash-uboot
root account has no password (maybe run passwd -l root)

I’m surprised about the root account?

unexpected last line in /etc/skel/.profile, should be: …
following files differ from how they are shipped by reform-tools: ??5??? /var/lib/alsa/asound.state
/boot/initrd.img--mnt-reform-arm64 doesn not belong to any installed kernel package
/boot/vmlinuz--mnt-reform-arm64 does not belong to any installed kernel package
currently running kernel is not installed as a package
/boot/boot.scr doesn’t exist
run ‘sudo flash-kernel’ to create it
no vmlinuz in /boot for the currently running kernel
no initrd.img in /boot for the currently running kernel
no dtb-* symlink for the currently running kernel
run ‘sudo flash-kernel’ to create it
/etc/modprobe.d/reform.conf is identical to /usr/lib/modprobe.d/reform.conf
and the latter succeeds the former. Consider deleting the former in favour of the latter.
/usr/lib/modules/6.10.7-mnt-reform-arm64/updates/dkms/reform2_lpc.ko{.xz} does not exist
the linux header package linux-headers-6.10.7-mnt-reform-arm64 for the currently running kernel is not installed

14

Can you be more verbose? What was the error?

That looks like your eMMC has no partitions on it. That’s okay, you can just put /boot onto your SD-card by re-using the first partition of the rescue sd-card you flashed. Just make sure to manually unmount /boot before performing the steps I outlined and then mount /dev/mmcblk0p1 to /mnt/boot.

That one as well as mmcblk1boot1 do not contain filesystems but u-boot itself. If you manage to write the wrong things onto those locations, you really softbrick your a311d. Better not mess with those unless you know what you are doing.

Good, that’s from August 9 2024.

That’s bad. This is from December 2023.

And that is the reason that your kernel is this old. You removed the meta-package which would’ve kept your kernel up-to-date.

Indeed you already are the superuser in the chroot environment, so no need to prefix the command with sudo. Apologies for that oversight.

The hostname should not have something to do with this. The pty assignment error is because of how /dev is now shared between the outside and inside of your chroot environment.

Instead of transcribing the result (you really have patience!) you could redirect the output to a file on your sd-card and then copy that file on another computer.

Your system image is quite old. At the time where you installed your reform, there was no reform-setup-wizard which allowed you to create a new user. Before that, there was a root account with an empty password. Since this is very dangerous, I added this check to reform-check.

That one is interesting.

You might be able to fix your situation by doing the following as superuser:

$ umount /boot
$ cryptsetup luksOpen /dev/nvme0n1 reform_crypt
$ vgchange -ay reformvg
$ mount /dev/reformvg/root /mnt
$ mount /dev/mmcblk0p1 /mnt/boot
$ mount -o bind /dev /mnt/dev
$ mount -t sysfs sys /mnt/sys
$ mount -t proc proc /mnt/proc
$ cp /etc/resolv.conf /mnt/etc
$ chroot /mnt apt update
$ chroot /mnt apt install linux-image-mnt-reform-arm64 linux-headers-mnt-reform-arm64
$ umount --recursive /mnt
$ vgchange -an reformvg
$ cryptsetup luksClose reform_crypt

Doing this will install the latest kernel meta-packages and with it, the latest kernel package. During the installation, a new initramfs and a new boot.scr will be created for you. Since you are downloading packages, you need an internet connection for this to work. Following these instructions will also make your rescue system unbootable as you are configuring the first partition on your sd-card to boot into your encrypted system instead. So be prepared to flash a new image to that sd-card in case following these steps does not have the desired effects.

15

Super. I’ll try this latest advice.

When I try to direct the reform-check output to file, it doesn’t seem to work.
reform-check > 20240909-r2-2-reformcheck.txt
Output stays on screen.

More results soon.

16

While Reform S/N 200 is installing linux-image-6.10.7-mnt-reform-arm64, apt install claims that two items are already newest versions:

  • linux-image-mnt-reform-arm64
  • linux-headers-mnt-reform-arm64

Still need to use luksOpen to use the NVME after this rebooted.

17

Sorry, please also redirect stderr like this:

reform-check > 20240909-r2-2-reformcheck.txt 2>&1

This is very surprising, because according to your earlier output of dpkg -l | grep linux-image-, you do not have the package linux-image-mnt-reform-arm64 installed. Are you sure that you are running the apt install command inside the chroot, that is, do you have apt install prefixed with chroot /mnt as in the instructions?

1 Like
18

Thanks, i always get the >&2 wrong.

??5??????   /var/lib/alsa/asound.state
I: Your platform name is: MNT Reform 2 with BPI-CM4 Module
I: You are running kernel version: 6.10.7-mnt-reform-arm64
I: Your installed kernel version: 6.10.7-1+reform20240902T112605Z
I: Your installed reform-tools version: 1.52
I: the following files differ from how they are shipped by reform-tools:
??5??????   /var/lib/alsa/asound.state

Strange thing is that apt still installed the files, with the error.

19

i did miss that last bit.

making sure the second apt command is chrooted give no error, upgrades 1 package and installs 6.

Update:
could not write log to /dev/pts for one of the packages, otherwise OK.

Except that rebooting still ends with black screen well past the LPC screen turns off.

I’ll add the reform-check next. After I get back in with a rescue SD.

20 
I: Your platform name is: MNT Reform 2 with BPI-CM4 Module
I: You are running kernel version: 6.10.7-mnt-reform-arm64
I: Your installed kernel version: 6.10.7-1+reform20240902T112605Z
I: Your installed reform-tools version: 1.49
I: not installed:  blueman firmware-misc-nonfree greetd pocket-reform-handbook tuigreet wayvnc wf-recorder 
E: your /boot has nothing mounted on it -- fix your /etc/fstab
W: /boot/flash.bin doesn't exist
W: You can download the latest version by running as root:
reform-flash-uboot
E: root account has no password (maybe run passwd -l root)
E: unexpected last line in /etc/skel/.profile, should be:
if [ "$(whoami)" = "root" ]; then reform-help --root; elif [ -z "$WAYLAND_DISPLAY" ]; then reform-help; fi
I: the following files differ from how they are shipped by reform-tools:
??5??????   /var/lib/alsa/asound.state
E: /boot/boot.scr doesn't exist
E: run 'sudo flash-kernel' to create it
E: no vmlinuz in /boot for the currently running kernel
E: no initrd.img in /boot for the currently running kernel
E: no dtb-* symlink for the currently running kernel
E: run 'sudo flash-kernel' to create it
W: /etc/modprobe.d/reform.conf is identical to /usr/lib/modprobe.d/reform.conf
W: and the latter succeeds the former. Consider deleting the former in favour of the latter.