No keyboard input with LS1028 board and latest image?

I was finally able to install the LS1028 module in my Reform. The fit for the holes in the module doesn’t seem to be perfect - installing the screws through the board was a bit of a struggle.

The flat flex attached to the board also hits the space of the coil marked 680 and need to be bend to be installed.

The images at https://mnt.re/system-image are no longer available - the link redirects to https://source.mnt.re/reform/reform-system-image/-/jobs/3875/artifacts/browse which shows a 404.

I downloaded images from job 3903 instead. I get normal boot output to the display.

I’m unable to log in. The system does not react to input from the keyboard or an attached USB keyboard.

I also failed to make an attached 3.3V USB to serial adapter work. I connected to S1 and S2 and tried both possible RX/TX pairings. I tried all common baud rates. The best I got was a magic sysrq display on the screen when I somehow triggered a break condition at 9600 baud.

Before I re-install the i.MX module, I wanted to ask if anyone has a reform with LS1028 who could test the image I downloaded. Does it work for anyone else?

Is there something else I should do to which might make the LS1028 work? Perhaps keyboard/LPC firmware update?; I’m at 20220621 for LPC and 20210927 for keyboard.

1 Like

I have an LS1028 which I’ve not yet tested. It’s the next module for which I want to get NixOS working, so I’ll get that installed, hopefully later today, and report back if you haven’t resolved your issue by then.

Edit: I’ve got the LS1028A module installed and booting. Works fine on my box, sorry to say. I’m running the 3903 image as well, which I found linked on the Reform documentation page. (I also tried to use the link in the handbook at first. Seems the /system-image redirect needs to be updated.)

My LPC and keyboard are both running the 2021-04-19 firmware.

I did notice that I had to disconnect the keyboard from the motherboard in order to comfortably install the eDP adapter, and the system does boot with it disconnected, though it (as one might expect) doesn’t accept keyboard input once booted. It might be a dumb question, but you do have that cable firmly connected, right?

Aside from that, my best guess at this point is no better than yours: try upgrading your firmware.

Edit 2: Might also be the physical connection from your keyboard to the module, aside from the connection at the jumper. I had a physical trace broken on my keyboard when I got it; was able to bodge around it, but still, it does apparently happen. Wouldn’t explain your serial troubles, granted. Have you tried connecting a USB keyboard to see if that works?

Thank you for testing and confirming there’s no issue with the image!

I reinstalled the i.MX module. Everything worked fine there. I also updated LPC and keyboard firmwares while I was at it and re-wrote the SD card.

Unfortunately keyboard input - from internal or external USB keyboard - still doesn’t work.

USB ports have power: my security key’s lights go on when plugging it into the reform. However, when I plug in a mouse or keyboard no lights go on. Plugging the same mouse into another machine immediately causes the light on the bottom to go on.

I suspect at least USB isn’t working correctly for me. While the serial port measures 3.3V between ground and TX and the display showed a SYSRQ during measuring, it doesn’t seem to work either.

SD cards work, ethernet works, mPCI (Wifi) seems to work as well since I get the undesired display flashes.

The LED H2 (corner towards SD card slot) on the board is on, the STATUS_LED H1 is not. Do you see a red light shining below the heat sink near the IMXRST button? Maybe the device tree file can tell me if it’s supposed to be on.

I have SOM_LS1028_rv02 with a botch between TP19 and N3. My mainboard dates from 2020-10-29 based on the silkscreen.

I’ll see if I can modify the image to permit remote access via SSH to get more diagnostic information.

This might not be of much use to @johkra but @lykso might find this interesting when getting NixOS to run on the ls1028a…

I’m running Gentoo on ls1028a and I used to have intermittent USB problems as well. On some boots, the keyboard would not work and I had to reboot to fix the issue. Some times it took a couple of reboots.

Over in the fediverse, mnt confirmed that this is a real issue, probably a bug in the driver.

In the official image, this is worked around by running this script that reloads the driver.

I ended up writing a custom dracut module to run the script in my initrd.

1 Like

Hi,

the ls1028a has issues with usb, specifically with the dwc3 module. Attached USB devices are just not picked up and you see this in dmesg:

[   23.742947] xhci-hcd xhci-hcd.0.auto: Error while assigning device slot ID: Command Aborted
[   23.742960] xhci-hcd xhci-hcd.0.auto: Max number of devices this xHCI host supports is 127.

One work around seems to be to rmmod and modprobe again the dwc3 module and that’s what reform-hw-setup does. But there are also attempts to fix this problem for good in the upstream kernel:

https://lore.kernel.org/linux-devicetree/20191121024206.32933-1-ran.wang_1@nxp.com/

@minute had prepared a simpler patch of an earlier version of the patch which can be found in this MR:

But when I tested it, it didn’t have the desired effect. So next I applied the updated upstream patch to our kernel in this new MR:

If somebody would like to help, I can later post a link to system images containing this patched kernel. Otherwise I’ll probably get around testing it by tomorrow.

1 Like

Thank you both! I enabled SSH on the SD card in a chroot via qemu and was able to connect.

I confirmed that I got the USB errors in dmesg josch showed. Unloading and reloading the module is very flaky (doing so multiple times didn’t help at first), but I eventually succeeded. The keyboard worked at this point.

I’m happy to test if you post system image links.

1 Like

I flashed the image and USB worked 0 times out of 4 reboots.

I still get the following in dmesg:

[   26.046829] xhci-hcd xhci-hcd.0.auto: Error while assigning device slot ID: Command Aborted
[   26.046862] xhci-hcd xhci-hcd.0.auto: Max number of devices this xHCI host supports is 127.
[   26.046893] usb usb1-port1: couldn't allocate usb_device
[   38.334814] xhci-hcd xhci-hcd.0.auto: Error while assigning device slot ID: Command Aborted
[   38.334843] xhci-hcd xhci-hcd.0.auto: Max number of devices this xHCI host supports is 127.
[   38.334870] usb usb2-port1: couldn't allocate usb_device

Rebooting (by typing reboot in a terminal) doesn’t work for me, either. The display never comes on after a reboot. The NIC lights show some activity from time to time, but no IP address is assigned.

Sorry, in the meantime user hramrach from IRC found that my commit was missing the corresponding changes to the device tree. So the fact that it didn’t work for you makes sense. I’ll write again once we’ve found the correct change to the dt.

1 Like

My LS1028 is in the same situation as OP, no keyboard input on build 4343 regardless of how many times I reboot. Any word on how this is going, or if there’s any images I could test?

FWIW I’m experiencing the same thing with latest LS1028A.

Sorry for that! I can reproduce the issue with 6.8.11, investigating.

You won’t believe it, but it seems like a timing issue that was revealed by us setting loglevel=3 in the latest images.

If you set the loglevel back to 7, it works again (most of the time at least, how it used to be). My hunch is that the log prints introduce tiny delays that are required for the functionality of dwc3 on LS1028A.

If you have serial connected you can test this even dynamically by doing something like this:

rmmod dwc3
dmesg -n 7
modprobe dwc3

Then USB should work. If you set dmesg -n 3, it should again cease to work.

This is so far the most ridiculous issue I have encountered. To root cause this we would need to find all the log print sites and sprinkle with delays (or it could also be cache flush related).

4 Likes

The easiest workaround right now, before the root cause is found, would be to add the following after line 78 in /sbin/reform-hw-setup:

dmesg -n 7

You could do this by mounting the SD card on another computer and editing the file, for example.

2 Likes

Thank you! The workaround should appear in the next version 1.43 of reform-tools:

3 Likes

Thank you so much for the fix!

Okay, so modifying the reform-hw-setup script to add dmesg does work with the latest v4 image.

However, if you then run reform-setup-encrypted-nvme and then reboot the machine, the keyboard still doesn’t work.

My guess is that reform-hw-setup gets run after luks is already setup.

1 Like

You are right, reform-hw-setup is not run early enough for the LUKS passphrase. The dmesg -n 7 would have to run in the initramfs before the LUKS passphrase prompt gets shown. Could you try applying this patch

diff --git a/initramfs-tools/scripts/reform b/initramfs-tools/scripts/reform
index b80dbd4..d92ca47 100755
--- a/initramfs-tools/scripts/reform
+++ b/initramfs-tools/scripts/reform
@@ -11,4 +11,6 @@ elif [ -e /proc/device-tree/model ]; then
        MACHINE=$(cat /proc/device-tree/model)
 fi
 
-# nothing to do
+case $MACHINE in
+       "MNT Reform 2 with LS1028A Module") dmesg -n 7;;
+esac

To /usr/share/initramfs-tools/scripts/init-top/reform and re-create your initramfs and reboot?

1 Like

I’ve applied the change, and chrooted into the reformsdroot and ran update-initramfs -c -k 6.8.11-mnt-reform-arm64.

I got errors related to not being able to find /boot/config-6.8.11-mnt-reform-arm64. So I tried to sudo mount --bind /boot /media/myusername/12345.

I’m now realizing that it’s being tricky because I probably need to decrypt the luks volume or handle it some special way prior to being able to modify it. Then maybe I can access it via /boot in the chroot because right now /boot is empty.

Currently looking at this thread for inspiration.

A shorter command to rebuild the current initramfs would be update-initramfs -u. This works correctly even in a chroot if you have all the things mounted. You can look at the reform-boot-config script which does exactly that. Essentialy you’d need to do:

mount /dev/myrootpart /mnt
mount /dev/mybootpart /mnt/boot
mount -o bind /dev /mnt/dev
mount -t sysfs sys /mnt/sys
mount -t proc proc /mnt/proc

Then update-initramfs -u and then unmount everything again. The script creating the reform system image does the same thing to create the initramfs without actually running on the reform itself but just in a chroot.

And yes, if your /dev/myrootpart is on an encrypted volume, you have to decrypt that first.

1 Like

Interesting. I’ve been using a custom dracut module to run the reform-hw-setup script in the initramfs. Still, this did not fully solve the issue. Every so often, I would still get stuck at the disk encryption password prompt with the keyboard not working. Curious to see if this fix will permanently resolve this.