"shutdown now" doesn't shut down the machine anymore

[O.o]

I upgraded the machine yesterday or Friday - not sure when - and now shutdown now doesn’t shut down the machine, and the keyboard stays up.

The kernel does say Power down but the machine stays running and pulling some 350mA:

[...]
   45.277537] EXT4-fs (nvme0n1p1): unmounting filesystem c676142f-ae95-4fb0-800e-a39f71a011ad.
[   45.551658] systemd-shutdown[1]: Syncing filesystems and block devices.       
[   45.649054] systemd-shutdown[1]: Sending SIGTERM to remaining processes...    
[   45.659555] systemd-journald[457]: Received SIGTERM from PID 1 (systemd-shutdow).
[   45.710703] systemd-shutdown[1]: Sending SIGKILL to remaining processes...    
[   45.720170] systemd-shutdown[1]: Unmounting file systems.                     
[   45.722085] (sd-umount)[2078]: Unmounting '/run/credentials/systemd-journald.service'.
[   45.724488] (sd-remount)[2079]: Remounting '/' read-only with options 'errors=remount-ro'.
[   45.767388] EXT4-fs (mmcblk0p2): re-mounted aa4942fb-f689-5188-baa8-5600ee52c104 ro. Quota mode: none.
[   45.783204] systemd-shutdown[1]: All filesystems unmounted.                   
[   45.783732] systemd-shutdown[1]: Deactivating swaps.                          
[   45.784241] systemd-shutdown[1]: All swaps deactivated.                       
[   45.784716] systemd-shutdown[1]: Detaching loop devices.                      
[   45.786620] systemd-shutdown[1]: All loop devices detached.                   
[   45.787128] systemd-shutdown[1]: Stopping MD devices.                         
[   45.787681] systemd-shutdown[1]: All MD devices stopped.                      
[   45.788178] systemd-shutdown[1]: Detaching DM devices.                        
[   45.788731] systemd-shutdown[1]: All DM devices detached.                     
[   45.789223] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[   45.791554] systemd-shutdown[1]: Syncing filesystems and block devices.       
[   45.792908] systemd-shutdown[1]: Powering off.                                
[   45.973098] kvm: exiting hardware virtualization                              
[   45.973681] reboot: Power down     

Anything I can do?

[O.o]

Update: poweroff -f sometimes works - i.e. turns off the machine and the keyboard - and sometimes doesn’t. I can’t see to figure out in which conditions it works and when it doesn’t.

There’s this in reform2_lpc.c:

/* FIXME: this overwrites something else that has already claimed pm_power_off on reform2 but it'll do for now */
poweroff_device = &spi->dev;
pm_power_off = lpc_power_off;           

I wonder if there’s a race when registering pm_power_off…

[O.o]

Oh and this is new:

And the keyboard is almost unusable now.

I’m guessing the LPC has gone silent

[O.o]

I reflashed the keyboard and the LPC. Now the keyboard seems to be back to normal. And the LPC correctly powers off the machine upon shutdown once every 3 to 5 times.

If anybody has an idea, I’m all ears.

[josch]

Disclaimer: I might be posting a lot on this forum but I’m just another contributor and not affiliated with MNT. I’m saying this to make it clear that I have no financial incentive or whatnot for you to do one thing or the other.

My suggestion to you is, to take a break and to come back to the Reform once @minute (or whoever finds the time to do it) fixed the bugs in the lpc/keyboard/dkms interaction. I have a rk3588 classic Reform from the same batch as your as I can reproduce some of your issues which is part of why I’m not doing much with my unit right now. I lack the bandwidth to fix this myself.

The lpc/keyboard/dkms combo received a major update just before the rk3588 classic reforms shipped:

• dkms: LPC driver: bump speed by 10x; simplify error handling; add uart command support (!127) · Merge requests · Reform / MNT Reform Tools · GitLab
• lpc: Classic Reform LPC SPI rework (!84) · Merge requests · Reform / reform · GitLab
• sysctl (pocket): Sysctl SPI rework (!42) · Merge requests · Reform / MNT Pocket Reform · GitLab

All of the above were merged at the same time at the end of May. For example for my RK3588 Pocket Reform went back to the sysctl version from before above changes happened because the sysctl version that got tagged after above MR were merged makes my RK3588 Pocket Reform reboot randomly every few hours of use. Super annoying.

From the perspective of a long term user and contributor to the Reform ecosystem, my suggestion for you is to take a small break. I’d love to see you keep your Reform because I love what you do with your unit and I would really like you to stick around to show us more crazy things you come up with. I share your frustration, it’s very annoying. You and me are not the only ones affected. We also get users in IRC who have similar issues like user reform18574 here. Maybe just give it some time. My hope is that we’ll soon see a fix to this mess.

Thank you for all your contributions so far in any case.

[O.o]

@josch

I’m just another contributor and not affiliated with MNT

I’m aware of this and I appreciate your feedback

Re the Reform and “taking a break”: that’s not an option and I’ll tell you why: I ordered the Reform because my old HP laptop was dying. I need a reliable daily driver and the Reform is long in the tooth, so I figured it was stable.

The Reform took so long to arrive that my HP laptop died. So I bought a cheap replacement to have something going until the Reform came. I promised to donate the replacement laptop to the Red Cross when the Reform arrived and I did.

So now I need a working laptop. The Reform isn’t some toy I’m having fun hacking on when it deigns working, it’s my daily driver. If it doesn’t work, I have to fix it, because at that price, I’d find having to buy yet another replacement laptop - and reinstalling everything yet again - quite hard to swallow.

But no matter. I’m a patient man. As long as it limps along, it’ll do I guess.

And I figured out why the keyboard was acting up: when the menus disappeared, I reset it. When it came back, it thought the LPC was off, and I forgot it acts up when it thinks the rest of the system is off. This really, REALLY needs to be documented somewhere because it’s hella confusing.

[josch]

For others it helped to avoid using the OLED menu. Does doing so get you some more reliability?

(also, very impressive story! I really, really hope we get this fixed soon)

[minute]

Hey, sorry and thanks for sticking with it despite these issues. The keyboard4-fw has some (by now) clear issues that I’ll start fixing on Monday. I couldn’t do it this week because we’re in Karlsruhe at the GPN since Wednesday.

The keyboard4-fw was forked off a longer while ago from the pocket reform keyboard fw (because of the move from atmega to rp2040 during COVID chip crisis). In the meantime, many stability improvements and refactorings went into the pocket keyboard fw that I’ll now port back into the keyboard4-fw. I’m also considering unifying the codebases to not let them drift apart again, but not sure this unification is doable for me in a few days, so I’ll first do a more surgical approach and will post updates here.

[O.o]

I can code. If I can help, let me know.

[O.o]

@josch

For others it helped to avoid using the OLED menu

How do you mean? How can you not use it? It’s needed to turn the machine on and off

There is something I noticed: because of the location of the Circle key, I keep hitting it instead of Backspace. And then of course I hit some other key. That’s how I discovered the donkey animation and the hello animation shortcuts. But I also discovered the trackpad-related shortcuts that do nothing with a trackball.

So I think what I’ll do today is modify the keyboard firmware so that:

• You need to press Circle-key, as in you need to keep Circle pressed for the key to do anything, so you don’t accidentally put the keyboard in programming mode or turn the machine off if you intended to type Backspace-X or Backspace-0
• Disable the TP-related keys with a compilation directive. Because I don’t know what they do exactly, but they do something on nonexistent hardware, so I’m not sure it’s very healthy.

[O.o]

Here’s a patch. It does two things:

• It fixes “Press CIRCLE for menu - Hold to power up” that didn’t work.
• It requires CIRCLE to be held down for certain dangerous or non-trivial functions to execute directly - that is, not from the drop-down menu.

diff --git a/reform2-keyboard4-fw/src/main.c b/reform2-keyboard4-fw/src/main.c
index b640168..04538fb 100644
--- a/reform2-keyboard4-fw/src/main.c
+++ b/reform2-keyboard4-fw/src/main.c
@@ -645,6 +645,16 @@ int is_repeatable(uint8_t keycode, int menu_repeat_count) {
   return 0;
 }
 
+// Menu keys that shouldn't do anything unless CIRCLE is pressed and held first
+int requires_press_and_hold_circle(uint8_t keycode) {
+  if (keycode == KEY_0 ||    // Power down
+      keycode == KEY_X ||    // Enter firmware update mode
+      keycode == KEY_Q ||    // Trackpad ATI
+      keycode == KEY_W       // Trackpad init
+      ) return 1;
+  return 0;
+}
+
 int process_keyboard(uint8_t* resulting_scancodes) {
   // how many keys are pressed this round
   uint8_t total_pressed = 0;
@@ -725,6 +735,7 @@ int process_keyboard(uint8_t* resulting_scancodes) {
 
         // circle? open OLED menu
         if (keycode == KEY_CIRCLE) {
+          circle_key = 1;
           if (!last_menu_key) {
             if (!active_menu_mode) {
               enter_menu_mode();
@@ -736,11 +747,11 @@ int process_keyboard(uint8_t* resulting_scancodes) {
               circle_long_press_start_ms = now_ms;
               // edge case
             }
-            if (now_ms - circle_long_press_start_ms > 1000) {
-              // turn on computer after 2 seconds of holding hyper + enter
+            // If CIRCLE is held over 1 second and the computer is off, turn it on
+            if (now_ms - circle_long_press_start_ms > 1000 && !remote_get_power_state()) {
               exit_menu_mode();
-              last_menu_key = KEY_1;
               circle_long_press_start_ms = 0;
+              execute_menu_function(KEY_1);
             }
           }
         } else if (keycode == KEY_HYPER) {
@@ -750,22 +761,25 @@ int process_keyboard(uint8_t* resulting_scancodes) {
           if (active_menu_mode) {
             // not holding the same key?
             if (total_pressed == 1 && (last_menu_key != keycode || is_repeatable(keycode, menu_repeat_count))) {
-              // hyper/menu functions
-              int stay_menu = execute_menu_function(keycode);
-              // don't repeat action while key is held down
-              // except for repeatable actions (allow each menu_repeat_count cycle)
-              last_menu_key = keycode;
-
-              // exit menu mode
-              if (!stay_menu) {
-                exit_menu_mode();
-              }
-
-              // after wake-up from sleep mode, skip further keymap processing
-              if (stay_menu == 2) {
-                reset_keyboard_state();
-                enter_menu_mode();
-                return 0;
+              // Is CIRCLE down if the function requires it to execute, or can the function execute regardless?
+              if (circle_key || !requires_press_and_hold_circle(keycode)) {
+                // hyper/menu functions
+                int stay_menu = execute_menu_function(keycode);
+                // don't repeat action while key is held down
+                // except for repeatable actions (allow each menu_repeat_count cycle)
+                last_menu_key = keycode;
+
+                // exit menu mode
+                if (!stay_menu) {
+                  exit_menu_mode();
+                }
+
+                // after wake-up from sleep mode, skip further keymap processing
+                if (stay_menu == 2) {
+                  reset_keyboard_state();
+                  enter_menu_mode();
+                  return 0;
+                }
               }
             }
           } else if (!last_menu_key) {

[O.o]

Re the poweroff issue:

I traced the reform2_lpc module and I’m 100% certain the kernel never calls pm_power_off.

I also converted the module to register a sys-off handler directly with register_sys_off_handler, and the handler is never called either.

I looked in the kernel in reboot.c and in the relevant bits in arch/arm64, and I fail to see why the handler isn’t called. Unless the MNT kernel has been patched and that code path has been broken somehow…

[O.o]

Here’s another patch for the keyboard firmware (independent of the one above, i.e. you can apply it on the stock keyboard firmware on its own).

This one watches F1 and F12, and if you press both for 10 seconds, it forces the computer off and will keep the OLED display off until both keys are released.

This is meant to work with my little shutdown systemd service and foam pad doohickey: the script shuts down the OS cleanly, then the kernel fails to turn off the power currently, as per above, but it’s okay because the keyboard does it itself eventually.

10 seconds is plently long enough for the OS to unmount stuff cleanly and halt.

It is primarily a workaround for the shutdown issue above. But it’s not that stupid actually: even if the kernel did power off the machine properly, in case it goes haywire for any reason, this patch turns the keyboard into a sort of “safety last resort shutoff” that works regardless of whether the system behaves properly.

Meaning you can slam the lid closed, stuff the computer in your laptop bag and be certain it won’t cook itself in the bag and empty its batteries no matter what.

Here’s the patch:

diff --git a/reform2-keyboard4-fw/src/main.c b/reform2-keyboard4-fw/src/main.c
index b640168..f8392f7 100644
--- a/reform2-keyboard4-fw/src/main.c
+++ b/reform2-keyboard4-fw/src/main.c
@@ -606,6 +606,9 @@ int active_menu_mode = 0;
 uint8_t last_menu_key = 0;
 uint32_t circle_long_press_start_ms = 0;
 
+uint32_t scuttle_countdown_start_ms = 0;
+uint32_t computer_scuttled = false;
+
 uint8_t* active_matrix = matrix;
 bool hyper_key = 0; // Am I holding HYPER?
 bool circle_key = 0; // Am I holding CIRCLE?
@@ -648,7 +651,9 @@ int is_repeatable(uint8_t keycode, int menu_repeat_count) {
 int process_keyboard(uint8_t* resulting_scancodes) {
   // how many keys are pressed this round
   uint8_t total_pressed = 0;
+  uint8_t total_scuttle_key_pressed = 0;
   uint8_t used_key_codes = 0;
+  uint32_t now_ms = board_millis();
 
   menu_repeat_count++;
   if (menu_repeat_count > MENU_REPEAT_RATE) menu_repeat_count = 0;
@@ -723,13 +728,17 @@ int process_keyboard(uint8_t* resulting_scancodes) {
       if (debounced_pressed) {
         total_pressed++;
 
+        // Track how many scuttle keys (F1 and F12) are pressed simultaneously
+        if (keycode == KEY_F1 || keycode == KEY_F12) {
+          total_scuttle_key_pressed++;
+        }
+
         // circle? open OLED menu
         if (keycode == KEY_CIRCLE) {
           if (!last_menu_key) {
             if (!active_menu_mode) {
               enter_menu_mode();
             }
-            uint32_t now_ms = board_millis();
             if (!now_ms) now_ms++;
 
             if (!circle_long_press_start_ms) {
@@ -789,14 +798,41 @@ int process_keyboard(uint8_t* resulting_scancodes) {
     }
   }
 
+  // Has the computer been scuttled yet?
+  if (computer_scuttled) {
+
+    // If all the scuttle keys are released, reset the scuttled flag
+    if (total_scuttle_key_pressed == 0) {
+      computer_scuttled = false;
+    }
+
+ // The computer hasn't been scuttled yet
+ } else {
+
+    // If all the scuttle keys haven't been depressed,reset the scuttle timer
+    if(total_scuttle_key_pressed < 2) {
+      scuttle_countdown_start_ms = now_ms;
+
+    // ...otherwise count down to scuttle the computer
+    } else {
+
+      // Press the scuttle keys (F1 and F12) for 10 seconds to turn the computer off
+      if ((now_ms > scuttle_countdown_start_ms && now_ms -  scuttle_countdown_start_ms > 10000) || (now_ms < scuttle_countdown_start_ms && 0xffffffff - scuttle_countdown_start_ms + now_ms + 1 > 10000)) {
+        execute_menu_function(KEY_0);
+        execute_menu_function(KEY_ESC);
+        exit_menu_mode();
+        computer_scuttled = true;
+      }
+    }
+  }
+
   // if no more keys are held down, allow a new menu command
   if (total_pressed<1) last_menu_key = 0;
 
-  // if device is off and user is pressing random keys,
-  // show a hint for turning on the device
+  // if device is off and user is pressing random keys, and the computer hasn't been scutled by the scuttle keys, show a hint for turning on the device
 #if KBD_MODE == KBD_MODE_LAPTOP
   if (!remote_get_power_state()) {
-    if (total_pressed>0 && !active_menu_mode && !circle_key && !last_menu_key) {
+    if (total_pressed>0 && !computer_scuttled && !active_menu_mode && !circle_key && !last_menu_key) {
       execute_menu_function(KEY_H);
     }
   }

So, my problem is solved in a roundabout kind of way

[samjc]

Rarely on Pocket Reform I’m having a similar issue; the keyboard and screen stay lit despite being “shutdown”. However, I can still use Hyper+Enter to enter the menu and power down.

[O.o]

Next time is happens to you, can you check the battery status page and see if the computer still draws power? If it’s does, it’s not just the keyboard staying lit: the machine is fully powered up and the CPU is halted.

I suppose if the keyboard is lit, it naturally means the LPC hasn’t powered off, since the keyboard polls its status all the time.

[chorc]

On the Pocket it happens to me all the time (I just tried shutting it down 3 times in a row - reproduced every time).
OLED battery status shows SBC drawing power, the top of the machine is not cooling down.

[O.o]

I have to trace the kernel to figure out why it doesn’t seem to execute (or perhaps even register) the LPC module’s PM handlers. I think there’s a JTAG port on the RK3588. The problem is, I don’t really have time for this. That’s why I brute-forced the power-off I need with the keyboard for the time being.

[samjc]

Yeah, still drawing 0.5A

One time this happened the keyboard menu stopped working too. It would light up and show but none of the options worked. I had to power down using the switch on the side

[glob]

Yep, I’ve been experiencing the same behaviour on my RK3588 Pocket for the past week or so. I haven’t had the time to investigate properly yet, but it sounds like what you’re describing.

[josch]

@minute asked me yesterday on IRC whether I also experience this bug. The answer is: yes, I see this issue on RK3588 classic Reform with motherboard 3.0 and the firmware from MR88 on my keyboard.

This means that my RK3588 classic Reform has the same problem as my RK3588 Pocket Reform. My A311D classic Reform is unaffected and shuts down just fine without me having to press circle+0.

minute expressed in IRC that it will get addressed and fixed soon.