Standby - Suspend to RAM (MNT Reform)

[doctorhoo]

I have just observed another crash and recently have seen one more. Main difference to my previous tests is the wait between resumes; before the crashes it was more than 24h while I usually power up the reform at least once a day. Last time the console showed an error related to the nvme:

SysRq key worked in that state, but I didn’t have time to connect a serial console (onto which the output of the SysRq dumps was sent last time I checked). But as least I could sync, umount and reboot. Might be an option for further investigations next time.

Before those recent crashes, the system resumed fine maybe 30–50 times. I usually only suspend and never shutdown the system.

[2disbetter]

@ruff after apt upgrading I am noticing that the ability to have the suspend process turn on and off the keyboard backlight is no longer working. Do you maybe have idea why this might be?

[ruff]

Not sure what exactly happened on apt upgrade but for me the mnt kbd rawhid became rawhid4. So I’ve created a small discovery sriptlet

hidraw=$(ls -1 /sys/devices/platform/soc@0/38100000.usb/xhci-hcd.0.auto/usb1/1-1/1-1.3//1-1.3\:1.0/0003\:03EB\:2042.0005/hidraw)
...
bklite_off() {
        echo -en "\0LITE0" > /dev/$hidraw
}
bklite_on() {
        echo -en "\0LITE6" > /dev/$hidraw
}

[2disbetter]

@ruff thanks! Are you using the bind and unbind elements from sigrid’s script? Since upgrading the script has been pretty temperamental.

Also I run your scriplet I get an error saying that that file / directory doesn’t exist. Instead of 2042.0005 there is only a 2042.0001 there. I changed that in the scriptlet, and I’ll see if that does anything.

edit: It did. Curious how the upgrade changed that directory.

[ruff]

Uhm, yes, that’s actually enumeration number and will be changing so need to be replaced with *. But since you have number 1 it means it should still be hidraw0, not sure why in hardcoded state it didn’t work.
I’m still heavily experimenting with dynamic power management (which does not actually work) so I didn’t try unbinding it (so far dynamic re-bind works as long as device re-enumerates on the bus, but it doesn’t if I enable pm)

[2disbetter]

@2disbetter

2042.0001

@ruff

so need to be replaced with *

so it would look like this:
2042.000*

in the script?

[ruff]

hidraw=$(ls -1 /sys/devices/platform/soc@0/38100000.usb/xhci-hcd.0.auto/usb1/1-1/1-1.3//1-1.3\:1.0/0003\:03EB\:2042.*/hidraw)

should do it, there won’t be more than one anyway

[2disbetter]

Ok, for the first time after upgrading, I am back to reliable suspend and resume cycles. I believe the changing of the hidraw resource was part of the instability I was seeing with it.

In combination with sigrid’s suggestion to unbind when suspending and rebind the keyboard and trackball when resuming, I believe that this script is robust enough to give most mostly reliable suspend sessions.

I just want to thank everyone that made this a priority and improved performance. My own Reform is so close to be my daily driver. When I was looking at the MNT Reform to begin with I was VERY skeptical that the SoC would be sufficient to make that possible. My experience, however, is just that it really can do just about everything else my other computers can.

It helps that I love the design of the machine and the build quality. One thing I think that is sorely missing on the internet is a solid video review of it. I think I have had enough time with it now to maybe do that. No promises.

Anyway thanks again for all the work on standby / suspend here!

[jfred]

Hmm - I’ve been trying this suspend script on NixOS without much luck. I’ve only just gotten NixOS running recently, so I’m not yet completely up to speed with everything I need for the Reform to run well.

The symptoms I’ve seen have varied; sometimes the NVMe controller gets disabled (seen in dmesg) and disk reads/writes start failing. On my last attempt I set no_console_suspend and my first resume succeeded, but the machine seemed to go unresponsive after the second resume. I haven’t yet hooked up a serial console during the resume yet though, so there may be debug info I haven’t yet seen.

[chaseadam]

@2disbetter

I believe that this script is robust enough to give most mostly reliable suspend sessions.

Any chance you can post your version of the suspend script with the changes? So far I have not been successful with suspend and am hoping I may have missed an important script change in the discussion.

[2disbetter]

Hi @chaseadam my script is not unique, it is what ruff posted a while back in this thread:

Standby - Suspend to RAM (MNT Reform)

I’m now testing modified suspend script which I plan to make a suspend target unit (to utilize systemd based suspend). The script is below, I have tested it successfully during the day, and left overnight - which also woke up ok. It draws suspiciously big power during sleep though - 240mA in active state and 100mA in suspend (so about 2.5W). I’d expect it to be lower. If I try to poweroff aux (mpcie, by PWR3 command) it does not affect power draw (actually makes it slightly bigger?!) but it al…

If you follow that post and the post right underneath it, you can bind the script to a key combo, and the system will invoke the script when you press it.

For me after upgrading there was a hassle of making sure things were cleared of local, etc. that I can’t quite remember right now. But honestly, this script works well.

I am finding that if I leave some applications open while suspending it increases the likelihood of a crash on resume. One of those applications is Librewriter. I find that if I close that BEFORE suspending the Reform it greatly increases the chances of resume working.

It might be nothing but this is what I have noticed.

[josch]

@2disbetter

Hi @chaseadam my script is not unique, it is what ruff posted a while back in this thread:

Note, that this is also what is done in the reform-standby script shipped by the reform-tools package and invoked by reform-tools.reform-sleep.service:

sbin/reform-standby · main · Reform / MNT Reform Tools · GitLab

If you find anything that is missing from that standby script, please write to me or open a new issue in the reform-tools repo so that everyone can profit from these improvements. Thanks!

[chaseadam]

It looks like it is no longer necessary to perform the steps in this thread as they are included in the image.

@josch

Note, that this is also what is done in the reform-standby script shipped by the reform-tools package and invoked by reform-tools.reform-sleep.service:

ii  reform-tools   1.6          all          MNT Reform System Tools

I performed the steps anyways, so I will undo and use the packaged version. Looks like the packaged script also unbinds the keyboard, which is not mentioned in the script above.

Either way, it looks like I am running what others are running, but I have yet to successfully resume. Power draw is ~100mA. Attempts to resume result in the OLED “Waking LPC 0%”, then shuts off and no other changes. I know debugging power states can be challenging. What is the recommended approach to troubleshoot? Serial console UART?

My environment is as follows:

• system-image v3 with latest updates to SD card (as provided in DIY kit)
• NVMe (as provided in DIY kit) installed, but not in use
• Wifi (as provided in DIY kit) installed and in use

0000:00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
0000:01:00.0 Network controller: Qualcomm Atheros AR928X Wireless Network Adapter (PCI-Express) (rev 01)
0001:00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
0001:01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)

The wakeup settings appeared to already be enabled (but the script sets them again). After fresh boot I see the following:

# cat /sys/devices/platform/soc@0/30800000.bus/30860000.serial/tty/ttymxc0/power/wakeup
enabled
# cat /sys/devices/platform/soc@0/30800000.bus/30890000.serial/tty/ttymxc1/power/wakeup
enabled
# cat /sys/devices/platform/soc@0/30800000.bus/30880000.serial/tty/ttymxc2/power/wakeup
enabled

Linux reform 5.18.0-reform2-arm64 #1 SMP Debian 5.18.2-1+reform1 (2022-06-07) aarch64 GNU/Linux

[humberto]

You’re resuming with Circle-space, right?

[chaseadam]

Good question (because if you use “power on” it will boot from scratch). Yes, I am using Circle-space.

[2disbetter]

You could use this command before you try to standby and resume, it will print out what the system is doing during the running on the script. It might be helpful for you to see what the issue is:

echo N | sudo tee /sys/module/printk/parameters/console_suspend

[chaseadam]

I set console_suspend to N, but the behavior didn’t appear to change for me. I noted that after a fresh boot, the value was set to N, so maybe theat is my default value (or it is sticky between reboots, but I don’t think so).

Should the screen stay on during the suspend to show me the script output or is the output to a serial port I should monitor with a second device?

[2disbetter]

Honestly when suspending I see text fly by in a split second. When resuming however, I see the console and can see it enabling wifi, etc.

I mean when resume fails, I’m not really getting anything from the console output, but maybe in your case it will help you to see something else running a miss.

[chaseadam]

@2disbetter

Honestly when suspending I see text fly by in a split second. When resuming however, I see the console and can see it enabling wifi, etc.

I see the following flick by (had to take a video and go frame by frame):

PM: suspend entry (deep)
Filesystem sync: 0.000 seconds
<screen reset>
Freezing userspace processes ... (elapsed 0.001 seconds) done.
OOM killer disabled.
Frezzing remaining freezable tasks ... (elapsed 0.001 seconds) done.
<screen off>

On Circle-space I see OLED activity, but no change from the screen (stays off).

I have installed quite a few packages on the system, so I am going to try a fresh system-image SD card and see if suspend works for me in a ‘pristine’ OS state.

[2disbetter]

Yeah, because if you are not seeing the screen come back on resume, that means that is not initializing, and why that would be happening is beyond me. I would be interested to know what happens when you go to stock image though.