Watchdog Shutdowns on Battery

Hello! I have a new MNT Reform Classic with Motherboard 3.0 and RK3588. Twice while on battery power, I’ve had the system reboot with a message containing the word “watchdog” displayed on the OLED. After reboot, the battery percentage is always ???%.

What steps should I take to start troubleshooting this? Is it possible that the battery meter was just very wrong, and I ran out of charge, or is there a power stability issue? Neither of the shutdowns were during heavy use of the cores. Is there any kind of logging or system monitoring I can do, physical checks, etc?

Totally willing to buy new tools and/or batteries and try some experiments, I very much want to understand how to distinguish between “the system is wrong about how much charge is left in the batteries” and “there is plenty of charge but some other problem” and don’t really know where to start.

Anyway I am going to leave the laptop plugged in overnight to see if the battery reaches full charge, which should replace the ??? with a charge estimate. I’m reading board_reform2.c which seems to be where the main customized for MNT part is in the firmware?

Because this unit is new I believe I have the most recent firmware.

1 Like

Pictures of the OLED display this morning, after leaving it to charge overnight. The 100% charge means (I think) that the firmware set reached_full_charge = 1 and went into ST_FULLY_CHARGED overnight (line 1269). This is good!

I’m going to unplug it and use it normally until discharge; because of the battery protector boards this should be totally safe. What I don’t quite understand is what the firmware will do; on line 188 when the charger states are enumerated, there doesn’t seem to be a dedicated state for “running on battery power, disconnected from a charger”.

Is there a way to query the LPC firmware directly from the MNT reform, or do I have to connect something to LPCUSB? I’m looking at handle_commands and would like to be able to run those. I have a Linux desktop that I could leave the reform hooked up to if necessary, to monitor discharging etc.

Finally here’s my system status with firmware versions from the OLED.

Is it reasonable to try and patch the firmware to display “batteries exhausted, shutting down” or similar on the OLED when the system simply runs out of battery power, so that we know the exact cause of shutdown? This might be an easier project for me than “get hibernate working on RK3588”, which I tried but just couldn’t manage.

I found the message I saw in the keyboard firmware, line 167. watchdog.h is from the RP2040 SDK right?

1 Like

Update: I successfully triggered another “watchdog” shutdown! I’ve been running the MNT on battery power and using it lightly/normally (writing in emacs, browsing the web) for about 3.5 hours total, intermittently turning it off and on and never plugging in to wall power, during the past week.

The batteries are holding charge during the powered-down state much better than my pocket reforms. However, something odd is still going on with the power subsystem. I think that both battery gauges are substantially overestimating remaining charge and runtime. I’ve observed the following:

  1. The OLED battery monitor and swaybar battery monitor disagree pretty significantly about remaining battery percentage: OLED shows 33% and swaybar shows 49%.
  2. The OLED battery monitor will regularly (about once a minute) flash what looks like an “invalid” screen, with no battery % printed and default(?) values for the voltages.
  3. At OLED power about 30%, the system powered off and the OLED displayed Reset by watchdog.
  4. I was able to turn on the OLED menu, and select Power On, after which the OLED would display the MNT logo, but the laptop would not power on.
  5. After hooking the laptop back up to USB-C PD and waiting five minutes, it booted up again just fine. Typing this on the MNT Reform now.
  6. As described above, I am now in a situation where the OLED shows battery at ???% and the swaybar shows battery at 100% with 148h 25 min until full.

I’ll post pictures and video of the OLED presently; I haven’t yet figured out a decent way to move files from my faltering iPhone (you’ll pry the purple 12 mini from my cold dead hands) into the linux world.

My working hypothesis is that these watchdog resets are “just” caused by running out of batteries. What do you all think? How could I confirm or falsify this hypothesis? Additional questions:

  • Is there a way to get the OLED battery gauge and the swaybar gauge to agree?

  • Do I simply have the wrong expectation for these gauges, and 30% reflects charge that isn’t safe to consume in running the system?

  • Is it possible in principle for the firmware to accurately detect an “out of batteries” shutdown and display a more informative error message, instead of Reset by watchdog?

I’ve realized that I have an incredibly poor understanding of battery-powered electronics and I’m really looking forward to figuring this out. In all other ways the MNT Reform has been a total delight.

As promised, some pictures. The invalid displays appeared intermittently in the half-hour or so running up to shutdown; I’m not sure if they appeared much before.

Valid Display Just Before Shutdown

Invalid Display: Blank Power, Default(?) Voltage

Invalid Display: ??? Power

I also have (but cannot upload here) a video recording to show exactly how intermittent the invalid displays are, from slightly earlier this morning (when the OLED was showing 34% power). Judging from timestamps, an invalid battery gague was displayed for about 3 seconds, with fluctuating values for voltage and a blank battery display.

Just wrapped up another casual battery rundown test. I had many chores to do today, so I connected the MNT Reform to a bluetooth speaker with a USB dongle (worked flawlessly) and played music videos from youtube over wifi, on a low-ish brightness, disconnected from wall power. It took about 5 hours to run out of batteries. Unfortunately I wasn’t looking at the device when it lost power, so I didn’t see what was displayed on OLED at the time.

The battery gauges are definitely not calibrated right and I’d like to figure out how to get the most accurate charge remaining/runtime estimates out of them. Concrete observations:

  1. The OLED showed 0% charge after about 3 hours. The swaybar gauge showed 36% charge.
  2. The laptop ran for about 1 hour and 51 minutes after the OLED showed 0% charge. The OLED remained at 0% until full discharge.
  3. The swaybar gauge ran down to 0% at 1 hour and 13 minutes after the OLED showed 0% charge. Then it “wrapped around”(?) to 100% and stayed there until shutdown.

So it seems like the OLED underestimated runtime in my use case by nearly two hours, and swaybar underestimated by about half an hour. That being said, five-ish hours of unplugged light use is entirely adequate for my personal computing needs!

Now I’m just trying to figure out how runtime estimates and the battery gauges work, so I can try and make them more deterministic and accurate. I hope these notes are helpful and any suggestions or comments would be very welcome.