Debugging power-related firmware

MNT Reform
1

How do I investigate and/or fix LPC firmware related to power and batteries?

Since beginning of April MNT Reform is my daily driver. At the same time I feel it is behaving strangely (I might be wrong here) - or at least I cannot understand its mode of operation.

Just before FOSDEM (beginning of February, when I was still treating MNT Reform more as an experiment) it stopped working: it was not turning on and when I connected power supply (both USB-C PD 20V and enclosed one 24V) D9 LED was dark and D8 was blinking few times a second. Lucas looked at it, did not find anything obvious and advised to send it for deeper investigation. Before I managed to do that - I disassembled everything, disconnecting batteries, and after one day “offline” everything started working again.

Beginning of April - my previous laptop died so I started using MNT Reform daily.

Beginning of May - batteries (still original ones) were discharging quickly. I disconnected everything, used “Analyze” mode of my external charger (fully charge, discharge, charge again) which fixed situation.

Then, beginning of July, one of original accumulators discharged fully, to 0.0V (I do not have new battery pack with discharge protection). My attempts to recover it failed, so I changed all accumulators to Eremit 2000mAh and recompiled firmware to take this change of capacity into consideration. OLED shows firmware version as MREF2LPCR320230703.

After that I can observe rather strange behavior, as if new firmware cannot keep charging state. When I turn on laptop after longer period of it being turned off (longer than 2-3h) OLED shows ???. When I connect
power supply and laptop charges fully, OLED displays 100%, and when I disconnect it, charge drops to 98%, 97%, and so on. If I connect power, charge grows. When I turn off laptop, and then turn it back on after 1h or so - charge is displayed properly. Only when laptop is turned off for longer time, it looks like it “forgets” charge state. Situation is the same for almost 2 months, so much longer than potential “learning period”. I even disconnected everything (including removal of small CR battery from motherboard) and left it over the weekend - it didn’t help. I noticed another strange behaviour. USB-PD 20V from Ada Fruit does not work (I described it shortly here). When I connect it, D9 stays dark while D8 blinks few times a second, similar to situation I described earlier. Laptop works otherwise, it’s just not being charged. OTOH 12V converter works without any problems. I’m not sure if this is problem with power supply or motherboard: from time to time, when I connect normal power source (24V), D8 is on while D9 is blinking once every few seconds (few seconds on, few seconds off). During such behaviour I can observe that OLED shows 0.00A and -1.47A. After few minutes normal charging resumes - but sometimes with -0.7A (or similar) and not usual -1.4A.

I tried to read source (reform2-lpc-fw/src/boards/reform2/board_reform2.c) but haven’t noticed anything obvious. OTOH I do not have experience with such hardware - I was not even able to determine if D8 and D9 are steered by firmware or directly by hardware.

Basically I’m a bit lost. I’m not sure if this is normal behavior or signs of potential problems. And if it’s latter - how can I check what is going on? I can compile new versions of firmware and observe its behavior but lack knowledge to know what to safely change. Any tips will be appreciated. If not - I’ll post if something changes.

2

Hi, some thoughts:

D8 blinks few times a second

This normally happens when the power adapter is not delivering enough current to Reform. It is advised to use a stronger power supply.

D8 is on while D9 is blinking once every few seconds (few seconds on, few seconds off).

This sounds normal, it should be the balancing behavior. Charging is turned off during balancing (when discharging overvolted cells).

D9 will go on when the LPC firmware turns off charging, see:

When I turn on laptop after longer period of it being turned off (longer than 2-3h) OLED shows ???.

I also have this recently. I would like to fix it, but didn’t have the time yet due to a perfect storm here of a lot of new products/batches being readied for testing and shipment.

I suspect it is the brownout reset being a bit overzealous. I activated it in a relatively recent version of LPC firmware to make sure the LPC never crashes/freezes/is able to recover from undervoltage crashes. Perhaps one can tweak its sensitivity. See the LPC11U24 reference manual.

See also:

LPC11U24 manual:

3

Also, when rereading i see that you don’t have the protected battery boards. I strongly, strongly suggest to get them.

4

I now have the new Reform and experience similar issues - so want to resume investigation to fix problem with charging.

To summarize: in original MNT Reform I had troubles with charging using USB-C 20V adapter from Ada Fruit, mentioned in another thread. When using it, Reform was not charging, D9 was off and D8 was blinking few times a second - according to description above it was a sign of not enough power being provided.

Original charger (24V 2.5A) never had troubles, and when connected OLED was displaying -1.4A. What’s interesting - lover voltages (e.g. USB-C 12V) never had troubles with charging, but it was taking longer: OLED displayed around -0.4A. I assume this is caused by need to change voltage from 12V to almost 30V - thus decreasing current.

Now I can observe really strange behavior; new Reform, motherboard v2.5, A311D, original LPC firmware (I haven’t yet recompiled it). First - A311D is much hotter than iMX8; currently cat /sys/class/hwmon/hwmon0/temp1_input shows 84900, while iMX8 was around 48-55C. It also uses more power: OLED usually shows around 0.560A when working on battery, while iMX8 was using around 0.250A. It also means that battery life dropped from 6h to slightly above 2h. I’m providing this data as it might be sign of potential troubles - but computer is fully working, except for WiFi which I haven’t yet got to run.

And now new machine shows similar behavior when charging, but now also with original 24V charger. Basically when it is discharged and running (not off - so CPU is power hungry) it has troubles getting enough power. And I cannot make sense of it: When I plug in barrel-jack to laptop and then connect plug to mains it usually cannot charge (D8 is blinking); but when I unplug charger from both sides, first connect 230V, and then connect 24V to laptop - it can charge.

Two times it started charging, went up to 80% and 79%, and stopped charging - again, D8 started blinking and OLED was showing 0.558A instead of -1.4A.

Today I turned on laptop, kept it on charger (so battery had 100%), then went shopping. When I returned, I resumed mail reading, and after some time checked power: it was working from battery which went down to 30%, and again D8 was blinking. I “restarted” charger, it was able to charge to 38% - and again switched to battery. This time 24V charger was not able to provide more power. I switched to USB-C 12V and now laptop is slowly charging (-0.16A displayed at the moment of writing).

How can I fix it? I can provide more details, or try new firmware, but do not have enough experience with low-level programming or power supply to propose my own experiments or explanations.

1 Like
5

Hi serpent,

I’m sorry to hear about these issues. I see two distinct problems here:

  1. Your A311D is running abnormally hot. Normally it should run at around 50 degrees with heatsink. And it can also run much longer without a heatsink than i.MX8MQ because it is much more efficient. So something is probably wrong at the hardware level (I don’t suppose you run all the cores at 100% all the time). Is the heatsink attached correctly incl the thermal pad, and the plastic removed from the thermal pad? If so, there might be some soldering issue on the RCM4 adapter. If you can’t find anything obvious and it still gets hot, please contact support@mntre.com and we’ll sort it out with a replacement. But also, the current is abnormal: 0.56A is too high for an idle desktop. For me it’s around 0.35-0.4A with normal use (browser, display at full brightness).

  2. There is a newly known issue with MB2.5 where it charges at a quite high peak current which can be more than 60W (more like 70W). So the default charger and USB-C chargers can be unhappy about this and turn off. But what works for me and another user is this patch where we wait a little bit after connecting the charger before enabling the full current, maybe you can give it a try: WIP: on mb25, soft-start charging, i.e. wait a bit after charger is connected (!66) · Merge requests · Reform / reform · GitLab

1 Like
6

Good news (mostly). Now after changes, my new machine behaves much better. Long story short - originally I used thermal pasta, and it looks like this was not the best choice. Today I removed paste, attached thermal pad, and computer uses much less energy (current reading is 0.268A), and does not get so warm (38.9C after 1h of work). To summarize - it is worth to read and follow instruction :slight_smile:

On Monday I compiled new firmware (with change from branch mb25-charge-delay) and charging behaves much better. I haven’t tested it fully with USB-C PD yet, but original charger does not have any troubles.
After first run (just after FW upgrade) shutdown -h now did not fully turned computer off - keyboard backlight was still on and I could not start system after putting in SD-card. But after circle-0 everything went OK. And once, after turning system off (with charger plugged in and battery at 100%), both D8 and D9 went dark. Not sure if this is sign of some problems, but wanted to let you know.

To summarize - system behaves much better. I’ll let you know how it behaves after some more observation.

1 Like
7

I’m also seeing the “cannot charge with default charger” issue on my new Reform (with BananaPi CM4). I’m pretty sure the hypothesis that the laptop draws a lot of power right after connecting the supply applies in my case too, as charging is stable once it has been succesfully started. It also seems to just be barely too much power draw for the stock charger, as system load and display backlight have a reproducible effect on it. If the system load is low, starting the charging process works. If the system load is high (for instance while compiling something), the charging process will fail with D8 blinking, unless the display backlight is switched off.

My CPU temperature also sometimes goes up to near 80°C while compiling software. This might however be due to a bad thermal contact with the heatsink, as I’m not certain that my thermal pad is still OK.

I’ll have a close look at power consumption and thermals next time I get to work on the laptop (probably the weekend), and I’ll give the “soft-start charging” change a try, to see if that allows me to start charging at 100% CPU load.

1 Like
8

Preliminary report from my side: I now flashed the firmware and tried plugging/unplugging the power supply several times while having all cores of the CM4 at full load. It seems the changes on the soft-start branch have resolved my charging issues.
In addition, I squished the thermal pad a bit to ensure that it has good contact to the chip, and CPU temperature is now stable, at about 60°C after an extended period of full load.

9

It has been a while, and I am still running the same firmware, with the delayed charging start. I’m sorry to report that today I ran into the same issue, that charging does not start, again.

So, it seems that this firmware change does not solve the problem completely, though it seems to be a noticeable improvement, as previously I could reproduce the issue reliably, and now it only happened once and does not happen again if I disconnect/reconnect the power supply.

1 Like
10

So far I had more luck - for me new firmware (looks like change we’re discussing was merged into main branch) solved problem: I haven’t had problems since installing mb25-charge-delay patch.
Situation from my original post (USB-C 20V cannot charge laptop, blinking D8, while 12V can) persists. But as long as at least some of the chargers work (including original one) I can live with that.

11

Firmware still works, I haven’t had problems with charging.
Few times Reform had problems with cold start. After Circle-1, OLED was showing percentage, went to 100% and displayed “No response from LPC”. After Circle-r (Reset) or 0+1 (Power Off and then On), Reform was able to start without more troubles. It’s not happening often (about 5 times since February) so I’m not sure if it’s worth much investigation. OTOH, if you prepare changes I can test those. But, as problem occurs rarely, I’m not sure how be sure if such a change helped.
Another strange issue, again more inconvenience than real problem. When I enable keyboard backlight and then connect something needing more power to USB (like spinning external drive, etc.), backlight turns off. I can enable it back (Circle-F2), but when connect something again - backlight turns off again. This occurs both when running on battery and connected to charger. I just wanted to write it down, in case someone observes something similar.

1 Like
12

I haven’t noticed the keyboard lights, but I’ve had trouble with spinning usb drives. I ended up having to plug in a powered usb hub to get at some backup disks.