This might not be of much use to @johkra but @lykso might find this interesting when getting NixOS to run on the ls1028a…
I’m running Gentoo on ls1028a and I used to have intermittent USB problems as well. On some boots, the keyboard would not work and I had to reboot to fix the issue. Some times it took a couple of reboots.
Over in the fediverse, mnt confirmed that this is a real issue, probably a bug in the driver.
In the official image, this is worked around by running this script that reloads the driver.
I ended up writing a custom dracut module to run the script in my initrd.