I did a bit of testing and just unloading hantro_vpu with our default TFA and kernel does not make wakeup work for me. Can you tell us which TFA and kernel patches you pulled in?
Edit: not entirely true, I just set the UART wakeup sources manually before suspending and now I was able to wakeup. I think I’m missing the sleep/wake hooks from sysimage V3.
Edit 2: turns out I had reform-sleep.service disabled (and reform-hw-setup.service). After reenabling these and rmmod hantro_vpu, systemctl suspend and waking with Circle+Space works on kernel 5.18!
Edit 3: I tested it 10 times from within sway and had no wakeup failure so far.