I know the MNT Pocket reform has an M.2 slot.
How feasible would this be to add an AI accelerator chip like M5 stacks to it:
Or perhaps the Hailo-8 M.2 AI Accelerator Module: https://www.newegg.com/p/3C6-005Y-01AC5
2 Likes
The Hailo looks more compatible as it doesn’t have a giant heatsink. The RK3588 also has a 6 TOPS NPU already, have you looked into it? I haven’t tested any of these as I’m quite upset about people using LLM coding agents causing much trouble for free and open source software in general, and the cloud AI datacenter hype ruining general purpose computing prices/supplies for many people, not to speak about the environmental impact and the web being filled with garbage generated content. That said, perhaps you have a very different application in mind, and you have of course the freedom to run any software/hardware you want in your MNT device.
9 Likes
There is no way these accelerators could run a LLM with any reasonable power. But what they can do is accelerate models like BERT and kNN algorithms. Vector word embeddings are really cool and I think you should check them out. My personal goal is to have the equivalent of Google circa 2010 (with all of wikipedia, wikidata, etc) on a single device.
I promise: no chat bots 
6 Likes
As far as I understood it, the Hailo 10H is capable of running LLMs.
Jeff Geerling wrote a nice post about it and the Raspberry PI 5: Raspberry Pi's new AI HAT adds 8GB of RAM for local LLMs - Jeff Geerling .
What I took away from Jeff Geerling’s post is that the Raspberry Pi can already run larger models and has more power to do so as compared to the Hailo 10H HAT. The AI HAT is nice if you want to offload AI workloads and use the CPU for something else.
@minute I believe the open source movement needs to figure out how to build an effective ecosystem of AI tools which are freedom respecting. That requires to run LLMs on hardware you own.
Further, I think there is a lot of potential to reduce the environmental impact of LLMs by optimizing hardware and software. Quantization can help on the software side: A 30B Qwen Model Walks Into a Raspberry Pi… and Runs in Real Time .
On the hardware side it would be nice to have accelerators which follow an open approach. My wishlist: run on free software, have open source drivers, are maintainable and are open hardware.
I would love to see an AI accelerator for the MNT Reform devices. One which follows open principles.
i agree with the environmental thing in that they need to be optimized, and i think as NPUs become more efficient, and models are slimmed down, they will be.
theoretically in the future, personal, locally run, and user-controlled AIs may become useful and common, but i doubt they’ll be anything more than an email auto reply assistant or similar until we can get NPUs with qbits in them, and the ability to run real AI, not just LLMs or similar VI, or virtualised intelligence, which i use since LLMs are just glorified GPIO machines when you get down to it, useful, but still.
regardless, i think right now LLMs are in a bad place with corporate control and closed source qualities, but things like ollama and hugging face are pushing things to be better, more open, more personal, and LLMs or similar VI, and eventually true AI, may become useful one day, but i think the only way they will be is if they’re user-first, locally run, and the hardware to run them is open from schematics to firmware to drivers, all the way up to the LLM itself.
theoretically, in my opinion the best outcome would be if we can achieve true AI one day, and learn to teach it to ally with humanity, and not go all skynet, and then make it so everyone can have their own personal AI. assuming society can be healthy about it, it may help supplement human need for connection and facilitate it by offering to call friends, or a therapist if you’re struggling, and it could then help mental health, and humanity as a whole, but that’s a difficult reality to make and relies on a difficult assumption.
1 Like