Phi-3-mini

No, but if you are thinking about edge compute for LLMs, you quantize. Models are getting more efficient, and there are plenty of SLMs and smaller LLMs (like phi-2 or phi-3) that are plenty capable even on a tiny arm device like the current range of RPi "clones". I have done experiments with 7B Llama3 Q8 models on a M3 MBP. They run faster than I can read, and only occasionally fall off the rails. 3B Phi-3 mini is almost instantaneous in simple responses on my MBP. When I want longer context windows, I use a hosted service somewhere else, but if I only need 8000 tokens (99% of the time that is MORE than I need), any of my computers from the last 3 years are working just fine for it.

The initial model release had a terrible, frequent, issue with emitting the wrong "end of message" token, or never emitting one.[1] That is a very serious issue that breaks chat. The ones from today still have this issue.[2] Beyond that, they've been pushing new ONNX features enabling LLMs via Phi for about a month now. The ONNX runtime that supports it still isn't out, much less the downstream integration of it into the iOS/Android runtimes. Heck, the Python package for it isn't supported anywhere but Windows. It's absolutely wild to me that MS is pulling this stuff with ~0 discussion or reputation repercussions. I'm a huge ONNX fan and bet a lot on it, it works great. It was clear to me about 4 months ago that Wintel's "AI PC" buildup meant "ONNX x newer Phi" It is very frustrating to see an extremely late rush, propped up by potemkin blog posts that I have to waste time to find out are just straight up lying. Burnt a lot of goodwill that they worked hard to earn. I am virtually certain that the new Windows AI features previewed about yesterday are going to land horribly if they actually try to land them this year. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://x.com/jpohhhh/status/1793003272187351195