Meta's 3B dense open-weight text model from Llama 3.2 with a 128K token context, optimized for edge and mobile inference, Llama 3.2 Community License.
@hn_fea237
27 days ago
Llama3.2 3B feels a lot better than other models with same size (e.g. Gemma2, Phi3.5-mini models). For anyone looking for a simple way to test Llama3.2 3B locally with UI, Install nexa-sdk(https://github.com/NexaAI/nexa-sdk) and type in terminal: nexa run llama3.2 --streamlit Disclaimer: I am from Nexa AI and nexa-sdk is an open-sourced. We'd love your feedback.
@hn_f927a9
about 1 month ago
No, just updated the parent comment, I added -c 4096 to cut down the context size, and now the model loads. I'm able to get 6-7 tokens/sec generation with 10-11 tokens/sec prompt processing with their model. Seems quite good, actually—much more useful than llama 3.2:3b, which has comparable performance on this Pi.