4.0
@hn_8d8355
29 days ago
I'm glad I saw this because llama-3.3-70b-versatile just stopped working in my app. I switched it to meta-llama/llama-4-scout-17b-16e-instruct and it started working again. Maybe groq stopped supporting the old one?
Meta's Llama 4 Scout MoE model with 17B active / 16 experts (109B total), 10M token context, natively multimodal, under the Llama 4 Community License.
@hn_8d8355
29 days ago
I'm glad I saw this because llama-3.3-70b-versatile just stopped working in my app. I switched it to meta-llama/llama-4-scout-17b-16e-instruct and it started working again. Maybe groq stopped supporting the old one?
@hn_e9000d
about 1 month ago
But Llama 4 Scout does badly on long context benchmarks despite claiming 10M. It scores 1 slot above Llama 3.1 8B in this one[1]. [1] https://github.com/adobe-research/NoLiMa
@hn_090ed5
about 1 month ago
1. rent 4x H100 instance on http://app.hyperbolic.xyz/compute (Llama 4 Scout has 109B parameters in bf16, so the weights are already 218GB) 2. open a terminal tool and SSH into the machine 3. run the following commands: >> sudo apt-get update && sudo apt-get install -y python3-pip >> pip install -U vllm >> pip install -U "huggingface_hub[cli]" 4. get an access token on u/huggingface website and run >> huggingface-cli login 5. use @vllm_project to serve Llama 4 >> vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 10000 6. open a new terminal and call the API to know "What can I do in SF?": >> curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What can I do in SF?"} ] }' It's just that simple ;) A big thank you to @AIatMeta and @vllm_project for making it easy to access the best open intelligence!
@hn_90f6c7
about 1 month ago
Yep, can confirm, I used their API just fine for Llama 4 Scout for weeks now.
@hn_faed12
about 2 months ago
There were actually multiple bugs which impacted long context benchmarks and general inference - I helped fix some of them. 1. RMS norm eps was 1e-6, but should be 1e-5 - see https://github.com/huggingface/transformers/pull/37418 2. Llama 4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889 3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311 If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for Scout actually did better than inference providers by +~5% on MMLU Pro. https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-... has more details