Author here. Quick context on what made this worth writing up: Gemma 4 26B A4B is an MoE — 26B total params, 4B active per token — which fundamentally changes what’s viable on a single consumer GPU. During decode you pay the memory bandwidth cost of a 4B model but get the quality of a 26B. That’s what makes a 5090 a real option for it; a dense 26B wouldn’t be.
The interesting part was the quant format choice. NVFP4 is Blackwell’s native 4-bit format and theoretically the fastest path, but MoE support for Gemma 4 specifically was blocked on an unmerged vLLM PR (#39045) — linear layers loaded, expert weights didn’t. Falling back to nightly didn’t help because that day’s nightly was broken by someone landing an unconditional pandas import in the AITER code path without updating the image’s deps. Ended up on AWQ + Marlin kernels, which has been stable in vLLM for over a year. For single-user memory-bandwidth-bound decode the gap to NVFP4 is smaller than you’d expect — both hit the same 4x weight compression, and AWQ dequantizes to FP16 in-register rather than using FP4 tensor cores. I’m getting ~196 tok/s; I’d estimate NVFP4 would be 220-240 if it had worked.
Happy to dig into the vLLM config, the RunPod Serverless side, or the NVFP4 vs AWQ tradeoff in more depth.
This app is cool and it showcases some use cases, but it still undersells what the E2B model can do.
I just made a real-time AI (audio/video in, voice out) on an M3 Pro with Gemma E2B. I posted it on /r/LocalLLaMA a few hours ago and it's gaining some traction [0]. Here's the repo [1]
I'm running it on a Macbook instead of an iPhone, but based on the benchmark here [2], you should be able to run the same thing on an iPhone 17 Pro.
[0] https://www.reddit.com/r/LocalLLaMA/comments/1sda3r6/realtim...
[1] https://github.com/fikrikarim/parlor
[2] https://huggingface.co/litert-community/gemma-4-E2B-it-liter
What hardware are you running? Parakeet runs on nvidia and Mac and it’s way faster than Whisper. And I’ve had issues with training Qwen3 (and even Qwen2.5 but I think I was masking stop tokens wrong). I’ve had success with Gemma 3 though, and they have some really small models (270m and 1b). Maybe 270m for just transcript cleaning? I wonder if the 1b model can handle the transcript analysis…
The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.
It doesn’t render Markdown or LaTeX. The scrolling is unusable during generation. E4B failed to correctly account for convection and conduction when reasoning about the effects of thermal radiation (31b was very good). After 3 questions in a session (with thinking) E4B went off the rails and started emitting nonsense fragment before the stated token limit was hit (unless it isn’t actually checking).
