GPT-OSS-20B

Public discussion, not a rating submitted on Rate My AI.

For reference in case it's interesting to someone, a 5090 on Windows 11 with CUDA 13.1 | model | size | params | backend | ngl | test | t/s | | --------------------- | ---------: |--------: | -------- | --: |------: |----------------: | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 999 | pp2048 | 10179.12 ± 52.86 | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 999 | tg128 | 326.82 ± 7.82 | | qwen35 27B Q6_K | 23.87 GiB | 26.90 B | CUDA | 999 | pp2048 | 3129.92 ± 5.12 | | qwen35 27B Q6_K | 23.87 GiB | 26.90 B | CUDA | 999 | tg128 | 53.45 ± 0.15 | build: 9d34231bb (8929) gpt-oss-20b-MXFP4.gguf Qwen3.6-27B-UD-Q6_K_XL.gguf Using MXFP4 of GPT-OSS because it was trained quantization-aware for this quantization type, and it's native to the 50xx.

Public discussion, not a rating submitted on Rate My AI.

I have 2 of them. I would advise against if you want to run things like vllm. I have had the cards for months and I still have not been able to create a uv env with trl and vllm. For vllm, it’s works fine in docker for some models. With one gpu, gpt-oss 20b decoding at a cumulative 600-800tps with 32 concurrent requests depending on context length but I was getting trash performance out of qwen3.5 and Gemma4 If I were to do it again, I’d probably just get a dgx spark. I don’t think it’s been worth the hassle.