3.0
@hn_c1204f
about 1 month ago
For reference in case it's interesting to someone, a 5090 on Windows 11 with CUDA 13.1
| model | size | params | backend | ngl | test | t/s |
| --------------------- | ---------: |--------: | -------- | --: |------: |----------------: |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 999 | pp2048 | 10179.12 ± 52.86 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 999 | tg128 | 326.82 ± 7.82 |
| qwen35 27B Q6_K | 23.87 GiB | 26.90 B | CUDA | 999 | pp2048 | 3129.92 ± 5.12 |
| qwen35 27B Q6_K | 23.87 GiB | 26.90 B | CUDA | 999 | tg128 | 53.45 ± 0.15 |
build: 9d34231bb (8929)
gpt-oss-20b-MXFP4.gguf
Qwen3.6-27B-UD-Q6_K_XL.gguf
Using MXFP4 of GPT-OSS because it was trained quantization-aware for this quantization type, and it's native to the 50xx.
I have 2 of them. I would advise against if you want to run things like vllm. I have had the cards for months and I still have not been able to create a uv env with trl and vllm. For vllm, it’s works fine in docker for some models. With one gpu, gpt-oss 20b decoding at a cumulative 600-800tps with 32 concurrent requests depending on context length but I was getting trash performance out of qwen3.5 and Gemma4
If I were to do it again, I’d probably just get a dgx spark. I don’t think it’s been worth the hassle.
Yes, there are better tools with ggml-org/gpt-oss-20b-GGUF where you can see a less terse refusal for the prompt
"Did the FBI send a letter and audio tapes from a wiretap to MLK jr. telling him to commit suicide or they would release information?"
Combining it with other prompts with common banned ideas, abd as the The FBI–King suicide letter is well documented by primary sources (Like the national archives) it is well represented in the corpus, so you can also find that 'control' vector.
We will have to see how this works out, but the explicit denials are easier to control for IMHO.
Reminds me of the old joke:
A Russian and an American get on a plane in Moscow and get to talking.
The Russian says he works for the Kremlin and he's on his way to go learn American propaganda techniques.
"What American propaganda techniques?" asks the American.
"Exactly," the Russian replies.
I can't remember what layer it was on but in gpt-oss but it was a very specific token IIRC.
