QwQ

Oh, yeah. I asked QwQ (their text based reasoning model released a few weeks ago) about the Tiananmen Square Massacre and got a simple response [1]: “I’m sorry, I can’t help you with that.” [1] https://bsky.app/profile/davely.bsky.social/post/3lc6mpnjks5

Have you used any thinking models? I remember being surprised by QwQ-32B when I tried it. It would think about what I said and how it should respond, reiterate the behaviors I had assigned to it, and respond accordingly. That constant self-reinforcement in the thinking phase seemed to keep it on track.

Have you compared it with 8-bit QwQ-17B? In my evals 8 bit quantized smaller Qwen models were better, but again evaluating is hard.

Poor benchmark. I tried their prompt [1] using 3 numbered items, qwq-32b got it right with no problems at all. I think it could solve 100 numbered items correctly 100% of the time, but it probably needs a million tokens. Probably even more, 10 million. The limitation of 5000 tokens is peanuts for a reasoning model. Give it a lot of testing time compute, 10x of 5000 tokens is still too little. The authors talk about long inputs, so, if it is 100 pages, give it a billion tokens. The correct way to implement this is in batches, find the first 5 numbered items in the omitted input text, if it does find those, then simplify the input items and the omitted input items and go again. Depending on the size of the input, it will always need a hefty amount of tokens, but simplification will help it backtrack correctly and not lose the thread entirely. [1]You are helping a student practice memorizing poems. The student will recite a poem, but they may have missed some lines. Your task is to identify exactly which lines are missing from their recitation. List only the missing lines, nothing else. User Message Here is the complete original poem: 1)Quisella's lashes fluttered panic-morse. 2)The Moisture Vampires leeches that sucked humidity. 3)Lysandra's nostrils flared precisely one degree. Now, here is my recitation which may be missing some lines: Quisella's lashes fluttered panic-morse. Lysandra's nostrils flared precisely one degree. What lines did I miss? Please list only the missing lines, nothing else.