There is bias in the training data as well as the fine-tuning. LLMs are stochastic, which means that every time you call it, there's a chance that it will accidentally not censor itself. However, this is only true for certain topics when it comes to DeepSeek-R1. For other topics, it always censors itself.
We're in the middle of conducting research on this using the fully self-hosted open source version of R1 and will release the findings in the next day or so. That should clear up a lot of speculation.
The following response is generated by an offline version with only 1.8 Gb of numerical weight values:
Since the model is run offline with fixed weights, the entire response is generated by next-word prediction using preset weights, without agents, web search or other crutches.
When the model is outputing thinking tokens, all previous thinking tokens within the context window are taken into account to predict each subsequent token.
So the model must have been explicitly trained to start answer in a meandering / "what-if-this" "what-if-that" manner. This expands the context of the prompt beyond what is actually given explicitly in the prompt.
Once the meandering / thinking phase is done "alright, that is enough thinking i think", the model can be primed ("so i will now summarize all that i have thought about") to provide a summary (i.e. final answer) based on all the thinking tokens it had previously outputed.
Two major reasons given why a stand-alone transformer is not sufficient to rival human intelligence is due to
Inability to perform tree-like search/calculation.
Lack of a good model of the world/problem given.
The first example seems to show that transformers can use existing world-model (compound interest formula) and perform calculations using that world-model it has internalized.
However, given a more complicated world:
So
The lack of a good world-model (chess board) is fatally hampering its ability to analyze the problem correctly.
This fixed weight implementation of the transformer is unable to learn after being given new information about its previous error. It is not clear if alternative implementation of fixed weight transformers is able to self-learn from newly given information/logic without additional help using agents, web search, etc.
It’s appeared on the Livecodebench leaderboard too. Performance on par with O4 Mini - https://livecodebench.github.io/leaderboard.html
I think the major advancements are outside the textbox in this last year: video generation, robotic models like Helix, world models, Genie 3.
Even for text, Deepseek R1 was this year, and agentic and coding AI has made progress on length/complexity of tasks. The rise of MoE architecture in the open/local model space has made it possible to run useful models locally on hardware under like $2K, something I didn’t expect for a long time.
The "linear" assumption here is worth interrogating. In work I've been doing on alignment evaluation, I find that linear probes can achieve high accuracy on refusal-relevant directions, but that probe accuracy is non-diagnostic for whether the model actually routes behavior through those directions at inference time.
DeepSeek-R1 and Qwen2.5-72B have cleanly separable routing layers (ablating the refusal direction recovers accurate outputs), but Qwen3-8B doesn't - it confabulates, suggesting knowledge and suppression are jointly encoded. Whether a linear alignment method holds up may depend heavily on which of those architectural regimes you're in.
