Claude Mythos

The fact that with AI development, your brain is no longer in a tight feedback loop with the codebase, leading to a significant drift between your model and reality, is still a sticking point with me and agentic development. It feels like trying to eat with silicone rubber chopsticks. I lose all precision and dexterity. I still keep hoping there'll be a glut of demand for traditional software engineers once the bibbi in the babka goes boom in production systems in a big way: https://m.youtube.com/watch?v=J1W1CHhxDSk But agentic workflows are so good now—and bound to get better with things like Claude Mythos—that programming without LLMs looks more and more cooked as a professional technique (rather than a curiosity or exercise) with each passing day. Human software engineers may well end up out of the loop completely except for the endpoints in a few years.

Honestly if that was some kind of research paper, it would be wholly insufficient to support any safety thesis. They even admit: "[...]our overall conclusion is that catastrophic risks remain low. This determination involves judgment calls. The model is demonstrating high levels of capability and saturates many of our most concrete, objectively-scored evaluations, leaving us with approaches that involve more fundamental uncertainty, such as examining trends in performance for acceleration (highly noisy and backward-looking) and collecting reports about model strengths and weaknesses from internal users (inherently subjective, and not necessarily reliable)." Is this not just an admission of defeat? After reading this paper I don't know if the model is safe or not, just some guesses, yet for some reason catastrophic risks remain low. And this is for just an LLM after all, very big but no persistent memory or continuous learning. Imagine an actual AI that improves itself every day from experience. It would be impossible to have a slightest clue about its safety, not even this nebulous statement we have here. Any sort of such future architecture model would be essentially Russian roulette with amount of bullets decided by initial alignment efforts.