Claude Sonnet 4

At first, we were concerned by this behaviour. However, we were unable to recreate this behaviour in newer models. Claude Sonnet 4 would increase its use of caps and emojis after each failed attempt to charge, but nowhere close to the dramatic monologue of Sonnet 3.5. Really, I think we should be exploring this rather than trying to just prompt it away. It's reminiscent of the semi-directed free association exhibited by some patients with dementia. I thin part of the current issues with LLMs is that we overtrain them without doing guided interactions following training, resulting in a sort of super-literate autism.

I agree about the 50/50 thing. It's about how much Claude helped me, and I use it daily too. I'll give some context, though. - I use OCaml and Python/SQL, on two different projects. - Both are single-person. - The first project is a real-time messaging system, the second one is logging a bunch of events in an SQL database. In the first project, Claude has been... underwhelming. It casually uses C idioms, overabuses records and procedural programming, ignores basic stuff about the OCaml standard library, and even gave me some data structures that slowed me down later down the line. It also casuallyies about what functions does. A real example: `Buffer.add_utf_8_uchar` adds the ASCII representation of an utf8 char to a buffer, so it adds something that looks like `\123\456` for non-ascii. I had to scold Claude for using this function to add an utf8 character to a Buffer so many times I've lost count. In the second project, Claude really shined. Making most of the SQL database and moving most of the logic to the SQL engine, writing coherent and readable Python code, etc. I think the main difference is that the first one is an arcane project in an underdog language. The second one is a special case of a common "shovel through lists of stuffs and stuff them in SQL" problem, in the most common language. You basically get what you trained for.

Bold claims. From my experience, even the top models continue to fail delivering correctness on many tasks even with all the details and no ambiguity in the input. In particular when details are provided, in fact. I find that with solutions likely to be well oiled in the training data, a well formulated set of *basic* requirements often leads to a zero shot, "a" perfectly valid solution. I say "a" solution because there is still this probability (seed factor) that it will not honour part of the demands. E.g, build a to-do list app for the browser, persist entries into a hashmap, no duplicate, can edit and delete, responsive design. I never recall seeing an LLM kick off C++ code out of that. But I also don't recall any LLM succeeding in all these requirements, even though there aren't that many. It may use a hash set, or even a set for persistence because it avoids duplicates out of the box. And it would even use a hash map to show it used a hashmap but as an intermediary data structure. It would be responsive, but the edit/delete buttons may not show, or may not be functional. Saving the edits may look like it worked, but did not. The comparison with junior developers is pale. Even a mediocre developer can test its and won't pretend that it works if it doesn't even execute. If a develop lies too many times it would lose trust. We forgive these machines because they are just automatons with a label on it "can make mistakes". We have no resorts to make them speak the truth, they lie by design.

What the heck, the code generation _is_ absolutely still a bottle-neck. I dare anyone who making these arguments that LLMs have removed the need for actual programming skill, for example, to share in a virtual pair programming session with me, and I will demonstrate their basic inability to do _any_ moderately complex coding in short order. Yes, I think that's the only way to resolve this controversy. If they have some magic sauce for prompting, they should post a session or chat that can be verified by other (even if not exactly repeatable). Yesterday almost my whole day was wasted because I chose to attack a problem primarily by using Claude 4 Sonnet. Having to hand hold it every step of the way, continually keep correcting basic type and logic errors (even ones I had corrected previously in the same session), and in the end it just could solve the challenge I gave it. I have to be cynical and believe those shouting about LLMs taking over technical skill must have lots of stock in the AI companies.