o3-mini

I’d agree with all of this, although I’d also point out o3-mini is very fast and cheap.

The whataboutism of that section was odd. The only non-handwavy argument presented is that due to the scale of LLM training that models' output should be treated like US-specific typeface forms' copyright, ie: non-applicable. It's interesting as typeface plagiarism became rampant beginning in the 70s when more accurate photo reproductions made it trivial. This was problematic for designers wanting to make a livelihood, which is something ITC sought to mitigate by better up-front payments (IIRC from U&lc's coverage) to incentivize quality typeface creation. There's a distinction though between literal plagiarism and just inspiration from elements. US copyright law doesn't protect either for typeface forms but ironically it does allow copyright for the code used in font files. I've seen OpenAI's o3-mini (their reasoning model) suggest verbatim code and comments that I found on Github predating LLMs by years. It seems the more times the same code and comment appears online the more likely this is to occur. I'd imagine there would be studies looking into the scope and frequency this occurs and how much is considered fair use.

Working on an embedded DSL for writing typed LLM workflows in Haskell: https://github.com/outervation/promptyped . It handles the "LLM in a loop" structuring for e.g. forcing an LLM to keep making changes in a loop until unit tests pass, and custom validation so the LLM is forced to retry a task if the validation fails (validation can include using a LLM to check the result, or plain Haskell function, or e.g. running a compiler/unit tests). It also allows structured context management, allowing the user to specify which files are initially provided as context to a task (which may itself be the result of a LLM query), as well as providing the LLM tools to open files it deems relevant, and prompting it to close the least important file when it has too many open (and compressing non-recently-modified source files when many are open). Currently heavily WIP though and only has a working Go compile/test backend (Go seems particularly easy for smaller LLMs like o3-mini to work with due to being a relatively non-complex language).

FWIW I just tried making a maze game and having an LLM solve it step by step, where each move the LLM has to choose from "move forward", "turn left", "turn right", and is provided with a current state e.g., "right: blocked, left: open, forward: open, back: blocked" after each move. I wanted to see if the LLM could maintain some internal context so as not to explore paths that it has already seen before. In chat mode, it didn't. It would get stuck going back and forth along already-explored paths indefinitely. So I tried a "thinking" mode, where after each move, I asked it to summarize what it knew about the maze thus far. Then instead of continuing the chat I'd start a new one, providing the LLM with its previous instance's summary. With gpt4, the summaries were very verbose, and each subsequent iteration would append to it and maybe rephrase a bit. It got very slow, and didn't seem to help a whole lot. They were so verbose, it was hard to tell if they accurately represented the maze or not. "On step N, you went forward, and you found walls on two sides but not the front, so I would recommend trying going forward again for your subsequent move". With o3-mini, the summaries were more concise and more mathematical looking, easy to understand. "Current position: (4,3) orientation: E. History: (0,0): walls N,E; open S,W. (0,1): walls ...". But it would get them wrong sometimes, so didn't help in the long run. I think maybe I could improve the results by playing around with the prompts a bit, but I'm moderately convinced at this point that they're not good enough to maintain an internal state nor externalize a scratchpad state that would allow them to solve a maze one step at a time efficiently. Much less, make sense out of a full RPG.