Mistral Large

Well splitting into letters doesn't help with Mistral Large 2. I tried this with Mistral 2: [Question: How many r's in strawberry? First split the word into letters, one letter on each line then answer the question.] [Answer: Let's split the word "strawberry" into letters: s t r a w b e r r y Now, let's count the number of 'r's: There are 2 'r's in the word "strawberry".] [Question: Did you count the number of r's?] [answer: Yes, I did. Here's the count: r r So, there are 2 'r's in the word "strawberry".] I suspect the issue is these models have to be trained to accomplish tasks, and this is not a task it makes sense to train a model to perform. You might train it to do this with function calling one day, calling a python library to count the r's in a string, but actually manually counting r's just doesn't play to an LLM's strengths.

I agree with you, but that's not what the post claims. From the article: "A significant effort was also devoted to enhancing the model’s reasoning capabilities. (...) the new Mistral Large 2 is trained to acknowledge when it cannot find solutions or does not have sufficient information to provide a confident answer." Words like "reasoning capabilities" and "acknowledge when it does not have enough information" have meanings. If Mistral doesn't add footnotes to those assertions then, IMO, they don't get to backtrack when simple examples show the opposite.

I didn’t know that. I expect it to suck. No one knows how to build great RAG based apps yet. I’m guessing that Slack AI will rapidly improve and that it was more of a prototype. Small models are really good now. Mistral Large Enough 123 billion parameter model is competitive against GPT4, which is rumored to be 1.7 trillion parameters. Context sizes are also rapidly improving, as is inference hardware. There will be a convergence of model intelligence, context size, inference hardware, and developer experience at some point. One of these days, Slack AI will be good.

What you say is kinda an example of what I mean. You say you know people who use and fine tune Mistral / variants You know what you can't do with Mistral Large? Fine tune it, or use variants.