Gemini 3.5 Flash

One time I told it “we are doing science” and I had DNA emoji everywhere and it so over enthusiastically embraced the science theme I was genuinely laughing. It finished one task with a flourish of several dna emoji and proclaimed: The Science is COMPLETE. I died. It really is a lot some of the time. And it’s chain of thought is hilarious a lot of the time.

Also, Gemini 3.5 Flash seems to be hallucinating, becoming more confident because it doesn't use its web search tool. And when I point it out, it takes it at face value without checking, not using its web search tool.

This understates the cost increase. 3.5 Flash also uses more tokens. artificialanalysis.ai shows these difference to run the whole eval, which I think is more realistic pricing: Gemini 2.5 flash (27 score): $172 (1.0x) Gemini 2.5 pro (35 score): $649 (3.8x) Gemini 3.0 Flash (46 score): $278 (1.6x) Gemini 3.5 Flash (55 score): $1,552 (9.0x or 2.4x compared to 2.5 pro) This is a massive price increase... 5.6x compared to Gemini 3.0 Flash

I'm seeing this too. I have a SQL agent and my tests with 3.5 are resulting in hitting query budget limits that have never been hit before. On average, to answer the same question, 3.5 is spending 10x more on SQL queries vs gemini-3-flash-preview. The query patterns can be extremely degenerate too. E.g. the agent will hit the semantic layer tool to pull the schema, then run `SELECT * FROM table LIMIT 1`, which hits the query budget limit and fails. I've only really been looking this morning, so I need to do a full eval, but the initial results match what your benchmark shows. --- Side note: your benchmark has an issue. On Q1 medium the model returned gross margin of 0.127 instead of 12.7 (%), and the benchmark failed it. The failures on Q9 and Q21 are the same (I didn't check other questions). Nowhere in the prompt did you specify you wanted the values converted to percentage points and rounded. If you asked me to write that SQL with that prompt, unless you were throwing it directly into a visualization I would format it the same way gemini-flash did. If I were pulling into a spreadsheet or vis tool this format is preferable because it's easier to format in a client application. The other failures like Q21 incorrectly averaging the list price are correct failures.