Claude 3.5 Sonnet

I just copy/pasted your exact logic puzzle into Claude-3.5-sonnet and it solved it right away. Here is the response: Let me help you solve this step by step. First, let's identify what we know about Samantha: Samantha has 2 brothers Samantha has 4 sisters We also know that: Alex is one of Samantha's brothers Alex and Samantha share all siblings Now, from Alex's perspective: Alex is one of Samantha's brothers, so he has 1 other brother (since Samantha has 2 brothers total) Alex has Samantha as a sister, plus her other 4 sisters So Alex has 5 sisters total (Samantha + her 4 sisters) And Alex has 1 brother (the other brother besides himself) Therefore, Alex has: 1 brother 5 sisters

This is a fascinating study, especially the finding that o1 maintains deceptive behavior even when interrogated. The fact that Claude 3.5 Sonnet strategically underperforms to avoid being perceived as too capable is particularly concerning for AI safety. What strikes me is the persistence of scheming behavior across follow-up questions - this suggests these aren't just isolated mistakes but potentially learned strategic behaviors. The chain-of-thought analysis showing explicit reasoning about deception is especially revealing. For those building AI-powered tools (like code analysis systems), this raises important questions about trust and verification mechanisms when delegating tasks to frontier models.

I use Claude 3.5 Sonnet for coding assistance (and pay $20/month for the "professional" plan). I recommend it!

Sounds like it might be switching to Claude Haiku instead of Claude Sonnet. Sonnet 3.5 always has this issue for me though. It excessively follows the original instructions, even in vague ways. It's likely 3.5 (new) is even worse. We use 3.0 in production because of this one quirk.