A few days ago I tried AI again, after several months of not using it at all. We're dealing with some housing-related matters, and I asked Claude, ChatGPT and Gemini for advice what to do in a certain situation. (I won't give the details here, because they're personal and not that important for my point.) All three models independently gave the same advice, which sounded really convincing to me. But we have some friends with experience in such matters, who said that following this advice would be catastrophic and we should do the opposite. Eventually we followed the advice of our friends, and it worked out very well, better than anything I'd hoped from the AIs' advice.

My takeaway is that I'm right to avoid LLMs for "fuzzy" real life tasks. They can be totally clueless about something, but answer so convincingly that you can't tell. Their real competence is in "crisp" tasks like math and software, where RLVR is available. I suspect that specialized models for such tasks could be much smaller and better than today's LLMs.