Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence.
In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons.[1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others.
My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional "component" to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this "adaptation") and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence.
To be clear, this doesn't mean I think extremely powerful AI isn't near: I expect full AI R&D automation[2] in <5.5 years[3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at everything including fluid intelligence[4].
Also, I think that at the point when AIs fully match humans in fully general-purpose fluid intelligence and learning, they will likely be extremely superhuman in most domains (because they'll have other advantages and be able to train for much longer). And most types of R&D will get fully or virtually fully automated earlier.
The reason I found this situation confusing for awhile (and am still somewhat confused) is that it seems like there is a lot of crystallized intelligence available from the Human Corpus about how to be smart. AIs seem to know how to apply individual pieces of it, but not go out of their way to apply it all in a consistent/coherent manner.
(Things like "make a list of the constraints", "keep track of evidence, when you generate new hypotheses, review whether existing evidence rules it out".)
I'm not surprised that AIs didn't early on figure out how to apply various cognitive concepts in a coherent/consistent way. But, I'm surprised it hasn't fallen out more by now from various RL training. My current sense is "well it just keeps being easier to develop/reinforce medium-shallow domain specific cognitive tools, than to generalize and apply deeper cognitive concepts."
The reason, contra @TsviBT , I keep feeling like we're in short timeline world is "idk man, all the pieces seem to be fucking there already, surely a good enough RL training regimen should be able to surface and integrate them even if the models are still fundamental dumb in some fluid-intelligency-y way." But, they do seem to keep being kinda dumb in ways that surprise me.
I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.