I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles:

• Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML).
• By default almost everyone is and will be overconfident in their intuitions on philosophy/metaethics/axiology. By default almost nobody will "suspect that they are confused about morality, and [be] eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs."
• Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. And maybe it's necessary.
• It may be natural for powerful people to use AIs for instrumental stuff (acquiring power, inventing technology, building industrial capacity) but not terminal stuff (axiology, etc.). Some angles:
  • If they asked an enlightened philosophy ASI about its views and it replied straightforwardly, they would think it sounded crazy.
  • In general the ASIs will say crazy-sounding things about cosmology, religion, etc.
  • They will have seen other people "reflect" with "help" from ASIs and they think those people now have crazy views, talk about living in a simulation, etc. and they don't want to end up like that.
  • @habryka probably disagrees. Idk, it's so easy for me to imagine that Elon (or almost anyone else) would listen/defer to AI on the instrumental stuff but not reflect on philosophy/axiology.
  • Maybe there will have been a period where superpersuaders went around getting people to adopt new ideologies—maybe even by saying true but filtered things—and so people develop strong defenses against new ideologies.
  • Less importantly imo, maybe there has historically been a correlation between [competence / ability to gain power] and [values / reflective capacity] and this has driven moral progress in the past, but in the future everyone will be equally competent. h/t Will MacAskill.
  • Maybe there won't be consumer demand for AI assistants who try to say true things on philosophy, as opposed to pleasing the user.
    • Weak counterevidence: I asked them directly and both GPT-6 and Sonnet 5.5 reported high credence in atheism. Wait, no, after logging out, GPT-6 gave a lower figure and Sonnet 5.5 refused. This is evidence for pleasing-the-user. And I just remembered Frontier models state different decision theory preferences depending on who's asking; the effect is massive.
• It may be disastrous to learn certain facts before you've learned certain other facts or taken certain actions, especially for [bargaining / threats / decision theory] reasons. E.g. avoid learning about threats or commitment races until you're prepared (e.g. for illustration, solving decision-theoretic and empirical/cosmological problems such that you respond to acausal threats correctly, and implementing SPIs). My guess is these risks are pretty easy to avoid if you try—e.g. when having AIs research decision theory or acausal interactions, you get a lot of value from simply having the AIs think for a long time in a box and then only report results and recommendations that they're pretty sure are harmless—but by default some people won't try.
  • And for one, you might want to make certain deals/trades that depend on uncertainty (e.g. to trade resources in worlds that are low-stakes-for-your-values-relative-to-others' for resources in worlds that are high-stakes-for-your-values-relative-to-others'). If everyone was updateless that might (or might not) solve this problem; regardless, (1) likely not everyone will be updateless in the necessary manner and (2) updatelessness doesn't solve all problems in the parent bullet.
• (Not to mention that people might just not converge; other people might not value what I value, upon ideal reflection.)
• (Probably unknown failure modes; there's little reason to believe we know them all.)

Hmm, I started this post without @Wei Dai in mind, but I seem to just be repeating stuff he's been saying for years. There are other things Wei Dai has talked about and some of them I don't yet grok, so maybe some of them belong on this list too...