Epistemic status: trying to feel out the shape of a concept and give it an appropriate name. Trying to make explicit some things that I think exist implicitly in many people's minds. This post makes truth claims, but its main goal is to not to convince you that they are true.

Here are some things I would expect any AGI to be able to do:

Operate over long intervals of time relative to its sensory bandwidth (e.g. months or years of ~30 fps visual input).
Remember specific sensory experiences from long ago that are relevant to what's happening to it now. (E.g. remember things it saw months or years ago.)
Retain or forget information and skills over long time scales, in a way that serves its goals. E.g. if it does forget some things, these should be things that are unusually unlikely to come in handy later.
Re-evaluate experiences that happened a long time ago (e.g. years ago) in light of newer evidence (observed in e.g. the last hour), and update its beliefs appropriately.
Continually adjust its world model in light of new information during operation.
- E.g. upon learning that a particular war has ended, it should act as though the war is not happening, and do so in all contexts/modalities.
- As with humans, this adaptation may take a nonzero amount of time, during which it might "forget" the new fact sometimes. However, adaptation should be rapid enough that it does not impede acting prudently on the most relevant implications of the new information.
- This may require regular "downtime" to run offline training/finetuning (humans have to sleep, after all). But if so, it should require less than 1 second of downtime per second of uptime, ideally much less.
Perform adjustments to itself of the kind described above in a "stable" manner, with a negligibly low rate of large regressions in its knowledge or capabilities.
- E.g. if it is updating itself by gradient descent, it should do so in a way that avoids (or renders harmless) the gradient spikes and other instabilities that cause frequent quality regression in the middle of training for existing models, especially large ones.
Keep track of the broader world context while performing a given task.
- E.g. an AGI playing a video game should not forget about its situation and goals in the world outside the game.
- It might "get distracted" by the game (as humans do), but it should have some mechanism for s

...

Autonomy and Choice