A domain is 'rich', relative to our own intelligence, to the extent that (1) its search space is too large and irregular for all of the best strategies to be searched by our own intelligence, and (2) the known mechanics of the domain do not permit us to easily place absolute bounds against some event occurring or some goal of interest being strategically obtainable.

For the pragmatic implications, see Cognitive uncontainability and Almost all real-world domains are rich.

Work in progress

spectrum: Logical tic-tac-toe, logical chess, logical go, real-world tic-tac-toe, human brain, Internet, real world

since this is about advanced safety, we have to assume that there's a smarter-than-us intelligence that could seek out and exploit the tiniest loophole in our reasoning about how safe we are, and accordingly be safe and conservative in what we think is a 'narrow' domain, and flag every time we rely on the assumption that the agent's strategy space is narrow and therefore we can cognitively contain something smarter than us

talk about that old example of the evolutionary algorithm that used transistors to form a radio to form a clock circuit; gravity means that everything interacts with everything

paradigmatic example: That Alien Message