'Nearest unblocked strategy' seems like it should be a foreseeable problem of trying to get rid of undesirable AI behaviors by adding specific penalty terms to them, or otherwise trying to exclude one class of observed or foreseen bad behaviors. Namely, if a decision criterion thinks is the best thing to do, and you add a penalty term that you think excludes everything inside the next-best thing to do may be a very similar thing which is the most similar thing to that doesn't trigger
Some very early proposals for AI alignment suggested that AIs be targeted on producing human happiness. Leaving aside various other objections, arguendo, imagine the following series of problems and attempted fixes:
The overall story is one where the AI's preferences on round denoted are observed to arrive at an attainable optimum which the humans see as undesirable. The humans devise a penalty term intended to exclude the undesirable parts of the policy space, and add this to creating a new utility function after which the AI's optimal policy settles into a new state that seems acceptable. However, after the next expansion of the policy space, settles into a new attainable optimum which is very similar to and makes the minimum adjustment necessary to evade the boundaries of the penalty term requiring a new penalty term to exclude this new misbehavior.
(The end of this story might not kill you if the AI had enough successful, advanced-safe corrigibility features that the AI would indefinitely go on checking novel policies and novel goal instantiations with the users, not strategically hiding its disalignment from the programmers, not deceiving the programmers, letting the programmers edit its utility function, not doing anything disastrous before the utility function had been edited, etcetera. But you wouldn't want to rely on this. You would not want in the first place to operate on the paradigm of 'maximize happiness, but not via any of these bad methods that we have already excluded'.)
Recurrence of a nearby unblocked strategy is argued to be a foreseeable difficulty given the following preconditions:
• The AI is a consequentialist, or is conducting some other search such that when the search is blocked at the search may happen upon a similar that fits the same criterion that originally promoted E.g. in an agent that selects actions on the basis of their consequences, if an event leads to goal but is blocked, then a similar may also have the property of leading to
• The search is taking place over a rich domain where the space of relevant neighbors around X is too complicated for us to be certain that we have described all the relevant neighbors correctly. If we imagine an agent playing the purely ideal game of logical Tic-Tac-Toe, then if the agent's utility function hates playing in the center of the board, we can be sure (because we can exhaustively consider the space) that there are no Tic-Tac-Toe squares that behave strategically almost like the center but don't meet the exact definition we used of 'center'. In the far more complicated real world, when you eliminate 'administer heroin' you are very likely to find some other chemical or trick that is strategically mostly equivalent to administering heroin. See "Almost all real-world domains are rich".
• From our perspective on value, the AI does not have an absolute identification of value for the domain, due to some combination of "the domain is rich" and "value is complex". Chess is complicated enough that human players can't absolutely identify winning moves, but since a chess program can have an absolute identification of which endstates constitute winning, we don't run into a problem of unending patches in identifying which states of the board are good play. (However, if we consider a very early chess program that (from our perspective) was trying to be a consequentialist but wasn't very good at it, then we can imagine that, if the early chess program consistently threw its queen onto the right edge of the board for strange reasons, forbidding it to move the queen there might well lead it to throw the queen onto the left edge for the same strange reasons.)
Although humans obeying the law make poor analogies for mathematical algorithms, in some cases human economic actors expect not to encounter legal or social penalties for obeying the letter rather than the spirit of the law. In those cases, after a previously high-yield strategy is outlawed or penalized, the result is very often a near-neighboring result that barely evades the letter of the law. This illustrates that the theoretical argument also applies in practice to at least some pseudo-economic agents (humans), as we would expect given the stated preconditions.
To a human, 'poisonous' is one word. In terms of molecular biology, the exact volume of the configuration space of molecules that is 'nonpoisonous' is very complicated. By having a single word/concept for poisonous-vs.-nonpoisonous, we're dimensionally reducing the space of edible substances - taking a very squiggly volume of molecule-space, and mapping it all onto a linear scale from 'nonpoisonous' to 'poisonous'.
There's a sense in which human cognition implicitly performs dimensional reduction on our solution space, especially by simplifying dimensions that are relevant to some component of our values. There may be some psychological sense in which we feel like "do X, only not weird low-value X" ought to be a simple instruction, and an agent that repeatedly produces the next unblocked weird low-value X is being perverse - that the agent, given a few examples of weird low-value Xs labeled as noninstances of the desired concept, ought to be able to just generalize to not produce weird low-value Xs.
In fact, if it were possible to encode all relevant dimensions of human value into the agent then we could just say directly to "do X, but not low-value X". By the definition of full_coverage, the agent's concept for 'low-value' includes everything that is actually of low value, so this one instruction would blanket all the undesirable strategies we want to avoid.
Conversely, the truth of the complexity of value thesis would imply that the simple word 'low-value' is dimensionally reducing a space of tremendous algorithmic complexity. Thus the effort required to actually convey the relevant dos and don'ts of "X, only not weird low-value X" would be high, and a human-generated set of supervised examples labeled 'not the kind of X we mean' would be unlikely to cover and stabilize all the dimensions of the underlying space of possibilities. Since the weird low-value X cannot be eliminated in one instruction or several patches or a human-generated set of supervised examples, the nearest unblocked strategy problem will recur incrementally each time a patch is attempted and then the policy space is widened again.
Nearest unblocked strategy being a foreseeable difficulty is a major contributor to worrying that short-term incentives in AI development, to get today's system working today, or to have today's system not exhibiting any immediately visible problems today, will not lead to advanced agents which are safe after undergoing significant gains in capability.
More generally, nearest unblocked strategy is a foreseeable reason why saying "Well just exclude X" or "Just write the code to not X" or "Add a penalty term for X" doesn't solve most of the issues that crop up in AI alignment.
Even more generally, this suggests that we want AIs to operate inside a space of conservative categories containing actively whitelisted strategies and goal instantiations, rather than having the AI operate inside a (constantly expanding) space of all conceivable policies minus a set of blacklisted categories.