2018 Nominations — AI Alignment Forum

Coherence arguments do not entail goal-directed behavior

One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.

We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent, since if we could notice these failures, so could the agent. So we should expect that powerful agents appear coherent to us. (Note that it is possible that the...

(Continue Reading - 1974 more words)

1habryka2d

I am sorry, I am still totally failing to see the relevance of any of this, or how it's responding to what I am saying. Like, there are like 15 things you seem to be assuming about what I am saying that I am not saying. Like, what is that last paragraph about? Where am I saying that agents always want to be arbitrarily coherent under arbitrary definitions of the term? Are you just tone-policing the term of the word "coherent"? What does that have to do with anything I say? Why is it important that "coherence" has exactly the right valence? Yes, sometimes you don't want to become more coherent, but competitiveness considerations push you in that direction. What does this have to do with e.g. ML models obviously being trained in a competitive context that encourages coherence? [...] It would be an even more confusing way to define the word "coherent" if you insist on it only ever being applied to positive things. In that case "coherent" would have to become fully synonymous with good. Like, sure, you can have whatever connotations you want with the word "coherent". I don't really understand what this has to do with how people de-facto use the word "coherence" and "coherence arguments" in practice. Basically every positively-valenced word will of course have instances where more of it is bad. Imagine someone being like "Hey man, you seem to use the word 'principled' in a way that is super weird. Like, here is a situation where someone would regret being more 'principled', but the word has positive valence, so it's very confusing to use such a confusing word like 'principled'." No, of course 'principled' and 'coherent' and basically every other word points to a bunch of structural properties in the world that of course sometimes are good and sometimes are bad. ---------------------------------------- And given that you keep asking questions that indicate that you aren't reading what I am writing, I am bowing out. I shall write my response one more time in all ca

2habryka2d

I honestly don't know what you are responding to here. What are you disagreeing with? How is this responding to anything I said? I have read this comment 10+ times and still can't figure out what you are saying or how this is a response to any of what I am saying. I feel like you are just repeatedly again and again and again trying to put words in my mouth that I am trying to make some kind of highly specific argument that I am not making and then trying to call that "the coherence argument" or something. [...] I don't understand. This is not a proof, this just an assertion that says "obviously". The standard proofs of this would route through arguments that require VNM rationality already, so this just totally begging the question. Also why on earth would the answer to this question be obvious? It is extremely non-obvious (and in many scenarios not even true). In-practice, in real-life, it is extremely common that directly pursuing an objective is less effective than pursuing nearby objectives for many different reasons. I also still don't understand how it is relevant at all. I said that the arguments for why markets select for coherence route through dutch books. Then you "proved" a completely different thing which is that "the market selects for agents that 'optimize' for eventual money". The market selects for many things. Coherence is one of them. Optimizing for money is a stronger thing it also selects for. I don't understand how this is responding at all to anything I said. Your proof doesn't say anything about coherence. There is a Lemma you could try to proof that says "well, clearly markets select for optimizing for money, and optimizing for something must clearly mean not being exploitable in being drained off the thing you are optimizing for, so markets select for unexploitability" and then, what a surprise, you would probably use dutch book arguments to show that. ---------------------------------------- If you are incoherent, the speed at whic

2Steven Byrnes2d

Sorry if I missed the mark on where you were coming from. We can drop this thread, I think the other conversation branch is more promising. (I went back and edited in the proof that I had described as “obvious”, for completeness, in case you still care.)

habryka2d*10

The proof you added still doesn't do anything. Like, at a mechanistic level in order to even define what it means for an agent to take actions that maximize expected eventual money, or anything else, you would need to have assumed non-dutch-bookness, because otherwise you don't get to create a utility function to maximize over.

We don't have a canonical definition of "optimization" for non-coherent agents.

if Agent A follows course of action CA, and Agent B follows course of action CB, and Agent A winds up with more eventual money than Agent B, then that’s

... (read more)

Specification gaming examples in AI

Vika

[Cross-posted from personal blog]

Various examples (and lists of examples) of unintended behaviors in AI systems have appeared in recent years. One interesting type of unintended behavior is finding a way to game the specified objective: generating a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent. This occurs when the objective is poorly specified, and includes reinforcement learning agents hacking the reward function, evolutionary algorithms gaming the fitness function, etc. While ‘specification gaming’ is a somewhat vague category, it is particularly referring to behaviors that are clearly hacks, not just suboptimal solutions.

Since these examples are currently scattered across several lists, I have put together a master list of examples collected from the various existing sources. This list is intended to be comprehensive and up-to-date, and serve as a resource for AI safety research and discussion. If you know of any interesting examples of specification gaming that are missing from the list, please submit them through this form.

Thanks to Gwern Branwen, Catherine Olsson, Alex Irpan, and others for collecting and contributing examples!

Nominated Posts for the 2018 Review

Nominated Posts for the 2018 Review

2018 Review Discussion