DanielFilan

Comments

Knowledge is not just mutual information

Seems like maybe the solution should perhaps be that you should only take 'the system' to be the 'controllable' physical variables, or those variables that are relevant for 'consequential' behaviour? Hopefully if one can provide good definitions for these, it will provide a foundation for saying what the abstractions should be that let us distinguish between 'high-level' and 'low-level' behaviour.

Challenge: know everything that the best go bot knows about go

Ah, understood. I think this is basically covered by talking about what the go bot knows at various points in time, a la this comment - it seems pretty sensible to me to talk about knowledge as a property of the actual computation rather than the algorithm as a whole. But from your response there it seems that you think that this sense isn't really well-defined.

Challenge: know everything that the best go bot knows about go

Actually, hmm. My thoughts are not really in equilibrium here.

AXRP Episode 7 - Side Effects with Victoria Krakovna

Not sure what the actual sentence you wanted to write was. "are not absolutely necessary" maybe?

You're quite right, let me fix that.

Challenge: know everything that the best go bot knows about go

(Also: such a rewrite would be a combination of 'what I really meant' and 'what the comments made me realize I should have really meant')

Challenge: know everything that the best go bot knows about go

OK, the parenthetical helped me understand where you're coming from. I think a re-write of this post should (in part) make clear that I think a massive heroic effort would be necessary to make this happen, but sometimes massive heroic efforts work, and I have no special private info that makes it seem more plausible than it looks a priori.

Challenge: know everything that the best go bot knows about go

In the parent, is your objection that the trained AlphaZero-like model plausibly knows nothing at all?

Challenge: know everything that the best go bot knows about go

Suppose you have a computer program that gets two neural networks, simulates a game of go between them, determines the winner, and uses the outcome to modify the neural networks. It seems to me that this program has a model of the 'go world', i.e. a simulator, and from that model you can fairly easily extract the rules and winning condition. Do you think that this is a model but not a mental model, or that it's too exact to count as a model, or something else?

Challenge: know everything that the best go bot knows about go

I think there's some communication failure where people are very skeptical of this for reasons that they think are obvious given what they're saying, but which are not obvious to me. Can people tell me which subset of the below claims they agree with, if any? Also if you come up with slight variants that you agree with that would be appreciated.

  1. It is approximately impossible to succeed at this challenge.
  2. It is possible to be confident that advanced AGI systems will not pose an existential threat without being able to succeed at this challenge.
  3. It is not obvious what it means to succeed at this challenge.
  4. It will probably not be obvious what it means to succeed at this challenge at any point in the next 10 years, even if a bunch of people try to work on it.
  5. We do not currently know what it means for a go bot to know something in operational terms.
  6. At no point in the next 10 years could one be confident that one knew everything a go bot knew, because we won't be confident about what it means for a go bot to know something.
  7. You couldn't know everything a go bot knows without essentially being that go bot.

[EDIT: 8. One should not issue a challenge to know everything a go bot knows without having a good definition of what it means for a go bot to know things.]

Load More