cousin_it — AI Alignment Forum

AI ALIGNMENT FORUM
AF

Replying toProblems I've Tried to Legibilize

I think AI offers a chance of getting huge power over others, so it would create competitive pressure in any case. In case of a market economy it's market pressure, but in case of countries it would be a military arms race instead. And even if the labs didn't get any investors and raced secretly, I think they'd still feel under a lot of pressure. The chance of getting huge power is what creates the problem, that's why I think spreading out power is a good idea. There would still be competition of course, but it would be normal economic levels of competition, and people would have some room to do the right things.

Replying toProblems I've Tried to Legibilize

cousin_it3mo*

Problems I've Tried to Legibilize

I think on the level of individual people, there's a mix of moral and self-interested actions. People sometimes choose to do the right thing (even if the right thing is as complicated as taking metaethics and metaphilosophy into account), or can be convinced to do so. But with corporations it's another matter: they choose the profit motive pretty much every time.

Making an AI lab do the right thing is much harder than making its leader concerned. A lab leader who's concerned enough to slow down will be pressured by investors to speed back up, or get replaced, or get outcompeted. Really you need to convince the whole lab and its investors. And... (read more)

Replying toProblems I've Tried to Legibilize

cousin_it3mo*

Problems I've Tried to Legibilize

I'm pretty slow to realize these things, and I think other people are also slow, so the window is already almost closed. But in any case, my current thinking is that we need to start pushing on the big actors from outside, try to reduce their power. Trying to make them see the light is no longer enough.

What it means in practical terms: - Make it clear that we frown on people who choose to work for AI labs, even on alignment. This social pressure (on LW and related forums maybe) might already do some good. - Make it clear that we're allied with the relatively poor majority of people outside the... (read more)

Replying toProblems I've Tried to Legibilize

cousin_it3mo

Problems I've Tried to Legibilize

I'm worried about the approach of "making decisionmakers realize stuff". In the past couple years I've switched to a more conflict-theoretic view: the main problem to me is that the people building AI don't want to build aligned AI. Even if we solved metaethics and metaphilosophy tomorrow, and gave them the solution on a plate, they wouldn't take it.

This is maybe easiest to see by looking at present harms. An actually aligned AI would politely decline to do such things as putting lots of people out of jobs or filling the internet with slop. So companies making AI for the market have to make it misaligned in at least these ways, otherwise... (read more)

Replying toGeometric UDT

cousin_it3mo*

Geometric UDT

Sure, but if we put a third "if" on top (namely, "it's a representation of our credences, but also both hypotheses are nosy neighbors that care about either world equally"), doesn't that undo the second "if" and bring us back to the first?

Replying toGeometric UDT

cousin_it3mo

Geometric UDT

I still don't completely understand what your assumptions are supposed to model, but if we take them on face value, then it seems to me that always making rainbows is the right answer. After all, if both hypotheses are "nosy neighbors" that don't care which universe we end up in, there's no point figuring out which universe we end up in: we should just make rainbows because it's cheaper. No?

Replying toPlans A, B, C, and D for misalignment risk

cousin_it4mo

Plans A, B, C, and D for misalignment risk

Yeah, that partly makes sense to me. I guess my intuition is like, if 95% of the company is focused on racing as hard as possible (and using AI leverage for that too, AI coming up with new unsafe tricks and all that), then the 5% who care about safety probably won't have that much impact.

Replying toPlans A, B, C, and D for misalignment risk

cousin_it4mo

Plans A, B, C, and D for misalignment risk

The OP says takeover risk is 45% under plan D and 75% under plan E. We're supposed to gain an extra 30% of safety from this feeble "build something by next week with 1% of compute"? Not happening.

My point is that if the "ten people on the inside" obey their managers, plan D will have a tiny effect at best. And if we instead postulate that they won't obey their managers, then there are no such "ten people on the inside" in the first place. So we should already behave as if we're in world E.

Replying toPlans A, B, C, and D for misalignment risk

cousin_it4mo

Plans A, B, C, and D for misalignment risk

Can you maybe describe in more detail how you imagine it? What specifically do the "ten people on the inside" do, if company leadership disagrees with them about safety?

Replying toPlans A, B, C, and D for misalignment risk

cousin_it4mo

Plans A, B, C, and D for misalignment risk

Do you know any people working at frontier labs who would be willing to do the kind of thing you describe in plan D, some kind of covert alignment against the wishes of the larger company? Who would physically press keys on their terminal to do it, as opposed to quitting or trying to sway the company? Not asking to name names, just my hunch is that there are very few such people now, maybe none at all. And if that's the case, we're in E world already.

If the housing crisis is caused by low-density rich neighborhoods blocking redevelopment of themselves (as seems the consensus on the internet now), could it be solved by developers buying out an entire neighborhood or even town in one swoop? It'd require a ton of money, but redevelopment would bring even more money, so it could be win-win for everyone. Does it not happen only due to coordination difficulties?

I just thought of a simple way to explain tensors. Imagine a linear function that accepts two numbers and returns a number, let's call it f(x,y). Except there are two ways to imagine it:

Linear in both arguments combined: f(1,2)+f(1,3)=f(2,5). Every such function has the form f(x,y)=ax+by for some a and b, so the space of such functions is 2-dimensional. We say that the Cartesian product of R^1 and R^1 is R^2, because 1+1=2.
Linear in each argument when the other is fixed: f(1,2)+f(1,3)=f(1,5). Every such function has the form f(x,y)=axy for some a, so the space of such functions is 1-dimensional. We say that the tensor product of R^1 and R^1 is R^1, because 1*1=1.

In this case the tensor product is lower dimensional than the Cartesian product. But if we take say R^3 and R^3, then the Cartesian product will be R^6 and the tensor product will be R^9, because it will have separate coefficients a_(ij)*x_i*y_j.

Edit: no point asking this question here.

(12/?)

The uncertainty principle, or something like it.

When you measure a qubit cos(φ)|0> + sin(φ)|1>, the result has variance (1-cos(4φ))/4. (I'm skipping over the trig calculations here and below.) If you have a choice between either measuring the qubit, or rotating it by an angle ψ and then measuring it, then the sum of variances of the two operations is at least (1-|cos(2ψ)|)/2. In particular, if ψ=π/4, the sum of variances is at least 1/2.

So no matter what state a qubit is in, and how precisely you know its state, there are two possible measurements such that the sum of variances of their results is at least 1/2. The reason is that the

... (read more)

(11/?)

Superdense coding.

Alice is told two classical bits of information and sends a qubit to Bob, who can then recover the two classical bits. Again, it relies on Alice and Bob sharing a prepared pair beforehand. It's the opposite of quantum teleportation, where Alice sends two classical bits and Bob can recover a qubit.

First let's talk about bases. This is the usual basis: |00>, |01>, |10>, |11>. This is the Bell basis: (|00> + |11>)/√2, (|00> - |11>)/√2, (|10> + |01>)/√2, (|10> - |01>)/√2. Check for yourself that each two vectors are orthogonal to each other. To "measure a state in a different basis" means to apply a rotation from one basis into

... (read more)

(10/?)

Quantum teleportation.

It's not a very good name, maybe we should call it "sending qubits over a classical channel using a quantum one-time pad". The idea is that Alice and Bob are far apart from each other and have a prepared pair of qubits, and Alice also has some qubit in unknown state. She can measure it in a clever way and send some classical bits to Bob, which allows him to recreate the unknown qubit. Both qubits of the prepared pair are spent in the process.

Let's say the unknown qubit is a|0>+b|1>, and the prepared pair is (|00> + |11>)/√2. So the whole state is (a|0> + b|1>) ⊗ (|00> + |11>)/√2

... (read more)

(9/?)

Black box functions and Grover's algorithm.

Say we want our quantum computer to make calls to a black box function f(x) from bit strings to bits. There's a couple ways to formalize that. Phase oracle: |x> becomes -|x> if f(x), otherwise |x>. Binary oracle: |x> ⊗ |b> becomes |x> ⊗ |f(x) xor b>. These definitions only say what happens to basis vectors, and are extended to other vectors by linearity.

We can convert from a binary oracle to a phase oracle by using an extra qubit. Start with a state |x> ⊗ (|0> - |1>)/√2 and apply a binary oracle, we end up with (1/√2) ⋅ (|x1> - |x0> if f(x), otherwise |x0> -

... (read more)

(8/?)

Decoherence.

Let's say you have a state ( |0> + |1> ) / √2 and want to rotate it by 45 degrees and measure it, expecting to get "1" with certainty. But a bug in your equipment affects some other unrelated qubit, which was |0> before and becomes |0 xor your qubit> afterward. So now you have ( |00> + |11> ) / √2. Now rotating your qubit leads to ( |00> + |10> + |11> - |01> ) / 2, and measuring gives "0" or "1" with probability 1/2 each. Some unrelated qubit being xor'ed with yours caused your qubit to become mixed.

In theory, this is recoverable: find the stray qubit and xor it with yours again. But what if you accidentally measured it? Then you'd have to xor your memory with the qubit, which sounds impractical. Worse, what if you measured it and learned about the problem only in case of "1"? This is why quantum computation is so fragile.

(7/?)

No-cloning and no-deleting theorems.

As a warmup, imagine the state ( |00> + |01> ) / √2 gets changed by a quantum operation into ( |00> + |11> ) / √2. The operation is "flip the first bit if the second is 1". After that, if we have only the first qubit to play with, it's statistically indistinguishable from a single qubit in the mixed state "|0> or |1> with probability 1/2 each". Parts of a pure state, considered in isolation, are mixed states.

Mixed states involve probabilities, which makes us think of measurement. And indeed, the operation above can be seen as either "use the first qubit to measure the second" or "entangle

... (read more)

(6/?)

The last comment didn't have enough rules for calculating with mixed states. Here's some more:

To measure a mixed state, the probabilities of all bit strings are on the main diagonal of the density matrix. For example, the pure state |0> has density matrix ((1,0),(0,0)), so measuring it leads to 0 with certainty. The mixed state “|0> or |1> with probability 1⁄2 each” has density matrix ((1/2,0),(0,1/2)), so the probability of each outcome is 1/2. And the pure state ( |0> + |1> ) / √2 has density matrix ((1/2,1/2),(1/2,1/2)), so the probability of each outcome is also 1/2.
To apply a reversible logical operation to a mixed state, recall that each possible bit

... (read more)

cousin_it's Shortform

cousin_it

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Announcement: AI alignment prize round 4 winners

cousin_it

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the fourth round of the AI Alignment Prize, funded by Paul Christiano. From July 15 to December 31, 2018 we received 10 entries, and are awarding four prizes for a total of $20,000.

The winners

We are awarding two first prizes of $7,500 each. One of them goes to Alexander Turner for Penalizing Impact via Attainable Utility Preservation; the other goes to Abram Demski and Scott Garrabrant for the Embedded Agency sequence.

We are also awarding two second prizes of $2,500 each: to Ryan Carey for Addressing three problems with counterfactual corrigibility, and to Wei Dai for Three AI Safety Related Ideas

cousin_it

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the third round of the AI Alignment Prize, funded by Paul Christiano. From April 15 to June 30 we received entries from 12 participants, and are awarding $10,000 to two winners.

We are also announcing the fourth round of the prize, which will run until December 31 of this year under slightly different rules. More details below.

The winners

First prize of $7,500 goes to Vanessa Kosoy for The Learning-Theoretic AI Alignment Research Agenda. We feel this is much more accessible than previous writing on this topic, and gives a lot of promising ideas for future research. Most importantly, it explains why... (read 172 more words →)

UDT can learn anthropic probabilities

cousin_it

Today I realized that UDT doesn't have to clash with "objective" views of anthropic probability, like SSA or SIA: instead it can, in some sense, learn which of these views is true!

The argument goes like this. First I'll describe a game where SSA and SIA lead to different decisions. Unlike Sleeping Beauty, my game doesn't involve memory loss, so someone can play it repeatedly and keep memories of previous plays. Then we'll figure out what UDT would do in that game, if it valued each copy's welfare using the average of SSA and SIA. It turns out that different instances of UDT existing at once will act differently: the instances whose memories... (read 754 more words →)

UDT as a Nash Equilibrium

cousin_it

I realized today that UDT doesn't really need the assumption that other players use UDT. In any game where all players have the same utility function, "everyone using UDT" is a Nash equilibrium that gives everyone their highest possible expected utility. So you can just use it unilaterally.

That covers such cases as Absent-Minded Driver, Psy-Kosh's problem, Wei's coordination problem. Notably it doesn't cover the Prisoner's Dilemma, because the players assign different utilities to the same outcome.

(Also this shows how Von Neumann-Morgenstern expected utility maximization is basically a restriction of UDT to single player games with perfect recall. For imperfect recall (AMD) or multiple players (Psy-Kosh) you need the full version.)

You can actually push game theory a bit further, and allow games where players have different utility functions, as long as some subset of players can jointly enforce their highest possible expected utilities. UDT doesn't support that kind of decision-making, but really it should be a no-brainer too...?

Beware of black boxes in AI alignment research

cousin_it

Over the course of the AI Alignment Prize I've sent out lots of feedback emails. Some of the threads were really exciting and taught me a lot. But mostly it was me saying pretty much the same thing over and over with small variations. I've gotten pretty good at saying that thing, so it makes sense to post it here.

Working on AI alignment is like building a bridge across a river. Our building blocks are ideas like mathematics, physics or computer science. We understand how they work and how they can be used to build things.

Meanwhile on the far side of the river, we can glimpse other building blocks that we imagine... (read more)

Announcing the AI Alignment Prize

cousin_it

Using modal fixed points to formalize logical causality

cousin_it

This post is a simplified introduction to existing ideas by Eliezer Yudkowsky, Wei Dai, Vladimir Nesov and myself. For those who already understand them, this post probably won't contain anything new. As always, I take no personal credit for the ideas, only for the specific mathematical model.

People usually think about decision-making in terms of causality, where an agent's action causes an outcome to happen. In this post I will outline a different idea of "causality", which can work even if regular causality isn't available. For example, in the world of natural numbers it doesn't make sense to say that the sentence $1 + 1 = 2$ somehow "causes" the sentence $2 + 2 = 4$ to be true. Yet

... (read 960 more words →)

A cheating approach to the tiling agents problem

cousin_it

(This post resulted from a conversation with Wei Dai.)

Formalizing the tiling agents problem is very delicate. In this post I'll show a toy problem and a solution to it, which arguably meets all the desiderata, but only by cheating in a new and unusual way.

Here's a summary of the toy problem: we ask an agent to solve a difficult math question and also design a successor agent. Then the successor solves another math question and designs its own successor, and so on. The questions get harder each time, so they can't all be solved in advance, and each of them requires believing in Peano arithmetic (PA). This goes on for a fixed

... (read 883 more words →)

Loebian cooperation in the tiling agents problem

cousin_it

The tiling agents problem is about formalizing how AIs can create successor AIs that are at least as smart. Here's a toy model I came up with, which is similar to Benya's old model but simpler. A computer program X is asked one of two questions:

Would you like some chocolate?
Here's the source code of another program Y. Do you accept it as your successor?

Program X will be called a "winner" if is says "yes" to chocolate, every program Y accepted as successor by X also says "yes" to chocolate, every program Z accepted as successor by Y also says "yes" to chocolate, and so on. Otherwise X will be called a "loser".

Let's

... (read 366 more words →)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Vladimir Slepnev

Announcement: AI alignment prize round 3 winners and next round

Announcement: AI alignment prize round 4 winners

UDT can learn anthropic probabilities

Beware of black boxes in AI alignment research

Vladimir Slepnev

cousin_it's Shortform

Announcement: AI alignment prize round 4 winners

Announcement: AI alignment prize round 3 winners and next round

UDT can learn anthropic probabilities

UDT as a Nash Equilibrium

Beware of black boxes in AI alignment research

Announcing the AI Alignment Prize

Vladimir Slepnev

Announcement: AI alignment prize round 3 winners and next round

Announcement: AI alignment prize round 4 winners

UDT can learn anthropic probabilities

Beware of black boxes in AI alignment research

Vladimir Slepnev

cousin_it's Shortform

Announcement: AI alignment prize round 4 winners

Announcement: AI alignment prize round 3 winners and next round

UDT can learn anthropic probabilities

UDT as a Nash Equilibrium

Beware of black boxes in AI alignment research

Announcing the AI Alignment Prize

The winners

The winners