AI Alignment Forum

3Vladimir_Nesov12h

The idea with the substrate/civilization split is not that the substrate owns the civilization, but that the civilization is the updateless core for the substrate, it defines the policies that the substrate follows (like in a UDT agent), without knowing some of the things the substrate gets to observe in the physical world. The civilization governance on the inside could take any form (doesn't need to be a singleton), and the substrate governance on the outside doesn't carry the historically familiar menacing connotations even if it runs the civilization in a perfect sandbox, because one way or another alignment is going to be settled at that point (which was never historically the case, and so direct control was always relevant). [...] Right, that's a thing for the outer governance to worry about (like that arcane material manufacturing process for the industrial tool), but not necessarily for the civilization.

3jessicata11h

Ok, to make my opinion on this clearer: (a) We don't really know much about what the eventual state looks like, whether it is a singleton, how locked down, how many have access to quantum ops. (b) My personal guess for (a) is that "stochastic Turing machines + Bell nonlocality" is a more relevant model than "deterministic Turing machines", though both are relevant. (c) How do we figure out about (a)? It seems like we need some theory of agent foundations to think about this well. That theory shouldn't start out with strong assumptions like "only deterministic Turing machines are relevant" or "everyone has access to a quantum RNG". But we should also use our personal guesses on this. So I'm thinking about agent foundations with stochastic TMs partially because of my (b) opinion. (d) Part of reasoning about (a) also involves reasoning about intermediate states. These intermediate states can be "messy" and therefore forecasting them involves non-agent-foundations reasoning. But agent foundations is still relevant here, since some intermediate states are smarter-than-human, "economically efficient", etc. (e) I think stochastic TMs are more important for (d) than deterministic TMs, as a model. Sure, both will have some relevance, but quantum RNGs are currently easily available, will become more available in the near future, and have cryptographic applications. Moreover, many RL algorithms, and decision theory (like CDT self-ratification, mixed Nash equilibrium, absent minded driver) assumes access to RNGs, so quantum RNGs are relevant as an idealized case. (f) Overall these considerations make me think stochastic TMs are a better model for agent foundations than deterministic TMs, but in theory both are relevant. My opinion here seems to significantly disagree with yours. (g) I think I have significantly more "dispersion" in my beliefs about the eventual state (a) than you do, such that I don't find the specific scenario you're talking about where agents are locke

4Vladimir_Nesov11h

My stance is "unclear", this subthread is about why it's unclear, why a weird possibility of determinism and a lack of access to quantum observations/randomness/computation could make sense. It's not load bearing for the post or how I think about this; even with quantum observations you still end up with intentionally designed computations in a central sense, rather than mere weather and stars. The backups motivation doesn't imply external observations shouldn't happen at all, only that they should be sufficiently old to already be in the distributed backups by the time they get to be observed on the inside. It's a values question whether creating a vulnerable branch determined by non-backed-up observations is a good thing.

jessicata11h30

Ah, ok. I agree on "unclear" so we disagree less than I thought. I could see an analogous case of: "Maybe in the future, there will be many low-intelligence agents, because there will be some governance system that forces all agents except one to be low-intelligence, for security reasons. We should be extensively studying low-intelligence agents in agent foundations for this reason." And some but not all reasons why this is not a compelling argument for studying low-intelligence agents (in agent foundations) translate to studying "agents with no quantum tech".

Notes on “We’re in Triage Mode for AI Safety” by Miles Brundage

Chris_Leong

13h

This is a linkpost for https://reflectionsonsafety.substack.com/p/notes-on-were-in-triage-mode-for

This reading was previously included Day 2 of the BlueDot Governance course. In the newest version of the BlueDot Governance course, this reading is now optional. I was sad to see this reading removed because I think it’s a great provocation. However, since it’s no longer part of the core curriculum content, I’m publishing my notes separately from the rest of my day 2 notes.

A short extract:

Miles argues that: “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition...

(Continue Reading - 1175 more words)

Monitorability evaluations and policy proposals

evhub

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.

First, a categorization. I currently think about four different types of monitorability:

Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber

...

(Continue Reading - 1281 more words)

1Alek Westover1d

Hi Evan, Thanks for your thoughts! I agree that labs should have red lines like “[models can’t] reliably evade detection when attempting dangerous sabotage actions”. (In fact, “reliably evades detection” is too high a bar IMO: it’s already bad if the model can evade detection 10% of the time.) I also agree that it would be concerning if an architectural change made models much less monitorable than the current “ECI vs monitorability trend” predicts (assuming such a trend exists). However, I worry that these measures won’t rule out the architectural changes I’m concerned about, and I still think that avoiding models with high opaque serial depth is a better approach. Below, I spell out where I disagree with you and why I’m concerned about your proposal. 1. It’s not clear to me that there is a nice continuous notion of monitorability that we can track. * Your proposal mentions an “ECI vs monitorability” trend. I’m skeptical that there is a clean trend here; I worry any trend would be very sensitive to which monitorability evaluations you include and which elicitation methods you use. Monitorability evals are very ad hoc, there isn’t a unified framework where we can easily scale the difficulty of the evasion task. For this to be a binding constraint now, we’d need the monitorability score to increase smoothly with ECI, but I’m not sure how we’ll make sure this happens. 2. We don’t have a ready to go open-source suite of monitorability evaluations that could be used in the proposal you’re mentioning, and developing this might be very labor intensive. * For your proposal to apply the same standard to all AI companies (which is a desirable property, because we want to coordinate around monitorability), we’d need to have a high quality open source suite of monitorability evals that make sense to run on models of hugely varying ECIs, and it would need to be robust to future models with larger ECIs. This seems hard to create. It’s also hard to open source fo

Lukas Finnveden1d32

Alek: What do you think about a proposal using "ECI vs. no-CoT capabilities" instead of "ECI vs. monitorability"? It seems to me more likely that there's a nice continuous notion there which isn't too difficult to get a dataset for, ameliorating 1, 2, and 7, and I think problem 3-6 straightforwardly don't apply.

This could potentially be coupled to Evan's proposal if there was a presumption that "ECI vs. no-CoT capabilities" was an easier-to-measure metric that's upstream of many monitorability metrics, such that AI companies would accept a higher burden of... (read more)

4Lukas Finnveden2d

I would phrase this somewhat differently: You need to trust that, whenever there is some plausible input that would cause the policy to behave in a bad way, the monitor (that's based on the same model as the policy) has been elicited to actually flag that bad behavior. (Granting that it does have the capability to do so.) In other words, I think that ruling out "collusion" may not be sufficient. There could be many other reasons that the monitor doesn't flag some particular misbehavior. This includes: that developers might never have trained it to flag that misbehavior, because the misbehavior is OOD from its training or because, during training, the humans who provided ground truth didn't understand that anything bad was happening in analogous situations, and therefore never provided a reward signal towards flagging the bad behavior as bad. You do propose to "evaluate the extent to which the model can tell apart your monitor evals (that are looking for monitor collusion) from real monitoring" which seems like it should help address the OOD problem in particular, though I'm not entirely clear on the details of how these experiments would work. ... Separately, if the monitor has limited compute, I also don't think it's strictly true that the compute-limited monitor will have the capability (elicited or not) to flag misbehavior whenever the policy has the capability to do something catastrophic. I think this is probably less important as a failure mode than the elicitation issues above, but one counterexample is: * Let's say there's 1000 places in a program to insert a backdoor. * Let's say that figuring out how to make a backdoor takes 100 units of compute, that making a benign edit or backdoored edit takes 1 unit of compute, and that detecting whether a specific edit contained a backdoor takes 10 units of compute. * Then the policy can spend 1100 units of compute to make 999 benign edits and 1 randomly placed backdoored edit, and the monitor would need 10*10

4Lukas Finnveden2d

Thanks for this. Redwood distinguishes between two types of control, that have been referred to using different names: * Control against "high-stakes" / "concentrated" threats — e.g. exfiltrating the weights or hacking the data center using a small number of actions. * Control against "low-stakes" / "non-concentrated" / "diffuse" threats — e.g. research sabotage where individual actions aren't clearly incriminating. (See eg here https://blog.redwoodresearch.org/p/how-can-we-solve-diffuse-threats.) I'll use "non-concentrated" in this comment because I think "low-stakes" is a confusing name and "diffuse" has an unfortunate name-clash with your point (4). What type of monitorability do you think is required to protect against non-concentrated threats? (Terminologically, it seems like it should go in "control monitorability", but this seems tough because individual actions aren't incriminating, so it would be hard to construct a linux-arena equivalent.)

My Current Take on Counterfactuals

abramdemski

[Epistemic status: somewhat lower confidence based on the fact that I haven't worked out a detailed theory along the lines I've suggested, yet.]

I've felt like the problem of counterfactuals is "mostly settled" (modulo some math working out) for about a year, but I don't think I've really communicated this online. Partly, I've been waiting to write up more formal results. But other research has taken up most of my time, so I'm not sure when I would get to it.

So, the following contains some "shovel-ready" problems. If you're convinced by my overall perspective, you may be interested in pursuing some of them. I think these directions have a high chance of basically solving the problem of counterfactuals (including logical counterfactuals).

Another reason for posting this rough write-up is...

(Continue Reading - 7188 more words)

0Vivek Hebbar5d

Maybe someone should hand it off to the LLMs at this point...

Vivek Hebbar1d*10

Not sure this matches Abram's vision, but I was curious what axioms we could have for counterfactuals such that they:

Support counterpossibility (we can counterfact into a logical impossibility without explosion)
Are never refuted by the real world (if A is actually true, then implies B)
- (this is inspired by Abram's dutch book showing that counterfactual expectations must match conditional expectations except on impossible actions)
Obey as many intuitive properties as possible subject to these constraints

Here's what GPT, Claude, and I came up with:

Write for

... (read more)

Operationalizing FDT

Vivek Hebbar

8mo

This post is an attempt to better operationalize FDT (functional decision theory). It answers the following questions:

given a logical causal graph, how do we define the logical do-operator?
what is logical causality and how might it be formalized?
how does FDT interact with anthropic updating?
why do we need logical causality? why FDT and not EDT?

Defining the logical do-operator

Consider Parfit's hitchhiker:

A logical causal graph for Parfit's hitchhiker, where blue nodes are logical facts

An FDT agent is supposed to reason as follows:

I am deciding the value of the node "Does my algorithm pay?"
If I set that node to "yes", then omega will save me and I will get +1000 utility. Also I will pay and lose 1 utility. Total is +999.
If I set that node to "no", then omega will not

...

(Continue Reading - 1545 more words)

Vivek Hebbar1d10

The operationalization of FDT given here can't support similarity-based cooperation between agents who aren't identical to each other, because in a causal graph we can't have two distinct nodes A and B both be upstream of the other. In my view this is a fatal flaw.

Coherence arguments do not entail goal-directed behavior

Rohin Shah

One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.

We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent, since if we could notice these failures, so could the agent. So we should expect that powerful agents appear coherent to us. (Note that it is possible that the...

(Continue Reading - 1974 more words)

1habryka2d

I am sorry, I am still totally failing to see the relevance of any of this, or how it's responding to what I am saying. Like, there are like 15 things you seem to be assuming about what I am saying that I am not saying. Like, what is that last paragraph about? Where am I saying that agents always want to be arbitrarily coherent under arbitrary definitions of the term? Are you just tone-policing the term of the word "coherent"? What does that have to do with anything I say? Why is it important that "coherence" has exactly the right valence? Yes, sometimes you don't want to become more coherent, but competitiveness considerations push you in that direction. What does this have to do with e.g. ML models obviously being trained in a competitive context that encourages coherence? [...] It would be an even more confusing way to define the word "coherent" if you insist on it only ever being applied to positive things. In that case "coherent" would have to become fully synonymous with good. Like, sure, you can have whatever connotations you want with the word "coherent". I don't really understand what this has to do with how people de-facto use the word "coherence" and "coherence arguments" in practice. Basically every positively-valenced word will of course have instances where more of it is bad. Imagine someone being like "Hey man, you seem to use the word 'principled' in a way that is super weird. Like, here is a situation where someone would regret being more 'principled', but the word has positive valence, so it's very confusing to use such a confusing word like 'principled'." No, of course 'principled' and 'coherent' and basically every other word points to a bunch of structural properties in the world that of course sometimes are good and sometimes are bad. ---------------------------------------- And given that you keep asking questions that indicate that you aren't reading what I am writing, I am bowing out. I shall write my response one more time in all ca

2habryka2d

I honestly don't know what you are responding to here. What are you disagreeing with? How is this responding to anything I said? I have read this comment 10+ times and still can't figure out what you are saying or how this is a response to any of what I am saying. I feel like you are just repeatedly again and again and again trying to put words in my mouth that I am trying to make some kind of highly specific argument that I am not making and then trying to call that "the coherence argument" or something. [...] I don't understand. This is not a proof, this just an assertion that says "obviously". The standard proofs of this would route through arguments that require VNM rationality already, so this just totally begging the question. Also why on earth would the answer to this question be obvious? It is extremely non-obvious (and in many scenarios not even true). In-practice, in real-life, it is extremely common that directly pursuing an objective is less effective than pursuing nearby objectives for many different reasons. I also still don't understand how it is relevant at all. I said that the arguments for why markets select for coherence route through dutch books. Then you "proved" a completely different thing which is that "the market selects for agents that 'optimize' for eventual money". The market selects for many things. Coherence is one of them. Optimizing for money is a stronger thing it also selects for. I don't understand how this is responding at all to anything I said. Your proof doesn't say anything about coherence. There is a Lemma you could try to proof that says "well, clearly markets select for optimizing for money, and optimizing for something must clearly mean not being exploitable in being drained off the thing you are optimizing for, so markets select for unexploitability" and then, what a surprise, you would probably use dutch book arguments to show that. ---------------------------------------- If you are incoherent, the speed at whic

2Steven Byrnes2d

Sorry if I missed the mark on where you were coming from. We can drop this thread, I think the other conversation branch is more promising. (I went back and edited in the proof that I had described as “obvious”, for completeness, in case you still care.)

habryka2d*10

The proof you added still doesn't do anything. Like, at a mechanistic level in order to even define what it means for an agent to take actions that maximize expected eventual money, or anything else, you would need to have assumed non-dutch-bookness, because otherwise you don't get to create a utility function to maximize over.

We don't have a canonical definition of "optimization" for non-coherent agents.

if Agent A follows course of action CA, and Agent B follows course of action CB, and Agent A winds up with more eventual money than Agent B, then that’s

... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Defining the logical do-operator