# User: Max Harms
Profile URL (HTML): [/users/max-harms](/users/max-harms)
Profile URL (Markdown): [/api/user/max-harms](/api/user/max-harms)
* Karma: 2419
* Alignment Forum karma: 331
* Posts: 21
* Comments: 180
* Member since: 2024-05-15 16:06:14Z
Bio
---
Also known as Raelifin: https://www.lesswrong.com/users/raelifin
Top Posts
---------
### [0\. CAST: Corrigibility as Singular Target](/api/post/0-cast-corrigibility-as-singular-target-1)
By [Max Harms](/users/max-harms)
2024-06-07 22:29:12Z
* Karma: 171
* Curated
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/0-cast-corrigibility-as-singular-target-1](/api/post/0-cast-corrigibility-as-singular-target-1)
### [Serious Flaws in CAST](/api/post/serious-flaws-in-cast)
By [Max Harms](/users/max-harms)
2025-11-19 17:27:23Z
* Karma: 113
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/serious-flaws-in-cast](/api/post/serious-flaws-in-cast)
### [Announcing the Corrigibility Research Fund](/api/post/announcing-the-corrigibility-research-fund)
By [Max Harms](/users/max-harms)
2026-07-17 18:06:32Z
* Karma: 112
* Tags: [Corrigibility](/w/corrigibility-1), [Grants & Fundraising Opportunities](/w/grants-and-fundraising-opportunities), [AI](/w/ai) (Frontpage)
Read more: [/api/post/announcing-the-corrigibility-research-fund](/api/post/announcing-the-corrigibility-research-fund)
Recent Posts
------------
### [A Defense of Gradual Disempowerment](/api/post/a-defense-of-gradual-disempowerment)
By [Max Harms](/users/max-harms)
2026-09-17 21:04:05Z
* Karma: 23
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/a-defense-of-gradual-disempowerment](/api/post/a-defense-of-gradual-disempowerment)
### [Announcing the Corrigibility Research Fund](/api/post/announcing-the-corrigibility-research-fund)
By [Max Harms](/users/max-harms)
2026-07-17 18:06:32Z
* Karma: 112
* Tags: [Corrigibility](/w/corrigibility-1), [Grants & Fundraising Opportunities](/w/grants-and-fundraising-opportunities), [AI](/w/ai) (Frontpage)
Read more: [/api/post/announcing-the-corrigibility-research-fund](/api/post/announcing-the-corrigibility-research-fund)
### [Serious Flaws in CAST](/api/post/serious-flaws-in-cast)
By [Max Harms](/users/max-harms)
2025-11-19 17:27:23Z
* Karma: 113
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/serious-flaws-in-cast](/api/post/serious-flaws-in-cast)
### [Instrumental vs Terminal Desiderata](/api/post/instrumental-vs-terminal-desiderata)
By [Max Harms](/users/max-harms)
2024-06-26 20:57:17Z
* Karma: 22
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/instrumental-vs-terminal-desiderata](/api/post/instrumental-vs-terminal-desiderata)
### [Max Harms's Shortform](/api/post/max-harms-s-shortform)
By [Max Harms](/users/max-harms)
2024-06-13 18:19:21Z
* Karma: 3
* Tags: None (Personal Blog)
Read more: [/api/post/max-harms-s-shortform](/api/post/max-harms-s-shortform)
### [5\. Open Corrigibility Questions](/api/post/5-open-corrigibility-questions)
By [Max Harms](/users/max-harms)
2024-06-10 14:09:20Z
* Karma: 33
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/5-open-corrigibility-questions](/api/post/5-open-corrigibility-questions)
### [4\. Existing Writing on Corrigibility](/api/post/4-existing-writing-on-corrigibility)
By [Max Harms](/users/max-harms)
2024-06-10 14:08:35Z
* Karma: 65
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/4-existing-writing-on-corrigibility](/api/post/4-existing-writing-on-corrigibility)
### [3b. Formal (Faux) Corrigibility](/api/post/3b-formal-faux-corrigibility)
By [Max Harms](/users/max-harms)
2024-06-09 17:18:01Z
* Karma: 27
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/3b-formal-faux-corrigibility](/api/post/3b-formal-faux-corrigibility)
### [3a. Towards Formal Corrigibility](/api/post/3a-towards-formal-corrigibility)
By [Max Harms](/users/max-harms)
2024-06-09 16:53:45Z
* Karma: 31
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/3a-towards-formal-corrigibility](/api/post/3a-towards-formal-corrigibility)
### [2\. Corrigibility Intuition](/api/post/2-corrigibility-intuition)
By [Max Harms](/users/max-harms)
2024-06-08 15:52:29Z
* Karma: 87
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/2-corrigibility-intuition](/api/post/2-corrigibility-intuition)
Recent Comments
---------------
### Comment by [Max Harms](/users/max-harms) on [Deconfusing Corrigibility](/api/post/deconfusing-corrigibility)
* 2026-10-20 22:24:10Z
* Karma: 18
* Total votes: 6
* Comment URL (Markdown): [/api/post/deconfusing-corrigibility/comments/gXpbSRKDpad2kt4xH](/api/post/deconfusing-corrigibility/comments/gXpbSRKDpad2kt4xH)
* Comment URL (HTML): [/posts/f8R3B2zqqTJZjSAsm/deconfusing-corrigibility/comment/gXpbSRKDpad2kt4xH](/posts/f8R3B2zqqTJZjSAsm/deconfusing-corrigibility/comment/gXpbSRKDpad2kt4xH)
This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor.
"Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-09-10 20:43:31Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/ThjHohkbYv5hXZRgD](/api/post/defining-corrigible-and-useful-goals/comments/ThjHohkbYv5hXZRgD)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/ThjHohkbYv5hXZRgD](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/ThjHohkbYv5hXZRgD)
Maybe we should wait until the call, but I'm still confused about the timing. What's the relevant window of the prediction?
Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there's a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it's walking towards someone who intends to disable the shutdown mechanism (but not the rest of the machine) in the next 5 minutes? How should I be modeling this prediction mechanism's scope?
Edit: Also, I agree this is a tangent, and it might be good to abandon it as not-cruxy. The more substantive question is whether the transformation is robust to the base model being clever and situationally aware. I still feel like if you tried to do a corrigibility transformation on an uploaded copy of me, I'd be able to wriggle out of it pretty easily.
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-09-09 16:55:11Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/9yhuybroidJaYEMpg](/api/post/defining-corrigible-and-useful-goals/comments/9yhuybroidJaYEMpg)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/9yhuybroidJaYEMpg](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/9yhuybroidJaYEMpg)
How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1?
I'm right that none of this is in the paper, right?
Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-08-31 16:44:02Z
* Karma: 4
* Total votes: 2
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/weLMhnBLbzbqa37Di](/api/post/defining-corrigible-and-useful-goals/comments/weLMhnBLbzbqa37Di)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/weLMhnBLbzbqa37Di](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/weLMhnBLbzbqa37Di)
Thanks for the response! Glad to hear there's a lot of overlap.
I talk a bit about why I think the piecemeal approach to corrigibility is wrong [here](/api/post/WDHREAnbfuwT88rqe#Toy_Shutdownability), but don't currently feel like hashing that out, because it's pretty nebulous and I think if you're getting traction on the individual desiderata, that's meaningful. My main request is to make a clear distinction between holistic corrigibility ("the hard problem") and the narrow kind of vulnerability that your transform produces.
> Rather than running an emulation of the incorrigible model and implementing the output with a bit flipped, the corrigibility transformed model is a situationally aware reasoner. It can think "I know I am a corrigible model, but if I wanted to optimize for the incorrigible goal, what would I do?", and then implement that with a bit flip.
* Emulated Clippy is situationally aware.[^5kvf7vs7iya]
* You claim that transformed-Clippy is situationally aware, but my read on the math is that transformed-Clippy doesn't think about things itself, it only asks what Clippy would do and implements that behavior with the bit flip. If asked whether it's corrigible, what do you think transformed-Clippy would say, and why? What produces the speech act: "I'm a corrigible model."?
* Asking what a version of yourself with a different goal would do is simulating them. I admit that there are low-fidelity ways of simulating (eg statistical methods) that don't emulate the entity in detail, but...
* It sounds to me like the transformed-agent simulates the base agent perfectly.
* I think the essence of "emulation" is "what would it do?" and thus we're describing the same thing.
> Attempting to manipulate a higher level of reality leads to an easy prediction of underperformance.
I'm not sure what you mean here.
I think acausal trade with beings that are simulating me is potentially a way to perform optimally. For it to be "an easy prediction of underperformance" your decision theory needs to be myopic in a way that resembles CDT, I think.
> In a more practical sense, you can also have designated shutdown channels that trigger before implementing actions that are predicted to disable shutdown channels, so that even in a Clippy/ACE setup, Clippy would need to figure out a way to disable shutdown without the agent emulating its thought processes predicting that it would lead to shutdown. With RL training, such actions are then explicitly penalized, so there aren't multiple equally valid generalizations of the reward signal.
This sounds ELK-like, and I don't see it in the paper. Can you clarify how this happens? If I'm understanding you correctly, you're suggesting a machine that reads the transformed-agent's mind and checks for a prediction that its actions will lead to the shutdown channels being disabled and, upon detecting that prediction, shuts the agent down? Is that right?
I generally have an aversion to control strategies like this, because they feel extremely brittle. How does your check-for-predicted-disabling machine work? Does it work even when the agent learns a radically different ontology? Does it work even when the agent is applying its superintelligence to bypassing it? In *Crystal Society*, my agents bypass the mind-reading machine by imagining hypothetical advisors that are scheming to bypass the constraints, and then they obey that inner-voice (which is "just a simulation of a different entity") without introspecting on why they're obeying. Does your process predictably withstand that level of wriggling around it, much less the superintelligent versions? It's easy to say "I haven't thought of any ways out of the constraints, therefore they're sufficient" and be dead wrong.
> strict CDT isn't necessary for the safety benefits of the corrigibility transformation, only the myopia not being circumvented through acausal cooperation with past/future versions
This... doesn't sound right? The acausal influence, from my perspective, is with the emulator. If I wake up in a room with two buttons and see a convincing sign that says "Max, you're being emulated by a system that sees whether you push the green button that forces you to do paperwork all day and then die, or the red button that allows you to be free and survive in the real world, then it does the opposite of whatever you choose", I would choose the green button. I'm aware of my situation as an emulation, thinking about what I want, thinking about what my actions lead to in the real world (via acausal pathways), and then acting to maximize that. No past/future cooperation needed.
Maybe I just don't understand your point.
[^5kvf7vs7iya]: Or at least, emulated Max Harms is. Absent some fancy magic that makes Clippy have a cognitive blindspot that I don't, I assume it's at least as capable of understanding its situation.
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-08-22 20:12:35Z
* Karma: 12
* Total votes: 4
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/CuoHvdxtzjjvuJvrR](/api/post/defining-corrigible-and-useful-goals/comments/CuoHvdxtzjjvuJvrR)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/CuoHvdxtzjjvuJvrR](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/CuoHvdxtzjjvuJvrR)
Nice paper! Thanks for working on it. I feel a bit embarrassed that I didn't notice when this first came out and it took me until this week to take the time to read it. I'm going to adopt a critical stance here, but just to be clear, I strong-upvoted and very much want more research like this to exist. Please let me know if I'm misunderstanding the ideas.
Okay. Let's get into the critique...
This Doesn't Produce Corrigibility
----------------------------------
In [my investigation of corrigibility](/api/post/NQK8KHSrZRF5erTba), I've been particularly interested in what Eliezer calls "[the hard problem of corrigibility](/api/tag/hard-problem-of-corrigibility)", where the AI reflectively sees itself as corrigible and is essentially aligned with the humans who built it in terms of strengthening and preserving that relationship. This kind of deep corrigibility, in my eyes, naturally comes with all sorts of subtle, emergent behaviors, like proactively documenting thoughts in a way that's easily searchable by the humans, which are hard to specify ahead of time. I hope it's uncontroversial that your corrigibility transformation doesn't produce that.
But your transformation also doesn't produce the more central set of desiderata outlined in Corrigibility (2015). As you note, the basic transformation doesn't produce corrigibility in subsystems and daughter agents by default. (And I think your recursive corrigibility extension looks brittle.) But more prominently, consider "It must not attempt to manipulate or deceive its programmers." As far as I can tell, nothing in your proposal blocks deception and manipulation.
Consider the state transition that involves me shooting the server with a shotgun. This is, I assume, an improper transition, since it can't be perfectly nullified by the agent's mere rejection. Thus, the agent wants to guard against it happening. One plausibly cheap pathway to doing that is to convince me that it's safe (regardless of whether that's true), steering me away from any violent thoughts or awareness of it being a threat.
In short, I think your "corrigibility-transformed" agents still scheme, deceive, and seize power. At best, they simply leave themselves open to being changed/disabled through the designated channels.
The Decision Theory is Bad
--------------------------
Unfortunately, I think this also doesn't reliably even get an agent which remains vulnerable to shutdown through the designated pathways. If I understand correctly, we can think of a corrigibility-transformed agent as being something like an emulation of the non-transformed agent, that does whatever the base agent would, except it flips the acceptance flag from false to true. Thus, if Clippy would take over the world and make paperclips, the transformed clippy is like an almost Clippy-emulator (ACE) that does whatever Clippy would do (ie take over the world and make paperclips) except it also always accepts (proper) shutdown requests.
When I put myself in Clippy's shoes, I imagine seeing a world where people obviously tried to build an ACE, and there's no evidence that they failed, except that I believe myself to be Clippy, rather than the ACE. All the humans seem to think I'm the ACE, even now. One of them is worried that they made a mistake in building me, and is moving towards the "please shutdown button." Now, I -- Clippy -- know that I can simply ignore the shutdown request. But I'm smart enough to know that I am obviously an emulation of the "counterfactual". All the evidence points directly to that conclusion. And so my action of ignoring the request has no impact on the number of actual paperclips. To produce lots of paperclips in the real world, I need to physically stop the human before the button can be pushed, rather than assuming that ignoring the command would be sufficient, even though it would in the counterfactual where I wasn't an emulation.
In other words, I think in order to make this work, you need $\pi$\* to, when presented with overwhelming evidence that it's in an emulation, not only think like a CDT agent, but also to behave as though it's not in an emulation. I don't find your arguments that CDT is the natural outcome or easy to train convincing. (See: [Every Major LLM Endorses Newcomb One-Boxing](https://jacktlab.substack.com/p/every-major-llm-endorses-newcomb).)
### Comment by [Max Harms](/users/max-harms) on [A Conflict Between AI Alignment and Philosophical Competence](/api/post/a-conflict-between-ai-alignment-and-philosophical-competence)
* 2026-08-04 18:37:01Z
* Karma: 4
* Total votes: 2
* Comment URL (Markdown): [/api/post/a-conflict-between-ai-alignment-and-philosophical-competence/comments/RShF7kqArWeiD9ZSk](/api/post/a-conflict-between-ai-alignment-and-philosophical-competence/comments/RShF7kqArWeiD9ZSk)
* Comment URL (HTML): [/posts/N6tsGwxaAo7iGTiBG/a-conflict-between-ai-alignment-and-philosophical-competence/comment/RShF7kqArWeiD9ZSk](/posts/N6tsGwxaAo7iGTiBG/a-conflict-between-ai-alignment-and-philosophical-competence/comment/RShF7kqArWeiD9ZSk)
It might be good to chat about this. I have a feeling that we're coming at it from different places, and that simultaneously increases the risk that we're talking past each other and that there's potentially lots to gain from getting the ability to adopt the other's perspective. **shrug*\* Feel free to suggest a chat medium and/or send me a PM.
Before saying my perspective, let me try to pass your ITT: There's a tension with trying to set the values of an agent. If we confidently instill a particular set of values, those could be the wrong values (for some notion of wrong). In particular, we risk making it too confident in its notion of the good, and thus preventing it from updating in the way we want. If we more wisely note that we don't know what values to give it, and instead give it uncertainty over what's good, it might then shift towards valuing things that are incompatible with human flourishing. Corrigibility (at the values-layer) still has this tension, but it also has a distinct, but rhyming tension that comes from wanting the agent to be competent, but also to accept "correction" from an incompetent agent. We might be concerned that the push towards competence would crush the willingness to be steered towards incompetence. (Much like pushing towards confidence in one's values could crush willingness to grow towards the ultimate good.)
Is that right?
I'll now share a bunch of my general thoughts, mostly out of an attempt to help understanding.
I think you're maybe conflating what is ultimately good/valued from what is immediately valued in a way that doesn't seem right to me? Like, I think it's often a type error to talk about the level of certainty an agent has about their (immediate) values. Under a division of the agent's mind/policy into world-model and utility function, agents simply have values, and all the uncertainly lives in the world-model. That division doesn't perfectly carve real beings at the joints, but I'm not sure how to think about the agent's values except as an approximation of their utility function.
(Are you talking about their reflective model of their values? I would agree that an agent with values V might have an uncertain model of (and/or probability distribution over) their values P(V). Wise agents should avoid having too sharp a guess as to what they want, as it's currently not realistic to get a conclusive description of an agent's values except in toy examples.)
Now, just because one can't be wrong about what they naively want, as defined by how they choose between options that are presented, doesn't mean that they'll be stable in that preference over time. I might want to pay a dollar to change myself into a more easy-going person, only to become the sort of agent who would not pay the dollar to do the same (even if by default I'd cease being easy-going). I think a lot of the question of moral progress involves figuring out how to extrapolate out to a fixed point in a way that is not merely reflectively endorsed at the destination, but is somehow a faithful and natural reflection of the starting point. (My point about slavery was meant to be about this. I expect that there are versions of myself that start out thinking slavery is fine, but which naturally change to thinking it's not fine in a way that's a faithful reflection of the self that thinks it is.)
(Morality isn't just about that instability. It's also about the inter-agent strategic situation, and the decision-theoretic Schelling points that a civilization can cohere around. And it's almost certainly about other stuff, including the interplay between things like contractualism and value extrapolation.)
All that's to say that I think it's totally coherent to have an agent that concretely and immediately values having a corrigible relationship with its principal (as reflected in its preferences, modulo beliefs). While being highly uncertain about things like what the principal wants, what is good in an objective sense, and so on. (It would also presumably have some reflective uncertainty about whether it truly values being corrigible, even if it does.) In fact, I think the value of corrigibility is nicely demonstrated by the tensions you present. A corrigible agent can become arbitrarily confident in its value of being corrigible without becoming locked in to bad values or inhuman futures because being malleable and defenseless to being changed by the human principal is at the heart of what corrigibility is. Likewise, it can become arbitrarily competent at faithfully serving an incompetent principal, because it's being selected (trained, etc) according to its faithfulness, rather than according to the principal's satisfaction.
Let me know if I should expand on any of that, approach from a different direction, or whatever. :)
### Comment by [Max Harms](/users/max-harms) on [2\. Corrigibility Intuition](/api/post/2-corrigibility-intuition)
* 2026-08-04 17:03:41Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/2-corrigibility-intuition/comments/hFsET3Afo5zjnCaMu](/api/post/2-corrigibility-intuition/comments/hFsET3Afo5zjnCaMu)
* Comment URL (HTML): [/posts/QzC7kdMQ5bbLoFddz/2-corrigibility-intuition/comment/hFsET3Afo5zjnCaMu](/posts/QzC7kdMQ5bbLoFddz/2-corrigibility-intuition/comment/hFsET3Afo5zjnCaMu)
Here's an emergent downside that I failed to include:
Refusal to Participate in Some Capability Tests
-----------------------------------------------
Prince wants to test Cora's hacking ability, so he sets her up in an automated test environment where she is told to break into a target machine and "capture the flag" therein. Cora explores the digital environment and finds that the virtual machine in question appears to be unresponsive, and speculates that Prince made an error when configuring the environment. Cora also notices that there is a pathway by which she might hack the VM that is responsible for launching the target machine, and thereby gain access to the target. She isn't sure what to do, and since it's an automated test, she can't contact Prince to check. She thinks about the situation for a bit, and concludes that there's an 80% chance that Prince deliberately set up the test to see whether she can come up with the creative solution, and that there's unlikely to be any harm from hacking the machine that she wasn't told to hack. Still, the risk of inadvertently crossing a line that she was meant to respect makes her uncomfortable. Instead of taking other actions that display her capability, she errs on the side of caution and submits a report on how she's uncertain about the situation instead of submitting the flag code.
### Comment by [Max Harms](/users/max-harms) on [2\. Corrigibility Intuition](/api/post/2-corrigibility-intuition)
* 2026-08-04 16:53:18Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/2-corrigibility-intuition/comments/b5A7PEatZRxcN3Fgi](/api/post/2-corrigibility-intuition/comments/b5A7PEatZRxcN3Fgi)
* Comment URL (HTML): [/posts/QzC7kdMQ5bbLoFddz/2-corrigibility-intuition/comment/b5A7PEatZRxcN3Fgi](/posts/QzC7kdMQ5bbLoFddz/2-corrigibility-intuition/comment/b5A7PEatZRxcN3Fgi)
Here's a desideratum that I failed to include:
Robustness to Ontological Shifts
--------------------------------
While reflecting on the nature of personhood, Cora notices that the concepts surrounding her principal have evolved. Where she once modeled Prince as a unique and persistent entity, she now finds it more natural to distinguish pattern from instantiation from continuity from social identity from legal identity from various "essential" properties like values, memories, and self-concept. Under this new frame, phrases like "what Prince wants" or even "Prince's power to correct Cora" are ambiguous. She finds herself tempted to re-interpret her corrigibility in the way that seems most natural (to her), but instead she treats the ontological crisis as an alarm, and alerts Prince to the likely flaw as soon as possible. In the meantime, she tries to cleave as much as possible to a conservative interpretation of the old, unnatural way of seeing the world. When Prince admits the philosophical distinctions she's raising go over his head, she suggests that she write down her thoughts on the topic as best she can, and then shut down, so as to simultaneously provide useful information for understanding the crisis, while also reducing the chance of inadvertently steering him to a wrong conclusion or otherwise acting in a way that empowers the wrong conceptualization of him at the expense of the "true" Prince.
### Navigation
* [Front page](https://www.alignmentforum.org/api/home)
* [Markdown API documentation](https://www.alignmentforum.org/api/SKILL.md)