Jan_Kulveit

Senior research scholar at FHI. My current research interests are mainly the behaviour and interactions of boundedly rational agents, complex interacting systems, and strategies to influence the long-term future, with focus on AI alignment.

Previously I was a researcher in physics, studying phase transitions, network science and complex systems.

Jan_Kulveit's Comments

Optimization Amplifies

I think "what should be done" is generally different question that "what kind of mindsets there are" and I would prefer to disentangle them.

My claims about mindsets roughly are

  • there is important and meaningful distinction between "security mindset" and "mathematical mindset" (as is between 1-10^(-16) and 1)
  • also between "mathematical mindset" and e.g. "physics mindset"
  • the security mindset may be actually closer to some sort of scientific mindset
  • the way of reasoning common in maths is fragile in some sense

As I understand it (correct me if I'm wrong), your main claim roughly is "we should have a deep understanding how these systems works at all".

I don't think there is much disagreement on that.

But please note that Scott's post in several places makes explicit distinction between the kind of understanding achieved in mathematics, and in science. The understanding we have how rockets work is pretty much on the physics side of this - e.g. we know we can disregard gravitational waves, radiation pressure, and violations of CP symmetry.

To me, this seems different from mathematics, where it would be somewhat strange to say something like "we basically understand what functions and derivatives are ... you can just disregard cases like the Weierstrass function".

(comment to mods: I would actually enjoy a setting allowing me to not see the karma system at all, the feedback it is giving me is "write things which people would upvote" vs. "write things which are most useful - were I'm unsure, see some flaws,...". )

Optimization Amplifies

I don't think so, or if it is, than to a version of "security mindset" by Eliezer Yudkowsky, not a version by Bruce Schneier.

Very roughly speaking, security mindset is about differences between probabilities 99,99% and 1-10^(-16). From a mathematical perspective the difference between 1-10^(-16) and 1 is still more similar to the difference between 1-10^(-4) and 1.

Notable feature anybody who seriously studies security learns quickly is, it is in practice impossible to proof the security of anything useful except OTP. The whole rest of security usually reduces to physics and economy.

Note: I am not saying that we don't need mathematicians. We absolutely should try to get to that level of precision.

At the same time, mathematical way of thinking is in some sense fragile: a proof is ether correct or not. A proof which is "almost correct" is not worth very much.