I've been thinking about out whether I can discover laws of agency and wield them to prevent AI ruin (perhaps by building an AGI myself in a different paradigm than machine learning).

So far I’ve looked into the history of the discovery of physical laws (gravity in particular) and mathematical laws (probability theory in particular). Here are 12 things I’ve learned or been surprised by.

1.

Data-gathering was a crucial step in discovering both gravity and probability theory. One rich dude had a whole island and set it up to have lenses on lots of parts of it, and for like a year he’d go around each day and note down the positions of the stars. Then this data was worked on by others who turned it into equations of motion.

2.

Relatedly, looking at the celestial bodies was a big deal. It was almost the whole game in gravity, but also a little helpful for probability theory (specifically the normal distribution was developed in part by noting that systematic errors in celestial measuring equipment followed a simple distribution).

It hadn’t struck me before, but putting a ton of geometry problems on the ceiling for the entire civilization led a lot of people to try to answer questions about it. (It makes Eliezer’s choice in That Alien Message apt.) I’m tempted in a munchkin way to find other ways to do this, like to write a math problem on the surface of the moon, or petition Google to put a prediction market on its home page, or something more elegant than those two.

3.

Probability theory was substantially developed around real-world problems! I thought math was all magical and ivory tower, but it was much more grounded than I expected.

After a few small things like accounting and insurance and doing permutations of the alphabet, games of chance (gambling) was what really kicked it off, with Fermat and Pascal trying to figure out the expected value of games (they didn’t phrase it like that, they put it more like “if the game has to stop before it’s concluded, how should the winnings be split between the players?“).

Other people who consulted with gamblers also would write down data about things like how often different winning hands would come up in different games, and discovered simple distributions, then tried to put equations to them. Later it was developed further by people trying to reason about gases and temperatures, and then again in understanding clinical trials or large repeated biological experiments.

Often people discovered more in this combination of “looking directly at nature” and “being the sort of person who was interested in developing a formal calculus to model what was going on”.

4.

Thought experiments about the world were a big deal too! Thomas Bayes did most of his math this way. He had a thought experiment that went something like this: his assistant would throw a ball on a table that Thomas wasn’t looking at. Then his assistant would throw more balls on the table, each time saying whether it ended up to the right or the left of the original ball. He had this sense that each time he was told the next left-or-right, he should be able to give a new probability that the ball was in any particular given region. He used this thought experiment a lot when coming up with Bayes’ theorem.

5.

Lots of people involved were full-time inventors, rich people who did serious study into a lot of different areas, including mathematics. This is a weird class to me. (I don’t know people like this today. And most scientific things are very institutionalized, or failing that, embedded within business.)

Here’s a quote I enjoyed from one of Pascal’s letters to Fermat when they founded the theory of probability. (For context: de Mere was the gambler who asked Pascal for help with a confusion he had.)

“I have not time to send you the demonstration of a difficulty which greatly astonished M. de Mere, for he has a very good mind, but he is not a geometer (this is, as you know, a great defect)...” – Blaise Pascal

6.

In Laplace’s seminal work putting probability theory on a formal footing, he has a historical section at the end praising all the people who did work, how great they were and how beautiful their work was. Then he has one line on Bayes where he calls his work “a little perplexing”.

“Bayes, in the Transactions Philosophiques of the year 1763, sought directly the probability that the possibilities indicated by past experiences are comprised within given limits; and he has arrived at this in a refined and very ingenious manner, although a little perplexing.”

Also, whenever you feel like you’ve missed out on your glorious youth, note that Thomas Bayes got interested in probability theory in his 50s, and died aged 59. He was not formally trained in math in his youth.

7.

I watched a talk by Pearl about his causal models, and I was struck by the extent to which he had a “philosophy” of counterfactual inference. It had seemed pretty possible to me he would have said “here was a problem, and here is my solution”, but instead he had a lot to say about counterfactuals and how he thought about them conceptually that wasn’t in the math.

It reminds me of my impression that Daniel Kahneman (and Amos Tversky) have strong models of how their minds work, of which the heuristics & biases literature is a legibilized component of, but certainly does not capture the whole thing.

Relatedly, in a lecture by Feynman on seeking new laws, he says that some people say “don’t talk about what you cannot measure”. He says he agrees insofar as your theories need measurable predictions, but he doesn’t agree that people should stop discussing their whole philosophies, as the philosophies seem to help some people come up with good guesses about laws.

I think in the past I could have found myself unable to justify my interest in the philosophy of something as more than a personal interest. Now I have a practical justification, which is that it helps me come up with guesses about how nature works! And my current guess is that many people who were successful at that had unique and well-developed philosophies.

8.

Pearl himself says that he has discovered two laws, and once you have them, you can fire him, because the rest is just algebra! And he calls it a calculus of counterfactuals, just like Newton and Bayes and everyone did. Fascinating.

I couldn’t find anything on what problems Pearl was thinking about when he came up with his calculus of counterfactuals. Like, was he personally trying to analyze clinical trials? Was he a mathematician who was friends with people doing large experiments and thought the math was interesting? I want to know what part of the world he was in contact with when developing it.

9.

I updated against expecting to resolve scientific disagreements at the time when the correct theory is known. Let me explain.

In the discovery of gravity, there were a lot of anomalies that didn’t fit the data. For instance, Jupiter didn’t follow the law: its orbit was a more elongated ellipse when it was further away. Uranus’s orbit would jiggle a bit sometimes. Also there were two stars who didn’t orbit their collective center of gravity, but instead some other point within the ellipse. At this point I would have been like “yeah, nice try, but your theory isn’t fitting the details”.

Want to know what they said at the time? (Spoilers ahead.) For the stars, they said that we were probably just looking at them at a funny angle and that’s why it didn’t work. For Uranus, they said there was an invisible planet that was knocking it off-course. And for Jupiter, they said the light was moving too slowly for the measurements to work out.

To me this seems like an awful lot of complexity cost weighing on a theory. Now it’s no longer just a theory, it’s also a lot of explaining exceptions with unlikely stories. The star-angle one doesn’t even seem testable, it gives me a Scott-Alexander-like sense of this explanation gives me so many degrees of freedom that I can probably explain away loads more anomalies with it.

Anyway… they were all right.

From Uranus’s wobbles, they found Neptune. The stars were indeed rotated at an angle. And they did some experiments and found out that light did have a speed and this explained the Jupiter issue, and opened up a whole new area of inquiry about light.

Very impressive in retrospect, but I feel like I couldn’t have gotten this right at the time.

My update is further in the direction that Jacob’s post The Copernican Revolution from the Inside argues for, which is that if two different people had different theories at the time, I do not anticipate the disagreement being able to be “clearly resolvable” at all, and do expect for it to involve a great number of judgment calls, in large part dependent on one’s “philosophy” of how to make those calls in this domain.

10.

Feynman has a wonderful quote on the art of guessing nature's laws that includes at least two paths not discussed above. That said I don’t understand them, in particular the ways that quantum mechanics was discovered. (I’m tempted to dig into that some.)

I’ve put the full quote in this footnote^[1], recommended.

11.

One confusion I wrote down in advance was “I still don’t quite know how to predict that there will not be a simple mathematical apparatus that explains something. Why the motion of the planets, why the game of chance, why not the color of houses in England or the number of hairs on a man’s head?"

Looking back on this, I don’t know whether I got a direct answer, but I now feel that my answer is something like “look for the places where Nature will show herself directly”. Obviously that’s not a very well-specified answer, but I feel like it points to a real distinction.

12.

I also made an advance prediction: “I guess I also make the advance prediction that most of the rest of the [probability] math was developed by people who liked symbol manipulation more than people doing real-world problem solving. But I would be interested to be surprised here.”

This prediction was false! It took both! All the probability math was developed by people who liked using math to reason rigorously about the world, and who were interested in understanding the real world! There were exceptions like Bayes who relied a great deal on thought-experiment, though sort of still “about” the world, not just about symbols.

When I thought of math previously I thought about my math friends in academia, who just sort of entered the abstract world as a starting point and lived in there. (“My professor does work in flat-spherical-manifold-density-vector-spaces, so I’m trying to prove something there too!“) Now I think of people trying to reason about particular parts of the world I live in, and who are trying to make an externalized symbolic calculus that can do that reasoning for them.

Next Step

The natural next step of my investigation is to learn more about how key discoveries in areas like optimization and information theory and game theory were made. How did nature show herself to these discoverers? I have written down a few advance predictions for if I continue seeking this information...

^{^}
Feynman, on the art of guessing nature’s laws, in his final lecture for BBC's Messenger Lectures:
“Or look at history, you first start out with Newton: he [was] in a situation where he had incomplete knowledge, and he was able to get the laws by putting together ideas which all were relatively close to experiment—there wasn’t a great distance between the observations and the test.”
“Now, the next guy who did something—another man who did something great—was Maxwell, who obtained the laws of electricity and magnetism. But what he did was this, he put together all the laws of electricity due to Faraday and other people that came before him, and he looked at them and he realized that they were mutually inconsistent; they were mathematically inconsistent. In order to straighten it out he had to add one term to an equation.”
“By the way, he did this by inventing a model for himself of idler wheels, and gears, and so on, in space. Then he found what the new law was, and nobody paid much attention, because they didn’t believe in the idler wheels. We don’t believe in the idler wheels today, but the equations that he obtained were correct. So the logic may be wrong, but the answer is all right.”
“In the case of relativity, the discovery of relativity was completely different: there was an accumulation of paradoxes; the known laws gave inconsistent results, and it was a new kind of thinking, a thinking in terms of discussing the possible symmetries of laws. It was especially difficult because it was for the first time realized how long something like Newton’s laws could be right—and still ultimately be wrong—and, second, that ordinary ideas of time and space that seem so instinctive could be wrong.”
“Quantum mechanics was discovered in two independent ways, which is a lesson. There, again, and even more so, an enormous number of paradoxes were discovered experimentally, things that absolutely couldn’t be explained in any way by what was known—not that the knowledge was incomplete, but the knowledge was too complete!: your prediction was, this should happen; it didn’t.
The two different routes were: one, by Schrodinger, who guessed the equations; another, by Heisenberg, who argued that you must analyze what’s measurable. So two different philosophical methods reduced to the same discovery in the end.”
“More recently, the discovery of the laws of this [weak decay] interaction, which are still only partly known, add quite a somewhat different situation: this time it was a case of incomplete knowledge, and only the equation was guessed. The special difficulty this time was that the experiments were all wrong—all the experiments were wrong.”
“Now, how can you guess the right answer when, when you calculate the results it disagrees with the experiment, and you have the courage to say the experiments must be wrong. I’ll explain where the courage comes from in a minute.”
“Now, I’m sure that history does not repeat itself in physics, as you see from this list, and the reason is this: any scheme—like, "Think of symmetry laws," or "Put the equations in mathematical form," or any of these schemes "Guess equations," and so on—are known to everybody now, and they’re tried all the time. So if the place where you get stuck is not that—and you try that right away: we try looking for symmetries; we try all the things that have been tried before, but we’re stuck-so it must be another way next time.
Each time that we get in this log jam of too many problems, it’s because the methods that we’re using are just like the ones we used before. We try all that right away, but the new discovery is going to be made in a completely different way—so history doesn’t help us very much.”