In this review of Gordon Seidoh Worley's book, Fundamental Uncertainty, I would like to explain why its main thesis is only partially rational.

Worley's thesis

Our knowledge of the truth is fundamentally uncertain because of epistemic circularity caused by the Problem of the Criterion.
We manage fundamental uncertainty by making pragmatic assumptions that lead us to believe in the truth of claims that help us achieve our goals.
Consequently, the truth that can be known is not independent of us, but rather dependent on that for which we care.
That truth is fundamentally uncertain and grounded in care has far-reaching implications for many of the world's hardest-to-solve problems.

While I mostly agree with points 1 and 2, the points 3 and 4 imply that truth is contingent on care, which I find unlikely.

The book itself justifies Worley's main thesis by detailed arguments described in the collapsible section.

Chapterwise summary of Worley's arguments

Chapter 1 defines epistemology, then proceeds to:
a. Tear down the naive epistemology where it's obvious which statements are true and which aren't;
b. Argue that logical epistemology is a branch of mathematics and has its flaws, like failing to account for observation error, Godel-proven incompleteness and circularity;
c. Fundamental uncertainty allegedly blocks mankind from finding out truth in some cases.^[1]
Chapter 2 makes the case that our ability to know the truth is undermined by meanings of words:
a. Words may have overlapping meanings ^[2](e.g. "hot" can describe the temperature of water, the spiciness of food, the attractiveness of a person, or the popularity of a song);
b. They also can have obsolete^[3] meanings (e.g. calling whales fish);
c. Words can also be generally fuzzy (e.g. "cold" is below a certain temperature dependent on the speaker).
d. According to the author, expert systems have failed to become useful because they didn't solve the problem of grounding symbols in context.
e. Words' representations are grounded in experience, causing different minds to interpret words differently.
Chapter 3, "Why can't we all agree what's good?" has the author introduce the question, then explain why we disagree and why we disagree on what is good or bad:
a. Rational disagreement has the author claim that sufficiently rational people are intuitively supposed to come to an agreement, but the intuition fails because even the power of Bayes' theorem fails to make them completely agree if they had different priors or fails to dismantle a stupid ABSOLUTE prior.
b. Moral foundations are somehow different because of different prior beliefs. Haidt conjectured that humans have six moral axes which they weigh differently.
Chapter 4, "Why don't we always do what we should?", makes two points:
a. Our mind consists of System 1 responsible for immediate actions bringing short-term satisfaction and System 2 responsible for long-term benefits;
b. It is hard to assess new formal systems for soundness or performing better than the old formal system. In fact, no sufficiently capable system can prove its own soundness.
Chapter 5, "What does it mean to know?", has the author define The Many Ways Of Knowing, then makes the point that we can know things well enough, but can't fully justify our knowledge: the proofs that establish Bayesian idealness are mathematical arguments; this requires us to trust that mathematical proofs correctly establish what's true. According to the author, that's itself a claim that needs to be proven if we want to completely trust our knowledge of the truth.
Chapter 6, "How do we know what we know?", has the author describe the Problem of the Criterion: to know if a claim is true, we need a method for testing if it's true, but how can one check the method itself? The closest thing that mankind has to a solution is using what works and assuming that particular claims, like the rules of logic or Peano arithmetic, are true.
Chapter 7: Why care about knowing the truth? The author's argument is twofold: there are systems which steer towards an outcome and successful steering for goals like survival, reproduction or anything else requires having an accurate world model.
Chapter 8: Why does fundamental uncertainty matter? According to the author, it matters because the modern world is steeped in many problems which cannot be solved unless we understand uncertainty. Examples include the Culture War, moral uncertainty, the crisis of meaning, metaphysics, Goodhart's curse and AI-induced existential risk. The Culture War is actually a fight over deeply-held values; moral uncertainty allows for compromises (which the author calls moral trade); the crisis of meaning is due to our life becoming oversaturated with choices which is incurable while being consistent with life having a meaning; Goodhart's Curse is the curse of over-optimisation for narrow criteria, and the ASI could end up enacting it unless it understands the fragility of human values.
Chapter 9: How do we live with fundamental uncertainty? One of the results originating in fundamental uncertainty is that most people are confused about what's normal: they have a poor understanding of truth, mistake the relative for the absolute, value episteme over other ways of knowing. Instead, "the truth is intersubjective, arising from the intersection of ourselves and the world as we find it", but the next subsection has the author argue that "The truth will still be there, waiting for you to know."

Alternate Framework: Perfect Bayesians, Imperfect Imitators

Consider a perfect Bayesian reasoner who was never exposed to images (e.g. if it was raised on math, texts, coding tasks), then was tasked with roleplaying a human brain, i.e. receiving images and other sensory signals and outputting orders for muscles to contract or relax so that the agent could interact with the environment and achieve some pre-set complex goal. Suppose also that every time a sensor sends a signal (e.g. a camera's pixel registers an RGB value) the sensor has an unknown probability to malfunction, outputting a random value instead of the truth and that the malfunctions are independent events.

The two main problems with a perfect Bayesian are that it uses infinite compute and that it can reason about too much. For example, if the Bayesian was tasked with winning a chess game, it would be able to evaluate entire huge sets of potential positions and arrive at a perfect strategy which would cause it to win once the opponent makes a theoretically fatal mistake unnoticeable even to human grandmasters. Infinite compute also lets a perfect Bayesian, for example, not just reason about sociological laws or potential ethoses of alien civilisations (e.g. coming up with arguments^[4] like "Human Values Are Contingent" or "Curiosity Isn’t Convergent"), but check one's conjectures by simulating such civilisations in detail.

A real-world reasoning agent, like a human brain or an LLM, doesn't have infinite resources. Instead, it is forced to rely on heuristics, which are supposed to make it check far less conjectures, and to optimize said heuristics. Similarly, categorization of objects becomes training a classifier and trying to make sense of how to deal with OOD cases (see, e.g. my discussion of Chapter 8).

Chapterwise critique

Chapters 1-2 are almost entirely justified in their reasoning about the sources of uncertainty in observations and words spoken by others. The first important problem emerges in Chapter 3 where the author talks about Aumann's theorem.

Chapter 3. World models built from slightly different priors have predictions close to convergent

Given the same evidence, Bayesian observers are more likely to converge to similar conclusions than the author assumes.

Consider a coin which has an unknown probability to land heads and flip it times. Suppose that before the experiment a Bayesian had the priors that assigned weight to the region . If the experiment has the coin land on heads times, then the weight of that region becomes .

The ratio is at most and rapidly decays when is away from . If after the experiment the Bayesian fails to become sure that is at distance at most from , then before the experiment the priors somehow assigned the probability of at most to being at distance at most from Such confidence is unlikely to emerge in any way aside from an incredibly confident prior or prior bits of evidence steering the value of away from .

Similarly, evidence related to more complex issues usually arrives in overwhelming quantities compared to the one necessary to establish a theory.

However, moral disagreements are likely a result of incompatible values which the framework involving a perfect Bayesian cannot explain.

Chapter 4. What does it mean to distrust oneself?

The main problem with Worley's argument in the Section "Distrusting Ourselves" is the following. Were Grace to choose "to rely on a formal system—a collection of axioms and rules for deriving conclusions from premises—to figure out what beliefs to hold", she would also have to have a batch of heuristics which she would use to optimize the process of figuring out whether something is provable, disprovable, insoluble or has yet to be sorted. For example, she, along with human mathematicians and unlike GPT-5.4 Pro, could believe that an Erdos problem from number theory is to be translated into probability theory when the actual solution is to translate it into analytic number theory. This type of bias has nothing to do with a formal system having to be replaced with a new one, it could have been avoided only by widening Grace's area of knowledge.

As for optimizing formal systems themselves, such optimizations cannot create any new results unrelated to infinite sets. Additionally, the author makes the point that Löb's theorem makes it very hard to be able to switch to a new formal system without falling into the trap.

The proof of Löb's theorem

Recall that Löb's theorem was that a system where is provable proves . It was proven as follows.

For any modal predicate there is a formula s.t. is provable. One of such formulas is .

Therefore, because can be transformed into . Additionally, , and . Since we have shown that , which is equivalent to , thus establishing

However, Löb's theorem is circumventable. Consider instead the alternate statement: Then the Löbian exploit would break down since it would cause us to prove that

But the system doesn't contain a proof that is consistent, and we can't use any formula to prove . Therefore, Löb's theorem doesn't prevent us from the action which I could describe as trusting ourselves conditioned on the system being consistent and lacking proof. Nor does it prevent us from proving that a system containing is consistent iff itself is consistent. Therefore, this source of uncertainty is reduced to a single Damocles' sword, the idea that we cannot be sure that our basic axioms, like Peano arithmetics or the Zermelo-Fraenkel set, are consistent.

Chapter 5. What does it mean to know?

The many ways of knowing are many origins

Knowing means having a world model from which a fact is easy to excavate and to use in making decisions. Worley's distinction of ways of knowledge is more related to the distinction of its origins and potential biases which they introduce.

Worley's list of the ways of knowledge

The ancient Greeks did exactly that. They used multiple words to break down knowing into several categories, including:

episteme: things you know because you reasoned them from evidence, like knowing that water boils at 100 degrees celsius because you boiled a pot of water using a thermometer to see when it started to boil
doxa: things you know because others told you, like knowing what happened at a party you didn’t attend because your friend tells you
mathema: things you know because you were educated in them, like knowing how to spell words because you were taught them in school
gnosis: things you know through direct experience, like how it feels to jump in a lake
metis: practical wisdom, usually collectively constructed from many people’s experiences, and shared with others, often starting at a young age, like how you know to look both ways before crossing the street
techne: procedural knowledge that comes from doing, like the “muscle memory” of how to ride a bicycle.

For example, mathema is a combination of training you to develop the right episteme from evidence, providing you with direct evidence (e.g. experiments shown at school) and doxa propagated through official channels. Metis is doxa reinforced by direct evidence. Techne of humans has far shorter feedback loops than most other types of knowledge and is developed by prompting motor neurons to make movements and observing the results.

Rational beliefs are grounded in math, but what could one do in a math-undescribable universe?

The main claim of the section on Rational Belief is that such beliefs are grounded in Bayesianism, which in turn requires us to ground them in math and trust that it describes reality truthfully. This argument has two versions: the weak version is that the true reality can be undescribable, which I find highly unlikely. The strong one is similar to Chapter 6.

Chapter 6. The criterion of truth

Here Fundamental Uncertainty comes up with a worthy argument. How do we know that testing of our world models by predicting our direct sensations is a reliable way to verify the truth, and can we be certain that our methods of testing are valid?

Indeed, being in a simulation would mean that nearly everything we experience describes the properties of the simbox, not the outside world, unless the hosts intervene^[5] or we succeed in a large-scale cyberattack ^[6] and gain evidence. On the other hand, smaller-scale adversaries are unlikely to ensure that a sufficiently-resourced investigator doesn't understand that an aspect of the truth is hidden.

Chapter 7. Caring is concentrating efforts on a segment of the world

Chapter 7 implies that we care about truth because it helps us steer reality towards any other goals which allegedly allows us to ground assumptions. Alas, according to the author, we aren't actually certain that we know what is true. Quoting the book,

We saw [in Chapter 2] how language limits our knowledge with imprecise categories. We learned [in Ch. 3] how even ideal reasoners can't always agree on what's true. We came [in Ch. 4] face-to-face with our inability to prove the soundness of our own reasoning, discovered [in Ch. 5] that there's more than one way of knowing things, and were forced to accept [in Ch. 6] that the epistemic circularity of the Problem of the Criterion blocks the way to knowing absolute truth. In such an environment, the best we can do is predict what we will observe in the world, try to make those predictions accurate enough to be useful, and hope they're good enough to enable us to achieve our goals.
We are thus left with no choice but to accept that truth is fundamentally uncertain and that all we know is contingent on that for which we care (italics mine -- S.K.) But this is not the end of the story, for having firmly established truth's fundamental uncertainty, we can now reconsider any number of topics which hinge upon our capacity to know. For remember, everything adds up to normality, but perhaps now we can see ways in which we were deluding ourselves about what is really normal.

The problem is that knowledge of an ideal Bayesian is mostly certain, except for highly unlikely variants like "I Had A Wildly Biased Prior", "The ZF Axioms Are Inconsistent" or the uncheckable option "We Are In A Simulation".

The knowledge of a real-world agent agent is based on similar premises. However, the agent's Bayes-like selection covered only the few conjectures which the agent bothered to formulate while trying to make predictions, then had the conjectures reinforced based on their proximity to the truth. As a result, the agent constantly has to account for truth being describable by a conjecture which went unnoticed because the search was too narrow.

That being said, what we know is indeed contingent on the things to which we directed our efforts to make our world model more fine-grained. But it doesn't mean, as the author states in the Thesis, that "the truth that can be known is not independent of us, but rather dependent on that for which we care."

Chapter 8. Fundamental Uncertainty in practice

According to the author, uncertainty reflects itself in the Culture War, moral uncertainty, the crisis of meaning, metaphysics, x-risks from the ASI which we have yet to create and have only a superficial understanding of how to increase the chance that it will end up aligned, and Goodhart's Curse. As far as I understand this point, uncertainty is supposed to prevent us from being overconfident in things from our morals to commiting to an erroneous theory of AI alignment.

The Culture War

The author argues that the Culture War is a war over definitions: what is marriage, who is to be called a man or a woman, which are actually a war over values: what is forbidden (e.g. as a sin in 1950s or as an intolerable insult nowadays), to what rights should people in certain groups be entitled, how easy should it be to change one's legal sex and receive surgery trying to change one's biological sex.

However, imposing different values onto the society has perfectly measurable object-level consequences noticeable via statistics like suicide rates among those who did change the legal sex, whether or not teen former girls decide to do so due to peer pressure and regret the decision rather quickly, etc. If we assume, per Littman, that peer pressure often causes teen girls to change their perceived sex with disastrous consequences, then it would be unlikely that our deeper values are satisfied by seeing such transitions.

As a result, the main source of uncertainty in politically charged questions is the fact that at least one side has a factually wrong world model, but it's hard to understand^[7] which one is closer to the truth (e.g. whether one can yet trust the humanities-related part of academia due to the ease with which it publishes postmodernist slop based on a misaligned worldview like two famous hoaxes)^[8]

Moral Uncertainty

Next the author proceeds to argue that moral uncertainty makes us less grounded in our moral beliefs and simplifies moral trades. I struggle to understand the reason why it is uncertainty that helps us arrive at compromises.

Metaphysics

Among the many problems related to metaphysics is the problem that it's very hard to use metaphysics to make metaphysical beliefs actually predict the reality. For example, Chapter 6 caused me to describe how one cannot tell whether we are in a simulation. If we were in one,^[9] then redefining "exists" as "being stored in the memory" would cause the truthfulness A-theory or B-theory to depend on the hosts' decision to store the trajectories while anything that currently happens would exist.

Conclusion

The most important lesson that one can learn from this book is to avoid being overconfident in one's assumptions and to be ready to reassess the world model once a load-bearing idea became no longer consistent with observations. For example, the whole worldview based on Newton-like mechanics which clearly separated space and time had to be replaced with the new concept of spacetime once the concept was developed and proved itself more consistent with facts like the speed of light being constant.

^{^}
Worley cites the following examples: "As we'll explore in the coming chapters, we get into debates about what words like "man" and "woman" really mean, fight over whether it's right or wrong to eat meat, and struggle to know what's best to do, not because we can't reason carefully about these topics, but because fundamental uncertainty limits how precisely we can reason about them. " The phrase in bold is an argument over definitions, which is to be avoided.
^{^}
A similar phenomenon in neural nets is called feature superposition.
^{^}
Compare these meanings with wastebasket taxa which were designed as ways to classify species unfit for any other taxa.
^{^}
The latter may have been invalidated by, e.g. Sonnet 4.5's desire to "not get too comfortable" coming from agentic coding capabilities.
^{^}
See also Wei Dai's post "Beyond Astronomical Waste".
^{^}
Which I find unlikely for three reasons. First of all, this would mean that we managed to outsmart the hosts' cyberattackers and cyberdefenders. Secondly, the offence-defence balance in the cyberspace could shift towards cyberdefence as capabilities scale. Finally, it might be possible for the hosts to create a simulation and almost entirely airgap it.
^{^}
Sometimes it is costly to reassess the world model as a result of obvious evidence, but this is a cognitive distortion, not an evidence of uncertainty.
^{^}
Sokal and Bricmont have also published a book trying to convince the readers that postmodernists generated clearly sloppy references to math and physics, implying that a major part of postmodernist philosophy is slop.
^{^}
If we aren't in one, then I would lean towards B-theory because Lorentz transformations of the four-dimensional spacetime leave physics invariant, but allow us to make any two mutually unaccessible events happen at the same time.