Outside of LessWrong, "humility" usually refers to "a modest or low view of one's own importance". In common parlance, to be humble is to be meek, deferential, submissive, or unpretentious, "not arrogant or prideful". Thus, in ordinary English "humility" and "modesty" have pretty similar connotations.
On LessWrong, Eliezer Yudkowsky has proposed that we instead draw a sharp distinction between two kinds of "humility" — social modesty, versus "scientific humility".
In The Proper Use of Humility (2006), Yudkowsky writes:
You suggest studying harder, and the student replies: “No, it wouldn’t work for me; I’m not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot.”
This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process.
If you ask someone to “be more humble,” by default they’ll associate the words to social modesty—which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You’d still double-check your calculations if you were wise.
On LW, then, we tend to follow the convention of using "humility" as a term of art for an important part of reasoning: combating overconfidence, recognizing and improving on your weaknesses, anticipating and preparing for likely errors you'll make, etc.
In contrast, "modesty" here refers to the bad habit of letting your behavior and epistemics be ruled by not wanting to look arrogant or conceited. Yudkowsky argues in Inadequate Equilibria (2017) that psychological impulses like "status regulation and anxious underconfidence" have caused many people in the effective altruism and rationality communities to adopt a "modest epistemology" that involves rationalizing various false world-models and invalid reasoning heuristics.
LW tries to create a social environment where social reward and punishment is generally less salient, and where (to the extent it persists) it incentivizes honesty and truth-seeking as much as possible. LW doesn't always succeed in this goal, but this is nonetheless the goal.
The most commonly cited explanation of scientific/epistemic humility on LW is found in Yudkowsky's "Twelve Virtues of Rationality" (2006):
The eighth virtue is humility.
To be humble is to take specific actions in anticipation of your own errors.
To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.
Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans.
Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their own abilities. But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse. If you compare yourself to others you will not see the biases that all humans share. To be human is to make ten thousand errors. No one in this world achieves perfection.
While humility is based on the general idea that you are fallible (and should try to be calibrated and realistic about this), modest epistemology makes stronger claims such as:
In contrast, Yudkowsky has argued:
I try to be careful to distinguish the virtue of avoiding overconfidence, which I sometimes call “humility,” from the phenomenon I’m calling “modest epistemology.” But even so, when overconfidence is such a terrible scourge according to the cognitive bias literature, can it ever be wise to caution people against underconfidence?
Yes. First of all, overcompensation after being warned about a cognitive bias is also a recognized problem in the literature; and the literature on that talks about how bad people often are at determining whether they’re undercorrecting or overcorrecting. Second, my own experience has been that while, yes, commenters on the Internet are often overconfident, it’s very different when I’m talking to people in person. My more recent experience seems more like 90% telling people to be less underconfident, to reach higher, to be more ambitious, to test themselves, and maybe 10% cautioning people against overconfidence. And yes, this ratio applies to men as well as women and nonbinary people, and to people considered high-status as well as people considered low-status.
The Sin of Underconfidence (2009) argues that underconfidence is one of the "three great besetting sins of rationalists" (the others being motivated reasoning / motivated skepticism and "cleverness").
In Taboo "Outside View" (2021), Daniel Kokotajlo notes that the original meaning of "outside view" (reference class forecasting) has become eroded as EAs have begun using "outside view" to refer to everything from reasoning by analogy, to trend extrapolation, to foxy aggregation, to bias correction, to "deference to wisdom of the many", to "anti-weirdness heuristics", to priors, etc.
Additionally, proponents of outside-viewing often behave as though there is a single obvious reference class to use -- "the outside view", as opposed to "an outside view" -- and tend to neglect the role of detailed model-building in helping us figure out which reference classes are relevant.
The lesson of this isn't "it's bad to ever use reference class forecasting, trend extrapolation, etc.", but rather that these tools are part and parcel of building good world-models and deriving good predictions from them, rather than being a robust replacement for world-modeling.
Likewise, the lesson isn't "it's bad to ever worry about overconfidence", but rather that overconfidence and underconfidence are both problems, neither is a priori worse than the other, and fixing them requires doing a lot of legwork and model-building about your own capabilities -- again, there isn't a royal road to 'getting the right answer without having to figure things out'.