I think this post is underapprecatiated on LW. It could be better written but I guess that is partly due to the topic and maybe another audience.
Favorite quote:
You see, if I were somehow convinced that an all-powerful being existed, my ability to assess that being’s honesty would drop to 0. No matter what its other qualities were, it could effortlessly appear to be absolutely anything it wanted, with perfectly persuasive evidence on any possible scale. Upon seeing such evidence, I would greatly increase my probability that the being was very powerful, also increase the probability of it being all-powerful (not sure I could tell the difference, though), but remain totally agnostic as to whether or not it was an honest being.
I think your ending considerations are correct: for an evidence to be a good evidence, it needs to be something that happens with high probability in the intended model and with low probability in other models. Spiritual experiences or miracles are often better explained with naturalistic assumptions, so they are particularly bad evidence.
On the other hand, it is perfectly Bayesian to believe in something with 100% certainty and not be moved by any argument. Here on LW it's considered bad form exactly because of this, since we believe in an imperfect brain and thus we shouldn't limit in any way what evidence presents us with.
You can with some evidence that happens only in your model and nowhere else:
P(A|E) = P(E|A)P(A) / (P(E|A)P(A) + P(E|-A)P(-A))
If P(E|A) = 1 and P(E|-A) = 0 then
P(A|E) = P(A) / P(A) = 1 (if A is not false)
Sure, this moves the problem from the evidence (P(A)) to the model (P(E|A)), but since the model is mental, you can set it anyway you want.
So, it would have to be an observation which could not possible occur except one way?
What possible piece of evidence couldn't have, at the very least, the alternative cause of, say some super meta-God from beyond logic itself that can cause any possible observation?
Y'know. Plus a trillion other equally plausible explanations
Yes, I feel your pain. Indeed, it is unreasonable to put some evidence as pertaining exclusively to a model. But still, you believe what you want to believe, and you can start from below certainty and go up to one.
I suspect also that this is what happens when people are converted by a supposed miracle and some lack of imagination.
You can't get to it from somewhere else, if you didn't start there. But he was saying that if you start there, you are following Bayes' rule by refusing to change it afterwards, and that this is a good reason not to start there.
Ugh. "all-powerful" is not defined, nor even reduced to "sufficiently powerful to X". Also, why focus on "honest" for such a crazy thing? There's no evidence possible for ANY trait of a sufficiently-powerful entity. Once it can change your perception, memory, and processes, you're pretty much lost, right?
Also, mind-killing topic. If I could downvote, I would.
I think you see the question of God very clearly.
But maybe more clearly than you see God-believers. Would this be more effective for them if I had reduced "all-powerful?" Can you see why focusing on honesty in particular might be especially effective for them?
You're not wrong philosophically. It's more about how to approach the issue.