Using fancy tools like neural nets, boosting and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a bandaid.

Larry Wasserman

Foreword

For some reason, statistics always seemed somewhat disjoint from the rest of math, more akin to a bunch of tools than a rigorous, carefully-constructed framework. I am here to atone for my foolishness.

This academic term started with a jolt - I quickly realized that I was missing quite a few prerequisites for the Bayesian Statistics course in which I had enrolled, and that good ol' AP Stats wasn't gonna cut it. I threw myself at All of Statistics, doing a good number of exercises, dissolving confusion wherever I could find it, and making sure I could turn each concept around and make sense of it from multiple perspectives.

I then went even further, challenging myself during the bits of downtime throughout my day to do things like explain variance from first principles, starting from the sample space, walking through random variables and expectation - without help.

All of Statistics

1: Introduction

2: Probability

In which sample spaces are formalized.

3: Random Variables

In which random variables are detailed and a multitude of distributions are introduced.

Conjugate Variables

Consider that a random variable $X$ is a function $X : Ω \to R$ . For random variables $X, Y$ , we can then produce conjugate random variables $X Y, X + Y$ , with

(X Y) (ω) = X (ω) Y (ω) (X + Y) (ω) = X (ω) + Y (ω) .

4: Expectation

Evidence Preservation

E (E (Y | X)) = E (Y)

is conservation of expected evidence (thanks to Alex Mennen for making this connection explicit).

Marginal Variance

V (Y) = E V (Y | X) + V E (Y | X)

Why does marginal variance have two terms? Shouldn't the expected conditional variance be sufficient?

This literally plagued my dreams.

Proof (of the variance; I cannot prove it plagued my dreams):

\begin{matrix} V (Y) & = E (Y - E (Y))^{2} = E ((Y - E (Y | X)) + (E (Y | X) - E (Y)))^{2} = E (Y - E (Y | X))^{2} + E (2 (Y - E (Y | X)) (E (Y | X) - E (Y))) + E (E (Y | X) - E (Y))^{2} = E V (Y | X) + 2 E ((Y - E (Y | X)) (E (Y | X) - E (Y))) + V E (Y | X) = E V (Y | X) + 2 E (Y E (Y | X) - Y E (Y) - E (Y | X)^{2} + E (Y | X) E (Y)) + V E (Y | X) = E V (Y | X) + 2 (E (Y E (Y | X)) - E (Y E (Y)) - E (E (Y | X)^{2}) + E (E (Y | X) E (Y))) + V E (Y | X) = E V (Y | X)      sample variance + V E (Y | X)      model variance . \end{matrix}

The middle term is eliminated as the expectations cancel out after repeated applications of conservation of expected evidence. Another way to look at the last two terms is the sum of the expected sample variance an... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,