Statistical Inference

Estimation, Bayesian updating and hypothesis testing.

Statistical Inference

How to learn about an unknown quantity from data: building estimators, judging them, Bayesian updating, confidence intervals and hypothesis tests.

Setup

Random sample – Data X1,…,XnX_1, \dots, X_n that are i.i.d. (independent and identically distributed) draws from some distribution. That distribution depends on an unknown parameter θ\theta, such as a mean or a probability.

Sample mean and variance – The usual summaries of a sample:

Xˉ=1n∑i=1nXi,s2=1n−1∑i=1n(Xi−Xˉ)2\bar X = \frac1n\sum_{i=1}^n X_i, \qquad s^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2

Dividing by n−1n - 1 instead of nn corrects for Xˉ\bar X being fitted to the same data.

Estimator – Any function θ^\hat\theta of the data used to guess θ\theta. Because the data is random, θ^\hat\theta is itself a random variable with its own mean and spread.

Judging an estimator

Bias – How far the estimator is off on average. It is unbiased if the bias is zero.

Bias⁡(θ^)=E[θ^]−θ\operatorname{Bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta

Mean squared error (MSE) – Average squared distance from the truth, which splits into bias and variance:

MSE⁡(θ^)=E[(θ^−θ)2]=Bias⁡(θ^)2+Var⁡(θ^)\operatorname{MSE}(\hat\theta) = \mathbb{E}\big[(\hat\theta - \theta)^2\big] = \operatorname{Bias}(\hat\theta)^2 + \operatorname{Var}(\hat\theta)

Consistency – θ^\hat\theta is consistent if it gets arbitrarily close to θ\theta, with probability approaching 1, as n→∞n \to \infty.

Standard error – The standard deviation of an estimator. For the sample mean it is σ/n\sigma/\sqrt n, where σ\sigma is the standard deviation of a single observation.

Maximum likelihood

Likelihood – The probability (or density) of the observed data viewed as a function of θ\theta: L(θ)=∏ip(xi∣θ)L(\theta) = \prod_i p(x_i \mid \theta). Working with its log turns the product into a sum.

Maximum likelihood estimator (MLE) – The parameter value that makes the observed data most probable:

θ^MLE=arg⁡max⁡θ∑i=1nlog⁡p(xi∣θ)\hat\theta_{\text{MLE}} = \arg\max_\theta \sum_{i=1}^n \log p(x_i \mid \theta)

Example: for coin flips with hh heads out of nn, the log-likelihood is hlog⁡p+(n−h)log⁡(1−p)h\log p + (n-h)\log(1-p). Setting its derivative to zero gives p^=h/n\hat p = h/n.

ModelMLE
Bernoulli(p)(p)p^=xˉ\hat p = \bar x
Poisson(λ)(\lambda)λ^=xˉ\hat\lambda = \bar x
Exponential(λ)(\lambda)λ^=1/xˉ\hat\lambda = 1/\bar x
Normal(μ,σ2)(\mu, \sigma^2)μ^=xˉ\hat\mu = \bar x, σ^2=1n∑(xi−xˉ)2\hat\sigma^2 = \frac1n\sum(x_i - \bar x)^2 (biased low)
Uniform(0,θ)(0, \theta)θ^=max⁡ixi\hat\theta = \max_i x_i (biased low; n+1nmax⁡ixi\frac{n+1}{n}\max_i x_i is unbiased)

Fisher information – How sharply the log-likelihood curves around the true value, per observation. More curvature means the data pins θ\theta down better.

I(θ)=−E[∂2∂θ2log⁡p(X∣θ)]I(\theta) = -\mathbb{E}\Big[\frac{\partial^2}{\partial\theta^2}\log p(X \mid \theta)\Big]

Properties of the MLE – For large nn it is approximately normal around the truth, θ^≈N(θ,1/(nI(θ)))\hat\theta \approx \mathcal{N}\big(\theta, 1/(nI(\theta))\big). It is also invariant: the MLE of g(θ)g(\theta) is g(θ^)g(\hat\theta).

Cramér–Rao bound – No unbiased estimator can have variance below 1/(nI(θ))1/(nI(\theta)), so the MLE is asymptotically the best possible.

Bayesian inference

Prior and posterior – Treat θ\theta as random. A prior p(θ)p(\theta) encodes belief before the data; Bayes' rule updates it to the posterior:

p(θ∣x)∝p(x∣θ) p(θ)p(\theta \mid x) \propto p(x \mid \theta)\,p(\theta)

Maximum a posteriori (MAP) estimate – The peak of the posterior, which is the MLE plus a penalty from the prior:

θ^MAP=arg⁡max⁡θ[log⁡p(x∣θ)+log⁡p(θ)]\hat\theta_{\text{MAP}} = \arg\max_\theta \big[\log p(x \mid \theta) + \log p(\theta)\big]

A Gaussian prior on regression weights gives an L2 (ridge) penalty, and a Laplace prior gives an L1 (lasso) penalty.

Conjugate prior – A prior whose posterior stays in the same family, so updating just changes its parameters.

Data modelPriorPosterior after the data
Bernoulli, hh successes and tt failuresBeta(α,β)(\alpha, \beta)Beta(α+h,β+t)(\alpha + h, \beta + t)
Poisson, nn countsGamma(α,β)(\alpha, \beta)Gamma(α+∑xi,β+n)(\alpha + \sum x_i, \beta + n)
Normal with known σ2\sigma^2Normal(μ0,τ2)(\mu_0, \tau^2)Normal, with mean a precision-weighted average of μ0\mu_0 and xˉ\bar x

With a uniform Beta(1,1)(1,1) prior, the posterior mean of a coin's bias after hh heads in nn flips is (h+1)/(n+2)(h+1)/(n+2) (Laplace's rule of succession).

Confidence intervals

Confidence interval – A range computed from the data that contains the true θ\theta in a fraction 1−α1 - \alpha of repeated experiments, e.g. 95% for α=0.05\alpha = 0.05. It is a statement about the procedure, not a probability that θ\theta lies in this particular interval.

Critical value – zα/2z_{\alpha/2} is the point with probability α/2\alpha/2 above it under the standard normal, so z0.025=1.96z_{0.025} = 1.96. tα/2, n−1t_{\alpha/2,\,n-1} is the same for Student's tt-distribution with n−1n - 1 degrees of freedom, which has heavier tails and is used when σ\sigma is estimated from a small sample.

SituationInterval for the mean μ\mu
σ\sigma known, or nn largexˉ±zα/2 σ/n\bar x \pm z_{\alpha/2}\,\sigma/\sqrt n (use ss for unknown σ\sigma)
Normal data, σ\sigma unknown, nn smallxˉ±tα/2, n−1 s/n\bar x \pm t_{\alpha/2,\,n-1}\,s/\sqrt n
Proportion pp, nn largep^±zα/2p^(1−p^)/n\hat p \pm z_{\alpha/2}\sqrt{\hat p(1-\hat p)/n}

Bootstrap – Resample the data with replacement many times, recompute the statistic each time, and use the spread of those values as its standard error or interval. It needs no formula for the statistic's distribution.

Hypothesis testing

Null and alternative – The null hypothesis H0H_0 is the default claim (e.g. "no difference"); the alternative H1H_1 is what we look for evidence of. A test statistic TT summarizes the data, and H0H_0 is rejected when TT is extreme.

p-value – The probability, assuming H0H_0 is true, of a test statistic at least as extreme as the one observed. Reject H0H_0 when it is below the chosen level α\alpha.

A p-value is not the probability that H0H_0 is true.

Errors and power – A type I error rejects a true H0H_0 (false positive) and happens with probability α\alpha. A type II error keeps a false H0H_0 (false negative) with probability β\beta. Power =1−β= 1 - \beta is the chance of detecting a real effect.

Sample size – To detect a difference δ\delta between two group means, each with standard deviation σ\sigma, each group needs about

n≈2(zα/2+zβ)2σ2δ2n \approx \frac{2(z_{\alpha/2} + z_\beta)^2\sigma^2}{\delta^2}

which is roughly 16σ2/δ216\sigma^2/\delta^2 for α=0.05\alpha = 0.05 and 80% power.

Common tests – A zz-test or tt-test compares means. A chi-squared (χ2\chi^2) test compares observed and expected counts, or checks independence in a table. An FF-test compares nested regression models. A permutation test shuffles group labels to build the null distribution without assumptions.

Multiple comparisons – Running mm tests at level α\alpha produces false positives by chance. The Bonferroni correction tests each at α/m\alpha/m. The Benjamini–Hochberg procedure instead controls the false discovery rate, the expected fraction of rejections that are false.

Bias–variance tradeoff

Decomposition – For a model f^\hat f fitted to random training data and a new point with noise variance σnoise2\sigma^2_{\text{noise}}, the expected squared prediction error is

E[(y−f^(x))2]=Bias⁡[f^(x)]2+Var⁡[f^(x)]+σnoise2\mathbb{E}\big[(y - \hat f(x))^2\big] = \operatorname{Bias}[\hat f(x)]^2 + \operatorname{Var}[\hat f(x)] + \sigma^2_{\text{noise}}

More flexible models lower bias but raise variance. Regularization, more data and averaging several models all reduce variance.