Statistical Inference
Estimation, Bayesian updating and hypothesis testing.
How to learn about an unknown quantity from data: building estimators, judging them, Bayesian updating, confidence intervals and hypothesis tests.
Setup
Random sample – Data that are i.i.d. (independent and identically distributed) draws from some distribution. That distribution depends on an unknown parameter , such as a mean or a probability.
Sample mean and variance – The usual summaries of a sample:
Dividing by instead of corrects for being fitted to the same data.
Estimator – Any function of the data used to guess . Because the data is random, is itself a random variable with its own mean and spread.
Judging an estimator
Bias – How far the estimator is off on average. It is unbiased if the bias is zero.
Mean squared error (MSE) – Average squared distance from the truth, which splits into bias and variance:
Consistency – is consistent if it gets arbitrarily close to , with probability approaching 1, as .
Standard error – The standard deviation of an estimator. For the sample mean it is , where is the standard deviation of a single observation.
Maximum likelihood
Likelihood – The probability (or density) of the observed data viewed as a function of : . Working with its log turns the product into a sum.
Maximum likelihood estimator (MLE) – The parameter value that makes the observed data most probable:
Example: for coin flips with heads out of , the log-likelihood is . Setting its derivative to zero gives .
| Model | MLE |
|---|---|
| Bernoulli | |
| Poisson | |
| Exponential | |
| Normal | , (biased low) |
| Uniform | (biased low; is unbiased) |
Fisher information – How sharply the log-likelihood curves around the true value, per observation. More curvature means the data pins down better.
Properties of the MLE – For large it is approximately normal around the truth, . It is also invariant: the MLE of is .
Cramér–Rao bound – No unbiased estimator can have variance below , so the MLE is asymptotically the best possible.
Bayesian inference
Prior and posterior – Treat as random. A prior encodes belief before the data; Bayes' rule updates it to the posterior:
Maximum a posteriori (MAP) estimate – The peak of the posterior, which is the MLE plus a penalty from the prior:
A Gaussian prior on regression weights gives an L2 (ridge) penalty, and a Laplace prior gives an L1 (lasso) penalty.
Conjugate prior – A prior whose posterior stays in the same family, so updating just changes its parameters.
| Data model | Prior | Posterior after the data |
|---|---|---|
| Bernoulli, successes and failures | Beta | Beta |
| Poisson, counts | Gamma | Gamma |
| Normal with known | Normal | Normal, with mean a precision-weighted average of and |
With a uniform Beta prior, the posterior mean of a coin's bias after heads in flips is (Laplace's rule of succession).
Confidence intervals
Confidence interval – A range computed from the data that contains the true in a fraction of repeated experiments, e.g. 95% for . It is a statement about the procedure, not a probability that lies in this particular interval.
Critical value – is the point with probability above it under the standard normal, so . is the same for Student's -distribution with degrees of freedom, which has heavier tails and is used when is estimated from a small sample.
| Situation | Interval for the mean |
|---|---|
| known, or large | (use for unknown ) |
| Normal data, unknown, small | |
| Proportion , large |
Bootstrap – Resample the data with replacement many times, recompute the statistic each time, and use the spread of those values as its standard error or interval. It needs no formula for the statistic's distribution.
Hypothesis testing
Null and alternative – The null hypothesis is the default claim (e.g. "no difference"); the alternative is what we look for evidence of. A test statistic summarizes the data, and is rejected when is extreme.
p-value – The probability, assuming is true, of a test statistic at least as extreme as the one observed. Reject when it is below the chosen level .
A p-value is not the probability that is true.
Errors and power – A type I error rejects a true (false positive) and happens with probability . A type II error keeps a false (false negative) with probability . Power is the chance of detecting a real effect.
Sample size – To detect a difference between two group means, each with standard deviation , each group needs about
which is roughly for and 80% power.
Common tests – A -test or -test compares means. A chi-squared () test compares observed and expected counts, or checks independence in a table. An -test compares nested regression models. A permutation test shuffles group labels to build the null distribution without assumptions.
Multiple comparisons – Running tests at level produces false positives by chance. The Bonferroni correction tests each at . The Benjamini–Hochberg procedure instead controls the false discovery rate, the expected fraction of rejections that are false.
Bias–variance tradeoff
Decomposition – For a model fitted to random training data and a new point with noise variance , the expected squared prediction error is
More flexible models lower bias but raise variance. Regularization, more data and averaging several models all reduce variance.