Probability Distributions
Common distributions, limit theorems and inequalities.
The standard probability distributions, how they relate to one another, and the limit theorems and inequalities that describe averages of random quantities.
Describing a distribution
Random variable – A function that assigns a number to each outcome of an experiment. It is discrete if it takes countably many values, such as counts, and continuous if it takes values in an interval.
Probability mass function (PMF) – For discrete , , with .
Probability density function (PDF) – For continuous , a function whose area gives probabilities, with total area 1:
Cumulative distribution function (CDF) – . It rises from 0 to 1, and in the continuous case.
Mean and variance – The mean is the probability-weighted average, or . The variance measures spread, and is the standard deviation.
Moment generating function (MGF) – . Its -th derivative at is the -th moment , and the MGF of a sum of independent variables is the product of their MGFs.
i.i.d. – Variables are independent and identically distributed when they are independent draws from the same distribution.
Discrete distributions
| Name and meaning | PMF | Mean | Variance |
|---|---|---|---|
| Bernoulli: one trial, success with probability | , | ||
| Binomial: successes in independent trials | |||
| Geometric: trials up to the first success | , | ||
| Poisson: count of rare events at rate |
Continuous distributions
| Name and meaning | Mean | Variance | |
|---|---|---|---|
| Uniform: equally likely on | |||
| Exponential: waiting time at rate | , | ||
| Normal : the bell curve | |||
| Gamma: total wait for events | |||
| Beta: a random probability in |
for integer , and is the constant that makes the Beta PDF integrate to 1.
How they relate
Poisson limit – Binomial approaches Poisson as with held fixed: many trials, each rarely successful.
Poisson process – Events arriving randomly in time at average rate . The number of events in a window of length is Poisson, and the gaps between events are i.i.d. Exponential. Merging independent processes adds their rates; keeping each event independently with probability gives rate .
Memorylessness – Having already waited doesn't change the remaining wait:
The exponential is the only continuous memoryless distribution, and the geometric the only discrete one.
Minimum of exponentials – For independent Exponential, the first to occur satisfies Exponential, and is first with probability .
Sums – i.i.d. Exponential sum to Gamma. Independent Poissons sum to a Poisson with the rates added. Independent normals sum to a normal with means and variances added.
Order statistics – Sort i.i.d. Uniform draws. The -th smallest is Beta with mean ; in particular the maximum has mean .
The normal distribution
Standardization – If then , the standard normal. About 68%, 95% and 99.7% of the mass lies within 1, 2 and 3 standard deviations; the exact 95% two-sided cutoff is .
Multivariate normal – A random vector with mean vector and covariance matrix , whose entry . Linear maps keep it normal:
For jointly normal variables, zero correlation implies independence. This fails for variables that are only individually normal.
Conditioning – Split into blocks with matching blocks of and . Given , is normal with
Lognormal – If , then . This follows from the normal MGF, .
High dimensions – A draw from , where is the identity matrix, has length close to , and two independent draws are nearly orthogonal.
Limit theorems
Let be i.i.d. with mean and variance , and let be the sample mean.
Law of large numbers – The sample mean converges to the true mean: as .
Central limit theorem – Whatever the shape of the original distribution, the standardized sample mean becomes normal:
The error of an average shrinks like , so halving it takes four times the data.
Delta method – A smooth function of the sample mean is also approximately normal, with variance scaled by the slope squared:
Inequalities
These bound probabilities using only a few facts about a distribution.
| Name | When it applies | Statement |
|---|---|---|
| Markov | , | |
| Chebyshev | finite variance, | |
| Jensen | convex (curving upward) | |
| Cauchy–Schwarz | any | |
| Hoeffding | i.i.d. | |
| Union bound | any events |