Probability Distributions

Common distributions, limit theorems and inequalities.

Probability Distributions

The standard probability distributions, how they relate to one another, and the limit theorems and inequalities that describe averages of random quantities.

Describing a distribution

Random variable – A function XX that assigns a number to each outcome of an experiment. It is discrete if it takes countably many values, such as counts, and continuous if it takes values in an interval.

Probability mass function (PMF) – For discrete XX, p(x)=P(X=x)p(x) = P(X = x), with ∑xp(x)=1\sum_x p(x) = 1.

Probability density function (PDF) – For continuous XX, a function f(x)≥0f(x) \ge 0 whose area gives probabilities, with total area 1:

P(a≤X≤b)=∫abf(x) dxP(a \le X \le b) = \int_a^b f(x)\,dx

Cumulative distribution function (CDF) – F(x)=P(X≤x)F(x) = P(X \le x). It rises from 0 to 1, and f=F′f = F' in the continuous case.

Mean and variance – The mean μ=E[X]\mu = \mathbb{E}[X] is the probability-weighted average, ∑xx p(x)\sum_x x\,p(x) or ∫xf(x) dx\int x f(x)\,dx. The variance σ2=E[(X−μ)2]\sigma^2 = \mathbb{E}[(X - \mu)^2] measures spread, and σ\sigma is the standard deviation.

Moment generating function (MGF) – M(t)=E[etX]M(t) = \mathbb{E}[e^{tX}]. Its kk-th derivative at t=0t = 0 is the kk-th moment E[Xk]\mathbb{E}[X^k], and the MGF of a sum of independent variables is the product of their MGFs.

i.i.d. – Variables are independent and identically distributed when they are independent draws from the same distribution.

Discrete distributions

Name and meaningPMF P(X=k)P(X = k)MeanVariance
Bernoulli(p)(p): one trial, success with probability pppk(1−p)1−kp^k(1-p)^{1-k}, k∈{0,1}k \in \{0,1\}ppp(1−p)p(1-p)
Binomial(n,p)(n,p): successes in nn independent trials(nk)pk(1−p)n−k\binom{n}{k}p^k(1-p)^{n-k}npnpnp(1−p)np(1-p)
Geometric(p)(p): trials up to the first success(1−p)k−1p(1-p)^{k-1}p, k≥1k \ge 11/p1/p(1−p)/p2(1-p)/p^2
Poisson(λ)(\lambda): count of rare events at rate λ\lambdae−λλk/k!e^{-\lambda}\lambda^k/k!λ\lambdaλ\lambda

Continuous distributions

Name and meaningPDF f(x)f(x)MeanVariance
Uniform(a,b)(a,b): equally likely on [a,b][a,b]1/(b−a)1/(b-a)(a+b)/2(a+b)/2(b−a)2/12(b-a)^2/12
Exponential(λ)(\lambda): waiting time at rate λ\lambdaλe−λx\lambda e^{-\lambda x}, x≥0x \ge 01/λ1/\lambda1/λ21/\lambda^2
Normal N(μ,σ2)\mathcal{N}(\mu,\sigma^2): the bell curve1σ2πe−(x−μ)2/2σ2\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/2\sigma^2}μ\muσ2\sigma^2
Gamma(k,λ)(k,\lambda): total wait for kk eventsλkxk−1e−λx/Γ(k)\lambda^k x^{k-1}e^{-\lambda x}/\Gamma(k)k/λk/\lambdak/λ2k/\lambda^2
Beta(α,β)(\alpha,\beta): a random probability in [0,1][0,1]xα−1(1−x)β−1/B(α,β)x^{\alpha-1}(1-x)^{\beta-1}/B(\alpha,\beta)αα+β\frac{\alpha}{\alpha+\beta}αβ(α+β)2(α+β+1)\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}

Γ(k)=(k−1)!\Gamma(k) = (k-1)! for integer kk, and B(α,β)=Γ(α)Γ(β)/Γ(α+β)B(\alpha,\beta) = \Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta) is the constant that makes the Beta PDF integrate to 1.

How they relate

Poisson limit – Binomial(n,p)(n,p) approaches Poisson(λ)(\lambda) as n→∞n \to \infty with np=λnp = \lambda held fixed: many trials, each rarely successful.

Poisson process – Events arriving randomly in time at average rate λ\lambda. The number of events in a window of length tt is Poisson(λt)(\lambda t), and the gaps between events are i.i.d. Exponential(λ)(\lambda). Merging independent processes adds their rates; keeping each event independently with probability pp gives rate pλp\lambda.

Memorylessness – Having already waited ss doesn't change the remaining wait:

P(X>s+t∣X>s)=P(X>t)P(X > s + t \mid X > s) = P(X > t)

The exponential is the only continuous memoryless distribution, and the geometric the only discrete one.

Minimum of exponentials – For independent Xi∼X_i \sim Exponential(λi)(\lambda_i), the first to occur satisfies min⁡iXi∼\min_i X_i \sim Exponential(∑iλi)(\sum_i \lambda_i), and XiX_i is first with probability λi/∑jλj\lambda_i / \sum_j \lambda_j.

Sums – kk i.i.d. Exponential(λ)(\lambda) sum to Gamma(k,λ)(k,\lambda). Independent Poissons sum to a Poisson with the rates added. Independent normals sum to a normal with means and variances added.

Order statistics – Sort nn i.i.d. Uniform(0,1)(0,1) draws. The kk-th smallest is Beta(k,n−k+1)(k, n-k+1) with mean k/(n+1)k/(n+1); in particular the maximum has mean n/(n+1)n/(n+1).

The normal distribution

Standardization – If X∼N(μ,σ2)X \sim \mathcal{N}(\mu, \sigma^2) then Z=(X−μ)/σ∼N(0,1)Z = (X - \mu)/\sigma \sim \mathcal{N}(0, 1), the standard normal. About 68%, 95% and 99.7% of the mass lies within 1, 2 and 3 standard deviations; the exact 95% two-sided cutoff is 1.961.96.

Multivariate normal – A random vector X∈RdX \in \mathbb{R}^d with mean vector μ\mu and covariance matrix Σ\Sigma, whose entry Σij=Cov⁡(Xi,Xj)\Sigma_{ij} = \operatorname{Cov}(X_i, X_j). Linear maps keep it normal:

AX+b∼N(Aμ+b,  AΣA⊤)AX + b \sim \mathcal{N}\big(A\mu + b,\; A\Sigma A^\top\big)

For jointly normal variables, zero correlation implies independence. This fails for variables that are only individually normal.

Conditioning – Split XX into blocks X1,X2X_1, X_2 with matching blocks of μ\mu and Σ\Sigma. Given X2=x2X_2 = x_2, X1X_1 is normal with

μ1∣2=μ1+Σ12Σ22−1(x2−μ2),Σ1∣2=Σ11−Σ12Σ22−1Σ21\mu_{1 \mid 2} = \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2), \qquad \Sigma_{1 \mid 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}

Lognormal – If log⁡Y∼N(μ,σ2)\log Y \sim \mathcal{N}(\mu, \sigma^2), then E[Y]=eμ+σ2/2\mathbb{E}[Y] = e^{\mu + \sigma^2/2}. This follows from the normal MGF, E[etX]=eμt+σ2t2/2\mathbb{E}[e^{tX}] = e^{\mu t + \sigma^2 t^2/2}.

High dimensions – A draw from N(0,Id)\mathcal{N}(0, I_d), where IdI_d is the identity matrix, has length close to d\sqrt d, and two independent draws are nearly orthogonal.

Limit theorems

Let X1,…,XnX_1, \dots, X_n be i.i.d. with mean μ\mu and variance σ2\sigma^2, and let Xˉn=1n∑iXi\bar X_n = \frac1n\sum_i X_i be the sample mean.

Law of large numbers – The sample mean converges to the true mean: Xˉn→μ\bar X_n \to \mu as n→∞n \to \infty.

Central limit theorem – Whatever the shape of the original distribution, the standardized sample mean becomes normal:

Xˉn−μσ/n  ⟶  N(0,1)\frac{\bar X_n - \mu}{\sigma/\sqrt n} \;\longrightarrow\; \mathcal{N}(0, 1)

The error of an average shrinks like σ/n\sigma/\sqrt n, so halving it takes four times the data.

Delta method – A smooth function gg of the sample mean is also approximately normal, with variance scaled by the slope squared:

g(Xˉn)≈N(g(μ),  g′(μ)2σ2n)g(\bar X_n) \approx \mathcal{N}\Big(g(\mu),\; \frac{g'(\mu)^2\sigma^2}{n}\Big)

Inequalities

These bound probabilities using only a few facts about a distribution.

NameWhen it appliesStatement
MarkovX≥0X \ge 0, a>0a > 0P(X≥a)≤E[X]/aP(X \ge a) \le \mathbb{E}[X]/a
Chebyshevfinite variance, k>0k > 0P(∣X−μ∣≥kσ)≤1/k2P(\lvert X - \mu\rvert \ge k\sigma) \le 1/k^2
Jensenff convex (curving upward)f(E[X])≤E[f(X)]f(\mathbb{E}[X]) \le \mathbb{E}[f(X)]
Cauchy–Schwarzany X,YX, YE[XY]2≤E[X2] E[Y2]\mathbb{E}[XY]^2 \le \mathbb{E}[X^2]\,\mathbb{E}[Y^2]
Hoeffdingi.i.d. Xi∈[a,b]X_i \in [a,b]P(∣Xˉn−μ∣≥t)≤2e−2nt2/(b−a)2P(\lvert\bar X_n - \mu\rvert \ge t) \le 2e^{-2nt^2/(b-a)^2}
Union boundany events AiA_iP(⋃iAi)≤∑iP(Ai)P(\bigcup_i A_i) \le \sum_i P(A_i)