Fisher information and the Cramér–Rao bound

Score. The score measures how a likelihood responds to a parameter change.

Information. Under the usual regularity conditions, its covariance is Fisher information; the inverse information bounds the covariance of unbiased estimators of the parameter.

Main route. We prove the bound, examine equality and failure of equality, and connect it to Gaussian regression.

Extensions. Geometric curvature supports the main route; the KL interpretation and Gauss–Markov extension are optional.

Model. Suppose that \(X\)$X$ is drawn (e.g., single observation or sequence of i.i.d. data) from \(P_{\theta} \in \mathcal{P} = \{P_\theta: \theta\in\Theta\}\)$P_{\theta} \in \mathcal{P} = \{P_\theta: \theta\in\Theta\}$, where the parameter space \(\Theta \subset \mathbb{R}^k\)$\Theta \subset \mathbb{R}^k$ is an open set.

Density. Also suppose that \(P_\theta\)$P_\theta$ has p.d.f. \(f_\theta\)$f_\theta$ w.r.t. a dominating measure \(\mu\)$\mu$ on the observation space for any \(\theta\in\Theta\)$\theta\in\Theta$, and \(f_\theta\)$f_\theta$ is differentiable in \(\theta\)$\theta$.

Meaning. The regularity condition is an annoying technicality that ensures the exchangeability of integrals with differentials (again, physicists don’t care).

Sufficient conditions. A useful sufficient condition is a parameter-independent support and locally integrable envelopes for the first two parameter derivatives.

Limitation. Smoothness alone, or compact support whose boundary moves with the parameter, does not suffice.

Examples. Regular full exponential families provide important examples.

Nonnegativity and additivity. Fisher information shares two familiar properties with quantities such as discrete entropy and KL divergence: it is nonnegative (positive semidefinite in the matrix case), and it is additive for independent observations under the stated mean-zero-score conditions. The additivity follows because independent centered scores have zero cross covariance.

Parameter scale. An important difference is its dependence on parametrization. Changing the scale of the parameter changes the score and therefore the numerical value of Fisher information. For a scalar change \(\theta=\psi(\eta)\)$\theta=\psi(\eta)$, \[I_\eta(\eta)=[\psi'(\eta)]^2 I_\theta(\psi(\eta)).\]$$I_\eta(\eta)=[\psi'(\eta)]^2 I_\theta(\psi(\eta)).$$ See Remark 1 for the matrix version and why the variance bound for a fixed target is unchanged.

Change of observations. For comparison, discrete entropy is unchanged by a one-to-one relabeling of outcomes, and KL divergence is unchanged when the same invertible transformation is applied to both distributions.

Two different transformations. These are changes of the observations, whereas the formula above changes the parameter. Fisher information for a fixed parameter is also unchanged by an invertible, parameter-independent transformation of the observations. The entropy comparison here concerns discrete entropy; differential entropy for continuous variables is not generally invariant. See the change-of-variables discussion and Fisher-information notes.

Positive definiteness. Recall that a symmetric matrix \(A\)$A$ is positive definite if \(u^\top A u > 0\)$u^\top A u > 0$ for all nonzero vectors \(u\)$u$, and it is positive semidefinite if \(u^\top A u \ge 0\)$u^\top A u \ge 0$ for all vectors \(u\)$u$.

Two equivalent notations. For symmetric matrices \(A\)$A$ and \(B\)$B$ of the same size, both \(A\ge B\)$A\ge B$ and \(A\succeq B\)$A\succeq B$ will mean that \(A-B\)$A-B$ is positive semidefinite, equivalently \(u^\top(A-B)u\ge0\)$u^\top(A-B)u\ge0$ for every vector \(u\)$u$.

Meaning of the inequality. This is the matrix order, not an entrywise comparison. We use \(\succeq\)$\succeq$ in Theorem 1 to make that meaning explicit.

Target and dimensions. Let \(\nu = g(\theta)\)$\nu = g(\theta)$ be the quantity of interest where \(g:\R^d\to\R^k\)$g:\mathbb{R}^d\to\mathbb{R}^k$ is differentiable; in this section \(\theta\in\R^d\)$\theta\in\mathbb{R}^d$ and \(Dg(\theta)\)$Dg(\theta)$ is the \(k\times d\)$k\times d$ Jacobian.

Moment condition. Assume \(T\)$T$ has finite covariance.

Unbiased estimator. Suppose that \(X\)$X$ is drawn from a distribution in \(\{P_{\theta}: \theta \in \Theta\}\)$\{P_{\theta}: \theta \in \Theta\}$ and \(T(X)\)$T(X)$ is an unbiased estimator of \(\nu\)$\nu$.

Density. Let \(f_{\theta}\)$f_{\theta}$ be the density function with respect to a reference measure \(\mu\)$\mu$.

Lower bound. Fisher information provides a lower bound on variance (and thus MSE) of unbiased estimators for a parametric family.

When it proves optimality. It may not always be achievable, but if we can find an estimator that achieves this lower bound, then we certify the optimality of this estimator in the class of unbiased estimators (namely certifying UMVUE).

Reference. See wiki for the multivariate case.

Design variables. Here \(Z_i\)$Z_i$ can be random or nonrandom.

Working assumption. For simplicity, we will treat \(Z_i\)$Z_i$ as fixed (If \(Z_i\)$Z_i$ is random and independent of the other random variables, we condition on \(Z_i\)$Z_i$ and conclusions remain the same.)

Matrix form. We can also write the matrix form. \[\begin{aligned} \underset{n\times 1}{X} = \underset{n\times p}{Z} \cdot \underset{p\times 1}{\beta} + \underset{n\times 1}{\epsilon}. \end{aligned}\]$$\begin{aligned} \underset{n\times 1}{X} = \underset{n\times p}{Z} \cdot \underset{p\times 1}{\beta} + \underset{n\times 1}{\epsilon}. \end{aligned}$$

Intercept. Note: Equation 4 subsumes the case with an intercept term: \(X_i=\beta_0+\beta^\top Z_i+\epsilon_i\)$X_i=\beta_0+\beta^\top Z_i+\epsilon_i$. In the matrix form, we include a column of ones in \(Z\)$Z$ for the intercept.

Model space and fit. The vectors \(Zb\)$Zb$, as \(b\)$b$ varies, fill the column space \(\mathcal C(Z)\subseteq\R^n\)$\mathcal C(Z)\subseteq\mathbb{R}^n$. Least squares finds the point in this space closest to the observation vector \(X\)$X$. Write \(P_Z\)$P_Z$ for the orthogonal projection onto \(\mathcal C(Z)\)$\mathcal C(Z)$. The fitted vector and residual vector are \[\widehat X=Z\hat\beta=P_ZX,\qquad r=X-\widehat X=(I_n-P_Z)X.\]$$\widehat X=Z\hat\beta=P_ZX,\qquad r=X-\widehat X=(I_n-P_Z)X.$$

Orthogonality. The residual is perpendicular to every column of \(Z\)$Z$: \(Z^\top r=0\)$Z^\top r=0$. For full column rank, \(P_Z=Z(Z^\top Z)^{-1}Z^\top\)$P_Z=Z(Z^\top Z)^{-1}Z^\top$; the geometric projection is still unique when the coefficient vector is not.

Pythagoras. For any coefficient \(b\)$b$, the two terms in \(X-Zb=r+(\widehat X-Zb)\)$X-Zb=r+(\widehat X-Zb)$ are orthogonal. Pythagoras gives \[\|X-Zb\|^2=\|r\|^2+\|\widehat X-Zb\|^2\ge\|r\|^2.\]$$\|X-Zb\|^2=\|r\|^2+\|\widehat X-Zb\|^2\ge\|r\|^2.$$

Example. For a numerical illustration, take \(Z=(1,1)^\top\)$Z=(1,1)^\top$ and \(X=(1,3)^\top\)$X=(1,3)^\top$. Then \(\hat\beta=(X_1+X_2)/2=2\)$\hat\beta=(X_1+X_2)/2=2$, and the squared error is \(2+2(b-2)^2\)$2+2(b-2)^2$: its minimum is \(2\)$2$, compared with \(4\)$4$ at \(b=1\)$b=1$.

Explained and unexplained parts. The fit captures the component along the model space; the residual is what remains in the perpendicular directions.

More pictures. For another visual explanation, see Strang’s MIT lecture on projections onto subspaces.

Gaussian assumptions. The geometry does not require Gaussian noise. To anticipate Theorem 3, now suppose \(X=Z\beta+\epsilon\)$X=Z\beta+\epsilon$, with fixed full-column-rank \(Z\)$Z$, \(n>p\)$n>p$, and \(\epsilon\sim N(0,\sigma^2 I_n)\)$\epsilon\sim N(0,\sigma^2 I_n)$.

Gaussian transformations. Recall from Lecture 1: affine transformations of Gaussian vectors that a linear transformation of a Gaussian vector is still Gaussian. In particular, \(P_ZX\)$P_ZX$ and \((I_n-P_Z)X\)$(I_n-P_Z)X$ are jointly Gaussian, and their cross-covariance is \[\operatorname{Cov}(P_ZX,(I_n-P_Z)X) =\sigma^2P_Z(I_n-P_Z)=0.\]$$\operatorname{Cov}(P_ZX,(I_n-P_Z)X) =\sigma^2P_Z(I_n-P_Z)=0.$$

Why independence follows. Joint Gaussianity turns this zero cross-covariance into independence. Orthogonality alone would not give independence for arbitrary noise.

Choose coordinates. Choose orthonormal columns \(Q_1\)$Q_1$ spanning \(\mathcal C(Z)\)$\mathcal C(Z)$ and complete them with \(Q_2\)$Q_2$ to an orthogonal matrix \(Q=(Q_1,Q_2)\)$Q=(Q_1,Q_2)$.

Separate the two groups. For \(G=\epsilon/\sigma\)$G=\epsilon/\sigma$, the vector \(Q^\top G\sim N(0,I_n)\)$Q^\top G\sim N(0,I_n)$ has independent standard Normal coordinates. The fitted vector depends only on \(Q_1^\top G\)$Q_1^\top G$ (and the fixed mean), while the residual depends only on \(Q_2^\top G\)$Q_2^\top G$.

Functions and residual length. Thus any function of the fit, including \(\ell^\top\hat\beta\)$\ell^\top\hat\beta$, is independent of any function of the residual, including \(\hat\sigma^2\)$\hat\sigma^2$. Moreover, \[\frac{\|r\|^2}{\sigma^2} =\|Q_2^\top G\|^2 =\sum_{j=p+1}^{n}(Q^\top G)_j^2\sim\chi^2_{n-p}.\]$$\frac{\|r\|^2}{\sigma^2} =\|Q_2^\top G\|^2 =\sum_{j=p+1}^{n}(Q^\top G)_j^2\sim\chi^2_{n-p}.$$

Connection to the theorem. This explains both the independence and the \(n-p\)$n-p$ degrees of freedom in Theorem 3.

Rank and identifiability. If \(Z\)$Z$ is not of full rank, \(I(\theta)\)$I(\theta)$ may not be invertible. Indeed, in the null space Ker(\(Z\)$Z$), two distributions \(P_\theta\)$P_\theta$, \(P_{\theta'}\)$P_{\theta'}$ are not distinguishable.

Attaining the bound. It is clear that LSE is the MLE for estimating \(\beta\)$\beta$ under our assumption, and it attains the Cramér–Rao lower bound; the corresponding question for \(\sigma^2\)$\sigma^2$ is the linked exercise below.