Why this part matters
When the number of parameters is comparable to the sample size, fixed-dimension intuition can fail. We first develop probability bounds that retain their sample-size and dimension dependence: subgaussian and sub-exponential tails, random-vector norms, and operator norms of random matrices.
These tools lead to geometric and statistical consequences. Gaussian vectors concentrate near a sphere; sample-covariance noise can remain substantial; PCA can have a signal threshold; and regression risk changes around interpolation. Read the assumptions together with each conclusion. The optional community-detection material shows another use of the same spectral ideas.
Learning objectives
Apply Hoeffding- and Bernstein-type bounds (Course outcome 4).
Control vector and matrix norms with the stated tail assumptions (Course outcome 4).
Relate covariance error to PCA behavior (Course outcome 5).
Compare fixed- and growing-dimension statistical regimes (Course outcome 5).
Roadmap
| Lecture | Purpose |
| 14. Concentration inequalities | Obtain finite-sample tail bounds for sums. |
| 15. Random vectors in high dimensions | Turn scalar bounds into random-vector geometry. |
| 16. Norms of random matrices | Control random matrices and covariance estimation. |
| 17. High-dimensional statistical phenomena | Study PCA and regression when dimension grows with sample size. |
The main sequence is Lectures \(14\to15\to16\to17\). Bernstein from Lecture 14 also feeds directly into the matrix bounds in Lecture 16.
What you need from earlier lectures
Where this leads
Optional shrinkage and sparsity ask how to design estimators for high-dimensional problems. The spectral tools also provide a foundation for reading modern statistical research. Continue to Part VI (optional).
Readings
Use the lecture notes together with the course texts and original notes. Exact textbook sections and proof references are linked in the lectures.
Part checkpoint
Try each question before opening its answer. This self-check reviews the linked material and adds no exam requirements; the lecture priorities and exceptions still apply.
Why does Bernstein have two deviation regimes?
Its exponent involves the minimum of a quadratic variance term and a linear tail-scale term. The quadratic term controls moderate deviations and the linear term controls sufficiently large deviations. Return to the explanation.
Why do union bounds cost a logarithm of the number of events?
Adding exponentially small failure probabilities over many events introduces their number as a prefactor. Solving for a prescribed total failure probability brings that factor into the exponent through its logarithm. Return to the explanation.
When can covariance noise fail to vanish?
If dimension is proportional to sample size, an operator-norm error need not tend to zero. The dimension/sample-size ratio is part of the guarantee, not an ignorable constant. Return to the explanation.
Does positive limiting PCA overlap mean consistency?
No. Positive overlap means the estimated direction retains some signal. Consistency requires the squared overlap to approach one; the high-dimensional spiked model can have a limit strictly between zero and one. Return to the explanation.
What changes at interpolation in high-dimensional regression?
The rank and conditioning of the design change how noise is amplified. Beyond interpolation, minimum norm selects among exact fits, introducing a geometric bias as well as variance. The risk depends on the stated design and signal assumptions. Return to the explanation.
You should now be able to carry out the learning objectives above and explain where each result needs its assumptions.