16 Gaussian vectors
16.1 Univariate Gaussian distribution
The standard Gaussian density is denoted by \(\phi\): \[ \phi(x) = \frac{\mathrm{e}^{- \frac{x^2}{2} }}{\sqrt{2 \pi}} \, . \] The corresponding cumulative distribution function is denoted by \(\Phi\). The survival function \(1- \Phi\) is denoted by \(\overline{\Phi}\). \[ \Phi(x) = \int_{- \infty}^x \phi(u) \mathrm{d}u \, . \] Let is denote by \(\mathcal{N} (0, 1)\) (expectation \(0\), variance \(1\)) the standard Gaussian probability distribution, that is the probability distribution defined by \(\phi\).
Any affine transform of a standard Gaussian random variable is distributed according to a univariate Gaussian distribution. If \(X \sim \mathcal{N} (0, 1)\) then \(\sigma X + \mu \sim \mathcal{N} \left( \mu, \sigma^2 \right)\) with density \(\frac{1}{\sigma}\phi\left(\frac{\cdot- \mu}{\sigma}\right)\), cumulative distribution function \(\Phi\left(\frac{\cdot - \mu}{\sigma}\right)\).
The standard Gaussian distribution is characterized by the next identity.
Proof. The proof relies on integration by parts. First note that replacing \(g\) by \(g - g(0)\) changes neither \(g'\), nor \(\mathbb{E}[Xg(X)]\). We may assume that \(g(0)=0\).
\[\begin{align*} \mathbb{E}[Xg(X)] & = \int_{\mathbb{R}} xg(x) \phi(x) \mathrm{d}x \\ & = \int_0^\infty xg(x) \phi(x) \mathrm{d}x + \int_{-\infty}^0 xg(x) \phi(x) \mathrm{d}x \\ & = \int_0^\infty x \int_0^\infty g'(y) \mathbb{I}_{y\leq x}\mathrm{d}y \phi(x) \mathrm{d}x -\int^0_{-\infty} x \int^0_{-\infty} g'(y) \mathbb{I}_{y\geq x}\mathrm{d}y \phi(x) \mathrm{d}x\\ & = \int_0^\infty g'(y) \int_0^\infty \mathbb{I}_{y\leq x} x\phi(x)\mathrm{d}x \mathrm{d}y - \int_{-\infty}^0 g'(y) \int^0_{-\infty} x \phi(x)\mathbb{I}_{y\geq x}\mathrm{d}x \mathrm{d}y \\ & = \int_0^\infty g'(y) \int_y^\infty x\phi(x)\mathrm{d}x \mathrm{d}y - \int_{-\infty}^0 g'(y) \int^y_{-\infty} x \phi(x)\mathrm{d}x \mathrm{d}y \\ & = \int_0^\infty g'(y) \phi(y) \mathrm{d}y - \int_{-\infty}^0 - g'(y) \phi(y)\mathrm{d}y \\ & = \int_{-\infty}^\infty g'(y) \phi(y) \mathrm{d}y \, . \end{align*}\]
The last inequality is justified by Tonelli-Fubini’s Theorem. Then, we rely on \(\phi'(x)=-x \phi(x)\).
The characteristic function is a very efficient tool when handling Gaussian distributions.
Proof. It is enough to check the proposition for \(\mathcal{N}(0,1)\). As \(\phi\) is even,
\[\begin{eqnarray*} \widehat{\Phi}(t) &= & \int_{-\infty}^{\infty} \mathrm{e}^{\imath t x} \frac{\mathrm{e}^{- \frac{x^2}{2}}}{\sqrt{2 \pi}} \mathrm{d} x \\ & = & \int_{-\infty}^{\infty} \cos(tx) \frac{\mathrm{e}^{- \frac{x^2}{2}}}{\sqrt{2 \pi}} \mathrm{d} x \, . \end{eqnarray*}\]
Derivation with respect to \(t\), interchanging derivation and expectation (why can we do that?)
\[\begin{eqnarray*} \widehat{\Phi}'(t) & = & \int_{-\infty}^{\infty} -x \sin(tx) \frac{\mathrm{e}^{- \frac{x^2}{2}}}{\sqrt{2 \pi}} \mathrm{d} x \, . \end{eqnarray*}\] Now relying on Stein’s Identity with \(g(x)=-\sin(tx)\) and \(g'(x)=-t\cos(tx)\) \[\begin{eqnarray*} \widehat{\Phi}'(t) & = & - t \int_{-\infty}^{\infty} \cos(tx) \phi(x) \mathrm{d} x \\ & = & -t \widehat{\Phi}(t) \, . \end{eqnarray*}\]
We immediately get \(\widehat{\Phi}(0)=1\), and solving the differential equation leads to \[ \log \widehat{\Phi}(t) = - \frac{t^2}{2} \, . \]
The fact that the characteristic function completely defines the probability distribution provides us with a converse of Lemma 16.1.
Proof. Consider the real \(\widehat{F}\) and the imaginary part \(\widehat{G}\) of the characteristic function of the distribution pf \(X\), the identity entails that \(\widehat{F}'(t) = -t \widehat{F}(t)\) and \(\widehat{G}'(t) = -t \widehat{G}(t)\) with \(\widehat{F}(0)=1\) and \(\widehat{G}(0)=0\). Solving the two differential equations leads to \(\widehat{F}(t) = \mathrm{e}^{-t^2/2}\) and \(\widehat{G}(t)=0\). We just checked that the characteristic function of the distribution of \(X\) is the characteristic function of \(\mathcal{N}(0,1).\)
It is now easy to check that the distribution of the sum of two independent Gaussian random variables is a Gaussian random variable.
If \(X\) and \(Y\) are two independent random variables distributed according to \(\mathcal{N} (\mu, \sigma^2)\) and \(\mathcal{N} (\mu', \sigma^{\prime 2})\) then \(X + Y\) is distributed according to \(\text{$\mathcal{N} (\mu + \mu', \sigma^2 + \sigma^{\prime 2})$}\).
Check and justify.
The moment generating function of a Gaussian random variable is given by \[ s \mapsto \mathbb{E} \left[ \mathrm{e}^{s X} \right] = \text{e}^{\frac{s^2}{2}} \, . \]
From Markov’s inequality, we obtain interesting upper bounds on the Gaussian tail function. Some calculus allows us to refine the tail bounds
Proof. The proof boils down to repeated integration by parts.
\[\begin{eqnarray*} \overline{\Phi}(x) & = & \int_x^{\infty} \frac{1}{ \sqrt{2 \pi}} \mathrm{e}^{- \frac{u^2}{2}} \mathrm{d} u\\ & = & \left[ - \frac{1}{ \sqrt{2 \pi} u} \mathrm{e}^{- \frac{u^2}{2}} \right]^{\infty}_x - \int_x^{\infty} \frac{1}{ \sqrt{2 \pi}} \frac{1}{u^2} \mathrm{e}^{- \frac{u^2}{2}} \mathrm{d} u . \end{eqnarray*}\]
As the second term is non-positive,
\[ \overline{\Phi}(x)\leq \left[ - \frac{1}{ \sqrt{2 \pi} u} \mathrm{e}^{- \frac{u^2}{2}} \right]^{\infty}_x = \frac{\phi(x)}{ x} . \]
This is the first part of the right-hand inequality, the other part comes from Markov’s inequality. For the left-hand inequality, we have to upper bound \(\int_x^{\infty} \frac{1}{ \sqrt{2 \pi}} \frac{1}{u^2} \mathrm{e}^{- \frac{u^2}{2}} \mathrm{d} u\).
\[\begin{eqnarray*} \int_x^{\infty} \frac{1}{ \sqrt{2 \pi}} \frac{1}{u^2} \mathrm{e}^{- \frac{u^2}{2}} \mathrm{d} u & = & \left[ \frac{- 1}{ \sqrt{2 \pi}} \frac{1}{u^3} \mathrm{e}^{- \frac{u^2}{2}} \right]_x^{\infty} - \int_x^{\infty} \frac{1}{ \sqrt{2 \pi}} \frac{3}{u^4} \mathrm{e}^{- \frac{u^2}{2}} \mathrm{d} u\\ & \leq & \frac{1}{ \sqrt{2 \pi}} \frac{1}{x^3} \mathrm{e}^{- \frac{x^2}{2}} . \end{eqnarray*}\]
Proof. Thanks to distributional symmetry, \(\mathbb{E} \left[ X^k \right]=0\) for all odd\(k\). We handle even powers using integration by parts:
\[\begin{eqnarray*} \mathbb{E} \left[ X^{k+2} \right] & = & (k+1) \mathbb{E} \left[ X^{k} \right] \, . \end{eqnarray*}\]
Induction on \(k\) leads to,
\[\begin{eqnarray*} \mathbb{E} \left[ X^{2k} \right] &= & \prod_{j=1}^k (2j-1) = \frac{(2k) !}{2^k k! } \, . \end{eqnarray*}\]
Note that \((2k)!/(2^k k!)\) is also the number of partitions of \(\{1, \ldots, 2k\}\) into subsets of cardinality \(2\).
The skewness is null, the kurtosis (ratio of fourth centred moment over squared variance equals \(3\):
\[ \mathbb{E}[X^4] = 3 \times \mathbb{E}[X^2]^2 \, . \]
16.2 Gaussian vectors
A Gaussian vector is a collection of univariate Gaussian random variables that satisfies a very stringent property:
Remark 16.1. Not every collection of Gaussian random variables forms a Gaussian vector.
The random vector \((X, \epsilon X)\) with \(X \sim \mathcal{N}(0.1)\), independent of \(\epsilon\) which is worth \(\pm 1\) with probability \(1/2\), is not a Gaussian vector although both \(X\) and \(\epsilon X\) are univariate Gaussian random variables.
Yet there are Gaussian vectors! A simple way to obtain a Gaussian vector is provided by the next proposition (checked by a characteristic function argument).
In the sequel, a standard Gaussian vector is a random vector with independent coordinates with each coordinate distributed according to \(\mathcal{N}(0,1)\).
We will see how to construct general Gaussian vectors. Before this, let us check that the joint distribution of a Gaussian random vector is completely characterized by its covariance matrix and its expectation vector.
Recall that the covariance of random vector \(X= (X_1, \ldots, X_n)^\top\) is the matrix \(K\) with dimension \(n \times n\) with coefficients
\[ K [i, j] = \operatorname{Cov} (X_i, X_j) = \mathbb{E} [X_i X_j] - \mathbb{E} [X_i] \mathbb{E} [X_j] . \]
Without loss of generality, we may assume that random vector \(X\) is centered For every \(\lambda = (\lambda_1, \ldots, \lambda_n)^\top \in \mathbb{R}^n\), we have:
\[ \operatorname{var}(\langle \lambda, X \rangle) = \lambda^\top K \lambda = \text{trace} (K \lambda \lambda^\top)\, \] (this is does not depend on any Gaussianity assumption).
Indeed,
\[\begin{eqnarray*} \operatorname{var}(\langle \lambda, X \rangle) & = & \mathbb{E} \left[ \left( \sum_{i=1}^n \lambda_i X_i\right)^2\right] \\ & = & \sum_{i,j=1}^n \mathbb{E} \left[\lambda_i \lambda_j X_i X_j \right] \\ & = & \sum_{i,j=1}^n \lambda_i \lambda_j K[i,j] \\ & = & \lambda^\top K \lambda\, . \end{eqnarray*}\]
The characteristic function of a Gaussian vector \(X\) with expectation vector \(\mu\) and covariance \(K\) satisfies
\[ \mathbb{E} \mathrm{e}^{\imath \langle \lambda, X \rangle } = \mathrm{e}^{\imath \langle \lambda, \mu \rangle - \frac{\lambda^\top K \lambda}{2}} \, \, . \]
A linear transform of a Gaussian vector is a Gaussian vector.
Proof. Without loss of generality, we assume \(Y\) is centred.
For any \(\lambda \in \mathbb{R}^p\), \(\langle \lambda , A Y \rangle = \langle A^\top \lambda, Y \rangle\) , thus \(A \times Y\) is Gaussian with variance
\[ \lambda^\top A K A^\top \lambda \, . \]
The covariance of \(A \times Y\) is determined by this observation.
To manufacture Gaussian vectors with general covariance matrices, we rely on an important notion from matrix analysis.
Proof. If \(X\) is a \(\mathbb{R}^k\)-valued random vector, with covariance \(K\), for any vector \(\lambda \in \mathbb{R}^n\),
\[ \lambda^\top K \lambda = \sum_{i,j\leq k} K_{i,j} \lambda_i \lambda_j = \operatorname{cov}(\langle \lambda, X \rangle, \langle \lambda, X \rangle) \] soit \(\lambda^\top K \lambda = \operatorname{var}(\langle \lambda, X \rangle)\). The variance of a univariate random variable is always non-negative.
The next observation is the key to the construction to general Gaussian vectors.
We do not check this proposition. This is a basic Theorem from matrix analysis. It can be established from the spectral decomposition theorem for symmetric matrices. It can also be established by a simple constructive approach: a positive definite matrix \(K\) admits a Cholesky decomposition, in other words, there exists a triangular matrix lower than \(L\) such that \(K = L \times L^\top\).
The next proposition is a corollary of the general formula for image densities.
Proof. The density formula is trivially correct for standard Gaussian vectors. For the general case, it is enough to invoke the image density formula to the image of the standard Gaussian vector by the bijective linear transformation defined by the Cholesky factorization of \(A\). The determinant of the Cholesky factor is the square root of the determinant of \(A\).
The Gaussian space is a real vector space. If \((\Omega, \mathcal{F},P)\) denotes the probability space, \(X\) lives on, the Gaussian space is a subspace of \(L^2_{\mathbb{R}}(\Omega, \mathcal{F},P)\). It inherits the inner product structure from \(L^2_{\mathbb{R}}(\Omega, \mathcal{F},P)\).
This inner-product is completely defined by the covariance matrix \(K\).
\[\begin{eqnarray*} \left\langle \sum_{i = 1}^n \lambda_i X_i, \sum_{i = 1}^n \lambda'_i X_i \right\rangle & \equiv & \mathbb{E}_P \left[ \left( \sum_{i = 1}^n \lambda_i X_i \right) \left( \sum_{i = 1}^n \lambda'_i X_i \right) \right]\\ & = & \sum^n_{i, i' = 1} \lambda_i \lambda_{i'}' K [i, i']\\ & = & (\lambda_1, \ldots, \lambda_n) K \left(\begin{array}{c} \lambda'_1\\ \vdots\\ \lambda'_n \end{array}\right) \\ & = & \text{trace} \left( K \left(\begin{array}{c} \lambda_1\\ \vdots\\ \lambda_n \end{array}\right) \left(\begin{array}{ccc} \lambda'_1 & \dots & \lambda'_n \end{array}\right) \right). \end{eqnarray*}\]
Remark 16.2. Different Gaussian vectors may generate the same Gaussian space. Explain how and why.
Gaussian spaces enjoy remarkable properties. Independence of random variables belonging to the same Gaussian space may checked very easily.
Without loss of generality, we assume covariance matrix \(K\) is positive definite.
Proof. Independence always implies orthogonality.
Without loss of generality, we assume that the Gaussian space is generated by a standard Gaussian vector, let \(Z = \sum_{i = 1}^n \lambda_i X_i\) and \(Y = \sum_{i = 1}^n \lambda'_i X_i\).
If \(Z\) and \(Y\) are orthogonal (or non-correlated)
\[\mathbb{E} [ZY] = \sum_{i = 1}^n \lambda_i \lambda_{i}' = 0 .\]
To show that \(Z\) and \(Y\) are independent, it is enough to check that for all \(\mu\) and \(\mu'\) in \(\mathbb{R}\) \[\mathbb{E} \left[ \mathrm{e}^{\imath \mu Z} \mathrm{e}^{\imath \mu' Y} \right] = \mathbb{E} \left[ \mathrm{e}^{\imath \mu Z} \right] \times \mathbb{E} \left[ \mathrm{e}^{\imath \mu' Y} \right] . \]
\[\begin{align*} \mathbb{E} \left[ \mathrm{e}^{\imath \mu Z} \mathrm{e}^{\imath \mu' Y} \right] & = \mathbb{E} \left[ \mathrm{e}^{\imath \mu \sum_i \lambda_i X_i} \mathrm{e}^{\imath \mu' \sum_i \lambda'_i X_i} \right]\\ & = \mathbb{E} \left[ \prod_{i = 1}^n \mathrm{e}^{\imath (\mu \lambda_i + \mu' \lambda'_i) X_i} \right]\\ & X_i \text{ are independent} \ldots\\ & = \prod_{i = 1}^n \mathbb{E} \left[ \mathrm{e}^{\imath (\mu \lambda_i + \mu' \lambda'_i) X_i} \right]\\ & = \prod_{i = 1}^n \mathrm{e}^{- (\mu \lambda_i + \mu' \lambda'_i) ^2 / 2}\\ & = \exp \left( - \frac{1}{2} \sum_{i = 1}^n \mu^2 \lambda_i^2 + 2 \mu \mu' \lambda_i \lambda'_i + \mu'^2 \lambda'^2_i \right)\\ & \text{orthogonality}\\ & = \exp \left( - \frac{1}{2} \sum_{i = 1}^n \mu^2 \lambda_i^2 + \mu'^2 \lambda'^2_i \right)\\ & \ldots\\ & = \mathbb{E} \left[ \mathrm{e}^{\imath \mu Z} \right] \times \mathbb{E} \left[ \mathrm{e}^{\imath \mu' Y} \right] . \end{align*}\]
The next proposition is a direct consequence.
16.3 Convergence of Gaussian vectors
Recall the Lévy continuity theorem (Theorem 15.2)), it relates weak convergence for probability measures and simple convergence for characteristic functions.
For every \(\vec{s} \in \mathbb{R}^k\), functions \(\vec{x}\mapsto \cos ( \left\langle \vec{s}, \vec{x} \right\rangle)\) and \(\vec{x} \mapsto \sin ( \left\langle \vec{s}, \vec{x}\right\rangle)\) are continuous and bounded, They are also infinitely many times differentiable.
It is remarkable and useful that weak convergence can be checked on this small set of functions.
The continuity condition at \(0\) is necessary. The characteristic function of a probability distribution is always continuous at \(0\). Continuity at \(0\) warrants the tightness of the sequence of probability distributions.
16.4 Gaussian conditioning
Let \((X_1,\ldots,X_n)^\top\) be a Gaussian vector with distribution \(\mathcal{N}(\mu, K)\) where \(K \in \textsf{DP}(n)\). The covariance matrix \(K\) is partitioned into blocks \[ K = \left[ \begin{array}{cc} A & B^\top \\ B & W \end{array} \right] \]
where \(A \in \textsf{DP}(k)\), \(1 \leq k < n\), and \(W \in \textsf{DP}(n-k)\).
We are interested in the conditional expectation of \((X_1, \ldots, X_k)^\top\) with repsect to \(\sigma(X_{k+1},\ldots,X_n)\) and in the conditional distribution of \((X_1, \ldots, X_k)^\top\) with respect to \(\sigma(X_{k+1},\ldots,X_n)\).
The Schur complement of \(A\) in \(K\) is defined as \[ W - B A^{-1} B^\top\, . \]
This definition makes sense for symmetric matrices when \(A\) is non-singular.
If \(K \in \textsf{DP}(n)\) then the Schur complement of \(A\) in \(K\) also belongs to \(\textsf{DP}(n-k)\)
In the statement of the next theorems, \(A^{-1/2}\) denotes the Cholesky factor of \(A^{-1}\): \(A^{-1} = A^{-1/2} \times (A^{-1/2})^\top\).
We will first study the conditional density, and, with a minimum amount of calculation, establish that it is Gaussian. Conditional expectation will be calculated as expectation under conditional distribution.
To characterize conditional density, we rely on a distributional representation argument (any Gaussian vector is distributed as the image of a standard Gaussian vector by an affine transformation) and a matrix analysis result that is at the core of the Cholesky factorization of positive semi-definite matrices.
\((X_1, \ldots, X_n)^\top\) is distributed as the image of standard Gaussian vector by a block triangular matrix
, et utiliser des propriétés des lois conditionnelles pour établir à la fois les deux résultats.
Proof. Without loss of generality, we check the statement on centered vectors. The Cholesky factorization of \(K\) allows us to write
\[ \begin{pmatrix} X_1 \\ \vdots \\ X_n \end{pmatrix} \sim \left[ \begin{array}{cc} L_1 & 0 \\ B (L_1^\top)^{-1} & L_2 \end{array} \right] \times \begin{pmatrix} Y_1 \\ \vdots \\ Y_n \end{pmatrix} \]
where \(( Y_1, \ldots, Y_n)^\top\) is a centered standard Gaussian vector.
In the sequel, we assume \((X_1, \ldots,X_n)^\top\) and \((Y_1,\ldots,Y_n)^\top\) live on the same probability space. As \(L_1\) is invertible, the \(\sigma\)-algebras generated by \((X_1, \ldots,X_k)^\top\) and \((Y_1, \ldots,Y_k)^\top\) are equal. We agree on \(\mathcal{G}=\sigma(X_1, \ldots,X_k)\). The conditional expectations and conditional distributions also coincide .
\[\begin{eqnarray*} \mathbb{E} \left[ \begin{pmatrix} X_{k+1} \\ \vdots \\ X_n \end{pmatrix} \mid \mathcal{G} \right] &= &\mathbb{E} \left[ B (L_1^\top)^{-1} \begin{pmatrix} Y_{1} \\ \vdots \\ Y_k \end{pmatrix} \mid \mathcal{G} \right] + \mathbb{E} \left[ L_2 \begin{pmatrix} Y_{k+1} \\ \vdots \\ Y_n \end{pmatrix} \mid \mathcal{G} \right] \\ & = & B (L_1^\top)^{-1} L_1^{-1}\begin{pmatrix} X_{1} \\ \vdots \\ X_k \end{pmatrix} = B A^{-1} \begin{pmatrix} X_{1} \\ \vdots \\ X_k \end{pmatrix} \, , \end{eqnarray*}\]
car \((Y_{k+1}, \ldots,Y_n)^\top\) is centered and independent from \(\mathcal{G}\).
Note that residuals
\[ \begin{pmatrix} X_{k+1} \\ \vdots \\ X_n \end{pmatrix} -\mathbb{E} \left[ \begin{pmatrix} X_{k+1} \\ \vdots \\ X_n \end{pmatrix} \mid \mathcal{G} \right] = L_2 \begin{pmatrix} Y_{k+1} \\ \vdots \\ Y_n \end{pmatrix} \]
are independent from \(\mathcal{G}\). This is a Gaussian property. For general square integrable random variables, we may only assert that residuals are orthogonal to \(\mathcal{G}\)-measurable random variables.
The conditional distribution of \((X_{k+1},\ldots, X_n)^\top\) with respect to \((X_1,\ldots, X_k)^\top\) coincides with the conditional distribution of
\[ B (L_1^\top)^{-1} \times \begin{pmatrix} Y_1\\ \vdots \\ Y_k \end{pmatrix} + L_2 \times \begin{pmatrix} Y_{k+1}\\ \vdots \\ Y_n \end{pmatrix} \]
conditionally on \((Y_1,\ldots, Y_k)^\top\). As \((Y_1,\ldots, Y_k)^\top = L_1^{-1}(X_1,\ldots,X_k)^\top\), the conditional distribution we are looking for is Gaussian with expectation \[B A^{-1} \times \begin{pmatrix} X_1\\ \vdots \\ X_k \end{pmatrix}\] (the conditional expectation) and variance \(L_2 \times L_2^\top = W - B A^{-1} B^\top\).
Example 16.1 If \((X,Y)^\top\) is a centered Gaussian vector with \(\operatorname{var}(X)=\sigma_x^2\), \(\operatorname{var}(Y)=\sigma^2_y\) and \(\operatorname{cov}(X,Y)= \rho \sigma_x \sigma_y\), the conditional distribution of \(Y\) with respect to \(X\) is
\[ \mathcal{N}\left( \rho \sigma_y/\sigma_x X, \sigma^2_y (1- \rho^2) \right) \, . \]
The quantity \(\rho\) is called the linear correlation coefficient between \(X\) and \(Y\). By Cauchy-Schwarz’s inequality, \(\rho \in [-1,1]\).
These two theorems are usually addressed in the order in which they are stated. Conditional expectation is characterized by adopting the \(L^2\) (predictive) viewpoint: the conditional expectation of the random vector \(Y\) knowing \(X\) is defined as the best \(X\)-measurable predictor of the vector \(Y\) with respect to quadratic error (the random vector \(Z\), \(X\)-measurable that minimizes \(\mathbb{E} \left[ \| Y- Z\|^2 \right]\)).
In order to characterize conditional expectation, we first compute the optimal affine predictor of \((X_{k+1},\ldots,X_n)^\top\) based on \((X_{1},\ldots,X_k)^\top\). This optimal affine predictor is
\[ \begin{pmatrix} \mu_{k+1} \\ \vdots \\ \mu_n \end{pmatrix} + \left(B A^{-1} \right) \times \left( \begin{pmatrix} X_{1} \\ \vdots \\ X_{k} \end{pmatrix} - \begin{pmatrix} \mu_{1} \\ \vdots \\ \mu_k \end{pmatrix}\right) \, , \]
(if Gaussian vectors are centred, this amounts to determine the matrix \(P\) with dimensions \((n-k)\times k\) which minimizes \(\text{trace}(PA P^\top -2 B P^\top\))). The optimal affine predictor is a Gaussian vector, one can check that the residual vector
\[ \begin{pmatrix} X_{k+1}\\ \vdots \\ X_n \end{pmatrix} - \left\{ \begin{pmatrix} \mu_{k+1} \\ \vdots \\ \mu_n \end{pmatrix} + \left(B A^{-1}\right) \times \left( \begin{pmatrix} X_{1} \\ \vdots \\ X_{k} \end{pmatrix} - \begin{pmatrix} \mu_{1} \\ \vdots \\ \mu_k \end{pmatrix}\right) \right\} \]
is also Gaussian and orthogonal to the affine predictor. The residual vector is independent from the affine predictor.
This is enough to establish that the affine predictor is the orthogonal projection of \((X_{k+1}, \ldots, X_n)^\top\) on the closed linear subspace of square-integrable \((X_{1},\ldots,X_k)^\top\)-measurable random vectors.
This proves that the affine predictor is the conditional expectation.
In the notes, we deal with a special case of linear conditioning.
To fugure out general linear conditioning, consider \(X \sim \mathcal{N}(0, {K})\) (we assume centering to alleviate notation and computations, translating does not change the relevant \(\sigma\)-algebras and thus conditioning), where \({K} \in \textsf{DP}(n)\), and a linear transformation defined by matrix \(H\) with dimensions \(m \times n\). \(H\) is assumed to have rank \(m\). Agree on \(Y= {H} X\). Considering the Gaussian vector \([ X^\top : Y^\top]\) with covariance matrix \[ \left[ \begin{array}{cc} {K} & {K} {H}^\top \\ {H}{K} & {H} {K} {H}^\top \end{array} \right] \]
and adapting the previous computations (the covariance matrix is not positive definite any more), we may check that the conditional distribution of \(X\) with respect to \(Y\) is Gaussian with expectation \[ K H^\top (HKH^\top)^{-1} \]
and variance
\[ K - K H^\top (HKH^\top)^{-1} H K \, . \]
The linearity of conditional expectation is a property of Gaussian vectors and linear conditioning. If you condition with respect to the norm \(\| X\|_2\), the conditional distribution is not Gaussian anymore.
16.5 About Gamma distributions
Investigating the norm of Gaussian vectors will prompt us to introduce \(\chi^2\) distributions, a sub-family of Gamma distributions.
If \(X \sim \text{Gamma}(p,1)\) then \(\sigma X \sim \text{Gamma}(p,1/\sigma)\) for \(\sigma>0\).
Euler’s \(\Gamma ()\) function interpolates the factorial. For every positive real \(p\), \(\Gamma (p + 1) = p \Gamma (p) .\) If \(p\) is integer, \(\Gamma (p + 1) = p!\)
The next proposition is a cornerstone of Gamma-calculus. The sum of two independent Gamma-distributed random variables is Gamma distributed if they have the same intensity (or scale) parameter.
Proof. The density of the distribution of \(X+Y\) is the convolution of the densities \(g_{p, \lambda}\) et \(g_{q, \lambda}\). \[\begin{eqnarray*}
g_{p, \lambda} \ast g_{q, \lambda} (x) & = & \int_{\mathbb{R_{}}}
g_{p, \lambda} (z) g_{_{q, \lambda}} (x - z) \mathrm{d} z\\
& = & \int_0^x g_{p, \lambda} (z) g_{_{q,
\lambda}} (x - z) \mathrm{d} z\\
& = & \int_0^x \frac{\lambda^p}{\Gamma (p)} z^{p - 1} \mathrm{e}^{- \lambda
z} \frac{\lambda^q}{\Gamma (q)} (x - z)^{q - 1} \mathrm{e}^{-
\lambda (x - z)} \mathrm{d} z\\
& = & \frac{\lambda^{p + q}}{\Gamma (p) \Gamma (q)} \mathrm{e}^{- \lambda x}
\int_0^x z^{p - 1} (x - z)^{q - 1} \mathrm{d} z\\
& & \operatorname{changement} \operatorname{de} \operatorname{variable} z = x u\\
& = & \frac{\lambda^{p + q}}{\Gamma (p) \Gamma (q)} \mathrm{e}^{- \lambda x}
x^{p + q - 1} \int_0^{1} u^{p-1} (1 - u)^{q - 1}
\mathrm{d} u\\
& = & g_{p + q, \lambda} (x) \frac{\Gamma(p+q)}{\Gamma(p)\Gamma(q)} \int_0^{1} u^{p-1} (1 - u)^{q - 1}
\mathrm{d} u \, .
\end{eqnarray*}\] We may pocket the next observation:
\[
B(p,q):= \int_0^{1} u^{p-1} (1 - u)^{q - 1}
\mathrm{d} u
\]
satisfies \(B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}.\)
Gamma distributions with parameters \((k / 2, 1 / 2)\) for \(k \in \mathbb{N}\) deserve to be named: they are \(\chi^2\) distributions with \(k\) degrees of freedom.
Proof. According to proposition Proposition 9.2), it suffices to establish the proposition \(k = 1.\)
Let \(X \sim \mathcal{N}(0,1)\), for \(t\geq 0\), \[\begin{align*} \mathbb{P} \left\{ X^2 \leq t\right\} & = & \Phi(\sqrt{t}) - \Phi(-\sqrt{t}) \\ & = & 2 \Phi(\sqrt{t}) - 1 \,. \end{align*}\] Now, differentiating with respect to \(t\), applying the chain rule provides us with a formula for the density: \[ 2 \frac{1}{2\sqrt{t}} \phi(\sqrt{t}) = \frac{1}{\sqrt{2\pi t}} \mathrm{e}^{-\frac{t}{2}} = \left(\frac{1}{2}\right)^{1/2} \frac{t^{-1/2}}{\Gamma(1/2)} \mathrm{e}^{-\frac{t}{2}} \,. \]
16.6 Norms of centred Gaussian vectors
The distribution of the squared Euclidean norm of a centered Gaussian vector only depends on the spectrum of its covariance matrix.
This is a corollary of an important property of standard Gaussian vectors: rotational invariance. The standard Gaussian distribution is invariant under orthogonal transform (a matrix \(O\) is orthogonal iff \(OO^\top=\text{Id})\).
Proof. Matrix \(A\) may be factorized as \(A = LL^\top\) (Cholesky), and \({X}\) is distributed like \(L {Y}\) where \({Y}\) is standard Gaussian. The quadratic form \({X}^\top M {X}\) is thus distributed like \({Y}^\top {L}^\top M {L} {Y}\). There exist an orthogonal transform \(O\) such that \(L^\top M L = O^\top \operatorname{diag} (\lambda_i) O.\) Random vector \(O {Y}\) is distributed like \(\mathcal{N} (0, I_n)\).
16.7 Norm of non-centred Gaussian vectors
The distribution of the squared norm of a Gaussian vector with covariance matrix \(\sigma^2 \operatorname{Id}\) depends on the norm of the expectation but does not depend on its direction. In addition, this distribution stochastically can be compared with the distribution of the squared norm of a centred Gaussian vector with the same covariance.
If \(X\) is stochastichally less than \(Y\) and if \(F\) and \(G\) denote the cumulative distribution functions of \(X\) and \(Y\), then for all \(x \in \mathbb{R}\), \(F(x)\geq G(x)\). Quantile functions \(F^\leftarrow, G^\leftarrow\) satisfy \(F^\leftarrow(p) \leq G^\leftarrow(p)\) for \(p \in (0,1)\).
Conversely.
The proof proceeds by a quantile coupling argument.
Proof. It is enough to endow \(([0,1], \mathcal{B}([0,1])\) with the uniform distribution. Let \(X (\omega)= F^{\leftarrow}(\omega)\), \(Y(\omega) = G^\leftarrow(\omega)\). Then the distribution of \(X\) (resp. \(Y\)) has cumulative distribution function \(F\) (resp. \(G\)) and the following holds:
\[ \mathbb{P} \{ X \leq Y\} = \mathbb{P} \{ F^{\leftarrow}(U) \leq G^{\leftarrow}(U)\} = 1 \, . \]
Proof. The Gaussian vector \(Y\) is distributed like \(\theta + X\). There exists an orthogonal transform \(O\) such that \[ O \theta = \begin{pmatrix} \| \theta\|_2 \\ 0 \\ \vdots \\ 0 \end{pmatrix} \, . \]
Vectors \(OY\) and \(OX\) respectively have the same norms as \(X\) and \(Y\).
The squared norm of \(Y\) is distributed as the squared norm of \(OY\), that is like \((Z_1+ \|\theta\|_2)^2 +\sum_{i=2}^d Z_i^2\). This proves the first part of the theorem.
To establish the second part of the theorem, it suffices to check that for every \(x\geq 0\), \[ \mathbb{P} \left\{ (Z_1+ \|\theta\|_2)^2 \leq x \right\} \leq \mathbb{P} \left\{ X_1^2 \leq x \right\} \, , \]
that is
\[
\mathbb{P} \left\{ |Z_1+ \|\theta\|_2| \leq \sqrt{x} \right\} \leq
\mathbb{P} \left\{ |X_1| \leq \sqrt{x} \right\} \, ,
\]
or \[ \Phi(\sqrt{x}- \|\theta\|_2) - \Phi(-\sqrt{x}-\|\theta\|_2) \leq \Phi(\sqrt{x}) - \Phi(-\sqrt{x}) \, . \]
For \(y>0\), the function mapping \([0,\infty)\) to \(\mathbb{R}\), defined by \(a \mapsto \Phi(y-a) - \Phi(-y-a)\) is non-increasing with respect to \(a\): it derivative with respect to \(a\) equals \(-\phi(y-a)+\phi(-y-a)=\phi(y+a)-\phi(y-a)\leq 0\). The conclusion follows
The last step of the proof reads as \[ \mathbb{P} \left\{ X \in \theta + C \right\} \leq \mathbb{P} \left\{ X \in C\right\} \]
where \(X \sim \mathcal{N}(0,\operatorname{Id}_1)\), \(\theta \in \mathbb{R}\) and \(C = [-\sqrt{x},\sqrt{x}]\). This inequality holds in dimension \(d\geq 1\) if \(C\) is compact, convex, symmetric. This (subtle) result is called Anderson’s Lemma.
16.8 Cochran Theorem and consequences
Proof. The covariance matrix of \(\pi_{E_j} X\) is \(\pi_{E_j} \pi_{E_j}^\top = \pi_{E_j}\). The eigenvalues of \(\pi_{E_j}\) are \(1\) with multiplicity \(\text{dim}(E_j)\) and \(0\). The statement about the distribution of \(\| \pi_{E_j} X\|_2^2\) is a corollary of ?prp-normgaussstand and ?prp-normespectre.
To prove stochastic independence, let us consider \(\mathcal{I}, \mathcal{J} \subset \{1,\ldots,k\}\) with \(\mathcal{I} \cap \mathcal{J} = \emptyset.\) It is enough to check that for all \((\alpha)_{j \in \mathcal{I}}, (\beta_j)_{j \in \mathcal{J}}\), the characteristic functions of \[ \left(\sum_{j\in \mathcal{I}} \langle \alpha_j, \pi_{E_j} X \rangle, \sum_{j\in \mathcal{J}} \langle \beta_j, \pi_{E_j} X \rangle\right) \]
can be factorized. It suffices to check that the two Gaussians are orthogonal.
\[\begin{eqnarray*} { \mathbb{E} \left[ \left(\sum_{j\in \mathcal{I}} \langle \alpha_j, \pi_{E_j} X \rangle \right) \times \left(\sum_{j'\in \mathcal{J}} \langle \beta_{j'}, \pi_{E_{j'}} X \rangle\right)\right]} & = & \sum_{j \in \mathcal{I}, j' \in \mathcal{J}} \alpha_j^\top \pi_{E_j} \pi_{E_{j'}} \beta_{j'} = 0 \, . \end{eqnarray*}\]
The next result is a cornerstone of statistical inference in Gaussian models. It is a corollary of Cochran’s Theorem.
Proof. Without loss of generality, we may assume that \(\mu=0\) et \(\sigma=1\).
As \[ \begin{pmatrix} \overline{X}_n \\ \vdots\\ \overline{X}_n \\ \end{pmatrix} = \frac{1}{n} \begin{pmatrix} 1 \\ \vdots\\ 1 \\ \end{pmatrix} \times \begin{pmatrix} 1 & \ldots & 1 \end{pmatrix} X \] the vector \((\overline{X}_n, \ldots , \overline{X}_n)^\top\) is the orthogonal projection of the standard Gaussian vector \(X\) on the line generated by \((1, \ldots, 1)^\top\).
Vector \((X_1- \overline{X}_n, \ldots , X_n -\overline{X}_n)^\top\) is the orthogonal projection fo Gaussian vector \(X\) on the hyperplane which is orthogonal to \((1, \ldots, 1)^\top\).
According to Cochran’s Theorem (Section 16.8), random vectors \((\overline{X}_n, \ldots , \overline{X}_n)^\top\), and \((X_1- \overline{X}_n, \ldots , X_n -\overline{X}_n)^\top\) are independent.
The distribution of \(\overline{X}_n\) is trivially Gaussian.
The distribution of \(V\) is characterized using Cochran’s Theorem.
16.9 Gaussian concentration
The very definition of Gaussian vectors characterizes he distribution of any affine function of a standard Gaussian vector. If the linear part of the affine function is defined by a vector \(\lambda\), we know that the variance will be \(\|\lambda\|^2_2\). What happens if we are interested in fairly regular functions of a standard Gaussian vector? for example if we consider \(L\)-lipschitzian functions? These are generalizations of affine functions. We cannot therefore expect a general increase in the variance of the \(L\)-Lipschitzian functions of a standard Gaussian vector better than \(L^2\) (in the linear case the Lipschitz constant is the Euclidean norm of \(\lambda\)). It is remarkable that the bound provided for linear functions extends to Lipschitzian functions. It is even more remarkable that this bound does not involve the dimension of the ambient space.
The proof relies on the next identity.
We start by checking this proposition on functions \(x \mapsto \mathrm{e}^{\imath \langle \lambda, x\rangle}, x \in \mathbb{R}^d\).
Proof. Let us first check Poincaré’s inequality.
We choose \(f=g\). Starting from the covariance identity, thanks to Cauchy-Schwarz’s inequality:
\[\begin{eqnarray*} \operatorname{var}(f(X) ) &= & \operatorname{cov}(f(X),f(X)) \\ & = & \int_0^1 \mathbb{E}\left\langle \nabla f(X) , \nabla f\left(\alpha X +\sqrt{1- \alpha^2} Y \right) \right\rangle \mathrm{d} \alpha \\ & \leq & \int_0^1 \left( \mathbb{E}\| \nabla f(X) \|^2\right)^{1/2} \times \left(\mathbb{E} \|\nabla f\left(\alpha X +\sqrt{1- \alpha^2} Y\right)\|^2 \right)^{1/2} \mathrm{d} \alpha \, . \end{eqnarray*}\]
The desired results follows by noticing that \(X\) and \(\alpha X + \sqrt{1- \alpha^2}Y\) are both \(\mathcal{N}(0,\text{Id})\)-distributed.
To obtain the exponential inequality, choose \(f\) differentiable and 1-Lipschitz, and \(g = \exp(\lambda f)\) pour \(\lambda\geq 0\). Without loss of generality, assume \(\mathbb{E}f(X)=0\). The covariance identity and the chain rule imply
\[\begin{eqnarray*} \operatorname{cov}\left(f(X),\mathrm{e}^{\lambda f(X)}\right) & = & \lambda \int_0^1 \mathbb{E}\left[\left\langle \nabla f(X) , \nabla f\left(\alpha X +\sqrt{1- \alpha^2} Y \right) \right\rangle \mathrm{e}^{\lambda f\left(\alpha X +\sqrt{1- \alpha^2} Y \right)}\right] \mathrm{d} \alpha \\ & \leq & \lambda L^2 \int_0^1 \mathbb{E}\left[ \mathrm{e}^{\lambda f\left(\alpha X +\sqrt{1- \alpha^2} Y \right)}\right] \mathrm{d} \alpha \\ & = & \lambda L^2 \mathbb{E}\left[ \mathrm{e}^{\lambda f\left(X\right)}\right] \end{eqnarray*}\]
Define \(F(\lambda):= \mathbb{E}\left[ \mathrm{e}^{\lambda f\left(X\right)}\right]\). Note that we have just established a differential inequality for \(F\), checking \(\operatorname{cov}( f , \mathrm{e}^{\lambda f})= F'(\lambda)\) since \(f\) is centred:
\[ F'( \lambda) \leq \lambda L^2 F(\lambda) \,. \]
Solving this differential inequality under \(F(0)=1\), for \(\lambda\geq 0\) \[ F( \lambda) \leq \mathrm{e}^{\frac{\lambda^2L^2}{2}} \, . \]
The same approach works for \(\lambda<0\). It is enough to invoke Markov’s exponential inequality and to optimize with respect to \(\lambda=t/L^2\).
Concentration inequalities describe the behavior of the norm of high-dimensional Gaussian vectors.
Proof. The Euclidean norm is \(1\)-Lipschitz (triangle inequality). The first inequality follows fron Poincaré’s inequality.
The upper bound on expectation follows from Jensen’s inequality.
The lower bound on expectation follows from \((\mathbb{E} \|X\|_2)^2 = \mathbb{E} \|X\|_2^2 - \operatorname{var}(\|X\|_2)= d -\operatorname{var}(\|X\|_2)\) and from the variance upper bound.
16.10 Bibliographic remarks
Gaussian literature is very abundant, see for example (janson1997gaussian?). Much of this literature is relevant to statistics.
The lemmas ?lem-stein and ?lem-steinbis that characterize the Gaussian standard are the starting point of Stein’s (Charles) method to demonstrate the central limit theorem (and many other results). This relatively recent development is described in (Ross, 2011).
Matrix analysis and algorithmics play an important role in Gaussian analysis and statistics. The books (Horn & Johnson, 1990), and if we wish to go further (Bhatia, 1997), provide an introduction to the concepts and techniques of matrix factorization and elements of perturbation theory.
There is a multi-dimensional version of the laws of \(\chi^2\) that appear when determining the law of variance empirical. These are the laws of Wishart. They were the subject of intensive studies in random matrix theory, see for example (Anderson, Guionnet, & Zeitouni, 2010)
Gaussian concentration plays an important role in non-parametric statistics and is a source of inspiration in statistical learning. M. Ledoux’s book (Ledoux, 2001) provides an elegant perspective on this issue.