Application of matrix congruence and similarity

A homework problem in statistical parameter estimation asks us to show that for two symmetric positive definite matrices \(\Sigma_1\) and \(\Sigma_2\).

If \(\Sigma_1 \leq \Sigma_2\), then \(\Sigma_1^{-1} \ge \Sigma_2^{-1}\)

Note that by assumption \(B=\Sigma_2 - \Sigma_1\) is positive semi-definite.  If I can show that the resolvent identity of  \(\Sigma_1^{-1} - \Sigma_2^{-1}\), \(\Sigma_2^{-1} (\Sigma_2 - \Sigma_1 ) \Sigma_1^{-1}\) is positive semi-definite, then the above statement is verified.

This requires the following two results:

Sylvester's Law of Inertia

Symmetric matrices \(A\) and \(B\) are congruent (i.e. there is a non-singular matrix \(C\) such that \(C^TAC=B\)) if and only if \(A\) and \(B\) have the same inertia.  They have the same number of positive, negative and zero eigenvalues.

Theorem 1.

The product of a symmetric positive definite matrix \(A\) and a symmetric matrix \(B\) has the same inertia as \(B\)

Proof

Note that \(A^{-1/2}ABA^{1/2} = A^{1/2}BA^{1/2}\).  The right hand side is similar to \(AB\), which means they have the same eigenvalues.  Since \(A^{1/2}\) is symmetric, the matrix \(A^{1/2}BA^{1/2}\) is congruent to \(B\).  By Sylvester's Law of inertia, the eignevalues of \(B\) have the same inertia as \(A^{1/2}BA^{1/2}\) and also of \(AB\).

The main point from Theorem 1 is that multiplying a positive definite matrix \(A\) to any symmetric matrix \(B\) will not change the inertia of the result.  Note that \(\Sigma_2^{-1}\) and \(\Sigma_1^{-1}\) are positive definite. We let \(\Sigma_2^{-1}= A_1\), \(\Sigma_1^{-1}=A_2\) and \(\Sigma_2-\Sigma_1 =B\), and applying Theorem 1 twice, leads to \(\Sigma_2^{-1} (\Sigma_2 - \Sigma_1 ) \Sigma_1^{-1}\) positive semi-definite.  We have shown \(\Sigma_1^{-1} - \Sigma_2^{-1}\) is indeed positive semi-definite.

Caution!

For real positive definite matrices, not all of them are symmetric!  For example, given a symmetric positive definite matrix \(B\) and a anti-symmetric matrix \(C\) (\(C^T=-C\)), the sum of which (\(A=B+C\)) is positive definite.   Extra care must be taken.  All references of positive definiteness are within the context of symmetric matrices.