Invariance and carry over properties of MLE

Review: Asymptotic properties of MLE

• Asymptotically efficient (attains CRLB as \(N\rightarrow\infty\))
• Asymptotically Gaussian (asymptotically normality)
• Asymptotically Unbiased
• Consistent (weakly and strongly)

First, the invariance property of MLE

The MLE of the parameter \(\alpha = g(\theta)\), where the PDF \(p(x;\theta)\) is paremeterized by \(\theta\), is given by
\[ \hat{\alpha} = g(\hat{\theta})\] where \(\hat{\theta}\) is the MLE of \(\theta\).

Consistency (in class) is defined as the weak convergence of the sequence of estimates to the true parameter as N gets large.

If \(g(\theta)\) is continuous in \(\theta\), the convergence properties (esp. convergence in prob.) carry over, i.e. the consistency of the estimator \(g(\hat{\theta})\)

However, biasedness of the estimator \(g(\hat{\theta})\) depends on the convexity of \(g\) and does not carry over from \(\hat{\theta}\).

Other properties of MLE

• If an efficient estimator exists, the ML method will produce it.
• Unlike the MVU estimator, MLE can be biased
• Note: CRLB applies to unbiased estimators, so when estimator is biased, it is possible it has variance smaller than \(I^{-1}(\theta)\)

Properties of a regular family of parameterized distribution

A family of parameterized distribution defined by

\[ \mathcal{P} = \{ p_\theta(y)  | \theta \in \Theta \subset \mathbb{R}^P \}\]

is regular if it satisfies the following conditions

1. Support of \(p_\theta(y)\) does not depend on \(\theta\) for all \(\theta \in \Theta\)
2. \(\frac{\partial}{\partial \theta} p_\theta(y) \) exists
3. Optional \( \frac{\partial^2}{\partial \theta^2} p_\theta(y) \) exists

Note \[ \frac{\partial }{ \partial \theta } \ln p_\theta(y)  = \frac{1}{p_\theta(y) } \frac{\partial }{ \partial \theta } p_\theta(y) \quad \quad (4) \]

Define the score function (log := natural log)

\[  S_\theta (y) := \nabla_\theta \log p_\theta(y) \]

Note also

\[ E_\theta \{ 1 \} = 1 = \int_\mathcal{Y} p_\theta(y) dy \]

As a result of the above we have (Kay's definition of regular)
\begin{align*}  0 &= \frac{\partial }{ \partial \theta }E_\theta \{1\} \\
&= \frac{\partial }{ \partial \theta }\int p_\theta(y) dy \\
&\overset{1}{=} \int \frac{\partial }{ \partial \theta }  p_\theta(y) dy \\
&\overset{2,4}{=} \int p_\theta(y)  \frac{\partial }{ \partial \theta } \log p_\theta(y) dy \\
&= E_\theta \{ S_\theta(y) \}
\end{align*}

Spectral Theorem for Diagonalizable Matrices

It occurs to me that most presentation of the spectrum theorem only concerns orthonormal basis.  This is a more general result from Meyer.

Theorem

A matrix \( \mathbf{A} \in \mathbb{R}^{n\times x}\) with spectrum \(\sigma(\mathbf{A}) = \{ \lambda_1, \dotsc, \lambda_k \} \) is diagonalizable if and only if there exist matrices \(\{ \mathbf{G}_1, \dotsc, \mathbf{G}_k\} \) such that  \[ \mathbf{A} = \lambda_1 \mathbf{G}_1 + \dotsb + \lambda_k  \mathbf{G}_k \] where the \(\mathbf{G}_i\)'s have the following properties

• \(\mathbf{G}_i\) is the projector onto \(\mathcal{N} (\mathbf{A} - \lambda_i \mathbf{I})  \) along \(\mathcal{R} ( \mathbf{A} - \lambda_i \mathbf{I} ) \). 
• \(\mathbf{G}_i\mathbf{G}_j = 0 \) whenever \( i \neq j \)
• \( \mathbf{G}_1 + \dotsb + \mathbf{G}_k = 1\)

The expansion is known as the spectral decomposition of \(\mathbf{A}\), and the \(\mathbf{G}_i\)'s are called the spectral projectors associated with \(\mathbf{A}\).

Note that being a projector \(\mathbf{G}_i\) is idempotent.

• \(\mathbf{G}_i = \mathbf{G}_i^2\)

And since \(\mathcal{N}(\mathbf{G}_i) = \mathcal{R}(\mathbf{A} - \lambda_i \mathbf{I} ) \) and \(\mathcal{R}(\mathbf{G}_i) = \mathcal{N}(\mathbf{A} - \lambda_i \mathbf{I} ) \), we have the following equivalent complimentary subspaces

• \(  \mathcal{R}(\mathbf{A} - \lambda_i \mathbf{I} ) \oplus \mathcal{N}(\mathbf{A} - \lambda_i \mathbf{I} ) \)
• \(  \mathcal{R}(\mathbf{G}_i) \oplus \mathcal{N}(\mathbf{A} - \lambda_i \mathbf{I} ) \)
• \(  \mathcal{R}(\mathbf{A} - \lambda_i \mathbf{I} ) \oplus  \mathcal{N}(\mathbf{G}_i) \)
• \(  \mathcal{R}(\mathbf{G}_i)  \oplus  \mathcal{N}(\mathbf{G}_i) \)