Properties of MMSE and MAP estimator (Bayesian)

The MMSE estimator is the mean of the posterior pdf $E(x|y)$ of $x$ given observation $y$.

1. The estimator is unbiased.
2. The covariance is reduced compared to the a priori information.
3. Commutes over affine transformation.
4. Additivity property for independent data sets.
5. Linear in the Gaussian case.
6. The estimator error is orthogonal to the space spanned by all Y-measurable functions (affine functions being a subset)

The MAP estimator $\textsf{arg max}_\theta \; p(\theta|x)$ given observation $x$

1. Jointly Gaussian case, MAP = MMSE (posterior is Gaussian, hence pdf unimodal and symmetric, mean = mode = median)
2. Do not commute over nonlinear transformation. (Invariant property does not hold, unlike ML)
3. Commutes over linear transformation.

MAP tends to ML when

• Prior is uninformative
• Large amount of information in data compared to prior

Gaussian linear model

Let the observed samples takes on the model

$$x = H\theta + w$$ with prior $\mathcal{N}(\mu_\theta, C_\theta)$ and noise vector $\mathcal{N}(0, C_w)$ independent of $\theta$, then the posterior is Gaussian with mean

$$E(\theta|x) = \mu_\theta + C_\theta H^T (H C_\theta H^T + C_w)^{-1} (x - H \mu_\theta)$$ and covariance $$C_{\theta|x} = C_\theta - C_\theta H^T (H C_\theta H^T + C_w)^{-1} H C_\theta$$ Contrary to the classical Gaussian linear model $H$ does not need to be full rank.  

In alternative form, 

$$E(\theta|x) = \mu_\theta + ( C_\theta^{-1} + H^T C_w^{-1} H )^{-1} H^T C_w^{-1} (x - H \mu_\theta)$$ and $$C_{\theta|x} = ( C_\theta^{-1} + H^T C_w^{-1} H )^{-1}$$  

LMMSE estimator $E^*[X|Y]$

1. A function of first and second order statistics only.   $$E^*[X|Y] = \mu_x + \Sigma_{xy} \Sigma_{yy}^{-1} ( y - \mu_y)$$ (inverse can be replaced with pseudo-inverse if necessary)
2. Jointly Gaussian case, $E^*[X|Y] = E[X|Y]$
3. Error orthogonal to subspace spanned by $Y$
4. Additivity property $$E^*[X|Y_1,\dotsc,Y_k] = \sum_{j=1}^k E^*[X|Y_j] - (k-1)\mu_x$$