Redoing a reverted merge

The consequence of reverting a merge and how it should be handled when the re-merging.

https://stackoverflow.com/questions/1078146/re-doing-a-reverted-merge-in-git

http://blog.jdwyah.com/2015/07/dealing-with-git-merge-revisions.html

Official source

https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/revert-a-faulty-merge.txt

Reverting a sequence of commits in git

This command lets you revert to any commit in the branch without modifying git history.  It appends the modifications to the current head of the branch.

$ git reset --hard A
$ git reset --soft D # (or ORIG_HEAD or @{1} [previous location of HEAD]), all of which are D
$ git commit

https://stackoverflow.com/questions/1463340/how-to-revert-multiple-git-commits/1470452#1470452

Return to blogging

I just found an awesome way to publish file that can support all the things I wanted to do with relative ease… This is so amazing, where were you all these years. Check out this awesome tool in the link below.

Written with StackEdit.

Notable courses Spring 2018

ECE271C Deep Learning 

• TuTh 11a-12:20p CENTR 214

ECE272B Stocastic Proc/Dynamic Systems

• ThTh 11a-12:20p WLH 2115

Notable IEEE conferences on signal processing and Communications

Communications

• GLOBECOM - Global communications conference. US (Dec.)
• ICC - International conference on communications (May)
• ISWCS - International symposium on wireless communication systems.  (Sept)
• WINCOM - International conference on wireless networks and mobile communications.  (Nov.)
• VNC - Vehicular networking conference (Nov.)
• ICSPCS - International conference on signal processing and communication systems (Dec.)
• WCNC - Wireless communications and networking conference (March)

Signal Processing

• ICASSP - International conference on acoustics speech and signal processing (March)
• GlobalSIP - Global conference on signal and information processing (Dec.)
• MLSP - Workshop on machine learning for signal processing (Sep.)
• SPAWC - Workshop on signal processing advances in wireless communications (July)

---------------------------

• WCSP - International conference on wireless communications and signal processing (China)

Definition of a distribution function

A function \(F: \mathbb{R} \mapsto [0,1]\) satisfying the following properties is a distribution function.

1. \(F\) is right continous;
2. \(F\) is monotone non-decreasing,
3. \(F\) has limits at \(\pm\infty\)
   \begin{align*} F(\infty) &:= \lim_{x\uparrow \infty} F(x) = 1 \\ F(-\infty) &:=\lim_{x\downarrow - \infty} F(x) = 0 \end{align*}

VMWare disk activity reduction methods

See this forum link

and also vmware knowledge base

Also try disabling swap in Linux guest (sudo swapoff -a)

Enabling copy/paste in vmware 12 player

1) sudo apt-get autoremove open-vm-tools

2) Install VMware Tools by following the usual method (Virtual Machine --> Reinstall VMWare Tools)

3) Reboot the VM

4) sudo apt-get install open-vm-tools-desktop

5) Reboot the VM, after the reboot copy/paste and drag/drop will work!

Properties of MMSE and MAP estimator (Bayesian)

The MMSE estimator is the mean of the posterior pdf \(E(x|y)\) of \(x\) given observation \(y\).

1. The estimator is unbiased.
2. The covariance is reduced compared to the a priori information.
3. Commutes over affine transformation.
4. Additivity property for independent data sets.
5. Linear in the Gaussian case.
6. The estimator error is orthogonal to the space spanned by all Y-measurable functions (affine functions being a subset)

The MAP estimator \( \textsf{arg max}_\theta \; p(\theta|x) \) given observation \(x\)

1. Jointly Gaussian case, MAP = MMSE (posterior is Gaussian, hence pdf unimodal and symmetric, mean = mode = median)
2. Do not commute over nonlinear transformation. (Invariant property does not hold, unlike ML)
3. Commutes over linear transformation.

MAP tends to ML when

• Prior is uninformative
• Large amount of information in data compared to prior

Gaussian linear model

Let the observed samples takes on the model

\[  x = H\theta + w\] with prior \(\mathcal{N}(\mu_\theta, C_\theta)\) and noise vector \(\mathcal{N}(0, C_w)\) independent of \(\theta\), then the posterior is Gaussian with mean

\[ E(\theta|x) = \mu_\theta + C_\theta H^T (H C_\theta H^T + C_w)^{-1} (x - H \mu_\theta) \] and covariance \[ C_{\theta|x} = C_\theta - C_\theta H^T (H C_\theta H^T + C_w)^{-1} H C_\theta \] Contrary to the classical Gaussian linear model \(H\) does not need to be full rank.  

In alternative form, 

\[ E(\theta|x) = \mu_\theta + ( C_\theta^{-1} + H^T C_w^{-1} H )^{-1} H^T C_w^{-1} (x - H \mu_\theta)\] and \[ C_{\theta|x} = ( C_\theta^{-1} + H^T C_w^{-1} H )^{-1} \] 

LMMSE estimator \( E^*[X|Y] \)

1. A function of first and second order statistics only.  \[E^*[X|Y] = \mu_x + \Sigma_{xy} \Sigma_{yy}^{-1} ( y - \mu_y) \] (inverse can be replaced with pseudo-inverse if necessary)
2. Jointly Gaussian case, \(E^*[X|Y] = E[X|Y]\)
3. Error orthogonal to subspace spanned by \(Y\)
4. Additivity property \[E^*[X|Y_1,\dotsc,Y_k] = \sum_{j=1}^k E^*[X|Y_j] - (k-1)\mu_x \]

Properties of the exponential family of distributions

From Dasgupta (see link)

One parameter Exponential family

Given the family of distribution \(\{ P_\theta, \theta \in \Theta \subset \mathbb{R} \} \), the pdf of which has the form

\[ f(x|\theta) = h(x) e^{\eta(\theta) T(x) - \psi^*(\theta)} \] 

If \(\eta(\theta)\) is a 1-1 function of  \(\theta\) we can drop \(\theta\) in the discussion.  Thus the family of distributions \(\{ P_\eta, \eta\in \Xi \subset \mathbb{R} \} \) is in canonical form.

\[ f(x|\theta) = h(x) e^{\eta T(x) - \psi(\eta)} \] and define the set 

\[ \mathcal{T} = \{ \eta : e^{\psi(\eta)} < \infty \}\]

\(\eta\) is the natural parameter, and \(\mathcal{T}\) the natural parameter space. 

The family is called the canonical one parameter Exponential family.

[Brown] The family is called full if \(\Xi = \mathcal{T}\), regular if \(\mathcal{T}\) is open.

[Brown] Let K be the convex support of the measure \(\nu\)

The family is minimal if \(\dim \Xi = \dim K = k\)

It is nonsingular if \( Var_\eta (T(X)) > 0 \) for all \(\eta \in \overset{\circ}{\mathcal{T}}\), the interior of \(\mathcal{T}\).

Theorem 1. \(\psi(\eta)\) is a convex function on \(\mathcal{T}\).

Theorem 2. \(\psi(\eta)\) is a cumulant generating function for any \( \eta \in \overset{\circ}{\mathcal{T}}\).

Note: 1st cumulant is the expectation, 2nd,3rd are the central moments (2nd being the variance), 4th and higher order cumulants are neither moments or central moments.

There are more properties...

Multi-parameter Exponential family

Given the family of distribution \(\{ P_\theta, \theta \in \Theta \subset \mathbb{R}^k \} \), the pdf of which has the form

\[ f(x|\theta) = h(x) e^{\sum_{i=1}^k \eta_i(\theta) T_i(x) - \psi^*(\theta)} \] is the k-parameter Exponential family.

Where we reparametrize using \(\eta_i = \eta_i(\theta)\), we have the k-parameter canonical family.

The assumption here is that the dimension of \(\Theta\) and dimension of the image of \(\Theta\) under the map \( (\theta) \rightarrow (\eta_1(\theta),\dotsc,\eta_k(\theta) )\) are equal to \(k\).

The canonical form is 

\[ f(x|\theta) = h(x) e^{\sum_{i=1}^k \eta_i T_i(x) - \psi(\eta)} \]

Theorem 7.  Given a sample having a distribution \(P_\eta, \eta\in\mathcal{T}\) in the canonical k-parameter Exponential family.  with  \( \mathcal{T} = \{ \eta \in \mathbb{R}^k : e^{\psi(\eta)} < \infty \} \)

\(\psi(\eta))\) the partial derivatives of any order exists  for any \(\eta \in \overset{\circ}{\mathcal{T}}\)  

Definition.  The family is full rank if at every \(\eta \in \overset{\circ}{\mathcal{T}}\) the covariance matrix \[ I(\eta) = \frac{\partial^2}{\partial \eta_i \partial \eta_j} \psi(\eta) \ge 0 \] is nonsingular.

Definition/Theorem.  If the family is nonsingular, then the matrix \(I(\eta)\) is called the Fisher information matrix at \(\eta\) (for the natural parameter).

Proof.  For canonical exponential family, we have \(L(x,\eta) = \log p_\eta(x) \doteq \langle \eta, T(x) \rangle - \psi(\eta) \), \(L'(x;\eta) = T(x) - \frac{\partial }{\partial \eta} \psi(\eta) \) and \( L''(x;\eta) = - \frac{\partial^2}{\partial \eta \partial \eta^T} \psi(\eta)\) is constant for fixed \(\eta\), so  

\[ I(\eta) = \frac{\partial^2}{\partial \eta \partial \eta^T} \psi(\eta)\]

Sufficiency and Completeness

Theorem 8.  Suppose a family of distribution \(\mathcal{F} = \{ P_\theta, \theta \in \Theta\} \) belongs to a k-parameter Exponential family and that the "true" parameter space \(\Theta\) has a nonempty interior, then the family \(\mathcal{F}\) is complete.

Theorem 9. (Basu's Theorem for the Exponential Family) In any k-parameter Exponential family \(\mathcal{F}\), with a parameter space \(\Theta\) that has a nonempty interior, the natural sufficient statistic of the family \(T(X)\) and any ancillary statistic \(S(X)\) are independently distributed under each \(\theta \in \Theta\).

MLE of exponential family

Recall, \(L(x,\theta) = \log p_\theta(x) \doteq \langle \theta, T(x) \rangle - \psi(\theta) \).  The solution of the MLE satisfies 

\[ S(\theta) = \left. \frac{\partial}{\partial \theta} L(x;\theta) \right\vert_{\theta = \theta_{ML}}= 0 \; \Longleftrightarrow  \; T(x) = E_{\theta_{ML}} [ T(X) ] \] where \(  \frac{\partial}{\partial \theta} \psi(\theta) =  E_\theta [ T(X) ]  \)

The second derivative gives us 

\[ \frac{\partial^2}{ \partial \theta \partial \theta^T} L(x;\theta) = - I(\theta) = - Cov_\theta [ T(X) ]  \] The right hand side is negative definite for full rank family.  Therefore the log likelihood function is strictly concave in \(\theta\).

Existence of conjugate prior

For likelihood functions within the exponential family, a conjugate prior can be found within the exponential family.  The marginalization to \(p(x) = \int p(x|\theta) p(\theta) d\theta \) is also tractable.

From Casella-Berger.  

Note that the parameter space is the "natural" parameter space.

Local convergence for exponential mixture family

From Redner, Walker 1984

Theorem 5.2.  Suppose that the Fisher information matrix \(I(\Phi)\) is positive definite at the true parameter \(\Phi^*\) and that \(\Phi^* = (\alpha_1^*, \dotsc, \alpha_m^*, \phi_1^*, \dotsc, \phi_m^*)\) is such that \(\alpha_i^* > 0 \text{ for } i = 1,\dotsc,m\).  For \(\Phi^{(0)} \in \Omega\), denote by \(\{\Phi^{(j)}\}_{j=0,1,2,\dotsc}\) the sequence in \(\Omega\) generated by the EM iteration.  Then with probability 1, whenever N is sufficiently large, the unique strongly consistent solution \(\Phi^N = (\alpha_1^N, \dotsc, \alpha_m^N, \phi_1^N, \dotsc, \phi_m^N)\) of the likelihood equations is well defined and there is a certain norm on \(\Omega\) in which  \(\{\Phi^{(j)}\}_{j=0,1,2,\dotsc}\) converges linearly to \(\Phi^N\) whenever \(\Phi^{(0)}\) is sufficiently near \(\Phi^N\), i.e. there is a constant \( 0 \leq \lambda < 1\), for which
\[ \lVert \Phi^{(j+1)} - \Phi^N \rVert \leq \lambda \lVert \Phi^{(j)} - \Phi^N \rVert, \quad j = 0,1,2,\dotsc \] whenever \(\Phi^{(0)}\) is sufficiently near \(\Phi^{N}\).

Differentiability of jump functions

Let
\[ j_n(x) =  \left\lbrace \begin{matrix}{} 0 & \text{if } x < x_n, \\ \theta_n & \text{if } x = x_n, \\ 1 & \text{if } x > x_n , \end{matrix} \right. \] For some \(0\leq \theta_n \leq 1\),  then the jump function is defined as
\[ J(x) = \sum_{n=1}^\infty \alpha_n j_n(x).\] with \(\sum_{n=1}^\infty \alpha_n < \infty\).
Theorem.  If  \(J\) is the jump function, then \(J'(x)\) exists and vanishes almost everywhere.  (non-zero in a set of measure zero, \( E = \{x : J'(x)\neq 0, x\in \mathcal{B} \}, m(E) = 0\) ).

Typical a probability distribution \(F\) is defined as a nondecreasing, right continuous function with \(F(-\infty) = 0,\; F(\infty)=1\).

Properties of Linear and Matrix Operators

Define the adjoint \(A^*\) of operator \(A\) such that
\[ \DeclareMathOperator{\rank}{rank} \langle y, Ax \rangle = \langle A^*y, x \rangle \]
We have the properties

• \(\mathcal{N}(A) = \mathcal{N}(A^*A)\) and \(\mathcal{R}(A^*) = \mathcal{R}(A^*A)\)
• \(\mathcal{N}(A^*) = \mathcal{N}(AA^*)\) and \(\mathcal{R}(A) = \mathcal{R}(AA^*)\)

And noting that \(\dim \mathcal{R}(A) = \dim \mathcal{R}(A^*)\), we have

• \(\rank(A^*A) = \rank ( AA^*) = \rank(A) = \rank(A^*) \)

For matrix operators, dimension of the column space is equal to the dimension of the row space

• column space: \(\dim (\mathcal{R}(A)) = r\)
• row space: \(\dim (\mathcal{R}(A^H)) = r\)
• Nullspace: \(\dim (\mathcal{N}(A)) = n -r\)
• Left nullspace: \(\dim (\mathcal{N}(A^H))= m-r\)

Characterization of matrix \(AB\)

For matrices A and B such that AB exists

1. \(\mathcal{N}(B) \subset \mathcal{N}(AB)\)
2. \(\mathcal{R}(AB) \subset \mathcal{R}(A)\)
3. \(\mathcal{N}(A^*) \subset \mathcal{N}((AB)^*)\)
4. \(\mathcal{R}((AB)^*) \subset \mathcal{R}(B^*)\)

From 2 and 4

\[ \rank(AB) \leq \rank(A), \quad \rank (AB) \leq \rank(B)  \]

Topology and Continuity concepts

Let \(S\) be a subset of a metric space \(M\)

• \(S\) is closed if it contains all its limits.
• \(S\) is open if for each \(p\in S\) there exists an \(r>0\) such that the open ball \(B(p,r)\) is entirely contained in \(S\)
• The complement of an open set is closed and vice versa.

The topology of \(M\) is the collection \(\mathcal{T}\) of all open subsets of \(M\).

\(\mathcal{T}\) has the following properties

• It is closed under arbitrary union of open sets
• It is closed under finite intersections
• \(\emptyset, M\) are open sets.

Corollary

• arbitrary intersection of closed sets is closed
• finite union of closed sets is closed
• \(\emptyset, M\) are closed sets.

A metric space \(M\) is complete if each Cauchy sequence in \(M\) converges to a limit in \(M\).  

• \(\mathbb{R}^n\) is complete

Every compact set is closed and bounded

Continuity of function \(f: M \rightarrow N\)

• The pre-image of each open set in \(N\) is open in \(M\) 
• Preserves convergence sequences under the transformation, i.e.
  \[ f( \lim x_n) = \lim f(x_n)\] for every convergent sequence \(\{x_n\}\)

Conditions that allows interchange of differentiation and integral

With reference to Forland's book

Theorem in Wiki

Continuous mapping theorem

Continuous mapping theorem on Wiki

where (i) is convergence in distribution, (ii) in probability and (iii) almost sure convergence.

Kalman filter

Define the system
\[ x_{k+1} = F_k x_k + G_k w_k + \Gamma_k u_k \quad (1) \\
    z_k = H_k' x_k + v_k \quad (2)\] \(\{u_k\}\) is known, \(x_0 \sim (\bar{x}_0, P_0) \) and \( \{w_k\}, \{v_k\} \) are random sequences with
\[ \begin{bmatrix} w_k \\ v_k \end{bmatrix} \sim
\left ( \begin{bmatrix} 0 \\ 0 \end{bmatrix},
 \begin{bmatrix} Q_k & S_k \\ S_k' & R_k \end{bmatrix} \right )  \]  with \( [w_k' \; v_k']' \) independent of other vectors indexed by \(l \neq k\) and \(x_0\)

One step predictor estimate

First we seek a recursive equation for \[ \hat{x}_{k|k-1} = E[x_k | Z^{k-1}] = E[x_k | \tilde{Z}^{k-1}] \] Define \(\tilde{x}_k = x_k - \hat{x}_{k|k-1}\), note that \(\{\tilde{x}_k\}\) is not an innovations sequence.  Because of the independence of the innovations we have
\[ E[x_{k+1}| \tilde{Z}^k] = E[x_{k+1} | \tilde{z}_k] +  E[x_{k+1}| \tilde{Z}^{k-1}] - \bar{x}_{k+1} \]
Where \( \bar{x}_k = E[x_k]\).  Recall
\[ \DeclareMathOperator{\cov}{cov} E[x_{k+1} | \tilde{z}_k] = \bar{x}_{k+1} + \cov(x_{k+1}, \tilde{z}_k) \cov^{-1}(\tilde{z}_k, \tilde{z}_k) \tilde{z}_k  \] Define the error covariance matrix \( \Sigma_{k|k-1} = E[\tilde{x}_k \tilde{x}_k' ] \) Then
\[ \begin{align*} \cov(x_{k+1}, \tilde{z}_k) &= \cov(F_k x_k + G_k w_k + \Gamma_k u_k, H_k' \tilde{x}_k + v_k) \\  &= E[ (F_k x_k + G_k w_k - F_k \bar{x}_k) (\tilde{x}_k' H_k + v_k') ]  \\ &= E[F_k x_k \tilde{x}_k' H_k ] + G_k S_k \\ &= F_k [ E(\hat{x}_{k|k-1} \tilde{x}_k') + E(\tilde{x}_k \tilde{x}_k')] H_k + G_k S_k \\ &= F_k \Sigma_{k|k-1} H_k + G_k S_k
\end{align*}
\] Observe that \( \hat{z}_{k|k-1} = H' \hat{x}_{k|k-1} \)  and subtracting from (2) gives \( \tilde{z}_k = H_k' \tilde{x}_k + v_k \).  Also note that \( E[\hat{x}_k \tilde{x}_k'] = 0\).  Next
\[ \begin{align*} \cov(\tilde{z}_k,\tilde{z}_k) &= \cov ( H_k' \tilde{x}_k + v_k, H_k' \tilde{x}_k + v_k) \\ &= H_k' \Sigma_{k|k-1} H_k + R_k  = \Omega_k \end{align*} \] We also have
\[ \begin{align*} E[x_{k+1} | \tilde{Z}_{k-1}] &= E[F_k x_k + G_k w_k + \Gamma_k u_k | \tilde{Z}_{k-1}] \\ &= F_k E[x_k | \tilde{Z}_{k=1} ] + \Gamma_k u_k \\ &= F_k \hat{x}_{k|k-1} + \Gamma_k u_k \end{align*} \] Collecting all terms above, the recursion becomes
\[  \hat{x}_{k+1|k} = F_k \hat{x}_{k|k-1} + \Gamma_k u_k + K_k (z_k - H_k' \hat{x}_{k|k-1}) \quad (9) \] with \(K_k = (F_k \Sigma_{k|k-1} H_k + G_k S_k ) \Omega_k^{-1} \)

The recursion of the error covariance is developed next.   From (1),(9), using the identity \(\tilde{x}_{k+1} = x_{k+1} - \hat{x}_{k+1|k} \) and expanding \(z_k\) using (2).
\[ \tilde{x}_{k+1} = (F_k - K_k H_k') \tilde{x}_k + G_k w_k - K_k v_k \] Since \(\tilde{x}_k\) and \( [w_k' v_k']' \) are independent and zero mean, we get
\[ \begin{multline*} E[\tilde{x}_{k+1} \tilde{x}_{k+1}'] = (F_k - K_k H_k') E(\tilde{x}_k \tilde{x}_k') ( F_k - K_k H_k')' \\ \times \begin{bmatrix} G_k &  -K_k \end{bmatrix} \begin{bmatrix} Q_k & S_k \\ S_k' & R_k \end{bmatrix} \begin{bmatrix} G_k' \\ -K_k' \end{bmatrix} \end{multline*} \] or
\[\begin{multline*} \Sigma_{k+1|k} = (F_k - K_k H_k') \Sigma_{k|k-1} (F_k - K_k H_k')' + G_k Q_k G_k' + K_k R_k K_k' \\ - G_k S_k K_k' - K_k S_k' G_k'  \end{multline*} \]
Filtered estimates

Defined in terms of \( \hat{x}_{k+1|k}\) and \( z_{k+1}\)
\[ \begin{align*} \hat{x}_{k+1|k+1} &= E[x_{k+1} | \tilde{Z}_{k+1}] \\ &= E[x_{k+1}|\tilde{z}_{k+1}] + E[x_{k+1}| \tilde{Z}_{k}] - \bar{x}_{k+1} \\ &= \bar{x}_{k+1} + \cov(x_{k+1}, \tilde{z}_{k+1}) \cov^{-1} (\tilde{z}_{k+1}, \tilde{z}_{k+1}) \tilde{z}_{k+1} + \hat{x}_{k+1|k} - \bar{x}_{k+1} \end{align*} \]
Now \[ \begin{align*} \cov(x_{k+1}, \tilde{z}_{k+1}) &= E[ (\tilde{x}_{k+1} + \hat{x}_{k+1|k} - \bar{x}_{k+1}) (\tilde{x}_{k+1} H_{k+1} + v_{k+1}) ] \\ &= E[ \tilde{x}_{k+1} \tilde{x}_{k+1}'] H_{k+1} \\ &= \Sigma_{k+1|k} H_{k+1} \end{align*} \]
From early results, we have \[ \cov(\tilde{z}_{k+1}, \tilde{z}_{k+1}) = H_{k+1}' \Sigma_{k+1|k} H_{k+1} + R_{k+1} = \Omega_{k+1}\]  The measurement-update (filtered estimate) is
\[ \hat{x}_{k+1|k+1} = \hat{x}_{k+1|k} + \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} (z_{k+1} - H_{k+1}' \hat{x}_{k+1|k}) \quad (6) \]
Define the uncorrelated input noise \( \tilde{w}_k = w_k - \hat{w}_k = w_k - S_k R_k^{-1} v_k\) such that
\[ \begin{bmatrix} \tilde{w}_k \\ v_k \end{bmatrix} \sim
\left ( \begin{bmatrix} 0 \\ 0 \end{bmatrix},
 \begin{bmatrix} Q_k - S_k R_k^{-1}S_k' & 0 \\  0 & R_k \end{bmatrix} \right )  \]
then we have
\[ \begin{align*} x_{k+1} &= F_k x_k + G_k \tilde{w}_k + G_k S_k R_k^{-1} v_k + \Gamma_k u_k \\ &= (F_k - G_k S_k R_k^{-1} H_k') x_k + G_k \tilde{w}_k + \Gamma_k u_k + G_k S_k R_k^{-1} z_k \end{align*} \] using the fact \(v_k = z_k - H_k' x_k\) .Noting that \( E[\tilde{w}_k v_k'] = 0 \),  the time update equation becomes
\[ \hat{x}_{k+1|k} = (F_k - G_k S_k R_k^{-1} H_k') \hat{x}_{k|k} + \Gamma_k u_k + G_k S_k R_k^{-1} z_k \quad (5) \]
Error covariance for filtered estimates
The error covariance is
\[ \Sigma_{k|k} = E[ (x_k - \hat{x}_{k|k}) (x_k - \hat{x}_{k|k})'] \]
From (6) we have
\[ (x_{k+1} - \hat{x}_{k+1|k+1}) + \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} \tilde{z}_{k+1} = x_{k+1} - \hat{x}_{k+1|k}  \]
By the orthogonality principle, \(x_{k+1} - \hat{x}_{k+1|k+1} \) is orthogonal to \(\tilde{z}_{k+1}\).  Therefore,
\[ \Sigma_{k+1|k+1} + \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} H_{k+1}' \Sigma_{k+1|k} = \Sigma_{k+1|k} \] or
\[ \Sigma_{k+1|k+1} = \Sigma_{k+1|k} -  \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} H_{k+1}' \Sigma_{k+1|k} \]
Lastly, we obtain the time-update error covariance, subtracting (5) from (1)
\[ x_{k+1} - \hat{x}_{k+1|k} = (F_k - G_k S_k R_k^{-1} H_k') (x_k - \hat{x}_{k|k}) + G_w \tilde{w}_k \] and using the orthogonality of \(\tilde{w}_k\) and \(x_k - \hat{x}_{k|k}\), we obtain
\[ \begin{multline*} \Sigma_{k+1|k} = (F_k - G_k S_k R_k ^{-1} H_k') \Sigma_{k|k} (F_k - G_k S_k R_k^{-1} H_k')' \\ + G_k(Q_k - S_k R_k^{-1} S_k') G_k \end{multline*} \]
Summary

Measurement update
\[\begin{align*} \hat{x}_{k+1|k+1} &= \hat{x}_{k+1|k} H_{k+1}' \Omega_{k+1}^{-1} ( z_{k+1} - H_{k+1}' \hat{x}_{k+1|k}) \\  \Sigma_{k+1|k+1} &=  \Sigma_{k+1|k} - \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} H_{k+1}' \Sigma_{k+1|k} \\ \Omega_{k+1} &= H_{k+1}' \Sigma_{k+1|k} H_{k+1} + R_{k+1} \end{align*} \]
Time update
\[ \begin{align*} \hat{x}_{k+1|k} &= ( F_k - G_k S_k R_k^{-1} H_k') \hat{x}_{k|k} + \Gamma_k u_k + G_k S_k R_k^{-1} z_k \\ \Sigma_{k+1|k} &= (F_k - G_k S_k R_k^{-1} H_k') \Sigma_{k|k} (F_k - G_k S_k R_k^{-1} H_k')' + G_k (Q_k - S_k R_k^{-1} S_k') G_k' \end{align*} \]
Time update with \(S_k = 0\)
\[ \begin{align*} \hat{x}_{k+1|k} &= F_k \hat{x}_{k|k} + \Gamma_k u_k \\ \Sigma_{k+1|k} &= F_k \Sigma_{k|k} F_k' + G_k Q_k G_k' \end{align*} \]
Combined update with \(S_k = 0\) for filtered state:
\[ \begin{align*} \hat{x}_{k+1|k+1} &= F_k \hat{x}_{k|k} + L_{k+1} ( z_{k+1} - H_{k+1}' F_k \hat{x}_{k|k} - H_{k+1}' \Gamma_k u_k)  \\  L_{k+1} &= \Sigma_{k+1|k} H_{k+1} \Omega_{k+1}^{-1} \\  \Omega_{k+1} &= H_{k+1}' \Sigma_{k+1|k} H_{k+1} + R_{k+1}\end{align*} \]

Innovations sequence

Definition 
Suppose \( \{z_k\} \) is a sequence of jointly Gaussian random elements.   The innovations process \(\{\tilde{z}_k\} \) is such that \(\tilde{z}_k\) consists of that part of \(z_k\) containing new information not carried in \(z_{k-1}, z_{k-2}, \dotsc\).
\[ \tilde{z}_k = z_k - E[z_k | z_0, \dotsc, z_{k-1} ]  = z_k - E[z_k | Z^{k-1}] \] with \( \tilde{z}_0 = z_0 - E[z_0] \).

Properties

1. \(\tilde{z}_k\) independent of \( z_0, \dotsc, z_{k-1}\) by definition
2. (1) implies \(E[ \tilde{z}_k' \tilde{z}_l] = 0, l \neq k \)
3. \(E[z_k | Z^{k-1}]\) is a linear combination of \(z_0, \dotsc, z_{k-1}\)
4. The sequence \(\{\tilde{z}_k\} \) can be obtained from \(\{z_k\} \) by a causal linear operation.
5. The sequence \(\{z_k\} \) can be reconstructed from \(\{\tilde{z}_k\} \) by a causal linear operation. 
6. (4) and (5) implies \( E[z_k | Z^{k-1}] = E[z_k | \tilde{Z}^{k-1}] \) or more generally  \( E[w | Z^{k-1}] = E[w | \tilde{Z}^{k-1}] \) for jointly Gaussian \(w, \{z_k\} \)
7. For zero mean Gaussian \(\tilde{x}_k\), \(\tilde{z}_k\), we have \[ E[x_k|Z^{k-1}] = E[x_k|\tilde{Z}^{k-1}] = E[x_k| \tilde{z}_0] + \dotsb + E[x_k| \tilde{z}_{k-1}]    \]

Properties of the exponential family distributions

Given exponential family \( \mathcal{P}=\{p_\theta(x) | \theta \in \Theta \} \), where
\[ p_\theta(x) = h(x) \exp ( q^T(\theta) T(x) - b(\theta)  )  I_{supp}(x), \quad Z = \exp(- b(\theta)) \]
Regular family (gives you completeness)
Conditions for regularity,

1. support \(p_\theta(x)\) independent of \(\theta\)
2. finite partition function \(Z(\theta) < \infty,\; \forall \theta\)
3. Interior of parameter space is solid, \( \mathring{\Theta} \neq \emptyset \), 
4. Interior of natural parameter space is solid \( \mathring{\mathcal{Q}} \neq \emptyset \)
5. Statistic vector function and the constant function are linearly independent.  i.e. \( [1, T_1(x),\dotsc,T_K(x)] \) linear indep. (gives you minimal statistic)
6. twice differentiable \( p_\theta(x) \) 

Curved family (only know statistic is minimal)
An exponential family where the dimension of the vector parameter \(\mathbf{\theta}=(\theta_1,\dotsc,\theta_r)\) is less than the dimension of the natural statistic \(\mathbf{T}(\mathbf{x}) \) is called a curved family.

Identifiability of parameter vector \( \mathbf{\theta} \).
When statistic is minimal, then it is a matter of ensuring \(q: \Theta \mapsto \mathcal{Q} \) defines a 1-1 mapping from desired parameter space to natural parameter space.

Invariance and carry over properties of MLE

Review: Asymptotic properties of MLE

• Asymptotically efficient (attains CRLB as \(N\rightarrow\infty\))
• Asymptotically Gaussian (asymptotically normality)
• Asymptotically Unbiased
• Consistent (weakly and strongly)

First, the invariance property of MLE

The MLE of the parameter \(\alpha = g(\theta)\), where the PDF \(p(x;\theta)\) is paremeterized by \(\theta\), is given by
\[ \hat{\alpha} = g(\hat{\theta})\] where \(\hat{\theta}\) is the MLE of \(\theta\).

Consistency (in class) is defined as the weak convergence of the sequence of estimates to the true parameter as N gets large.

If \(g(\theta)\) is continuous in \(\theta\), the convergence properties (esp. convergence in prob.) carry over, i.e. the consistency of the estimator \(g(\hat{\theta})\)

However, biasedness of the estimator \(g(\hat{\theta})\) depends on the convexity of \(g\) and does not carry over from \(\hat{\theta}\).

Other properties of MLE

• If an efficient estimator exists, the ML method will produce it.
• Unlike the MVU estimator, MLE can be biased
• Note: CRLB applies to unbiased estimators, so when estimator is biased, it is possible it has variance smaller than \(I^{-1}(\theta)\)