Processing math: 100%

MathJax

Monday, August 29, 2016

Properties of Linear and Matrix Operators

Define the adjoint A of operator A such that
y,Ax=Ay,x
We have the properties

  • N(A)=N(AA) and R(A)=R(AA)
  • N(A)=N(AA) and R(A)=R(AA)
And noting that dimR(A)=dimR(A), we have
  • rank(AA)=rank(AA)=rank(A)=rank(A)

For matrix operators, dimension of the column space is equal to the dimension of the row space
  • column space: dim(R(A))=r
  • row space: dim(R(AH))=r
  • Nullspace: dim(N(A))=nr
  • Left nullspace: dim(N(AH))=mr
Characterization of matrix AB
For matrices A and B such that AB exists
  1. N(B)N(AB)
  2. R(AB)R(A)
  3. N(A)N((AB))
  4. R((AB))R(B)
From 2 and 4
rank(AB)rank(A),rank(AB)rank(B)

Thursday, August 25, 2016

Topology and Continuity concepts

Let S be a subset of a metric space M

  • S is closed if it contains all its limits.
  • S is open if for each pS there exists an r>0 such that the open ball B(p,r) is entirely contained in S
  • The complement of an open set is closed and vice versa.
The topology of M is the collection T of all open subsets of M.

T has the following properties
  • It is closed under arbitrary union of open sets
  • It is closed under finite intersections
  • ,M are open sets.
Corollary
  • arbitrary intersection of closed sets is closed
  • finite union of closed sets is closed
  • ,M are closed sets.
A metric space M is complete if each Cauchy sequence in M converges to a limit in M.  
  • Rn is complete
Every compact set is closed and bounded

Continuity of function f:MN
  • The pre-image of each open set in N is open in M 
  • Preserves convergence sequences under the transformation, i.e.
    f(limxn)=limf(xn) for every convergent sequence {xn}

Wednesday, August 17, 2016

Continuous mapping theorem

Continuous mapping theorem on Wiki


where (i) is convergence in distribution, (ii) in probability and (iii) almost sure convergence.


Friday, August 12, 2016

Kalman filter

Define the system
xk+1=Fkxk+Gkwk+Γkuk(1)zk=Hkxk+vk(2) {uk} is known, x0(ˉx0,P0) and {wk},{vk} are random sequences with
[wkvk]([00],[QkSkSkRk])  with [wkvk] independent of other vectors indexed by lk and x0

One step predictor estimate

First we seek a recursive equation for ˆxk|k1=E[xk|Zk1]=E[xk|˜Zk1] Define ˜xk=xkˆxk|k1, note that {˜xk} is not an innovations sequence.  Because of the independence of the innovations we have
E[xk+1|˜Zk]=E[xk+1|˜zk]+E[xk+1|˜Zk1]ˉxk+1
Where ˉxk=E[xk].  Recall
E[xk+1|˜zk]=ˉxk+1+cov(xk+1,˜zk)cov1(˜zk,˜zk)˜zk Define the error covariance matrix Σk|k1=E[˜xk˜xk] Then
cov(xk+1,˜zk)=cov(Fkxk+Gkwk+Γkuk,Hk˜xk+vk)=E[(Fkxk+GkwkFkˉxk)(˜xkHk+vk)]=E[Fkxk˜xkHk]+GkSk=Fk[E(ˆxk|k1˜xk)+E(˜xk˜xk)]Hk+GkSk=FkΣk|k1Hk+GkSk Observe that ˆzk|k1=Hˆxk|k1  and subtracting from (2) gives ˜zk=Hk˜xk+vk.  Also note that E[ˆxk˜xk]=0.  Next
cov(˜zk,˜zk)=cov(Hk˜xk+vk,Hk˜xk+vk)=HkΣk|k1Hk+Rk=Ωk We also have
E[xk+1|˜Zk1]=E[Fkxk+Gkwk+Γkuk|˜Zk1]=FkE[xk|˜Zk=1]+Γkuk=Fkˆxk|k1+Γkuk Collecting all terms above, the recursion becomes
ˆxk+1|k=Fkˆxk|k1+Γkuk+Kk(zkHkˆxk|k1)(9) with Kk=(FkΣk|k1Hk+GkSk)Ω1k

The recursion of the error covariance is developed next.   From (1),(9), using the identity ˜xk+1=xk+1ˆxk+1|k and expanding zk using (2).
˜xk+1=(FkKkHk)˜xk+GkwkKkvk Since ˜xk and [wkvk] are independent and zero mean, we get
E[˜xk+1˜xk+1]=(FkKkHk)E(˜xk˜xk)(FkKkHk)×[GkKk][QkSkSkRk][GkKk] or
Σk+1|k=(FkKkHk)Σk|k1(FkKkHk)+GkQkGk+KkRkKkGkSkKkKkSkGk
Filtered estimates

Defined in terms of ˆxk+1|k and zk+1
ˆxk+1|k+1=E[xk+1|˜Zk+1]=E[xk+1|˜zk+1]+E[xk+1|˜Zk]ˉxk+1=ˉxk+1+cov(xk+1,˜zk+1)cov1(˜zk+1,˜zk+1)˜zk+1+ˆxk+1|kˉxk+1
Now cov(xk+1,˜zk+1)=E[(˜xk+1+ˆxk+1|kˉxk+1)(˜xk+1Hk+1+vk+1)]=E[˜xk+1˜xk+1]Hk+1=Σk+1|kHk+1
From early results, we have cov(˜zk+1,˜zk+1)=Hk+1Σk+1|kHk+1+Rk+1=Ωk+1  The measurement-update (filtered estimate) is
ˆxk+1|k+1=ˆxk+1|k+Σk+1|kHk+1Ω1k+1(zk+1Hk+1ˆxk+1|k)(6)
Define the uncorrelated input noise ˜wk=wkˆwk=wkSkR1kvk such that
[˜wkvk]([00],[QkSkR1kSk00Rk])
then we have
xk+1=Fkxk+Gk˜wk+GkSkR1kvk+Γkuk=(FkGkSkR1kHk)xk+Gk˜wk+Γkuk+GkSkR1kzk using the fact vk=zkHkxk .Noting that E[˜wkvk]=0,  the time update equation becomes
ˆxk+1|k=(FkGkSkR1kHk)ˆxk|k+Γkuk+GkSkR1kzk(5)
Error covariance for filtered estimates
The error covariance is
Σk|k=E[(xkˆxk|k)(xkˆxk|k)]
From (6) we have
(xk+1ˆxk+1|k+1)+Σk+1|kHk+1Ω1k+1˜zk+1=xk+1ˆxk+1|k
By the orthogonality principle, xk+1ˆxk+1|k+1 is orthogonal to ˜zk+1.  Therefore,
Σk+1|k+1+Σk+1|kHk+1Ω1k+1Hk+1Σk+1|k=Σk+1|k or
Σk+1|k+1=Σk+1|kΣk+1|kHk+1Ω1k+1Hk+1Σk+1|k
Lastly, we obtain the time-update error covariance, subtracting (5) from (1)
xk+1ˆxk+1|k=(FkGkSkR1kHk)(xkˆxk|k)+Gw˜wk and using the orthogonality of ˜wk and xkˆxk|k, we obtain
Σk+1|k=(FkGkSkR1kHk)Σk|k(FkGkSkR1kHk)+Gk(QkSkR1kSk)Gk
Summary

Measurement update
ˆxk+1|k+1=ˆxk+1|kHk+1Ω1k+1(zk+1Hk+1ˆxk+1|k)Σk+1|k+1=Σk+1|kΣk+1|kHk+1Ω1k+1Hk+1Σk+1|kΩk+1=Hk+1Σk+1|kHk+1+Rk+1
Time update
ˆxk+1|k=(FkGkSkR1kHk)ˆxk|k+Γkuk+GkSkR1kzkΣk+1|k=(FkGkSkR1kHk)Σk|k(FkGkSkR1kHk)+Gk(QkSkR1kSk)Gk
Time update with Sk=0
ˆxk+1|k=Fkˆxk|k+ΓkukΣk+1|k=FkΣk|kFk+GkQkGk
Combined update with Sk=0 for filtered state:
ˆxk+1|k+1=Fkˆxk|k+Lk+1(zk+1Hk+1Fkˆxk|kHk+1Γkuk)Lk+1=Σk+1|kHk+1Ω1k+1Ωk+1=Hk+1Σk+1|kHk+1+Rk+1

Wednesday, August 10, 2016

Innovations sequence

Definition 
Suppose {zk} is a sequence of jointly Gaussian random elements.   The innovations process {˜zk} is such that ˜zk consists of that part of zk containing new information not carried in zk1,zk2,.
˜zk=zkE[zk|z0,,zk1]=zkE[zk|Zk1] with ˜z0=z0E[z0].

Properties

  1. ˜zk independent of z0,,zk1 by definition
  2. (1) implies E[˜zk˜zl]=0,lk
  3. E[zk|Zk1] is a linear combination of z0,,zk1
  4. The sequence {˜zk} can be obtained from {zk} by a causal linear operation.
  5. The sequence {zk} can be reconstructed from {˜zk} by a causal linear operation. 
  6. (4) and (5) implies E[zk|Zk1]=E[zk|˜Zk1] or more generally  E[w|Zk1]=E[w|˜Zk1] for jointly Gaussian w,{zk}
  7. For zero mean Gaussian ˜xk˜zk, we have E[xk|Zk1]=E[xk|˜Zk1]=E[xk|˜z0]++E[xk|˜zk1]



Friday, August 05, 2016

Properties of the exponential family distributions

Given exponential family P={pθ(x)|θΘ}, where
pθ(x)=h(x)exp(qT(θ)T(x)b(θ))Isupp(x),Z=exp(b(θ))
Regular family (gives you completeness)
Conditions for regularity,

  1. support pθ(x) independent of θ
  2. finite partition function Z(θ)<,θ
  3. Interior of parameter space is solid, ˚Θ
  4. Interior of natural parameter space is solid ˚Q
  5. Statistic vector function and the constant function are linearly independent.  i.e. [1,T1(x),,TK(x)] linear indep. (gives you minimal statistic)
  6. twice differentiable pθ(x) 

Curved family (only know statistic is minimal)
An exponential family where the dimension of the vector parameter θ=(θ1,,θr) is less than the dimension of the natural statistic T(x) is called a curved family.

Identifiability of parameter vector θ.
When statistic is minimal, then it is a matter of ensuring q:ΘQ defines a 1-1 mapping from desired parameter space to natural parameter space.