12 Covariance

12.1 Definition of Covariance

For any two random variables \(X\) and \(Y\), the covariance of \(X\) and \(Y\) is defined as

\[\mathrm{Cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)]\]

12.2 Theorems on Covariance

12.2.1 Theorem

Let \(X\) be a random variable. Then \[\mathrm{Cov}(X,X) = V(X)\]

Proof:

\[\begin{aligned} \mathrm{Cov}(X,X) &= E[(X-\mu)(X-\mu)] \\ &= E[(X-\mu)^2] \\ &= V(X) \end{aligned}\]

12.2.2 Theorem

Let \(X\) and \(Y\) be random variables. Then

\[\mathrm{Cov(X,Y)} = E(XY)-E(X)E(Y)\]

Proof:

\[\begin{aligned} \mathrm{Cov(X,Y)} &= E[(X-\mu_x)(Y-\mu_Y)] \\ &= E[XY - X\mu_y - Y\mu_X + \mu_X\mu_Y] \\ &= E(XY) - E(X)\mu_Y - \mu_XE(Y) + \mu_X\mu_Y \\ &= E(XY) - E(X)E(Y) - E(X)E(Y) + E(X)E(Y) \\ &= E(XY) - 2E(X)E(Y) + E(X)E(Y) \\ &= E(XY) - E(X)E(Y) \end{aligned}\]

12.2.3 Covariance

Let \(X\) and \(Y\) be random variables and let \(a\) and \(b\) be constants. Then

\[\mathrm{Cov(aX,bY)} = ab \mathrm{Cov}(X,Y)\]

Proof:

\[\begin{aligned} \mathrm{Cov}(aX,bY) &= E(aXbY) - E(aX)E(bY) \\ &= abE(XY) - abE(X)E(Y) \\ &= ab[E(XY) - E(X)E(Y)] \\ &= ab \cdot \mathrm{Cov}(X,Y) \end{aligned}\]

12.2.4 Theorem

Let \(X_1 , X_2 , \ldots , X_n\) be random variables with \(E(X_i) = \mu_i\) for \(i = 1,2,\ldots,n\) and let \(Y_1,Y_2,\ldots,Y_m\) be random variables with \(E(Y_j) = \phi_j\) for \(j=1,2,\ldots,m\). Also, let \(a_1,a_2,\ldots,a_n\) and \(b_1,b_2,\ldots,b_m\) be constants.\

If \(U_1 = \sum\limits_{i=1}^{n}a_iX_i\) and \(U_2 = \sum\limits_{i=1}^{m}b_iY_i\), then

\[\mathrm{Cov}(U_1,U_2) = \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{m}a_ib_j \cdot \mathrm{Cov}(X_i,Y_j)\]

Proof:

\[\begin{aligned} \mathrm{Cov}(U_1,U_2) &= E[(U_1-E(U_1))(U_2-E(U_2))] \\ &= E\Big[\big(\sum\limits_{i=1}^{n}a_iX_i-\sum\limits_{i=1}^{n}a_i\mu_i\big) \big(\sum\limits_{j=1}^{m}b_jY_j-\sum\limits_{j=1}^{m}b_j\phi_j\big)\Big] \\ &= E\Big[\big(\sum\limits_{i=1}^{n}a_i(X_i-\mu_i)\big)\big(\sum\limits_{j=1}^{m}b_j(Y_j-\phi_j)\big)\Big] \\ &= E\Big[\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{m}a_ib_j(X_i-\mu_i)(Y_j-\phi_j)\Big] \\ &= \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{m}a_ib_jE[(X_i-\mu_i)(Y_j-\phi_j)] \\ &= \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{m}a_ib_j\ \mathrm{Cov}(X_i,Y_j) \end{aligned}\]

12.2.5 Theorem

Let \(X_1,X_2,\ldots,X_n\) be random variables with \(E(X_i)=\mu_i\) for \(i=1,2,\ldots,n\) and let \(a_1,a_2,\ldots,a_n\) be constants.

If \(Y = \sum\limits_{i=1}^{n}a_iX_i\) then

\[V(Y) = \sum\limits_{i=1}^{n}a_i^2V(X_i)+2\sum\limits_{\ \ i<}\sum\limits_{j\ \ }a_ia_j\ \mathrm{Cov}(X_i,X_j)\]

Proof:

\[\begin{aligned} V(Y) &= E[(Y-\mu_Y)^2] \\ &= E[(Y-\mu_Y)(Y-\mu_Y)] \\ &= E\Big[\big(\sum\limits_{i=1}^{n}a_iX_i-a_i\mu_i\big) \big(\sum\limits_{n=1}^{n}a_jX_j-a_j\mu_j\big)\Big] \\ &= \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n}a_ia_jE[(X_i-\mu_i)(X_j-\mu_j)] \\ &= \sum\limits_{i=1}^{n}a_i^2\ \mathrm{Cov}(X_i,X_i)+ \sum\limits_{\ \ i\neq}\sum\limits_{j\ \ \ \ }a_ia_jE[(X_i-\mu_i)(X_j-\mu_j)] \\ &= \sum\limits_{i=1}^{n}a_i^2V(X_i)+ 2\sum\limits_{\ \ i<}\sum\limits_{j\ \ \ \ }a_ia_j\ \mathrm{Cov}(X_i,X_j) \end{aligned}\]