Variance of Sum of Two Random Variables
The variance of the sum of two random variables X and Y is given by:
\begin{align}
\mathbf{var(X + Y) = var(X) + var(Y) + 2cov(X,Y)}
\end{align}
where cov(X,Y) is the covariance between X and Y.
Proof. \begin{align} \mathrm{var(X + Y)} &= \mathbb{E}[\left\lbrace(X + Y) - \mathbb{E}[X + Y]\right\rbrace^2] \\ &= \mathbb{E}[\left\lbrace(X + Y) - (\mathbb{E}[X] + \mathbb{E}[Y])\right\rbrace^2] \\ &= \mathbb{E}[(X + Y - \mathbb{E}[X] - \mathbb{E}[Y])^2] \\ &= \mathbb{E}[( X - \mathbb{E}[X] + Y - \mathbb{E}[Y])^2] \\ &= \mathbb{E}[(X - \mathbb{E}[X])^2 + (Y - \mathbb{E}[Y])^2 \\ \notag & \,\,\,\,\,\,\,\,\,\,\,\, + 2(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])] \\ &= \mathbb{E}[(X - \mathbb{E}[X])^2] + \mathbb{E}[(Y - \mathbb{E}[Y])^2] \\ \notag & \,\,\,\,+ 2\mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])] \\ &= \mathrm{var}(X) + \mathrm{var}(Y) + 2\mathrm{cov(X,Y)} \end{align} which proves (1).(3) follows from the linearity of expectation and (8) follows from the definition of variance and covariance. □
Exercise 2.3 Variance of a sum.
Proof. \begin{align} \mathrm{var(X + Y)} &= \mathbb{E}[\left\lbrace(X + Y) - \mathbb{E}[X + Y]\right\rbrace^2] \\ &= \mathbb{E}[\left\lbrace(X + Y) - (\mathbb{E}[X] + \mathbb{E}[Y])\right\rbrace^2] \\ &= \mathbb{E}[(X + Y - \mathbb{E}[X] - \mathbb{E}[Y])^2] \\ &= \mathbb{E}[( X - \mathbb{E}[X] + Y - \mathbb{E}[Y])^2] \\ &= \mathbb{E}[(X - \mathbb{E}[X])^2 + (Y - \mathbb{E}[Y])^2 \\ \notag & \,\,\,\,\,\,\,\,\,\,\,\, + 2(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])] \\ &= \mathbb{E}[(X - \mathbb{E}[X])^2] + \mathbb{E}[(Y - \mathbb{E}[Y])^2] \\ \notag & \,\,\,\,+ 2\mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])] \\ &= \mathrm{var}(X) + \mathrm{var}(Y) + 2\mathrm{cov(X,Y)} \end{align} which proves (1).(3) follows from the linearity of expectation and (8) follows from the definition of variance and covariance. □
References:
Kevin P. Murphy. Machine Learning: A Probabilistic PerspectiveExercise 2.3 Variance of a sum.