The Pennsylvania State University, Spring 2021 Stat 415-001, Hyebin Song

Probability Review

Go to course main page

Elements of probability

Sample space $\Omega$: The set of all the outcomes of a random experiment.

Event: Any subset of the sample space $\Omega$. In other words, a collection of some possible outcomes

Probability: A function that takes an event as an input and returns a value from $[0,1]$ as an output, which satisfies the following three properties (Axioms of Probability).

1. For any event $E$, $P(E)\geq 0$.
2. $P(\Omega)=1$.
3. If $A_1,A_2,…$ are disjoint events, then $P(\cup_i A_i)=\sum_iP(A_i)$.

Example Tossing a fair coin twice.

Sample space $\Omega$ = $\{(H,H),(H,T),(T,H),(T,T)\}$

Event : the event of having two heads $\{(H,H)\}$, the event of having at least one tail $\{(T,H),(H,T)\}$, the event of having a head on the first toss $\{(H,T),(H,H)\}$, etc.

Random variables

A random variable is a variable whose outcome is determined by a random experiment.

• More formally, a random variable is a function which takes an element of the sample space $\Omega$ as an input and returns a real number.
• We denote random variables using upper case letters $X(\omega)$, or simply $X$ .

Events can be defined in terms of random variables:

For example, the following are events:

• $\{X=x\}$ for a number $x\in \mathbb{R}$
• $\{a\leq X \leq b\}$ for $-\infty<a<b<\infty$

Therefore it makes sense to write $P(X=x)$ or $P(a\leq X \leq b)$.

More formally,

• $\{X=x\} = \{\omega\in \Omega; X(\omega)=x\}$. Similarly,
• $\{a\leq X \leq b\} =\{\omega \in \Omega; a\leq X(\omega) \leq b\}$.

Also note, $\{a\le X \le b\} = \{\frac{a-c_1}{c_2} \le \frac{X-c_1}{c_2}\le \frac{b-c_1}{c_2}\}$ for any constants $c_1,c_2$ with $c_2>0$.

Example Tossing a fair coin twice. Let $X$ be the number of heads. Compute $P(X=2)$ and $P(X/2=1$).

Sample space $\Omega$ = $\{(H,H),(H,T),(T,H),(T,T)\}$

$P(X=2) = P(\{\omega \in \Omega; X(\omega)=2\}) = P((H,H)) = 1/4$.

The event that $X/2=1$ is the same as the event that $X=2$.

Therefore $P(X/2=1) = P(X=2) = 1/4$.

We often use the indicator variable of an event $A$, $1[A]$, which takes $1$ if $A$ happens and $0$ otherwise.

For example, $1[X=2]$ is a random variable which takes $1$ if $X=2$ and $0$ otherwise.

Class Question: what is the distribution of the random variable $1[X=2]$?

Distribution functions

A (cumulative) distribution function of a random variable $X$ is a function $F_X:\mathbb{R}\rightarrow [0,1]$ such that

It is often easier to handle the pmf or pdf than the cdf, where

• a probability mass function (pmf) when $X$ is discrete,

  • $p_X(t) = P(X=t)$

• a probability density function (pdf) when $X$ is continuous.

  • $f_X(t)$ is a function such that for any $B \subset \mathbb{R}$,

    $$P(X\in B) = \int_{x\in B} f_X(x) dx$$

    In particular,

    $$P(t-\epsilon/2<X<t+\epsilon/2 ) = \int_{t-\epsilon/2}^{t+\epsilon/2}f_X(x)dx \approx \epsilon f_X(t)$$

    for a small $\epsilon>0$.

  • $f_X(t) = \frac{d}{dx}F_X(t)$

• cdf $\leftrightarrow$ pmf/pdf

• Some pmfs or pdfs have names!

Remark We can compute any probability involving $X$ if we know the cdf or pmf/pdf of $X$.

• We specify the distribution of $X$ if we specify the cdf or pmf/pdf of $X$.

  • Notation: $X \sim F_X$ or $X \sim {\rm Name(param)}$

Example

• $F_X(t) = \int_{-\infty}^t \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}s^2} ds, \,\,t\in \mathbb{R}$.
• $p_X(t) = (0.4)^t (0.6)^{1-t},\,\, t\in\{0,1\}$.
• $f_X(t) = 3e^{-3t},\,\, t>0$.

Three main quantities of interest

For a random variable $X$ and any function $g$, three main quantities of interest are

• $P(a\leq g(X) \leq b)$
• $E(g(X))$, average location of $g(X)$
• ${\rm Var} (g(X))$, average spread of $g(X)$

Expectation

For any random variable $X$ with pmf $p_X(t)$ or pdf $f_X(t)$ and any function $g$,

• $E[g(X)] = \sum_{x } g(x) p_X(x)$ if $X$ discrete
• $E[g(X)] = \int_{\mathbb{R}} g(x)f_X(x) dx$ if $X$ continuous

Remark: The expectation of $g(X)$ can be thought of as a "weighted average" of the values that $g(x)$ with the weights $p_X(x)$ or $f_X(x)$.

Properties

• $E[a]=a$ for any constant $a\in \mathbb{R}$.
• $E[ag(X)]=aE[g(X)]$ for any constant $a \in \mathbb{R}$.
• (Linearity of Expectation) $E[f(X)+g(X)]=E[f(X)]+E[g(X)]$

Exercise We toss an unbiased coin twice. Let $N$ be the number of heads from two tosses, and let $X = 1[N=2]$ (the indicator variable for $\{N=2\}$). Find $E[X^2 +3X+2]$.

We have $X \sim Ber(p)$ where $p = P(N=2) = .25$. $E[X^2 +3X+2] = E[X^2]+E[3X]+E[2] = E[X^2]+3E[X]+2$

$E[X^2] = \sum_{x=0}^1 x^2 p_X(x) = p_X(1) = .25$

$E[X] = \sum_{x=0}^1 xp_X(x) = p_X(1) = .25.$

Therefore, $.25+.75+2 = 3$.

Variance

For any random variable $X$ and any function $g$,

Remark: The variance of a random variable $g(X)$ is a measure of how concentrated the distribution of a random variable $g(X)$ is around its mean $E[g(X)]$.

Properties

• ${\rm Var}[a]=0$ and ${\rm Var}(g(X)+a) = {\rm Var}(g(X))$ for any constant $a\in \mathbb{R}$.

• ${\rm Var(g(X))} = E[g(X)^2] - E[g(X)]^2$

• ${\rm Var}[ag(X)]=a^2{\rm Var}[g(X)]$ for any constant $a \in \mathbb{R}$.

• ${\rm Var}[f(X)+g(X)]={\rm Var}[f(X)]+{\rm Var}[g(X)] + 2{\rm Cov}(f(X),g(X))$

   

Exercise We toss an unbiased coin twice. Let $N$ be the number of heads from two tosses, and let $X = 1[N=2]$. Find ${\rm Var}[3X+2]$.

We have $X \sim {\rm Ber}(.25)$.

${\rm Var}[3X+2] = 9{\rm Var}(X) = 9(.25)(.75)$

Some common random variables

• Discrete random variables: Bernoulli(p), Binomial($n,p$), Poisson($\lambda$), Geometric($p$), NegBin($r,p$), ...
• Continuous random variables: Normal($\mu,\sigma^2$), $\chi^2(n)$, Unif($a,b$), Beta($a,b$), Exponential($\lambda$), Gamma($\alpha,\lambda$), $t(d)$, $F(d_1,d_2)$, ...

Example The trunk diameter of a pine tree is normally distributed with a mean of $\mu = 150$ cm and a variance of $\sigma^2 = 900$ cm$^2$. Given that the mean of the trunk diameters of 100 randomly selected pine tree is greater than 156 cm, calculate the conditional probability that the sample mean exceeds 158 cm.

Let $X_i$ be the trunk diameter of the ith selected pine tree. The question is

$P(\bar{X}\ge 158 | \bar{X}\ge 156)$.

By definition,

$$P(\bar{X}\ge 158 | \bar{X}\ge 150) = \frac{P(\bar{X}\ge 156)}{P(\bar{X}\ge 150)}$$

Also we have,

$\bar{X}= \frac{1}{100}(X_1+\dots X_n) \sim N(150, \frac{900}{100})$

Therefore, $P(\bar{X}\ge 158) = P(\frac{\bar{X}-150}{3}\ge \frac{158-150}{3}) = 1-\Phi(8/3).$, and

$P(\bar{X}\ge 156 )= P(\frac{\bar{X}-150}{3}\ge \frac{156-150}{3}) = 1-\Phi(2).$

The answer is $P(\bar{X}\ge 158 | \bar{X}\ge 156) = \frac{1-\Phi(8/3)}{1-\Phi(2)}$.

Two or more random variables

A random vector is a collection of random variables.

Examples

• $(X,Y)$ is a random vector of size $2$.
• $(X_1,\dots,X_n)$ is a random vector of size $n$.

Distribution functions

A joint (cumulative) distribution function of a random vector $(X_1,\dots,X_n)$ is defined by

Remark: by knowing the joint cumulative distribution function, the probability of any event involving $X_1,\dots,X_n$ can be calculated.

As in the univariate case, the distribution of a random vector $(X_1,\dots,X_n)$ is completely specified if we specify the joint distribution function.

Alternatively, we can specify the joint pmf (discrete $X_i$) or pdf (continuous $X_i$).

• Joint pmf $p$: $p(x_1,\dots,x_n) = P(X_1=x_1,\dots,X_n=x_n)$.

• Joint pdf $f$: $f(x_1,\dots,x_n)$ is a function such that for any $B \subset \mathbb{R}^n$,

  $$P((X_1,X_2,\dots,X_n)\in B) = \int\int\dots\int_{(x_1,\dots,x_n)\in B} f(x_1,\dots,x_n)d x_1 dx_2\dots dx_n$$

In particular, when $n=2$,

• Joint pmf $p$: $p(x_1,x_2) = P(X_1=x_1,X_2=x_2)$.

• Joint pdf $f$: $f(x_1,x_2)$ is a function such that for any $B \subset \mathbb{R}^2$,

  $$P((X_1,X_2)\in B) = \int\int_{(x_1,x_2)\in B} f(x_1,x_2)d x_1 dx_2$$

The pmf/pdf of each $X_i$ (a.k.a the marginal pmf/pdf) can be obtained by summing/integrating over other variables.

For example, when $n=2$,

• Marginal pmf $p_{X_1}$: $p_{X_1}(x_1)=\sum_{x_2} p(x_1,x_2)$
• Marginal pdf $f_{X_1}$: $f_{X_1}(x_1) = \int_{\mathbb{R}} f(x_1,x_2)dx_2$

Conditional distributions

Suppose we have a random vector $(X,Y)$. The conditional distribution of $X$ given $Y=y$ tells us the probability distribution of $X$ when we know that $Y$ takes a certain value $y$.

Similarly as before, the probability distribution is completely specified if we know the cdf or pmf/pdf. The cdf or pmf/pdf will usually depend on $y$.

• When $X$, $Y$ are discrete, the conditional pmf of $X$ given $Y=y$ is

  $$p_{X|Y} (x|y) = P(X=x|Y=y) = \frac{p(x,y)}{p_Y(y)}$$

  for $y$ such that $p_Y(y)>0$ .

• When $X$, $Y$ are continuous, the conditional pdf of $X$ given $Y=y$ is

  $$f_{X|Y}(x|y) = \frac{f(x,y)}{f_Y(y)}$$

  for $y$ such that $f_Y(y)>0$.

Example Toss an unbiased coin once. $X$ = number of heads $Y$ = number of tails. Compute the joint pmf and conditional pmf of $X$ given $Y=1$

Recall the definition $p(x,y) = P(X=x,Y=y)$

p(x,y)	x=0	x=1
y=0	p(0,0)=0	p(1,0)=1/2
y=1	p(0,1)=1/2	p(1,1)=0

We have, $p_Y(y) = \sum_x p(x,y)$

• $p_Y(0) = \sum_x p(x,0)=p(0,0)+p(1,0) = 1/2$
• $p_Y(1) = \sum_x p(x,1) = p(0,1)+p(1,1) = 1/2$

Therefore,

$$p_{X|Y}(x|1) = \frac{p(x,1)}{p_Y(1)} = \begin{cases} 0 & {\rm if } \,\, x=1\\ 1 & {\rm if } \,\, x=0 \end{cases}$$

Independence

Random variables $(X_1,\dots,X_n)$ are (jointly) independent if

for all values of $x_1,\dots,x_n \in \mathbb{R}$. Equivalently,

• For discrete random variables, $p(x_1,\dots,x_n)=p_{X_1}(x_1)\dots p_{X_n}(x_n)$.
• For continuous random variables, $f(x_1,\dots,x_n)=f_{X_1}(x_1)\dots f_{X_n}(x_n)$

Example We have a random sample of size $n$, $(X_1,\dots,X_n)$, such that each $X_i$ follows a Poisson distribution with parameter $\lambda$. Write the joint pmf of $(X_1,\dots,X_n)$.

$$p(x_1,\dots,x_n) = \prod_{i=1}^n p_{X_i}(x_i) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{n\lambda}\lambda^{\sum_{i=1}^n x_i}}{\prod_{i=1}^n x_i!}$$

Expectation and Covariance

Suppose we have two discrete random variables $X$,$Y$. In addition to measuring the average location and spread of $X$ or $Y$, we can measure the covariance of $X$ and $Y$.

Covariance is a measure of linear dependence between $X$ and $Y$, defined as

Properties:

for any function $f$ and $g$ such that $f,g:\mathbb{R}^2 \rightarrow \mathbb{R}$,

• (Linearity of expectation) $E[f(X,Y)+g(X,Y)]=E[f(X,Y)]+E[g(X,Y)]$

• ${\rm Var}[X+Y]={\rm Var}[X]+{\rm Var}[Y]+2{\rm Cov}[X,Y]$

• ${\rm Cov}(X,Y) = E[XY]-E[X]E[Y]$.

• ${\rm Cov}(aX,bY) = ab{\rm Cov}(X,Y)$, for any constants $a,b\in\mathbb{R}$.

• ${\rm Cov}(X,X) = {\rm Var}(X)$.

• If $X$ and $Y$ are independent, then ${\rm Cov}[X,Y]=0$

  • ${\rm Cov}(X,Y)=0$ does not imply that $X$ and $Y$ are independent. On the contrary, if ${\rm Cov}(X,Y)\neq 0$, then $X$ and $Y$ are not independent.

• If $X$ and $Y$ are independent, then $E[f(X)g(Y)]=E[f(X)]E[g(Y)]$.

Example Toss an unbiased coin once. $X$ = number of heads $Y$ = number of tails. Compute ${\rm Var}(2X+3Y)$.

$$\begin{align} {\rm Var}(2X+3Y) &= {\rm Var}(2X) + {\rm Var}(3Y) + {\rm} 2{\rm Cov}(2X,3Y)\\ &= 4{\rm Var}(X)+ 9{\rm Var}(Y) + 12{\rm Cov}(X,Y) \end{align}$$

We have,

$X \sim Ber(0.5)$, $Y \sim Ber(0.5)$. ${\rm Var}(X) = {\rm Var}(Y) = 0.25$.

${\rm Cov}(X,Y) = E[XY] - E[X]E[Y] =-0.5^2$.

Therefore,

${\rm Var}(2X+3Y) = 4/4 +9/4 - 12/4 = 1/4$.

Moment generating functions

For a random variable $X$, the moment generating function $M_X(t)$ of the random variable $X$ is defined for all $t\in\mathbb{R}$ by

Similarly, for a random vector $[Math Processing Error](X_1,\dots,X_n)$, the moment generating function $M_{X_1,\dots,X_n}(t)$ of the random vector $(X_1,\dots,X_n)$ is defined for all $t=[t_1,\dots,t_n] \in \mathbb{R}^n$ by

One of the most important results regarding moment generating functions (mgfs) is the following uniqueness theorem, which says that random variables from the same probability distribution have the same moment generating function.

Uniqueness Theorem Let $X$ and $Y$ be two random variables with mgf $M_X$ and $M_Y$. If $M_X(t) = M_Y(t)$ for all $t \in [-C,C]$ for some $C>0$, then $X$ and $Y$ have the same distribution.

Remark Moment generating functions characterize distributions. Similarly as in the case of pmf and pdf, by looking at the moment generating function of $X$, we can identify the distribution of $X$.

• For example, if the mgf of $X$ is $M_X(t) = e^{\lambda(e^t -1)}$, $t \in \mathbb{R}$, then by comparing $M_X$ with mgfs of random variables from known distributions, we can identify that $X$ follows a Poisson distribution with parameter $\lambda$.

Example We have a random vector $(X_1, X_2,..., X_n)$, i.i.d., where each $X_i$, for $i = 1, 2,..., n$, is distributed as ${\rm Poisson}(2)$, i.e., $X_i\sim {\rm Poisson}(2)$. Find the distribution of $X_1+\dots+X_n$.

Let $Z = X_1+X_2+\dots,X_n$. We want to compute the mgf of $Z$.

$M_Z(t) = E[e^{tZ}] = E[e^{t(X_1+\dots+X_n)}] = E[e^{tX_1}e^{tX_2}\dots e^{tX_n}]$

Since $X_1,\dots,X_n$ are independent,

$M_Z(t) = E[e^{tX_1}]E[e^{tX_2}]\dots E[e^{tX_n}] = M_X(t)^n = e^{n\lambda(e^{t}-1)}$

Therefore, $Z\sim {\rm Poisson}(n\lambda)$.

Some limit theorems

1. Weak Law of Large Numbers (WLLN)

   For $X_1,\dots,X_n \sim F$ i.i.d., with $E[X_i] = \mu, \, {\rm Var}(X_i)=\sigma^2<\infty$, such that $\mu,\sigma^2<\infty$,

   for any $\epsilon>0$, we have,

   $$P(|\bar{X}-\mu|\geq \epsilon) \underset{n\rightarrow \infty}{\rightarrow} 0.$$

   In words, with high probability, the sample mean of $n$ i.i.d random variables is close to the population mean.

2. Central Limit Theorem (CLT)

   For $X_1,\dots,X_n \sim F$ i.i.d., with $E[X_i] = \mu, \, {\rm Var}(X_i)=\sigma^2<\infty$, such that $\mu,\sigma^2<\infty$,

   for any $t\in\mathbb{R}$$, we have,

   $$P(\frac{\bar{X}-E[\bar{X}]}{\sqrt{{\rm Var}(\bar{X})}}\le t) = P(\frac{\bar{X}-\mu}{\sigma/\sqrt{n}}\le t) \underset{n\rightarrow \infty}{\rightarrow} \Phi(t) \tag{1}$$

   where $\Phi(t)$ is a cdf of a standard normal distribution. Equivalently,

   $$P (\frac{S_n-E[S_n]}{\sqrt{{\rm Var}(S_n)}}\leq t)=P(\frac{S_n-n\mu}{\sqrt{n}\sigma}\le t) \underset{n\rightarrow \infty}{\rightarrow} \Phi(t) \tag{2}$$

where $S_n = \sum_{i=1}^n X_i$.

In words, the distribution of the standardized sample mean or sum is close to the normal distribution.

We also write (1) and (2) as

or equivalently,