The Pennsylvania State University, Spring 2021 Stat 415-001, Hyebin Song

Point Estimation

Go to course main page

Introduction to Point Estimation

Learning objectives

1. Understand the goal of point estimation
2. Understand bias and mean square error of point estimators

Recap

Setting: $X_1,\dots,X_n\sim F_\theta$, i.i.d., where $\theta$ is an unknown value in the parameter space $\Omega$.

• we have a sample $(X_1,\dots,X_n)$ where $X_i$ is independent and identically distributed (i.i.d), and the distribution of $X_i$ is known up to an unknown parameter value. That is, we know that the distribution function of $X_i$ is in the collection of distribution functions $\{F_\theta; \theta \in \Omega\}$, but we do not know which one because we do not know the value of $\theta$.
• the goal is to choose a "plausible" $\theta$ in $\Omega$ based on the sample $(X_1,\dots,X_n)$.

Recall the definitions:

1. A point estimator of $\theta$:
2. A point estimate of $\theta$:

Notation:

Recall that for a continuous random variable $X$ with a pdf $f_\theta$, the probability of an event ${X\in B}$, the expectation of $X$, and the variance of $X$ are computed as

• $P(X\in B) = \int_B f_\theta(x)dx$
• $E[X] = \mu= \int_\mathbb{R} x f_\theta(x)dx$
• ${\rm Var}[X]= \int_\mathbb{R} (x-\mu)^2 f_\theta(x) dx$

Note: for a discrete random variable $X$, we can replace the pdf with a pmf and the integrations with summations.

We sometimes use subscripts and write $P_\theta(X\in B)$, $E_\theta[X]$ and ${\rm Var}_\theta[X]$ to emphasize that the expectation and variance are computed using a pdf with a particular value of $\theta$.

The bias and mean squared error of point estimators

Definition (Biased and unbiased estimators)

• ${\rm Bias}(\widehat{\theta}; \theta) = E_\theta[\widehat{\theta}(X_1,\dots,X_n)]-\theta$
• If ${\rm Bias}(\widehat{\theta}; \theta$) $=0$ for all $\theta \in \Omega$, then $\widehat{\theta}(X_1,\dots,X_n)$ is unbiased. Otherwise, the estimator is biased.

Example Suppose we have $X_1,\dots,X_n \sim N(\mu,1)$, i.i.d. where the parameter space $\Omega = \mathbb{R}$. We consider the following three two estimators

Are three estimators $\widehat{\theta}_1,\widehat{\theta}_2, \widehat{\theta}_3$ unbiased?

For any $\mu \in \mathbb{R}$,

1. we have ${\rm Bias}(\bar{X};\mu) = E_\mu[\bar{X}]-\mu =0$ . Therefore $\bar{X}$ is an unbiased estimator of $\mu$.
2. ${\rm Bias}((X_1+X_2)/2;\mu) = E_\mu[(X_1+X_2)/2]-\mu =0$ . Therefore $(X_1+X_2)/2$ is an unbiased estimator of $\mu$.
3. ${\rm Bias}(\bar{X}+1;\mu) = E_\mu[\bar{X}+1]-\mu =1\neq 0$ Therefore $\bar{X}+1$ is not an unbiased estimator of $\mu$.

Mean Squared Error (MSE)

• Definition: ${\rm MSE}(\widehat{\theta}; \theta) = E_\theta[(\widehat{\theta}(X_1,\dots,X_n)-\theta)^2]$

• MSE captures both bias and variance

  ${\rm MSE}(\widehat{\theta};\theta) = {\rm Var}_\theta(\widehat{\theta})+{\rm Bias}(\widehat{\theta};\theta)^2$

  Proof. $E_\theta[(\widehat{\theta}-\theta)^2] = E_\theta[(\widehat{\theta}-E_\theta(\hat{\theta})+E_\theta(\hat{\theta})-\theta)^2]=E_\theta[(\widehat{\theta}-E_\theta(\hat{\theta}))^2]+(E_\theta(\hat{\theta})-\theta)^2$

• In particular, if an estimator $\widehat{\theta}$ is unbiased, ${\rm MSE}(\widehat{\theta};\theta) = {\rm Var}_\theta(\widehat{\theta})$.

Example (cont'd) Compute MSE of $\widehat{\theta}_1,\widehat{\theta}_2, \widehat{\theta}_3$.

1. ${\rm MSE}(\widehat{\theta}_1;\mu) = {\rm Var}_\mu(\bar{X}) = \frac{1}{n}$
2. ${\rm MSE}(\widehat{\theta}_2;\mu) = {\rm Var}_\mu(\frac{(X_1+X_2)}{2}) = \frac{1}{2}$
3. ${\rm MSE}(\widehat{\theta}_3;\mu) = {\rm Bias}(\hat{\theta}_3;\mu)^2+{\rm Var}_\mu(\bar{X}+1) = \frac{1}{n}+1$

Method of Moments Estimation

Learning objectives

1. Understand how to compute method of moments estimators

Setting: $X_1,\dots,X_n\sim F_\theta$, i,i,d., where $\theta$ is an unknown value in the parameter space $\Omega$.

• Recall the $k$th (population) moment of $X$ is defined as $\mu_k = E[X^k]$.
• Define the $k$th (sample) moment of a sample $(X_1,\dots,X_n)$ as $M_k =\frac{1}{n}\sum_{i=1}^{n}X_i^k$

Idea: Substitution principle

• From the WLLN, we know $M_k=\frac{1}{n}\sum_{i=1}^n X_i^k \approx \mu_k$ for $k=1,2,3,\dots$
• If $\theta = g(\mu_1,\dots,\mu_p)$, then $\widehat{\theta}^{MoM}(X_1,\dots,X_n) = g(M_1,\dots,M_p)$

Example. $X_1,\dots,X_n\sim N(\mu,1)$, i,i,d. We want to estimate $\mu$.

$\theta = \mu$. $\widehat{\theta}^{MoM}(X_1,\dots,X_n) = \frac{1}{n}\sum_{i=1}^n X_i$.

Procedures to obtain Method of Moments (MoM) estimators of $\boldsymbol{\theta} = (\theta_1,\dots,\theta_p)$

1. Write $\mu_1, \dots, \mu_p$ as functions of $\theta_1,\dots,\theta_p$.

2. Solve (1) with respect to $\theta_1,\dots,\theta_p$. That is, find $g_1,g_2,\dots,g_p$ such that

3. Substituted population moments with sample moments

Example $X_1,\dots,X_n\sim N(\mu,\sigma^2)$, i,i,d. We want to estimate $\boldsymbol{\theta}=[\mu,\sigma^2]$.

Write $\mu_1,\mu_2$ as functions of $\theta_1,\theta_2$.

1. $\mu_1 = \mu = \theta_1$

2. $\mu_2 = \sigma^2 + \mu^2 = \theta_2+\theta_1^2$.

3. Solve 1 with respect to $\theta_1,\theta_2$.

   1. $\theta_1= \mu_1$
   2. $\theta_2 = \mu_2 - \mu_1^2$.

4. Substitute population moments with sample moments

   1. $\widehat{\theta}_1^{MoM} = M_1$
   2. $\widehat{\theta}_2^{MoM} = M_2 - M_1^2$

Therefore,

1. $\widehat{\theta}_1^{MoM} = \bar{X}$
2. $\widehat{\theta}_2^{MoM} = \frac{1}{n}\sum_{i=1}^n X_i^2 - \bar{X}^2 = \frac{1}{n}\sum_{i=1}^n(X_i - \bar{X})^2$

Remark: The Method of Moments estimator for $\mu$ is unbiased, but the Method of Moments estimator for $\sigma^2$ is biased

Maximum Likelihood Estimation

Learning objectives

1. Understand how to compute maximum likelihood estimators (MLE)

2. Understand invariance property of MLE

    

Setting: $X_1,\dots,X_n\sim F_\theta$, i,i,d., where $\theta$ is an unknown value in the parameter space $\Omega$.

Idea: Choose the value of $\theta \in \Omega$ that is most likely to have given rise to the observed data

Example: Suppose $X_1,\dots,X_n\sim {\rm Ber}(p)$. We have the observed sample $(1,1,0,1)$. Suppose the parameter space $\Omega = \{0.2, 0.7\}$ (so the parameter space contains only 2 values).

Given each $p \in \Omega$, what is the probability of observing $(1,1,0,1)$?

1. when $p=0.2$, $P_{p=0.2}(X_1=1,X_2=1,X_3=0,X_4=1) = (0.2)^3 (0.8) = 0.0064$
2. when $p=0.7$, $P_{p=0.7}(X_1=1,X_2=1,X_3=0,X_4=1) = (0.7)^3 (0.3) = 0.1029$

Thus, since $0.1029>0.0064$, $p=0.7$ is the maximum likelihood estimate of $p$.

Remark:

1. For any $p \in [0,1]$, $P_{p}(X_1=1,X_2=1,X_3=0,X_4=1) = p^3 (1-p)$ is a function of $p$.
2. Finding a maximum likelihood estimate can be viewed as finding a maximizer of the function $L(p) = p^3(1-p)$ for $p = 0.2,0.7$.
3. We call $L(p; 1,1,0,1) = p^3(1-p)$ the likelihood function.

Definition (Likelihood function)

• The likelihood function of $\boldsymbol{\theta} = [ \theta_1,\dots,\theta_p]$ based on the observed values $(x_1,\dots,x_n)$ from a random sample $(X_1,\dots,X_n)$ with pmf $p(\cdot; \theta_1,\dots,\theta_p)$ or pdf $f(\cdot; \theta_1,\dots,\theta_p)$ is

• The log likelihood function of $\boldsymbol{\theta} = [ \theta_1,\dots,\theta_p]$ is $\log L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$.

Definition (Maximum Likelihood Estimator)

• The maximum likelihood estimate is a value $\boldsymbol{\theta} \in \Omega$ which maximizes the likelihood function $L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$. In other words,

  $\hat{\boldsymbol{\theta}}(x_1,\dots,x_n) = (\hat{\theta_1},\dots,\hat{\theta_p})$ such that

  $L(\hat{\theta_1},\dots,\hat{\theta_p}; x_1,\dots,x_n)\geq L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$, for any $\boldsymbol{\theta} \in \Omega$.

• The maximum likelihood estimator is $\hat{\boldsymbol{\theta}}(X_1,\dots,X_n)$.

• Since $\log(\cdot)$ is a strictly increasing function, the maximizer of the likelihood $L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$ is the same as the maximizer of the log-likelihood $\log L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$. Maximizing the log-likelihood is often easier.

Example. Likelihood and log-likelihood function of $p$ based on the observed sample $(x_1,\dots,x_n)$, when $X_i \sim {\rm Ber}(p)$ i.i.d., $\Omega= [0,1]$.

$p(x_i ;p) = p^{x_i} (1-p)^{1-x_i}$.

Therefore,

• $L(p;x_1,\dots,x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} = p^{\sum_{i=1}^n x_i} (1-p)^{n-\sum_{i=1}^n x_i}$
• $\log L(p;x_1,\dots,x_n) = {\sum_{i=1}^n x_i} \log p + (n-{\sum_{i=1}^n x_i})\log (1-p)$

Procedures to obtain Maximum Likelihood Estimators (MLE) of $\boldsymbol{\theta} = (\theta_1,\dots,\theta_p)$

1. Compute the log-likelihood function $\log L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$

   1. Find the pdf or pmf of $X_i$
   2. Find the log-likelihood function

2. Find a maximizer $\hat{\boldsymbol{\theta}}(x_1,\dots,x_n) = (\hat{\theta_1},\dots,\hat{\theta_p})$ of the log-likelihood function $\log L(\theta_1,\dots,\theta_p; x_1,\dots,x_n)$

   1. Compute stationary points of the log-likelihood.

      • When the likelihood is differentiable (most cases), find the solution of the equation $\triangledown \log L(\theta_1,\dots,\theta_p; x_1,\dots,x_n) =0$ inside the parameter space.

   2. Find a maximizer among candidate points (stationary points and boundary points).

    

3. For a given sample $(x_1,...,x_n)$, the maximum likelihood estimate is $\hat{\boldsymbol{\theta}}(x_1,\dots,x_n)$. The maximum likelihood estimator is $\hat{\boldsymbol{\theta}}(X_1,\dots,X_n)$, which is a random variable.

Example (Bernoulli MLE) Compute the MLE of $p$ for a sample $(X_1,\dots,X_n)$, when $X_i \sim {\rm Ber}(p)$ i.i.d., $\Omega= [0,1]$.

1. $\log L(p;x_1,\dots,x_n) = {\sum_{i=1}^n x_i} \log p + (n-{\sum_{i=1}^n x_i})\log (1-p)$

 

2. a. Find stationary points in $(0,1)$.

$\frac{d}{dp} \log L(p;x_1,\dots,x_n) = {\sum_{i=1}^n x_i} \frac{1}{p}- (n-{\sum_{i=1}^n x_i})\frac{1}{1-p}=0$

Solving for $p$, we get the solution of $\hat{p} = \bar{x}$ .

When $0<\sum_{i=1}^n x_i<n$, $\bar{x}$ is the unique stationary point in $(0,1)$.

When $\sum_{i=1}^n x_i=0$ or $n$, there exist no stationary points in $(0,1)$.

 

b. When $0<\sum_{i=1}^n x_i<n$, $\hat{p} = \bar{x}$ is the global maximizer, since

$\frac{d^2}{dp^2} \log L(p;x_1,\dots,x_n) = -{\sum_{i=1}^n x_i} \frac{1}{p^2}- (n-{\sum_{i=1}^n x_i})\frac{1}{(1-p)^2}<0$, for $0<p<1$, and

$\lim_{p\rightarrow \pm\infty}\log L(p;x_1,\dots,x_n) = -\infty$.

When $\sum_{i=1}^n x_i=0$ or $n$,

$$\log L(p;x_1,\dots,x_n) =\begin{cases} n \log (1-p) & {\rm if}\,\, \sum_{i=1}^n x_i = 0\\ n\log p & {\rm if}\,\,\sum_{i=1}^n x_i = n\end{cases}$$

and it is straightforward to verify that both functions are monotone and the maximum achieves at $p=0$ and $p=1$. Then again, $\hat{p} = \bar{x}$ is the global maximizer.

Therefore, the maximum likelihood estimate of $p$: $\hat{p}(x_1,\dots,x_n) = \bar{x}$.

 

3. The maximum likelihood estimator of $p$: $\hat{p}(X_1,\dots,X_n) = \bar{X}$.

Example (Normal MLE, both $\mu$ and $\sigma^2$ unknown) Compute the MLE of $p$ for a sample $(X_1,\dots,X_n)$, when $X_i \sim N(\theta_1,\theta_2)$ i.i.d., $\Omega= \{(\theta_1,\theta_2); -\infty<\theta_1<\infty, 0<\theta_2<\infty\}$

We have $\boldsymbol{\theta} = [\theta_1,\theta_2]$

1. a. pdf of $X_i$: $f(x_i; \boldsymbol{\theta}) = \frac{1}{\sqrt{2\pi\theta_2}}e^{-\frac{1}{2\theta_2}(x_i-\theta_1)^2}$

   b. Log-likelihood $\log L(\boldsymbol{\theta}; x_1,\dots,x_n) = \sum_{i=1}^n \{ -\frac{1}{2}\log (2\pi)-\frac{1}{2}\log(\theta_2)-\frac{1}{2\theta_2} (x_i-\theta_1)^2 \}$

2. a. Find stationary points in $\Omega$

   $$\triangledown \log L(\boldsymbol{\theta}; x_1,\dots,x_n) = \begin{bmatrix} \frac{1}{\theta_2}\sum_{i=1}^n (x_i-\theta_1)\\ -\frac{n}{2\theta_2} +\sum_{i=1}^n\frac{1}{2\theta_2^2} (x_i-\theta_1)^2 \end{bmatrix} =0$$

   we get $\hat{\theta}_1 = \bar{x}$, $\hat{\theta}_2 = \frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2$.

 

b. we can verify the solution in a is indeed the unique global maximizer by using a second derivative condition (positive determinant and negative first element of the hessian matrix) and checking that there is no maximum at infinity.

3. The maximum likelihood estimator $\widehat{\boldsymbol{\theta}} = [\bar{X}, n^{-1}\sum_{i=1}^n(X_i-\bar{X})^2]$

Example (Uniform MLE) Compute the MLE of $\theta$ for a sample $(X_1,\dots,X_n)$, when $X_i \sim Unif(0,\theta)$ i.i.d., $\Omega= \{\theta;\theta>0\}$.

1. a. pdf of $X_i$:

   $$f(x_i;\theta) = \begin{cases}\frac{1}{\theta} & 0\leq x_i \leq \theta\\ 0 & {\rm elsewhere}\end{cases}$$

   Joint pdf $f$ of $(X_1,\dots,X_n)$:

   $$f(x_1,\dots,x_n;\theta) = \begin{cases}(\frac{1}{\theta})^n & 0\leq x_1,\dots,x_n \leq \theta\\ 0 & {\rm elsewhere}\end{cases}$$

   b. Likelihood:

   $$L(\theta;x_1,\dots,x_n) = \begin{cases}(\frac{1}{\theta})^n & \theta \geq \max_{1\le i \le n}x_i \\ 0 & {\rm elsewhere}\end{cases}$$

   Log-likelihood:

   $$\log L(\theta;x_1,\dots,x_n) = \begin{cases} -n\log \theta & \theta \geq \max_{1\le i \le n}x_i\\ -\infty & {\rm elsewhere}\end{cases}$$

2. a. Find stationary points in $\Omega$.

   For $\theta > \max_{1\le i \le n} x_i$,

   $\frac{d}{d\theta}\log L(\theta;x_1,\dots,x_n) = -\frac{n}{\theta}<0$. No stationary points exist. The log-likelihood is a decreasing function of $\theta$.

   Therefore, the log-likelihood function is maximized at $\hat{\theta} = \max_{1\le i \le n} x_i$.

   The maximum likelihood estimator $\hat{\theta} = \max_{1\le i \le n }X_i$.

Theorem (Invariance) If $\widehat{\theta}$ is the MLE of $\theta$, then for any function $g$, $g(\widehat{\theta})$ is the MLE of $g(\theta)$.

Remark Theorem 6.4-1 in HTZ requires that $g$ is one-to-one. When $g$ is not one-to-one, the discussion becomes more subtle, but we will not worry about this point in this class.

Proof (when $g$ is one-to-one)

Let $\eta = g(\theta)$. We have that $\widehat{\theta}$ is the maximizer of $L(\theta; x_1,\dots,x_n) = L(g^{-1}(\eta);x_1,\dots,x_n)$.

This function has the largest value $L(\widehat{\theta}; x_1,\dots,x_n) = L(g^{-1}(g(\widehat{\theta})); x_1,\dots,x_n)$.

Therefore, $\hat{\eta} = g(\hat{\theta})$ is the maximizer of the function $L(g^{-1}(\eta);x_1,\dots,x_n)$.

Example (Bernoulli MLE) Compute the MLE of ${\rm Var}(X_i)$ for a sample $(X_1,\dots,X_n)$, when $X_i \sim {\rm Ber}(p)$ i.i.d., $\Omega= [0,1]$.

The parameter of interest $\eta = {\rm Var}(X_i) = p(1-p)$.

Since the MLE of $p$ is $\bar{X}$, the MLE of $\eta$ is $\bar{X}(1-\bar{X})$.

Properties of Point Estimators

Learning objectives

1. Understand various finite sample (unbiasedness, sufficiency) and large $n$ properties (consistency, asymptotic normality/efficiency) of point estimators
2. Understand optimal properties of maximum likelihood estimators

Finite sample properties

• Unbiasedness

  • $\widehat{\theta}$ is an unbiased estimator if ${\rm Bias}(\widehat{\theta}; \theta$) $=0$ for all $\theta \in \Omega$.

   

  Example: for a sample $(X_1,\dots,X_n)$, when $X_i \sim {\rm Poisson}(\lambda)$, i.i.d., we consider two estimators for $\lambda$, $\widehat{\theta}_1 = \bar{X}$ and $\hat{\theta}_2 = X_1$.

  For $\lambda >0$, ${\rm Bias}(\hat{\theta_1};\lambda) = E_\lambda[\hat{\theta}_1]-\lambda = E_\lambda[\bar{X}]-\lambda = 0$. Similarly, ${\rm Bias}(\hat{\theta_1};\lambda) = E_\lambda[\hat{\theta}_2]-\lambda = E_\lambda[X_1]-\lambda = 0$.

  Both are unbiased.

  Among unbiased estimators $\widehat{\theta}_1$ and $\widehat{\theta}_2$, which estimator should we use?

• Relative Efficiency

  • Definition: Given two unbiased estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ of a parameter $\theta$, the efficiency of $\hat{\theta}_1$ relative to $\hat{\theta}_2$ , denoted by ${\rm eff}( \hat{\theta}_1,\hat{\theta}_2 )$, is defined to be the ratio
  $${\rm eff}( \hat{\theta}_1,\hat{\theta}_2 ) = \frac{{\rm Var}_\theta(\hat{\theta}_2)}{{\rm Var}_\theta(\hat{\theta}_1)}$$

  • $\hat{\theta}_1$ is more efficient than $\hat{\theta}_2$ iff ${\rm eff}( \hat{\theta}_1,\hat{\theta}_2 )>1$. Obviously, given any two unbiased estimators, we would prefer a more efficient estimator.

    Pushing this idea even further, we may consider the "best" unbiased estimator, which we define in the following way.

     

  Example: for a sample $(X_1,\dots,X_n)$, when $X_i \sim {\rm Poisson}(\lambda)$, i.i.d., we consider two estimators $\widehat{\theta}_1 = \bar{X}$ and $\hat{\theta}_2 = X_1$ for $\lambda$.

  $${\rm eff}( \hat{\theta}_1,\hat{\theta}_2 ) = \frac{{\rm Var}_\lambda(\hat{\theta}_2)}{{\rm Var}_\lambda(\hat{\theta}_1)} = \frac{\lambda}{\lambda/n} = n >1.$$

  $\hat{\theta}_1$ is more efficient than $\hat{\theta}_2$.

• Definition (Minimum Variance Unbiased Estimator (MVUE)) An estimator $\hat{\theta}$ is a best unbiased estimator of $\theta$ if $\hat{\theta}$ is unbiased (i.e., $E_\theta[\hat{\theta}] = \theta$, for all $\theta \in \Omega$), and has a minimum variance (i.e., for any other unbiased estimator $T$, ${\rm Var}_\theta(\hat{\theta}) \leq {\rm Var}_\theta(T)$, for all $\theta \in \Omega$.)

   

  Finding a best unbiased estimator (if it exists!) is not an easy task for a variety of reasons. One obvious challenge is that we cannot check the variances of all unbiased estimators. The following theorem provides a lower bound on the variance of any unbiased estimator of $\theta$. That is, if we find an unbiased estimator for which the variance matches with the lower bound, then we know that we have found the MVUE.

   

  • Theorem (Cramer-Rao Inequality) $X_1,\dots,X_n\sim F_\theta$, i,i,d. with pdf (or pmf) $f_\theta(x)$. Let $\hat{\theta}=\hat{\theta}(X_1,\dots,X_n)$ be any unbiased estimator of $\theta$. Under very general conditions,
  $${\rm Var}_\theta(\hat{\theta}(X_1,\dots,X_n)) \geq \frac{1}{I_n(\theta)}, \,\, \mbox{where } I_n(\theta)= nE_\theta[-\frac{\partial^2 \log f_\theta(X)}{\partial \theta^2}]$$
  • We define the efficiency of an unbiased estimator of $\hat{\theta}$ as
  $${\rm eff}(\hat{\theta}) = \frac{I_n(\theta)^{-1}}{{\rm Var}_\theta(\hat{\theta})}.$$

  • We say an unbiased estimator $\hat{\theta}$ is efficient if ${\rm eff}(\hat{\theta})=1$, i.e., the variance of $\hat{\theta}$ equals the Cramer-Rao lower bound $I_n(\theta)^{-1}$.

     

    Example: Let $X_1,X_2,...,X_n$ be a random sample from a Poisson distribution with parameter $\lambda >0$. Determine the Cramer-Rao lower bound.

    $$\log p_\lambda(x_i) = -\lambda+x_i\log \lambda - \log x_i!$$

    $$I_n(\lambda ) = nE_\lambda [ -\frac{\partial^2 \log p_\lambda(X)}{\partial \lambda^2}] = E_\lambda[x_i \frac{1}{\lambda^2}] = \frac{n}{\lambda}.$$

    Therefore, the Cramer-Rao lower bound is $\lambda/n$.

    We note ${\rm Var}_\lambda (\bar{X}) = \lambda/n$. Therefore, the efficiency of $\bar{X}$ is $1$, i.e., $\bar{X}$ is efficient.

• Sufficiency

  • A statistic $T(X_1,\dots,X_n)$ is said to be sufficient for a parameter $\theta$ if the statistic captures all the information in a sample about a target parameter $\theta$ (the formal definition will be presented later).
  • Sufficient statistics are not unique. In fact, the random sample itself $(X_1,\dots,X_n)$ is a sufficient statistic. We would like to find a sufficient statistic that reduces the data in the sample as much as possible. Such a statistic is called minimal sufficient.
  • Often, "good" estimators are functions of a minimal sufficient statistic.

Large sample (asymptotic) properties

• Consistency: If $P_\theta(|\widehat{\theta}(X_1,\dots,X_n) - \theta|\geq \epsilon) \underset{n\rightarrow \infty}{\rightarrow} 0, \forall \epsilon>0$, then $\widehat{\theta}_n=\widehat{\theta}(X_1,\dots,X_n)$ is a consistent estimator.

  Remark: consistency is often considered as a "minimum requirement" an estimator should meet.

   

  Theorem If ${\rm MSE}(\hat{\theta};\theta) \underset{n\rightarrow\infty}{\rightarrow} 0$, then $\hat{\theta}$ is consistent.

  proof. For any $\epsilon>0$, $P_\theta(|\widehat{\theta} - \theta|\geq \epsilon) =P_\theta(|\widehat{\theta} - \theta|^2\geq \epsilon^2)\leq \frac{E[|\hat{\theta}-\theta|^2]}{\epsilon^2}\underset{n\rightarrow\infty}{\rightarrow} 0$.

   

  Example: for a random sample $(X_1,\dots,X_n)$, when $X_i \sim N(\mu,1)$, is $\bar{X}$ a consistent estimator of $\mu$? How about $(X_1+X_2)/2$?

  1. By WLLN, $P_\mu(|\bar{X}-\mu|\geq\epsilon) \underset{n\rightarrow \infty}{\rightarrow} 0, \forall \epsilon>0$.

     Or, ${\rm MSE}(\bar{X};\mu) = {\rm Var}_\mu (\bar{X}) = 1/n \underset{n\rightarrow\infty}{\rightarrow} 0$ . Thus $\bar{X}$ is a consistent estimator of $\mu$.

  2. We have $T=(X_1+X_2)/2 \sim N(\mu,1/4)$. Let $\epsilon$ be any positive number. We have,

     $$P(|T - \mu| \geq \epsilon)= P(\frac{|T - \mu|}{1/4} \geq 4\epsilon) = \Phi(4\epsilon)-\Phi(-4\epsilon)$$

     Therefore, $T$ is not a consistent estimator of $\mu$.

   

• Asymptotic Normality: we say an estimator $\widehat{\theta}_n$ is asymptotically normal if the distribution of $\widehat{\theta}_n \approx N(\theta, V_n(\theta))$ when $n$ is large.

  The distribution of $\hat{\theta}_n$ when $n$ is large is called an asymptotic distribution of $\hat{\theta}_n$. An estimator $\widehat{\theta}_n$ is asymptotically normal if its asymptotic distribution is Normal, and we denote it as