Rao-Blackwell Theorem and it’s applications 2


Data : X=(X_1, ..., X_n) \sim P \in \mathbb{P}= \left \{ P_\theta | \theta \in \Theta \right \}, i.e. we assume the data comes from some probability distribution which can be defined using the parameters \theta. Now we want to make inference about the unknown parameter \theta.

Some Definitions:

  • An estimator T(X) is a statistic (or a function of X).
  • For a given observed value of x of X, T(x) is an estimate of \theta.
  • Mean Squared Error: MSE_\theta(T)=E_\theta(T(X)-g(\theta))^2 .

We may define an estimator T_0  to be the estimator of \theta if MSE_\theta(T_0) \leq MSE_\theta(T).

Unfortunately, such an estimator does not exist (Think Why!!!). Hence we have to approach from another direction. One way is to restrict attention to a class of estimators showing some degree of impartiality with a hope that we can find a best estimate in the restricted class. One such restriction is Unbiasedness.

  • A statistic T(X) is said to be unbiased for estimating a parametric function g(\theta) if E_\theta(T(X))=g(\theta) \forall \theta. We say that g(\theta) is unbiasedly estimable if such an estimator exists.

It can be easily seen that MSE_\theta(T)=V_\theta(T)+(b(T,\theta))^2 where b(T,\theta)=E_\theta(T)-g(\theta) is called the bias of T. (Check!!!)

  • An unbiased estimator T_\theta of g(\theta) is uniformly minimum variance unbiased estimator (UMVUE) if :V_\theta(T_0) \leq V_\theta(T) \forall \theta \in \Theta and for any unbiased estimator T of g(\theta).


An Inequality: For a random variable Z,

E(Z-c)^2 \geq (E(Z)-c)^2  where c is a fixed real number. (Check!!!)

Consider now conditional distribution of a random variable U given another random variable T. Then we can see:

E[(U-c)^2|T] \geq [E(U|T)-c]^2, thereby implying:

E(U-c)^2=EE[(U-c)^2|T] \geq E[E(U|T)-c]^2

Rao-Blackwell Theorem:

Let U be an unbiased estimator of g(\theta). Consider sufficient statistics T for \theta and let \phi(T)=E[U|T]. Then \phi(T) is an unbiased estimator of g(\theta) and V_\theta[\phi(T)] \leq V_\theta(U).

Proof: First note that \phi(T) is free of \theta as T is sufficient. Then \phi(T) can be used as an estimator of  g(\theta)Also, V_\theta(\phi(T))=E_\theta[\phi(T)-g(\theta)]^2 \leq E_\theta[U-g(\theta)]^2 = V_\theta(U) \forall \theta

Thus given any unbiased estimator, there exists an unbiased estimator \phi(T) based on T, which is “as good as” U in the sense that: V_\theta(\phi(T)) \leq V_\theta(U) \forall \theta.

Lemma 1Let X \sim P_\theta, \theta \in \Theta, and T(X) be a complete sufficient statistic for \theta. Then every U-estimable parametric function g(\theta) has one and only one unbiased estimator based on T in the sense that if \phi_1(T) and \phi_2(T) are two unbiased estimators based on T, then:

\phi_1(T)=\phi_2(T) \textrm{ a.e. } [P_\theta] \forall \theta

Lemma 2Let T be a complete sufficient statistic. An Unbiased Estimator \phi(T) of g(\theta)based on T, is also an UMVUE.

(Try to Prove the above lemma.)

Some Examples:


Example 1Let’s first think about the Bernoulli case. X_1,...X_n iid Ber(p). 0 \lt p \lt 1

Here T=T(X)= \sum_{i=1}^n X_i is a complete sufficient statistic.

  • UMVUE of p: 

\sum_{i=1}^n E_p(T)=np \forall p, i.e. T/n is an unbiased estimator based on the complete sufficient statistics T , implying T/n is UMVUE.

  • UMVUE of p^2:

Check that: E_p[T(T-1)]=n(n-1)p^2. Thus UMVUE of p is \frac{[T(T-1)]}{n(n-1)}.

  • UMVUE of p^r:

UMVUE of p^r = \frac{T(T-1)(T-2)...(T-r+1))}{n(n-1)(n-2)...(n-r+1)}.

It can be easily seen that any U-estimable function is a polynomial in p of degree \leq n.


Example 2: Let’s now think about Poisson case. X_1,...,X_n \textrm{ iid }Poi(\lambda).Then, we know, T=\sum_{i=1}^nX_i is complete sufficient.

It is very easy to get UMVUE of \lambda^r for r integer.

Let’s try to find UMVUE of P[X=k].

  • U=\mathbb{I}[X_1=k].
  • \phi(T)=E(U|T)
  • \phi(t)=E[U|T=t]=_{k}^{t}\textrm{C}\frac{(n-1)^{t-k}}{n^t}

Exercise: X \sim Poi(\lambda). Find UMVUE of e^{\sqrt{2}\lambda} .                                               [Hint: E(t^X)=e^{(t-1)\lambda}]


Example 3: X_1,...,X_n \sim N(\mu, \sigma^2). Here, \bar{X}=sample mean is complete sufficient.

  • Case 1: (\sigma  is known)  So, \bar{X} is UMVUE of \mu. Similarly, UMVUE of \mu^2 is \bar{X}^2- \frac{\sigma^2}{n}

Suppose \sigma  =1. Want to estimate: P[X_1 \leq u].

Start with U= \mathbb{I}[X_1 \leq u], then condition by \bar{X}.

P[X_1 \leq u | \bar{X}= \bar{x}]= P[X_1-\bar{X}\leq u-\bar{X} | \bar{X}=\bar{x}]=P[X_1-\bar{X} \leq u-\bar{x}], using the fact that X_1-\bar{X} and \bar{X} are independent (by Basu’s Theorem).

Hence, UMVUE: \Phi[\sqrt{\frac{n}{n-1}}(u-\bar{x})], where \Phi  is cdf of N(0,1). (Check!!!)

  • Case 2: (\mu known) Want UMVUE of \sigma^r

T=\sum(X_i-\mu)^2 is complete sufficient statistic for \sigma^2.

Y=\frac{T}{\sigma^2} \sim \chi^2_n, thereby giving E(T/\sigma^2)=n, i.e T/n is UMVUE of \sigma^2.

E[Y^r]=\frac{1}{2^{\frac{n}{2}}\Gamma(\frac{n}{2})}\int_0^\infty y^ry^{\frac{n}{2}-1}e^{-\frac{y}{2}}dy=\frac{2^r\Gamma(\frac{n}{2}+r)}{\Gamma(\frac{n}{2})}, if \frac{n}{2}+r \gt 0.

Replacing r by r/2, we get the required UMVUE of \sigma^r as: \frac{\Gamma(\frac{n}{2})}{2^{\frac{r}{2}}\Gamma(\frac{n+r}{2})}T^{\frac{r}{2}} if n \gt -r.

  • Case 3: (both \mu and \sigma  unknown)

Now, (\bar{X},T) is complete sufficient for (\mu, \sigma^2), where T=\sum(X_i-\mu)^2.

Now, E(\bar{X}^2)= E(\bar{X}-\mu)^2+ \mu^2=\frac{\sigma^2}{n} + \mu^2, , which gives UMVUE of \mu^2 as: \bar{X}^2-\frac{T}{n(n-1)} (Chcek!!!)

Similarly, UMVUE of \mu^3 is: \bar{X}^3-3\bar{X}\frac{T}{n(n-1)}  (Since \bar{X} and T are independent) (Check!!!)

Also, UMVUE of \sigma^r is \frac{\Gamma(\frac{n-1}{2})}{2^{\frac{r}{2}}\Gamma(\frac{n+r-1}{2})}T^{\frac{r}{2}}, the small change from the previous version is due to change in degrees of freedom.

Thus, now we can get UMVUE of \frac{\mu}{\sigma}, which turns out to be: \frac{\bar{X}\Gamma(\frac{n-1}{2})\sqrt{2}}{\Gamma(\frac{n}{2}-1)}T^{-\frac{1}{2}}.

Now, to estimate quantiles: let: P[X_1 \leq u]=p, we get the UMVUE of u, the p-th quantile, as: \bar{X}+\hat{\sigma}\Phi^{-1}(p), where \hat{\sigma} is the UMVUE of \sigma.


Example 4X_1,...X_n  iid Exp(\theta), i.e. with density  f(x)=\theta e^{-\theta x}\textrm{ } \mathbb{I}[x \gt 0].Want UMVUE of P_\theta(X_1 \gt k).

Let U=I[X_1 \gt k]. A complete sufficient statistics is T=\sum X_i.

E[U |T=t]=P[X_1 \gt k | T=t]= P[\frac{X_1}{T} \gt \frac{k}{T} | T=t]= P[\frac{X_1}{T} \gt \frac{k}{t} ], since \frac{X_1}{T} is ancillary.

Now we know, if X_1 \sim Gamma(\alpha, p_1) and X_2 \sim Gamma(\alpha, p_2), then

  •  X_1+X_2 \sim Gamma(\alpha, p_1+p_2)
  • \frac{X_1}{X_1+X_2} \sim Beta(p_1,p_2)

Therefore:  E[U | T=t]= P[V \gt \frac{k}{t}]= (1-\frac{k}{t})^{{n-1}

Conclusion: Thus, in this article, we have gathered some basic knowledge about parametric inference, and a detailed discussion on UMVUE and some examples. Let me finish this topic with an example, which would illustrate why UMVUE does not guarantee minimum Mean Square Error.

Concluding Example:  X_1,...,X_n \sim N(\theta,1), where -5 \leq \theta \leq 5. We have learnt that the UMVUE of \theta is \bar{X}.  Now let us consider the statistic \hat{\theta}, which takes \bar{X} if -5 \lt \bar{X} \lt 5, otherwise take –-5 if \bar{X} \leq -5 and  5  if  \bar{X} \geq 5.

Then, we can see that MSE(\hat{\theta}) \leq MSE(\bar{X}), even though \bar{X} is UMVUE.


So, let’s bid adieu for now. I will be back with some other interesting topic very soon.

Leave a comment

Your email address will not be published. Required fields are marked *

2 thoughts on “Rao-Blackwell Theorem and it’s applications

    • djghosh Post author

      Thanks for the comment…This was mainly on Rao-Blackwell Theorem…Will try to post an introductory post on Parametric Inference with motivations later.