Data : , i.e. we assume the data comes from some probability distribution which can be defined using the parameters . Now we want to make inference about the unknown parameter .
- An estimator T(X) is a statistic (or a function of X).
- For a given observed value of x of X, T(x) is an estimate of .
- Mean Squared Error: .
We may define an estimator to be the estimator of if .
Unfortunately, such an estimator does not exist (Think Why!!!). Hence we have to approach from another direction. One way is to restrict attention to a class of estimators showing some degree of impartiality with a hope that we can find a best estimate in the restricted class. One such restriction is Unbiasedness.
- A statistic T(X) is said to be unbiased for estimating a parametric function if . We say that is unbiasedly estimable if such an estimator exists.
It can be easily seen that where is called the bias of T. (Check!!!)
- An unbiased estimator of is uniformly minimum variance unbiased estimator (UMVUE) if : and for any unbiased estimator T of .
An Inequality: For a random variable Z,
where c is a fixed real number. (Check!!!)
Consider now conditional distribution of a random variable U given another random variable T. Then we can see:
, thereby implying:
Let U be an unbiased estimator of . Consider sufficient statistics T for and let . Then is an unbiased estimator of and .
Proof: First note that is free of as T is sufficient. Then can be used as an estimator of . Also,
Thus given any unbiased estimator, there exists an unbiased estimator based on T, which is “as good as” U in the sense that: .
Lemma 1: Let , and T(X) be a complete sufficient statistic for . Then every U-estimable parametric function has one and only one unbiased estimator based on T in the sense that if and are two unbiased estimators based on T, then:
Lemma 2: Let T be a complete sufficient statistic. An Unbiased Estimator of , based on T, is also an UMVUE.
(Try to Prove the above lemma.)
Example 1: Let’s first think about the Bernoulli case. iid Ber(p).
Here T=T(X)= is a complete sufficient statistic.
- UMVUE of p:
, i.e. T/n is an unbiased estimator based on the complete sufficient statistics T , implying T/n is UMVUE.
- UMVUE of :
Check that: . Thus UMVUE of p is .
- UMVUE of :
UMVUE of = .
It can be easily seen that any U-estimable function is a polynomial in p of degree .
Example 2: Let’s now think about Poisson case. .Then, we know, is complete sufficient.
It is very easy to get UMVUE of for r integer.
Let’s try to find UMVUE of P[X=k].
Exercise: . Find UMVUE of . [Hint: ]
Example 3: . Here, =sample mean is complete sufficient.
- Case 1: ( is known) So, is UMVUE of . Similarly, UMVUE of is .
Suppose =1. Want to estimate: .
Start with , then condition by .
, using the fact that and are independent (by Basu’s Theorem).
Hence, UMVUE: , where is cdf of N(0,1). (Check!!!)
- Case 2: ( known) Want UMVUE of .
is complete sufficient statistic for .
, thereby giving , i.e T/n is UMVUE of .
, if .
Replacing r by r/2, we get the required UMVUE of as: if .
- Case 3: (both and unknown)
Now, is complete sufficient for , where .
Now, , , which gives UMVUE of as: (Chcek!!!)
Similarly, UMVUE of is: (Since and T are independent) (Check!!!)
Also, UMVUE of is , the small change from the previous version is due to change in degrees of freedom.
Thus, now we can get UMVUE of , which turns out to be: .
Now, to estimate quantiles: let: , we get the UMVUE of u, the p-th quantile, as: , where is the UMVUE of .
Example 4: iid Exp(), i.e. with density .Want UMVUE of .
Let . A complete sufficient statistics is .
, since is ancillary.
Now we know, if and , then
Conclusion: Thus, in this article, we have gathered some basic knowledge about parametric inference, and a detailed discussion on UMVUE and some examples. Let me finish this topic with an example, which would illustrate why UMVUE does not guarantee minimum Mean Square Error.
Concluding Example: , where . We have learnt that the UMVUE of is . Now let us consider the statistic , which takes if , otherwise take – if and if .
Then, we can see that , even though is UMVUE.
So, let’s bid adieu for now. I will be back with some other interesting topic very soon.