What is Estimation Theory ?

A main portion of the statistics deals with the estimation of parameter or distribution. But, what is estimation?  Googling the word ‘estimation‘ gives the meaning “a rough calculation of the value, number, quantity, or extent of something”, which is practically what we do in estimation theory of Statistics. Suppose there is something unknown and we will try to ‘estimate’ that based on the information we have.

For example, consider my favorite city Kolkata, the city of joy. There are approximately 2 crore households in Kolkata. Suppose I wish to know (don’t know why though) what is the annual average income of the households of Kolkata. What should I do ? One obvious way is to go to every households and ask for their family income, get 2 crore data and take the average. Of course in this way we will get the best answer ! But, is it feasible at all ?  Yeah, you guessed it correctly, the answer is a big big NO ! Why ? There are numerous of reasons including:

  1. Visiting all the households of Kolkata is absurd in terms of man power and time .
  2. Income is a very confidential thing and many of them will not respond at all, many of them will  try to undermine the amount.

So what do we do? Dealing with the second problem is more difficult and needs more sophisticated statistical thinking, so we will skip this problem for the time being.

For the first problem, what we can do is to sample some houses (say 200 or 300 many), get their income and then try to “estimate” the average income. Now how we should sample the households is a problem that is answered in the topics like ‘Sample Survey’. Borrowing their techniques judiciously, lets say we have sampled 300 households. The unknown average income which we want to estimate is say \theta. The data from 300 households is denoted by say y_1, y_2, \cdots , y_{300}.

Now what should we do ? Let’s plot the data and see how they behave !

This is plot of density of data using the software R, which is very useful in statistical analysis. The X-axis is income in thousands. This is how the density of the data looks like !

Can we guess the distribution, at-least the family of the distribution? The answer may be ‘Yes’ or ‘No’, and based on this answer, there are two different fields of estimation theory of statistics! From this, one can guess that ,”Oh yes! This is exponential distribution !” Again may be you are correct, may be not, but let’s assume we have decoded the family of the distribution successfully. Then? We need to know the parameters of this particular distribution to get hold of the distribution completely! Now, our next job is to ‘estimate the parameter‘. Their comes the estimation theory. Estimating them statistically let’s say we get that the income actually follows Exp (0.5, 1). Then we can say that as the income data has this distribution, the average income will be mean of the distribution i.e. 1.5 thousand !  Now depending on whether we can guess the family of the distribution or not we have two different theories in statistics:


  • Parametric Estimation:  Here we assume that we have data x_1, \cdots , x_n \sim F and we know the family of F (Say Normal or Exponential) but we don’t know the exact value of the parameters. We try to estimate the parameters from the data to get hold of the distribution as correctly as possible and then try to infer about our question. The name is “Parametric Estimation” because here we are trying to estimate the unknown parameters of the known distribution family.


  • Non-Parametric Estimation: Now suppose we have data x_1, \cdots , x_n \sim F but we don’t even know the family of F ! Then the analysis is totally different from the previous one. There are several approaches from several point of views. One type of approaches is to first estimate the distribution F from the data then try to infer what we wish to learn. Another approach is to estimate directly the unknown quantity of interest without guessing anything about the distribution ! (Like Re-sampling Techniques which we will learn later).


One think you should observe that in case of Parametric Estimation we know the family of the distribution almost by free! (Either by eye estimation or GOD has told so), but in case of Non-parametric Estimation we don’t have this kind of luxury! So, in the second case, we need more data for the same accuracy which can be obtained using less data in the former one ! So when should we avail the first and when the second? Basically, the thumb rule is, whenever we have the information about the family of distribution we should always go for Parametric Estimation, as here we have better accuracy with smaller sample size. But, if we don’t have a good idea about the family, then go for the Non-parametric! But again, as we don’t know the family, (basically like Jon Snow we know nothing here!!!), we need larger amount of data for better approximation.

Leave a comment

Your email address will not be published. Required fields are marked *