Expert: Scott A Wilson - 12/9/2008

Question
I understand that a sample is a subset of a population of interest and a census is the entire population of interest. I'm interested in tests of statistical significance. I'd think it's extremely unlikely for 2 populations to have identical means. Yet, you often hear that the difference between the means of group A and B are or are not statistically significant. Statistical significance is a function of the magnitude of the difference and the sample size. Yet, it seems to me that it is likely that there will be a difference between populations on most things (1.0612 v. 1.0616). Are you still dealing with matters of statistical significance when you interview (or survey) entire populations? This issue came up in a conversation with a buddy. I suggested that there could be a difference between substantive differences and statistically significant differences in that some statistically significant differences can be so small to be almost meaningless. We argued about whether a sample could ever become too big. So, if you interview entire populations and find a difference, are you still talking about a statistically significant difference, or is statistical significance only in the realm of sampling? Thanks. I hope this isn't too confusing.

Answer
When a sample is taken from a population, it only matters how big the sample was to determine the accuracy of the results.  You would not expect two samples to ever be the same, but that's why there's a margin of error involved.  You may get, for some measurement, that the average is 30.3% with a standard deviation of 2.1%.

This means that the chance of the true average being between  µ±1σ is only 0.6826.  The interval would be (µ-σ,µ+σ) = (28.2, 32.4).  That means that 31.74 percent of the time, the measurement would not be right.

The chance that the average is between µ±2σ 0.9544.  The range would be (26.1, 34.5).  This would mean that over 95% of the time, the true average would be in this interval.

Note that the sample can be as big as it is, but the the statistics present the same results when measured on a set of n people.

THe accuracy of the data only depends on whether or not the sample taken was truly random.  For example, if you wanted to measure the average age of everyone in the US, you couldn't rely on data collected in New York to generate this.  You could say what the mean and standard deviation are in the state of New York, but applying that to the entire US would not be possible.

What statistical sampling does is state that the mean is between two values a certain percent of the time.  The greater the difference between the limits, the higher probability they are correct.

As an another example, you might test a population something twice and get measurements of the average being 25.6 with the standard deviation being 5.3.  Another test would give 24.8 with a standard deviation of 5.3 again.  The difference betwen the is only 0.8 and the standard deviation is 5.3, so you can say there is not a statistical difference between the samples.

So in summary, to see if there is a difference in populations, it all depends on the sample size n and if the statistical sampling was truly randomly spread through the entire population.  There is almost always a difference, but according to statistics, it may or may not be significant.  The sample average may be slightly different, but with statistics it can be shown that neither one is exactly right (almost all the time), but only close.

Did that answer your question?  Remember, no matter how large the sample size, the difference only depends on the number in the sample and if the sample was a truly random representation of the entire sample.  No matter how large, the results only are affected by the sample size.  The average will most likely never come out the same, but statistically, there is not significance in the difference.

I'm not sure this answers what for which you are looking.  Perhaps you might send me an example and I could tell  you about the statistics on that sample.