Question QUESTION: In trying to verify a set of test results using the F & t tests. The F test which checks the variance of the test is run to see if there is a significant difference between tests A & B. Suppose you would have 10 tests for "A" and 5 tests for "B" and you are trying to verify the "A" tests. You determine that there is no significant difference using a significance level of 0.05. However your probability to detect a 1.5 standard deviation ratio is only 10%.
Therefore you assume the variances are equal and do a pooled t test to determine whether the means are assumed to be from the same population. You still get no significant difference, but the probability of detecting a difference is approximately 72%.
Finally my question is, with the probability of only 10% for the F test to detect a difference of 1.5, how can I feel comfortable that their is no difference in the tests? In order to get a minimum of 70% probability for the F test Iwould need two sets of 40 samples each,which is not reasonable in this case. Is the t test more important in determining that two sets of results are different.
ANSWER: phil -
what you point out is that tests based on small samples can have low power (= the chance of detecting a specified difference).
in your case, as an alternative to first doing an F-test for the variances and then pooling and doing a pooled-variance 2-sample t-test for the means, you can just do a modified sort of 2-sample t-test on the means, without having to assume anything about the variances.
this can be done using Excel, for example, using the TTEST function found in the "Statistical" part of the "Insert Function" menu: put the cursor on an empty cell and click the "Insert Function" icon under the menu bar (it looks like f_x or f subscript x). you can also click on the Insert menu and choose Function. in either case, select the category "Statistical" and then TTEST.
after highlighting TTEST with the cursor on the Insert Function window that opens, if you then click on the Help link in the lower left corner of the Insert Function window, you will get an explanation of what arguments have to be filled in to carry out the test. the syntax for the test is
TTEST(array1, array2, tails, type).
as explained in the Help, tails = 1 or 2 (1 or 2-sided test) and you want to set type=3 to carry out the t-test without assuming equal variances.
the Excel help does not go into more detail about what is being done, but you can consult various stats texts to get some further details about it. one such reference is: Agresti and Franklin, Statistics - the Art and Science of Learning from Data (Pearson/Prentice Hall), pp442-443. many other introductory stats texts also discuss this test and i'm sure you can find online posting about it as well. it goes by the name of welch's t-test. wikipedia has an entry for it and some relevant references.
altho this test avoids having to assume equal variances, there is no free lunch. it too will have low power for detecting moderate differences between the true means with small sample sizes.
---------- FOLLOW-UP ----------
QUESTION: Using excel to determine the F test I've noticed that the value calculated is the percent probability of detecting whether the variances are different, with a low probability indicating a less likely chance of having the same variance. First, am I interpreting this correctly, and if so, what probability would be considered as acceptable. In the past if F<F-critical, there was assumed to be no significant difference. It appears that this method would be less powerful than the probability to detect a difference. I don't know whether I am interpreting this correctly. Please advise.
Answer phil -
you have to describe the F-test you are trying to carry out in a context.
F-tests are used in various situations (such as ANOVA and regression) and just
how the test is carried out depends on the particulars of the situation.
if you tell me the scenario, the hypothesis (or hypotheses) you are trying to
test and other possibly relevant information, perhaps i'll be able to help.
