Sec 64 Statistical versus Practical Significance

Question 1

Statistical vs. Practical Significance

Answer 1

PRACTICAL SIGNIFICANCE – A statistical result is of practical significance when it has direct implications for application in the real world.

Statistically significant differences are NOT necessarily large differences. Likewise, a statistically significant correlation coefficient is not necessarily large.

• Just because a result is STATISTICALLY SIGNIFICANT does NOT NECESSARILY mean that it is PRACTICALLY SIGNIFICANT – meaning that the difference might be REAL, but simply not LARGE enough to matter in real life.
  
  • Important to carefully consider the descriptive statistics (including measures of EFFECT SIZE) before drawing conclusions and considering the implications of statistically significant results.
• Ex: An researcher administered an experimental test to a random sample of the incoming college freshmen in September. At the end of their freshman year, he correlated the admissions test scores with freshman-year GPA and obtained a Pearson r of 0.20.
  
  • For the sample size used, the value of 0.20 was found statistically significant at the .05 probability level.
  • To interpret the results further, the researcher calculated the Coefficient of Determination r² (a descriptive statistic), which equals 0.04 in this case. Multiplying 0.04 by 100%, the researcher learned that the amount of variation in GPA accounted for by the experimental test was only 4.0%.
    
    • That means that only 4% or 1/20th of the variance in GPA was predicted by the test scores. As a result, the researcher subjectively concluded the test was NOT very helpful and so, was not an important predictor of freshman GPA, even though the underlying correlation coefficient was statistically significant.
• Ex: A researcher administered a math test to two random samples of third graders, one in a large urban school district, one in a suburban school district.
  
  • The difference between the two means (45.75 and 55.75 respectively) was found statistically significant at the .01 probability level.
  • To better analyze the results, the researcher calculated the T-Scores (Transformed standard scores, which make the data more intuitive) so that they could range from 20.00 to 80.00 – a 60 point range.
  • The researcher reported that the 10-point difference between means (45.75 to 55.75) was not only statistically significant but in this case, they were also PRACTICALLY SIGNIFICANT.
  • She concluded that the difference was educationally important because it was 1/6 of the total possible difference that could be obtained (i.e., the 10-point difference was 1/6 of the maximum possible difference of 60 points between 20.00 and 80.00),
  • Note that the STATISTICAL SIGNIFICANCE is a mathematical calculation, but the determination of PRACTICAL SIGNIFICANCE is usually a SUBJECTIVE decision based on whatever criteria the user has determined would be USEFUL for their REAL-LIFE purposes.
• When using easily interpreted T-Scores, comparing the possible maximum range of the difference (60 points in the 2nd example above) with the obtained difference between means (10 points in the same example) is helpful in reaching conclusions.
  
  • For instance, IF the 10-point difference in means above were based on a possible range of 600 points (just 10/600 = about 1.7% of the range), then it probably would NOT be very useful or important in real life. But because the range is only 60, and the amount of that range is 1/6th (or almost 17%) of the available range, that might be regarded as very important.

NOTE: statistical significance is NOT necessary for a result to be of practical significance (Useful).

• Ex: Suppose you’re testing two systems to see which one you should purchase, one being far more expensive than the other. When both systems are tested for accuracy, it is found that the DIFFERENCE in the results acquired from the two systems is STATISTICALLY INSIGNIFICANT. As a result, your choice is clear – buy the less expensive system for the same results as the more expensive system.

SUMMARY:

• DESCRIPTIVE STATISTICS summarize and organize the results of a study so that they can be understood more fully and communicated concisely.
• INFERENTIAL STATISTICS tell us whether any differences are reliable in light of the possibility that the differences were created by random error.
• Determining PRACTICAL SIGNIFICANCE is the last step in the research process, and it is based on sound judgment that goes beyond mathematical reasoning.

Question 2

Determining Practical Significance (4 Considerations)

Answer 2

DETERMINING PRACTICAL SIGNIFICANCE requires FOUR CONSIDERATIONS (in addition to direct statistical considerations):

1. COST-BENEFIT ANALYSIS – Ask yourself, is the benefit worth the cost of obtaining that benefit?
   
   • Ex: Suppose a computer program comes out that helps you increase your batting average from .290 to .295 (a BENEFIT of +0.005 with statistical significance), should you buy it?
     
     • It depends. If the software costs $15 or maybe even $150, then it’s a great deal and you should buy it. HOWEVER, if the software costs $150,000 then you might think it was NOT worth it – that the BENEFIT was NOT worth the COST.
     • (In this case, the cost per unit can be determined by calculating how much more the new program would cost than the old program and dividing the difference by 6.) If the cost per unit is subjectively judged to be excessive, the researcher would conclude that the new program is not of practical significance.
     • Essentially, this looks at the COST PER UNIT of BENEFIT.
2. SIDE-EFFECTS – Researchers who study the effects of prescription drugs look at two things 1) the drug’s effectiveness, and 2) negative side effects.
   
   • This is another type of cost-benefit analysis: Even if the drug is very effective (the BENEFIT) the negative side-effects might be so severe (the COST) that the drug is unusable.​
3. ACCEPTABILITY – The statistically significant option must also be acceptable to the parties involved.
   
   • Ex: If statistics indicate that there are benefits in requiring public school students to wear uniforms but the parents in a particular school district are strongly opposed to uniform requirements, a policy requiring them might not be politically acceptable to the school board.
4. LEGAL and ETHICAL ISSUES – Programs, treatments, and other procedures that are found to be statistically superior but are illegal or unethical obviously should be avoided.