W12L15&16 - Method Comparison and Evaluation Flashcards
Aspects to Consider for Quantifiable Measurements
Precision Linearity Limits of detection and quantification Specificity and interference Reagent/sample carry over How results compare between measures
When might you evaluate/compare new methods?
Current method unsatisfactory/could be improved
- analytical (simplicity, imprecision, sensitivity, throughput etc.)
- cost of resources
- reliability of supplier/distributors/tech support
- safety
Acquiring new instrument to complement existing or for a new setting/POCTs
New disease marker
Laboratory reorganisation/restructure/take-over
Verification method/instrument manufacturer’s claims requirement for compliance
How do you compare analytical performance?
T-test - summary statistics Chi-squared test X-Y regression analysis Difference plot Linearity/interferences/recovty
Principles to Successful Evaluation of New Method
Consider the clinical perspective Consider the analytical goals required Select analytical method Conduct the experiments/evaluation Use appropriate statistical tools and methodology to estimate differences between methods
Strategy in Analytical Performance Goal Setting: Stockholm’s Consensus on Hierarchy
- Evaluation of the effect of analytical performance on clinical outcomes in specific clinical settings
- Evaluation of the effect of analytical performance on clinical decisions in general:
- data based on components of biological variation
- data based on analysis of clinician’s opinions - Published professional recommendations
- Performance goals set by:
- regulatory bodies
- organisers of External Quality Assessment schemes - Goals based on the current state of the art
Milan Model
Model 1: based on the effect of analytical performance on clinical outcomes
- direct/indirect outcome studies
Model 2: based on components of biological variation of the measurand
Model 3: based on state-of-the-art
- relates to highest level of analytical performance technically achievable
Comparing Results between Methods
Visual representations
- scatterplot
- difference plots
Statisitical approaches
- T test to test for differences between means
- calculating correlation coefficient
- regression analysis
Context
- how differences are interpreted depends on context
- sometimes results expected to be different
T-Test
Test for differences between means
Paired and unpaired t-tests
Paired, same sample assayed by different methods
Very easily done, but only summarises overall bias
Gives equal weight to all differences
Not that useful
Correlation Coefficient
Not that useful by itself
Gives a measure of the strength of the linear relationship between two variables
Doesn’t give a measure of the degree of agreement or the relationship between two sets of data
Expect a high correlation
Regression Analysis
Much more useful than correlation
Regression line defined as the line where the square of deviations from the drawn line to the data is the least
Several types including normal and Deming regression
Gives:
- measurement of agreement
- relationship between X and Y including degrees of constant and proportional bias and scatter about regression line
Causes of Systematic Error
Change in reagent or calibrator lot numbers
Wrong calibrator values
Improperly prepared reagents
Deterioration of reagents/calibrators
Inappropriate storage of reagents/calibrators
Pipetting misalignments
Causes of Random Error
Air bubbles in reagent Improperly mixed reagents Reagent lines, sampling or reagent syringes Improperly fitting pipette tips Clogged or imprecise pipette Fluctuations in power supply
What is R squared?
A statistical measure of how close the data are to the fitted regression line
Also known as the coefficient of determination
Tells you how much of the variability of the y variable is predicted by the x variable
The higher the R-squared, the better the model fits your data
