W4 Readings

Question 1

Validity and reliability of measurement instruments used in researched
Kimberlite article

Question 2

Reliability

Answer 2

The obtained and observed score through a measuring Instrument is composed of both the true score which is unknown and error in the measurement process

Question 3

True score

Answer 3

The score that a person would have received if the measurement were perfectly accurate

Question 4

Pre-testing or pilot testing

Answer 4

An instrument allows for the identification of such scores refinement of the instrument than focusses on minimizing measurement error

Question 5

Reliability estimates are used to evaluate
1

Answer 5

1.The stability of measures administered at different times to the same individual
2The equivalent of sets of items from the same test (internal consistency) Or of different observers scoring a behaviour or event using the same Instrument (Interrater reliability)

Question 6

Reliability coefficient range from…..

Answer 6

0-1
Higher coefficient means higher levels of reliability

Question 7

Stability

Answer 7

Is determined by administering the test at two different points in time to the same individual and determining the correlation or strength of Association of the two sets of scores

Question 8

Internal consistency

Answer 8

Gives an estimate of the equivalent of sets of items from the same test
Ex,A set of questions aimed at assessing quality of life or disease severity
The coefficient of internal consistency provides an estimate of the reliability of measurement and is based on the assumption that items measuring the same contract should correlate

Question 9

What is the widely used method for estimating internal consistency reliability

Answer 9

Cronbatch alpha

Question 10

What is cronbatch’s alpha¿

What is it used for?

What makes cronbatch Alpha high? Major gains?

Answer 10

A function of the average intercorrelations of items and the number of items in the scale

Used for summmated scales such as quality of life Instruments

All things being equal, the greater the number of items in the summated scale

Additional items up to approximately 10, when the increase in reliability for each additional item levels off. This is one reason why the use of a single item to measure a construct is not optimal

Having multiple items to measure a construct aids in the determination of the reliability of measurement and in general improves the reliability or precision of the measurement

Question 11

Interrater reliability /Intero observer agreement

Answer 11

Establishes the equivalent of readings obtained with an instrument when used by different observers

If a measurement process involves judgements or ratings by observers, a reliable measurement will require consistency between different raters

Requires completely independent ratings of the same event by more than one rater

No discussion or collaboration can occur when reliability is being tested

Reliability is determined by the correlation of the scores from two or more independent raters Or the coefficient of agreement of the judgements of the Raters 

Question 12

For categorical variables, _______ _____ is commonly used to determine the coefficient of agreement.

______ Is used when two raters or observes Classify events or observations into categories based on rating criteria


Answer 12

For categorical variables, Cohens Kappa is commonly used to determine the coefficient of agreement.

Kappa Is used when two raters or observes Classify events or observations into categories based on rating criteria

Rather than a simple percent agreement, KAPPA takes into account the agreement that could be excepted by chance alone

Question 13

Interrater reliability should be establish when datas are abstracted from medical charts or when diagnosis or assessments are made for research purposes
It also depends on…

Answer 13

Developing precise operational definitions of variables being measured as well as having observers well trained to use the instrument

it is optimized when criteria are explicit and raters are trained to apply the criteria

Question 14

Raters must be trained how to make a decision

Answer 14

I that and event has occurred or how to determine which point on the scale measuring strength or degree a phenomenon should be applied.

Question 15

The more that individual judgement is involved in the reading, the more crucial it is the independent observers agreed when applying the scoring criteria.

Question 16

Training should include multiple cases such as

Answer 16

They respond to simulated situation they will encounter and rate, interrater reliability is calculated disagreements or clarified and a criterion level of agreement is Met

Question 17

Interrater reliability you should be verified

Answer 17

Throughout the study even when established observational instruments are being used or criteria or explicit research That relies on observations or judgements should check reliability and the study protocol should include procedures to determine the level of observer agreement

Question 18

I percentage of observation such as number of charts reviewed his randomly selected for scoring by two independent readers rather than requiring that to raiders judge all observation
Data to establish the consistency…

Answer 18

With which the primary rater applies the criteria overtime are important for establishing the reliability of an instrument

Question 19

Rater drift can occur when 

Answer 19

An individual rater alters the way he or she applies the scoring criteria such as becoming more lenient or stringent over time 

Question 20

Investigators who build and reliability checks throughout the study is data or collected rather than waiting until the end of data collection can identify instances where into regular reliability has become to deteriorate perhaps to the 

Answer 20

Rater drift

Question 21

Validity

Answer 21

The extent to which an instrument measures what purports to measure

Question 22

Validity requires

Answer 22

That’s an instrument is reliable but an instrument can be reliable without being valid

For example the scale that is incorrectly calibrated may healed exactly the same but in accurate weight values

Multiple choice test intended to evaluate the counselling skills of pharmacy students May yield reliable scores but it may actually Evaluate drug knowledge rather than the ability to communicate effectively with patients and making a recommendation

Question 23

Validity is not a property of the test itself instead Validity is…

Answer 23

The extent to which the interpretation of the results of a test are warranted which depend on the test intended use such as measurement of the underlying construct

Question 24

Much of the research conducted in healthcare involves quantifying attributes that cannot be measured directly. Instead hypothetical or abstract concepts (________)

Answer 24

Constructs Such as severity of disease, drug efficiency, drug safety, burden of illness, patient satisfaction, health literacy, quality of life, quality of provider patient communication, and adherence to medical regimes Are measured 

Question 25

Hypothetical constructs cannot be

Answer 25

Measured directly and can only be inferred from observations of specified behaviours or phenomenon that are thought to be indicators of the presence of the contract

Question 26

Measurement of a construct requires that the…

Answer 26

A conceptual definition will be translated into an operational definition

Question 27

An operational definition of a construct…

Answer 27

Links to conceptual or theoretical definition to more concrete indicators that have numbers applied to signify the amount of the construct

Example, efficiency of new drug product. The ability to improve a patient’s health may be measured by the decrease of certain symptoms, the delay in onset of certain diseases, length of remissions, or the prevention of certain clinical complication. Like why is the theoretical construct of medication adherence may be operational define as a one month recording of number of miss those as measured by medication event monitoring system Which include micro processors That record occurrence and time of each opening up a prescription file

Question 28

an operational definition of patient satisfaction with healthcare might be

Answer 28

Patient self reported responses to items on the 18th item short form version of the patient statisfication questionnaire

Question 29

Crocker and Algina Have pointed to the importance of a theoretical foundation by noting

Answer 29

That Constructs cannot be defined only in terms of operational definitions but also have demonstrated relationships to other contstucts or observe able phenomenal

Question 30

Construct validity

Answer 30

A judgement based on the accumulation of evidence from numerous studies using a specific measuring instrument

Evaluation Of construct validity requires examining the relationship of the measures being evaluated with the variables known to be related or theoretically related to the construct measured by the instrument

For example a measure of quality of life would be expected to result in lower scores for chronically ill patients then for healthy college students. Correlations to fit the expected pattern contribute evidence of Construct validity.

All evidence of validity including content and criterion related validity, contributes  to the evidence Of construct validity

Question 31

Content validity

Answer 31

Addresses how well the items developed to operationalize a construct provide an adequate and representative sample of all the items that might Measure the construct of interest

Because there is no statistical test to determine whether a measure adequately covers the content area or adequately represents a construct….

Content validity usually depends on the judgement of experts in the field

Question 32

Criterion related validity

Answer 32

Provides evidence about how well scores on the new measure correlate with other measures of the same construct or very similar underlying constructs that theoretically should be related
Crucial that these criterion measures are valid themselves 

Question 33

With one type of criterion related validity ________ ________ The criterion measurement is obtained at sometime after the administration of the test, and the ability of the test to accurately predict the criterion is evaluated
What is an example of this?

Answer 33

Predictive validly

For example surrogate Out come such as blood pressure and cholesterol levels are based on their predictive validity In projecting the risk of cardiovascular disease, even though some of these associations have been recently question

Question 34

Another type of criterion related validity is_________,Scores on an instrument or correlated with scores on another criterion measure of the same construct were highly related construct that is measured concurrently in the same subjects. Ideally, the criterion measure would be considered to be the gold standard measure of the construct. The strategy of determining the validity of a measure might be seen in a situation in which a new instrument has some advantage over the gold standard measure. Disadvantages would justify the time and effort involved in the development and validation of a new instrument.

Example, a clinical research are wanting to use a brief screening instrument for condition, such as depression, instead of administering a more expensive measure.

Cost of administration of the best criterion measures they also have a barrier 

Answer 34

-concurrent validity

Question 35

Responsiveness

Answer 35

The ability of a measure to detectChange over time in the construct of interest

For outcome measures intended to evaluate the effects of medical or educational interventions, Responsiveness two changes that result from the intervention is required

Question 36

What is a crucial component of responsiveness?

Answer 36

Reliability. The noise that is due to measurement Error mask changes that may in fact be attributed to the intervention

For example, using a scale manufactured to weigh trucks will not be helpful when evaluating a new weight loss drug In humans because the estimates will be too imprecise to identify small changes. The measurement will be valid yet unreliable or imprecise

Question 37

Responsiveness to change can legitimately different from one population to another, which is why the measure must be appropriate to the subjects being studied. Give an example of this

Answer 37

A measure of activities of daily living that includes the ability to dress or wash oneself may be responsive to change among an elderly population of patients undergoing physical therapy or cardiac rehabilitation. However, it would probably not be sensitive to change due to a ceiling affect among oh younger group of newly diagnosed hypertensive patients who have not experienced significant disability due to disease or to the ageing process

Question 38

Selecting an Existing instrument 

Answer 38

Before developing a new test or measure an investigator should identify existing instruments that measure the construct of interest. Using an existing instrument that has Substantial evidence of reliability and validity in a variety of populations is more cost effective than starting from scratch to develop and validate an instrument.

Question 39

In selecting an instrument the following questions should be addressed:

Answer 39

1.Do instruments already exist that measure a construct the same or very similar to the one you wish to measure?Before you begin searching for instruments, you must have a clearly defined construct or concept that you wish to measure, along with operational definitions and some evidence that the construct can be measured as defined
2. How well do the constructs in the instruments you have identified match the construct you have conceptually defined in your study? And evaluating whether there is congruence, do not rely on the title of the measure or on the operational definition of the construct that appears in a research article or the description of variables in a secondary data base.Real understanding of the measure usually requires an examination of the actual items or questions and the way dad were generated or documented.
3.Is the evidence of reliability and validity well established? Has a measurement evaluated using various types of reliability instruments such as both internal consistency and test retest and varied strategies for establishing validity such as content and concurrent validity as well as more extensive evidence of construct validity in very population? Has it been validated in a population similar to the one you will be studying?
4.In previous research, was there variability In scores with no floor or ceiling affect? The previous studies have a large amount of missing data, either on the measure itself or on items within the measure?
5.If the measure is to be used to evaluate health outcomes, effects of intervention or changes over time are there Studies that establish the instruments responsiveness to change in the construct of interest? Obviously it is important that change in measurement we do the change in the construct rather than the instability of scores such as lacking reliability of the measure itself. In addition it would be helpful if there were data on how much change in scores would be required to be considered clinically meaningful.
7.How expensive is it to use the instrument? Email questionnaire cost less and require less time to administer them do telephone or face-to-face interviews.However, electronic data may not contain information that is available on patient charts so a good understanding of the limitation to data available as well as requirements of measurements for your study is important
8.If the instrument is administered by an interviewer Or if the measures require uses of judges or experts,How much expertise or specific training is required to administer the instrument?
9.Will the instrument be acceptable to subjects? Does the test require invasive procedures? Is the reading level appropriate? Is the respondents burden, included complexity of questions and time needed to complete the instrument, unlikely to affect response rates of the quality of responses?

Question 40

Reliability and validity evidence from establish instruments is a applicable only…

Answer 40

If you use the instrument in the same form and follow the same administration procedures has used in the validation study.Modifications of validate instrumentsMay require permission from developers and also require validating the modified instrument as if it were a new instrument

Question 41

Researchers may be tempted to conclude that..

Answer 41

They must develop their own instruments.
They may view the measures they want to develop as being so straightforward such as a few questions measuring patient knowledge or specific item from a medical chart, that they do not need to conduct a pilot Test to determine reliability and validity.

Researchers me then go to considerable After collecting data only to find at the end of the study that subject to not very much and their responses to the instrument or the documentation in the charts was in adequate, so the measure was not able to correlate with any other variable of interest

Subjects may miss interpret questions.
Responses may be highly skilled
Internal consistency maybe so low the item responses cannot Reasonably be combined into a single summmated score.

Question 42

And other studies,

Answer 42

Researchers may obtain biased results by incorrectly assuming the diagnostic codes are valid without determining the relationship to other measures that should indicate the presence of the disease. Assuming medical records adequately capture the information needed to construct a measure in the chart review were’s will interpret information uniformly can also threaten the validity of findings.
Careful attention to the development of instruments regardless of how straightforward to measure may seem, along with pilots testing to determine the reliability and validity is crucial to conduct of quality research

Question 43

Item response theory

Answer 43

Provide an alternate framework for understanding measurement and alternative strategies for judging the quality of a measuring instruments.

Building item pools and developing questionnaires that measure key health outcomes related to many chronic diseasesIncluding measures such as fatigue and pain. These items will be available to investigators, and the Repository will become a resource for accurate and efficient measurements of patient reported symptoms and other health outcomes in clinician practises

Question 44

Measurements using self-report

Answer 44

With surveys, researchers rely on responses to questions to provide measurements of the constructs of interest. While self reports of behaviours beliefs and attitudes are prone to know when biases, they are no acceptable alternative means of measurement for many constructs such as level of pain depression patients abdication with care quality of life

Self-report questions May elicit An estimation of behavioural frequency rather than the recall and count response desired by the researcher.It is the use of estimation rather than recall which is a function of how information is retrieved from memory, how frequency response scales are formulatedAnd other specific aspects of instrument.

Behaviours that occur with high frequency such as dietary intake or taking a scheduled medication for a chronic condition or not likely to be a specific in memory for a very long period of time. If it is desired that specific events be recalled rather than estimated, the timeframe must be a very short duration in the immediate past

Therefore ask impatiens how many doses of medication they missed in the past month or past year will likely result in an estimate or educated guess where as a questionable the past 24 hours or past three days may reflect actual recall

Response choices require subject pride their own judgement about frequency using undefined response alternatives such as an ordinal scale from seldom to frequency

When I ask you questions or frequency of behavior, it is usually best to let the subject fill in the blank on an item with a clearly defined reference period
Ex,How many doses of specific medication have you missed taking completely in the past three days?Open format requires a specific description of the behaviour interest as well as a specific time frame


Question 45

Use of self-report or poorly designed measures can result in…

Answer 45

Misclassification bias which is error in classifying either exposure status or effect (e.g disease ) in patients or subjects

Patient recall of Previous drug exposure, for example, has been shown to be subject to error

In case - control studies, Recall biases of concern when there are no objective markers of exposure. Individuals with the disease or outcome of interest are more likely to remember relevant exposures than are healthy controls.

One approach that is recommended to address this recall bias is to have a control groupAffected by a disease different from that of cases to introduce a similar biased toward recall of exposure

Question 46

Use a secondary data
What does this include?

Answer 46

Data originally gathered for a different purpose are often used to answer a research question

These data may have addressed a different research question Or may have been gathered for clinical, billing, or legal purpose

This includes Pharmacy records, electronic or paper medical records, patient registries, and insurance claims data.

Question 47

What is the first consideration when deciding whether secondary data can be used?

Answer 47

Is to verify that the data set appropriately measures the variables require to answer the research question. If the data elements are not present, consideration can be given to weather appropriate proxy measures a variables of interest are available.

Question 48

The use a proxy measures requires

Answer 48

Careful conceptual analysis OfHow closely the variables of interest and proxy measures are associated. For example it seems intuitive that the claim database could be used to identify all patients who suffered a stroke during a certain time. As long as they were eligible for benefits. However strokes may have been silent and required no medical intervention, patient may have died before medical care could be sought. Stroke may have been missed diagnosed, or certain medical services may not have been covered by the insurance company and us may not appear in the billing databases

Question 49

Use of surrogate measures

Answer 49

Surrogate endpoint of clinical trials as a Laboratory measurement or a physical sign used as a substitute for a clinically meaningful and point that directly measures how patient feels, functions, or survived.

Changes induced by a therapy on a surrogate How come expected to reflect changes in clinically meaningful end point

Do use of surrogate outcomes to operationally defined a construct, such as drug efficacy Has become increasingly popular, as application of these measures is typically faster and less costly

Results are obtained after shorter follow up periods, and the number of patients and length of time patients have to participate experiments are reduced.

For a surrogate I’ll come to be valid, it should be in the direct pathophysiological pathway of a disease End it should be reasonable to expect that the pharmacologic action of the new drug is mediated through this pathway. If these two conditions are true, the drug affect on the surrogate Outcome can be Extrapolated toward true measures of morbidity or mortality. However even well-established surrogate outcomesHave recently been questioned

Surrogate outcomes Remain nothing more than substitutes and can only approximate the truth

Question 50

In healthcare and social science research, many of the variables of interest and outcomes that are important are abstract concept known as _______ _________

Answer 50

Theoretical constructs

Using test or instruments that are valid and reliable to measure such constructs is a crucial component to research quality