Coding and Reliability

Question 1

Why are coding schemes applied?

Answer 1

To produce data - data is what you get out. this word is not standardised in psychological literature, people think of data in different ways

Question 2

What is a coding scheme?

Answer 2

A measurement instrument

Question 3

What are the principles of measurement?

Answer 3

No such thing as a perfect measurement, but they should be:

Accurate - correct or valid measurement, degree to which they capture phenomena of interest

Precise - degree to which measurements are reliable or replicable

Question 4

Problems with some coding schemes

Answer 4

if a baby shows distress, code yes, otherwise no:
misses change in intensity of crying
uses arbitrary clip boundaries, instead of relevant events
scheme does not capture differences in the intensity of baby’s responses to different social contexts

Question 5

How can you adjust a coding scheme so it isn’t just yes or no?

Answer 5

Define the observational intervals in terms of changes in social context - but still misses intensity

Define different levels of intensity of apparent distress

Question 6

What are the principles of design?

Answer 6

Codes should be:
mutually exclusive - each behaviour can only be put into one category
exhaustive - all possible behaviours have categories

Question 7

How can you ensure a coding scheme is exhaustive?

Answer 7

By having an other category - record things you didn’t anticipate

Question 8

Should you use established coding schemes?

Answer 8

No, because coding schemes are designed to answer questions, so if you are using someone else coding scheme, you will be asking the same questions they did - if you want to ask new questions, you need to design your own coding scheme

Question 9

What is reliability concerned with?

Answer 9

The precision of the coding scheme - not the precision of the observer. it is ambiguous from the reliability estimate whether the low reliability is due to weaknesses in the coding scheme themselves, lack of observation, poor training etc

Question 10

What can reliability only be applied too?

Answer 10

A specific coding scheme, by specific observers, in a specific context - demonstrating good reliability of a scheme in one context, doesn’t mean that it will be good with other observers in different contexts

Question 11

What is intra-observer reliability?

Answer 11

The same observer coded the same behavioural record at different times (only possible with video, audio)

Question 12

What is inter-observer reliability?

Answer 12

Different observers independently coded the same behavioural record at the same time or different times

Question 13

How much of a record should you code?

Answer 13

15% of the record

Question 14

When can reliability be assessed?

Answer 14

Beginning of a coding scheme - once reliability is demonstrated between 2 or more coders, they can independently code the rest of the behavioural record

End of coding

During the coding job, with a portion of the behavioural record assigned to two or more coders

Question 15

What are consensus estimates?

Answer 15

Based on the assumption that two or more coders can come to an exact agreement (percent agreement or kappa)
Usually used on nominal data, where different codes represent qualitatively distinct ideas

Question 16

What are consistent estimates?

Answer 16

Based on the assumption that it is unnecessary for two or more coders to interpret a scale identically, but the coders will consistently clarify phenomena with their understanding of the scale (persons r, cronbachs alpha)
Used on ordinal to continuous scale

Question 17

How can you measure consensus?

Answer 17

Percent agreement - does not correct for agreement by random chance
Cohens kappa - agreement after corrections for agreement by random chance

Question 18

How can you measure consistency?

Answer 18

Correlation coefficient - does not take into account variance between coders
Cronbachs a - corrects for variance between coders, can get reliability for more than two coders

Question 19

How do you calculate percent agreement?

Answer 19

Number of agreements (a+d) divided by total number of observations times 100

Question 20

Problem with percent agreement

Answer 20

Does not correct for agreement by random choice

Question 21

What is included in a confusion matrix?

Answer 21

A = number of agreements, yes yes
B = observer 1 no, observer 2 yes
C = observer 1 yes, observer 2 no
D = number of disagreements, no no

Question 22

When is kappa good enough?

Answer 22

There is no standard, but just need to cite sources
0 = no agreement
0.0.2 = slight 
0.2=0.4 = fair
0.4-0.6 = moderate
0.6-0.8 = substantial
0.8-1 = perfect

Question 23

Why is cronbach’s alpha good?

Answer 23

Because it corrects for the variance between observers, when there are more than 2 raters