Content analysis - reliability Flashcards
what is inter-rater reliability?
the correlation between different raters using the same code scheme
What might codebook variables look like?
Textual - name of country, city etc
Numerical - number of perpetrators, victims, deaths, injuries, hostages etc
Categorical - qualitative options eg weapon type - hand gun, long gun, bladed weapon, improvised weapon etc
Dichotomous - did X happen? e.g. ramson demand, hostages taken etc
Difference between objective and subjective variable
Objective - facts - eg date of an incident, location, weapons used etc
Subjective - need careful defining as open to significant interpretation - eg motives
Rules to construct variable definitions (5)
- Don’t over interpret your data
- Avoid scales
- Categorical variables must be inclusive
- The categories of categorical variables must be mutually exclusive
- Are you dealing with missing data?
Define - 1. Don’t over interpret your data
- Don’t read into things…
- For example, avoid making assumptions about why the perpetrator is doing something (unless this has been established externally some how, e.g. in a note they left).
- Stick to observables that can be said to have happened or not.
- For example, while preparation and planning are potentially important, identify the specific behaviours that you are taking as indicators of these things and turn them into variables rather than the more abstract concepts themselves.
- For example, any of these phrases in a definition will affect your reliability.
* “More violence than was necessary”
* “A great deal of violence”
* “Were highly aggressive”
• Because they require your second coder to make a judgement which will introduce potential unreliability
Define - 2. Avoid scales
Avoid anything that involves judgements.
• Scales (e.g. 1 to 5) require an opinion from the coder.
• E.g. low violence, medium violence, high violence.
• This is just a judgement call.
• You need operational definitions.
• You need to specify exactly what coders should take into consideration.
• Try to make it as easy as deciding whether something happened or it did not.
Define - 3. Categorical variables must be inclusive
- If your variable is categorial you must account for all eventualities.
- Not just your 3 cases here – imagine this will be a real research project with many more cases to be included.
- Your scheme must be able to accommodate many more other cases that may arise.
- Often the other categories will only become apparent as you start to code a larger sample.
- This is fine – you can alter it as you go along – until your data are ready for analysis.
- Many schemes (e.g. GTD) have an “other” category for things they haven’t expected.
You can change with time, BUT must change the historical data when you do
Define - 4. The categories of categorical variables must be mutually exclusive
- This is a problem for many published schemes.
- They appear to have started with categories that are NOT mutually exclusive.
- And then they have had to solve it in less than optimal ways.
- Because it would be too difficult to go back and change things when you have already invested in the scheme.
- You see it in the GTD. e.g. weapons - what if they have more than one weapon type?
Does no mean no or just ‘not reported’?
- Are you dealing with missing data?
- Do you need a “unknown” category?
- Is it really a no or is it just not reported?
- For example, did the hostage takers verbally abuse the hostages? Yes/no
- The first case you read says that they did, so you can code “yes”.
- The next case says that they were very polite to the hostages, so you can code “no” they definitely didn’t.
- The third case doesn’t mention anything at all about what they said to the hostages.
- So it can’t be a “yes” or a “no”, it must be unknown?
- You need a third category for unknown.
- Keep as much data as possible when first coding because you can always collapse the categories later but you can’t get the original back again.
