BRM Flashcards
5 levels of evidence strength?
- meta analysis
- experimental studies
- correlational studies (longitudinal and cross sectional)
- qualitative studies
- ad hoc personal observations
2 most important concepts of study design?
validity| reliability
Define validity
Are we really measuring what we say we are measuring?
Define reliability
Can our measure produce the same results everytime?
- Provided the thing we are measuring is not subject to change.
- For example, personality is supposed to be a stable trait and therefore a measure of personality should get the same (or very similar) results when a person is tested at different times.
- However, some things such as attitudes can change and the very point of our study may be to see if a certain intervention changes attitudes.
- For example, we might want to see if we can change attitudes to engaging in pro-environmental behaviours.
Existing data advantages
- If open source it is easy to access.
- Most often does not require lengthy ethical assessments.
- For certain types of study, it is the only practical source of information (see Lecture 2).
Existing data disadvantages
We must think carefully about its validity – is it really a measure of what we think it is?
- Qualitative data needs skilled manipulations - especially in relation to reliability (see Lectures 3 and 4).
Define coding
the process of classifying observables such as behaviours into specifically defined categories for data analysis
What is a code book/scheme?
descriptive document that explains how data has been defined and classified in order to be converted into numerical (often categorical) data
Observational studies - advantages
Observations allow us to study what people actually do.
Observational studies - disadvantages
Can generate enormous quantities of qualitative data.
- It can be hard to decide what is important.
- Can be very difficult and time consuming to convert into data.
- Requires experience to develop reliable data coding schemes.
- If the observer is present (or known about) it can change the behaviour of the observed.
- If the observer is not known about it there are a lot of ethical issues to be addressed.
- Because the participants have not given consent to be observed.
Questionnaires - advantages
- A relatively large number of people’s responses can be collected.
- The participants may be widely geographically distributed.
- Reliable scales can be developed.
- Scales allow direct comparability between subgroups’ responses.
- Or direct comparability at two or more different time points.
They are less time consuming than interviews.
Questionnaires - disadvantages
- They require a lot of skill to do well.
- The issues of concern are pre-determined by the researcher.
- There is no opportunity to explore issues in more depth.
- They have poor response rates.
Interviews - advantages
Allow the researcher to get detailed information about what the participants think.
- Are not totally based on the researchers’ preconceptions and allow follow up of unforeseen issues.
Interviews - disadvantages
- All the same problems as other qualitative data collection techniques, i.e. what to do with large quantities of descriptive data.
- Responses may be affected by the individual interviewer.
- Participants may not want to reveal information about sensitive subjects in a one-to-one situation.
- Very time consuming.
- Often difficult to make comparisons between participants.
Sorting tasks - advantages
Can access the way people think about a specific domain without researchers’ preconceptions.
- Can reveal concepts that participants find difficult to verbalise (e.g. expert knowledge).
- Useful for understanding knowledge outside the researchers’ area of expertise.
sorting tasks - disadvantages
Difficult to initially develop.
- Time consuming.
- Difficult to analyse.
- Only suitable for small samples of participants.15-25
What is a profile?
The results sorted items produce against the categorical measures they are assessing.
It shows the way people think about the items
What is the Likert scale?
an odd number of responses to elicit a response about an emotional attitude or behaviour. usually 1 to 5 or 1 to 7.
What is a t-test?
The t-test assesses whether the means of two groups are statistically different from each other. This analysis is appropriate whenever you want to compare the means of two groups, and especially appropriate as the analysis for the posttest
-only two-group randomized experimental design e.g. time taken to walk in vs out of a dentist
What types of data might we never have?
That which is withheld for security reasons.
- In-house security research has the full data.
- Those of us working in academia have to live with it.
- Develop and test models based on what we have (see Lecture 5 BRM for example).
Where those present are killed.
- Both victims and offenders.
- A potential systematic bias – (e.g. does victim resistance work) – we have little or no data for those cases with the very worst outcomes.
Other biases: Eye witness and victim testimony.
- Victims and witnesses of violent and traumatic incidents have unreliable memory.
- Many academic studies.
What is Crenshaws list?
unofficial list of reputable news sources:
BBC
Telegraph
Observer
Independent
Times
Guardian
Washington Post
New York Times
CNN
Types of media bias
- Occur near to base (theirs),
- Have the potential to affect their own readers,
- Are “unusual”,* Or result in loss of life.
- “Low key” incidents and unsuccessful missions may not be covered.
- Results in public perception that (for example) hijacks don’t happen any more and that terrorism is aimed only at killing people.
- E.g. Many terrorist attacks are against property – but these are rarely reported.
- Is the media “forcing” terrorists to take more serious action?
- Media does not report unsuccessful crimes very often.
Selective reporting makes violent crime seem more prevalent than it really is.
- There is a whole literature on “Fear of Crime”.
- Those who are most fearful are not those most likely to be victims.
- Contagion or Copycat crimes.
- There are a number of crime types that are thought to be prone to so called “contagion” effects.
- There are studies that demonstrate this statistically. - E.g. Hijackings, product tampering, mass shootings. (And suicide).
- News restrictions/blackouts.
Basic issues of media sources
Choose the most reliable news sources possible – “Crenshaw’s List”
- Use more than one account for each crime studied.
- We tend to use three reputable sources and compare the content.
- “Triangulation” of sources.
- What if they don’t agree?
- Two out of three agree? If they all disagree get a fourth.
Press Contradictions on detail - reasons?
Breaking news may not yet have very many details available to report.
- Therefore contradictions between reports may be time related…
- Were the reports published at different stages of the incident meaning new information may have only just come to light?
- This is why for your exercise I said choose reports close to the time the crimes were committed.
- But in real research you will rarely be creating data on breaking news and have time to sample reports over a period of time.
- GTD allows for updates if more information becomes known. But only uploaded once a year.
Reasons for omissions in news reported data
That is one source reports information that another source does not.
- This demonstrates why more sources provide better coverage.
- Creating data this way is incredibly time consuming.
- Which is why most researchers just reach for an established data base.
- Even if it is not quite right for their research questions (more on this in later weeks).
Security reasons
WHat is triangulation of sources?
where you take three sources and go with the majority (minority report) and if all 3 contradict, then widen the number of sources used till a correlation is found (using Crenshaws list as a limiting factor)
is there such a thing as a terrorist profile?
NO
Can offender profiling be used to predicit if someone will commit a crime
No
what is inter-rater reliability?
the correlation between different raters using the same code scheme
What might codebook variables look like?
Textual - name of country, city etc
Numerical - number of perpetrators, victims, deaths, injuries, hostages etc
Categorical - qualitative options eg weapon type - hand gun, long gun, bladed weapon, improvised weapon etc
Dichotomous - did X happen? e.g. ransom demand, hostages taken etc
Difference between objective and subjective variable
Objective - facts - eg date of an incident, location, weapons used etc
Subjective - need careful defining as open to significant interpretation - eg motives
Rules to construct variable definitions (5)
- Don’t over interpret your data
- Avoid scales
- Categorical variables must be inclusive
- The categories of categorical variables must be mutually exclusive
- Are you dealing with missing data?
Define - 1. Don’t over interpret your data
- Don’t read into things…
- For example, avoid making assumptions about why the perpetrator is doing something (unless this has been established externally some how, e.g. in a note they left).
- Stick to observables that can be said to have happened or not.
- For example, while preparation and planning are potentially important, identify the specific behaviours that you are taking as indicators of these things and turn them into variables rather than the more abstract concepts themselves.
- For example, any of these phrases in a definition will affect your reliability.
- “More violence than was necessary”
- “A great deal of violence”
- “Were highly aggressive”
- Because they require your second coder to make a judgement which will introduce potential unreliability
Define - 2. Avoid scales
Avoid anything that involves judgements.
- Scales (e.g. 1 to 5) require an opinion from the coder.
- E.g. low violence, medium violence, high violence.
- This is just a judgement call.
- You need operational definitions.
- You need to specify exactly what coders should take into consideration.
- Try to make it as easy as deciding whether something happened or it did not.
Define - 3. Categorical variables must be inclusive
- If your variable is categorial you must account for all eventualities.
- Not just your 3 cases here – imagine this will be a real research project with many more cases to be included.
- Your scheme must be able to accommodate many more other cases that may arise.
- Often the other categories will only become apparent as you start to code a larger sample.
- This is fine – you can alter it as you go along – until your data are ready for analysis.
- Many schemes (e.g. GTD) have an “other” category for things they haven’t expected.You can change with time, BUT must change the historical data when you do
Define - 4. The categories of categorical variables must be mutually exclusive
- This is a problem for many published schemes.
- They appear to have started with categories that are NOT mutually exclusive.
- And then they have had to solve it in less than optimal ways.
- Because it would be too difficult to go back and change things when you have already invested in the scheme.
- You see it in the GTD. e.g. weapons - what if they have more than one weapon type?Does no mean no or just ‘not reported’?
- Are you dealing with missing data?
- Do you need a “unknown” category?
- Is it really a no or is it just not reported?* For example, did the hostage takers verbally abuse the hostages? Yes/no
- The first case you read says that they did, so you can code “yes”.
- The next case says that they were very polite to the hostages, so you can code “no” they definitely didn’t.
- The third case doesn’t mention anything at all about what they said to the hostages.
- So it can’t be a “yes” or a “no”, it must be unknown?
- You need a third category for unknown.
- Keep as much data as possible when first coding because you can always collapse the categories later but you can’t get the original back again.
Why do we study expert decision makers?
to improve decision making|
to understand it
to develop decision support systems
Why is it difficult to understand expert decision makers?
Researchers do not get access to the people
the commissioning might be through the decision makers themselves
Access to the data might be difficult to manage
What is a retospective study?
where you go back over a closed case for people to talk through how decisions were made.
but difficult due to confidentiality
What is factor analysis?
Statistical analysis comparing highly correlated variables
Eg I like going to parties might link to I enjoy meeting new people, and with strong agreement with other statements
can lead to identification of personality traits
What are 2 types of factor analysis?
Exploratory| Confirmatory
What is exploratory analysis?
No set hypothesis
Use data to make groups
What is confirmatory analysis?
Start with groups and see if data fits the mold
Define survey
A Survey is a general term for any kind of data collection on people’s opinions or behaviour.
E.g. A market researcher asking how you found the facilities at an airport.
Define questionnaire
A questionnaire is a set of questions (usually written) that people are asked to respond to – usually on a rating scale.
Thus, a questionnaire can be used as part of a survey
What people do, not how they think
What, how, when, how many…
What simple response categories can be used in questionnaires?
- Open ended: the person writes their answer.
- Which brand of toothpaste do you buy?
- How old are you?
- Category tick boxes: Are you aged 18-25, 26-35, etc.
- When should you have categories not actual replies?
- “Sensitive” data; Arbitrary banding; Unable to recover detail.
- Which of these news sources do you read? (tick all that apply)
- How often do you read them? (daily, weekly, monthly).
- If the question needs them to choose only option from a selection…
- Then just like your variables for Content Analysis the categories must be exhaustive and mutually exclusive.
- The use of “Other” as a category.
- Transport to work example…all that apply? Or the longest distance?
What value is used for a missing answer for questionnaires?
either 9 or mode
Ways to deliver a questionnaire?
telephone survey
researcher led questioning
paper based questionnaires
Online surveys
Telephone surveys pros and cons
- Can get to a wide variety of people geographically.
- How will you select your sample?
- Random number dialling?
- Not everyone has a landline or a mobile.
- And this is related to age – i.e. a biased sample.
- People are often annoyed by cold callers.
- It will rarely be a good time for them.
- You could pre-arrange a good time.
- With postal questionnaires, at least they can fill it inwhen they choose. But they mainly forget or throwit away.
- Telephone surveys are not used very often.
Researcher read questionnaire pros and cons
The researcher reads the questions to the participants and fills in the answers for them.
- Can be pre-arranged by appointment…
- Or opportunistic, e.g. asking people in a public place.
- Often used by market researchers on the street.
- This is the same as a structured interview with pre-determined response choices.
- The challenge is getting people to stop and take part.
- Useful if the respondent has difficulties with reading/writing.
Paper based questionnaires. pros and cons
- Paper based questionnaires. Can be:
- Handed out to people face to face, and:
- Completed there and then (e.g. in a class).
- Taken away and return by post or to a central point.
- Sent out in the post.
- The challenge is getting people to send it back.
- See response rates.
- You must report your response rates – i.e how many you sent out compared to how many you got back.
- Your response rates will be low.
