Lecture 2 Flashcards
Validity of measurement
Research question of characteristic of media content
Each character requires valid measurement
Measurement validity: substantive accuracy of a measure - degree to which a measure covers a concepts content (reality, meaning) correctly and fully
Accurate measurement is harder for more latent characteristics
Manifest content
Directly observable (concrete) on its surface
Unambiguous meaning
Simple operationalisation
Single-item measure
Latent content
Not directly observable (abstract) below the surface must be revealed
Deeper ambiguous meaning
Multiple operationalisation - indicators
Measurement instrument with multiple items
The more latent a characteristic of media content:
The more meaningful and influential the context
The more meaning is judged constructed instead of observed by the coder
The more its meaning becomes dependent in pre existing mental schemes (frames) of the coder - projection
The more the focus of meaning making shifts from content to coder
More subjective (potential problem)
Manifest
AI or specific term (gender of actor) // middle is frames // latent: beauty
Intersubjectivity
Strong social norm
Operationalisation of latent media content (constructs)
/ Translation into a set of more manifest characteristics of content (indicators)
/ To break up -> to objectify quantitative
/ Indicators items combined must be covered the construct fully
Natural starting point:
Existing literature
Theoretical insights and empirical evidence
THeory must lead if it does not exist
Empirical ways to evaluate validity of measurement
(Instrument - validation)
Factor analysis
Construct validity (dichotomous in CA often - negative: they are not good for factor analysis)
Association between measurement instrument and:
Other version of same type of measurement (concurrent validity)
Plausible outcome variable (predictive validity)
Comparison outcomes with those of very different instruments (convergent validity)
(cross check among producers or consumers of content)
Social validity:
The social significance and meaning of the analysed media content beyond the academic community
/ among intended receivers of the content
/ combination of convergent and ecological validity: are researchers and users on the same substantive frequency?
Understand differently / audience understand in different ways than researchers
Should researchers determine the standard for meaning of content? Theoretical question
Is the chosen standard representative for that of the receivers of the content? Empirical question
Wider community beyond academic society?
Validity vs reliability of measurement
Reliability is a criterion for evaluating the quality of measurement
Agreement about categorising media content (among coders)
Reliability is a necessary but not sufficient condition for validity (reliability is a precondition for validity - no reliability > no validity)
Complication: In CA> both criteria of measurement quality are in tension with one another / raising one level might come on expense on of the other
Reliability» provide more instructions (1)
More coding rules
More detailed coding rules
Do more coder training
Creator mental focus on the rules
Consequence> coders will approach media content less naturally - like normal users
The social validity (ecological validity) of the measurement is impaired
(2) Simplify coding task:
Stronger focus on easiest observable aspects of latent
Remove difficult coding decisions
Reduce number of coding options / negative or positive
Consequence: The content reality of measured concept is too reduced
Content validity of measurement is impaired
Be realistic: a perfectly reliable and valid measurement instrument for latent media is impossible
Don’t fixate on just one criterion:
It’s self-defeating: a measurement must always satisfy both criteria: keep an eye on both
Content analysis still tends to be less attentive to validity
Don’t underestimate validity - matters more than reliability. Look for balance!
Coding process
Codebook: document containing all information relevant to the process of coding media content:
All variables to be coded (coding categories) for the study - full measurement instrument
Instruction to coders about when and how to code variables
Including coding rule and procedures that need to be followed systematically (quantitative)
Registration unit (observation unit):
Unit you judge when you code a variable - what is the unit level you make a decision about when you code a variable?
Reg unit can be: Textual media content: entire publication, web page, article, paragraph, sentence, word etc
Audiovisual media content (reg unit):
Entire production - movie, act, sequence, scene, shot etc
Reg unit
The choice of reg unit depends on the required level of detail for the study: more detail = more coding work!
Researchers can measure variables at different levels of registration within the same study (article level, or lower level)
Context unit:
Part of the content you take into consideration when you code a variable for each registration unit
Distinguished from reg unit only if the broader physical or temporal context of the media content you judge might influence the interpretation of users of that content > validity.
Mainly an issue with small reg units. Smaller the units, greater the importance of context
For large reg units: news article, often non-issue: reg and content coincide. (no broader context to take into consideration)
Does the photograph depict a human expression of sadness? Is it actually that? Broader context changed our interpretation on how to code the reg unit.
Reg unit should also include a caption.
Codebook
The codebook contains all variables to be coded (in the most logical coding order)
Variables needed to answer the research question
Optional: control variables (internal validity) Rule out alternative explanations
Mandatory: administrative variables (highly important)
Other variables than these: waste of time and effort!
Administrative variables
Registration unit ID variable
Unique identification number per coded reg unit (article ID)
Coder ID
ID of number by coder
Coder comments variable (string variable)
Text input to record doubts or problems regarding coding
Critical for dinding, correcting etc
BOTH ID necessary
Codebook
General and specific coding instructions that are as clear and comprehensive as possible:
Definitions of reg and context units, measured concepts and coding categories
Rules and procedures: what to decide in which coding situation? Systematic
Examples of coding situations and media content to illustrate rules and procedures
Anticipate a range of coding scenarios - explorative coding phase helps
Goal:
Switch off or standardise individual frames of reference of coders: objectivity
Assume prior substantive disagreement on everything
Keep an eye on validity: everything can be measured reliably, but is it valid?
Coding scheme: Instrument to input collect data during coding process
Online questionnaire: qualtrics
Mere purpose: offer coders an efficient way to code - per media message samples - all variables in the manner and order indicated in the codebook
All coder instructions belongs in the codebook only - keep at hand while completing the coding scheme
Validity of measurement:
Use the existing literature to inform before you design instrument
Consider all types of validity of measurement. Validity is priority
Take time when designing codebooks.
Explorative coding phase to refine code book.
Explicitly indicate the reg unit in your codebook and paper. Multiple? Group the variables per unit(level) in the codebook.
Reliability
What is reliability? The extent to which a measuring procedure yields the same results on repeated trials.
Consistency: among coders, between coders..etc.
Consistent over time.
Instrument must be consistent over time, place, coder and circumstance
Other codes - can replicate > same output because the codebook is so good. How replicable a study is.
Why brother with reliability?
Without knowing how reliable, content analysis measures are useless.
Outcome of study reflects beliefs of coders and random
Many studies fail to report on reliability or not transparent enough
Reliability is determined by
The quality of coding book
The particular judges (coders) making the ratings (train coders)
Give examples on every coding category
All starts with a clear operationalization:
What we mean by violence: explain explicitly (social validity: is the beheading of a snowman - violence? How is it perceived by people?)
All things must be plausible and credible
Explain as much as possible to easen the process
Troubleshooting coder differences
Due to codebook
Unclear or incomplete instructions
Multiple dimensions in a single variable
Unclear answer categories
Due to coders
Coders do not follow procedure (Insufficient training)
Context problems: context is unclear, different background knowledge
Fatigue after time
Solution for multidimensional
Start with operational definition that specifies the dimensions
Create or find indicators or instruction
Degree of social relevance, focus on private life etc
Break up every characteristic
