Lecture 2

Question 1

Validity of measurement

Answer 1

Research question of characteristic of media content
Each character requires valid measurement
Measurement validity: substantive accuracy of a measure - degree to which a measure covers a concepts content (reality, meaning) correctly and fully
Accurate measurement is harder for more latent characteristics

Question 2

Manifest content

Answer 2

Directly observable (concrete) on its surface
Unambiguous meaning
Simple operationalisation
Single-item measure

Question 3

Latent content

Answer 3

Not directly observable (abstract) below the surface must be revealed
Deeper ambiguous meaning
Multiple operationalisation - indicators
Measurement instrument with multiple items

Question 4

The more latent a characteristic of media content:

Answer 4

The more meaningful and influential the context
The more meaning is judged constructed instead of observed by the coder
The more its meaning becomes dependent in pre existing mental schemes (frames) of the coder - projection
The more the focus of meaning making shifts from content to coder
More subjective (potential problem)

Question 5

Manifest

Answer 5

AI or specific term (gender of actor) // middle is frames // latent: beauty

Question 6

Intersubjectivity

Answer 6

Strong social norm

Question 7

Operationalisation of latent media content (constructs)

Answer 7

/ Translation into a set of more manifest characteristics of content (indicators)
/ To break up -> to objectify quantitative
/ Indicators items combined must be covered the construct fully

Question 8

Natural starting point:

Answer 8

Existing literature
Theoretical insights and empirical evidence
THeory must lead if it does not exist

Question 9

Empirical ways to evaluate validity of measurement

Answer 9

(Instrument - validation)

Question 10

Factor analysis

Answer 10

Construct validity (dichotomous in CA often - negative: they are not good for factor analysis)

Question 11

Association between measurement instrument and:

Answer 11

Other version of same type of measurement (concurrent validity)

Plausible outcome variable (predictive validity)
Comparison outcomes with those of very different instruments (convergent validity)
(cross check among producers or consumers of content)

Question 12

Social validity:

Answer 12

The social significance and meaning of the analysed media content beyond the academic community

/ among intended receivers of the content

/ combination of convergent and ecological validity: are researchers and users on the same substantive frequency?

Understand differently / audience understand in different ways than researchers

Question 13

Should researchers determine the standard for meaning of content? Theoretical question

Answer 13

Is the chosen standard representative for that of the receivers of the content? Empirical question

Wider community beyond academic society?

Question 14

Validity vs reliability of measurement

Answer 14

Reliability is a criterion for evaluating the quality of measurement

Agreement about categorising media content (among coders)

Reliability is a necessary but not sufficient condition for validity (reliability is a precondition for validity - no reliability > no validity)

Complication: In CA> both criteria of measurement quality are in tension with one another / raising one level might come on expense on of the other

Question 15

Reliability» provide more instructions (1)

Answer 15

More coding rules
More detailed coding rules
Do more coder training
Creator mental focus on the rules

Consequence> coders will approach media content less naturally - like normal users
The social validity (ecological validity) of the measurement is impaired

Question 16

(2) Simplify coding task:

Answer 16

Stronger focus on easiest observable aspects of latent

Remove difficult coding decisions

Reduce number of coding options / negative or positive

Consequence: The content reality of measured concept is too reduced
Content validity of measurement is impaired

Question 17

Be realistic: a perfectly reliable and valid measurement instrument for latent media is impossible

Don’t fixate on just one criterion:
It’s self-defeating: a measurement must always satisfy both criteria: keep an eye on both

Question 18

Content analysis still tends to be less attentive to validity
Don’t underestimate validity - matters more than reliability. Look for balance!

Question 19

Coding process

Answer 19

Codebook: document containing all information relevant to the process of coding media content:

All variables to be coded (coding categories) for the study - full measurement instrument

Instruction to coders about when and how to code variables

Including coding rule and procedures that need to be followed systematically (quantitative)

Question 20

Registration unit (observation unit):

Answer 20

Unit you judge when you code a variable - what is the unit level you make a decision about when you code a variable?
Reg unit can be: Textual media content: entire publication, web page, article, paragraph, sentence, word etc

Question 21

Audiovisual media content (reg unit):

Answer 21

Entire production - movie, act, sequence, scene, shot etc

Question 22

Reg unit

Answer 22

The choice of reg unit depends on the required level of detail for the study: more detail = more coding work!

Researchers can measure variables at different levels of registration within the same study (article level, or lower level)

Question 23

Context unit:

Answer 23

Part of the content you take into consideration when you code a variable for each registration unit

Distinguished from reg unit only if the broader physical or temporal context of the media content you judge might influence the interpretation of users of that content > validity.

Mainly an issue with small reg units. Smaller the units, greater the importance of context

For large reg units: news article, often non-issue: reg and content coincide. (no broader context to take into consideration)

Question 24

Does the photograph depict a human expression of sadness? Is it actually that? Broader context changed our interpretation on how to code the reg unit.
Reg unit should also include a caption.

Question 25

Codebook

Answer 25

The codebook contains all variables to be coded (in the most logical coding order)
Variables needed to answer the research question
Optional: control variables (internal validity) Rule out alternative explanations
Mandatory: administrative variables (highly important)
Other variables than these: waste of time and effort!

Question 26

Administrative variables

Answer 26

Registration unit ID variable

Unique identification number per coded reg unit (article ID)

Coder ID
ID of number by coder

Coder comments variable (string variable)
Text input to record doubts or problems regarding coding

Critical for dinding, correcting etc
BOTH ID necessary

Question 27

Codebook

Answer 27

General and specific coding instructions that are as clear and comprehensive as possible:

Definitions of reg and context units, measured concepts and coding categories

Rules and procedures: what to decide in which coding situation? Systematic

Examples of coding situations and media content to illustrate rules and procedures
Anticipate a range of coding scenarios - explorative coding phase helps

Question 28

Goal:

Answer 28

Switch off or standardise individual frames of reference of coders: objectivity

Assume prior substantive disagreement on everything

Keep an eye on validity: everything can be measured reliably, but is it valid?

Question 29

Coding scheme: Instrument to input collect data during coding process
Online questionnaire: qualtrics

Answer 29

Mere purpose: offer coders an efficient way to code - per media message samples - all variables in the manner and order indicated in the codebook

All coder instructions belongs in the codebook only - keep at hand while completing the coding scheme

Question 30

Validity of measurement:

Answer 30

Use the existing literature to inform before you design instrument

Consider all types of validity of measurement. Validity is priority

Take time when designing codebooks.

Explorative coding phase to refine code book.

Explicitly indicate the reg unit in your codebook and paper. Multiple? Group the variables per unit(level) in the codebook.

Question 31

Reliability

Answer 31

What is reliability? The extent to which a measuring procedure yields the same results on repeated trials.

Consistency: among coders, between coders..etc.

Consistent over time.
Instrument must be consistent over time, place, coder and circumstance

Question 32

Other codes - can replicate > same output because the codebook is so good. How replicable a study is.

Question 33

Why brother with reliability?

Answer 33

Without knowing how reliable, content analysis measures are useless.
Outcome of study reflects beliefs of coders and random
Many studies fail to report on reliability or not transparent enough

Question 34

Reliability is determined by

Answer 34

The quality of coding book

The particular judges (coders) making the ratings (train coders)

Give examples on every coding category

Question 35

All starts with a clear operationalization:

Answer 35

What we mean by violence: explain explicitly (social validity: is the beheading of a snowman - violence? How is it perceived by people?)

Question 36

All things must be plausible and credible

Answer 36

Explain as much as possible to easen the process

Question 37

Troubleshooting coder differences

Answer 37

Due to codebook

Unclear or incomplete instructions
Multiple dimensions in a single variable
Unclear answer categories

Question 38

Due to coders

Coders do not follow procedure (Insufficient training)
Context problems: context is unclear, different background knowledge
Fatigue after time

Question 39

Solution for multidimensional

Answer 39

Start with operational definition that specifies the dimensions

Create or find indicators or instruction

Degree of social relevance, focus on private life etc

Break up every characteristic