Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Content analysis > Chapter 6 > Flashcards

Chapter 6 Flashcards

1
Q

Reliability: Quality of data:

A

Reflects reliability of measurement process. Crucial for the validity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Core:

A

Measurement instruments (protocols) applied to observations must be consistent over time, place etc
No distortions

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Ca:

A

Operationalisation of content protocol, training of coders, and mathematical measures.
Reliability in ca: consistency among coders applying protocol to categorize content

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Defined variables in protocol=

A

Control assignment of number to content units by coders.
Variable and category don’t control assignment of content: human biases.
Replicability: crucial

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Reliability in ca

A

Defining variables and categories. Ends with reliability tests, numerically how well the concept definitions have controlled the assignment of content to analytic categories
Coder bias: alternate interpretation failure to achieve: Replication attempts of dubious value.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Conceptual and operational definitions -

A

Specify how concepts can be recognised in the content of interest
Variable: operationalised definition of the concept.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Concept complexity and number of variables:

A

Harder to achieve reliability when its more complex. Better be easy and straightforward
Easier when: concept is more manifest. observable on its face.
Must address somewhat latent. Two problems affect reliability: agreement more difficult when proportion of meaning in latent increases

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Variables difficult coder decisions:

A

Affects reliability (coder complexity or limited common meaning)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Content analysis protocol:

A

Purpose of the protocol: rules governing the study
Defines and measures the content of interest. Rules must stay the same through the study - archival record of studies operations and definitions.

Protocol; other researchers can interpret results and replicate the study. Replication strengthens the ability of science.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Protocol organisation: coherent matter

A

3-part approach works well for organisation:
1. intro specifying the goals and major concepts
2. specifies the procedures governing how the content was to be processed.
3. each variable used in the CA, weight of the protocol

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Coding sheet

A

Each variable needs to be related unambiguously to the actual coding sheet used to record content attributes
Should be coder friendly.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Paper:

A

Flexibility when coding, no computers and interruptions, good with large CA, takes more time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Coding sheets two types:

A

Single case and multiple case.
Single: one or more pages for each case or recording unit. Each variable and a response number with a letter or number
Multiple case coding: more than one case to a page. Along the rows and variables listed in the columns

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Coders

A

Coder change as they engage in successive encounters with content of interest and way its captured.. protocol goes through drafts during pretesting as variables are refined, measures specified and procedures worked though

Depends on the number of coders. Researchers carry mental baggage, one coder may not notice many coders more likely to hammer out operational definitions more clear

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Familiarise them with content analysed - increase comfort level,
minimise coder differences: procedure that coders follow in dealing with the content. Familiarise with with the protocol, dealing with problems

A

-

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Protocol problems

A

Differences because of inadequate category definitions: serious. Category ambiguous or poorly articulated?
Does the coder not understand? revising its definition to remove sources of ambiguity or confusion
Can they be broken down to easier parts? dropped from study?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Coder problems

A

Between-coder reliability: easy to identify problem coders by comparing agreement of all pairs.
Some coders needs to be educated differences in cultural understanding? More latent

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Cross-country CA

A

Coder trainers agree in their coding with one another, coders within the country agree with one another, coders agree with coding of their trainers

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Symbolic complexity

A

Can have impact on levels of reliability and achieving consistent coding. more latent: more difficult
Visual: more ambiguous than text. Difficulty: generated by research questions.

Improve: protocol well developed and coders have training.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Reliability assessment

A

Assess the degree to which protocol can be applied.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

3 types: stability, reproducibility, accuracy (for reliability)

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Stability:

A

Coder consistency to the same set of content at two points (within coder tests) important for longer periods of testing

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Reproductivity:

A

Two or more coders can apply same to content.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Accuracy:

A

Consistent with external standard for content

25
Q

Coder training

A

Reliability pretest. Formal coder reliability. Period of coder training to see when proceed with the study

26
Q

Actual assessment of reliability when coding process starts.

A

-

27
Q

Two issues:

A

Selection of content used in reliability assessment; actual statistical reliability tests used.

28
Q

Coding after protocol:

A

Inflate reliability because creators share biases.

29
Q

Selection of content for testing

A

Units small: two or more coders.
otherwise: coders randomly select samples for testing.

30
Q

Random is good for:

A

Controls for human biases; procedure produces a sample that reflects appropriate characteristics in overall population.

Without random: inference that reliability outcome represents all content cannot be supported.

31
Q

Probability sampling

A

Take advantage of sampling theory on how much materials tested. Random can specify sampling error at knows levels. Bigger sample: smaller error and more precise estimate of agreement. Requirement of agreement: 80%.

32
Q

Selection procedures:

A

Content for reliability selected randomly: how many units of content selected?

Several factors: total numbers of units to be coded, desired degree on confidence in reliability assessment, degree of precision desired in the assessment.

33
Q

Estimate of actual agreement

A

Five percentage higher than estimate of actual agreement, be set five points higher than the maximum required reliability for the test.

34
Q

Procedure

A

Compute number of content cases required for reliability test. Survey population: formula for standard error of proportion to estimate a minimal sample size given confidence level. CA population far smaller. Possible to correct for a finite population size the sample makes up 20% or more of the population effect of reducing standard error and more precise estimate of reliability

35
Q

Standard error

A

Can be manipulated to solve for the sample size to achieve a given level of confidence. Standard error gives confidence level desired in the test. 95% or 99% (using a one-tailed test) N: population size, P is estimate of agreement, Q is 1 minus the estimate.

36
Q

Krippendorf, table with reliability sample sizes that is a function of the researchers selection of acceptable minimum level for Krippendorff’s alpha, and acceptable p-value.

A

-

37
Q

All study content

A

Eliminates sampling error, must be randomly selected.
Regardless of sampling process: sample should be checked to verify that all categories for all variables have been selected at least once

38
Q

When to conduct reliability?

A

Two types of test: series of pretests during training. Reproducibility pretesting serves as part of the coding process, examining reality, adjusting protocol etc.
Until an acceptable level. Formal reliability pretests will determine when a point is reached.

39
Q

Content for reporting reliability should be coded by all coders, do not know which content units are being coded by everyone. Blind approach yield a better representation. More than a month: test stability of the process = randomly selected.

A

-

40
Q

Sufficient:

A

Does not have to be as large as in the initial test. 30 to 50 is sufficient.
Does not usually need more than two reliability tests. Third tests: 80%-90% then sufficient

41
Q

Initial test is high

A

Deterioration (the process of becoming progressively worse) will likely reflect problems with coders

42
Q

Degree of reliability applies to each variables:

A

Reported with a reliability coefficient - summary statistic for often coders using the protocol agreed on the classification of the content units
more than 30 reliability coefficients.

Four most often used. Scott’s pi, Cohen’s kappa, and Krippendorff’s alpha

43
Q

Percentage of agreement:

A

Agreement of two or more coders first to evaluate reliability.
each possible pair of coders - agreement or disagreement.

3 coders -dichotomous: four coders yield six decisions.

Percentage of agreement: coefficient overestimates reliability because it does not control for the influence of agreement due to accident or error.

Agreements: most results of well developed and good training

44
Q

Percentages of agreement can inflate: however: protocol development and coder training identify where and why disagreements are occurring.

A

-

45
Q

Corrects change agreements:

A

Scott’s pi, two coders and with nominal data. correcting: calculation of expected agreement using probability theory.

Scott’s pi: computes expected agreement by using proportion of times particular values of a category used in given test

46
Q

OA: observed agreement (agreement in reliability test)
EA: expected agreement (expected by chance)

A
47
Q

Scott; squared the observed proportions used each value of category assuming all coders using those values.

Kappa: expected agreement based on proportion of a particular value of a category used by one coder multiplied by the proportion for that value used by the other coder. Kappa sometimes higher reliability

A

Kappa: nominal

Krippendorff alpha: depends on level of measurement. can be used with non-nominal and multiple coders - corrects for small samples.

48
Q

Reliability coefficient can be adequate measure of reliability under three conditions:

A
  • Two or more coders independently
  • Coefficient treats coders as interchangeable
  • Reliability coefficient must control for agreement for chance.
49
Q

Pearson’s product moment correlation:

A

Check for accuracy of measurement with interval and ratio level data. Degree to which two variables or coders vary together
Coders are the variables, recording units are the cases
(Warning to use correlations for reliability association is not same as agreement)

50
Q

Other forms of alpha

A

Variables that are predefined (labeled alpha) used to calculate reliability coefficient when units have not been defined= conversation among people

Krippendorff: MV alpha: calculating reliability for variables that are applied to units with multi-values.

51
Q

Debate:

A

Calculating accepted agreement using the concept of chance agreement: assumptions underlying the

Calculation of coefficients. pi, kappa, alpha criticised; produce low coefficients despite high agreements and correct for max level of chance agreement

52
Q

Ovbservations: replication

A

Use original protocol. More about this and training than disagreement
generate valid data - adequate rel. - help to apply valid.
Reach the highest level of validity

53
Q

Chance agreement

A

Very few occur by chance - most likely educated guesses

54
Q

Two-value measure (pi):

A

Frequent occurrence of one value and infrequent occurrence of other creates imbalance: overcorrected in chance agreement.

55
Q

New coefficient:

A

AC: agree and dis in two groups on four conditions
1: both coders assign values based on application of protocol,
2: assign based on random,
3,4: one coder random and other on correct application.
AC: better estimate of expected agreement.

56
Q

Coding difficulty

A

Determined chance agreement, and not distribution of agreements and categories

Alpha1: calculate chance agreement based on disagreements; highest agreement with this

57
Q

CA:

A

Describe content without examining relations.

58
Q

Selecting a reliability coefficient

A
  • Report RC that corrects for chance agreements when calculating expected agreement for the variables. Replication.
  • Census or probability sample to calculate RC and report sampling error for reliability coefficients
  • Not skewed: report kripp alpa and level of agreement
    .8 is reliable. also .667 can be acceptable.
  • Established research: reach .8. new areas. .7,.8 is okey.
  • High level of simple agreement, but reliability do not reach accepted: report GWET AC2 as coefficient.
  • Uncomfortable relying on alpha C; report alpha c with other RC, and explain. Also reach acceptance
  • Adequate sample size randomly
  • Variable reliability across time: improved measurement and consistent reliability
59
Q

Summary:

A

Reporting reliability is necessity:
- Full info about CA available for others
- Also on reliability with protocol definitions and procedures, training of judges, content items tested and selected.
- Specific coder reliability tests applied and achieved numeric reliability, confidence intervals included.
- Collect and share scholars and improves applying reli tests: randomly select a number of units for tests and decide if acceptable reliability levels (based on coefficients that error agreements into consideration and report simple agreement in endnote for replication)
- Failure: invalidates usefulness.

Content analysis (11 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions