Empirical methods exam

Question 1

Induction and Deduction

Answer 1

Data collection in the field of empiricism
▪ From these data researchers extract general
statements (theory) by induction.
▪ From theory, in turn, statements about single
cases can be derived by using deduction.

Question 2

Finding a Research Question - Inductive

Answer 2

Question 3

Finding a Research Question - Deductive + I & D reasoning

Answer 3

Question 4

Inductive reasoning

Answer 4

Specific observation -> Pattern recognition -> General conclusion

Question 5

What is Research Design?

Answer 5

Question 6

Research Criteria

Answer 6

Question 7

Correlational Research

Answer 7

The goal of correlational research is to determine whether two or
more variables are related.

Question 8

Experimental Design

Answer 8

Experimental research involves comparing two groups on one outcome
measure to test some hypothesis regarding causation

Question 9

Types of experimental Research Designs

Answer 9

Question 10

Cross sectional

Answer 10

Question 11

Descriptive Statistics: Graphic representation of data

Answer 11

Descriptive statistics is the idea of quantitatively describing data and you can do that through various means. For example, through visualization techniques like: * graphical representation * tabular representation * summary statistics The idea here is that you crunch the data, you work with the data and come up with (1 or 2 or 3 or 4) different numbers that summarized the data for you.

Question 12

Bar graph

Answer 12

There are dozens of charts and graphs you can make from data. * Which one you choose depends on what kind of data you have and what you want to display. * If you wanted to display relationships between data in categories, you could make a bar graph

Question 13

Histograms

Answer 13

Question 14

Pie chart

Answer 14

shows how categories in your data relate to the whole set

Question 15

Scatter plots

Answer 15

Scatter plots are a good way to display data points. * It shows the relationship between two variables * The position of each dot on the horizontal and vertical axis indicates values for an individual data point * The dots in a scatter plot report patterns when the data are taken as a whole.

Question 16

Why are descriptive statistics important?

Answer 16

Question 17

Basic Concepts of Descriptive Statistics

Answer 17

Question 18

Measures of central tendency

Answer 18

A measure of central tendency (also referred to as measures of centre or central location): * Is a summary measure that attempts to describe a whole set of data with a single value that represents the middle or centre of its distribution. There are three main measures of central tendency: −the mode, the median and the mean. Each of these measures describes a different indication of the typical or central value in the distribution.

Question 19

Mean

Answer 19

Question 20

Median

Answer 20

Question 21

Mode

Answer 21

Question 22

Shapes of Distributions - Peaks

Answer 22

Graphs often display peaks, or local maximums. It can be seen from the graph that the data count is visibly higher in certain sections of the graph. 1. one clear peak is called a unimodal distribution. 2. two clear peaks are called a bimodal distribution. Here, the term “mode” is used to describe a local maximum in a chart (such as the midpoint of a peak interval in a histogram). It does not necessarily refer to the most frequently appearing score, as in the “central tendency mode”. * Single peak at the center is called bell shaped distribution. Note: A bell shaped graph (bell curve), is a frequency distribution that resembles the outline of a bell when plotted on a graph

Question 23

What is a Distribution?

Answer 23

The statistical distribution shows which values are common and uncommon in your data * A statistical distribution, or probability distribution, describes how values are distributed for a field * There are many kinds of statistical distributions, including the bell-shaped normal distribution. * We use a statistical distribution to determine how likely a particular value is.

Question 24

Shapes of Distributions

Answer 24

When graphed, the data in a set is arranged to show how the points are distributed throughout the set. * These distributions show the spread (dispersion, variability, scatter) of the data. * The spread may be stretched (covering a wider range) or squeezed (covering a narrower range). * The shape of a distribution is described by its number of peaks and by its possession of symmetry, its tendency to skew, or its uniformity. * Distributions that are skewed have more points plotted on one side of the graph than on the other.

Question 25

Shapes of Distribution - Skewness

Answer 25

Skewness is a measure of a lack of symmetry in a distribution. * A standard normal distribution is perfectly symmetrical and has zero skew

Question 26

Measure of Variability

Answer 26

Variability describes how far apart data points lie from each other and from the center of a distribution. Range: the difference between the highest and lowest values. Interquartile range: the range of the middle half of a distribution. Standard deviation: average distance from the mean. Variance: average of squared distances from the mean

Question 27

Measure of Variability: Range

Answer 27

* The range is the difference between the highest and lowest values within a set of numbers.
* To calculate range, subtract the smallest number from the largest number in the set. * Range shows how much the numbers in a set vary

Question 28

Measure of Variability: Interquartile Range

Answer 28

The middle fifty or midspread of a set of numbers, removes the outliers (highest and lowest numbers in a set) * If there is a large set of numbers, divide them evenly into lower and higher numbers. * Then find the median of each of these groups. * Find the interquartile range by subtracting the lower median from the higher median.

Question 29

Measure of Variability: Variance

Answer 29

Variance measures how far a data set is spread out. It is mathematically defined as the average of the squared differences from the mean. * Finding the mean (the average). * Subtracting the mean from each number in the data set and then squaring the result. * The results are squared to make the negatives positive. Otherwise, negative numbers would cancel out the positives in the next step. It’s the distance from the mean that’s important, not positive or negative numbers. * Averaging the squared differences.

Question 30

Measure of Variability: Dispersion

Answer 30

Measures of dispersion help to describe the variability in data. * Dispersion is a statistical term that can be used to describe the extent to which data is scattered. * Thus, measures of dispersion are certain types of measures that are used to quantify the dispersion of data. * The variance helps to draw a comparison between the two data sets A and B on the basis of variability.

Question 31

Types and Examples of Quantitative Research Questions

Answer 31

Descriptive
* Used to collect participants’ opinions about the variable that you want to quantify
* Descriptive research questions begin with “ how much?” “how often?” “what percentage?” “what proportion?”
* Example: How often do middle-class adults go on vacation yearly? (variable: vacation; group: middleclass adults)

Comparative
* Help you identify the difference between two or more groups based on one or more variables Example: What is the difference in the usage of TikTok between male and female Cambodian university students? (variable: usage of TikTok; group 1: male Cambodian university students; group 2: female Cambodian university students)

Relationship-based: * Used to identify trends, causal relationships, or associations between two or more variables.
Example: What is the relationship between salary and shopping habits among the women of Australia? (independent variable: salary; dependent variable: shopping habits; group: Australians)

Question 32

Survey Questions - Types

Answer 32

Survey questions should be designed in a way that
any respondent can understand them.
Survey questions can be classified into 4 subtypes:
− 1. Fact-based questions
− 2. Knowledge-based questions
− 3. Attitudinal and opinion questions
− 4. Behavioural questions

Question 33

Checklist Survey Questions

Answer 33

Target: to formulate every single question in such a way that no biased responses, misunderstandings or ambiguities can occur.
✓ Is the question clear for everybody?
✓ Is the issue that the question is based on precisely and comprehensively formulated?
✓ Can everyone clearly understand (and then describe) the issue in the question?
✓ Do the questions impose a overly high level of verbal skills?
✓ Is the question one-dimensionally formulated, is it clear what the respondent needs to answer?
✓ Is it possible that the question might overload the memory of the respondent?

✓Is the respondent still able to answer the question even if he is already exhausted?
✓ Does the question induce social desirability by the way it is formulated?
✓ Does the question contain suggestive wording that could lead to a certain response behaviour?
✓ Might the question be too intimate?
✓ Would it be possible for respondents to feel embarrassed by the response (and therefore give an untruthful answer)?

Question 34

Function Questions

Answer 34

Function questions control the course of the questionnaire without bringing any contribution to the actual result interest. These questions guarantee that the survey questions are applied correctly. 1. Ice-breaker questions 2. Transfer and Resting questions 3. Filter and Funnel questions 4. Verification questions

Question 35

Ice-Breaker Questions

Answer 35

Ice-Breaker Questions * Are used to establish a relationship with the respondent. * In order to make the interview atmosphere more relaxing, the interviewer asks in the beginning a question which is unimportant in comparison to all other questions and will not be analysed. * Example: “What is your opinion about the current TV programme?“

Question 36

Transfer and Resting Questions

Answer 36

Transfer and Resting Questions * Are meant to distinguish subject areas in a survey in a clear manner for the survey participant without focusing on the relevance of the content. * They allow the respondent to talk freely.

Question 37

Filter and Funnel Questions

Answer 37

• Make sure that only those people who are truly concerned answer the survey questions.
• Funnel questions sort out all the respondents who cannot or should not give an answer to the actual questions (e.g. Pay-TV users).
• Filter questions work by the same logic which differentiates uncoupling and bifurcation.
• By uncoupling, unnecessary questions are skipped.
• Bifurcations are integrated to receive information about all respondents divided into subgroups.

Question 38

Verification & Credibility Questions

Answer 38

Verification Questions Are integrated into the questionnaire together with alternating questions.

Aim: to verify the consistency of response behavior

Credibility Questions
▪ Are meant to verify the credibility of the participants.
▪ The questions refer to issues which everybody has allegedly experienced in his life.
▪ If the respondent gives an answer which is highly unlikely, then this is a proof that the respondent might sometimes give untruthful responses.

Question 39

Socio-demo-graphic Questions

Answer 39

Sociodemographic questions are not content-related and are usually placed at the end of the survey. The following aspects are usually asked for: * Age * Gender * Education * Head of household * Net income * Religion * Marital status

Question 40

Standardization of Surveys

Answer 40

In-depth/Guideline Interview
Develop a guideline and define specific question sequence. Semi-structured or semi-standardized interview.

Special type of survey: group interview Conduct group interview according to a set of guidelines. In group discussion, opinions are encouraged and generated. Participants mutually influence their response behaviour. Example: Introducing new products etc. Special attention is needed with interpretation. Therefore, usually it is used as a complementary instrument.

Question 41

Bias

Answer 41

Bias can occur when respondents want to be liked and to avoid embarrassment are very strong. This can affect how people answer questions asked by strangers. * Question wording must facilitate unambiguous, fully accurate communication * Leading questions are the most obvious culprit

Question 42

Counter potential social desirability bias

Answer 42

Use normalizing statements:
* Example: Some people like to follow politics closely and others aren’t as interested in politics. How closely do you like to follow politics? * Closed-ended questions (questions that give answers for respondents to select from) are susceptible to response set bias.

Question 43

Open-ended questions

Answer 43

Can give respondents freedom to answer how they choose * Remove any potential for response set bias * Allow for rich, in-depth responses if a respondent is motivated enough. * Challenge: respondents can be ambiguous; can give responses that indicate the question was misunderstood * Researcher is giving power to respondents to structure the data

Question 44

Closed-ended questions: characteristics

Answer 44

• Answers are unambiguous * Data are easy to manage. * Challenge: researcher is structuring the data, which keeps things nice and tidy

Question 45

Survey Mode

Answer 45

All surveys are conducted in one of the three survey modes: * Face-to-Face Interview * Written interview * Telephone Interview / Survey. Additionally, there is a version of written interviews that has established itself in recent years: the online survey.

Question 46

Face-to-Face-Interviews

Answer 46

Face-to-Face-Interviews:
* Personal interviews require qualified interviewers.
* Result high costs relatively high time consumption Pro * Fully used sample has very low refusal and abandon rate
* Return rate (40-70 percent) is considerably higher in comparison to written, telephone and especially online surveys.

Question 47

Telephone Interviews

Answer 47

Telephone Interviews:
Due to technical development, telephone interviews are becoming more and more popular and are extensively used as a current survey mode in professional survey research institutes. Disadvantage: * Visual auxiliary material cannot be used * Risk of overloading (seven-level scale) * Abandon rate is higher than in face-to-face interviews

Question 48

Written Survey

Answer 48

Written Survey (Paper & Pencil) Disadvantages of surveys by conventional mail: * low return rate (partly under 10 %) * motivation of the respondents is usually low, the social contact is missing * field phase is considerably longer than in other survey modes Higher rates when: * the respondents have an additional benefit (winning competition) * the questionnaires are notified by telephone * there is a follow-up campaign. Groups of people who are hardly reachable can be approached more easily via written surveys. (e.g. politicians)

Question 49

Online Survey

Answer 49

Online Surveys are Internet-based survey methods, which can be classified into three groups: The questionnaire … * is saved on a server and is filled out online. * can be downloaded from a server and sent back via E-Mail * is sent via E-Mail and returned the same way after filling it out. Option 1 with the possibility to directly fill out the questionnaire online enjoys the highest popularity at the moment. There are different software on the market to help create and use online survey tools. (e.g. Lime Survey, SurveyMonkey).

Question 50

Survey Development: definition

Answer 50

Survey development is a collaborative and iterative process where researchers meet to discuss drafts of the questionnaire several times over the course of its development.
* Conducting frequent tests with new survey questions ahead of time through qualitative research methods such as focus groups, cognitive interviews, pretesting
* Choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent. It is important that all respondents interpret the question the same way.

Question 51

Question Order

Answer 51

We must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions * Two main types of order effects for closed-ended opinion questions: * Contrast effects (where the order results in greater differences in responses) * Assimilation effects (where responses are more similar as a result of their order). How to avoid: The best way to combat these errors is randomization. You can randomize your answer options for every respondent so that e ach option has a fair chance of being picked. As for the question order bias, make sure that your questions don’t affe ct the answers to the following questions.

Example: You ask your employees about their issues with the reporting manager and then ask if they are happy in the workplace. Now, there is a possibility that they would have answered the questions differently if the order had been reversed. Their second answer will be influenced by the answer to the first question.

Question 52

Closed-ended Questions

Answer 52

Is made up of pre-populated answer choices for the respondent to choose from * Multiple choice * Drop down * Checkbox * Ranking question * Questions that are closed-ended are conclusive in nature as they are designed to create data that is easily quantifiable * Closed-ended questions allows researchers to categorize respondents into groups based on the options they have selected (allows for demographic studies) * Major drawback to closed-ended questions is that a researcher must already have a clear understanding of the topic of their questions and how they tie into the overall research problem before they are created.

Question 53

Likert Scale Questions

Answer 53

Always a close-ended question * A survey scale represents a set of answer options—either numeric or verbal—that cover a range of opinions on a topic. * Will let us uncover degrees of opinion that could make a real difference in understanding the feedback you are getting * Are structured to provide quantifiable answer options that make analyzing data easier * Each series of questions in the survey focused around the same topic

Question 54

What is a Likert Scale?

Answer 54

The Likert scale is a five (or seven) point scale that is used to allow an individual to express how much they agree or disagree with a particular statement. Simple questions: the less thinking required from respondents the better the response rate. Example: How satisfied were you with your in-store experience? It was easy to navigate the website to find what I was looking for.

Question 55

Writing a Likert Scale Question

Answer 55

Accuracy: likert-type questions must be phrased correctly in order to avoid confusion and increase their effectiveness * Descriptive words need to be easily understood and should not be ambiguous * There should be no confusion about which grade is higher or bigger than the next * Better ask a question than let people rate a statement

Question 56

Single answer (closed): example

Answer 56

What is your marital status? Are you … * married and living with your spouse * married and separated * widowed * divorced * unmarried

Question 57

Single answer (semi-open)

Answer 57

What is your marital status? Are you … ▪married and living with your spouse ▪married and separated ▪widowed ▪divorced ▪unmarried ▪I have a different marital status, which is: _________________

Question 58

Multiple answer (complex)

Answer 58

Which of the devices on the list are in your household? (multiple answers possible)
▪washing machine
▪dishwasher
▪food processor
▪espresso machine
▪none of these

Question 59

Multiple answer (closed)

Answer 59

Which of the devices on the list are in your household? (multiple answers possible) ▪washing machine ▪dishwasher ▪food processor ▪espresso machine ▪vacuum cleaner

Question 60

Ordering scale: example

Answer 60

Please put these 7 properties in the order that best describes your diet. please indicate the order by assigning numbers from 1 = most important to 7 = least important __ must eat meat __ must not eat meat __ give me sweets __ comes out of a wrapping __ circumnavigates food intolerances __ any hipster label here __ super duper healthy

Question 61

Multiple response question

Answer 61

Of these 7 properties, which best describe your own diet? Chose 3 at most. (max. 3 items) ▪must eat meat ▪must not eat meat ▪give me sweets ▪comes out of a wrapping ▪circumnavigates food intolerances ▪any hipster label here ▪super duper healthy

Question 62

Pitfalls When Designing a Survey

Answer 62

• Biased or leading questions: Phrasing questions in a way that suggests a particular answer can skew results. Ensure questions are neutral and don’t lead respondents to a specific response.
• Ambiguous questions: Vague or unclear questions can confuse respondents and lead to inaccurate responses. Questions should be precise and easy to understand.
• Overly long or complex surveys: Lengthy surveys can lead to respondent fatigue and lower completion rates. Keep surveys concise and focused on relevant topics.
• Limited response options: Providing insufficient response options can limit the range of answers and fail to capture diverse perspectives. Include a variety of response options, including open-ended questions when appropriate.
• Order bias: The order in which questions are presented can influence responses. Be mindful of question sequencing and consider randomizing or alternating question order to mitigate bias.
• Social desirability bias: Respondents may provide answers that they perceive as socially acceptable rather than their true opinions or behaviors. Use anonymous surveys and assure respondents of confidentiality to minimize this bias.
• Sampling bias: If the survey sample is not representative of the target population, results may not be generalizable. Ensure the survey sample accurately reflects the demographics and characteristics of the population of interest.
• Non-response bias: If certain groups are more likely to participate in the survey than others, results may be skewed. Employ strategies to encourage participation from underrepresented groups.

Question 63

Process of Quantitative Research

Answer 63

1. Theory
2. Hypothesis
3. Research design
4. Devise measures of concepts
5. Select research sites (s)
6. Select research subjects/respondents
7. Administer research instruments/collect data
8. Process data
9. Analyse data
10. Findings/ conclusion
11. Write up findings/ conclusion

Question 64

Measuring and Counting

Answer 64

• Empirical social research is interested in conducting a systematic and intersubjectively traceable analysis of cases attributed to a phenomenon (our research question). * Quantifying any type of variable through a systemic method Definition: − Measuring is the systematic assignment of an amount of numbers or symbols to the values of a variable, hence also to the objects.

Question 65

Measuring and Counting Classification

Answer 65

Classification occurs by looking at the value of the variables. Nominal (categorical, qualitative) data * Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. (e.g. gender) Ordinal data (rank variable) * Variables that have two or more values which can also be ordered or ranked. That means that variable values have a natural sequence. (e.g. school grades). Metric data * Values that can be organized according to size AND can represent a multiple of a unit (e.g. height)

Question 66

Levels of measurement or scales of measurement –
classification principle of variables I

Answer 66

Levels of measurement or scales of measurement – classification principle of variables I − Nominal data / variables (e.g. hair colour/nationality) − Ordinal data (variables have natural, ordered categories: e.g. hot, hotter, hottest) − Metric data (e.g. age)

Question 67

Classification principle of variables

Answer 67

− Discrete variables - counting numbers with
specific values (e.g. age)
− Continuous variables - measurable number e.g.
1.7 feet

Question 68

Empirical and Numerical Relative

Answer 68

Эмпирические и численные данные

• Measuring is the process by which the empirical relative, e.g. the characteristic value “person in dress” is systematically assigned a numerical relative, e.g. the number 1. * The numerical relatives contain numerical values or scores, the empirical relatives define objects or objects of analysis * Each element of the empirical variables is assigned a numerical value in such a way that the relations in the empirical relatives correspond or equal the relations in the numerical relatives. * This rule of assigning values is also called scale.

Question 69

Terms

Answer 69

Object of
analysis
Attribute/
Variable
Value

Question 70

Coding Characteristics

Answer 70

Question 71

Scale Level: types Classification Principle of Variables

Answer 71

Scales: The one-to-one transfer of empirical relatives into numerical relatives leads to the formation of scales. We distinguish between three scales of measurement. The classification is done according to the value of the variables: * Nominal variables/ data/divide into class/numbers * Ordinal variables/ data/Scal in order/ 1st, 2nd, 3rd/Low, medium, moderate * Metric variables/ data/exam score/grading system 1-4

Question 72

Nominal Data - Qualitative Variables

Answer 72

Nominal Data: Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. ▪ The categories must be defined precisely. ▪ The categories must mutually exclude themselves. ▪ The categories must describe the variable exhaustively.

Question 73

Ordinal Data

Answer 73

▪ Variables that can take two or more values which can be ordered or ranked in a sequence.
▪ That means that variable values have a natural sequence.
▪ The measured values on an ordinal scale can be put into a logical sequence.

Question 74

Metric Data

Answer 74

▪ Variable values that can be organized according to size AND can represent a multiple of a unit. ▪ Values that have been measured on a metrical scale can take any numerical value (or score). ▪ The range or the intervals between the possible measured values have to be always the same. E.g age

Question 75

Nominal Data dichotomous variables

Answer 75

▪ Dichotomous (dividing into two parts) variables are variables which have
exactly two values, e.g yes/no
▪ Example: Do you have a mobile phone?

Question 76

Nominal Data - polytomous variables

Answer 76

▪ Polytomous variables display more than two values. E.g divide in class, brands of mobile phone, bus, tv ▪ Can be used to measure traits and abilities with items that have unordered response categories. ▪ It can identify the empirical ordering of response categories where that ordering is unknown but of interest ▪ Can be used to check whether the expected ordering of response categories is supported in data.

Question 77

Classification Principle of Variables

Answer 77

Discrete variables * A discrete variable is one that has specific values and cannot have values between these specific values. (e.g. school grades, gender, number of mistakes) Continuous variables * There is in theory an infinite number of values between any two values that a continuous variable can take. (e.g. size of screws, height, age)

Question 78

Index: Questionnaire & Survey difference

Answer 78

An index is formed by summing up several single indicators. Example: Press Freedom Index by Reporters Without Borders. − The annual ranking list assesses the situation of press freedom in almost 180 countries worldwide. Basis/Foundation for the ranking: An extensive questionnaire Different indicators (pressure, corruption), moderators, impact factors

A questionnaire is the term used to describe
the set of questions you are asking an
individual.
A survey is the process of collecting,
analysing and interpreting data from many
individuals. It aims to determine insights
about a group of people.

Question 79

Example - Press Freedom Index

Answer 79

Based on the questionnaire, each country is assigned a score for the following categories: − Plurality of opinion − Independent media − Journalistic work environment and self-censorship − Legal framework − Institutional transparency − Production infrastructure Do we have categories for our GMF research? In addition to these, there comes a more important factor – violent assaults upon journalists. Total score between 0 = total press freedom and 100 = no press freedom

Question 80

Indices characteristics

Answer 80

Indices represent an area of variables, i.e. in an index there are already several items summed up mathematically: Indices reduce the area of variables. The specific description of individual indicators is lost. * The index is the sum of all measured variables and hence it is itself a new variable. * The index needs to cover all aspects of the variables being studied, and it should do so in a straightforward way, without getting too complicated or including unnecessary elements. * The index should be able to measure everything important about the topic in a clear and simple manner. * The measurements that enter an index must all have the same measuring level, e.g. nominal, dichotomous scales.

Question 81

Difference between Index and Scale

Answer 81

• An index is like a summary score that combines lots of different things into one number, giving you a big picture. * For example: an index of happiness might include factors like income, health, and social connections, all rolled into one score.
• A scale is like a ruler with different points to measure one specific thing. * It is more detailed and focused. * A scale measuring happiness might ask you to rate how happy you feel on a scale from 1 to 10, giving you a precise measure of your happiness level

Question 82

Scales: terms difference

Answer 82

Attention: Different usage of the term “scale“.

• Scale level: Diversity of measurements
• Scale: Summation of several individual measurements into one total value.
• In this case it is used similarly to the term index. The difference might be more about the specific terminology used rather than a fundamental conceptual distinction

Question 83

Types of Scales

Answer 83

▪ Likert scale:
▪ Consists of several at least 5-stage items (interval scaled), which are combined to an index by summation.
▪ Semantic differential:
▪ Consists normally of about 10 to 20 opposing pairs.

Typical usage of a Likert scale with the aim to empirically determine the meaning of a concept. Multi-point rating option

Question 84

Why do we conduct a Pre-test?

Answer 84

Catch any problems with it before you conduct it Problems could be:
* questions that don’t make sense
* technical glitches
* potential sources of bias
➢ Pre-test are worth their time: better data quality, less drop-outs and less effort during the analysis and interpretation.
➢ Even under time pressure a pre-test should not be missed out on.

Question 85

Pre-Test

Answer 85

• Make revisions based on information from pre-testing * Edit questions and response categories if needed * Simplify language/ be more explicit if needed * In case of major revisions consult with survey development experts and conduct a retesting of the survey

Question 86

Pre-test Focus

Answer 86

Comprehension
* Do respondents understand the aim of the survey and the wording of the introduction and questions? Logic and flow
* Do items in the questionnaire follow a logical order?
* Is anything out of place or confusing?
* Do question order set up conditions for bias?

Acceptability
* Do questions cause offense or touch on sensitive subjects in an inappropriate way?

Length and adherence
* Does it take a lot of effort to fill in the survey?
* Can respondents make it through to the end without losing interest and focus?

Technical quality
* Does the survey platform operates smoothly? Introduction and gaining consent
* Do we have an introduction declaring aim and scope of the research clearly? Can respondents give informed consent?

Question 87

How to Pre-Test?

Answer 87

• Find 5-10 people from your target group to pre-test your survey * Try to get a range of different people who are representative of your target group * Ask them to complete the survey while thinking out loud (not down if they understand or do not understand a question) * The person should comment on anything that crosses their mind (method of loud thinking);
• Let them tell you if something is unclear or ambiguous. * Take notes of every comment made – do not comment or justify * If you are uncertain whether a question was understood correctly, you can ask your tester directly what they understood. * In case your testers were satisfied with everything at first contact, then you haven’t sufficiently asked them for criticism. * Make improvements based on the results

Question 88

Piloting a Survey

Answer 88

Question 89

Pre-test repetition

Answer 89

• Once you have incorporated all your changes, you should repeat the pre-test
• You need to recruit different people, since the original testers know how to understand your questions
• All questionnaires need a couple of alterations.
• Usually, 2-4 pre-test rounds are needed, to turn a good questionnaire into a really good one. Conduct a full pilot before starting actual data collection You need to test all the survey steps from start to finish with a reasonably large sample
• The size of the pilot sample depends on: * how big your actual sample is * how many data collectors you have

Question 90

Benefits of Survey Testing

Answer 90

• Uncovering Problems: survey testing can identify all types of problems, from intimidating survey length to ambiguously worded questions, poor design to flawed survey logic
• Pretesting helps avoid the time-consuming process of scrapping a flawed survey
• Improving Reputation: if you release a flawed survey, it can reflect poorly on your research and you might also waste the time and effort of respondents.
• Research Problem: if your survey is flawed, you will release inaccurate conclusions based on data accumulated through a flawed survey and your research will be unreliable
  Testing will improve your decision making
• Inaccurate data can lead to make poor decisions based on your research that ends up with incorrect conclusions.
• Testing a survey to be sure respondents understand it will result in more accurate responses when the final version is sent out, improving decision making. Safety and Compliance
• Testing your survey before releasing it, we can catch technical glitches and security issues and fix them before any damage is done. Improving Productivity

Question 91

History of Pretesting

Answer 91

• Modern sample survey started in the mid-1930s * Scholar started working on pretesting from 1940 when pretests were well established. * “The American Institute of Public Opinion pretested their questions to avoid phrasings which will be unintelligible to the public and to avoid issues unknown to the man on the street” * It usually takes about 12–25 cases to reveal major difficulties and weaknesses in a questionnaire

Question 92

Why Pretest?

Answer 92

• Questionnaire problems will be signaled either by the answers that the questions elicit (e.g., “don’t knows” or refusals)… * … Or by some other visible consequence of asking the questions (e.g., hesitation or discomfort in responding) * Moser and Kalton (1971, p. 50) judged, “Almost the most useful evidence of all on the adequacy of a questionnaire is the individual fieldworker’s [i.e., interviewer’s] report on how the interviews went, what difficulties were encountered, what alterations should be made, and so forth.” * Sudman and Bradburn’s (1982, p. 49) add: “A careful pilot test conducted by sensitive interviewers is the most direct way of discovering these problem words”

Question 93

Changes in Pretests in Recent Years

Answer 93

• Shift in the goals of testing: from an exclusive focus on identifying and fixing overt problems experienced by interviewers and respondents to a broader concern for improving data quality so that measurements meet a survey’s objectives. * New testing methods have been developed or adapted from other uses ➢ cognitive interviews ➢ behavior coding ➢ response latency ➢ vignette analysis ➢ formal respondent debriefings ➢ Experiments ➢ and statistical modeling The development of these methods raises issues of how they might best be used in combination, as well as whether they in fact lead to improvements in survey measurement.

Question 94

Pretesting: Methods of Testing

Answer 94

Pilot Testing
* Involves formally testing your complete, structured survey with a small sample of respondents.
* Rather than ask evaluation questions, you give respondents the survey as-is and ask them to complete it.
* Pilot testing your survey provides a sense of the kind of responses you will receive and any issues that may arise during the real survey period.

Data Analysis
* It can help to complete some data analysis.
* This involves looking at patterns in responses to see where confusion, hesitation, disengagement, or drop-out has occurred.
* You can often be discovered by identifying straight-lining (the same answer is always checked regardless of the question), unanswered questions, and inconsistent or unrealistic responses. Read more about these analysis techniques in our Data.

Question 95

Cognitive Interviews

Answer 95

• Cognitive interviewing can get to the bottom of what is going on in your respondent’s mind.
• These interviews are conducted in person with a small sample of respondents (generally about 5-15). As they answer each survey question, a facilitator asks them to think aloud, describing their thought processes, emotional responses, and understanding of what each question means.
• These interviews can help determine if a question is ambiguous, confusing, or makes people uncomfortable due to its content.
• They can also spot drop-out risks by identifying if respondents are getting bored or agitated by the survey.

Ordinary interviews focus on producing codable responses to the questions. Cognitive interviews aim to show how people think when they answer questions.
* Concurrent or retrospective “think-alouds” and/or probes are used to produce reports of the thoughts respondents have either as they answer the survey questions or immediately after. The objective is to reveal the thought processes involved in interpreting a question and arriving at an answer. These thoughts are then analyzed to diagnose problems with the
* Research on cognitive interviews has begun to reveal how the methods used to conduct the interviews shape the data produced.

Question 96

Behavior coding: Supplements to Conventional Pretests

Answer 96

Behavior coding
* Involves monitoring interviews or reviewing taped interviews (or transcripts) for a subset of the interviewer’s and respondent’s verbal behavior in the question asking and answering interaction.
* Questions marked by high frequencies of certain behaviors (e.g., the interviewer did not read the question verbatim or the respondent requested clarification) might need repair.
* Measurement of “response latency,” the time it takes a respondent to answer a question. Since most questions are answered rapidly, latency measurement requires the kind of precision (to fractions of a second) that is almost impossible without computers.
* Draisma and Dijkstra (2004) reasoned that longer delays signal respondent uncertainty, and they tested this idea by comparing the latency of accurate and inaccurate answers (with accuracy determined by information from another source).

Question 97

Vignettes

Answer 97

Hypothetical scenarios that respondents evaluate. They may be incorporated in either undeclared or participating pretests. Vignette analysis appears well suited to
1. explore how people think about concepts
2. test whether respondents’ interpretations of concepts are consistent with those that are intended
3. analyze the dimensionality of concepts
4. diagnose other question wording problems. Martin offers evidence of vignette analysis’s validity by drawing on evaluations of questionnaire changes made on the basis of the method.

Question 98

Expert Evaluation

Answer 98

• It may help to have an expert opinion, especially if your survey is asking questions about a complex subject matter.
• Topic experts can tap into their deep well of knowledge to evaluate your questions in order to help shape or reshape the content.
• Survey methodologists may also be called upon to help determine the best ways to collect accurate data for research questions.

Question 99

Focus Group

Answer 99

A focus group is a semi-structured discussion with a small group of people, usually around ten or so, led by a moderator. The members of the group can be very helpful in forming questions and addressing some of the following issues:
* Subject matter relevancy to the group
* Characteristics of the target population
* Understanding of questions and concepts
* Difficulty of the survey and abilities of the group
* General group reaction to the survey wording and design Focus groups can be very helpful but there is the danger of “group speak,” wherein problems can be built up and exaggerated as group members “feed off” one another during discussions. Focus groups should never be the sole means of survey testing

Question 100

Focus Group: how to process

Answer 100

Focus group is a semi-structured discussion with the purpose of stimulating conversation around a specific topic.
It is led by a facilitator who poses questions, and the participants give their thoughts and opinions.
It creates the possibility to cross check one individual’s opinion with other opinions gathered. A well organized and facilitated Focus Group Discussion is more than a question and answer session.
In a group situation, members tend to be more open and the dynamics within the group and interaction can enrich the quality and quantity of information needed

Question 101

Focus group: principles

Answer 101

• Define Objectives Clearly outline the goals and objectives of the focus group. Participant Selection Identify and recruit participants who represent your target audience or key stakeholders. Ensure diversity in demographics and perspectives Create a Moderator’s Guide Develop a structured moderator’s guide with openended questions to guide the discussion. This helps maintain consistency while allowing for flexibility to explore unexpected insights.
• Create Open-Ended Questions Develop a set of open-ended questions that encourage participants to share their thoughts, experiences, and opinions. Avoid leading questions that may bias responses, and allow for flexibility to explore unexpected insights that may arise during the discussion. Sequence Questions Logically Organize the questions in a logical sequence that flows naturally, gradually moving from general and introductory topics to more specific and in-depth areas. This helps build rapport with participants and ensures a smooth and focused discussion.
• Include Probes and Follow-Up Prompts Incorporate probes and follow-up prompts to delve deeper into participants’ responses. Can be designed to produce more detailed information, clarification, or diverse viewpoints. Probes help maintain the conversational flow and encourage participants to express themselves fully. Consider Group Dynamics Account for group dynamics in the moderator’s guide. Include strategies for managing potential challenges, such as dominating participants or a lack of engagement. Design questions that encourage all participants to contribute and ensure that the guide allows for flexibility in addressing unexpected topics that may arise during the discussion.
• Choose a Suitable Venue Select a comfortable and neutral venue conducive to open conversation. Ensure that the space allows for audio recording and video observation, if necessary. Moderator Skills Appoint a skilled and impartial moderator who can facilitate the discussion, keep participants engaged, and guide the conversation toward the research objectives. The moderator should be able to manage group dynamics effectively.
• Establish Ground Rules Set clear ground rules at the beginning of the session to create a respectful and inclusive environment. Emphasize the importance of confidentiality active listening avoiding dominant behavior. Limited Group Size Keep the focus group size manageable, typically ranging from 6 to 12 participants. This ensures everyone has the opportunity to contribute, and the discussion remains focused.
• Recording and Documentation Record the focus group session (with participant consent) for accurate analysis. Additionally, take detailed notes during the discussion to capture non-verbal cues, participant reactions, and key themes. Data Analysis Employ qualitative analysis methods to extract meaningful insights from the data. This may involve coding, categorizing responses, and identifying patterns or recurring themes. Triangulation and Validation Enhance the credibility of your findings by triangulating the results with other sources of data or by conducting multiple focus groups. This helps validate the reliability and consistency of the gathered insights.

Question 102

Interviewer Check-list

Answer 102

Question 103

Respondents Desirability Bias: what, how to avoid

Answer 103

Desirability Bias * Respondents often over-report desirable and underreport undesirable behaviours to portray themselves positively. * The key to avoiding it * Minimise the aspects of answering that may seem threatening to respondents * Increase the more salient features that influence comfort within the respondent to reveal private or sensitive information. * This can be done by increasing privacy and anonymity, for instance.

Question 104

Interviewers Bias: reasons

Answer 104

Bias
* Interviewers have a significant influence on survey reports.
* An interviewers’ race, gender, and expectations can all have an impact on responses.
* An interviewer’s conduct can also impact data quality, especially when it strays from the standardised procedures.
* This leaves room for measurement error caused by interviewer-respondent interactions which can result in biased responses and a decreased reliability of answers.
* The mere presence of interviewers influences survey respondents and their answers.

Question 105

Standardized Interviews: why, how

Answer 105

Standardized interviewer behavior is necessary for reasons of reliability:
* Survey methodology works from a stimulus-response model of interaction
* Every respondent should be presented with the same verbal input
* Interviewers should read aloud each question exactly as worded, preserve the order of the questions, and probe in a non-leading manner
* Consistency ensures that differences in responses can be related to respondent characteristics, rather than to characteristics of the interviewing procedure

• A change in the formulation or the order of questions and response options can have an effect on the responses * It is crucial that interviewers do not alter anything in the questionnaire Interviewers need to follow the rules of standardized interviewing: * Read the questions exactly as worded * If the respondent’s answer is incomplete or inadequate, probe for clarification or elaboration in a non-directive way * Record the answers without interviewer’s discretion. * Do not provide any positive or negative feedback regarding the specific content of respondent’s answers

Several studies show that interviewers often do not act according to these rules Reasons: Question may be ambiguous or difficult to read aloud * Can be solved in pre-testing Respondents may not understand what they are expected to do * They often do not act their roles as prescribed * Interviewers need to explain clearly to respondents the rules of the game

Question 106

Exploring Inadequate Answers: reactions, decisions

Answer 106

• Respondents can provide non satisfactory answers and interviewers must probe for more adequate or more complete ones * Do this in a non-directive way. Interviewers’ probes should not increase the chance of one answer over another * Each probe that can be answered by “yes” or “no” is directive

We need to establish rules that interviewers should follow when probing answers: * When respondents miss part of the question, interviewers should reread the entire question * When respondents do not understand a term or concept in the question because a term turns out to be ambiguous, interviewers should not provide an explanation. They can reread the definition, if the questionnaire provides one. * Otherwise, respondents must be told to interpret the question to their own liking and answer the question. * When probing answers, interviewers should make sure that answers meet the questions’ objectives.

Question 107

Administer the Survey: interviewer behaviour

Answer 107

• With interviewer-administered surveys, the interviewer provides feedback to the respondent * If an invalid response is provided, the interviewer should repeat the question or response options in order to get an accurate response * When a respondent asks for clarification, the question should be restated using the same words * When interviewers try to clarify the question on the fly, these explanations can change the meaning of the question. (This problem can be avoided by careful development of the questionnaire and pretesting the instrument.) * Recording answers accurately and completely is important. When using a paper questionnaire, the answers must be readable

Question 108

Introductory Script: The Outline and Format

Answer 108

• Introduction to the survey * Title of the study * Purpose * Duration of survey and length of questionnaire * Guarantee of confidentiality * Brief information about the company or organization (IMS)

Question 109

Preparing the Introductory Script and Support Materials

Answer 109

Introductory script and other support materials must be prepared before implementing the survey * The introductory script is used by the interviewer to schedule the * Date * Time * Location for the interview * Interviewers have about 20–30 seconds to obtain a respondent’s agreement to participate; * The scripts should be as brief as possible while also communicating critical information about the purpose and importance of the survey.

Question 110

Input Mask

Answer 110

• An input mask makes it much easier for users to figure out the required format for filling out the form * Input masks are a way to constrain data that users enter into form fields and enforce specific formatting * Use Google Forms to design your survey

Question 111

Choices: Designing an Input Mask

Answer 111

• User must enter a digit (0 to 9). * User can enter a digit (0 to 9). * User must enter a letter. * User can enter a letter. * User must enter a letter or a digit. * User can enter a letter or a digit. * User must enter either a character or a space. * User can enter characters or spaces. * Decimal and thousands placeholders, date and time separators. The character you select depends on your regional settings (in Germany commas and full stops).

Question 112

Sampling procedures: 2 ways

Answer 112

We have to provide information about certain characteristic features of statistical populations.

Two ways
Census:
▪ Complete count: It is a study of every unit, everyone or everything, in a population

Partial sampling (sub-samples): ▪ A sample, which is a subset of units in a population
▪ Selected to represent all units in a population of interest. ▪ Information from the sampled units is used to estimate the characteristics for the entire population of interest.

Question 113

Reasons for Sampling

Answer 113

1. Cost saving ▪ Collecting smaller amounts of elements is cheaper than conducting a census (complete enumeration).
2. Time saving ▪ Data collection in the case of sampling takes less time than a census.
3. A census or complete enumeration is practically impossible ▪ It is theoretically imaginable, but practically not reasonable or possible.

Question 114

Sampling Frame

Answer 114

Question 115

Required Sample Size and Sample Mean

Answer 115

• To achieve a certain level of confidence and accuracy a sample size is required to be at least a certain size * To calculate the required sample size we need to use these factors: population size, parameter variation, degree of accuracy, and degree of confidence * Our sample mean should be close to the population mean, the sample inter-quartile range to be close to the population interquartile range, and so on

Question 116

Parameter Variation and Confidence Level

Answer 116

Parameter Variation: * Is the process of changing the input parameters (for example: height, age, etc.) of a mathematical model or simulation to investigate the effect of each parameter on the output. * We want to determine which input parameters have the most significant impact on the output and how the output responds to changes in these parameters. Confidence Level * Is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way

Question 117

Degree of Accuracy

Answer 117

• Sample accuracy refers to the extent to which sample statistics correctly estimate the population parameter. * We typically use the terms biased and unbiased to describe the accuracy of sample statistics. Example: we take many thousands of samples from the same population. For each sample, we calculate a statistic (e.g., the mean). ➢ Unbiased sample: If the average of the sample statistics (e.g., sample means) equals the population parameter (e.g., population mean) then we refer to the statistic as being unbiased. ➢ Biased sample: If the average of the sample statistics (e.g., sample means) does not equal the population parameter (e.g., population mean) then we refer to the statistic as being biased

Question 118

Sampling

Answer 118

Question 119

Arbitrary Sampling

Answer 119

▪ In arbitrary sampling, the researcher selects individuals or units of study because they are available, convenient and represent some characteristic the researcher seeks to study without following any specific system, e.g. street surveys. ▪ Representativity (the sample is a reduced, structurally similar representation of the population under study) is problematic.

Question 120

Probability Sampling Designs

Answer 120

• Every case in the population has a known, greater-than-zero probability of being selected for the sample. For example quantified summaries of characteristics of the sample, like the median of a variable or the correlation between two variables. * With probability sampling design, we can use statistics to estimate the parameters (the corresponding quantified characteristics of the population) with known levels of confidence and accuracy.

Question 121

Random Sampling

Answer 121

Reliable selection of individuals or units of study from the population.
▪ The intent of random (or probability) sampling is to choose individuals or units for the sample so that any unit has an equal probability of being selected from the population.
▪ The more often one takes random samples from the same population, the closer one will get to the actual value in this population.
▪ No pre-specific characteristics
▪ The empirical social research accepts in general a 5 percent probability of error. The scores it provides always contain this sampling error.

Example: lottery procedure, i.e. the random extraction of elements from a limited population. Common selection or sampling procedures: ▪Choose every n-th element (or individual) in the population (e.g. selecting the file with the number 100 from a student file system) The criterion according to which a systematic random sampling is being applied needs to be distributed independently from the features that are studied in the population.

Question 122

Random Cluster Sampling

Answer 122

Cluster sampling is a spatiotemporal (time and space) defined conglomeration of elements of the population which form a structurally reduced representation of the respective population. Typical clusters: households, school classes, apartment buildings. Elements included: persons, pupils, households. Procedure: you choose by random sampling a number of clusters and analyse all units of study that occur in these clusters

Question 123

Simple Random Sample

Answer 123

Most basic kind of probability sampling design Each case in the population has a known and equal probability of being selected for the sample. Goal: construct a sample that is like the population so that we can use what we learn about the sample to generalize to the population.

Example: If you want a sample of 50,000 graduates using age range, the proportionate stratified random sample will be obtained using a formula: (sample size/population size) × stratum size. A stratum (plural strata) refers to a subset (part) of the population (entire collection of items under consideration) which is being sampled. The table below assumes a population size of 180,000 MBA graduates per year.

Calculation: The strata sample size for MBA graduates in the age range of 24 to 28 years old is calculated as (50,000/180,000) × 90,000 = 25,000. Same method is used for the other age-range groups. 25,000 graduates from the 24–28 age group will be selected randomly from the entire population 16,667 graduates from the 29–33 age range will be selected from the population randomly, and so on

Question 124

Probability Sampling and Nonprobability

Answer 124

Sampling Design Sampling frame: list of cases we select our sample from— ideally, a list of all cases in the population Select cases for sample through a sampling design. Two basic varieties of sampling designs: ➢Probability sampling designs and nonprobability sampling designs.

Question 125

Nonprobability Sampling

Answer 125

In nonprobability or purposeful sampling, you select individuals or units of study according to how “useful” or essential their analysis is for finding an answer to the research problem. ▪ These sampling procedures can be problematic with regard to representativity. − Four procedures: 1. Sampling typical cases 2. Sampling extreme groups 3. Sampling according to concentration principle 4. and quota sampling

Question 126

Typical Sampling:

Answer 126

▪ Especially appropriate in qualitative research methods, where only few individual cases are analysed. ▪ The sample consists of units of study (e.g. autistic people) who are especially characteristic for all units of study in the population

Question 127

Sampling according to the Concentration Principle

Answer 127

Sampling according to the Concentration Principle ▪ In a sampling procedure, the researcher focuses on the part of the population where they suspect the predominant part of these elements to be. − Example: Investigation about German skiers, −95% of all German skiers live in Bavaria – using a random sample only from the Bavarian population Cut-Off-Procedure = The less productive or rich part of the study population is being cut off.

Question 128

Quota Sampling

Answer 128

In quota sampling, you select individuals or units of study according to some fixed quota. ▪ Units of study are selected based on pre-specified characteristics so that the total sample has the same distribution of characteristics assumed to exist in the population that is studied. ▪ Quotas generally rely on demographic characteristics. Example: the sample displays, according to its quota-based characteristics, a structurally identical representation of the studied population

Question 129

Sample Drop-outs

Answer 129

Sample drop-out refers to all cases where an element of a sample could not be analysed. We distinguish between random and systematic deniers. Random drop-out: Relocation, illness… Systematic drop-out: Persons who deliberately refuse to take part in the survey e.g. highly educated people, people with low language knowledge etc.

Question 130

Sampling Variation: Problems with Sampling

Answer 130

Sampling variation is how much the results change when you take different samples from a group. Because of this, there is always some uncertainty when you try to make conclusions about the whole group based on those samples. Sampling variation is about how much a number you find might change when you look at different samples of things. (e.g there is no blue person in the sample) ➢ If you measure something in a few different samples, the result might be a little different each time. This means there is some uncertainty when you try to say something about the whole big group based on just those smaller ones.

Sampling variability is how much an estimate varies between samples. ➢ Variability between samples indicates the range of values differs between samples. ➢ If you are measuring something in different samples, you might notice that the numbers you get can be quite different from one group to another. ➢ That is what we call sampling variability—it is like the range of values you see across those different groups.

Question 131

How can we know how much something varies?

Answer 131

How can we know how much something varies? * Assume maximum variation which places the highest demand on sample size * Assuming maximum variation in a sample size means imagining that each sample you pick is as different as possible from the others. * Imagine every sample represents the extreme ends of whatever you are studying. This helps ensure that your sample size is big enough to cover all the possible variations in the population you’re interested in. It is a way to make sure you are not missing out on any important differences that might affect your results

Question 132

Why do we specify the variation?

Answer 132

Variation is a way to show how data is dispersed or spread out. * To specify variation in a sample size, you typically consider factors such as * the variability of the population you are studying the level of precision you need in your estimates any constraints on resources like time and budget.
Understand the Population Variation: * Consider the range and variability of the characteristics you are interested in within the entire population. This helps you gauge how much variation might exist in your samples. Define Precision Level * Determine the level of precision you need in your estimates. This depends on factors like the desired confidence level and margin of error. Choose a Confidence Level: * The choice of confidence level depends on how confident you need to be in your results, the potential consequences of being wrong, and any industry standards you may need to adhere to. For most purposes, 95% is a safe and commonly accepted level * Decide on the confidence level you want for your estimates. Common choices are 95% or 99%.

Question 133

Calculate Sample Size

Answer 133

• Take into account the population variation, desired precision, and confidence level. * Take into account any practical constraints such as budget, time, or feasibility of collecting data. Sometimes you may need to balance statistical considerations with these constraints. * If necessary, adjust the sample size based on factors like the complexity of the analysis, potential non-response rates, or the need for subgroup analysis. * Validate Sample Size: After collecting data, validate whether your sample size was sufficient to provide reliable estimates. You can do this by checking if the confidence intervals are narrow enough to meet your precision requirements.

Question 134

Variability and Sampling Error

Answer 134

A closely related term (almost a synonym) is sampling error. An error in sampling isn’t a mistake — it’s a measure of how much a value differs from the “true” value. Example: − Let’s say the true weight of a population is 150 lbs. − You take a sample and find the mean weight for the sample is 151 lbs. − The 1 lb difference is an “error.” − If you sample again, you might get different mean weights of 148 lbs, or 150.5 lbs, or 153 lbs. − The different errors — 1/2 lb, 1 lb, 2 lbs, 3 lbs — are a reflection of the variability between your samples, or sampling variability.

Question 135

Sampling Error

Answer 135

• Increasing or decreasing sample sizes leads to changes in the variability of samples. * For example: a sample size of 10 people taken from the same population of 1,000 will very likely give you a very different result than a sample size of 100. * There is no “perfect” sample size that will give you accurate estimates for the sample mean, variance and other statistics. Instead, you take your best “guess” — using standardized statistical procedures. In general, estimates will change from sample to sample and will probably never exactly match the population parameter.

Question 136

Why do we need to verify a hypothesis?

Answer 136

• Hypothesis testing is one of the most important processes for measuring the validity and reliability * It helps to provide links to the underlying theory and specific research questions * Verifying a hypothesis provides a reliable framework for making any data decisions for our population of interest * It helps to successfully extrapolate data from the sample to a larger population * Verifying a hypothesis allows us to determine whether the data from the sample is statistically significant

Question 137

Hypothesis testing

Answer 137

A hypothesis is a proposed explanation for a phenomenon. The term hypothesis is a statement about something that is supposed to be true. The logic of a hypothesis test is to compare two statistical data sets A hypothesis test involves two hypothesis: * the null hypothesis * and the alternative hypothesis

Question 138

Null hypothesis and alternative hypothesis

Answer 138

Question 139

Conducting a Survey

Answer 139

• Face-to-face surveys follow a standardized script without deviation * Face-to-face surveys also offer advantages in terms of data quality. * A face-to-face survey allows researchers a high degree of control over the data collection process and environment. Interviewers can ensure that respondents do not skip ahead * Responds rate of face-to-face survey is higher * There is also evidence that questions of a personal nature are less likely to be answered fully and honestly in a face-to-face survey

Question 140

Advantages of Face to Face Surveys

Answer 140

• Capture verbal and non-verbal cues (discomfort or enthusiasm)
• Helps keep respondents focused
• A surveyor has control over the interview, they can help ensure the interviewee remains focused and on track to complete their survey
• Can allow for more complex questions as interviewer is present to clarify wording and probe for more information
• Can generally be longer than a phone interview
• Arranged to be convenient for the participant to take part in
• Can access samples that phone and Internet questionnaires might not be able to reach
• Ability to incorporate visual stimuli into the questionnaire
• Generally have a higher response rate than other methods

Question 141

Conducting a Survey - Interviewer bias

Answer 141

• Interviewer is exposed to the potentially biasing effect of the respondent’s appearance and environment in addition to their voice * Interviewer may inadvertently give respondents nonverbal as well as verbal cues about how they should respond

Question 142

Interviewers - Team Composition

Answer 142

The Team composition and the number of interviewers per team need to be decided based on a number of factors: * the expected duration of interviews * the content of the survey * the size of clusters, etc. On the average each team will need * one supervisor * one runner/editor * three to five interviewers (depending on how many people an interviewer can complete)

Question 143

Team Lead’s Job

Answer 143

• Identify the clusters to be surveyed * Supervise interviewers as they perform the survey * Ensure that the interviewers follow instructions * Answer interviewers’ questions as they arise * Control data quality by checking for errors during interviewing, checking that forms are completed fully and correctly, and checking that all respondents are answering the questions * Identify problems and retrain interviewers who are doing their job incorrectly.

Question 144

The field runner/editor’s job is

Answer 144

• Monitoring interviewer performance by: * Observing several interviews, especially during the early stages of fieldwork * Conducting regular review sessions with interviewers (if needed) * Compiling completed questionnaires from a cluster and packing them up

Question 145

The interviewer’s job is to

Answer 145

• Identify the specific households to be surveyed * Gain the consent of respondents to be interviewed * Conduct interviews using the standard questionnaire * Maintain standard procedures in conducting the interviews and recording the answers.

Question 146

Data Cleaning Specific to our Survey

Answer 146

Missing Data
* Open Ended Questions: * For these questions, you may need to categorize the write-in responses to create a consistent coding scheme. * Sometimes participants did not answer all the survey questions due to any reason * This partial filled data can create some noise in pure data analysis. It is advised to remove the partial completed responses from the overall data before starting analysis. * Check Spelling - Merge Doubling

Skipped Questions
* Identify if any questions were consistently skipped by a large number of respondents. * This might indicate a problem with the question itself or the survey flow.

Question 147

Inconsistent Data

Answer 147

• Age (Q2) and Employment Status (Q5): There might be inconsistencies (e.g., someone selecting “Student” under employment but an age outside the typical student range). * You may need to decide how to handle these discrepancies (e.g., contacting respondents for clarification, excluding outliers). Response Options * “Prefer not to say” (Q1): Decide how to handle this option. * Similar to “Prefer not to say”, decide how to categorize this for respondents who might not work in media (e.g., separate category).

Question 148

Data Cleaning Techniques

Answer 148

Standardization * Ensure consistent formatting for written responses (e.g., all uppercase or lowercase). Coding * Create numerical codes for answer choices in multiple-choice questions to simplify analysis. Outlier Detection * Identify and potentially exclude extreme values that might skew the results (consider reviewing these cases individually).

Question 149

Respondents Who Give Inconsistent Responses

Answer 149

• When a respondent’s answer contradicts their response to another question, it’s clear that they’re either being dishonest or careless (or even both!). * You may be able to find these inconsistencies by applying multiple filters. * For instance, say that one of your survey questions asked respondents how much TV they watch per week. When the responses come back, you filter by those who said they watch at least a little. Document Decisions * Keep a log of the decisions made during data cleaning to ensure transparency and replicability. Remember, data cleaning is an iterative process. As you explore the data, you may identify new cleaning needs

Question 150

Remove Straightliners and Unrealistic and Nonsensical Answers

Answer 150

• Researchers needs to be cautious about straightline respondents and responses must be removed before analysis. * Straightlining is when respondents choose similar answer option frequently (such as first/last option etc.). * There might be higher possibility that the respondent has not responded the answers honestly. * Imagine asking respondents how much TV they watch per week, on average. If a respondent writes in 165 hours, they’re likely exaggerating (Hint: there are only 168 hours in a week). * Having a response like: “Fdsklj” might make you smile, but it isn’t going to get you far in your analysis.

Question 151

Remove Outliers - Remove Fake Or Manipulated Answers

Answer 151

• Sometimes you have encountered responses falling under unrealistic range for example “In which sport you see yourself as pro” and one respondent selected all the sports from option. * This kind of responses are called outliers. * This becomes tricky sometimes but researchers should always be beware of fake or manipulated responses from participants. * There are many ways to check fake responses such as, using open ended questions — check is any response contain unreadable or meaningless response like ‘fgfgfh’ type text etc.

Question 152

Conceptualizing your Qualitative Research Question

Answer 152

• Should be inductive, exploratory * Is framed as a question, aim or objective but not as a hypothesis * Should focus on a single phenomenon, concept or idea * Start with a verb and stating your goal (e.g. characterize, understand etc.) * Identify your topic of interest * Use a language that is non-directional and neutral to be exploratory * Define your sample and your setting * Example: Understand interpersonal factors relevant to relationships formed online by retirees - in private homes in Bavaria

Question 153

Methodology

Answer 153

Question 154

How to Analyze Surveys?

Answer 154

1.Look at the results of your survey as a whole 2.Take a look at the demographics of those who responded 3.Compare responses to different questions to find deviations 4.Find connections between specific data points with layered data 5.Compare new data with past data

Question 155

Survey Data – Connection to Research Question

Answer 155

− To what extent do media practitioners perceive AI tools in improving productivity, credibility, and creativity? * Filter results by cross-tabulating subgroups * Filter your results based on specific types of respondents, or subgroups * Look at different attendees (age, gender)

• What are the most common responses to a specific question? * Which responses are affecting/impacting us the most? * What did respondents with a particular job title say? * Which group of respondents are most affected by a specific issue? * Have conference visitors noticed something outstanding? * What do people say about a specific topic?

4. Find connections between specific data points with layered data − Understand connections − Look for causation and correlation − Look for confounding variables 5. Compare new data with old data − Check for changes and try to find explanations for them

Question 156

Differences between Codes, Categories and Themes

Answer 156

Meaning unit: It is the smallest unit of analysis that captures a single idea or concept in the data.
Condensed meaning unit: It is a shortened version of the meaning unit that retains the essential information.
Code: It is a label assigned to condensed meaning units to categorize and organize the data.
Category: It is a group of similar codes that represent a broader concept or idea.
Theme: It is a higher-level abstraction that emerges from grouping or categorizing related codes and categories.