Empirical methods exam Flashcards

Question

Shapes of Distribution - Skewness

Answer 1

Skewness is a measure of a lack of symmetry in a distribution. * A standard normal distribution is perfectly symmetrical and has zero skew

Answer 2

Variability describes how far apart data points lie from each other and from the center of a distribution. Range: the difference between the highest and lowest values. Interquartile range: the range of the middle half of a distribution. Standard deviation: average distance from the mean. Variance: average of squared distances from the mean

Answer 3

*** The range is the difference between the highest and lowest values within a set of numbers**. * To calculate range, subtract the smallest number from the largest number in the set. * Range shows how much the numbers in a set vary

Answer 4

The middle fifty or midspread of a set of numbers, removes the outliers (highest and lowest numbers in a set) * If there is a large set of numbers, divide them evenly into lower and higher numbers. * Then find the median of each of these groups. * Find the interquartile range by subtracting the lower median from the higher median.

Answer 5

Variance measures how far a data set is spread out. It is mathematically defined as the average of the squared differences from the mean. * Finding the mean (the average). * Subtracting the mean from each number in the data set and then squaring the result. * The results are squared to make the negatives positive. Otherwise, negative numbers would cancel out the positives in the next step. It’s the distance from the mean that’s important, not positive or negative numbers. * Averaging the squared differences.

Answer 6

Measures of dispersion help to describe the variability in data. * Dispersion is a statistical term that can be used to describe the extent to which data is scattered. * Thus, measures of dispersion are certain types of measures that are used to quantify the dispersion of data. * The variance helps to draw a comparison between the two data sets A and B on the basis of variability.

Answer 7

***Descriptive*** * Used to collect participants’ opinions about the variable that you want to quantify * Descriptive research questions begin with “ how much?” “how often?” “what percentage?” “what proportion?” * Example: How often do middle-class adults go on vacation yearly? (variable: vacation; group: middleclass adults) ***Comparative*** * Help you identify the difference between two or more groups based on one or more variables Example: What is the difference in the usage of TikTok between male and female Cambodian university students? (variable: usage of TikTok; group 1: male Cambodian university students; group 2: female Cambodian university students) ***Relationship-based***: * Used to identify trends, causal relationships, or associations between two or more variables. Example: What is the relationship between salary and shopping habits among the women of Australia? (independent variable: salary; dependent variable: shopping habits; group: Australians)

Answer 8

Survey questions should be designed in a way that any respondent can understand them. Survey questions can be classified into 4 subtypes: − 1. Fact-based questions − 2. Knowledge-based questions − 3. Attitudinal and opinion questions − 4. Behavioural questions

Answer 9

Target: to formulate every single question in such a way that no biased responses, misunderstandings or ambiguities can occur. ✓ Is the question clear for everybody? ✓ Is the issue that the question is based on precisely and comprehensively formulated? ✓ Can everyone clearly understand (and then describe) the issue in the question? ✓ Do the questions impose a overly high level of verbal skills? ✓ Is the question one-dimensionally formulated, is it clear what the respondent needs to answer? ✓ Is it possible that the question might overload the memory of the respondent? ✓Is the respondent still able to answer the question even if he is already exhausted? ✓ Does the question induce social desirability by the way it is formulated? ✓ Does the question contain suggestive wording that could lead to a certain response behaviour? ✓ Might the question be too intimate? ✓ Would it be possible for respondents to feel embarrassed by the response (and therefore give an untruthful answer)?

Answer 10

Function questions control the course of the questionnaire without bringing any contribution to the actual result interest. These questions guarantee that the survey questions are applied correctly. 1. Ice-breaker questions 2. Transfer and Resting questions 3. Filter and Funnel questions 4. Verification questions

Answer 11

Ice-Breaker Questions * Are used to establish a relationship with the respondent. * In order to make the interview atmosphere more relaxing, the interviewer asks in the beginning a question which is unimportant in comparison to all other questions and will not be analysed. * Example: “What is your opinion about the current TV programme?“

Answer 12

Transfer and Resting Questions * Are meant to distinguish subject areas in a survey in a clear manner for the survey participant without focusing on the relevance of the content. * They allow the respondent to talk freely.

Answer 13

* Make sure that only those people who are truly concerned answer the survey questions. * Funnel questions sort out all the respondents who cannot or should not give an answer to the actual questions (e.g. Pay-TV users). * Filter questions work by the same logic which differentiates uncoupling and bifurcation. * By uncoupling, unnecessary questions are skipped. * Bifurcations are integrated to receive information about all respondents divided into subgroups.

Answer 14

Verification Questions Are integrated into the questionnaire together with alternating questions. Aim: to verify the consistency of response behavior Credibility Questions ▪ Are meant to verify the credibility of the participants. ▪ The questions refer to issues which everybody has allegedly experienced in his life. ▪ If the respondent gives an answer which is highly unlikely, then this is a proof that the respondent might sometimes give untruthful responses.

Answer 15

Sociodemographic questions are not content-related and are usually placed at the end of the survey. The following aspects are usually asked for: * Age * Gender * Education * Head of household * Net income * Religion * Marital status

Answer 16

***In-depth/Guideline Interview*** Develop a guideline and define specific question sequence. Semi-structured or semi-standardized interview. ***Special type of survey***: group interview Conduct group interview according to a set of guidelines. In group discussion, opinions are encouraged and generated. Participants mutually influence their response behaviour. Example: Introducing new products etc. Special attention is needed with interpretation. Therefore, usually it is used as a complementary instrument.

Answer 17

Bias can occur when respondents want to be liked and to avoid embarrassment are very strong. This can affect how people answer questions asked by strangers. * Question wording must facilitate unambiguous, fully accurate communication * Leading questions are the most obvious culprit

Answer 18

***Use normalizing statements***: * Example: Some people like to follow politics closely and others aren’t as interested in politics. How closely do you like to follow politics? * Closed-ended questions (questions that give answers for respondents to select from) are susceptible to response set bias.

Answer 19

Can give respondents freedom to answer how they choose * Remove any potential for response set bias * Allow for rich, in-depth responses if a respondent is motivated enough. * Challenge: respondents can be ambiguous; can give responses that indicate the question was misunderstood * Researcher is giving power to respondents to structure the data

Answer 20

* Answers are unambiguous * Data are easy to manage. * Challenge: researcher is structuring the data, which keeps things nice and tidy

Answer 21

All surveys are conducted in one of the three survey modes: * Face-to-Face Interview * Written interview * Telephone Interview / Survey. Additionally, there is a version of written interviews that has established itself in recent years: the online survey.

Answer 22

Face-to-Face-Interviews: * Personal interviews require qualified interviewers. * Result high costs relatively high time consumption Pro * Fully used sample has very low refusal and abandon rate * Return rate (40-70 percent) is considerably higher in comparison to written, telephone and especially online surveys.

Answer 23

Telephone Interviews: Due to technical development, telephone interviews are becoming more and more popular and are extensively used as a current survey mode in professional survey research institutes. Disadvantage: * Visual auxiliary material cannot be used * Risk of overloading (seven-level scale) * Abandon rate is higher than in face-to-face interviews

Answer 24

Written Survey (Paper & Pencil) Disadvantages of surveys by conventional mail: * low return rate (partly under 10 %) * motivation of the respondents is usually low, the social contact is missing * field phase is considerably longer than in other survey modes Higher rates when: * the respondents have an additional benefit (winning competition) * the questionnaires are notified by telephone * there is a follow-up campaign. Groups of people who are hardly reachable can be approached more easily via written surveys. (e.g. politicians)

Answer 25

Online Surveys are Internet-based survey methods, which can be classified into three groups: The questionnaire … * is saved on a server and is filled out online. * can be downloaded from a server and sent back via E-Mail * is sent via E-Mail and returned the same way after filling it out. Option 1 with the possibility to directly fill out the questionnaire online enjoys the highest popularity at the moment. There are different software on the market to help create and use online survey tools. (e.g. Lime Survey, SurveyMonkey).

Answer 26

Survey development is a collaborative and iterative process where researchers meet to discuss drafts of the questionnaire several times over the course of its development. * Conducting frequent tests with new survey questions ahead of time through qualitative research methods such as focus groups, cognitive interviews, pretesting * Choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent. It is important that all respondents interpret the question the same way.

Answer 27

We must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions * Two main types of order effects for closed-ended opinion questions: * Contrast effects (where the order results in greater differences in responses) * Assimilation effects (where responses are more similar as a result of their order). How to avoid: The best way to combat these errors is randomization. You can randomize your answer options for every respondent so that e ach option has a fair chance of being picked. As for the question order bias, make sure that your questions don’t affe ct the answers to the following questions. Example: You ask your employees about their issues with the reporting manager and then ask if they are happy in the workplace. Now, there is a possibility that they would have answered the questions differently if the order had been reversed. Their second answer will be influenced by the answer to the first question.

Answer 28

Is made up of pre-populated answer choices for the respondent to choose from * Multiple choice * Drop down * Checkbox * Ranking question * Questions that are closed-ended are conclusive in nature as they are designed to create data that is easily quantifiable * Closed-ended questions allows researchers to categorize respondents into groups based on the options they have selected (allows for demographic studies) * Major drawback to closed-ended questions is that a researcher must already have a clear understanding of the topic of their questions and how they tie into the overall research problem before they are created.

Answer 29

Always a close-ended question * A survey scale represents a set of answer options—either numeric or verbal—that cover a range of opinions on a topic. * Will let us uncover degrees of opinion that could make a real difference in understanding the feedback you are getting * Are structured to provide quantifiable answer options that make analyzing data easier * Each series of questions in the survey focused around the same topic

Answer 30

The Likert scale is a five (or seven) point scale that is used to allow an individual to express how much they agree or disagree with a particular statement. Simple questions: the less thinking required from respondents the better the response rate. Example: How satisfied were you with your in-store experience? It was easy to navigate the website to find what I was looking for.

Answer 31

Accuracy: likert-type questions must be phrased correctly in order to avoid confusion and increase their effectiveness * Descriptive words need to be easily understood and should not be ambiguous * There should be no confusion about which grade is higher or bigger than the next * Better ask a question than let people rate a statement

Answer 32

What is your marital status? Are you … * married and living with your spouse * married and separated * widowed * divorced * unmarried

Answer 33

What is your marital status? Are you … ▪married and living with your spouse ▪married and separated ▪widowed ▪divorced ▪unmarried ▪I have a different marital status, which is: _________________

Answer 34

Which of the devices on the list are in your household? (multiple answers possible) ▪washing machine ▪dishwasher ▪food processor ▪espresso machine ▪none of these

Answer 35

Which of the devices on the list are in your household? (multiple answers possible) ▪washing machine ▪dishwasher ▪food processor ▪espresso machine ▪vacuum cleaner

Answer 36

Please put these 7 properties in the order that best describes your diet. please indicate the order by assigning numbers from 1 = most important to 7 = least important __ must eat meat __ must not eat meat __ give me sweets __ comes out of a wrapping __ circumnavigates food intolerances __ any hipster label here __ super duper healthy

Answer 37

Of these 7 properties, which best describe your own diet? Chose 3 at most. (max. 3 items) ▪must eat meat ▪must not eat meat ▪give me sweets ▪comes out of a wrapping ▪circumnavigates food intolerances ▪any hipster label here ▪super duper healthy

Answer 38

* Biased or leading questions: Phrasing questions in a way that suggests a particular answer can skew results. Ensure questions are neutral and don't lead respondents to a specific response. * Ambiguous questions: Vague or unclear questions can confuse respondents and lead to inaccurate responses. Questions should be precise and easy to understand. * Overly long or complex surveys: Lengthy surveys can lead to respondent fatigue and lower completion rates. Keep surveys concise and focused on relevant topics. * Limited response options: Providing insufficient response options can limit the range of answers and fail to capture diverse perspectives. Include a variety of response options, including open-ended questions when appropriate. * Order bias: The order in which questions are presented can influence responses. Be mindful of question sequencing and consider randomizing or alternating question order to mitigate bias. * Social desirability bias: Respondents may provide answers that they perceive as socially acceptable rather than their true opinions or behaviors. Use anonymous surveys and assure respondents of confidentiality to minimize this bias. * Sampling bias: If the survey sample is not representative of the target population, results may not be generalizable. Ensure the survey sample accurately reflects the demographics and characteristics of the population of interest. * Non-response bias: If certain groups are more likely to participate in the survey than others, results may be skewed. Employ strategies to encourage participation from underrepresented groups.

Answer 39

1. Theory 2. Hypothesis 3. Research design 4. Devise measures of concepts 5. Select research sites (s) 6. Select research subjects/respondents 7. Administer research instruments/collect data 8. Process data 9. Analyse data 10. Findings/ conclusion 11. Write up findings/ conclusion

Answer 40

* Empirical social research is interested in conducting a systematic and intersubjectively traceable analysis of cases attributed to a phenomenon (our research question). * Quantifying any type of variable through a systemic method Definition: − Measuring is the systematic assignment of an amount of numbers or symbols to the values of a variable, hence also to the objects.

Answer 41

Classification occurs by looking at the value of the variables. Nominal (categorical, qualitative) data * Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. (e.g. gender) Ordinal data (rank variable) * Variables that have two or more values which can also be ordered or ranked. That means that variable values have a natural sequence. (e.g. school grades). Metric data * Values that can be organized according to size AND can represent a multiple of a unit (e.g. height)

Answer 42

Levels of measurement or scales of measurement – classification principle of variables I − Nominal data / variables (e.g. hair colour/nationality) − Ordinal data (variables have natural, ordered categories: e.g. hot, hotter, hottest) − Metric data (e.g. age)

Answer 43

− Discrete variables - counting numbers with specific values (e.g. age) − Continuous variables - measurable number e.g. 1.7 feet

Answer 44

Эмпирические и численные данные * Measuring is the process by which the empirical relative, e.g. the characteristic value “person in dress” is systematically assigned a numerical relative, e.g. the number 1. * The numerical relatives contain numerical values or scores, the empirical relatives define objects or objects of analysis * Each element of the empirical variables is assigned a numerical value in such a way that the relations in the empirical relatives correspond or equal the relations in the numerical relatives. * This rule of assigning values is also called scale.

Answer 45

Object of analysis Attribute/ Variable Value

Answer 46

Scales: The one-to-one transfer of empirical relatives into numerical relatives leads to the formation of scales. We distinguish between three scales of measurement. The classification is done according to the value of the variables: * Nominal variables/ data/divide into class/numbers * Ordinal variables/ data/Scal in order/ 1st, 2nd, 3rd/Low, medium, moderate * Metric variables/ data/exam score/grading system 1-4

Answer 47

Nominal Data: Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. ▪ The categories must be defined precisely. ▪ The categories must mutually exclude themselves. ▪ The categories must describe the variable exhaustively.

Answer 48

▪ Variables that can take two or more values which can be ordered or ranked in a sequence. ▪ That means that variable values have a natural sequence. ▪ The measured values on an ordinal scale can be put into a logical sequence.

Answer 49

▪ Variable values that can be organized according to size AND can represent a multiple of a unit. ▪ Values that have been measured on a metrical scale can take any numerical value (or score). ▪ The range or the intervals between the possible measured values have to be always the same. E.g age

Answer 50

▪ Dichotomous (dividing into two parts) variables are variables which have exactly two values, e.g yes/no ▪ Example: Do you have a mobile phone?

Answer 51

▪ Polytomous variables display more than two values. E.g divide in class, brands of mobile phone, bus, tv ▪ Can be used to measure traits and abilities with items that have unordered response categories. ▪ It can identify the empirical ordering of response categories where that ordering is unknown but of interest ▪ Can be used to check whether the expected ordering of response categories is supported in data.

Answer 52

Discrete variables * A discrete variable is one that has specific values and cannot have values between these specific values. (e.g. school grades, gender, number of mistakes) Continuous variables * There is in theory an infinite number of values between any two values that a continuous variable can take. (e.g. size of screws, height, age)

Answer 53

An index is formed by summing up several single indicators. Example: Press Freedom Index by Reporters Without Borders. − The annual ranking list assesses the situation of press freedom in almost 180 countries worldwide. Basis/Foundation for the ranking: An extensive questionnaire Different indicators (pressure, corruption), moderators, impact factors A ***questionnaire*** is the term used to describe the set of questions you are asking an individual. A ***survey*** is the process of collecting, analysing and interpreting data from many individuals. It aims to determine insights about a group of people.

Answer 54

Based on the questionnaire, each country is assigned a score for the following categories: − Plurality of opinion − Independent media − Journalistic work environment and self-censorship − Legal framework − Institutional transparency − Production infrastructure Do we have categories for our GMF research? In addition to these, there comes a more important factor – violent assaults upon journalists. Total score between 0 = total press freedom and 100 = no press freedom

Answer 55

Indices represent an area of variables, i.e. in an index there are already several items summed up mathematically: Indices reduce the area of variables. The specific description of individual indicators is lost. * The index is the sum of all measured variables and hence it is itself a new variable. * The index needs to cover all aspects of the variables being studied, and it should do so in a straightforward way, without getting too complicated or including unnecessary elements. * The index should be able to measure everything important about the topic in a clear and simple manner. * The measurements that enter an index must all have the same measuring level, e.g. nominal, dichotomous scales.

Answer 56

* An **index is like a summary score that combines lots of different things into one number, giving you a big picture**. * For example: an index of happiness might include factors like income, health, and social connections, all rolled into one score. * **A scale is like a ruler with different points to measure one specific thing**. * It is more detailed and focused. * A scale measuring happiness might ask you to rate how happy you feel on a scale from 1 to 10, giving you a precise measure of your happiness level

Answer 57

Attention: Different usage of the term “scale“. * Scale level: Diversity of measurements * Scale: Summation of several individual measurements into one total value. * In this case it is used similarly to the term index. The difference might be more about the specific terminology used rather than a fundamental conceptual distinction

Answer 58

▪ Likert scale: ▪ Consists of several at least 5-stage items (interval scaled), which are combined to an index by summation. ▪ Semantic differential: ▪ Consists normally of about 10 to 20 opposing pairs. Typical usage of a Likert scale with the aim to empirically determine the meaning of a concept. Multi-point rating option

Answer 59

Catch any problems with it before you conduct it Problems could be: * questions that don’t make sense * technical glitches * potential sources of bias ➢ Pre-test are worth their time: better data quality, less drop-outs and less effort during the analysis and interpretation. ➢ Even under time pressure a pre-test should not be missed out on.

Answer 60

* Make revisions based on information from pre-testing * Edit questions and response categories if needed * Simplify language/ be more explicit if needed * In case of major revisions consult with survey development experts and conduct a retesting of the survey

Answer 61

**Comprehension** * Do respondents understand the aim of the survey and the wording of the introduction and questions? Logic and flow * Do items in the questionnaire follow a logical order? * Is anything out of place or confusing? * Do question order set up conditions for bias? **Acceptability** * Do questions cause offense or touch on sensitive subjects in an inappropriate way? **Length and adherence** * Does it take a lot of effort to fill in the survey? * Can respondents make it through to the end without losing interest and focus? **Technical quality** * Does the survey platform operates smoothly? Introduction and gaining consent * Do we have an introduction declaring aim and scope of the research clearly? Can respondents give informed consent?

Answer 62

* Find 5-10 people from your target group to pre-test your survey * Try to get a range of different people who are representative of your target group * Ask them to complete the survey while thinking out loud (not down if they understand or do not understand a question) * The person should comment on anything that crosses their mind (method of loud thinking); * Let them tell you if something is unclear or ambiguous. * Take notes of every comment made – do not comment or justify * If you are uncertain whether a question was understood correctly, you can ask your tester directly what they understood. * In case your testers were satisfied with everything at first contact, then you haven't sufficiently asked them for criticism. * Make improvements based on the results

Answer 63

* Once you have incorporated all your changes, you should repeat the pre-test * You need to recruit different people, since the original testers know how to understand your questions * All questionnaires need a couple of alterations. * Usually, 2-4 pre-test rounds are needed, to turn a good questionnaire into a really good one. Conduct a full pilot before starting actual data collection You need to test all the survey steps from start to finish with a reasonably large sample * The size of the pilot sample depends on: * how big your actual sample is * how many data collectors you have

Answer 64

* Uncovering Problems: survey testing can identify all types of problems, from intimidating survey length to ambiguously worded questions, poor design to flawed survey logic * Pretesting helps avoid the time-consuming process of scrapping a flawed survey * Improving Reputation: if you release a flawed survey, it can reflect poorly on your research and you might also waste the time and effort of respondents. * Research Problem: if your survey is flawed, you will release inaccurate conclusions based on data accumulated through a flawed survey and your research will be unreliable Testing will improve your decision making * Inaccurate data can lead to make poor decisions based on your research that ends up with incorrect conclusions. * Testing a survey to be sure respondents understand it will result in more accurate responses when the final version is sent out, improving decision making. Safety and Compliance * Testing your survey before releasing it, we can catch technical glitches and security issues and fix them before any damage is done. Improving Productivity

Answer 65

* Modern sample survey started in the mid-1930s * Scholar started working on pretesting from 1940 when pretests were well established. * “The American Institute of Public Opinion pretested their questions to avoid phrasings which will be unintelligible to the public and to avoid issues unknown to the man on the street” * It usually takes about 12–25 cases to reveal major difficulties and weaknesses in a questionnaire

Answer 66

* Questionnaire problems will be signaled either by the answers that the questions elicit (e.g., “don’t knows” or refusals)… * … Or by some other visible consequence of asking the questions (e.g., hesitation or discomfort in responding) * Moser and Kalton (1971, p. 50) judged, “Almost the most useful evidence of all on the adequacy of a questionnaire is the individual fieldworker’s [i.e., interviewer’s] report on how the interviews went, what difficulties were encountered, what alterations should be made, and so forth.” * Sudman and Bradburn’s (1982, p. 49) add: “A careful pilot test conducted by sensitive interviewers is the most direct way of discovering these problem words”

Answer 67

* Shift in the goals of testing: from an exclusive focus on identifying and fixing overt problems experienced by interviewers and respondents to a broader concern for improving data quality so that measurements meet a survey’s objectives. * New testing methods have been developed or adapted from other uses ➢ cognitive interviews ➢ behavior coding ➢ response latency ➢ vignette analysis ➢ formal respondent debriefings ➢ Experiments ➢ and statistical modeling The development of these methods raises issues of how they might best be used in combination, as well as whether they in fact lead to improvements in survey measurement.

Answer 68

Pilot Testing * Involves formally testing your complete, structured survey with a small sample of respondents. * Rather than ask evaluation questions, you give respondents the survey as-is and ask them to complete it. * Pilot testing your survey provides a sense of the kind of responses you will receive and any issues that may arise during the real survey period. Data Analysis * It can help to complete some data analysis. * This involves looking at patterns in responses to see where confusion, hesitation, disengagement, or drop-out has occurred. * You can often be discovered by identifying straight-lining (the same answer is always checked regardless of the question), unanswered questions, and inconsistent or unrealistic responses. Read more about these analysis techniques in our Data.

Answer 69

* Cognitive interviewing can get to the bottom of what is going on in your respondent’s mind. * These interviews are conducted in person with a small sample of respondents (generally about 5-15). As they answer each survey question, a facilitator asks them to think aloud, describing their thought processes, emotional responses, and understanding of what each question means. * These interviews can help determine if a question is ambiguous, confusing, or makes people uncomfortable due to its content. * They can also spot drop-out risks by identifying if respondents are getting bored or agitated by the survey. Ordinary interviews focus on producing codable responses to the questions. Cognitive interviews aim to show how people think when they answer questions. * Concurrent or retrospective “think-alouds” and/or probes are used to produce reports of the thoughts respondents have either as they answer the survey questions or immediately after. The objective is to reveal the thought processes involved in interpreting a question and arriving at an answer. These thoughts are then analyzed to diagnose problems with the * Research on cognitive interviews has begun to reveal how the methods used to conduct the interviews shape the data produced.

Answer 70

Behavior coding * Involves monitoring interviews or reviewing taped interviews (or transcripts) for a subset of the interviewer’s and respondent’s verbal behavior in the question asking and answering interaction. * Questions marked by high frequencies of certain behaviors (e.g., the interviewer did not read the question verbatim or the respondent requested clarification) might need repair. * Measurement of “response latency,” the time it takes a respondent to answer a question. Since most questions are answered rapidly, latency measurement requires the kind of precision (to fractions of a second) that is almost impossible without computers. * Draisma and Dijkstra (2004) reasoned that longer delays signal respondent uncertainty, and they tested this idea by comparing the latency of accurate and inaccurate answers (with accuracy determined by information from another source).

Answer 71

Hypothetical scenarios that respondents evaluate. They may be incorporated in either undeclared or participating pretests. Vignette analysis appears well suited to 1. explore how people think about concepts 2. test whether respondents’ interpretations of concepts are consistent with those that are intended 3. analyze the dimensionality of concepts 4. diagnose other question wording problems. Martin offers evidence of vignette analysis’s validity by drawing on evaluations of questionnaire changes made on the basis of the method.

Answer 72

* It may help to have an expert opinion, especially if your survey is asking questions about a complex subject matter. * Topic experts can tap into their deep well of knowledge to evaluate your questions in order to help shape or reshape the content. * Survey methodologists may also be called upon to help determine the best ways to collect accurate data for research questions.

Answer 73

A focus group is a semi-structured discussion with a small group of people, usually around ten or so, led by a moderator. The members of the group can be very helpful in forming questions and addressing some of the following issues: * Subject matter relevancy to the group * Characteristics of the target population * Understanding of questions and concepts * Difficulty of the survey and abilities of the group * General group reaction to the survey wording and design Focus groups can be very helpful but there is the danger of “group speak,” wherein problems can be built up and exaggerated as group members “feed off” one another during discussions. Focus groups should never be the sole means of survey testing

Answer 74

Focus group is a semi-structured discussion with the purpose of stimulating conversation around a specific topic. It is led by a facilitator who poses questions, and the participants give their thoughts and opinions. It creates the possibility to cross check one individual’s opinion with other opinions gathered. A well organized and facilitated Focus Group Discussion is more than a question and answer session. In a group situation, members tend to be more open and the dynamics within the group and interaction can enrich the quality and quantity of information needed

Answer 75

* Define Objectives Clearly outline the goals and objectives of the focus group. Participant Selection Identify and recruit participants who represent your target audience or key stakeholders. Ensure diversity in demographics and perspectives Create a Moderator's Guide Develop a structured moderator's guide with openended questions to guide the discussion. This helps maintain consistency while allowing for flexibility to explore unexpected insights. * Create Open-Ended Questions Develop a set of open-ended questions that encourage participants to share their thoughts, experiences, and opinions. Avoid leading questions that may bias responses, and allow for flexibility to explore unexpected insights that may arise during the discussion. Sequence Questions Logically Organize the questions in a logical sequence that flows naturally, gradually moving from general and introductory topics to more specific and in-depth areas. This helps build rapport with participants and ensures a smooth and focused discussion. * Include Probes and Follow-Up Prompts Incorporate probes and follow-up prompts to delve deeper into participants' responses. Can be designed to produce more detailed information, clarification, or diverse viewpoints. Probes help maintain the conversational flow and encourage participants to express themselves fully. Consider Group Dynamics Account for group dynamics in the moderator's guide. Include strategies for managing potential challenges, such as dominating participants or a lack of engagement. Design questions that encourage all participants to contribute and ensure that the guide allows for flexibility in addressing unexpected topics that may arise during the discussion. * Choose a Suitable Venue Select a comfortable and neutral venue conducive to open conversation. Ensure that the space allows for audio recording and video observation, if necessary. Moderator Skills Appoint a skilled and impartial moderator who can facilitate the discussion, keep participants engaged, and guide the conversation toward the research objectives. The moderator should be able to manage group dynamics effectively. * Establish Ground Rules Set clear ground rules at the beginning of the session to create a respectful and inclusive environment. Emphasize the importance of confidentiality active listening avoiding dominant behavior. Limited Group Size Keep the focus group size manageable, typically ranging from 6 to 12 participants. This ensures everyone has the opportunity to contribute, and the discussion remains focused. * Recording and Documentation Record the focus group session (with participant consent) for accurate analysis. Additionally, take detailed notes during the discussion to capture non-verbal cues, participant reactions, and key themes. Data Analysis Employ qualitative analysis methods to extract meaningful insights from the data. This may involve coding, categorizing responses, and identifying patterns or recurring themes. Triangulation and Validation Enhance the credibility of your findings by triangulating the results with other sources of data or by conducting multiple focus groups. This helps validate the reliability and consistency of the gathered insights.

Answer 76

Desirability Bias * Respondents often over-report desirable and underreport undesirable behaviours to portray themselves positively. * The key to avoiding it * Minimise the aspects of answering that may seem threatening to respondents * Increase the more salient features that influence comfort within the respondent to reveal private or sensitive information. * This can be done by increasing privacy and anonymity, for instance.

Answer 77

Bias * Interviewers have a significant influence on survey reports. * An interviewers’ race, gender, and expectations can all have an impact on responses. * An interviewer’s conduct can also impact data quality, especially when it strays from the standardised procedures. * This leaves room for measurement error caused by interviewer-respondent interactions which can result in biased responses and a decreased reliability of answers. * The mere presence of interviewers influences survey respondents and their answers.

Answer 78

Standardized interviewer behavior is necessary for reasons of reliability: * Survey methodology works from a stimulus-response model of interaction * Every respondent should be presented with the same verbal input * Interviewers should read aloud each question exactly as worded, preserve the order of the questions, and probe in a non-leading manner * Consistency ensures that differences in responses can be related to respondent characteristics, rather than to characteristics of the interviewing procedure * A change in the formulation or the order of questions and response options can have an effect on the responses * It is crucial that interviewers do not alter anything in the questionnaire Interviewers need to follow the rules of standardized interviewing: * Read the questions exactly as worded * If the respondent’s answer is incomplete or inadequate, probe for clarification or elaboration in a non-directive way * Record the answers without interviewer’s discretion. * Do not provide any positive or negative feedback regarding the specific content of respondent’s answers Several studies show that interviewers often do not act according to these rules Reasons: Question may be ambiguous or difficult to read aloud * Can be solved in pre-testing Respondents may not understand what they are expected to do * They often do not act their roles as prescribed * Interviewers need to explain clearly to respondents the rules of the game

Answer 79

* Respondents can provide non satisfactory answers and interviewers must probe for more adequate or more complete ones * Do this in a non-directive way. Interviewers’ probes should not increase the chance of one answer over another * Each probe that can be answered by “yes” or “no” is directive We need to establish rules that interviewers should follow when probing answers: * When respondents miss part of the question, interviewers should reread the entire question * When respondents do not understand a term or concept in the question because a term turns out to be ambiguous, interviewers should not provide an explanation. They can reread the definition, if the questionnaire provides one. * Otherwise, respondents must be told to interpret the question to their own liking and answer the question. * When probing answers, interviewers should make sure that answers meet the questions’ objectives.

Answer 80

* With interviewer-administered surveys, the interviewer provides feedback to the respondent * If an invalid response is provided, the interviewer should repeat the question or response options in order to get an accurate response * When a respondent asks for clarification, the question should be restated using the same words * When interviewers try to clarify the question on the fly, these explanations can change the meaning of the question. (This problem can be avoided by careful development of the questionnaire and pretesting the instrument.) * Recording answers accurately and completely is important. When using a paper questionnaire, the answers must be readable

Answer 81

* Introduction to the survey * Title of the study * Purpose * Duration of survey and length of questionnaire * Guarantee of confidentiality * Brief information about the company or organization (IMS)

Answer 82

Introductory script and other support materials must be prepared before implementing the survey * The introductory script is used by the interviewer to schedule the * Date * Time * Location for the interview * Interviewers have about 20–30 seconds to obtain a respondent's agreement to participate; * The scripts should be as brief as possible while also communicating critical information about the purpose and importance of the survey.

Answer 83

* An input mask makes it much easier for users to figure out the required format for filling out the form * Input masks are a way to constrain data that users enter into form fields and enforce specific formatting * Use Google Forms to design your survey

Answer 84

* User must enter a digit (0 to 9). * User can enter a digit (0 to 9). * User must enter a letter. * User can enter a letter. * User must enter a letter or a digit. * User can enter a letter or a digit. * User must enter either a character or a space. * User can enter characters or spaces. * Decimal and thousands placeholders, date and time separators. The character you select depends on your regional settings (in Germany commas and full stops).

Answer 85

We have to provide information about certain characteristic features of statistical populations. Two ways Census: ▪ Complete count: It is a study of every unit, everyone or everything, in a population Partial sampling (sub-samples): ▪ A sample, which is a subset of units in a population ▪ Selected to represent all units in a population of interest. ▪ Information from the sampled units is used to estimate the characteristics for the entire population of interest.

Answer 86

1. Cost saving ▪ Collecting smaller amounts of elements is cheaper than conducting a census (complete enumeration). 2. Time saving ▪ Data collection in the case of sampling takes less time than a census. 3. A census or complete enumeration is practically impossible ▪ It is theoretically imaginable, but practically not reasonable or possible.

Answer 87

* To achieve a certain level of confidence and accuracy a sample size is required to be at least a certain size * To calculate the required sample size we need to use these factors: population size, parameter variation, degree of accuracy, and degree of confidence * Our sample mean should be close to the population mean, the sample inter-quartile range to be close to the population interquartile range, and so on

Answer 88

Parameter Variation: * Is the process of changing the input parameters (for example: height, age, etc.) of a mathematical model or simulation to investigate the effect of each parameter on the output. * We want to determine which input parameters have the most significant impact on the output and how the output responds to changes in these parameters. Confidence Level * Is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way

Answer 89

* Sample accuracy refers to the extent to which sample statistics correctly estimate the population parameter. * We typically use the terms biased and unbiased to describe the accuracy of sample statistics. Example: we take many thousands of samples from the same population. For each sample, we calculate a statistic (e.g., the mean). ➢ Unbiased sample: If the average of the sample statistics (e.g., sample means) equals the population parameter (e.g., population mean) then we refer to the statistic as being unbiased. ➢ Biased sample: If the average of the sample statistics (e.g., sample means) does not equal the population parameter (e.g., population mean) then we refer to the statistic as being biased

Answer 90

▪ In arbitrary sampling, the researcher selects individuals or units of study because they are available, convenient and represent some characteristic the researcher seeks to study without following any specific system, e.g. street surveys. ▪ Representativity (the sample is a reduced, structurally similar representation of the population under study) is problematic.

Answer 91

* Every case in the population has a known, greater-than-zero probability of being selected for the sample. For example quantified summaries of characteristics of the sample, like the median of a variable or the correlation between two variables. * With probability sampling design, we can use statistics to estimate the parameters (the corresponding quantified characteristics of the population) with known levels of confidence and accuracy.

Answer 92

Reliable selection of individuals or units of study from the population. ▪ The intent of random (or probability) sampling is to choose individuals or units for the sample so that any unit has an equal probability of being selected from the population. ▪ The more often one takes random samples from the same population, the closer one will get to the actual value in this population. ▪ No pre-specific characteristics ▪ The empirical social research accepts in general a 5 percent probability of error. The scores it provides always contain this sampling error. Example: lottery procedure, i.e. the random extraction of elements from a limited population. Common selection or sampling procedures: ▪Choose every n-th element (or individual) in the population (e.g. selecting the file with the number 100 from a student file system) The criterion according to which a systematic random sampling is being applied needs to be distributed independently from the features that are studied in the population.

Answer 93

Cluster sampling is a spatiotemporal (time and space) defined conglomeration of elements of the population which form a structurally reduced representation of the respective population. Typical clusters: households, school classes, apartment buildings. Elements included: persons, pupils, households. Procedure: you choose by random sampling a number of clusters and analyse all units of study that occur in these clusters

Answer 94

Most basic kind of probability sampling design Each case in the population has a known and equal probability of being selected for the sample. Goal: construct a sample that is like the population so that we can use what we learn about the sample to generalize to the population. Example: If you want a sample of 50,000 graduates using age range, the proportionate stratified random sample will be obtained using a formula: (sample size/population size) × stratum size. A stratum (plural strata) refers to a subset (part) of the population (entire collection of items under consideration) which is being sampled. The table below assumes a population size of 180,000 MBA graduates per year. Calculation: The strata sample size for MBA graduates in the age range of 24 to 28 years old is calculated as (50,000/180,000) × 90,000 = 25,000. Same method is used for the other age-range groups. 25,000 graduates from the 24–28 age group will be selected randomly from the entire population 16,667 graduates from the 29–33 age range will be selected from the population randomly, and so on

Answer 95

Sampling Design Sampling frame: list of cases we select our sample from— ideally, a list of all cases in the population Select cases for sample through a sampling design. Two basic varieties of sampling designs: ➢Probability sampling designs and nonprobability sampling designs.

Answer 96

In nonprobability or purposeful sampling, you select individuals or units of study according to how “useful” or essential their analysis is for finding an answer to the research problem. ▪ These sampling procedures can be problematic with regard to representativity. − Four procedures: 1. Sampling typical cases 2. Sampling extreme groups 3. Sampling according to concentration principle 4. and quota sampling

Answer 97

▪ Especially appropriate in qualitative research methods, where only few individual cases are analysed. ▪ The sample consists of units of study (e.g. autistic people) who are especially characteristic for all units of study in the population

Answer 98

Sampling according to the Concentration Principle ▪ In a sampling procedure, the researcher focuses on the part of the population where they suspect the predominant part of these elements to be. − Example: Investigation about German skiers, −95% of all German skiers live in Bavaria – using a random sample only from the Bavarian population Cut-Off-Procedure = The less productive or rich part of the study population is being cut off.

Answer 99

In quota sampling, you select individuals or units of study according to some fixed quota. ▪ Units of study are selected based on pre-specified characteristics so that the total sample has the same distribution of characteristics assumed to exist in the population that is studied. ▪ Quotas generally rely on demographic characteristics. Example: the sample displays, according to its quota-based characteristics, a structurally identical representation of the studied population

Answer 100

Sample drop-out refers to all cases where an element of a sample could not be analysed. We distinguish between random and systematic deniers. Random drop-out: Relocation, illness… Systematic drop-out: Persons who deliberately refuse to take part in the survey e.g. highly educated people, people with low language knowledge etc.

Answer 101

Sampling variation is how much the results change when you take different samples from a group. Because of this, there is always some uncertainty when you try to make conclusions about the whole group based on those samples. Sampling variation is about how much a number you find might change when you look at different samples of things. (e.g there is no blue person in the sample) ➢ If you measure something in a few different samples, the result might be a little different each time. This means there is some uncertainty when you try to say something about the whole big group based on just those smaller ones. Sampling variability is how much an estimate varies between samples. ➢ Variability between samples indicates the range of values differs between samples. ➢ If you are measuring something in different samples, you might notice that the numbers you get can be quite different from one group to another. ➢ That is what we call sampling variability—it is like the range of values you see across those different groups.

Answer 102

How can we know how much something varies? * Assume maximum variation which places the highest demand on sample size * Assuming maximum variation in a sample size means imagining that each sample you pick is as different as possible from the others. * Imagine every sample represents the extreme ends of whatever you are studying. This helps ensure that your sample size is big enough to cover all the possible variations in the population you're interested in. It is a way to make sure you are not missing out on any important differences that might affect your results

Answer 103

Variation is a way to show how data is dispersed or spread out. * To specify variation in a sample size, you typically consider factors such as * the variability of the population you are studying the level of precision you need in your estimates any constraints on resources like time and budget. Understand the Population Variation: * Consider the range and variability of the characteristics you are interested in within the entire population. This helps you gauge how much variation might exist in your samples. Define Precision Level * Determine the level of precision you need in your estimates. This depends on factors like the desired confidence level and margin of error. Choose a Confidence Level: * The choice of confidence level depends on how confident you need to be in your results, the potential consequences of being wrong, and any industry standards you may need to adhere to. For most purposes, 95% is a safe and commonly accepted level * Decide on the confidence level you want for your estimates. Common choices are 95% or 99%.

Answer 104

* Take into account the population variation, desired precision, and confidence level. * Take into account any practical constraints such as budget, time, or feasibility of collecting data. Sometimes you may need to balance statistical considerations with these constraints. * If necessary, adjust the sample size based on factors like the complexity of the analysis, potential non-response rates, or the need for subgroup analysis. * Validate Sample Size: After collecting data, validate whether your sample size was sufficient to provide reliable estimates. You can do this by checking if the confidence intervals are narrow enough to meet your precision requirements.

Answer 105

A closely related term (almost a synonym) is sampling error. An error in sampling isn’t a mistake — it’s a measure of how much a value differs from the “true” value. Example: − Let’s say the true weight of a population is 150 lbs. − You take a sample and find the mean weight for the sample is 151 lbs. − The 1 lb difference is an “error.” − If you sample again, you might get different mean weights of 148 lbs, or 150.5 lbs, or 153 lbs. − The different errors — 1/2 lb, 1 lb, 2 lbs, 3 lbs — are a reflection of the variability between your samples, or sampling variability.

Answer 106

* Increasing or decreasing sample sizes leads to changes in the variability of samples. * For example: a sample size of 10 people taken from the same population of 1,000 will very likely give you a very different result than a sample size of 100. * There is no “perfect” sample size that will give you accurate estimates for the sample mean, variance and other statistics. Instead, you take your best “guess” — using standardized statistical procedures. In general, estimates will change from sample to sample and will probably never exactly match the population parameter.

Answer 107

* Hypothesis testing is one of the most important processes for measuring the validity and reliability * It helps to provide links to the underlying theory and specific research questions * Verifying a hypothesis provides a reliable framework for making any data decisions for our population of interest * It helps to successfully extrapolate data from the sample to a larger population * Verifying a hypothesis allows us to determine whether the data from the sample is statistically significant

Answer 108

A hypothesis is a proposed explanation for a phenomenon. The term hypothesis is a statement about something that is supposed to be true. The logic of a hypothesis test is to compare two statistical data sets A hypothesis test involves two hypothesis: * the null hypothesis * and the alternative hypothesis

Answer 109

* Face-to-face surveys follow a standardized script without deviation * Face-to-face surveys also offer advantages in terms of data quality. * A face-to-face survey allows researchers a high degree of control over the data collection process and environment. Interviewers can ensure that respondents do not skip ahead * Responds rate of face-to-face survey is higher * There is also evidence that questions of a personal nature are less likely to be answered fully and honestly in a face-to-face survey

Answer 110

* Capture verbal and non-verbal cues (discomfort or enthusiasm) * Helps keep respondents focused * A surveyor has control over the interview, they can help ensure the interviewee remains focused and on track to complete their survey * Can allow for more complex questions as interviewer is present to clarify wording and probe for more information * Can generally be longer than a phone interview * Arranged to be convenient for the participant to take part in * Can access samples that phone and Internet questionnaires might not be able to reach * Ability to incorporate visual stimuli into the questionnaire * Generally have a higher response rate than other methods

Answer 111

* Interviewer is exposed to the potentially biasing effect of the respondent’s appearance and environment in addition to their voice * Interviewer may inadvertently give respondents nonverbal as well as verbal cues about how they should respond

Answer 112

The Team composition and the number of interviewers per team need to be decided based on a number of factors: * the expected duration of interviews * the content of the survey * the size of clusters, etc. On the average each team will need * one supervisor * one runner/editor * three to five interviewers (depending on how many people an interviewer can complete)

Answer 113

* Identify the clusters to be surveyed * Supervise interviewers as they perform the survey * Ensure that the interviewers follow instructions * Answer interviewers’ questions as they arise * Control data quality by checking for errors during interviewing, checking that forms are completed fully and correctly, and checking that all respondents are answering the questions * Identify problems and retrain interviewers who are doing their job incorrectly.

Answer 114

* Monitoring interviewer performance by: * Observing several interviews, especially during the early stages of fieldwork * Conducting regular review sessions with interviewers (if needed) * Compiling completed questionnaires from a cluster and packing them up

Answer 115

* Identify the specific households to be surveyed * Gain the consent of respondents to be interviewed * Conduct interviews using the standard questionnaire * Maintain standard procedures in conducting the interviews and recording the answers.

Answer 116

Missing Data * Open Ended Questions: * For these questions, you may need to categorize the write-in responses to create a consistent coding scheme. * Sometimes participants did not answer all the survey questions due to any reason * This partial filled data can create some noise in pure data analysis. It is advised to remove the partial completed responses from the overall data before starting analysis. * Check Spelling - Merge Doubling Skipped Questions * Identify if any questions were consistently skipped by a large number of respondents. * This might indicate a problem with the question itself or the survey flow.

Answer 117

* Age (Q2) and Employment Status (Q5): There might be inconsistencies (e.g., someone selecting "Student" under employment but an age outside the typical student range). * You may need to decide how to handle these discrepancies (e.g., contacting respondents for clarification, excluding outliers). Response Options * "Prefer not to say" (Q1): Decide how to handle this option. * Similar to "Prefer not to say", decide how to categorize this for respondents who might not work in media (e.g., separate category).

Answer 118

Standardization * Ensure consistent formatting for written responses (e.g., all uppercase or lowercase). Coding * Create numerical codes for answer choices in multiple-choice questions to simplify analysis. Outlier Detection * Identify and potentially exclude extreme values that might skew the results (consider reviewing these cases individually).

Answer 119

* When a respondent’s answer contradicts their response to another question, it’s clear that they’re either being dishonest or careless (or even both!). * You may be able to find these inconsistencies by applying multiple filters. * For instance, say that one of your survey questions asked respondents how much TV they watch per week. When the responses come back, you filter by those who said they watch at least a little. Document Decisions * Keep a log of the decisions made during data cleaning to ensure transparency and replicability. Remember, data cleaning is an iterative process. As you explore the data, you may identify new cleaning needs

Answer 120

* Researchers needs to be cautious about straightline respondents and responses must be removed before analysis. * Straightlining is when respondents choose similar answer option frequently (such as first/last option etc.). * There might be higher possibility that the respondent has not responded the answers honestly. * Imagine asking respondents how much TV they watch per week, on average. If a respondent writes in 165 hours, they’re likely exaggerating (Hint: there are only 168 hours in a week). * Having a response like: “Fdsklj” might make you smile, but it isn’t going to get you far in your analysis.

Answer 121

* Sometimes you have encountered responses falling under unrealistic range for example “In which sport you see yourself as pro” and one respondent selected all the sports from option. * This kind of responses are called outliers. * This becomes tricky sometimes but researchers should always be beware of fake or manipulated responses from participants. * There are many ways to check fake responses such as, using open ended questions — check is any response contain unreadable or meaningless response like ‘fgfgfh’ type text etc.

Answer 122

* Should be inductive, exploratory * Is framed as a question, aim or objective but not as a hypothesis * Should focus on a single phenomenon, concept or idea * Start with a verb and stating your goal (e.g. characterize, understand etc.) * Identify your topic of interest * Use a language that is non-directional and neutral to be exploratory * Define your sample and your setting * Example: Understand interpersonal factors relevant to relationships formed online by retirees - in private homes in Bavaria

Answer 123

1.Look at the results of your survey as a whole 2.Take a look at the demographics of those who responded 3.Compare responses to different questions to find deviations 4.Find connections between specific data points with layered data 5.Compare new data with past data

Answer 124

− To what extent do media practitioners perceive AI tools in improving productivity, credibility, and creativity? * Filter results by cross-tabulating subgroups * Filter your results based on specific types of respondents, or subgroups * Look at different attendees (age, gender) * What are the most common responses to a specific question? * Which responses are affecting/impacting us the most? * What did respondents with a particular job title say? * Which group of respondents are most affected by a specific issue? * Have conference visitors noticed something outstanding? * What do people say about a specific topic? 4. Find connections between specific data points with layered data − Understand connections − Look for causation and correlation − Look for confounding variables 5. Compare new data with old data − Check for changes and try to find explanations for them

Answer 125

Meaning unit: It is the smallest unit of analysis that captures a single idea or concept in the data. Condensed meaning unit: It is a shortened version of the meaning unit that retains the essential information. Code: It is a label assigned to condensed meaning units to categorize and organize the data. Category: It is a group of similar codes that represent a broader concept or idea. Theme: It is a higher-level abstraction that emerges from grouping or categorizing related codes and categories.