Empirical methods exam Flashcards
Induction and Deduction
Data collection in the field of empiricism
▪ From these data researchers extract general
statements (theory) by induction.
▪ From theory, in turn, statements about single
cases can be derived by using deduction.
Finding a Research Question - Inductive
Finding a Research Question - Deductive + I & D reasoning
Inductive reasoning
Specific observation -> Pattern recognition -> General conclusion
What is Research Design?
Research Criteria
Correlational Research
The goal of correlational research is to determine whether two or
more variables are related.
Experimental Design
Experimental research involves comparing two groups on one outcome
measure to test some hypothesis regarding causation
Types of experimental Research Designs
Cross sectional
Descriptive Statistics: Graphic representation of data
Descriptive statistics is the idea of quantitatively describing data and you can do that through various means. For example, through visualization techniques like: * graphical representation * tabular representation * summary statistics The idea here is that you crunch the data, you work with the data and come up with (1 or 2 or 3 or 4) different numbers that summarized the data for you.
Bar graph
There are dozens of charts and graphs you can make from data. * Which one you choose depends on what kind of data you have and what you want to display. * If you wanted to display relationships between data in categories, you could make a bar graph
Histograms
Pie chart
shows how categories in your data relate to the whole set
Scatter plots
Scatter plots are a good way to display data points. * It shows the relationship between two variables * The position of each dot on the horizontal and vertical axis indicates values for an individual data point * The dots in a scatter plot report patterns when the data are taken as a whole.
Why are descriptive statistics important?
Basic Concepts of Descriptive Statistics
Measures of central tendency
A measure of central tendency (also referred to as measures of centre or central location): * Is a summary measure that attempts to describe a whole set of data with a single value that represents the middle or centre of its distribution. There are three main measures of central tendency: −the mode, the median and the mean. Each of these measures describes a different indication of the typical or central value in the distribution.
Mean
Median
Mode
Shapes of Distributions - Peaks
Graphs often display peaks, or local maximums. It can be seen from the graph that the data count is visibly higher in certain sections of the graph. 1. one clear peak is called a unimodal distribution. 2. two clear peaks are called a bimodal distribution. Here, the term “mode” is used to describe a local maximum in a chart (such as the midpoint of a peak interval in a histogram). It does not necessarily refer to the most frequently appearing score, as in the “central tendency mode”. * Single peak at the center is called bell shaped distribution. Note: A bell shaped graph (bell curve), is a frequency distribution that resembles the outline of a bell when plotted on a graph
What is a Distribution?
The statistical distribution shows which values are common and uncommon in your data * A statistical distribution, or probability distribution, describes how values are distributed for a field * There are many kinds of statistical distributions, including the bell-shaped normal distribution. * We use a statistical distribution to determine how likely a particular value is.
Shapes of Distributions
When graphed, the data in a set is arranged to show how the points are distributed throughout the set. * These distributions show the spread (dispersion, variability, scatter) of the data. * The spread may be stretched (covering a wider range) or squeezed (covering a narrower range). * The shape of a distribution is described by its number of peaks and by its possession of symmetry, its tendency to skew, or its uniformity. * Distributions that are skewed have more points plotted on one side of the graph than on the other.
Shapes of Distribution - Skewness
Skewness is a measure of a lack of symmetry in a distribution. * A standard normal distribution is perfectly symmetrical and has zero skew
Measure of Variability
Variability describes how far apart data points lie from each other and from the center of a distribution. Range: the difference between the highest and lowest values. Interquartile range: the range of the middle half of a distribution. Standard deviation: average distance from the mean. Variance: average of squared distances from the mean
Measure of Variability: Range
* The range is the difference between the highest and lowest values within a set of numbers.
* To calculate range, subtract the smallest number from the largest number in the set. * Range shows how much the numbers in a set vary
Measure of Variability: Interquartile Range
The middle fifty or midspread of a set of numbers, removes the outliers (highest and lowest numbers in a set) * If there is a large set of numbers, divide them evenly into lower and higher numbers. * Then find the median of each of these groups. * Find the interquartile range by subtracting the lower median from the higher median.
Measure of Variability: Variance
Variance measures how far a data set is spread out. It is mathematically defined as the average of the squared differences from the mean. * Finding the mean (the average). * Subtracting the mean from each number in the data set and then squaring the result. * The results are squared to make the negatives positive. Otherwise, negative numbers would cancel out the positives in the next step. It’s the distance from the mean that’s important, not positive or negative numbers. * Averaging the squared differences.
Measure of Variability: Dispersion
Measures of dispersion help to describe the variability in data. * Dispersion is a statistical term that can be used to describe the extent to which data is scattered. * Thus, measures of dispersion are certain types of measures that are used to quantify the dispersion of data. * The variance helps to draw a comparison between the two data sets A and B on the basis of variability.
Types and Examples of Quantitative Research Questions
Descriptive
* Used to collect participants’ opinions about the variable that you want to quantify
* Descriptive research questions begin with “ how much?” “how often?” “what percentage?” “what proportion?”
* Example: How often do middle-class adults go on vacation yearly? (variable: vacation; group: middleclass adults)
Comparative
* Help you identify the difference between two or more groups based on one or more variables Example: What is the difference in the usage of TikTok between male and female Cambodian university students? (variable: usage of TikTok; group 1: male Cambodian university students; group 2: female Cambodian university students)
Relationship-based: * Used to identify trends, causal relationships, or associations between two or more variables.
Example: What is the relationship between salary and shopping habits among the women of Australia? (independent variable: salary; dependent variable: shopping habits; group: Australians)
Survey Questions - Types
Survey questions should be designed in a way that
any respondent can understand them.
Survey questions can be classified into 4 subtypes:
− 1. Fact-based questions
− 2. Knowledge-based questions
− 3. Attitudinal and opinion questions
− 4. Behavioural questions
Checklist Survey Questions
Target: to formulate every single question in such a way that no biased responses, misunderstandings or ambiguities can occur.
✓ Is the question clear for everybody?
✓ Is the issue that the question is based on precisely and comprehensively formulated?
✓ Can everyone clearly understand (and then describe) the issue in the question?
✓ Do the questions impose a overly high level of verbal skills?
✓ Is the question one-dimensionally formulated, is it clear what the respondent needs to answer?
✓ Is it possible that the question might overload the memory of the respondent?
✓Is the respondent still able to answer the question even if he is already exhausted?
✓ Does the question induce social desirability by the way it is formulated?
✓ Does the question contain suggestive wording that could lead to a certain response behaviour?
✓ Might the question be too intimate?
✓ Would it be possible for respondents to feel embarrassed by the response (and therefore give an untruthful answer)?
Function Questions
Function questions control the course of the questionnaire without bringing any contribution to the actual result interest. These questions guarantee that the survey questions are applied correctly. 1. Ice-breaker questions 2. Transfer and Resting questions 3. Filter and Funnel questions 4. Verification questions
Ice-Breaker Questions
Ice-Breaker Questions * Are used to establish a relationship with the respondent. * In order to make the interview atmosphere more relaxing, the interviewer asks in the beginning a question which is unimportant in comparison to all other questions and will not be analysed. * Example: “What is your opinion about the current TV programme?“
Transfer and Resting Questions
Transfer and Resting Questions * Are meant to distinguish subject areas in a survey in a clear manner for the survey participant without focusing on the relevance of the content. * They allow the respondent to talk freely.
Filter and Funnel Questions
- Make sure that only those people who are truly concerned answer the survey questions.
- Funnel questions sort out all the respondents who cannot or should not give an answer to the actual questions (e.g. Pay-TV users).
- Filter questions work by the same logic which differentiates uncoupling and bifurcation.
- By uncoupling, unnecessary questions are skipped.
- Bifurcations are integrated to receive information about all respondents divided into subgroups.
Verification & Credibility Questions
Verification Questions Are integrated into the questionnaire together with alternating questions.
Aim: to verify the consistency of response behavior
Credibility Questions
▪ Are meant to verify the credibility of the participants.
▪ The questions refer to issues which everybody has allegedly experienced in his life.
▪ If the respondent gives an answer which is highly unlikely, then this is a proof that the respondent might sometimes give untruthful responses.
Socio-demo-graphic Questions
Sociodemographic questions are not content-related and are usually placed at the end of the survey. The following aspects are usually asked for: * Age * Gender * Education * Head of household * Net income * Religion * Marital status
Standardization of Surveys
In-depth/Guideline Interview
Develop a guideline and define specific question sequence. Semi-structured or semi-standardized interview.
Special type of survey: group interview Conduct group interview according to a set of guidelines. In group discussion, opinions are encouraged and generated. Participants mutually influence their response behaviour. Example: Introducing new products etc. Special attention is needed with interpretation. Therefore, usually it is used as a complementary instrument.
Bias
Bias can occur when respondents want to be liked and to avoid embarrassment are very strong. This can affect how people answer questions asked by strangers. * Question wording must facilitate unambiguous, fully accurate communication * Leading questions are the most obvious culprit
Counter potential social desirability bias
Use normalizing statements:
* Example: Some people like to follow politics closely and others aren’t as interested in politics. How closely do you like to follow politics? * Closed-ended questions (questions that give answers for respondents to select from) are susceptible to response set bias.
Open-ended questions
Can give respondents freedom to answer how they choose * Remove any potential for response set bias * Allow for rich, in-depth responses if a respondent is motivated enough. * Challenge: respondents can be ambiguous; can give responses that indicate the question was misunderstood * Researcher is giving power to respondents to structure the data
Closed-ended questions: characteristics
- Answers are unambiguous * Data are easy to manage. * Challenge: researcher is structuring the data, which keeps things nice and tidy
Survey Mode
All surveys are conducted in one of the three survey modes: * Face-to-Face Interview * Written interview * Telephone Interview / Survey. Additionally, there is a version of written interviews that has established itself in recent years: the online survey.
Face-to-Face-Interviews
Face-to-Face-Interviews:
* Personal interviews require qualified interviewers.
* Result high costs relatively high time consumption Pro * Fully used sample has very low refusal and abandon rate
* Return rate (40-70 percent) is considerably higher in comparison to written, telephone and especially online surveys.
Telephone Interviews
Telephone Interviews:
Due to technical development, telephone interviews are becoming more and more popular and are extensively used as a current survey mode in professional survey research institutes. Disadvantage: * Visual auxiliary material cannot be used * Risk of overloading (seven-level scale) * Abandon rate is higher than in face-to-face interviews
Written Survey
Written Survey (Paper & Pencil) Disadvantages of surveys by conventional mail: * low return rate (partly under 10 %) * motivation of the respondents is usually low, the social contact is missing * field phase is considerably longer than in other survey modes Higher rates when: * the respondents have an additional benefit (winning competition) * the questionnaires are notified by telephone * there is a follow-up campaign. Groups of people who are hardly reachable can be approached more easily via written surveys. (e.g. politicians)
Online Survey
Online Surveys are Internet-based survey methods, which can be classified into three groups: The questionnaire … * is saved on a server and is filled out online. * can be downloaded from a server and sent back via E-Mail * is sent via E-Mail and returned the same way after filling it out. Option 1 with the possibility to directly fill out the questionnaire online enjoys the highest popularity at the moment. There are different software on the market to help create and use online survey tools. (e.g. Lime Survey, SurveyMonkey).
Survey Development: definition
Survey development is a collaborative and iterative process where researchers meet to discuss drafts of the questionnaire several times over the course of its development.
* Conducting frequent tests with new survey questions ahead of time through qualitative research methods such as focus groups, cognitive interviews, pretesting
* Choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent. It is important that all respondents interpret the question the same way.
Question Order
We must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions * Two main types of order effects for closed-ended opinion questions: * Contrast effects (where the order results in greater differences in responses) * Assimilation effects (where responses are more similar as a result of their order). How to avoid: The best way to combat these errors is randomization. You can randomize your answer options for every respondent so that e ach option has a fair chance of being picked. As for the question order bias, make sure that your questions don’t affe ct the answers to the following questions.
Example: You ask your employees about their issues with the reporting manager and then ask if they are happy in the workplace. Now, there is a possibility that they would have answered the questions differently if the order had been reversed. Their second answer will be influenced by the answer to the first question.
Closed-ended Questions
Is made up of pre-populated answer choices for the respondent to choose from * Multiple choice * Drop down * Checkbox * Ranking question * Questions that are closed-ended are conclusive in nature as they are designed to create data that is easily quantifiable * Closed-ended questions allows researchers to categorize respondents into groups based on the options they have selected (allows for demographic studies) * Major drawback to closed-ended questions is that a researcher must already have a clear understanding of the topic of their questions and how they tie into the overall research problem before they are created.
Likert Scale Questions
Always a close-ended question * A survey scale represents a set of answer options—either numeric or verbal—that cover a range of opinions on a topic. * Will let us uncover degrees of opinion that could make a real difference in understanding the feedback you are getting * Are structured to provide quantifiable answer options that make analyzing data easier * Each series of questions in the survey focused around the same topic
What is a Likert Scale?
The Likert scale is a five (or seven) point scale that is used to allow an individual to express how much they agree or disagree with a particular statement. Simple questions: the less thinking required from respondents the better the response rate. Example: How satisfied were you with your in-store experience? It was easy to navigate the website to find what I was looking for.
Writing a Likert Scale Question
Accuracy: likert-type questions must be phrased correctly in order to avoid confusion and increase their effectiveness * Descriptive words need to be easily understood and should not be ambiguous * There should be no confusion about which grade is higher or bigger than the next * Better ask a question than let people rate a statement
Single answer (closed): example
What is your marital status? Are you … * married and living with your spouse * married and separated * widowed * divorced * unmarried
Single answer (semi-open)
What is your marital status? Are you … ▪married and living with your spouse ▪married and separated ▪widowed ▪divorced ▪unmarried ▪I have a different marital status, which is: _________________
Multiple answer (complex)
Which of the devices on the list are in your household? (multiple answers possible)
▪washing machine
▪dishwasher
▪food processor
▪espresso machine
▪none of these
Multiple answer (closed)
Which of the devices on the list are in your household? (multiple answers possible) ▪washing machine ▪dishwasher ▪food processor ▪espresso machine ▪vacuum cleaner
Ordering scale: example
Please put these 7 properties in the order that best describes your diet. please indicate the order by assigning numbers from 1 = most important to 7 = least important __ must eat meat __ must not eat meat __ give me sweets __ comes out of a wrapping __ circumnavigates food intolerances __ any hipster label here __ super duper healthy
Multiple response question
Of these 7 properties, which best describe your own diet? Chose 3 at most. (max. 3 items) ▪must eat meat ▪must not eat meat ▪give me sweets ▪comes out of a wrapping ▪circumnavigates food intolerances ▪any hipster label here ▪super duper healthy
Pitfalls When Designing a Survey
- Biased or leading questions: Phrasing questions in a way that suggests a particular answer can skew results. Ensure questions are neutral and don’t lead respondents to a specific response.
- Ambiguous questions: Vague or unclear questions can confuse respondents and lead to inaccurate responses. Questions should be precise and easy to understand.
- Overly long or complex surveys: Lengthy surveys can lead to respondent fatigue and lower completion rates. Keep surveys concise and focused on relevant topics.
- Limited response options: Providing insufficient response options can limit the range of answers and fail to capture diverse perspectives. Include a variety of response options, including open-ended questions when appropriate.
- Order bias: The order in which questions are presented can influence responses. Be mindful of question sequencing and consider randomizing or alternating question order to mitigate bias.
- Social desirability bias: Respondents may provide answers that they perceive as socially acceptable rather than their true opinions or behaviors. Use anonymous surveys and assure respondents of confidentiality to minimize this bias.
- Sampling bias: If the survey sample is not representative of the target population, results may not be generalizable. Ensure the survey sample accurately reflects the demographics and characteristics of the population of interest.
- Non-response bias: If certain groups are more likely to participate in the survey than others, results may be skewed. Employ strategies to encourage participation from underrepresented groups.
Process of Quantitative Research
- Theory
- Hypothesis
- Research design
- Devise measures of concepts
- Select research sites (s)
- Select research subjects/respondents
- Administer research instruments/collect data
- Process data
- Analyse data
- Findings/ conclusion
- Write up findings/ conclusion
Measuring and Counting
- Empirical social research is interested in conducting a systematic and intersubjectively traceable analysis of cases attributed to a phenomenon (our research question). * Quantifying any type of variable through a systemic method Definition: − Measuring is the systematic assignment of an amount of numbers or symbols to the values of a variable, hence also to the objects.
Measuring and Counting Classification
Classification occurs by looking at the value of the variables. Nominal (categorical, qualitative) data * Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. (e.g. gender) Ordinal data (rank variable) * Variables that have two or more values which can also be ordered or ranked. That means that variable values have a natural sequence. (e.g. school grades). Metric data * Values that can be organized according to size AND can represent a multiple of a unit (e.g. height)
Levels of measurement or scales of measurement –
classification principle of variables I
Levels of measurement or scales of measurement – classification principle of variables I − Nominal data / variables (e.g. hair colour/nationality) − Ordinal data (variables have natural, ordered categories: e.g. hot, hotter, hottest) − Metric data (e.g. age)
Classification principle of variables
− Discrete variables - counting numbers with
specific values (e.g. age)
− Continuous variables - measurable number e.g.
1.7 feet
Empirical and Numerical Relative
Эмпирические и численные данные
- Measuring is the process by which the empirical relative, e.g. the characteristic value “person in dress” is systematically assigned a numerical relative, e.g. the number 1. * The numerical relatives contain numerical values or scores, the empirical relatives define objects or objects of analysis * Each element of the empirical variables is assigned a numerical value in such a way that the relations in the empirical relatives correspond or equal the relations in the numerical relatives. * This rule of assigning values is also called scale.
Terms
Object of
analysis
Attribute/
Variable
Value
Coding Characteristics
Scale Level: types Classification Principle of Variables
Scales: The one-to-one transfer of empirical relatives into numerical relatives leads to the formation of scales. We distinguish between three scales of measurement. The classification is done according to the value of the variables: * Nominal variables/ data/divide into class/numbers * Ordinal variables/ data/Scal in order/ 1st, 2nd, 3rd/Low, medium, moderate * Metric variables/ data/exam score/grading system 1-4
Nominal Data - Qualitative Variables
Nominal Data: Variables whose value cannot be organized in a logical sequence and which can only be distinguished by their name. ▪ The categories must be defined precisely. ▪ The categories must mutually exclude themselves. ▪ The categories must describe the variable exhaustively.
Ordinal Data
▪ Variables that can take two or more values which can be ordered or ranked in a sequence.
▪ That means that variable values have a natural sequence.
▪ The measured values on an ordinal scale can be put into a logical sequence.
Metric Data
▪ Variable values that can be organized according to size AND can represent a multiple of a unit. ▪ Values that have been measured on a metrical scale can take any numerical value (or score). ▪ The range or the intervals between the possible measured values have to be always the same. E.g age
Nominal Data dichotomous variables
▪ Dichotomous (dividing into two parts) variables are variables which have
exactly two values, e.g yes/no
▪ Example: Do you have a mobile phone?
Nominal Data - polytomous variables
▪ Polytomous variables display more than two values. E.g divide in class, brands of mobile phone, bus, tv ▪ Can be used to measure traits and abilities with items that have unordered response categories. ▪ It can identify the empirical ordering of response categories where that ordering is unknown but of interest ▪ Can be used to check whether the expected ordering of response categories is supported in data.
Classification Principle of Variables
Discrete variables * A discrete variable is one that has specific values and cannot have values between these specific values. (e.g. school grades, gender, number of mistakes) Continuous variables * There is in theory an infinite number of values between any two values that a continuous variable can take. (e.g. size of screws, height, age)
Index: Questionnaire & Survey difference
An index is formed by summing up several single indicators. Example: Press Freedom Index by Reporters Without Borders. − The annual ranking list assesses the situation of press freedom in almost 180 countries worldwide. Basis/Foundation for the ranking: An extensive questionnaire Different indicators (pressure, corruption), moderators, impact factors
A questionnaire is the term used to describe
the set of questions you are asking an
individual.
A survey is the process of collecting,
analysing and interpreting data from many
individuals. It aims to determine insights
about a group of people.
Example - Press Freedom Index
Based on the questionnaire, each country is assigned a score for the following categories: − Plurality of opinion − Independent media − Journalistic work environment and self-censorship − Legal framework − Institutional transparency − Production infrastructure Do we have categories for our GMF research? In addition to these, there comes a more important factor – violent assaults upon journalists. Total score between 0 = total press freedom and 100 = no press freedom
Indices characteristics
Indices represent an area of variables, i.e. in an index there are already several items summed up mathematically: Indices reduce the area of variables. The specific description of individual indicators is lost. * The index is the sum of all measured variables and hence it is itself a new variable. * The index needs to cover all aspects of the variables being studied, and it should do so in a straightforward way, without getting too complicated or including unnecessary elements. * The index should be able to measure everything important about the topic in a clear and simple manner. * The measurements that enter an index must all have the same measuring level, e.g. nominal, dichotomous scales.
Difference between Index and Scale
- An index is like a summary score that combines lots of different things into one number, giving you a big picture. * For example: an index of happiness might include factors like income, health, and social connections, all rolled into one score.
- A scale is like a ruler with different points to measure one specific thing. * It is more detailed and focused. * A scale measuring happiness might ask you to rate how happy you feel on a scale from 1 to 10, giving you a precise measure of your happiness level
Scales: terms difference
Attention: Different usage of the term “scale“.
- Scale level: Diversity of measurements
- Scale: Summation of several individual measurements into one total value.
- In this case it is used similarly to the term index. The difference might be more about the specific terminology used rather than a fundamental conceptual distinction
Types of Scales
▪ Likert scale:
▪ Consists of several at least 5-stage items (interval scaled), which are combined to an index by summation.
▪ Semantic differential:
▪ Consists normally of about 10 to 20 opposing pairs.
Typical usage of a Likert scale with the aim to empirically determine the meaning of a concept. Multi-point rating option
Why do we conduct a Pre-test?
Catch any problems with it before you conduct it Problems could be:
* questions that don’t make sense
* technical glitches
* potential sources of bias
➢ Pre-test are worth their time: better data quality, less drop-outs and less effort during the analysis and interpretation.
➢ Even under time pressure a pre-test should not be missed out on.
Pre-Test
- Make revisions based on information from pre-testing * Edit questions and response categories if needed * Simplify language/ be more explicit if needed * In case of major revisions consult with survey development experts and conduct a retesting of the survey
Pre-test Focus
Comprehension
* Do respondents understand the aim of the survey and the wording of the introduction and questions? Logic and flow
* Do items in the questionnaire follow a logical order?
* Is anything out of place or confusing?
* Do question order set up conditions for bias?
Acceptability
* Do questions cause offense or touch on sensitive subjects in an inappropriate way?
Length and adherence
* Does it take a lot of effort to fill in the survey?
* Can respondents make it through to the end without losing interest and focus?
Technical quality
* Does the survey platform operates smoothly? Introduction and gaining consent
* Do we have an introduction declaring aim and scope of the research clearly? Can respondents give informed consent?
How to Pre-Test?
- Find 5-10 people from your target group to pre-test your survey * Try to get a range of different people who are representative of your target group * Ask them to complete the survey while thinking out loud (not down if they understand or do not understand a question) * The person should comment on anything that crosses their mind (method of loud thinking);
- Let them tell you if something is unclear or ambiguous. * Take notes of every comment made – do not comment or justify * If you are uncertain whether a question was understood correctly, you can ask your tester directly what they understood. * In case your testers were satisfied with everything at first contact, then you haven’t sufficiently asked them for criticism. * Make improvements based on the results
Piloting a Survey
Pre-test repetition
- Once you have incorporated all your changes, you should repeat the pre-test
- You need to recruit different people, since the original testers know how to understand your questions
- All questionnaires need a couple of alterations.
- Usually, 2-4 pre-test rounds are needed, to turn a good questionnaire into a really good one. Conduct a full pilot before starting actual data collection You need to test all the survey steps from start to finish with a reasonably large sample
- The size of the pilot sample depends on: * how big your actual sample is * how many data collectors you have
Benefits of Survey Testing
- Uncovering Problems: survey testing can identify all types of problems, from intimidating survey length to ambiguously worded questions, poor design to flawed survey logic
- Pretesting helps avoid the time-consuming process of scrapping a flawed survey
- Improving Reputation: if you release a flawed survey, it can reflect poorly on your research and you might also waste the time and effort of respondents.
- Research Problem: if your survey is flawed, you will release inaccurate conclusions based on data accumulated through a flawed survey and your research will be unreliable
Testing will improve your decision making - Inaccurate data can lead to make poor decisions based on your research that ends up with incorrect conclusions.
- Testing a survey to be sure respondents understand it will result in more accurate responses when the final version is sent out, improving decision making. Safety and Compliance
- Testing your survey before releasing it, we can catch technical glitches and security issues and fix them before any damage is done. Improving Productivity
History of Pretesting
- Modern sample survey started in the mid-1930s * Scholar started working on pretesting from 1940 when pretests were well established. * “The American Institute of Public Opinion pretested their questions to avoid phrasings which will be unintelligible to the public and to avoid issues unknown to the man on the street” * It usually takes about 12–25 cases to reveal major difficulties and weaknesses in a questionnaire
Why Pretest?
- Questionnaire problems will be signaled either by the answers that the questions elicit (e.g., “don’t knows” or refusals)… * … Or by some other visible consequence of asking the questions (e.g., hesitation or discomfort in responding) * Moser and Kalton (1971, p. 50) judged, “Almost the most useful evidence of all on the adequacy of a questionnaire is the individual fieldworker’s [i.e., interviewer’s] report on how the interviews went, what difficulties were encountered, what alterations should be made, and so forth.” * Sudman and Bradburn’s (1982, p. 49) add: “A careful pilot test conducted by sensitive interviewers is the most direct way of discovering these problem words”
Changes in Pretests in Recent Years
- Shift in the goals of testing: from an exclusive focus on identifying and fixing overt problems experienced by interviewers and respondents to a broader concern for improving data quality so that measurements meet a survey’s objectives. * New testing methods have been developed or adapted from other uses ➢ cognitive interviews ➢ behavior coding ➢ response latency ➢ vignette analysis ➢ formal respondent debriefings ➢ Experiments ➢ and statistical modeling The development of these methods raises issues of how they might best be used in combination, as well as whether they in fact lead to improvements in survey measurement.
Pretesting: Methods of Testing
Pilot Testing
* Involves formally testing your complete, structured survey with a small sample of respondents.
* Rather than ask evaluation questions, you give respondents the survey as-is and ask them to complete it.
* Pilot testing your survey provides a sense of the kind of responses you will receive and any issues that may arise during the real survey period.
Data Analysis
* It can help to complete some data analysis.
* This involves looking at patterns in responses to see where confusion, hesitation, disengagement, or drop-out has occurred.
* You can often be discovered by identifying straight-lining (the same answer is always checked regardless of the question), unanswered questions, and inconsistent or unrealistic responses. Read more about these analysis techniques in our Data.
Cognitive Interviews
- Cognitive interviewing can get to the bottom of what is going on in your respondent’s mind.
- These interviews are conducted in person with a small sample of respondents (generally about 5-15). As they answer each survey question, a facilitator asks them to think aloud, describing their thought processes, emotional responses, and understanding of what each question means.
- These interviews can help determine if a question is ambiguous, confusing, or makes people uncomfortable due to its content.
- They can also spot drop-out risks by identifying if respondents are getting bored or agitated by the survey.
Ordinary interviews focus on producing codable responses to the questions. Cognitive interviews aim to show how people think when they answer questions.
* Concurrent or retrospective “think-alouds” and/or probes are used to produce reports of the thoughts respondents have either as they answer the survey questions or immediately after. The objective is to reveal the thought processes involved in interpreting a question and arriving at an answer. These thoughts are then analyzed to diagnose problems with the
* Research on cognitive interviews has begun to reveal how the methods used to conduct the interviews shape the data produced.
Behavior coding: Supplements to Conventional Pretests
Behavior coding
* Involves monitoring interviews or reviewing taped interviews (or transcripts) for a subset of the interviewer’s and respondent’s verbal behavior in the question asking and answering interaction.
* Questions marked by high frequencies of certain behaviors (e.g., the interviewer did not read the question verbatim or the respondent requested clarification) might need repair.
* Measurement of “response latency,” the time it takes a respondent to answer a question. Since most questions are answered rapidly, latency measurement requires the kind of precision (to fractions of a second) that is almost impossible without computers.
* Draisma and Dijkstra (2004) reasoned that longer delays signal respondent uncertainty, and they tested this idea by comparing the latency of accurate and inaccurate answers (with accuracy determined by information from another source).
Vignettes
Hypothetical scenarios that respondents evaluate. They may be incorporated in either undeclared or participating pretests. Vignette analysis appears well suited to
1. explore how people think about concepts
2. test whether respondents’ interpretations of concepts are consistent with those that are intended
3. analyze the dimensionality of concepts
4. diagnose other question wording problems. Martin offers evidence of vignette analysis’s validity by drawing on evaluations of questionnaire changes made on the basis of the method.
Expert Evaluation
- It may help to have an expert opinion, especially if your survey is asking questions about a complex subject matter.
- Topic experts can tap into their deep well of knowledge to evaluate your questions in order to help shape or reshape the content.
- Survey methodologists may also be called upon to help determine the best ways to collect accurate data for research questions.
Focus Group
A focus group is a semi-structured discussion with a small group of people, usually around ten or so, led by a moderator. The members of the group can be very helpful in forming questions and addressing some of the following issues:
* Subject matter relevancy to the group
* Characteristics of the target population
* Understanding of questions and concepts
* Difficulty of the survey and abilities of the group
* General group reaction to the survey wording and design Focus groups can be very helpful but there is the danger of “group speak,” wherein problems can be built up and exaggerated as group members “feed off” one another during discussions. Focus groups should never be the sole means of survey testing
Focus Group: how to process
Focus group is a semi-structured discussion with the purpose of stimulating conversation around a specific topic.
It is led by a facilitator who poses questions, and the participants give their thoughts and opinions.
It creates the possibility to cross check one individual’s opinion with other opinions gathered. A well organized and facilitated Focus Group Discussion is more than a question and answer session.
In a group situation, members tend to be more open and the dynamics within the group and interaction can enrich the quality and quantity of information needed
Focus group: principles
- Define Objectives Clearly outline the goals and objectives of the focus group. Participant Selection Identify and recruit participants who represent your target audience or key stakeholders. Ensure diversity in demographics and perspectives Create a Moderator’s Guide Develop a structured moderator’s guide with openended questions to guide the discussion. This helps maintain consistency while allowing for flexibility to explore unexpected insights.
- Create Open-Ended Questions Develop a set of open-ended questions that encourage participants to share their thoughts, experiences, and opinions. Avoid leading questions that may bias responses, and allow for flexibility to explore unexpected insights that may arise during the discussion. Sequence Questions Logically Organize the questions in a logical sequence that flows naturally, gradually moving from general and introductory topics to more specific and in-depth areas. This helps build rapport with participants and ensures a smooth and focused discussion.
- Include Probes and Follow-Up Prompts Incorporate probes and follow-up prompts to delve deeper into participants’ responses. Can be designed to produce more detailed information, clarification, or diverse viewpoints. Probes help maintain the conversational flow and encourage participants to express themselves fully. Consider Group Dynamics Account for group dynamics in the moderator’s guide. Include strategies for managing potential challenges, such as dominating participants or a lack of engagement. Design questions that encourage all participants to contribute and ensure that the guide allows for flexibility in addressing unexpected topics that may arise during the discussion.
- Choose a Suitable Venue Select a comfortable and neutral venue conducive to open conversation. Ensure that the space allows for audio recording and video observation, if necessary. Moderator Skills Appoint a skilled and impartial moderator who can facilitate the discussion, keep participants engaged, and guide the conversation toward the research objectives. The moderator should be able to manage group dynamics effectively.
- Establish Ground Rules Set clear ground rules at the beginning of the session to create a respectful and inclusive environment. Emphasize the importance of confidentiality active listening avoiding dominant behavior. Limited Group Size Keep the focus group size manageable, typically ranging from 6 to 12 participants. This ensures everyone has the opportunity to contribute, and the discussion remains focused.
- Recording and Documentation Record the focus group session (with participant consent) for accurate analysis. Additionally, take detailed notes during the discussion to capture non-verbal cues, participant reactions, and key themes. Data Analysis Employ qualitative analysis methods to extract meaningful insights from the data. This may involve coding, categorizing responses, and identifying patterns or recurring themes. Triangulation and Validation Enhance the credibility of your findings by triangulating the results with other sources of data or by conducting multiple focus groups. This helps validate the reliability and consistency of the gathered insights.
Interviewer Check-list
Respondents Desirability Bias: what, how to avoid
Desirability Bias * Respondents often over-report desirable and underreport undesirable behaviours to portray themselves positively. * The key to avoiding it * Minimise the aspects of answering that may seem threatening to respondents * Increase the more salient features that influence comfort within the respondent to reveal private or sensitive information. * This can be done by increasing privacy and anonymity, for instance.
Interviewers Bias: reasons
Bias
* Interviewers have a significant influence on survey reports.
* An interviewers’ race, gender, and expectations can all have an impact on responses.
* An interviewer’s conduct can also impact data quality, especially when it strays from the standardised procedures.
* This leaves room for measurement error caused by interviewer-respondent interactions which can result in biased responses and a decreased reliability of answers.
* The mere presence of interviewers influences survey respondents and their answers.
Standardized Interviews: why, how
Standardized interviewer behavior is necessary for reasons of reliability:
* Survey methodology works from a stimulus-response model of interaction
* Every respondent should be presented with the same verbal input
* Interviewers should read aloud each question exactly as worded, preserve the order of the questions, and probe in a non-leading manner
* Consistency ensures that differences in responses can be related to respondent characteristics, rather than to characteristics of the interviewing procedure
- A change in the formulation or the order of questions and response options can have an effect on the responses * It is crucial that interviewers do not alter anything in the questionnaire Interviewers need to follow the rules of standardized interviewing: * Read the questions exactly as worded * If the respondent’s answer is incomplete or inadequate, probe for clarification or elaboration in a non-directive way * Record the answers without interviewer’s discretion. * Do not provide any positive or negative feedback regarding the specific content of respondent’s answers
Several studies show that interviewers often do not act according to these rules Reasons: Question may be ambiguous or difficult to read aloud * Can be solved in pre-testing Respondents may not understand what they are expected to do * They often do not act their roles as prescribed * Interviewers need to explain clearly to respondents the rules of the game
Exploring Inadequate Answers: reactions, decisions
- Respondents can provide non satisfactory answers and interviewers must probe for more adequate or more complete ones * Do this in a non-directive way. Interviewers’ probes should not increase the chance of one answer over another * Each probe that can be answered by “yes” or “no” is directive
We need to establish rules that interviewers should follow when probing answers: * When respondents miss part of the question, interviewers should reread the entire question * When respondents do not understand a term or concept in the question because a term turns out to be ambiguous, interviewers should not provide an explanation. They can reread the definition, if the questionnaire provides one. * Otherwise, respondents must be told to interpret the question to their own liking and answer the question. * When probing answers, interviewers should make sure that answers meet the questions’ objectives.
Administer the Survey: interviewer behaviour
- With interviewer-administered surveys, the interviewer provides feedback to the respondent * If an invalid response is provided, the interviewer should repeat the question or response options in order to get an accurate response * When a respondent asks for clarification, the question should be restated using the same words * When interviewers try to clarify the question on the fly, these explanations can change the meaning of the question. (This problem can be avoided by careful development of the questionnaire and pretesting the instrument.) * Recording answers accurately and completely is important. When using a paper questionnaire, the answers must be readable
Introductory Script: The Outline and Format
- Introduction to the survey * Title of the study * Purpose * Duration of survey and length of questionnaire * Guarantee of confidentiality * Brief information about the company or organization (IMS)
Preparing the Introductory Script and Support Materials
Introductory script and other support materials must be prepared before implementing the survey * The introductory script is used by the interviewer to schedule the * Date * Time * Location for the interview * Interviewers have about 20–30 seconds to obtain a respondent’s agreement to participate; * The scripts should be as brief as possible while also communicating critical information about the purpose and importance of the survey.
Input Mask
- An input mask makes it much easier for users to figure out the required format for filling out the form * Input masks are a way to constrain data that users enter into form fields and enforce specific formatting * Use Google Forms to design your survey
Choices: Designing an Input Mask
- User must enter a digit (0 to 9). * User can enter a digit (0 to 9). * User must enter a letter. * User can enter a letter. * User must enter a letter or a digit. * User can enter a letter or a digit. * User must enter either a character or a space. * User can enter characters or spaces. * Decimal and thousands placeholders, date and time separators. The character you select depends on your regional settings (in Germany commas and full stops).
Sampling procedures: 2 ways
We have to provide information about certain characteristic features of statistical populations.
Two ways
Census:
▪ Complete count: It is a study of every unit, everyone or everything, in a population
Partial sampling (sub-samples): ▪ A sample, which is a subset of units in a population
▪ Selected to represent all units in a population of interest. ▪ Information from the sampled units is used to estimate the characteristics for the entire population of interest.
Reasons for Sampling
- Cost saving ▪ Collecting smaller amounts of elements is cheaper than conducting a census (complete enumeration).
- Time saving ▪ Data collection in the case of sampling takes less time than a census.
- A census or complete enumeration is practically impossible ▪ It is theoretically imaginable, but practically not reasonable or possible.
Sampling Frame
Required Sample Size and Sample Mean
- To achieve a certain level of confidence and accuracy a sample size is required to be at least a certain size * To calculate the required sample size we need to use these factors: population size, parameter variation, degree of accuracy, and degree of confidence * Our sample mean should be close to the population mean, the sample inter-quartile range to be close to the population interquartile range, and so on
Parameter Variation and Confidence Level
Parameter Variation: * Is the process of changing the input parameters (for example: height, age, etc.) of a mathematical model or simulation to investigate the effect of each parameter on the output. * We want to determine which input parameters have the most significant impact on the output and how the output responds to changes in these parameters. Confidence Level * Is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way
Degree of Accuracy
- Sample accuracy refers to the extent to which sample statistics correctly estimate the population parameter. * We typically use the terms biased and unbiased to describe the accuracy of sample statistics. Example: we take many thousands of samples from the same population. For each sample, we calculate a statistic (e.g., the mean). ➢ Unbiased sample: If the average of the sample statistics (e.g., sample means) equals the population parameter (e.g., population mean) then we refer to the statistic as being unbiased. ➢ Biased sample: If the average of the sample statistics (e.g., sample means) does not equal the population parameter (e.g., population mean) then we refer to the statistic as being biased
Sampling
Arbitrary Sampling
▪ In arbitrary sampling, the researcher selects individuals or units of study because they are available, convenient and represent some characteristic the researcher seeks to study without following any specific system, e.g. street surveys. ▪ Representativity (the sample is a reduced, structurally similar representation of the population under study) is problematic.
Probability Sampling Designs
- Every case in the population has a known, greater-than-zero probability of being selected for the sample. For example quantified summaries of characteristics of the sample, like the median of a variable or the correlation between two variables. * With probability sampling design, we can use statistics to estimate the parameters (the corresponding quantified characteristics of the population) with known levels of confidence and accuracy.
Random Sampling
Reliable selection of individuals or units of study from the population.
▪ The intent of random (or probability) sampling is to choose individuals or units for the sample so that any unit has an equal probability of being selected from the population.
▪ The more often one takes random samples from the same population, the closer one will get to the actual value in this population.
▪ No pre-specific characteristics
▪ The empirical social research accepts in general a 5 percent probability of error. The scores it provides always contain this sampling error.
Example: lottery procedure, i.e. the random extraction of elements from a limited population. Common selection or sampling procedures: ▪Choose every n-th element (or individual) in the population (e.g. selecting the file with the number 100 from a student file system) The criterion according to which a systematic random sampling is being applied needs to be distributed independently from the features that are studied in the population.
Random Cluster Sampling
Cluster sampling is a spatiotemporal (time and space) defined conglomeration of elements of the population which form a structurally reduced representation of the respective population. Typical clusters: households, school classes, apartment buildings. Elements included: persons, pupils, households. Procedure: you choose by random sampling a number of clusters and analyse all units of study that occur in these clusters
Simple Random Sample
Most basic kind of probability sampling design Each case in the population has a known and equal probability of being selected for the sample. Goal: construct a sample that is like the population so that we can use what we learn about the sample to generalize to the population.
Example: If you want a sample of 50,000 graduates using age range, the proportionate stratified random sample will be obtained using a formula: (sample size/population size) × stratum size. A stratum (plural strata) refers to a subset (part) of the population (entire collection of items under consideration) which is being sampled. The table below assumes a population size of 180,000 MBA graduates per year.
Calculation: The strata sample size for MBA graduates in the age range of 24 to 28 years old is calculated as (50,000/180,000) × 90,000 = 25,000. Same method is used for the other age-range groups. 25,000 graduates from the 24–28 age group will be selected randomly from the entire population 16,667 graduates from the 29–33 age range will be selected from the population randomly, and so on
Probability Sampling and Nonprobability
Sampling Design Sampling frame: list of cases we select our sample from— ideally, a list of all cases in the population Select cases for sample through a sampling design. Two basic varieties of sampling designs: ➢Probability sampling designs and nonprobability sampling designs.
Nonprobability Sampling
In nonprobability or purposeful sampling, you select individuals or units of study according to how “useful” or essential their analysis is for finding an answer to the research problem. ▪ These sampling procedures can be problematic with regard to representativity. − Four procedures: 1. Sampling typical cases 2. Sampling extreme groups 3. Sampling according to concentration principle 4. and quota sampling
Typical Sampling:
▪ Especially appropriate in qualitative research methods, where only few individual cases are analysed. ▪ The sample consists of units of study (e.g. autistic people) who are especially characteristic for all units of study in the population
Sampling according to the Concentration Principle
Sampling according to the Concentration Principle ▪ In a sampling procedure, the researcher focuses on the part of the population where they suspect the predominant part of these elements to be. − Example: Investigation about German skiers, −95% of all German skiers live in Bavaria – using a random sample only from the Bavarian population Cut-Off-Procedure = The less productive or rich part of the study population is being cut off.
Quota Sampling
In quota sampling, you select individuals or units of study according to some fixed quota. ▪ Units of study are selected based on pre-specified characteristics so that the total sample has the same distribution of characteristics assumed to exist in the population that is studied. ▪ Quotas generally rely on demographic characteristics. Example: the sample displays, according to its quota-based characteristics, a structurally identical representation of the studied population
Sample Drop-outs
Sample drop-out refers to all cases where an element of a sample could not be analysed. We distinguish between random and systematic deniers. Random drop-out: Relocation, illness… Systematic drop-out: Persons who deliberately refuse to take part in the survey e.g. highly educated people, people with low language knowledge etc.
Sampling Variation: Problems with Sampling
Sampling variation is how much the results change when you take different samples from a group. Because of this, there is always some uncertainty when you try to make conclusions about the whole group based on those samples. Sampling variation is about how much a number you find might change when you look at different samples of things. (e.g there is no blue person in the sample) ➢ If you measure something in a few different samples, the result might be a little different each time. This means there is some uncertainty when you try to say something about the whole big group based on just those smaller ones.
Sampling variability is how much an estimate varies between samples. ➢ Variability between samples indicates the range of values differs between samples. ➢ If you are measuring something in different samples, you might notice that the numbers you get can be quite different from one group to another. ➢ That is what we call sampling variability—it is like the range of values you see across those different groups.
How can we know how much something varies?
How can we know how much something varies? * Assume maximum variation which places the highest demand on sample size * Assuming maximum variation in a sample size means imagining that each sample you pick is as different as possible from the others. * Imagine every sample represents the extreme ends of whatever you are studying. This helps ensure that your sample size is big enough to cover all the possible variations in the population you’re interested in. It is a way to make sure you are not missing out on any important differences that might affect your results
Why do we specify the variation?
Variation is a way to show how data is dispersed or spread out. * To specify variation in a sample size, you typically consider factors such as * the variability of the population you are studying the level of precision you need in your estimates any constraints on resources like time and budget.
Understand the Population Variation: * Consider the range and variability of the characteristics you are interested in within the entire population. This helps you gauge how much variation might exist in your samples. Define Precision Level * Determine the level of precision you need in your estimates. This depends on factors like the desired confidence level and margin of error. Choose a Confidence Level: * The choice of confidence level depends on how confident you need to be in your results, the potential consequences of being wrong, and any industry standards you may need to adhere to. For most purposes, 95% is a safe and commonly accepted level * Decide on the confidence level you want for your estimates. Common choices are 95% or 99%.
Calculate Sample Size
- Take into account the population variation, desired precision, and confidence level. * Take into account any practical constraints such as budget, time, or feasibility of collecting data. Sometimes you may need to balance statistical considerations with these constraints. * If necessary, adjust the sample size based on factors like the complexity of the analysis, potential non-response rates, or the need for subgroup analysis. * Validate Sample Size: After collecting data, validate whether your sample size was sufficient to provide reliable estimates. You can do this by checking if the confidence intervals are narrow enough to meet your precision requirements.
Variability and Sampling Error
A closely related term (almost a synonym) is sampling error. An error in sampling isn’t a mistake — it’s a measure of how much a value differs from the “true” value. Example: − Let’s say the true weight of a population is 150 lbs. − You take a sample and find the mean weight for the sample is 151 lbs. − The 1 lb difference is an “error.” − If you sample again, you might get different mean weights of 148 lbs, or 150.5 lbs, or 153 lbs. − The different errors — 1/2 lb, 1 lb, 2 lbs, 3 lbs — are a reflection of the variability between your samples, or sampling variability.
Sampling Error
- Increasing or decreasing sample sizes leads to changes in the variability of samples. * For example: a sample size of 10 people taken from the same population of 1,000 will very likely give you a very different result than a sample size of 100. * There is no “perfect” sample size that will give you accurate estimates for the sample mean, variance and other statistics. Instead, you take your best “guess” — using standardized statistical procedures. In general, estimates will change from sample to sample and will probably never exactly match the population parameter.
Why do we need to verify a hypothesis?
- Hypothesis testing is one of the most important processes for measuring the validity and reliability * It helps to provide links to the underlying theory and specific research questions * Verifying a hypothesis provides a reliable framework for making any data decisions for our population of interest * It helps to successfully extrapolate data from the sample to a larger population * Verifying a hypothesis allows us to determine whether the data from the sample is statistically significant
Hypothesis testing
A hypothesis is a proposed explanation for a phenomenon. The term hypothesis is a statement about something that is supposed to be true. The logic of a hypothesis test is to compare two statistical data sets A hypothesis test involves two hypothesis: * the null hypothesis * and the alternative hypothesis
Null hypothesis and alternative hypothesis
Conducting a Survey
- Face-to-face surveys follow a standardized script without deviation * Face-to-face surveys also offer advantages in terms of data quality. * A face-to-face survey allows researchers a high degree of control over the data collection process and environment. Interviewers can ensure that respondents do not skip ahead * Responds rate of face-to-face survey is higher * There is also evidence that questions of a personal nature are less likely to be answered fully and honestly in a face-to-face survey
Advantages of Face to Face Surveys
- Capture verbal and non-verbal cues (discomfort or enthusiasm)
- Helps keep respondents focused
- A surveyor has control over the interview, they can help ensure the interviewee remains focused and on track to complete their survey
- Can allow for more complex questions as interviewer is present to clarify wording and probe for more information
- Can generally be longer than a phone interview
- Arranged to be convenient for the participant to take part in
- Can access samples that phone and Internet questionnaires might not be able to reach
- Ability to incorporate visual stimuli into the questionnaire
- Generally have a higher response rate than other methods
Conducting a Survey - Interviewer bias
- Interviewer is exposed to the potentially biasing effect of the respondent’s appearance and environment in addition to their voice * Interviewer may inadvertently give respondents nonverbal as well as verbal cues about how they should respond
Interviewers - Team Composition
The Team composition and the number of interviewers per team need to be decided based on a number of factors: * the expected duration of interviews * the content of the survey * the size of clusters, etc. On the average each team will need * one supervisor * one runner/editor * three to five interviewers (depending on how many people an interviewer can complete)
Team Lead’s Job
- Identify the clusters to be surveyed * Supervise interviewers as they perform the survey * Ensure that the interviewers follow instructions * Answer interviewers’ questions as they arise * Control data quality by checking for errors during interviewing, checking that forms are completed fully and correctly, and checking that all respondents are answering the questions * Identify problems and retrain interviewers who are doing their job incorrectly.
The field runner/editor’s job is
- Monitoring interviewer performance by: * Observing several interviews, especially during the early stages of fieldwork * Conducting regular review sessions with interviewers (if needed) * Compiling completed questionnaires from a cluster and packing them up
The interviewer’s job is to
- Identify the specific households to be surveyed * Gain the consent of respondents to be interviewed * Conduct interviews using the standard questionnaire * Maintain standard procedures in conducting the interviews and recording the answers.
Data Cleaning Specific to our Survey
Missing Data
* Open Ended Questions: * For these questions, you may need to categorize the write-in responses to create a consistent coding scheme. * Sometimes participants did not answer all the survey questions due to any reason * This partial filled data can create some noise in pure data analysis. It is advised to remove the partial completed responses from the overall data before starting analysis. * Check Spelling - Merge Doubling
Skipped Questions
* Identify if any questions were consistently skipped by a large number of respondents. * This might indicate a problem with the question itself or the survey flow.
Inconsistent Data
- Age (Q2) and Employment Status (Q5): There might be inconsistencies (e.g., someone selecting “Student” under employment but an age outside the typical student range). * You may need to decide how to handle these discrepancies (e.g., contacting respondents for clarification, excluding outliers). Response Options * “Prefer not to say” (Q1): Decide how to handle this option. * Similar to “Prefer not to say”, decide how to categorize this for respondents who might not work in media (e.g., separate category).
Data Cleaning Techniques
Standardization * Ensure consistent formatting for written responses (e.g., all uppercase or lowercase). Coding * Create numerical codes for answer choices in multiple-choice questions to simplify analysis. Outlier Detection * Identify and potentially exclude extreme values that might skew the results (consider reviewing these cases individually).
Respondents Who Give Inconsistent Responses
- When a respondent’s answer contradicts their response to another question, it’s clear that they’re either being dishonest or careless (or even both!). * You may be able to find these inconsistencies by applying multiple filters. * For instance, say that one of your survey questions asked respondents how much TV they watch per week. When the responses come back, you filter by those who said they watch at least a little. Document Decisions * Keep a log of the decisions made during data cleaning to ensure transparency and replicability. Remember, data cleaning is an iterative process. As you explore the data, you may identify new cleaning needs
Remove Straightliners and Unrealistic and Nonsensical Answers
- Researchers needs to be cautious about straightline respondents and responses must be removed before analysis. * Straightlining is when respondents choose similar answer option frequently (such as first/last option etc.). * There might be higher possibility that the respondent has not responded the answers honestly. * Imagine asking respondents how much TV they watch per week, on average. If a respondent writes in 165 hours, they’re likely exaggerating (Hint: there are only 168 hours in a week). * Having a response like: “Fdsklj” might make you smile, but it isn’t going to get you far in your analysis.
Remove Outliers - Remove Fake Or Manipulated Answers
- Sometimes you have encountered responses falling under unrealistic range for example “In which sport you see yourself as pro” and one respondent selected all the sports from option. * This kind of responses are called outliers. * This becomes tricky sometimes but researchers should always be beware of fake or manipulated responses from participants. * There are many ways to check fake responses such as, using open ended questions — check is any response contain unreadable or meaningless response like ‘fgfgfh’ type text etc.
Conceptualizing your Qualitative Research Question
- Should be inductive, exploratory * Is framed as a question, aim or objective but not as a hypothesis * Should focus on a single phenomenon, concept or idea * Start with a verb and stating your goal (e.g. characterize, understand etc.) * Identify your topic of interest * Use a language that is non-directional and neutral to be exploratory * Define your sample and your setting * Example: Understand interpersonal factors relevant to relationships formed online by retirees - in private homes in Bavaria
Methodology
How to Analyze Surveys?
1.Look at the results of your survey as a whole 2.Take a look at the demographics of those who responded 3.Compare responses to different questions to find deviations 4.Find connections between specific data points with layered data 5.Compare new data with past data
Survey Data – Connection to Research Question
− To what extent do media practitioners perceive AI tools in improving productivity, credibility, and creativity? * Filter results by cross-tabulating subgroups * Filter your results based on specific types of respondents, or subgroups * Look at different attendees (age, gender)
- What are the most common responses to a specific question? * Which responses are affecting/impacting us the most? * What did respondents with a particular job title say? * Which group of respondents are most affected by a specific issue? * Have conference visitors noticed something outstanding? * What do people say about a specific topic?
- Find connections between specific data points with layered data − Understand connections − Look for causation and correlation − Look for confounding variables 5. Compare new data with old data − Check for changes and try to find explanations for them
Differences between Codes, Categories and Themes
Meaning unit: It is the smallest unit of analysis that captures a single idea or concept in the data.
Condensed meaning unit: It is a shortened version of the meaning unit that retains the essential information.
Code: It is a label assigned to condensed meaning units to categorize and organize the data.
Category: It is a group of similar codes that represent a broader concept or idea.
Theme: It is a higher-level abstraction that emerges from grouping or categorizing related codes and categories.
