PROCTOR EXAM 1 - QUAL AND SAMPLING Flashcards
A sample is often used to study a target population. the larger our sample size the smaller our margin of error and our estimate of the population is more likely to be accurate.
A sample is often used to study a target population. the larger our sample size the smaller our margin of error and our estimate of the population is more likely to be accurate.
Stages of the sampling process
- Defining the target population
- Deciding on the best way to reach the population
- Determining a sample frame
- Choosing a sampling method
- Determining a sample size and carrying out the sampling process
Stages of the sampling process
- Defining the target population
- Deciding on the best way to reach the population
- Determining a sample frame
- Choosing a sampling method
- Determining a sample size and carrying out the sampling process
One of the biggest decisions within the sampling process will be choosing between a probability vs. a non-probability sampling method.
This decision will depend on the purpose of the research, the need for accuracy, the budget, and timing of the project.
One of the biggest decisions within the sampling process will be choosing between a probability vs. a non-probability sampling method.
This decision will depend on the purpose of the research, the need for accuracy, the budget, and timing of the project.
Knowing the incidence of our target population within our sampling frame will help us predict how many interviews need to be conducted
to achieve the sample size we want and how much money the study will cost.
Knowing the incidence of our target population within our sampling frame will help us predict how many interviews need to be conducted
to achieve the sample size we want and how much money the study will cost.
It is important for market researchers to treat research buyers ethically by being transparent regarding sampling practices and treat the survey participants ethically by being honest with them and fulfilling any promises to them
It is important for market researchers to treat research buyers ethically by being transparent regarding sampling practices and treat the survey participants ethically by being honest with them and fulfilling any promises to them
Market Research is utilized to make decisions that reduce business risks. To have con dence in the decisions that are being made, we
need to have a sample that we are con dent in. Without that, all the results produced from our study could be called into question.
Market Research is utilized to make decisions that reduce business risks. To have con dence in the decisions that are being made, we
need to have a sample that we are con dent in. Without that, all the results produced from our study could be called into question.
The principles we have laid out in this course highlight that the sampling methods used to produce the research are as important to
interpreting the findings as the questions being asked. The answers to these questions can help
us formulate an idea of how accurate a secondary data set may be and how appropriate it would be to use the data set for our current need.
The principles we have laid out in this course highlight that the sampling methods used to produce the research are as important to interpreting the findings as the questions being asked. The answers to these questions can help
us formulate an idea of how accurate a secondary data set may be and how appropriate it would be to use the data set for our current need.
A Random Digit Dialing (RDD) sample consists of phone
numbers randomly generated from a computer.
A Random Digit Dialing (RDD) sample consists of phone
numbers randomly generated from a computer.
RDD - However, this is not the most efficient way to produce a
phone sample, as many of the numbers that are chosen
could be non-working or lead to a place of business.
RDD - However, this is not the most efficient way to produce a
phone sample, as many of the numbers that are chosen
could be non-working or lead to a place of business.
Often systematic sampling is used for phone—where all of the
phone numbers within the defined sample frame are put in numerical order, then the first number is chosen at random and then every
nth number the next number is chosen.
By performing a systematic sample via the phone numbers, it can help to guarantee disbursement of numbers across the population.
Often systematic sampling is used for phone—where all of the
phone numbers within the defined sample frame are put in numerical order, then the first number is chosen at random and then every
nth number the next number is chosen.
By performing a systematic sample via the phone numbers, it can help to guarantee disbursement of numbers across the population.
Stratification would be another form of telephone sampling that may also produce a more accurate sample than an RDD sample, in certain conditions. Because telephone numbers tell you information about location, it may make sense to stratify the sample by specific geography.
Stratification would be another form of telephone sampling that may also produce a more accurate sample than an RDD sample, in certain conditions. Because telephone numbers tell you information about location, it may make sense to stratify the sample by specific geography.
Landlines remain an important part of telephone sampling today, largely due to the cost associated with dialing mobile telephones.
Landlines remain an important part of telephone sampling today, largely due to the cost associated with dialing mobile telephones.
Mixing across these two separate frames (landline and mobile), requires care to apply various screenouts and/or data
weighting , as the people with both mobile and landline telephones will be more likely to be chosen, because they exist in both frames.
Mixing across these two separate frames (landline and mobile), requires care to apply various screenouts and/or data
weighting , as the people with both mobile and landline telephones will be more likely to be chosen, because they exist in both frames.
Autodialers can be used to dial landline telephone numbers. These computer based dialers will call many telephone numbers quickly,
passing the phone to a live interviewer once someone has answered.
Autodialers can be used to dial landline telephone numbers. These computer based dialers will call many telephone numbers quickly,
passing the phone to a live interviewer once someone has answered.
E ective sampling is important in order to gather meaningful data that will drive successful decisions. Taking care to consider alternatives
and make informed choices in each step of the sampling process helps to ensure success.
E ective sampling is important in order to gather meaningful data that will drive successful decisions. Taking care to consider alternatives
and make informed choices in each step of the sampling process helps to ensure success.
One of the most common questions you will hear in the Market Research industry is “How large does my sample need to be in order to be representative?” However, it is not the sample size that determines representativeness. It is the other elements discussed in this course, such as defining the target population correctly, and selecting valid sample in a way that will assure representativeness
One of the most common questions you will hear in the Market Research industry is “How large does my sample need to be in order to be representative?” However, it is not the sample size that determines representativeness. It is the other elements discussed in this
course, such as defining the target population correctly, and selecting valid sample in a way that will assure representativeness
Sample size is very often used interchangeably with the terms base size and the number of completed interviews.
Sample size is very often used interchangeably with the terms base size and the number of completed interviews.
Response rates - (% of those responding to an
invitation)
Response rates- (% of those responding to an
invitation)
Participation rates - (% of those responding and willing to complete the survey)
Participation rates - (% of those responding and willing to complete the survey)
Base or Base Size
The number of sample units used in analysis. This could be all the completed interviews, a subset of the completed interviews, or
even all of the started interviews. This term is sometimes used interchangeably with sample size when presenting research results.
Base or Base Size
The number of sample units used in analysis. This could be all the completed interviews, a subset of the completed interviews, or
even all of the started interviews. This term is sometimes used interchangeably with sample size when presenting research results.
Sample
A sub-group of the population of interest that has been selected for study. The term is also used to refer to a different number of people who completed the study.
Sample
A sub-group of the population of interest that has been selected for study. The term is also used to refer to a different number of people who completed the study.
Sample Size
The number of sample units selected for a study. The term is sometimes used interchangeably with base size.
Sample Size
The number of sample units selected for a study. The term is sometimes used interchangeably with base size.
A margin of error (a range around a sample finding that is expected to contain
the true finding within the population). We measure through statistical analysis how likely the truth is to be within this margin.
A margin of error (a range around a sample finding that is expected to contain
the true finding within the population). We measure through statistical analysis how likely the truth is to be within this margin.
The larger the sample size, the smaller the impact additional sample has on the margin of error.
The larger the sample size, the smaller the impact additional sample has on the margin of error.
The confidence level produces a confidence interval , which is the margin of error on either side of our result. The higher the confidence level we test at, the greater our confidence interval and margin of error become.
The confidence level produces a confidence interval , which is the margin of error on either side of our result. The higher the confidence level we test at, the greater our confidence interval and margin of error become.
Significance testing is often used in cross-tabulation to measure whether or not two results have a statistically significant difference.
This testing uses confidence level and margin of error to make a determination, but it does so often by comparing two results with
different base sizes. This testing is done to confirm that differences are likely to be real, rather than due to sample error
Significance testing is often used in cross-tabulation to measure whether or not two results have a statistically significant difference.
This testing uses confidence level and margin of error to make a determination, but it does so often by comparing two results with
different base sizes. This testing is done to confirm that differences are likely to be real, rather than due to sample error
The screening section of the questionnaire is where we qualify participants as part of the target population, removing people from the
original sample in the process.
The screening section of the questionnaire is where we qualify participants as part of the target population, removing people from the
original sample in the process.
People who begin the survey, but do not complete the survey are considered as participants who have dropped from the survey or
incompletes. This is measured as a proportion of the sample that was
able to start the survey, but did not complete the survey, even though
they were not turned away. It is important that this drop rate stays relatively consistent over time and that it is not tied to a specific element of the survey, which could potentially create a bias in the sample.
People who begin the survey, but do not complete the survey are considered as participants who have dropped from the survey or
incompletes. This is measured as a proportion of the sample that was
able to start the survey, but did not complete the survey, even though
they were not turned away. It is important that this drop rate stays relatively consistent over time and that it is not tied to a specific element of the survey, which could potentially create a bias in the sample.
Market researchers may evaluate the quality of the data collected through the questionnaire and decide that certain survey participants need to be removed from the data (also known as data cleaning). The data is assessed in a variety of ways, from speeding (finishing the questionnaire too quickly), straight lining (giving the same answers over and over), terse or
nonsense responses from open-ended questions, and other quality control measures. This is done is to reduce the noise in the data from these invalid responses, increasing the overall validity of the survey findings.
Market researchers may evaluate the quality of the data collected through the questionnaire and decide that certain survey participants need to be removed from the data (also known as data cleaning). The data is assessed in a variety of ways, from speeding (finishing the questionnaire too quickly), straight lining (giving the same answers over and over), terse or
nonsense responses from open-ended questions, and other quality control measures. This is done is to reduce the noise in the data from these invalid responses, increasing the overall validity of the survey findings.
Sampling internationally is not easy
Sampling internationally is not easy
When doing international
sampling, you must understand cultural differences. You also have to understand behavioral differences
When doing international
sampling, you must understand cultural differences. You also have to understand behavioral differences
“What do clients mean when they ask for census balanced”? Research buyers will often use the word “census” when they mean
characteristics of the overall population.
“What do clients mean when they ask for census balanced”? Research buyers will often use the word “census” when they mean
characteristics of the overall population.
Online is also a self-completion mode where the participants do not interact with an interviewer. This method is often the preferred means to conduct surveys
Online is also a self-completion mode where the participants do not interact with an interviewer. This method is often the preferred means to conduct surveys
It is possible to achieve a probability sample from online when
the sample frame gives complete coverage of the target population
It is possible to achieve a probability sample from online when
the sample frame gives complete coverage of the target population
An online sample that is trying to measure the general population is a form of convenience sampling.
An online sample that is trying to measure the general population is a form of convenience sampling.
Often participants who are recruited to take part in online surveys are recruited to join research panels, where they have given their
contact information and will be invited to take part in multiple research studies over their tenure.
Often participants who are recruited to take part in online surveys are recruited to join research panels, where they have given their
contact information and will be invited to take part in multiple research studies over their tenure.
A technique known as a
“ River Sample ” recruits participants from online advertisements directly into surveys, as opposed to using existing online panels. The difference being that online panels know more about the survey participant, tend to have a more consistent sample frame, and can adhere to a higher level of quality in this self-completion environment. However, proponents of river sampling point out that not everyone wants to join a research panel and that river sample extends the coverage in an online sample.
A technique known as a
“ River Sample ” recruits participants from online advertisements directly into surveys, as opposed to using existing online panels. The difference being that online panels know more about the survey participant, tend to have a more consistent sample frame, and can adhere to a higher level of quality in this self-completion environment. However, proponents of river sampling point out that not everyone wants to join a research panel and that river sample extends the coverage in an online sample.
At its core online sampling is a convenience sample. Because online sampling has become the most popular form of sampling and widely
accepted across the Market Research industry, there has been large investment into the quality, methodology , and best practices
around online sampling.
At its core online sampling is a convenience sample. Because online sampling has become the most popular form of sampling and widely
accepted across the Market Research industry, there has been large investment into the quality, methodology , and best practices
around online sampling.
The goal of recruiting through online channels should be to minimize potential bias, by recruiting through a diverse set of sources and by
trying to maximize participation.
The goal of recruiting through online channels should be to minimize potential bias, by recruiting through a diverse set of sources and by
trying to maximize participation.
Many advantages to routers 1. Efficient way to allocate sample 2. Has the opportunity to reduce cost 3. Better user experience, especially for those who are always screened out of a survey
Many advantages to routers 1. Efficient way to allocate sample 2. Has the opportunity to reduce cost 3. Better user experience, especially for those who are always screened out of a survey
The concern when it comes to employing routers is a router bias occurring. The router bias is most concerning when two
studies are competing for a similar audience.
The concern when it comes to employing routers is a router bias occurring. The router bias is most concerning when two
studies are competing for a similar audience.
Although survey routing is clearly a form of convenience sample, there are arguments that survey routing is an improved methodology
for online panels over email invitations directly into studies. The routing technology reduces the opportunity for self-selection that could
come from an email invitation. The routing technology allows panel companies to increase reach by bringing in sample from other
sources, which could be other panel providers, river sample, other non-email based sources, such as app based panels and social
communities.
Although survey routing is clearly a form of convenience sample, there are arguments that survey routing is an improved methodology
for online panels over email invitations directly into studies. The routing technology reduces the opportunity for self-selection that could
come from an email invitation. The routing technology allows panel companies to increase reach by bringing in sample from other
sources, which could be other panel providers, river sample, other non-email based sources, such as app based panels and social
communities.
Their overall conclusion is that routers do not show measurable levels of bias
relative to any online sample, as long as they are employed correctly.
Their overall conclusion is that routers do not show measurable levels of bias
relative to any online sample, as long as they are employed correctly.
In online research targeting can be used to help with the efficiency of the study
In online research targeting can be used to help with the efficiency of the study
One concern users of online research panels have is prior survey exposure. This is a conditioning effect that occurs when information from a previous survey informs the participant and influences their responses to future surveys.
One concern users of online research panels have is prior survey exposure. This is a conditioning effect that occurs when information from a previous survey informs the participant and influences their responses to future surveys.
This leads us to the question of how, if at all, margin of error can be applied in online sampling, when a random sample of everyone
online cannot be employed. There are three schools of thought.
1. Margin of error should never be applied to anything other than a probability based sample.
2. Margin of error can be applied in online sampling, but only to measure differences between the current sample and future
testing that is done within the same sample frame.
3. Online samples, though not probability based, are trying to measure a marketplace. Therefore, if a business believes that the
sample they created online accurately reflects the marketplace, they can use margin of error to project to that marketplace.
This leads us to the question of how, if at all, margin of error can be applied in online sampling, when a random sample of everyone
online cannot be employed. There are three schools of thought.
1. Margin of error should never be applied to anything other than a probability based sample.
2. Margin of error can be applied in online sampling, but only to measure differences between the current sample and future
testing that is done within the same sample frame.
3. Online samples, though not probability based, are trying to measure a marketplace. Therefore, if a business believes that the
sample they created online accurately reflects the marketplace, they can use margin of error to project to that marketplace.
Probability based online panels are online panels of survey participants that are recruited via probability samples using offline
means, such as address-based sampling and/or telephone. These panels are expensive to recruit and maintain
Probability based online panels are online panels of survey participants that are recruited via probability samples using offline
means, such as address-based sampling and/or telephone. These panels are expensive to recruit and maintain
A sample can be developed from frames that are fundamentally different (email lists, street addresses, etc.). Similarly, studies can be implemented with more than one mode of administration (online, phone, etc.).
By having these multiple modes of contact we can work to increase response rates for our study.
A sample can be developed from frames that are fundamentally different (email lists, street addresses, etc.). Similarly, studies can be implemented with more than one mode of administration (online, phone, etc.).
By having these multiple modes of contact we can work to increase response rates for our study.
Probability sampling is a
rigorous, scientific, and theory-based approach to sampling. Non-probability
sampling is less rigorous but faster and significantly less expensive.
Probability sampling is a
rigorous, scientific, and theory-based approach to sampling. Non-probability
sampling is less rigorous but faster and significantly less expensive.
A key part of almost every Market Research project involves the collection and/or processing of personal data (sometimes called PII for “personally identifiable information”), that is, data that can be used either directly or indirectly (e.g., by combining it with other data) to identify a specific individual. Protecting the identify of those who participate in research is recognized as a fundamental responsibility that must be met by every researcher.
A key part of almost every Market Research project involves the collection and/or processing of personal data (sometimes called PII for “personally identifiable information”), that is, data that can be used either directly or indirectly (e.g., by combining it with other data) to identify a specific individual. Protecting the identify of those who participate in research is recognized as a fundamental responsibility that must be met by every researcher.
In addition, researchers are required to strictly adhere to any relevant regional, national and local laws or regulations, and industry/professional
codes of conduct that may set a higher standard. They must review and comply with the national data protection and market research self-
regulatory requirements of each country where they plan to collect or process data, as there may be differences in how basic principles are
implemented within a specific country.
In addition, researchers are required to strictly adhere to any relevant regional, national and local laws or regulations, and industry/professional
codes of conduct that may set a higher standard. They must review and comply with the national data protection and market research self-
regulatory requirements of each country where they plan to collect or process data, as there may be differences in how basic principles are
implemented within a specific country.
Market researchers have an ethical obligation to
both the entity they are doing the research for
and the survey participants who are taking part
in the research.
Market researchers have an ethical obligation to
both the entity they are doing the research for
and the survey participants who are taking part
in the research.
Being transparent with the
research user about the sampling procedure
that will be used for the study is the most
important ethical obligation.
Being transparent with the
research user about the sampling procedure
that will be used for the study is the most
important ethical obligation.
Many of those regulations have to do with
protecting people’s PII (Personally Identi able
Information). PII, includes, without limitation, information such as name, address, and social security number.
Many of those regulations have to do with
protecting people’s PII (Personally Identi able
Information). PII, includes, without limitation, information such as name, address, and social security number.
Much of Market Research is conducted with the assurance to participants of work to protect people’s PII and keep their identities a secret.
Much of Market Research is conducted with the assurance to participants of work to protect people’s PII and keep their identities a secret.
A sample frame is the list of contacts that the sample will be drawn from
A sample frame is the list of contacts that the sample will be drawn from
The most important thing to consider regarding the sampling frame is coverage.
The most important thing to consider regarding the sampling frame is coverage.
Coverage can be defined as the proportion of the target population that exists within the sampling frame
Coverage can be defined as the proportion of the target population that exists within the sampling frame
Basically there are three criteria, related to different kinds of objectives that affect decisions
about sample size:
Precision - how accurate do your estimates (means, percentages) need to be?
Power - how much do you need to be able to identify differences when they exist?
Statistical modeling - what samples are required to run various statistical models?
Basically there are three criteria, related to different kinds of objectives that affect decisions
about sample size:
Precision - how accurate do your estimates (means, percentages) need to be?
Power - how much do you need to be able to identify differences when they exist?
Statistical modeling - what samples are required to run various statistical models?
To decide on a base size, we need to first understand what result will be significant to the marketing decision
and then choose a base size that makes sure that result would be statistically significant.
To decide on a base size, we need to first understand what result will be significant to the marketing decision
and then choose a base size that makes sure that result would be statistically significant.
We must also weigh the practicality of finding the individuals in our target population. If we are dealing with a population that
is very small, then the desire for a large sample will be logically impossible
We must also weigh the practicality of finding the individuals in our target population. If we are dealing with a population that
is very small, then the desire for a large sample will be logically impossible
If the research team defines their target too narrowly then they limit their ability to answer some key business questions
If the research team defines their target too narrowly then they limit their ability to answer some key business questions
if the research team defines the population too broadly, they may “waste” interviews and potentially marketing programs with potential customers who would not be very likely to use their product or service.
if the research team defines the population too broadly, they may “waste” interviews and potentially marketing programs with potential customers who would not be very likely to use their product or service.
The first step is to define the target population. The decision of what the target population should be is tied directly to the goal of the
research. When making the decision of what population to measure, we do not
want to exclude anyone who could be relevant to our study, but we also
do not want to include anyone not needed.
The first step is to define the target population. The decision of what the target population should be is tied directly to the goal of the
research. When making the decision of what population to measure, we do not
want to exclude anyone who could be relevant to our study, but we also
do not want to include anyone not needed.
It is important to have a discussion between the research user and research company carefully defining the business issues to address, and characteristics of the population that best relate to those issues. Defining the target group is a critical step and will impact the usefulness of results to achieve business decisions and goals.
It is important to have a discussion between the research user and research company carefully defining the business issues to address, and characteristics of the population that best relate to those issues. Defining the target group is a critical step and will impact the usefulness of results to achieve business decisions and goals.
Data weighting is a mathematical correction made to balance the sample and improve its validity.
Data weighting is a mathematical correction made to balance the sample and improve its validity.
Weighting factors below a one, weight the data down (we reduce their representation in the data).
Weighting factors below a one, weight the data down (we reduce their representation in the data).
Weighting factors
above a one, weight the data up (we increase their representation in the data).
Weighting factors
above a one, weight the data up (we increase their representation in the data).
When weighting by a single variable, the weighting factor
is easy to compute. Take the target value you want to achieve and divide it by the value you have.
When weighting by a single variable, the weighting factor
is easy to compute. Take the target value you want to achieve and divide it by the value you have.
The researcher usually decides if the data needs to be weighted and they come up with the weighting targets (also called the weighting scheme)
In probability sampling, it can be used as a way to control for non-response
Weighting can also be used in any situation where a quota is applied
We may be trying to balance the population to make
it look more like the population we are trying to study.
We could also be weighting samples to targets to make them more comparable.
Take care when weighting the data, as it is manipulation of the result.
Weighting methods should be communicated to the research buyer
and others using the data. For ethical reasons, weighting should not be done on the dependent variables, but rather than independent
variables.
The researcher usually decides if the data needs to be weighted and they come up with the weighting targets (also called the weighting scheme)
In probability sampling, it can be used as a way to control for non-response
Weighting can also be used in any situation where a quota is applied
We may be trying to balance the population to make
it look more like the population we are trying to study.
We could also be weighting samples to targets to make them more comparable.
Take care when weighting the data, as it is manipulation of the result.
Weighting methods should be communicated to the research buyer
and others using the data. For ethical reasons, weighting should not be done on the dependent variables, but rather than independent
variables.
Incidence is how often something occurs within the
population.
Incidence is how often something occurs within the population.
From an analytics point of view, incidence can be stated as how often something occurs within the population
From an analytics point of view, incidence can be stated as how often something occurs within the population
It is important to note that some types of convenience sampling do not have a measurable response rate, because there are not a known number of individuals invited to take part in the study.
It is important to note that some types of convenience sampling do not have a measurable response rate, because there are not a known number of individuals invited to take part in the study.
Non-response error becomes especially problematic when different groups in the population respond at different rates.
Non-response error becomes especially problematic when different groups in the population respond at different rates.
Researchers frequently use incentives to increase response rates, which are discussed in more detail in the next section
Researchers frequently use incentives to increase response rates, which are discussed in more detail in the next section
Well-designed and short questionnaires help keep refusal rates down
Well-designed and short questionnaires help keep refusal rates down
Participants are the lifeblood of the opinion and Market Research industry. They should be respected, valued, and protected. The
researcher’s interactions with participants should be characterized by honesty and integrity. The researcher’s ultimate objective is to
ensure that with every contact:
Participants will participate in surveys willingly
Participants are informed about the survey’s intentions
Participants will be satisfied with the survey experience
Participants will be willing to participate again
Participants are the lifeblood of the opinion and Market Research industry. They should be respected, valued, and protected. The
researcher’s interactions with participants should be characterized by honesty and integrity. The researcher’s ultimate objective is to
ensure that with every contact:
Participants will participate in surveys willingly
Participants are informed about the survey’s intentions
Participants will be satisfied with the survey experience
Participants will be willing to participate again
Incentives are given to entice people to take part in survey research and to complete the survey once it begins.
Incentives are given to entice people to take part in survey research and to complete the survey once it begins.