Test Questions

Question 1

See the image below:

Answer 1

A: Marketing Mix Modeling

Question 2

Answer 2

A

Question 3

Answer 3

D

Question 4

Answer 4

C

Question 5

Answer 5

D

Question 6

Answer 6

B

Question 7

Answer 7

C

Question 8

Answer 8

A & C

Question 9

See the image below:

Answer 9

Location

Question 10

An advertiser is running a Conversion Lift test on a new media platform. In discussing how the campaign will be measured, the media platform’s team says that it will compare the conversion rates for the test group exposed to ads vs. a pre-selected control group. The platform does have the ability to run a public service announcement instead, but feels that its methodology is valid.

What concern should the analyst have with the test design?

Randomization is required for valid experimental design

Pre and post measurements will provide a more accurate assessment

There may be contamination between the test and control groups

An A/B test will provide a more accurate assessment

Answer 10

Randomization is required for valid experimental design

Question 11

An advertiser’s primary product offering is a series of subscription boxes. The advertiser wants to increase user retention. Approximately 25% of customers who buy a three-month subscription do NOT buy another subscription the following year.

A key objective is to reduce the churn rate by 5%. A data scientist develops a model to identify users who have a high probability of churning and to create exclusive offers designed to entice these users to buy another subscription.

Which type of model should the data scientist use?

Logistic regression

Linear regression

Multinomial regression

Support vector regression

Answer 11

Logistic regression

Question 12

An online shoe brand ran two Conversion Lift studies. Test 1 resulted in the control group showing a 20% conversion rate while the test group showed a 25% conversion rate. The results were significant at 60%. Test 2 resulted in a 20% conversion rate in the control group while the test group showed a 30% conversion rate. These results were 95% significant.

What can the analyst conclude?

Test 2 results are more reliable than Test 1

Test 1 results are more reliable than Test 2

Both tests are equally reliable

A comparison cannot be made between Test 1 and Test 2

Answer 12

Test 2 results are more reliable than Test 1

Question 13

A brand runs a multi-cell experiment to confirm whether its campaign is generating sales lift. The results were as follows.

• Cell A
  
  • sales lift = 4.0%
  • 90% confidence interval = (0.010,0.070)
• Cell B
  
  • sales lift = 4.5%
  • 90% confidence interval = (-0.010, 0.084)
• Difference between Cell A and Cell B
  
  • sales lift = -0.5%
  • 90% confidence interval = (-0.023, 0.001)

What is the correct interpretation of the results?

Cell A performed positively, Cell B did not perform positively, Cell A performed better than Cell B

Cell A performed positively, Cell B performed positively, Cell B performed better than Cell A

Cell A performed positively, Cell B performed positively, and it is not possible to tell which was better

Cell A performed positively, Cell B did not perform positively, and it is not possible to tell which was better

Answer 13

Cell A performed positively, Cell B did not perform positively, and it is not possible to tell which was better

The correct answer was “Cell A performed positively, Cell B did not perform positively, and it is not possible to tell which was better”.

A confidence interval containing zero means we’re uncertain whether there is an effect. Having zero in the confidence interval implies that the effect could have a positive or negative effect on the outcome of interest. This means we can’t conclude that cell B resulted in a positive outcome, nor can we conclude there was any difference between cells A and B.

Question 14

Refer to the chart.

An analyst in a technology company aggregates 42 Brand Lift studies that contained favorability questions. The analyst plots these studies against the engagement rate that the ads received. The engagement rate is a summation of likes, comments, and shares over the total reach in each campaign.
Each dot in the chart is a separate campaign.

What is the correct interpretation from this chart regarding the correlation between the engagement rates and favorability lift?

No correlation

Weak positive correlation

Weak negative correlation

Strong positive correlation

Answer 14

No correlation

Question 15

An automotive manufacturer launches a campaign to sell its newest model. The manufacturer uses a last-click attribution model with a 90-day conversion window to evaluate its media investment.

The attribution results show lower-than-expected online conversions on Facebook. The manufacturer needs to validate these results.

In order to validate the results, what should they be compared to?

The results from last-click attribution model with a conversion window of 30 days

The number of offline conversions

The results from Facebook Data-Driven Attribution

The results from Facebook Brand Lift tests that were run during the campaign

Answer 15

The results from Facebook Data-Driven Attribution

Question 16

An ecommerce advertiser sells activewear. The advertiser needs to assess which proportion of the advertising budget should be spent on re-targeting previous buyers versus targeting consumers who visited its website but did NOT make a purchase. To make this decision, the advertiser needs to measure the proportion of sales generated by each targeting strategy to learn which strategy generates the largest proportion of additional sales.

The advertiser pauses its usual campaigns and sets up an A/B test using two ad sets. Both ad sets are optimized for purchases. The only difference is the targeting strategy used. The advertiser runs the ad sets for seven days. The A/B test results are based on attributed conversions using a last-touch attribution model to identify the winning strategy.

This measurement approach does NOT accurately measure causality.

What is causing this measurement issue?

The test should be set up to use an even credit attribution model

The test should be set up to prevent contamination to the test group

The test should run for a minimum of three weeks to collect adequate data

The test needs to run randomized test and control groups

Answer 16

The test needs to run randomized test and control groups

The correct answer was “The test needs to run randomized test and control groups”.

Conducting an A/B test will not measure causality; instead, an RCT (lift test) should be used to more accurately measure tthe causal impact of each strategy.

Question 17

An online fashion retailer wants to test the Video Views objective against the Brand Awareness objective. The goal is to see which objective generates better purchase intent among the target audience.

Which method should the retailer use to achieve this?

Multi-cell Conversion Lift test to test Video Views versus Brand Awareness

A/B test to measure Video Views versus Brand Awareness

Multi-cell Brand Lift test to test Purchase Intent with Brand Awareness versus Video Views

A/B test to measure estimated ad recall for each objective

Answer 17

Multi-cell Brand Lift test to test Purchase Intent with Brand Awareness versus Video Views

Question 18

An ecommerce company wants to understand the impact of reach on their incremental ROAS results when running randomized control trial experiments using their ads.

The company gathers their historical campaign performance and creates a scatterplot to compare reach and ROAS for each campaign run in the past year. Their marketing strategy has changed frequently over the past year, with adjustments to targeting, bidding strategies and optimization metrics. The marketing team decides to increase reach as much as possible to increase their incremental ROAS.

Refer to the chart.

What conclusion should the analytics team make in respect to these findings?

These results are correlative, not causal

There is not a discernible relationship between reach and ROAS

Reach should be optimized for 2.5 million unique users because that drove the highest incremental ROAS

Higher reach causes higher ROAS

Answer 18

These results are correlative, not causal

Question 19

An advertiser is planning on running a campaign that includes TV, Facebook and search, with the goal of increasing sales. The attribution results from that campaign will determine how they run their future campaigns.

What should the advertiser do based on the attribution results?

Reallocate budget to the channel driving the highest number of conversions

Stop spend entirely on the channel with the lowest number of conversions

Allocate budget evenly across all three channels

Conduct a multi-cell Brand Lift test to validate the attribution results

Answer 19

Reallocate budget to the channel driving the highest number of conversions

Question 20

A cruise brand advertises on Facebook, Instagram and YouTube. The brand is evaluating performance of these channels in Ads Manager and Google Analytics to determine how to allocate its increased budget for the upcoming season. The brand suspects the impact of Facebook platforms is over-represented in Ads Manager and under-represented in Google Analytics.

How should the company measure the causal contribution of the Facebook platforms accurately?

Adopt multi-touch attribution models for all platforms

Use Ads Manager results to evaluate Facebook platforms

Use Google Analytics to evaluate Facebook platforms

Run an ad-account level randomized control trial on Facebook

Answer 20

Run an ad-account level randomized control trial on Facebook

The correct answer was “Run an ad-account level randomized control trial on Facebook”.

Google Analytics can only show you correlated measures, whereas running a randomized control trial (RCT) can show you causal impact.

Question 21

A TV advertiser needs to attract more sports fans to tune into their upcoming live events. According to a global index survey, 90% of sports fans online report that they use another device while watching TV. More than 50% of sports fans reported that their top activities were using social media and chatting with friends.

To maximize ad reach, which hypothesis should the advertiser test?

Placing ads on popular social media platforms is the most cost efficient way to generate conversions

Placing ads on top sports channels is the most cost efficient way to generate reach

Messaging apps provide incremental conversions

Social media platforms provide incremental reach to TV

Answer 21

Social media platforms provide incremental reach to TV

Question 22

A large gaming company wants to know if its Facebook campaign is increasing purchase consideration due to ad exposure on Facebook.

What measurement solution should be used?

Facebook Conversion Lift

Facebook Brand Lift

Facebook A/B test

Facebook Analytics

Answer 22

Facebook Brand Lift

Question 23

A financial services company is launching a month-long Facebook campaign to promote its new credit card. The audience strategy is to reach people who have previously applied for one of the company’s current credit cards but did not get approved.

The campaign has been set up with a single-cell Conversion Lift test to determine how effective the campaign is in getting people to apply. The company would like a post-campaign report that highlights the total budget spent and number of approved applications.

What should the analyst recommend for the campaign?

The schedule of the campaign should be longer than one month

A multi-cell Conversion Lift test should be used instead

The KPI should be the number of applications submitted

The KPI should be the number of people reached

Answer 23

The KPI should be the number of applications submitted

The correct answer was “The KPI should be the number of applications submitted”.

Since the goal of the campaign is for people to apply for the new credit card, the metric the company should focus on is the number of applications submitted.

Question 24

An online food brand that ships high-quality meat has a loyal audience of people, ages 35+. The brand conducts market research and learns that younger consumers are starting to shop for food online more frequently, but are extremely health conscious. The brand is worried they will struggle to be top-of-mind among this new prospective young audience.

The brand runs a multi-cell Brand Lift test for eight weeks to compare how its current creative performs against people, ages 35+ versus people, ages 18-34. Both cells had the same budget and used the same optimization.

The results are in the image.

What action should the client take to improve performance with the younger audience?

Develop new creative

Keep serving the existing creative

Rerun the test

Increase the spend

Answer 24

Develop new creative

Question 25

An advertiser is running monthly individual conversion lift tests across multiple digital channels.

The latest lift tests showed the channels A, B and C had statistically significant positive lift results whereas channel D did not have statistically significant results. The tests were all run targeting the advertiser’s previous customers (CRM). They also hope to run future campaigns similarly using multiple channels with new customers. One of the blind spots the digital marketers want to clear up is how much overlap there is between the digital channels and whether efficiencies can be gained as a result of that finding.

What should the analyst pursue to most effectively answer the digital marketing team’s question?

Use multi-touch attribution to identify where multiple platforms are contributing to conversions

Use last-touch attribution to avoid double-counting conversions that were captured on multiple platforms

Run future tests where audiences are not overlapping so that there is no ability to see ads on multiple platforms

Identify correlations in daily sales data between channels using a marketing mix model

Answer 25

Use multi-touch attribution to identify where multiple platforms are contributing to conversions

Question 26

A shoe brand recently ran a multi-cell Conversion Lift test to compare evergreen digital marketing campaigns against a new promotional campaign using dynamic ads. The hypothesis was that the dynamic ads campaign would outperform the evergreen campaign on conversion and sales lift. This test ran for six weeks during the holiday periods between November and December.

Test results:

• The dynamic ads campaign created a 4% lift in overall conversions and $250,000 in sales
• The evergreen campaign created a 1.5% lift in overall conversions and $100,000 in sales
• The dynamic ads campaign resulted in a $120.43 cost per incremental conversion and a 1.5x ROAS
• The evergreen campaign resulted in a $35.45 cost per incremental conversion and 2x ROAS

What should an analyst recommend as next steps to use these insights?

Shift total budget to the evergreen digital campaign

Continue running both the dynamic ads and evergreen campaigns, but reduce spend on dynamic ads

Reduce spend from both campaigns because they did not result in efficient lift

Shift total budget to the dynamic ads campaign

Answer 26

Continue running both the dynamic ads and evergreen campaigns, but reduce spend on dynamic ads

Question 27

A large quick-serve restaurant brand launches a new holiday themed dessert. The brand runs an always-on campaign on Facebook to maintain its brand awareness. They launch a two-week campaign to drive traffic to a landing page with a discount offer for the dessert during the holiday period. The discount campaign is running the two weeks prior to the holiday period.

The brand runs Brand Lift tests every quarter to measure the impact of their always-on campaign and are planning to run a Conversion Lift test to see the incremental coupon downloads generated by the discount-offer campaign. The Conversion Lift test will be set up as a nested test within the Brand Lift test setup.

Which rationale should the brand use to document that the Conversion Lift test results will be unbiased?

The nested setup accounts for possible contamination across studies

The discount-offer campaign is unlikely to reach the same audience as the always-on campaign

The Conversion Lift test is measuring a different KPI than the Brand Lift test

The Brand Lift test only polls 1000 respondents and is unlikely to be impacted by the discount-offer campaign

Answer 27

The nested setup accounts for possible contamination across studies

Question 28

Sales of a dine-in restaurant brand continue to decline. This trend occurs across the dine-in category as new competitors enter the market.

To help bring more customers to the restaurant locations, the brand runs a series of promotional campaigns that feature existing products at a discounted price. The campaign has successfully brought customers into the locations. However, customers tend to order the promoted items only. As a result, the average spend size remains very low, which reduces profit.

Which insight should an analyst provide to the brand?

The brand may have an issue with value perception

The customers do not want a dine-in experience

The competitors have better promotions in market

The customers are not having a good customer experience

Answer 28

The brand may have an issue with value perception

Question 29

An insurance company needs to use data to identify the appropriate communication strategy for its brands. Recently, the company commissioned a creative agency to produce longer and more sophisticated video ads. The company is currently focusing on the amount of additional leads it is able to generate. Each evaluated strategy consists of a set of different combinations of reach levels, ad formats and campaign costs. The company has already run a large number of lift tests to measure additional conversions so it also has the average lift scores for each strategy.

The average outcomes of three strategies standout from the rest:

• Short video formats: 1 lift point, 5 million people reached and an average cost of 25,000 USD
• Mix of short videos and static formats: 0.5 lift points, 14 million people reached and an average cost of 30,000 USD
• Long video formats: 2 lift points, 3 million people reached and an average cost of 50,000 USD

Which action should be taken to improve the communication strategy based on the available data?

Use long videos and increase the investment

Use short videos instead of longer ones

Use a mix of short videos with static formats

Use long videos and maintain the current level of investment

Answer 29

Use a mix of short videos with static formats

The correct answer was “Use a mix of short videos with static formats”.

The mix of the short videos and static images provides the most cost-efficient ROI.

Question 30

Refer to the tables.

Which query will retrieve all results for movies released only after the year 2000?

SELECT release_year FROM movies WHERE movies > 2000;

SELECT movies FROM movies WHERE release_year > 2000;

SELECT * FROM movies WHERE release_year > 2000;

SELECT release_year > 2000 FROM movies;

Answer 30

SELECT * FROM movies WHERE release_year > 2000

Question 31

An agency for a film studio has historically focused on driving awareness and consideration for upcoming movie releases. They currently run a mixture of brand and direct response campaigns. Sixty percent of sales are from in-app and online ticket purchases, and 40% are offline.

To evaluate the success of their current strategy, the agency runs a Brand Lift test to poll the client’s audience on consideration, purchase intent, and ad recall. The test revealed strong results with a 6% lift in consideration and 5% lift in purchase intent. They want to use this evidence to request increased investment from their client.

The client’s analyst argues that this does not provide adequate evidence that the strategy drives sales.

Which test should be run to make the strongest case that the direct response campaigns are driving incremental sales?

Account-level Conversion Lift including all brand and direct response campaigns

Campaign-level Conversion Lift including only direct response campaigns

Campaign-level Brand Lift including only direct response campaigns

Campaign-level Conversion Lift including brand and direct response campaigns

Answer 31

Campaign-level Conversion Lift including only direct response campaigns

The correct answer was “Campaign-level Conversion Lift including only direct response campaigns”.

Since the goal is to understand the effectiveness of direct response campaigns only, a Conversion Lift test of direct response campaigns would be the most appropriate test to run since it measures incremental sales and conversions caused by the campaign.

Question 32

An automotive brand runs a $1,000,000 branding campaign on Facebook for its newest model. The goal is to encourage men and women, ages 18+ to believe that it is “the car for everyone.” A single-cell Brand Lift test runs and obtains 150 responses for each test and control group. The results are as follows:

• 45% of the test group respondents agree with the statement that the car is “the car for everyone”
• 40% of the control group respondents agree with the statement that the car is “the car for everyone”
• The results are statistically significant at the 80% confidence level

What is a potential action that the brand could take to increase power of the Brand Lift test?

Increase the target sample size

Run a multi-cell Brand Lift test

Poll men and women separately

Decrease the target sample size

Answer 32

Increase the target sample size