Business Question

Introduction: This portfolio work project will help you evaluate the results of advanced analytics projects and identify potential errors or issues with the application of the methods or the alignment between methods and business problems, and propose possible solutions.

Scenerio: Recently, the executive team in your organization has made some major business decisions based on the results of the analytics group’s projects and has seen mixed results. Some of the decisions made based on the analytics group’s results have been successful, but others have been huge failures and very costly for the organization. The executives are now questioning their trust in this group’s ability to analyze data appropriately to address business problems. They have put the analytics group into a probationary period of 90 days, after which they will decide whether to retain the advanced analytics group, or to disband the group. As head of the marketing department, they have placed you in charge of this group’s performance for the next 90 days, and have asked that you review all work that comes from this department prior to it being sent to the executives.

You will apply this scenario to either Option A or Option B described in the Requirements section.

Your role: You are the head of the marketing department and have been placed in charge of the analytics group for their 90-day probationary period. You are now responsible for the quality, accuracy, and reliability of the results reported out from the analytics group, and are responsible for reviewing all work prior to it being presented to the executive committee.

Requirements: Choose one of the following options:

OPTION A

An analyst has conducted an A/B test to compare the revenue per visitor for the same banner ad media run on two different websites. The results of this analysis are included here. You have decided to evaluate the results and use it as an example in a training session for the analytics group on how to conduct a critical evaluation of results from an analysis.

Resources:

Option A data [XLSX]

Download Option A data [XLSX]

.

Option A results [DOCX]

Download Option A results [DOCX]

.

OPTION B

An analyst has run a regression analysis to attempt to predict whether someone will pre-pay their car loan (resulting in a loss of interest revenue for the organization). The results of this analysis are included here. You have decided to evaluate the results and use the evaluation as an example in a training session for the analytics group on how to conduct a critical evaluation of results from an analysis.

Resources:

Option B data [XLSX]

Download Option B data [XLSX]

.

Option B results [DOCX]

Download Option B results [DOCX]

.

FOR EITHER OPTION A OR B

Using the data provided for the option you have selected, evaluate the results of the analysis, paying attention to the alignment of the method selected to the business problem being addressed, the testing of assumptions required to use the selected method, and potential issues with generalizations, and the usability of the results.

Yahoo
CNN
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
20
49
24
20
32
23
18
47
20
9
15
15
45
22
2
4
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
8
86
55
63
66
104
58
104
119
73
56
81
40
41
93
27
32
13
37
11
28
1
23
50
31
9
9
52
36
49
9
47
78
102
89
116
56
Running head: YAHOO AND CNN A/B TEST
Yahoo and CNN A/B Test
Analyst One
Company ABCD
1
YAHOO AND CNN A/B TEST
2
Abstract
Upon conducting an a/b test to determine whether running the company’s banner advertisement
on Yahoo or on CNN was more effective at bringing in revenue, it was found that there is a
significant difference in the mean purchase amount by customers who clicked on the banner
advertisement on Yahoo versus CNN.
Keywords: Yahoo, CNN, a/b test, hypothesis testing
YAHOO AND CNN A/B TEST
3
Yahoo and CNN A/B Test
An a/b test was conducted to determine which site brings in more revenue from the
company’s banner advertisement. The intention is to reduce the number of sites the company has
to manage advertising relationships with, without using a third-party service provider to manage
the sites and advertisement for the company. Below are the results of the analysis.
Yahoo vs CNN Test
The test was conducted over a 1-week period and resulted in a total of 63 clicks on the
Yahoo banner advertisement, and 51 clicks on the CNN banner advertisement. Out of these, 31
of each site’s clicks did not result in a purchase, so the purchase amount is 0, and 32 on Yahoo
and 20 on CNN did result in a purchase (i.e. a non-zero purchase amount).
Hypothesis Test
The results from the test of hypothesis (a/b test) for a difference in the average purchase amount
between Yahoo and CNN are included below.
Hypotheses.
The null hypothesis is that the average purchase amount between Yahoo and CNN is the
same.
The alternative hypothesis is that the average purchase amount between Yahoo and CNN
is different.
The two hypotheses that were tested during this process were:
Ho: 𝜇𝑌𝑎ℎ𝑜𝑜 = 𝜇𝐶𝑁𝑁
Ha: 𝜇𝑌𝑎ℎ𝑜𝑜 ≠ 𝜇𝐶𝑁𝑁
YAHOO AND CNN A/B TEST
4
Appropriate Statistical Test.
The test was conducted using Microsoft Excel’s Data Analysis Toolpak Add-In and the
“t-Test: Two-Sample Assuming Equal Variances” option.
Conducting the Statistical Test.
The test results are included in the table below. Histograms and box plots the two sets of
data, and a scatterplot, are also included as a check of the assumptions.
t-Test: Two-Sample Assuming Equal
Variances
Mean
Variance
Observations
Pooled Variance
Hypothesized Mean Difference
df
t Stat
P(T