USA: +1(669)289-8683 
Sign in
Need The exercises at the the end of the PDF’s done by 3/12/13 3pm EST
Will need by Tuesday, The exercises are at the the end of the PDF done.
Section8
Ensure Valid Test and Survey Results Trough
Proper
Sample Size Estimation
Rhonda Knehans Drake
Associate Professor, New York University
Data Analytics, Interpretation and Reporting
Copyright © 2013
2
• Prior to conducting your surveys, marketing tests, or obtaining
your estimates we need to determine how large of a sample is
required.
• The more accuracy required in our estimates, the more we will
need to sample.
• However, we should only sample enough to obtain the level of
accuracy needed to help us make a decision.
• Some situations will require more accuracy than others.
• The key here is to determine how much accuracy you will
need to make the decision and only sample the required number
of names.
• We will discuss 4 sample size formulas in this section.
Introduction
3
We will learn the following formulas for Sample Size estimation
when concerned with:
1. The error associated with a single sample mean
2. The error associated with a single sample proportion or
response rate
3. The error associated with the difference between 2 sample
means or averages
4. The error associated with the difference between 2 sample
proportions or response rates
Sample Size Estimation
4
1. Sample Size Formula When Concerned with the Accuracy of a
Single Sample Mean.
n = Z2S2
E2
Where:
• S is an estimate of the standard deviation.
If unsure use 25% of your estimated average.
• E is the ± error you can tolerate
• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence level.
Single Sample Mean I
5
Example
Every year you conduct a survey to determine student satisfaction at NYU.
The scale is 1 -10 (1 = extremely unhappy and 10 = extremely Happy)
Last year the survey yielded an average and standard deviation of 7.5 and 1.
2.
Your goal for this year was to increase satisfaction by 1 full point to 8.5. If
unsuccessful you will not receive your bonus of $20,000.
You work hard at increasing satisfaction over the course of the year by holding
town hall meetings with students, putting suggestion boxes in all dorms,
upgrading housing conductions, enhancing the student union with free coffee
and snacks, etc.
Single Sample Mean II
6
Example (continue…)
Based on what you are hearing students say, you believe you will just meet
your goal of moving the needle one point to a new average of 8.5. But, you do
not believe it will be much higher than this value.
How many students should you survey to ensure a tight read here with a
maximum error of ± .1 with 99% confidence.
Single Sample Mean II
7
• The resulting sample size is:
Single Sample Mean III
n = (2.575)2(1.2)2
(0.1)2
= (6.63)*(1.44)
0.01
= 954
1.
2.
3.
8
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for Averages”
Select “One Sample”
Single Sample Mean IV
9
Input the required info.
Single Sample Mean V
10
See the answer.
Single Sample Mean VI
11
• Is a 99% confidence level appropriate here in your opinion? What
would be a more appropriate level to use?
• Had you felt you had moved the needle by almost 1.5 points instead of
only one point, could you have been able to sample less names and
tolerate more error while not putting your bonus in jeopardy? Explain.
Single Sample Mean VII
A 99% confidence interval is a bit extreme for a survey.
A 90% – 95% interval is more appropriate.
An error of + 0.5 is still tolerable.
n = (6.63)*(1.44)
(0.25)
n = 38.2
12
2. Sample Size Formula When Concerned with the Accuracy of a
Single Sample Proportion or Response Rate
n = Z2(p)·(1-p)
E2
Where:
• P is estimate of population proportion. You will base this
figure on prior experience.
• E is the ± error you can tolerate
• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence level.
Single Sample Proportion I
13
Example
You are about to test a new prospect list
You expect the response rate of this new list to be some where
around 1% based on your list brokers experience.
Your break-even (the lowest response rate you can tolerate) for
prospecting is .9%.
How many names should you sample so that should the response
rate come in at 1% you will be able to make a decision regarding
using the entire list?
Single Sample Proportion II
14
• So
n = (1.96)2·(.01)(.99) = (3.8416)(.0099) = 38,0
32
(.01-,009)2 .000001
• Do so will ensure should the response rate of the test come in at 1%
the resulting confidence interval will look like
.01 ± (1.96) ·√ (.01)(.99)/38,032 = .01 ± .000877
(.9%, 1.1%)
and we can make our decision with actually same worse case our
response rate is at or above break even!
Single Sample Proportion III
n = (1.96)2(0.01)·(0.99)
(0.001)2
n = (3.8416)(0.0099)
(0.000001)
n = 38,032
1.
2.
3.
15
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for
Percentages”
Select “One Sample”
Single Sample Proportion IV
16
Input the required info.
Single Sample Proportion V
17
See the answer.
Single Sample Proportion VI
18
3. Sample Size Estimation When Concerned with Accurately
Measuring the Difference Between 2 Means or Averages
n1 = n2 = Z2(S12 + S22)
d2
Where:
•d is the minimum difference you wish to detect as
significant should it be observed.
•S1 and S2 are estimates of the standard deviation
associated with each sample. In most cases you will use
the same estimate for both samples and if unsure, you
will use 25% of your expected average.
•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence
level
Difference Between Sample Means I
19
Example
You work for MasterCard and you wish to test an incentive for new
card members to increase spend over their first 3 months as a
card holder.
Based on a break even analysis you will need spend to increase by
$5 to cover the costs of your incentive (a few bonus sky miles that
is costing you about 10 cents per card member) Currently, new
card members spend on average $325 over the first 3 months with
a standard deviation of $2
5.
How many names should we sample to ensure that if we find the
test to yield a spending level of $330 we can read the results as
significant?
Difference Between Sample Means II
20
Difference Between Sample Means III
n1 = n2 = (1.96)2(252 + 252)
52
n1 = n2 = (3.8416)*(625 + 625)
25
n1 = n2 = 192
1.
2.
3.
21
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for
Averages”
Select “Test vs. Control”
Difference Between Sample Means IV
22
Input the required info.
Difference Between Sample Means V
23
See the answer.
Difference Between Sample Means VI
24
4. Sample Size Estimation When Concerned with Accurately
Measuring the Difference Between 2 Proportions or Response
Rates
n1 = n2 = (Z2) (p1)(1-p1) + (p2)(1-p2)
d2
Where:
•p1 is an estimate of one of the samples. You typically know the
response rate of your control group.
•d is difference you wish to detect as significant.
•p2 is p1 + d
•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99% confidence
level
Difference Between Sample Proportions I
25
Example
You are testing the addition of a premium to your control package.
Based on a break even analysis, you determine that you need two
additional order per thousand names mailed to break even with the
control.
Your control response rate is typically 1.00%.
How many names should we sample to ensure that if we obtain
two additional orders for our test package we will be able to detect
it as a significant increase.
Difference Between Sample Proportions II
26
Difference Between Sample Proportions III
n1 = n2 = (1.962) (0.01)(0.99) + (0.012)(0.988)
(0.002)2
n1 = n2 = (3.8416) (0.0099) + (0.011856)
(0.000004)
n1 = n2 = 20,894
1.
2.
3.
27
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for
Percentages”
Select “Test vs. Control”
Difference Between Sample Proportions IV
28
Input the required info.
Difference Between Sample Proportions V
29
See the answer.
Difference Between Sample Proportions VI
30
There are two break-evens we typically calculate as marketers:
• The break-even response rate required for a new list or product
test such that profit exactly offsets revenue – breakeven.
• The break-even for a new and more expensive format or creative
test such that net profit generated equals that of the control
format.
Break-Even Analysis
31
The break-even response rate for a new list or product test is the lowest
response rate you can tolerate and not lose any money. It is easily
calculated. It is the response rate such that:
Revenue – Costs = $0, or
(MQ x RR x PPP) – (MQ x PC) = $0
Where: MQ = Mail Quantity
RR = Response Rate
PPP = Profit Prior Promotional Costs
PC = Promotional Costs
Break-Even Response I
32
By rearranging the formula and solving for the RR, we find that the
break-even response rate is equal to:
RR = PC / PPP
Break-Even Response II
33
Consider the following example:
Assume you sell collector plates via direct mail. The average profit per
order before promotional costs is $55.00. You are planning to test a
new list on the market that will cost you $650.00 per 1,000 names
promoted.
What is the minimum response rate you must achieve on this list test in
order to break-eve and not lose any money?
And, if you typically have never seen a response rate above 1.00%
historically (regardless of how good the list is) do you recommend
testing the list?
Break-Even Response III
34
Break-even is calculated as:
Our decision to test this list is:
Break-Even Response IV
RR = PC / PPP
= 0.65/55
= 0.118 or 1.18%
No. we will most likely not see this level of response, so do not
test.
35
The increase in response that must be obtained on a new and more
expensive format test in order to generate at least the same profit as the
control format is the response rate such that:
Test Rev – Test Cost = Control Rev – Control Cost, or
(MQ*RRT*PPP) – (MQ*PCT) = (MQ*RRC*PPP) – (MQ*PCC)
Where: MQ = Mail Quantity
RRT = Test Response Rate
RRC = Control Response Rate
PPP = Profit Prior Promotional Costs
PCT = Test Promotional Costs
PCC = Control Promotional Costs
Increase in Response Required Break – Even
36
Consider the following example:
Your current control format is known to yield a 5% response rate and
has a promotional cost of $1 per piece. The profit prior promotional
costs per order is $30.
Your creative director has come up with a new format but it is quite
expensive. This new format will cost you $1.75 per piece to mail.
What is the increase in response required for this new format to break-
even wit the control format?
Increase in Response Required Break – Even
37
Break-even for the new format test is calculated as:
Increase in Response Required Break – Even
(1,000*RRT*30) – (1,000*1.75) = (1,000*0.05*30) – (1,000*1)
Divide both sides by 1,000
(30*RRT) – (1.75) = (1.5) – (1)
30*RRT = 2.25
RRT = 0.075 or 7.50%
1.
3.
2.
4.
5.
38
8.1 A researcher wants to determine a 95% confidence interval for the mean
number of hours that high school students spend doing homework per
week. She believes based on prior research that the average study time
per week is about 20 hours with a standard deviation of 7 hours. How
large a sample should the researcher select this year so that the estimate
will be within 1.5 hours of the population mean?
Do by hand and using the Plan-alyzer.
8.2 A U.S. government agency wants to estimate at a 95% confidence level
the mean speed for all cars traveling on Interstate Highway I-95. From a
previous study last year, the agency knows that the average is about 63
miles per hour with a standard deviation of 3.5 miles per hour. What
sample size should the agency choose this year so that the estimate will
be within 1.5 miles per hour of the population?
Do by hand and using the Plan-alyzer.
Section 8 Exercises I
39
8.3 Tony’s Pizza guarantees all pizza deliveries within 30 minutes of the
placement of orders. The Federal Trade Commission is concerned with
Tony’s advertisements and feels, based on customer complaints, that they
only meet their guarantee about 50% of the time. As such the FTC has
requested that Tony conduct a study. What sample size should the FTC
require of Tony’s to ensure the estimate obtained is within 2% of the true
percentage with 99% confidence?
Do by hand and using the Plan-alyzer.
8.4 A consumer agency wants to estimate the proportion of all drivers who
wear seat belts while driving. Assume that a preliminary study has shown
that 76% of drivers wear seat belts while driving. How large should the
sample size be so that a 99% confidence interval for the population
proportion has a maximum error of .03?
Do by hand and using the Plan-alyzer.
Section 8 Exercises II
40
8.5 The marketing director at ACME Direct is planning to test the addition of a
4-color flyer to his current direct mail control format. The 4-color flyer will
contain testimonials from famous celebrities praising the product being
offered. The control format is expected to yield a 4.50% response rate. In
order to cover the cost of the flyer (break-even) the test format will need to
yield an additional 3 orders per thousand names promoted. To ensure the
marketing director will be able to read the break-even response rate with
statistical significance, how large should each test panel be? Assume a
95% confidence level.
Do by hand and using the Plan-alyzer.
Section 8 Exercises III
41
8.6 Jet Music is a direct marketer of music packages covering all genres. Their active
music buyer market is shrinking and fast. There is much competition. Based on prior
mailings, one of Jet Music’s most popular CD packages, “Dance Till You Drop” is
known to yield a net response rate of 3.63% at a $9.97 price point. In an effort to
help increase response rates, the marketing manager has tested this title at a $1
lower price. Order intake is just beginning to come in for this test. After two weeks of
intake the net response rate is approaching 3.95% and climbing. At least 6 more
weeks of intake is expected. It is looking good.
The marketing managers boss is curious if the test of a $1 price decrease is at the
break-even response rate level yet. Calculate the minimum net order rate required to
break-even with the $1 price decrease test so that the marketing manager can
answer her boss.
Section 8 Exercises IV
42
8.7 You are the marketing manager at ACME Publishing. You test promoted a new
cookbook concept to a very large compiled list file and received a response rate of
2.54%. Based on an examination of age information, you notice that for those over 50
years of age you received a response rate of 4.30% (an index of 169 to total or a 69%
gain over total).
Assume the following:
– Cook book profit prior promotion costs = $9.92
– Promotion costs including list rental costs = $0.4217 per
Should you promote those on this complied list file that are over the age of 50 if your
goal of this promotion is to break even?
Section 8 Exercises IV
