Need The exercises at the the end of the PDF’s done by 3/12/13 3pm EST

Will need by Tuesday, The exercises are at the the end of the PDF done.

Section8

Ensure Valid Test and Survey Results Trough

Proper

Sample Size Estimation

Rhonda Knehans Drake

Associate Professor, New York University

Data Analytics, Interpretation and Reporting

Copyright © 2013

2

• Prior to conducting your surveys, marketing tests, or obtaining

your estimates we need to determine how large of a sample is

required.

• The more accuracy required in our estimates, the more we will

need to sample.

• However, we should only sample enough to obtain the level of

accuracy needed to help us make a decision.

• Some situations will require more accuracy than others.

• The key here is to determine how much accuracy you will

need to make the decision and only sample the required number

of names.

• We will discuss 4 sample size formulas in this section.

Introduction

3

We will learn the following formulas for Sample Size estimation

when concerned with:

1. The error associated with a single sample mean

2. The error associated with a single sample proportion or

response rate

3. The error associated with the difference between 2 sample

means or averages

4. The error associated with the difference between 2 sample

proportions or response rates

Sample Size Estimation

4

1. Sample Size Formula When Concerned with the Accuracy of a

Single Sample Mean.

n = Z2S2

E2

Where:

• S is an estimate of the standard deviation.

If unsure use 25% of your estimated average.

• E is the ± error you can tolerate

• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%

confidence level.

Single Sample Mean I

5

Example

Every year you conduct a survey to determine student satisfaction at NYU.

The scale is 1 -10 (1 = extremely unhappy and 10 = extremely Happy)

Last year the survey yielded an average and standard deviation of 7.5 and 1.

2.

Your goal for this year was to increase satisfaction by 1 full point to 8.5. If

unsuccessful you will not receive your bonus of $20,000.

You work hard at increasing satisfaction over the course of the year by holding

town hall meetings with students, putting suggestion boxes in all dorms,

upgrading housing conductions, enhancing the student union with free coffee

and snacks, etc.

Single Sample Mean II

6

Example (continue…)

Based on what you are hearing students say, you believe you will just meet

your goal of moving the needle one point to a new average of 8.5. But, you do

not believe it will be much higher than this value.

How many students should you survey to ensure a tight read here with a

maximum error of ± .1 with 99% confidence.

Single Sample Mean II

7

• The resulting sample size is:

Single Sample Mean III

n = (2.575)2(1.2)2

(0.1)2

= (6.63)*(1.44)

0.01

= 954

1.

2.

3.

8

• Let’s do the previous

example again but using

the Plan-alyzer.

1.

2.

3.

Select the tab “Table of

Calculators”

Select “Sample Size

Calculators for Averages”

Select “One Sample”

Single Sample Mean IV

9

Input the required info.

Single Sample Mean V

10

See the answer.

Single Sample Mean VI

11

• Is a 99% confidence level appropriate here in your opinion? What

would be a more appropriate level to use?

• Had you felt you had moved the needle by almost 1.5 points instead of

only one point, could you have been able to sample less names and

tolerate more error while not putting your bonus in jeopardy? Explain.

Single Sample Mean VII

A 99% confidence interval is a bit extreme for a survey.

A 90% – 95% interval is more appropriate.

An error of + 0.5 is still tolerable.

n = (6.63)*(1.44)

(0.25)

n = 38.2

12

2. Sample Size Formula When Concerned with the Accuracy of a

Single Sample Proportion or Response Rate

n = Z2(p)·(1-p)

E2

Where:

• P is estimate of population proportion. You will base this

figure on prior experience.

• E is the ± error you can tolerate

• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%

confidence level.

Single Sample Proportion I

13

Example

You are about to test a new prospect list

You expect the response rate of this new list to be some where

around 1% based on your list brokers experience.

Your break-even (the lowest response rate you can tolerate) for

prospecting is .9%.

How many names should you sample so that should the response

rate come in at 1% you will be able to make a decision regarding

using the entire list?

Single Sample Proportion II

14

• So

n = (1.96)2·(.01)(.99) = (3.8416)(.0099) = 38,0

32

(.01-,009)2 .000001

• Do so will ensure should the response rate of the test come in at 1%

the resulting confidence interval will look like

.01 ± (1.96) ·√ (.01)(.99)/38,032 = .01 ± .000877

(.9%, 1.1%)

and we can make our decision with actually same worse case our

response rate is at or above break even!

Single Sample Proportion III

n = (1.96)2(0.01)·(0.99)

(0.001)2

n = (3.8416)(0.0099)

(0.000001)

n = 38,032

1.

2.

3.

15

• Let’s do the previous

example again but using

the Plan-alyzer.

1.

2.

3.

Select the tab “Table of

Calculators”

Select “Sample Size

Calculators for

Percentages”

Select “One Sample”

Single Sample Proportion IV

16

Input the required info.

Single Sample Proportion V

17

See the answer.

Single Sample Proportion VI

18

3. Sample Size Estimation When Concerned with Accurately

Measuring the Difference Between 2 Means or Averages

n1 = n2 = Z2(S12 + S22)

d2

Where:

•d is the minimum difference you wish to detect as

significant should it be observed.

•S1 and S2 are estimates of the standard deviation

associated with each sample. In most cases you will use

the same estimate for both samples and if unsure, you

will use 25% of your expected average.

•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%

confidence

level

Difference Between Sample Means I

19

Example

You work for MasterCard and you wish to test an incentive for new

card members to increase spend over their first 3 months as a

card holder.

Based on a break even analysis you will need spend to increase by

$5 to cover the costs of your incentive (a few bonus sky miles that

is costing you about 10 cents per card member) Currently, new

card members spend on average $325 over the first 3 months with

a standard deviation of $2

5.

How many names should we sample to ensure that if we find the

test to yield a spending level of $330 we can read the results as

significant?

Difference Between Sample Means II

20

Difference Between Sample Means III

n1 = n2 = (1.96)2(252 + 252)

52

n1 = n2 = (3.8416)*(625 + 625)

25

n1 = n2 = 192

1.

2.

3.

21

• Let’s do the previous

example again but using

the Plan-alyzer.

1.

2.

3.

Select the tab “Table of

Calculators”

Select “Sample Size

Calculators for

Averages”

Select “Test vs. Control”

Difference Between Sample Means IV

22

Input the required info.

Difference Between Sample Means V

23

See the answer.

Difference Between Sample Means VI

24

4. Sample Size Estimation When Concerned with Accurately

Measuring the Difference Between 2 Proportions or Response

Rates

n1 = n2 = (Z2) (p1)(1-p1) + (p2)(1-p2)

d2

Where:

•p1 is an estimate of one of the samples. You typically know the

response rate of your control group.

•d is difference you wish to detect as significant.

•p2 is p1 + d

•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99% confidence

level

Difference Between Sample Proportions I

25

Example

You are testing the addition of a premium to your control package.

Based on a break even analysis, you determine that you need two

additional order per thousand names mailed to break even with the

control.

Your control response rate is typically 1.00%.

How many names should we sample to ensure that if we obtain

two additional orders for our test package we will be able to detect

it as a significant increase.

Difference Between Sample Proportions II

26

Difference Between Sample Proportions III

n1 = n2 = (1.962) (0.01)(0.99) + (0.012)(0.988)

(0.002)2

n1 = n2 = (3.8416) (0.0099) + (0.011856)

(0.000004)

n1 = n2 = 20,894

1.

2.

3.

27

• Let’s do the previous

example again but using

the Plan-alyzer.

1.

2.

3.

Select the tab “Table of

Calculators”

Select “Sample Size

Calculators for

Percentages”

Select “Test vs. Control”

Difference Between Sample Proportions IV

28

Input the required info.

Difference Between Sample Proportions V

29

See the answer.

Difference Between Sample Proportions VI

30

There are two break-evens we typically calculate as marketers:

• The break-even response rate required for a new list or product

test such that profit exactly offsets revenue – breakeven.

• The break-even for a new and more expensive format or creative

test such that net profit generated equals that of the control

format.

Break-Even Analysis

31

The break-even response rate for a new list or product test is the lowest

response rate you can tolerate and not lose any money. It is easily

calculated. It is the response rate such that:

Revenue – Costs = $0, or

(MQ x RR x PPP) – (MQ x PC) = $0

Where: MQ = Mail Quantity

RR = Response Rate

PPP = Profit Prior Promotional Costs

PC = Promotional Costs

Break-Even Response I

32

By rearranging the formula and solving for the RR, we find that the

break-even response rate is equal to:

RR = PC / PPP

Break-Even Response II

33

Consider the following example:

Assume you sell collector plates via direct mail. The average profit per

order before promotional costs is $55.00. You are planning to test a

new list on the market that will cost you $650.00 per 1,000 names

promoted.

What is the minimum response rate you must achieve on this list test in

order to break-eve and not lose any money?

And, if you typically have never seen a response rate above 1.00%

historically (regardless of how good the list is) do you recommend

testing the list?

Break-Even Response III

34

Break-even is calculated as:

Our decision to test this list is:

Break-Even Response IV

RR = PC / PPP

= 0.65/55

= 0.118 or 1.18%

No. we will most likely not see this level of response, so do not

test.

35

The increase in response that must be obtained on a new and more

expensive format test in order to generate at least the same profit as the

control format is the response rate such that:

Test Rev – Test Cost = Control Rev – Control Cost, or

(MQ*RRT*PPP) – (MQ*PCT) = (MQ*RRC*PPP) – (MQ*PCC)

Where: MQ = Mail Quantity

RRT = Test Response Rate

RRC = Control Response Rate

PPP = Profit Prior Promotional Costs

PCT = Test Promotional Costs

PCC = Control Promotional Costs

Increase in Response Required Break – Even

36

Consider the following example:

Your current control format is known to yield a 5% response rate and

has a promotional cost of $1 per piece. The profit prior promotional

costs per order is $30.

Your creative director has come up with a new format but it is quite

expensive. This new format will cost you $1.75 per piece to mail.

What is the increase in response required for this new format to break-

even wit the control format?

Increase in Response Required Break – Even

37

Break-even for the new format test is calculated as:

Increase in Response Required Break – Even

(1,000*RRT*30) – (1,000*1.75) = (1,000*0.05*30) – (1,000*1)

Divide both sides by 1,000

(30*RRT) – (1.75) = (1.5) – (1)

30*RRT = 2.25

RRT = 0.075 or 7.50%

1.

3.

2.

4.

5.

38

8.1 A researcher wants to determine a 95% confidence interval for the mean

number of hours that high school students spend doing homework per

week. She believes based on prior research that the average study time

per week is about 20 hours with a standard deviation of 7 hours. How

large a sample should the researcher select this year so that the estimate

will be within 1.5 hours of the population mean?

Do by hand and using the Plan-alyzer.

8.2 A U.S. government agency wants to estimate at a 95% confidence level

the mean speed for all cars traveling on Interstate Highway I-95. From a

previous study last year, the agency knows that the average is about 63

miles per hour with a standard deviation of 3.5 miles per hour. What

sample size should the agency choose this year so that the estimate will

be within 1.5 miles per hour of the population?

Do by hand and using the Plan-alyzer.

Section 8 Exercises I

39

8.3 Tony’s Pizza guarantees all pizza deliveries within 30 minutes of the

placement of orders. The Federal Trade Commission is concerned with

Tony’s advertisements and feels, based on customer complaints, that they

only meet their guarantee about 50% of the time. As such the FTC has

requested that Tony conduct a study. What sample size should the FTC

require of Tony’s to ensure the estimate obtained is within 2% of the true

percentage with 99% confidence?

Do by hand and using the Plan-alyzer.

8.4 A consumer agency wants to estimate the proportion of all drivers who

wear seat belts while driving. Assume that a preliminary study has shown

that 76% of drivers wear seat belts while driving. How large should the

sample size be so that a 99% confidence interval for the population

proportion has a maximum error of .03?

Do by hand and using the Plan-alyzer.

Section 8 Exercises II

40

8.5 The marketing director at ACME Direct is planning to test the addition of a

4-color flyer to his current direct mail control format. The 4-color flyer will

contain testimonials from famous celebrities praising the product being

offered. The control format is expected to yield a 4.50% response rate. In

order to cover the cost of the flyer (break-even) the test format will need to

yield an additional 3 orders per thousand names promoted. To ensure the

marketing director will be able to read the break-even response rate with

statistical significance, how large should each test panel be? Assume a

95% confidence level.

Do by hand and using the Plan-alyzer.

Section 8 Exercises III

41

8.6 Jet Music is a direct marketer of music packages covering all genres. Their active

music buyer market is shrinking and fast. There is much competition. Based on prior

mailings, one of Jet Music’s most popular CD packages, “Dance Till You Drop” is

known to yield a net response rate of 3.63% at a $9.97 price point. In an effort to

help increase response rates, the marketing manager has tested this title at a $1

lower price. Order intake is just beginning to come in for this test. After two weeks of

intake the net response rate is approaching 3.95% and climbing. At least 6 more

weeks of intake is expected. It is looking good.

The marketing managers boss is curious if the test of a $1 price decrease is at the

break-even response rate level yet. Calculate the minimum net order rate required to

break-even with the $1 price decrease test so that the marketing manager can

answer her boss.

Section 8 Exercises IV

42

8.7 You are the marketing manager at ACME Publishing. You test promoted a new

cookbook concept to a very large compiled list file and received a response rate of

2.54%. Based on an examination of age information, you notice that for those over 50

years of age you received a response rate of 4.30% (an index of 169 to total or a 69%

gain over total).

Assume the following:

– Cook book profit prior promotion costs = $9.92

– Promotion costs including list rental costs = $0.4217 per

Should you promote those on this complied list file that are over the age of 50 if your

goal of this promotion is to break even?

Section 8 Exercises IV