USA: +1(669)289-8683 
Sign in
U2D2 – Levels of Measurement and Creating Survey Questions
After reading about levels of measurement in Chapter 4, search the Internet or the Research Library for examples of questions that measure each type of
variable.
Complete the following in your discussion post:
• Use the construct of yearly income to write a survey question, along with the possible answers, for each type of variable level of measurements; nominal, ordinal, interval, and ratio.
• Clearly label which survey question is nominal, ordinal, interval, or ratio.
Readings
• Program Evaluation Life Cycle – read transcript
Use your Program Evaluation and Performance Measurement text to complete the following:
• Read Chapter 4.
INTRODUCTION – Unit 2 – Program Evaluation: Quantitative Research Design
This unit continues your study of program evaluation. The activities will apply your new knowledge of research design to the evaluation of program
effectiveness.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
1. Explain strategies for measuring program effectiveness.
2. Understand variable level of measurement.
3. Explain the reliability and validity of measures.
4. Communicate through writing that is concise, balanced, and logically organized.
Program Evaluation Life Cycle
Evaluation is the systematic acquisition and assessment of information to provide useful feedback about a
career development center’s programs or services.
The general goal of evaluations in career development is to provide “useful feedback” to a variety of audiences including center staff, internal
and external interested parties, students, job seekers, adults in transition, community members, state and local leaders, federal entities, and
other relevant constituencies that address accountability. Most often, feedback is perceived as “useful” if it aids in decision-making and
making the programs and services more effective. Presently these strategies are used in a number ways to address efficacy, identify cost
effectiveness, and move to adequately address the changing nature of work and careers.
Two Types of Evaluation
Formative Evaluation in Career Centers Summative Evaluation in Career Centers
Needs Assessment
Determines who needs the career center programs and services,
how great the need is for these, and what might work to meet the
need at this time and place.
Outcome Evaluations
Investigates whether the particular career program or web based
career resource caused demonstrable effects on specifically defined
and targeted program outcomes.
Evaluability Assessment
Determines whether an evaluation is feasible and how the advisory
committee of internal and external members can help shape its
usefulness.
Impact Evaluation
Broadly assesses the overall or net effects—intended or
unintended—of the career program or web based career resources
as a whole.
Structured Conceptualization
Uses the advisory committee of internal and external members to
define the career center programs or web based career resources,
the target populations, and the possible outcomes of the programs
and services.
Cost-effectiveness and Cost-benefit Analysis
Addresses questions of efficiency by standardizing career center
outcomes in terms of their dollar costs and values.
Implementation Evaluation
Monitors the extent of which the career program aligns with the
initial vision and goals.
Secondary Analysis
Reexamines existing data on the career center to address new
questions or use methods not previously employed by the career
center leadership or staff.
Process Evaluation
Considers the process of delivering the career center program or
web based career information through technology, including
alternative delivery (i.e. social networking, blogging, twitter).
Meta-analysis
Integrates the outcome estimates from multiple studies to arrive at
an overall or summary judgment on an evaluation question about
the career center.
Phases of Evaluation
Phase One: Planning Phase
Formulation
Defining the major objectives, goals, and hypotheses of the career center program or web based center.
Conceptualization
Operationalizing the major components of the evaluation to be done of the career center – the career programs, participants, settings
these are used in, and measures to be used to see if these are effective.
Detailing
How the components of the evaluation will be coordinated. For example, what’s the timeline for evaluation? Will all programs be queried or
just some? Who will conduct the evaluations from the career center, or will there be an outside source of help? Is training needed for
career center staff or administrative leadership in preparation for the evaluation?
Evaluation
Assessment of the alternatives and the selection of the best one; and the implementation of the selected alternative.
Phase Two: Evaluation Phase
Formulation
Create major objectives, goals, and hypotheses of the career center program or web based career resources/technology.
Two Types of Evaluation
Conceptualization
Operationalization of the major components of the evaluation—the career programs, participants, settings, and measures (what you are
trying to capture).
Design
Who will design the evaluation? How will you validate it? What questions do you want to ask? What questions would external entities need
to know?
Analysis
Qualitative and quantitative. How will you write the implications? Conclusions? Areas of future study? Next steps in the career center’s
movement once this information is available?
Utilization
How will you use the evaluation results? What are the next steps in the career center’s movement having this data and information? Who
will it be shared with internally and externally?
L i c e n s e d u n d e r a C r e a t i v e C o m m o n s A t t r i b u t i o n 3 . 0 L i c e n s e .
This unit continues your study of program evaluation. The activities will apply your new knowledge of research design to the evaluation of program
effectiveness.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
1. Explain strategies for measuring program effectiveness.
2. Understand variable level of measurement.
3. Explain the reliability and validity of measures.
4. Communicate through writing that is concise, balanced, and logically organized.
Unit 2 – Program Evaluation: Quantitative Research Design
INTRODUCTION
[u02s1] Unit 2 Study 1
• Program Evaluation Life Cycle – read transcript
Readings
Use your Program Evaluation and Performance Measurement text to complete the following:
• Read Chapter 3.
◦ Pay attention to question 5 on page 137. The content this question addresses will be relevant for the first discussion in this unit.
• Read Chapter 4.
[u02d2] Unit 2 Discussion 2 – Levels of Measurement and Creating Survey Questions
After reading about levels of measurement in Chapter 4, search the Internet or the Research Library for examples of questions that measure each type of
variable.
Complete the following in your discussion post:
• Use the construct of yearly income to write a survey question, along with the possible answers, for each type of variable level of
measurements; nominal, ordinal, interval, and ratio.
• Clearly label which survey question is nominal, ordinal, interval, or ratio.
N
Causal validity
Exists when a conclusion that A leads to our results in B is correct (p. 50)
Authenticity
When the understanding of a social process or social setting is one that reflects fairly the various perspectives of
participants in that setting (p. 50)
Sample
A subset of a population that is used to study the population as a whole (p. 149)
Program theory
A descriptive or prescriptive model of how a program operates and produces effects (p. 411)
Random sampling
A method of sampling that relies on a random, or chance, selection method so that every element of the sampling
frame has a known probability of being selected (p. 157)
Systematic bias
Overrepresentation or underrepresentation of some population characteristics in a sample due to the method used to
select the sample (p. 158)
Control group
A comparison group that receives no treatment (p. 223)
Quantitative methods
Methods such as surveys and experiments that record variation in social life in terms of quantities (p. 17)
Qualitative methods
Methods such as participant observation, intensive interviewing, and focus groups that are designed to capture social
life as participants experience it rather than in categories predetermined by the researcher (p. 17)
Evaluation research
Research that describes or identifies the impact of social policies and programs (p. 395)
Selection bias
When characteristics of the experimental and comparison groups differ or when the group under study has some
characteristics that biases their responses (in surveys or focus groups) (p. 238)
4
CBPR—Step by Step
ow that you have an understanding of the underpinnings of CBPR, it is valuable to walk step by step through a
project in order to understand the approach. As discussed in prior chapters, the fundamental steps in CBPR revolve
around some salient concepts: community assessment, strategic goal setting, identification of problems, formulating
research design, research conduct, analysis/interpretation, and dissemination, including action. In this chapter, we will
present “How to do CBPR” utilizing case examples and walking the reader through the various stages of a CBPR
project.
1. First stage: Defining the community, engaging the community, community needs assessment, identifying the
research question
2. Second stage: Design/hypothesis testing, roles and responsibilities in the conduct of the research
3. Third stage: Analysis, interpretation and results, dissemination and action
The first case example is from Everett, Massachusetts, where a CBPR study of the impact of Immigration and
Customs Enforcement on immigrant health occurred in 2010. The second case is from Cambridge, Massachusetts,
where a CBPR project examined weight disparities among the African American population. The third case example is
from a suicide cluster investigation in Somerville, Massachusetts, from 2002 to 2006. All three cases were presented in
Chapter 1 and are presented here again for the reader’s convenience.
Example 1. Immigrant Health in Everett, Massachusetts
Picture 4.1 Everett Community Partners
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
As described in Chapter 1, Everett, Massachusetts, has seen an influx of immigrants coming from countries such as
Brazil, Haiti, Guatemala, and Morocco. Everett is a small city of about 37,000 with reasonable rents and proximity to
Boston. While there have been tensions in the community about issues related to immigration, such as housing and
parking, it is only recently that the increased activity of Immigration and Customs Enforcement (ICE) has created
challenges for the immigrant community. In particular, with increases in deportation and detention, immigrants fear that
they will be picked up by authorities and deported. Stories of immigrants missing health appointments because ICE was
in the vicinity or having stress-related conditions such as sleeplessness, headaches, and weight loss were common.
These concerns were raised by several of the immigrant advocacy groups and Everett community leaders, who felt that
getting concrete information about this issue would facilitate changes in local policy. So they approached a familiar
academic partner to join them in an investigation of the problem. Their goal was to learn more about the issue and then
solve the problem and develop policy or programmatic interventions that would alleviate some of the stress that
immigrants were experiencing.1
In this scenario, members of the target community approached the researcher to assist in answering what they saw as
a pressing health and social issue in their community. While this may be perceived as a preferable CBPR initiator, there
are many cases of CBPR researchers approaching a community with a research topic, particularly when they are
familiar with members of that community.
Example 2: BMI Disparities in Cambridge, Massachusetts
Figure 4.1 The Cambridge H.E.L.P. Project
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
Our second case as previously described, took place in Cambridge, Massachusetts over a 10-year period, schools,
public health agencies, and researchers had combined forces to track childhood indicators of obesity. Annual height and
weight measurements of children were taken by school personnel and provided to parents each year in the form of a
healthy weight progress report.2 Data accumulated through this process were available and allowed researchers to
examine trends in childhood obesity as well as disparities among racial/ethnic groups.3 The researcher noted that
despite downward trends in BMI in the population overall, there were persistent and glaring disparities by
race/ethnicity, with both Blacks and Hispanics carrying an undue burden of obesity. The researcher and a community
colleague with whom she already had a relationship joined forces to pursue this line of inquiry. They started by
discussing their concerns with several African American colleagues who were leaders in the community and were eager
to work with them. Together, they approached other key leaders in the community, including a principal of a school
with a large Black population. After exploring the existing data, they obtained a small amount of pilot funding to
conduct several key stakeholder interviews with “positive deviants”—local African Americans who had made
significant lifestyle changes and who had made progress toward attaining healthy weight. Using the results of these
interviews, they applied for a larger pilot grant to conduct further analysis of existing data, collect more qualitative,
contextual data, and to work with community partners to build appropriate interventions.
Example 3: Suicide Cluster in Somerville, Massachusetts: Real-Time Health Crisis
Figure 4.2 Suicide Attempts and Completed Suicides Among Somerville Residents Ages 10 to 24 Years
Source: Reproduced by permission from Crisis; Vol. 29(2): 86–95. © 2008 Hogrefe & Huber Publishers, www.hogrefe.com
As previously presented in Chapter 1, our third case took place in Somerville, Massachusetts, an urban city of 70,000
people that borders Cambridge, Massachusetts. Historically, Somerville has been home to working-class populations
and in recent years, between gentrification and new immigration, the city’s demography has changed substantially.
Somerville has also been affected by long-term substance abuse problems, especially heroin and alcohol. In 2001, a
young person took his own life, and this was followed soon after by oxycodone overdoses of two high school students.
A local researcher with an interest and experience in adolescent suicide was concerned that this might represent the
beginning of a suicide cluster. She had prior relationships with community partners and so approached the Health
Department director and mayor to discuss her concerns and interest.
Loss of youth life to suicide and overdose sends enormous ripples of concern through any community, and in
Somerville, the Health and School Departments examined data from their biannual teen health survey to determine if
suicidal behaviors had changed. The teen survey noted that 21% of the students had seriously considered suicide, and
14% had attempted suicide during the last 12 months. This was substantially elevated over previous years and higher
than the average for the state overall.
In order to respond to the situation and investigate further, the mayor convened several task forces and asked the
researcher to join with community members and colead one of the task forces along with the Health Department
Director. Other members included representation from the schools, the police, and community members, as well as
additional experts in suicide clusters. The question posed by the community to the researcher was “How do we stop the
youth suicide?” The researcher helped to reframe the question into two fundamental questions:
• Was this suicide and overdose activity significantly elevated from baseline?
• Were there common links between victims and was this a contagion/cluster?2
FIRST STAGE: DEFINING THE COMMUNITY, ENGAGING THE COMMUNITY, COMMUNITY
NEEDS ASSESSMENT, IDENTIFYING THE RESEARCH QUESTION
Each case presents a slightly different way to consider defining community. As an investigator, you need to consider
what community you want to approach; this may have to do with your content area or with your current locale. Either
way, you need to be aware of the boundaries around the community that you want to engage before taking your next
steps. Community can be defined by geography, by condition, or by other common concerns/characteristics. If you want
to work with immigrants, think through the groups that might represent this population and contact them. If you want to
work with a geographic community, consider the organizations or institutions that serve the community. In the Everett
case, the community essentially defined itself, and community members with a particular concern approached a familiar
researcher. The group of people representing the immigrant community and those that provided services to them
included a broad representation of the leaders from immigrant groups and local institutions, including schools, city
government, and police. This self-defined community, however, did not fully represent the population of interest, and as
the work proceeded, the researcher and the community members made efforts to expand their group to include more
members from the target population: immigrant residents of Everett. Through community partners’ social networks,
introductions were made, and new members joined the CBPR project.
In contrast to this approach, it is not unusual for a researcher who is interested in exploring a specific topic to
approach community partners. The challenge is whether the researcher’s area of interest is also of interest to the
community at large. If the researcher has done an adequate job of assessing the needs of a given community, this should
not pose a problem. In the Cambridge case, it was the researcher who approached the African American community
leaders in Cambridge, but several important points should be noted. First, the project was built on prior existing and
trusting relationships. Thus, the researcher had excellent knowledge of the key stakeholders from the Black community.
These stakeholders were well connected and had the potential to open doors to other community members to further the
work. Second, the topic chosen for study expanded prior work that was already deemed to be of interest to this
community—namely, despite increased community activities for healthy eating and active living, Black youth were still
twice as likely to be overweight or obese compared to their White peers. The practice of CBPR is best built on mutual
interest and existing relationships, as illustrated here. The researcher who has not done her or his homework to
understand the issues facing the community will find CBPR more challenging and is less likely to be successful in such
a pursuit.
In the Somerville example, the community was defined geographically, and the researcher and the community both
identified the problem as it was occurring. The researcher pointed out the emerging problem to local leaders, including
the mayor. Then, as the community began to coalesce around the issue, the researcher was brought in as a pivotal
addition to the community task force and was able to add expertise. It was during the crisis that the researcher and
community partners built new relationships and developed others. In this case, the researcher was really in the thick of
the situation and acted as both a community partner and an investigator toward trying to find solutions.
In all three of these cases, it is hard to separate the “defining the community stage” from the “engagement stage.” The
very process of defining the community leads the researcher to develop relationships, walk the streets, learn the culture,
and immerse him- or herself in the community of interest.
Steps 1 and 2: Defining and Engaging the Community and Community Needs Assessment
Once you have determined your target community, you will need to begin the engagement process. Start by
examining any data available on the community of interest, then contact the local service and civic organizations and
ask for an informational meeting, explore their websites, and learn what they provide. Census data are an excellent
source when thinking about geographic populations. This information is available on the Census website3 and can
provide information on economic conditions as well as demography. In addition, local newspapers are helpful in
learning about local politics. For specific populations, there are likely to be other sources available: immigrant advocacy
groups, national associations, prior publications, and the like. Educating yourself about the community of interest
before you try to engage its members will be helpful in your approach. In addition, in CBPR, don’t go into your
community engagement process with the particular research agenda in mind. You will need to better understand the
community concerns before you can really focus on a particular research question. Otherwise, there is a chance your
interest will not be of interest to your community partners, and you will find the mismatch of priorities frustrating.
How can one learn about current community concerns? In general, as noted above, existing data on community
health and social issues are available through local sources such as school department reports, public health
assessments, or statewide information on localities. In our Everett project, learning about Everett through its citywide
website gave us information on the current issues and community resources. In addition, state data on health issues were
available through the Massachusetts Department of Public Health. These high-level data are helpful as you develop an
understanding of the community, but nothing can replace showing up in the community and getting to know the local
culture and important leaders. This is also the best strategy with which to begin your community engagement activities.
Start with a basic needs assessment. Contact leaders of active community groups such as schools, community-based
organizations, advocacy groups, religious groups, and service agencies. Meet with the directors and find out what they
perceive as the challenges in their community. Think about where the likely places are that serve the population you are
interested in studying. For example, if you are interested in maternal infant health, try working with local pediatricians
or preschool programs. If you are interested in the elderly, contact senior centers. If you are interested in substance
abuse, learn about the local substance abuse provider agencies. Visit the programs and meet their directors. Ask
questions like: What was the stimulus for starting your organization? What do you see as the major issues facing your
community? This process may seem onerous, but the results will provide an important foundation for future work.
These early introductions will serve as the foundations of partnership as you and your community partners learn about
one another. Your approach to these meetings is very important, as a dominant, self-centered approach will likely
negatively impact future potential. Practice cultural humility and bring your listening skills. Academics can be viewed
as aloof and even intimidating to community members. In CBPR, we need to break down these barriers and move
toward equitable partnerships. The engagement process can be very time intensive and may take several years as the
community learns to trust the researcher. Thus, for many early CBPR researchers, it may be easier to build on existing
relationships established by successful CBPR researchers rather than starting from scratch. These investigators may be
helpful in opening doors to the community and in vetting you as someone that can be trusted.
This needs assessment process is an important first step in developing a CBPR agenda, particularly if the researcher
is unfamiliar with the community. Getting to know a community and those who are identified as leaders is step one in
the engagement process. The process of meeting with individuals begins the foundational partnerships that are the basis
of CBPR work. Community members have a chance to vet the researcher and vice versa. In communities where
historical projects have left a distrust of research in general, this is particularly important. Why are you different from
others? What will you provide for the community? How will this project benefit the citizens? This process can be
challenging at first but is ultimately one of the most rewarding components of CBPR.
The data that a researcher obtains in this engagement process are instrumental for community mapping, that is,
developing an in-depth understanding of the political organizational structure, the assets within the community, and the
issues that concern community members. By identifying interpersonal connections, social networks, and the existing
political landscape and infrastructure, one may also understand how things get done, what the opportunities for change
are, and who can help sustain this change. Remember, CBPR is not only about the research itself but also about
resulting action at the community level. In our Cambridge example, the researcher started with the data that had been
collected over many years on schoolchildren’s weight disparities. Along with a public health nutritionist and a doctoral
student, she approached four longtime Cambridge colleagues who were considered “movers and shakers” in the Black
community. These individuals had a commitment to the work and included a school principal, staff from the public
health department, and a local nonprofit. They became “community investigators” on the project and, along with the
researcher and public health nutritionist, identified themselves as the Healthy Eating and Living Project (H.E.L.P.) and
began to investigate the underlying causes for the disparities. Through these relationships, the researcher was able to
identify others who were not only concerned about the issue but were also likely to be able to make programmatic
change. The success of the project relied heavily on this core of community investigators.
In the Somerville example, the engagement process took place at many levels. This included the mayoral level and
other political and institutional leaders. It also included lay leaders in the community most affected by the suicides.
And, finally, it included the youth who were friends of the deceased. Not all engagement proceeded at the same pace,
and there were differing levels of trust. While the crisis demanded immediate teamwork, the relationships at the
grassroots level—that is, the lay community leaders and the youth themselves—were the most difficult to establish. The
researcher ended up in true partnership with the lay leaders but never developed strong partnerships with the youth, in
part because this was ill advised, according to the lay community leaders. Instead, these leaders acted as liaisons, and
the researcher did not push an alternative agenda. These lay leaders were pivotal advisors throughout the crisis.
In a similar fashion, most CBPR projects rely heavily on a community advisory board (CAB) or coalition that will
participate in the research process and function as the liaison to the community. As you build your relationships, you
should be thinking about who will serve on a community advisory group. The CAB will help to hone the research
question and methodology and provide outreach to the population of interest as well as supply knowledge of local
politics, pitfalls, and strategies. Choosing a group of community members to actively become part of your CAB is
critical to the success of the project. Who are you working with? Are they committed to the project? Are they
representative of the community of interest? Have you worked together previously and can you work together going
forward? Do they want to be part of a CBPR project? What are the expectations for participation and how will
information be communicated? In our Everett example, the community members who were already meeting were the
natural CAB. In the Cambridge example, the researcher started with a community partner, and then, together, they
identified and connected with other community members to form the advisory group. In our Somerville example, there
was a task force that the mayor had convened, and this group served as the CAB for the investigation. Most CABs will
meet regularly throughout the project. If grant dollars are available, they should be allocated to support this activity,
including payment for community member time needed to participate.
Step 3: Refine the Research Question
Honing the research question starts with the topic of interest. In many cases, the community questions will be broad,
reflecting timely concerns. “How do we stop drug abuse?” “Why is there youth violence?” “Why is autism increasing?”
Community members are very helpful in identifying the topic of interest but may not have the skills to hone the
research question. That is where the researcher’s skills are most applicable. You should work with community partners
to help focus the research questions. Your knowledge of the literature will help in this process, and by supplying the
existing evidence, you can help the partnership consider what has already been discovered and think about new and
emerging questions that have yet to be answered. Conduct a literature review and provide the important papers to the
CAB. Part of your role is to educate the CAB about what is already known about the subject from a research
perspective, just as the CAB will educate you about the subject from a community perspective. In our Everett example,
the community asked if and how ICE was impacting immigrant health. The researchers identified prior literature on
related subjects and provided this information to the group. This work also influenced the development of the
conceptual framework. The final research questions were more nuanced than the original broad questions and included
an exploration of how local police were perceived and whether ICE had an impact on health care access, on emotional
well-being, or on chronic disease.
In the Somerville, Massachusetts example, in the midst of a health crisis, the community wanted to know how to stop
the crisis. But first, it was important to understand how the crisis was proceeding. Was this a suicide cluster? Were the
young people involved connected to one another? Was activity elevated over baseline? The researchers worked with the
community to focus first on these questions and also to understand how other communities might have dealt with
similar circumstances. The researcher plays an important role in translating the evidence into real-life situations.
Similarly, in Cambridge, the questions were honed to a set of specific goals for the project: (A) understand the socio-
demographic factors associated with obesity among Black school-aged children; (B) understand the social, cultural, and
behavioral barriers and influences on diet and physical activity of Cambridge families; (C) identify a culturally relevant
intervention to promote healthy weight in Black children and their families. This process of taking a community
concern and reframing it into research questions and hypotheses is challenging. In Table 4.1, several examples are
noted.
Table 4.1 Examples of Questions From Community and Researcher Perspectives and Resulting Joint Hypotheses
Important First-Stage Considerations
• Learn about the community you are interested in through an informal needs assessment.
• Go into the community and meet people, begin conversations, get to know the culture.
• Work with a CAB to refine the questions for study.
• Serve as an educator and evidence translator.
SECOND STAGE: DESIGN/HYPOTHESIS TESTING, ROLES AND RESPONSIBILITIES,
CONDUCT OF THE RESEARCH
Step 4: Design and Methods
Once there is consensus on a research question, the CAB and investigator should work together to frame the
hypotheses to be tested and determine the best strategies for answering the research question. This will take place in a
series of meetings of the CAB. While the investigator is likely to know more about the possible methods for
investigation, the community partners possess the expertise to determine the feasibility of these methods. For example,
even though an experimental approach may be the most valuable to answer the question, community members may
have concerns regarding experimental methods due to other issues (time frames, access to participants, competing
priorities). Community partners will need to reflect on how methods can or cannot be utilized and whether they are
practical. In our Everett example, the initial work was exploratory in nature, and so methods that would allow for this
type of exploration were chosen, including interviews, focus groups, and surveys. But while the investigators suggested
that they hold several focus groups in only two languages so they could compare and contrast them, community
members wanted to work with all the dominant language groups in their community in order to get a broad
representation. The research team agreed to conduct six focus groups and changed its perspective to assess information
across immigrant groups. So too, the researchers thought that documented and undocumented immigrants should be
separated since they assumed these populations would have very different opinions on the topic. Community members
felt this posed several problems. Undocumented immigrants were unlikely to come forward if they thought they would
be singled out, and in many cases, families and social networks were made up of both documented and undocumented
individuals. This information was then incorporated into the design. Research design required a negotiation in which the
researcher introduced methods and community members considered the implications and practicality while also
contributing real world evidence.
In our Somerville example, researchers identified methods for monitoring suicide and overdose activity while the
crisis was occurring. This included examining death certificates and using 911 call data from the fire department.
Community members were quick to point out that these sources had their own sets of problems. The death certificate
data were often delayed if the body went to the medical examiner and the immediate cause of death was unknown. The
911 call data provided only initial impressions from the scene and did not provide follow-up of incidents. In turn,
community members noted that youth were using websites to monitor cases themselves and introduced the researchers
to these data sources, which provided information on connections of victims that was unknown to the CAB. As a result,
researchers expanded their data collection to include website monitoring.
In the Cambridge example, parent/child interviews with local Black families were conducted. Conversations were
held with various Black leadership groups (Men’s Health League, Black Pastors Association) and parents and students
at the high school and middle school. A survey of families at two schools was conducted. Introduction and access to
these groups was provided by members of the CAB.
Step 5: Roles and Responsibilities
Once there is consensus on methods, roles and responsibilities for research conduct should be delineated. How will
community partners participate in the research conduct? However, it is also important to remember that roles and
responsibilities require financial resources, so prior to writing any grants, budgetary needs of both researcher and
community member should be considered. In partnership, decide who will be involved in each step of the research
process. This will be dependent to some extent on skills, expertise, and time. In the Everett project, community
members were actively involved in tool development (focus group guides, surveys, and interview guides). They took
the lead on focus group facilitation and participant recruitment. Researchers provided training sessions in focus group
conduct and human subjects. Community members were less involved in analysis; they participated in initial
identification of thematic content, and then researchers took over statistical and qualitative analysis. Results were
presented to the CAB for interpretation and refinement.
In the Somerville example, community members led the task forces and researchers provided them with ongoing data
for data-driven decisions. Then community members provided the context and the interpretation of the data.
Community members were the main investigators when it came to outreach with youth, and researchers played more of
a technical support role.
In the Cambridge example, the researcher and her team did the original quantitative analysis to understand some of
the factors leading to disparities in BMI. The data were presented to the CAB in an iterative fashion, and CAB members
assisted with interpretation. The CAB was also involved in designing all the tools for data collection, including
interview guides and surveys, and members were involved in conducting the interviews. Throughout the entire project,
the CAB members were intimately involved in the interpretation of the research findings.
Each of these examples demonstrates a slightly different approach to roles and responsibilities. However, it is
important to recognize what partners bring to the table in the way of skills and expertise and build from there. Thinking
through how the partnership will work throughout a project is an important part of the grant-writing and project
development process. Transparent communication about expectations is also important, and using memoranda of
agreement (MOA) or other strategies can be helpful to clearly articulate these expectations. Details of items to be
included in an MOA are shown in Table 4.2.
Table 4.2 Components for a Memorandum of Agreement (MOA)
• Describe partners (who are the parties that are entering into this agreement?)
• Dates for MOA (beginning and end of project)
• Roles and responsibilities (who will actually do what; what each party agrees to do)
• Scope of work
• Timeline
• Deliverables (when and what is expected of each partner)
• Budget (how much, billing procedures)
• Publication rights and authorship
• Use of names of partners
• Data ownership (who owns data and how they can be used)
Step 6: Conduct Research
Conducting the research is largely dependent on preparation—that is, how well the roles and responsibilities were
defined and how well procedures were outlined. In addition, adequate training is necessary to ensure accurate and
complete data collection. There must also be a clear understanding of who is responsible for monitoring progress
toward goals and dealing with unexpected events that can range from staff departures to ethical dilemmas to incomplete
data. These responsibilities most often fall to the researchers but may be divided among community members as well.
In the Everett example, community members were responsible for conducting the focus groups and taking notes. But
the research team was responsible for ensuring that the dates for focus groups were set, that the community partners
were trained in focus group conduct and ethics, and that notes were delivered in a timely fashion. While there were
focus groups that lagged behind, frequent meetings of the CAB helped the project adhere to its timeline. A variety of
tools to help in data collection and to decrease variability in data collection were developed, including templates for
focus group notes, short surveys to collect first impressions after the focus groups, grids for analysis, and themes and
scripts for interviews. In CBPR, academics and community partners are mutually dependent. The potential pitfalls are
many. As is often the case, the researcher served as the overall principal investigator for the project and had ultimate
responsibility for adhering to the institutional review board approval. In CBPR, regardless of who leads the project,
there are many moving parts, which only increases the need for strong management. However, decisions are shared, and
conflicts that arise during conduct will need to be collaboratively resolved.
In our Somerville example, all data were collected by the community. These data sources included the youth risk
behavior survey that had been done annually and the 911 data, which were collected monthly by the fire department.
The researchers analyzed and mapped the data. Community members also assisted with the examination of death
certificate data. Today, in part as a result of this project, community members continue to collect these data for ongoing
surveillance.
In our Cambridge example, secondary data on childhood obesity were collected by school physical education
teachers, and the resulting dataset was used for analysis. Community members were involved in the development of all
new data collection tools, and a community member joined a researcher for each of the parent/child interviews. The
researcher, however, was responsible for the overall project and for moving the project forward.
There are many ways to engage the community in the conduct of CBPR. Their involvement in data collection can
build their individual skills and the capacity of the community around data overall. However, in a participatory process,
management is required, and this is always harder when responsibilities are shared. The MOA plays an important role
in setting the stage for CBPR, but it will be important to review progress regularly at CAB meetings, to deal with
unexpected challenges, and to ensure the work gets done.
Second-Stage Considerations
• Work in partnership with community members to design the research and choose appropriate methods: balance
rigor and practicality.
• Decide who is going to do what and make sure the resources are in place to get the work done.
• Monitor the conduct of the research, “make sure the trains are running on time,” and develop strategies for
addressing unexpected pitfalls.
THIRD STAGE: ANALYSIS, INTERPRETATION, AND DISSEMINATION
Step 7: Analysis and Interpretation
Once roles have been clarified, think through the analysis and interpretation plan and discuss how both community
members and researchers can be involved. In CBPR, it is not necessary that everyone is involved in everything to the
same extent; however, it is important to recognize and address issues of data ownership. Remember, in CBPR, data
ownership is something that should be negotiated up front, as data are not only the property of the researcher but also of
the community partners. Negotiating decisions about how data will be used, when they can be shared with those
external to the CAB, and who will house and “own” the data in the present and future will take some investment but
will help solve potential future problems. Memoranda of agreement (see Table 4.2), referred to previously, can be
helpful in documenting how these decisions should be made.
Roles in analysis often depend on expertise, time, and interest. For example, analyzing quantitative data using
statistical methods is often best left to the researcher and the academic team, who have knowledge of statistical
programs. However, there may be community members who want to learn more. In our Cambridge obesity example,
the CAB decided on a preliminary analysis plan, that is, what variables they would explore and what statistical methods
would be used. Then the researchers conducted analysis and brought frequencies back to the team prior to doing more
extensive analysis to understand the questions that community members might observe. This assigning of meaning is
one of the most important and valuable contributions of CBPR. The community partners will have an understanding of
the context and meaning of these results. Their questions may be very different than those of the researchers. Their
insight is extremely critical for later community action. Once additional decisions about analysis were made, the
research team ran the analyses and once again brought it to the CAB. This iterative process helped to move the analysis
forward in a manner that incorporated the multiple perspectives and enriched the end product. The CAB will be
instrumental in interpreting results of any CBPR project as members bring their own understanding of the community
issues to the process.
In our Somerville example, at the request of community partners, researchers took data from 911 calls and mapped
them onto a map of Somerville using mapping software, and this revealed that certain areas of the community had
higher levels of overdose and suicide activity than others. Community members could easily point out why these areas
might have higher activity; for example, the area around a local college had a high level of alcohol 911 calls, while the
area around the local housing project was known for drug activity. This information about the context of the community
was particularly important in framing the meaning of the data and their interpretation. As a researcher involved in
CBPR, recognizing what community members know about their community can be extremely helpful in this phase of
the research.
Qualitative data may be somewhat more approachable than quantitative data for community partners. Listening to
constituents is something that they do regularly. However, few are used to collecting and analyzing this type of data in a
systematic manner, and it will be important to warn against jumping to conclusions based on one interview or focus
group. The researcher can help facilitate the analysis of qualitative data by integrating community partners into the
process. Again, this can be an iterative process, with initial work being done by the researcher and then presented to the
CAB for refinement, reanalysis, and presentation again until all are satisfied with the codebook. This was the process
undertaken with the focus groups in the Everett example. After completion of the focus groups, the CAB identified
initial themes. Then the data were entered into a software package and further categorized, then presented to the CAB
and modified and reanalyzed again. The process took about 3 months but benefited from the multiple perspectives of
both researchers and CAB members. In short, while analysis may seem a cumbersome process to share between
researcher and community, it can be done through creative strategies that ultimately enrich the process and improve the
relevancy of the work.
Step 8: Dissemination
Lastly, the dissemination of results as a final step in the CBPR process often takes multiple forms. As a researcher,
you are likely to be interested in peer-reviewed journals, while community partners are much more likely to want
immediate dissemination so that they can utilize the results in action. This can present many conflicts for researcher and
community alike. For example, in Everett, the community wanted a forum at which results could be presented and
recommendations discussed. The researchers wanted to produce a paper for publication and were concerned about how
release of data would impact their ability to publish. They agreed on a compromise that resulted in several strategies.
First, a large community forum with a PowerPoint presentation of the high-level results was presented in a digestible
form at the end of the project. Community members took the lead on orchestrating the forum, including the invitation
list, the food, and the space. The researchers took the lead on the initial draft of the presentation, and then the
PowerPoint was circulated among all partners for critique and edits. The forum also had an interactive component in
which community members had a chance to develop recommendations for action based on the presentation. The list of
recommendations was finalized and presented to institutional leaders (mayor, police chief, and schools). Advocacy
agencies took the recommendations and incorporated them into their activities. Simultaneously, the researcher
developed an outline for a peer-reviewed paper and asked all those on the CAB if they wanted to be authors. Authorship
brought responsibilities, including some editing and writing. Not everyone wanted to participate, but over the course of
the next year, all the authors (there were 10 in total) had a chance to review drafts, comment, and provide feedback.
Two papers from the project were accepted for publication. The community, meanwhile, acted on some of the
recommendations. They began meeting with local police to address traffic stops, which changed the dynamics between
the immigrant community and the police overall. Almost a year later, the police were no longer “arresting” people who
did not have driver’s licenses. Instead, they would issue citations. This meant that undocumented immigrants were able
to avoid fingerprinting and reporting to ICE. The police/immigrant relations improved as regular meetings with police
were held. One immigrant advocacy leader noted that the number of complaints about police from the immigrant
community had fallen dramatically.
Dissemination in the Cambridge example was handled somewhat differently but with similar attributes. A formal
report was made to the chief public health officer for the city and to the CAB. A final written report was drafted and
vetted by community partners.4 In turn, this was presented to other Cambridge leaders to get their perspectives on
potential action steps. The CAB was able to obtain some additional funding to maintain its work as it applied for
additional funds to pilot test an intervention.
Overall, the dissemination process should be considered in two simultaneous manners: that is, what and how to
disseminate results for the benefit of the community members while also publishing results in peer-reviewed journals
for generalizability. Both academic and community partners are driven by real incentives, and these end products
highlight the differences between them. While these two agendas can conflict, there are ways in which both parties can
be satisfied and gain from the process. The two dissemination strategies may also have different timelines; the
community may want immediate results that can be utilized in practical applications at either the policy or
programmatic level. The time for article production may be much longer and somewhat irrelevant to the community but
important to the researcher. Straddling these potentially conflicting goals is difficult, but as long as communication is
open and transparent, the dissemination process can flow smoothly and obstacles can be addressed. There is no question
that group dissemination is cumbersome, but the differing perspectives enrich the final products and make them not
only more relevant to the community but also more likely to have sustainable outcomes and a “life” after the CBPR
project is completed.
Identifying action steps is really in the hands of the community partners. While it is part of the dissemination process,
the researcher may or may not be involved. However, the researcher can play an important role in sustaining the CBPR
collaboration. In our three examples, action steps resulting from the CBPR projects included the following:
1. Everett: regular meetings held with immigrants and police to enhance communication; health care providers
improved awareness how ICE affects their immigrant patients.
2. Somerville: regular monitoring of 911 activity; development of a citywide trauma response network.
3. Cambridge: recommendations developed for future interventions, including: encourage youth to drink water,
expand programming for Black girls to be physically active, limit TV consumption, and avoid using food to
comfort children when they are upset.
Third-Stage Considerations
• Determine how partners will be involved in both analysis and interpretation.
• Use an iterative process to assign meaning and generate consensus on that meaning.
• Utilize multiple dissemination strategies important to both community members and academics.
• Identify action steps.
CONCLUSION
In summary, there are clear steps in the CBPR process that should be considered (see Table 4.3). Engagement of the
community, defining the questions, conducting the research, and disseminating findings all take time and should be
addressed in a participatory manner. The CAB provides an excellent organizing strategy for working with community
partners, but the research team of community members and academics must map out communication and roles and
responsibilities and conduct the study in an environment of mutual respect.
Table 4.3 Summary of CBPR Steps
Stage 1
Community Engagement5–7
• Identify the community of interest.
• Conduct a needs assessment to understand the issues for potential research.
• Meet the stakeholders.
• Develop relationships.
• Assemble a CAB.
• Choose an area for research.
• Conduct a literature review.
• Hone the research questions and hypotheses.
Stage 2
Research Design, Roles, and Responsibilities
• Discuss methods.
• Assess feasibility.
• Define roles for community and for researcher.
• Provide education and skill development.
• Conduct study.
Stage 3
Analysis, Interpretation, and Dissemination
• Assess skills.
• Develop iterative processes.
• Work with CAB for interpretation and context.
• Determine dissemination modes.
• Develop processes that allow all to participate.
• Identify action steps based on evidence.
QUESTIONS AND ACTIVITIES
The purpose of this assignment is to help students put ideas about CBPR into practice. If you can actually assign your
students to work with local community partners, you can provide them with real-world experience. If that is not an
option, have them work in groups to develop a CBPR research proposal that addresses one of the community health
problems described below (i.e., childhood obesity, youth suicide, disparities in mortality among women living with
HIV, or disparities in cardiovascular disease among Black men).
Guidance
1. Describe your CBPR approach.
2. Who are your community partners and how will you engage them?
3. What is your research design?
• Sample
• Recruitment
• Methods
• Analysis (optional)
4. How will you disseminate findings? What will you do with the findings?
5. Identify potential challenges or limitations.
Scenario 1—Youth Suicide
Community Context
Setting: a community that is suburban with a population of about 80,000. The suburb is fairly wealthy, with average
home costs hovering around $500,000. It has a reasonable tax base and is home to an outstanding university, where you
work. The population is by and large White, and there is not much diversity economically, but the population is
changing as educated South Asians move into the community.
Description of a health problem: In 2010, one teenager who was attending the local high school committed suicide
via a pill overdose. Within several months, her best friend also left a note, and she too committed suicide, but she
jumped from a building in the downtown area. Parents of high school children are in an uproar, demanding that the
school do something to prevent another suicide. Teens are holding vigils for their lost friends.
Partners That Came to You With the Problem
Members of the school committee have convened a task force of parents, teachers, and members of the local
churches.
Health Question
What is happening in their community that could be leading to such a situation? They also want to understand if these
two suicides represent a “cluster” or “contagious” situation. They feel they need this information to inform their next
steps.
Scenario 2—Cardiovascular Disease in Men
Community Context
This is a city in the Midwest of about 1 million. There has been a real economic downturn here, and the
unemployment rate is upward of 10%. There is a large Black population in one area of the city, which is also an area
with high rates of poverty and even higher rates of unemployment.
Health Problem
The rates of stroke and heart attacks in the Black male population are alarming and far outstrip those for Whites in
the city. There are two hospitals, one catering to the small economically well-off population and one that is the
“poverty” hospital.
Partners That Came to You With the Problem
Providers at the poverty hospital have seen ever-increasing rates of young men coming in with strokes. They want to
do something to change the situation, but many men do not seek health care in part because they are uninsured and
unemployed.
Health Question
How can the health care system identify and reach out to men of color who are at risk for cardiovascular disease?
Scenario 3—Childhood Obesity
Community Context
This is a middle-income city on the West Coast with a population of approximately 100,000 persons. Approximately
30% of the population is under the age of 18, the majority of whom attend public schools across the city. The city is
racially and ethnically diverse. There is a large immigrant community from Mexico and other Central American
countries. There is also a growing Southeast Asian population.
Health Problem
In 2004, all of the city’s schools began measuring students’ BMI and reporting back to parents the results of the
assessment. Educational materials have been sent home with the reports, and school nurses are available to follow up
with parents who are interested in learning more about what they can do to improve their children’s health. Since 2004,
BMI scores have steadily increased among every racial and ethnic group in the city. BMI scores are particularly high in
several elementary schools.
Partners That Came to You With the Problem
The PE teacher in one school with particularly high BMI scores among students came to you with the data the school
has been collecting over the last few years. The PE teacher has been in touch with the school’s family council, PTA,
teachers, and administrators.
Potential Health
Questions
Why are BMI scores increasing at a higher rate among students in some schools? What underlies the increase in BMI
scores?
Questions
1. What are some of the factors to consider in engaging community partners in analysis of data?
2. How does the process of dissemination differ in a CBPR project from a non-CBPR process?
3. In what ways would community partners use data from a CBPR project to take action?
NOTES
1. Hacker K, Chu J, Leung C, Marra R, Pirie A, Brahimi M, English M, Beckmann J, Acevedo-Garcia D, Marlin RP. The impact of
Immigration and Customs Enforcement on immigrant health: perceptions of immigrants in Everett, Massachusetts, USA. Social Science &
Medicine. 2011 Aug;73(4):586–94.
2. Hacker K, Collins J, Gross-Young L, Almeida S, Burke N. Coping with youth suicide and overdose: one community’s efforts to
investigate, intervene, and prevent suicide contagion. Crisis. 2008;29(2):86–95.
3. U.S. Census Bureau. 2006–2008 American Community Survey 3-Year Estimates. 2008 [cited 2010 October 3]; Available from
http://factfinder2.census.gov.
4. Chomitz V, Arsenault L, Banks C, et al. H.E.L.P. Culminating Report. Cambridge, MA: Institute for Community Health; 2011.
5. Clinical and Translational Science Awards Consortium. Community Engagement Key Function Committee Task Force on the Principles of
Community Engagement. Principles of Community Engagement. 2nd ed. Rockville, MD: NIH; 2011.
6. Israel BA, Parker EA, Rowe Z, et al. Community-based participatory research: lessons learned from the Centers for Children’s
Environmental Health and Disease Prevention Research. Environmental Health Perspectives. 2005 Oct; 113(10):1463–71.
7. Minkler M, Wallerstein N., eds. Community-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass; 2003.
