USA: +1(669)289-8683 
Sign in
U2D1 – Reliable and Valid Program Evaluation
For this discussion, refer to your readings of Chapters 3 and 4 in Program Evaluation and Performance Measurement . Pay close attention to
question 5 on page 137.
Answer the following questions in your discussion post:
• What are the hallmarks of a reliable and valid evaluation that shows change over time?
◦ For example, how do you measure program effectiveness?
Readings
• Program Evaluation Life Cycle – read transcript
Use your Program Evaluation and Performance Measurement text to complete the following:
• Read Chapter 3.
◦ Pay attention to question 5 on page 137. The content this question addresses will be relevant for the first discussion in this unit.
• Read Chapter 4.
INTRODUCTION – Unit 2 – Program Evaluation: Quantitative Research Design
This unit continues your study of program evaluation. The activities will apply your new knowledge of research design to the evaluation of program
effectiveness.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
1. Explain strategies for measuring program effectiveness.
2. Understand variable level of measurement.
3. Explain the reliability and validity of measures.
4. Communicate through writing that is concise, balanced, and logically organized.
Program Evaluation Life Cycle
Evaluation is the systematic acquisition and assessment of information to provide useful feedback about a
career development center’s programs or services.
The general goal of evaluations in career development is to provide “useful feedback” to a variety of audiences including center staff, internal
and external interested parties, students, job seekers, adults in transition, community members, state and local leaders, federal entities, and
other relevant constituencies that address accountability. Most often, feedback is perceived as “useful” if it aids in decision-making and
making the programs and services more effective. Presently these strategies are used in a number ways to address efficacy, identify cost
effectiveness, and move to adequately address the changing nature of work and careers.
Two Types of Evaluation
Formative Evaluation in Career Centers Summative Evaluation in Career Centers
Needs Assessment
Determines who needs the career center programs and services,
how great the need is for these, and what might work to meet the
need at this time and place.
Outcome Evaluations
Investigates whether the particular career program or web based
career resource caused demonstrable effects on specifically defined
and targeted program outcomes.
Evaluability Assessment
Determines whether an evaluation is feasible and how the advisory
committee of internal and external members can help shape its
usefulness.
Impact Evaluation
Broadly assesses the overall or net effects—intended or
unintended—of the career program or web based career resources
as a whole.
Structured Conceptualization
Uses the advisory committee of internal and external members to
define the career center programs or web based career resources,
the target populations, and the possible outcomes of the programs
and services.
Cost-effectiveness and Cost-benefit Analysis
Addresses questions of efficiency by standardizing career center
outcomes in terms of their dollar costs and values.
Implementation Evaluation
Monitors the extent of which the career program aligns with the
initial vision and goals.
Secondary Analysis
Reexamines existing data on the career center to address new
questions or use methods not previously employed by the career
center leadership or staff.
Process Evaluation
Considers the process of delivering the career center program or
web based career information through technology, including
alternative delivery (i.e. social networking, blogging, twitter).
Meta-analysis
Integrates the outcome estimates from multiple studies to arrive at
an overall or summary judgment on an evaluation question about
the career center.
Phases of Evaluation
Phase One: Planning Phase
Formulation
Defining the major objectives, goals, and hypotheses of the career center program or web based center.
Conceptualization
Operationalizing the major components of the evaluation to be done of the career center – the career programs, participants, settings
these are used in, and measures to be used to see if these are effective.
Detailing
How the components of the evaluation will be coordinated. For example, what’s the timeline for evaluation? Will all programs be queried or
just some? Who will conduct the evaluations from the career center, or will there be an outside source of help? Is training needed for
career center staff or administrative leadership in preparation for the evaluation?
Evaluation
Assessment of the alternatives and the selection of the best one; and the implementation of the selected alternative.
Phase Two: Evaluation Phase
Formulation
Create major objectives, goals, and hypotheses of the career center program or web based career resources/technology.
Two Types of Evaluation
Conceptualization
Operationalization of the major components of the evaluation—the career programs, participants, settings, and measures (what you are
trying to capture).
Design
Who will design the evaluation? How will you validate it? What questions do you want to ask? What questions would external entities need
to know?
Analysis
Qualitative and quantitative. How will you write the implications? Conclusions? Areas of future study? Next steps in the career center’s
movement once this information is available?
Utilization
How will you use the evaluation results? What are the next steps in the career center’s movement having this data and information? Who
will it be shared with internally and externally?
L i c e n s e d u n d e r a C r e a t i v e C o m m o n s A t t r i b u t i o n 3 . 0 L i c e n s e .
This unit continues your study of program evaluation. The activities will apply your new knowledge of research design to the evaluation of program
effectiveness.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
1. Explain strategies for measuring program effectiveness.
2. Understand variable level of measurement.
3. Explain the reliability and validity of measures.
4. Communicate through writing that is concise, balanced, and logically organized.
Unit 2 – Program Evaluation: Quantitative Research Design
INTRODUCTION
[u02s1] Unit 2 Study 1
• Program Evaluation Life Cycle – read transcript
Readings
Use your Program Evaluation and Performance Measurement text to complete the following:
• Read Chapter 3.
◦ Pay attention to question 5 on page 137. The content this question addresses will be relevant for the first discussion in this unit.
• Read Chapter 4.
[u02d1] Unit 2 Discussion 1 – Reliable and Valid Program Evaluation
For this discussion, refer to your readings of Chapters 3 and 4 in Program Evaluation and Performance Measurement . Pay close attention to
question 5 on page 137.
Answer the following questions in your discussion post:
• What are the hallmarks of a reliable and valid evaluation that shows change over time?
◦ For example, how do you measure program effectiveness?
N
Causal validity
Exists when a conclusion that A leads to our results in B is correct (p. 50)
Authenticity
When the understanding of a social process or social setting is one that reflects fairly the various perspectives of
participants in that setting (p. 50)
Sample
A subset of a population that is used to study the population as a whole (p. 149)
Program theory
A descriptive or prescriptive model of how a program operates and produces effects (p. 411)
Random sampling
A method of sampling that relies on a random, or chance, selection method so that every element of the sampling
frame has a known probability of being selected (p. 157)
Systematic bias
Overrepresentation or underrepresentation of some population characteristics in a sample due to the method used to
select the sample (p. 158)
Control group
A comparison group that receives no treatment (p. 223)
Quantitative methods
Methods such as surveys and experiments that record variation in social life in terms of quantities (p. 17)
Qualitative methods
Methods such as participant observation, intensive interviewing, and focus groups that are designed to capture social
life as participants experience it rather than in categories predetermined by the researcher (p. 17)
Evaluation research
Research that describes or identifies the impact of social policies and programs (p. 395)
Selection bias
When characteristics of the experimental and comparison groups differ or when the group under study has some
characteristics that biases their responses (in surveys or focus groups) (p. 238)
4
CBPR—Step by Step
ow that you have an understanding of the underpinnings of CBPR, it is valuable to walk step by step through a
project in order to understand the approach. As discussed in prior chapters, the fundamental steps in CBPR revolve
around some salient concepts: community assessment, strategic goal setting, identification of problems, formulating
research design, research conduct, analysis/interpretation, and dissemination, including action. In this chapter, we will
present “How to do CBPR” utilizing case examples and walking the reader through the various stages of a CBPR
project.
1. First stage: Defining the community, engaging the community, community needs assessment, identifying the
research question
2. Second stage: Design/hypothesis testing, roles and responsibilities in the conduct of the research
3. Third stage: Analysis, interpretation and results, dissemination and action
The first case example is from Everett, Massachusetts, where a CBPR study of the impact of Immigration and
Customs Enforcement on immigrant health occurred in 2010. The second case is from Cambridge, Massachusetts,
where a CBPR project examined weight disparities among the African American population. The third case example is
from a suicide cluster investigation in Somerville, Massachusetts, from 2002 to 2006. All three cases were presented in
Chapter 1 and are presented here again for the reader’s convenience.
Example 1. Immigrant Health in Everett, Massachusetts
Picture 4.1 Everett Community Partners
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
As described in Chapter 1, Everett, Massachusetts, has seen an influx of immigrants coming from countries such as
Brazil, Haiti, Guatemala, and Morocco. Everett is a small city of about 37,000 with reasonable rents and proximity to
Boston. While there have been tensions in the community about issues related to immigration, such as housing and
parking, it is only recently that the increased activity of Immigration and Customs Enforcement (ICE) has created
challenges for the immigrant community. In particular, with increases in deportation and detention, immigrants fear that
they will be picked up by authorities and deported. Stories of immigrants missing health appointments because ICE was
in the vicinity or having stress-related conditions such as sleeplessness, headaches, and weight loss were common.
These concerns were raised by several of the immigrant advocacy groups and Everett community leaders, who felt that
getting concrete information about this issue would facilitate changes in local policy. So they approached a familiar
academic partner to join them in an investigation of the problem. Their goal was to learn more about the issue and then
solve the problem and develop policy or programmatic interventions that would alleviate some of the stress that
immigrants were experiencing.1
In this scenario, members of the target community approached the researcher to assist in answering what they saw as
a pressing health and social issue in their community. While this may be perceived as a preferable CBPR initiator, there
are many cases of CBPR researchers approaching a community with a research topic, particularly when they are
familiar with members of that community.
Example 2: BMI Disparities in Cambridge, Massachusetts
Figure 4.1 The Cambridge H.E.L.P. Project
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
Our second case as previously described, took place in Cambridge, Massachusetts over a 10-year period, schools,
public health agencies, and researchers had combined forces to track childhood indicators of obesity. Annual height and
weight measurements of children were taken by school personnel and provided to parents each year in the form of a
healthy weight progress report.2 Data accumulated through this process were available and allowed researchers to
examine trends in childhood obesity as well as disparities among racial/ethnic groups.3 The researcher noted that
despite downward trends in BMI in the population overall, there were persistent and glaring disparities by
race/ethnicity, with both Blacks and Hispanics carrying an undue burden of obesity. The researcher and a community
colleague with whom she already had a relationship joined forces to pursue this line of inquiry. They started by
discussing their concerns with several African American colleagues who were leaders in the community and were eager
to work with them. Together, they approached other key leaders in the community, including a principal of a school
with a large Black population. After exploring the existing data, they obtained a small amount of pilot funding to
conduct several key stakeholder interviews with “positive deviants”—local African Americans who had made
significant lifestyle changes and who had made progress toward attaining healthy weight. Using the results of these
interviews, they applied for a larger pilot grant to conduct further analysis of existing data, collect more qualitative,
contextual data, and to work with community partners to build appropriate interventions.
Example 3: Suicide Cluster in Somerville, Massachusetts: Real-Time Health Crisis
Figure 4.2 Suicide Attempts and Completed Suicides Among Somerville Residents Ages 10 to 24 Years
Source: Reproduced by permission from Crisis; Vol. 29(2): 86–95. © 2008 Hogrefe & Huber Publishers, www.hogrefe.com
As previously presented in Chapter 1, our third case took place in Somerville, Massachusetts, an urban city of 70,000
people that borders Cambridge, Massachusetts. Historically, Somerville has been home to working-class populations
and in recent years, between gentrification and new immigration, the city’s demography has changed substantially.
Somerville has also been affected by long-term substance abuse problems, especially heroin and alcohol. In 2001, a
young person took his own life, and this was followed soon after by oxycodone overdoses of two high school students.
A local researcher with an interest and experience in adolescent suicide was concerned that this might represent the
beginning of a suicide cluster. She had prior relationships with community partners and so approached the Health
Department director and mayor to discuss her concerns and interest.
Loss of youth life to suicide and overdose sends enormous ripples of concern through any community, and in
Somerville, the Health and School Departments examined data from their biannual teen health survey to determine if
suicidal behaviors had changed. The teen survey noted that 21% of the students had seriously considered suicide, and
14% had attempted suicide during the last 12 months. This was substantially elevated over previous years and higher
than the average for the state overall.
In order to respond to the situation and investigate further, the mayor convened several task forces and asked the
researcher to join with community members and colead one of the task forces along with the Health Department
Director. Other members included representation from the schools, the police, and community members, as well as
additional experts in suicide clusters. The question posed by the community to the researcher was “How do we stop the
youth suicide?” The researcher helped to reframe the question into two fundamental questions:
• Was this suicide and overdose activity significantly elevated from baseline?
• Were there common links between victims and was this a contagion/cluster?2
FIRST STAGE: DEFINING THE COMMUNITY, ENGAGING THE COMMUNITY, COMMUNITY
NEEDS ASSESSMENT, IDENTIFYING THE RESEARCH QUESTION
Each case presents a slightly different way to consider defining community. As an investigator, you need to consider
what community you want to approach; this may have to do with your content area or with your current locale. Either
way, you need to be aware of the boundaries around the community that you want to engage before taking your next
steps. Community can be defined by geography, by condition, or by other common concerns/characteristics. If you want
to work with immigrants, think through the groups that might represent this population and contact them. If you want to
work with a geographic community, consider the organizations or institutions that serve the community. In the Everett
case, the community essentially defined itself, and community members with a particular concern approached a familiar
researcher. The group of people representing the immigrant community and those that provided services to them
included a broad representation of the leaders from immigrant groups and local institutions, including schools, city
government, and police. This self-defined community, however, did not fully represent the population of interest, and as
the work proceeded, the researcher and the community members made efforts to expand their group to include more
members from the target population: immigrant residents of Everett. Through community partners’ social networks,
introductions were made, and new members joined the CBPR project.
In contrast to this approach, it is not unusual for a researcher who is interested in exploring a specific topic to
approach community partners. The challenge is whether the researcher’s area of interest is also of interest to the
community at large. If the researcher has done an adequate job of assessing the needs of a given community, this should
not pose a problem. In the Cambridge case, it was the researcher who approached the African American community
leaders in Cambridge, but several important points should be noted. First, the project was built on prior existing and
trusting relationships. Thus, the researcher had excellent knowledge of the key stakeholders from the Black community.
These stakeholders were well connected and had the potential to open doors to other community members to further the
work. Second, the topic chosen for study expanded prior work that was already deemed to be of interest to this
community—namely, despite increased community activities for healthy eating and active living, Black youth were still
twice as likely to be overweight or obese compared to their White peers. The practice of CBPR is best built on mutual
interest and existing relationships, as illustrated here. The researcher who has not done her or his homework to
understand the issues facing the community will find CBPR more challenging and is less likely to be successful in such
a pursuit.
In the Somerville example, the community was defined geographically, and the researcher and the community both
identified the problem as it was occurring. The researcher pointed out the emerging problem to local leaders, including
the mayor. Then, as the community began to coalesce around the issue, the researcher was brought in as a pivotal
addition to the community task force and was able to add expertise. It was during the crisis that the researcher and
community partners built new relationships and developed others. In this case, the researcher was really in the thick of
the situation and acted as both a community partner and an investigator toward trying to find solutions.
In all three of these cases, it is hard to separate the “defining the community stage” from the “engagement stage.” The
very process of defining the community leads the researcher to develop relationships, walk the streets, learn the culture,
and immerse him- or herself in the community of interest.
Steps 1 and 2: Defining and Engaging the Community and Community Needs Assessment
Once you have determined your target community, you will need to begin the engagement process. Start by
examining any data available on the community of interest, then contact the local service and civic organizations and
ask for an informational meeting, explore their websites, and learn what they provide. Census data are an excellent
source when thinking about geographic populations. This information is available on the Census website3 and can
provide information on economic conditions as well as demography. In addition, local newspapers are helpful in
learning about local politics. For specific populations, there are likely to be other sources available: immigrant advocacy
groups, national associations, prior publications, and the like. Educating yourself about the community of interest
before you try to engage its members will be helpful in your approach. In addition, in CBPR, don’t go into your
community engagement process with the particular research agenda in mind. You will need to better understand the
community concerns before you can really focus on a particular research question. Otherwise, there is a chance your
interest will not be of interest to your community partners, and you will find the mismatch of priorities frustrating.
How can one learn about current community concerns? In general, as noted above, existing data on community
health and social issues are available through local sources such as school department reports, public health
assessments, or statewide information on localities. In our Everett project, learning about Everett through its citywide
website gave us information on the current issues and community resources. In addition, state data on health issues were
available through the Massachusetts Department of Public Health. These high-level data are helpful as you develop an
understanding of the community, but nothing can replace showing up in the community and getting to know the local
culture and important leaders. This is also the best strategy with which to begin your community engagement activities.
Start with a basic needs assessment. Contact leaders of active community groups such as schools, community-based
organizations, advocacy groups, religious groups, and service agencies. Meet with the directors and find out what they
perceive as the challenges in their community. Think about where the likely places are that serve the population you are
interested in studying. For example, if you are interested in maternal infant health, try working with local pediatricians
or preschool programs. If you are interested in the elderly, contact senior centers. If you are interested in substance
abuse, learn about the local substance abuse provider agencies. Visit the programs and meet their directors. Ask
questions like: What was the stimulus for starting your organization? What do you see as the major issues facing your
community? This process may seem onerous, but the results will provide an important foundation for future work.
These early introductions will serve as the foundations of partnership as you and your community partners learn about
one another. Your approach to these meetings is very important, as a dominant, self-centered approach will likely
negatively impact future potential. Practice cultural humility and bring your listening skills. Academics can be viewed
as aloof and even intimidating to community members. In CBPR, we need to break down these barriers and move
toward equitable partnerships. The engagement process can be very time intensive and may take several years as the
community learns to trust the researcher. Thus, for many early CBPR researchers, it may be easier to build on existing
relationships established by successful CBPR researchers rather than starting from scratch. These investigators may be
helpful in opening doors to the community and in vetting you as someone that can be trusted.
This needs assessment process is an important first step in developing a CBPR agenda, particularly if the researcher
is unfamiliar with the community. Getting to know a community and those who are identified as leaders is step one in
the engagement process. The process of meeting with individuals begins the foundational partnerships that are the basis
of CBPR work. Community members have a chance to vet the researcher and vice versa. In communities where
historical projects have left a distrust of research in general, this is particularly important. Why are you different from
others? What will you provide for the community? How will this project benefit the citizens? This process can be
challenging at first but is ultimately one of the most rewarding components of CBPR.
The data that a researcher obtains in this engagement process are instrumental for community mapping, that is,
developing an in-depth understanding of the political organizational structure, the assets within the community, and the
issues that concern community members. By identifying interpersonal connections, social networks, and the existing
political landscape and infrastructure, one may also understand how things get done, what the opportunities for change
are, and who can help sustain this change. Remember, CBPR is not only about the research itself but also about
resulting action at the community level. In our Cambridge example, the researcher started with the data that had been
collected over many years on schoolchildren’s weight disparities. Along with a public health nutritionist and a doctoral
student, she approached four longtime Cambridge colleagues who were considered “movers and shakers” in the Black
community. These individuals had a commitment to the work and included a school principal, staff from the public
health department, and a local nonprofit. They became “community investigators” on the project and, along with the
researcher and public health nutritionist, identified themselves as the Healthy Eating and Living Project (H.E.L.P.) and
began to investigate the underlying causes for the disparities. Through these relationships, the researcher was able to
identify others who were not only concerned about the issue but were also likely to be able to make programmatic
change. The success of the project relied heavily on this core of community investigators.
In the Somerville example, the engagement process took place at many levels. This included the mayoral level and
other political and institutional leaders. It also included lay leaders in the community most affected by the suicides.
And, finally, it included the youth who were friends of the deceased. Not all engagement proceeded at the same pace,
and there were differing levels of trust. While the crisis demanded immediate teamwork, the relationships at the
grassroots level—that is, the lay community leaders and the youth themselves—were the most difficult to establish. The
researcher ended up in true partnership with the lay leaders but never developed strong partnerships with the youth, in
part because this was ill advised, according to the lay community leaders. Instead, these leaders acted as liaisons, and
the researcher did not push an alternative agenda. These lay leaders were pivotal advisors throughout the crisis.
In a similar fashion, most CBPR projects rely heavily on a community advisory board (CAB) or coalition that will
participate in the research process and function as the liaison to the community. As you build your relationships, you
should be thinking about who will serve on a community advisory group. The CAB will help to hone the research
question and methodology and provide outreach to the population of interest as well as supply knowledge of local
politics, pitfalls, and strategies. Choosing a group of community members to actively become part of your CAB is
critical to the success of the project. Who are you working with? Are they committed to the project? Are they
representative of the community of interest? Have you worked together previously and can you work together going
forward? Do they want to be part of a CBPR project? What are the expectations for participation and how will
information be communicated? In our Everett example, the community members who were already meeting were the
natural CAB. In the Cambridge example, the researcher started with a community partner, and then, together, they
identified and connected with other community members to form the advisory group. In our Somerville example, there
was a task force that the mayor had convened, and this group served as the CAB for the investigation. Most CABs will
meet regularly throughout the project. If grant dollars are available, they should be allocated to support this activity,
including payment for community member time needed to participate.
Step 3: Refine the Research Question
Honing the research question starts with the topic of interest. In many cases, the community questions will be broad,
reflecting timely concerns. “How do we stop drug abuse?” “Why is there youth violence?” “Why is autism increasing?”
Community members are very helpful in identifying the topic of interest but may not have the skills to hone the
research question. That is where the researcher’s skills are most applicable. You should work with community partners
to help focus the research questions. Your knowledge of the literature will help in this process, and by supplying the
existing evidence, you can help the partnership consider what has already been discovered and think about new and
emerging questions that have yet to be answered. Conduct a literature review and provide the important papers to the
CAB. Part of your role is to educate the CAB about what is already known about the subject from a research
perspective, just as the CAB will educate you about the subject from a community perspective. In our Everett example,
the community asked if and how ICE was impacting immigrant health. The researchers identified prior literature on
related subjects and provided this information to the group. This work also influenced the development of the
conceptual framework. The final research questions were more nuanced than the original broad questions and included
an exploration of how local police were perceived and whether ICE had an impact on health care access, on emotional
well-being, or on chronic disease.
In the Somerville, Massachusetts example, in the midst of a health crisis, the community wanted to know how to stop
the crisis. But first, it was important to understand how the crisis was proceeding. Was this a suicide cluster? Were the
young people involved connected to one another? Was activity elevated over baseline? The researchers worked with the
community to focus first on these questions and also to understand how other communities might have dealt with
similar circumstances. The researcher plays an important role in translating the evidence into real-life situations.
Similarly, in Cambridge, the questions were honed to a set of specific goals for the project: (A) understand the socio-
demographic factors associated with obesity among Black school-aged children; (B) understand the social, cultural, and
behavioral barriers and influences on diet and physical activity of Cambridge families; (C) identify a culturally relevant
intervention to promote healthy weight in Black children and their families. This process of taking a community
concern and reframing it into research questions and hypotheses is challenging. In Table 4.1, several examples are
noted.
Table 4.1 Examples of Questions From Community and Researcher Perspectives and Resulting Joint Hypotheses
Important First-Stage Considerations
• Learn about the community you are interested in through an informal needs assessment.
• Go into the community and meet people, begin conversations, get to know the culture.
• Work with a CAB to refine the questions for study.
• Serve as an educator and evidence translator.
SECOND STAGE: DESIGN/HYPOTHESIS TESTING, ROLES AND RESPONSIBILITIES,
CONDUCT OF THE RESEARCH
Step 4: Design and Methods
Once there is consensus on a research question, the CAB and investigator should work together to frame the
hypotheses to be tested and determine the best strategies for answering the research question. This will take place in a
series of meetings of the CAB. While the investigator is likely to know more about the possible methods for
investigation, the community partners possess the expertise to determine the feasibility of these methods. For example,
even though an experimental approach may be the most valuable to answer the question, community members may
have concerns regarding experimental methods due to other issues (time frames, access to participants, competing
priorities). Community partners will need to reflect on how methods can or cannot be utilized and whether they are
practical. In our Everett example, the initial work was exploratory in nature, and so methods that would allow for this
type of exploration were chosen, including interviews, focus groups, and surveys. But while the investigators suggested
that they hold several focus groups in only two languages so they could compare and contrast them, community
members wanted to work with all the dominant language groups in their community in order to get a broad
representation. The research team agreed to conduct six focus groups and changed its perspective to assess information
across immigrant groups. So too, the researchers thought that documented and undocumented immigrants should be
separated since they assumed these populations would have very different opinions on the topic. Community members
felt this posed several problems. Undocumented immigrants were unlikely to come forward if they thought they would
be singled out, and in many cases, families and social networks were made up of both documented and undocumented
individuals. This information was then incorporated into the design. Research design required a negotiation in which the
researcher introduced methods and community members considered the implications and practicality while also
contributing real world evidence.
In our Somerville example, researchers identified methods for monitoring suicide and overdose activity while the
crisis was occurring. This included examining death certificates and using 911 call data from the fire department.
Community members were quick to point out that these sources had their own sets of problems. The death certificate
data were often delayed if the body went to the medical examiner and the immediate cause of death was unknown. The
911 call data provided only initial impressions from the scene and did not provide follow-up of incidents. In turn,
community members noted that youth were using websites to monitor cases themselves and introduced the researchers
to these data sources, which provided information on connections of victims that was unknown to the CAB. As a result,
researchers expanded their data collection to include website monitoring.
In the Cambridge example, parent/child interviews with local Black families were conducted. Conversations were
held with various Black leadership groups (Men’s Health League, Black Pastors Association) and parents and students
at the high school and middle school. A survey of families at two schools was conducted. Introduction and access to
these groups was provided by members of the CAB.
Step 5: Roles and Responsibilities
Once there is consensus on methods, roles and responsibilities for research conduct should be delineated. How will
community partners participate in the research conduct? However, it is also important to remember that roles and
responsibilities require financial resources, so prior to writing any grants, budgetary needs of both researcher and
community member should be considered. In partnership, decide who will be involved in each step of the research
process. This will be dependent to some extent on skills, expertise, and time. In the Everett project, community
members were actively involved in tool development (focus group guides, surveys, and interview guides). They took
the lead on focus group facilitation and participant recruitment. Researchers provided training sessions in focus group
conduct and human subjects. Community members were less involved in analysis; they participated in initial
identification of thematic content, and then researchers took over statistical and qualitative analysis. Results were
presented to the CAB for interpretation and refinement.
In the Somerville example, community members led the task forces and researchers provided them with ongoing data
for data-driven decisions. Then community members provided the context and the interpretation of the data.
Community members were the main investigators when it came to outreach with youth, and researchers played more of
a technical support role.
In the Cambridge example, the researcher and her team did the original quantitative analysis to understand some of
the factors leading to disparities in BMI. The data were presented to the CAB in an iterative fashion, and CAB members
assisted with interpretation. The CAB was also involved in designing all the tools for data collection, including
interview guides and surveys, and members were involved in conducting the interviews. Throughout the entire project,
the CAB members were intimately involved in the interpretation of the research findings.
Each of these examples demonstrates a slightly different approach to roles and responsibilities. However, it is
important to recognize what partners bring to the table in the way of skills and expertise and build from there. Thinking
through how the partnership will work throughout a project is an important part of the grant-writing and project
development process. Transparent communication about expectations is also important, and using memoranda of
agreement (MOA) or other strategies can be helpful to clearly articulate these expectations. Details of items to be
included in an MOA are shown in Table 4.2.
Table 4.2 Components for a Memorandum of Agreement (MOA)
• Describe partners (who are the parties that are entering into this agreement?)
• Dates for MOA (beginning and end of project)
• Roles and responsibilities (who will actually do what; what each party agrees to do)
• Scope of work
• Timeline
• Deliverables (when and what is expected of each partner)
• Budget (how much, billing procedures)
• Publication rights and authorship
• Use of names of partners
• Data ownership (who owns data and how they can be used)
Step 6: Conduct Research
Conducting the research is largely dependent on preparation—that is, how well the roles and responsibilities were
defined and how well procedures were outlined. In addition, adequate training is necessary to ensure accurate and
complete data collection. There must also be a clear understanding of who is responsible for monitoring progress
toward goals and dealing with unexpected events that can range from staff departures to ethical dilemmas to incomplete
data. These responsibilities most often fall to the researchers but may be divided among community members as well.
In the Everett example, community members were responsible for conducting the focus groups and taking notes. But
the research team was responsible for ensuring that the dates for focus groups were set, that the community partners
were trained in focus group conduct and ethics, and that notes were delivered in a timely fashion. While there were
focus groups that lagged behind, frequent meetings of the CAB helped the project adhere to its timeline. A variety of
tools to help in data collection and to decrease variability in data collection were developed, including templates for
focus group notes, short surveys to collect first impressions after the focus groups, grids for analysis, and themes and
scripts for interviews. In CBPR, academics and community partners are mutually dependent. The potential pitfalls are
many. As is often the case, the researcher served as the overall principal investigator for the project and had ultimate
responsibility for adhering to the institutional review board approval. In CBPR, regardless of who leads the project,
there are many moving parts, which only increases the need for strong management. However, decisions are shared, and
conflicts that arise during conduct will need to be collaboratively resolved.
In our Somerville example, all data were collected by the community. These data sources included the youth risk
behavior survey that had been done annually and the 911 data, which were collected monthly by the fire department.
The researchers analyzed and mapped the data. Community members also assisted with the examination of death
certificate data. Today, in part as a result of this project, community members continue to collect these data for ongoing
surveillance.
In our Cambridge example, secondary data on childhood obesity were collected by school physical education
teachers, and the resulting dataset was used for analysis. Community members were involved in the development of all
new data collection tools, and a community member joined a researcher for each of the parent/child interviews. The
researcher, however, was responsible for the overall project and for moving the project forward.
There are many ways to engage the community in the conduct of CBPR. Their involvement in data collection can
build their individual skills and the capacity of the community around data overall. However, in a participatory process,
management is required, and this is always harder when responsibilities are shared. The MOA plays an important role
in setting the stage for CBPR, but it will be important to review progress regularly at CAB meetings, to deal with
unexpected challenges, and to ensure the work gets done.
Second-Stage Considerations
• Work in partnership with community members to design the research and choose appropriate methods: balance
rigor and practicality.
• Decide who is going to do what and make sure the resources are in place to get the work done.
• Monitor the conduct of the research, “make sure the trains are running on time,” and develop strategies for
addressing unexpected pitfalls.
THIRD STAGE: ANALYSIS, INTERPRETATION, AND DISSEMINATION
Step 7: Analysis and Interpretation
Once roles have been clarified, think through the analysis and interpretation plan and discuss how both community
members and researchers can be involved. In CBPR, it is not necessary that everyone is involved in everything to the
same extent; however, it is important to recognize and address issues of data ownership. Remember, in CBPR, data
ownership is something that should be negotiated up front, as data are not only the property of the researcher but also of
the community partners. Negotiating decisions about how data will be used, when they can be shared with those
external to the CAB, and who will house and “own” the data in the present and future will take some investment but
will help solve potential future problems. Memoranda of agreement (see Table 4.2), referred to previously, can be
helpful in documenting how these decisions should be made.
Roles in analysis often depend on expertise, time, and interest. For example, analyzing quantitative data using
statistical methods is often best left to the researcher and the academic team, who have knowledge of statistical
programs. However, there may be community members who want to learn more. In our Cambridge obesity example,
the CAB decided on a preliminary analysis plan, that is, what variables they would explore and what statistical methods
would be used. Then the researchers conducted analysis and brought frequencies back to the team prior to doing more
extensive analysis to understand the questions that community members might observe. This assigning of meaning is
one of the most important and valuable contributions of CBPR. The community partners will have an understanding of
the context and meaning of these results. Their questions may be very different than those of the researchers. Their
insight is extremely critical for later community action. Once additional decisions about analysis were made, the
research team ran the analyses and once again brought it to the CAB. This iterative process helped to move the analysis
forward in a manner that incorporated the multiple perspectives and enriched the end product. The CAB will be
instrumental in interpreting results of any CBPR project as members bring their own understanding of the community
issues to the process.
In our Somerville example, at the request of community partners, researchers took data from 911 calls and mapped
them onto a map of Somerville using mapping software, and this revealed that certain areas of the community had
higher levels of overdose and suicide activity than others. Community members could easily point out why these areas
might have higher activity; for example, the area around a local college had a high level of alcohol 911 calls, while the
area around the local housing project was known for drug activity. This information about the context of the community
was particularly important in framing the meaning of the data and their interpretation. As a researcher involved in
CBPR, recognizing what community members know about their community can be extremely helpful in this phase of
the research.
Qualitative data may be somewhat more approachable than quantitative data for community partners. Listening to
constituents is something that they do regularly. However, few are used to collecting and analyzing this type of data in a
systematic manner, and it will be important to warn against jumping to conclusions based on one interview or focus
group. The researcher can help facilitate the analysis of qualitative data by integrating community partners into the
process. Again, this can be an iterative process, with initial work being done by the researcher and then presented to the
CAB for refinement, reanalysis, and presentation again until all are satisfied with the codebook. This was the process
undertaken with the focus groups in the Everett example. After completion of the focus groups, the CAB identified
initial themes. Then the data were entered into a software package and further categorized, then presented to the CAB
and modified and reanalyzed again. The process took about 3 months but benefited from the multiple perspectives of
both researchers and CAB members. In short, while analysis may seem a cumbersome process to share between
researcher and community, it can be done through creative strategies that ultimately enrich the process and improve the
relevancy of the work.
Step 8: Dissemination
Lastly, the dissemination of results as a final step in the CBPR process often takes multiple forms. As a researcher,
you are likely to be interested in peer-reviewed journals, while community partners are much more likely to want
immediate dissemination so that they can utilize the results in action. This can present many conflicts for researcher and
community alike. For example, in Everett, the community wanted a forum at which results could be presented and
recommendations discussed. The researchers wanted to produce a paper for publication and were concerned about how
release of data would impact their ability to publish. They agreed on a compromise that resulted in several strategies.
First, a large community forum with a PowerPoint presentation of the high-level results was presented in a digestible
form at the end of the project. Community members took the lead on orchestrating the forum, including the invitation
list, the food, and the space. The researchers took the lead on the initial draft of the presentation, and then the
PowerPoint was circulated among all partners for critique and edits. The forum also had an interactive component in
which community members had a chance to develop recommendations for action based on the presentation. The list of
recommendations was finalized and presented to institutional leaders (mayor, police chief, and schools). Advocacy
agencies took the recommendations and incorporated them into their activities. Simultaneously, the researcher
developed an outline for a peer-reviewed paper and asked all those on the CAB if they wanted to be authors. Authorship
brought responsibilities, including some editing and writing. Not everyone wanted to participate, but over the course of
the next year, all the authors (there were 10 in total) had a chance to review drafts, comment, and provide feedback.
Two papers from the project were accepted for publication. The community, meanwhile, acted on some of the
recommendations. They began meeting with local police to address traffic stops, which changed the dynamics between
the immigrant community and the police overall. Almost a year later, the police were no longer “arresting” people who
did not have driver’s licenses. Instead, they would issue citations. This meant that undocumented immigrants were able
to avoid fingerprinting and reporting to ICE. The police/immigrant relations improved as regular meetings with police
were held. One immigrant advocacy leader noted that the number of complaints about police from the immigrant
community had fallen dramatically.
Dissemination in the Cambridge example was handled somewhat differently but with similar attributes. A formal
report was made to the chief public health officer for the city and to the CAB. A final written report was drafted and
vetted by community partners.4 In turn, this was presented to other Cambridge leaders to get their perspectives on
potential action steps. The CAB was able to obtain some additional funding to maintain its work as it applied for
additional funds to pilot test an intervention.
Overall, the dissemination process should be considered in two simultaneous manners: that is, what and how to
disseminate results for the benefit of the community members while also publishing results in peer-reviewed journals
for generalizability. Both academic and community partners are driven by real incentives, and these end products
highlight the differences between them. While these two agendas can conflict, there are ways in which both parties can
be satisfied and gain from the process. The two dissemination strategies may also have different timelines; the
community may want immediate results that can be utilized in practical applications at either the policy or
programmatic level. The time for article production may be much longer and somewhat irrelevant to the community but
important to the researcher. Straddling these potentially conflicting goals is difficult, but as long as communication is
open and transparent, the dissemination process can flow smoothly and obstacles can be addressed. There is no question
that group dissemination is cumbersome, but the differing perspectives enrich the final products and make them not
only more relevant to the community but also more likely to have sustainable outcomes and a “life” after the CBPR
project is completed.
Identifying action steps is really in the hands of the community partners. While it is part of the dissemination process,
the researcher may or may not be involved. However, the researcher can play an important role in sustaining the CBPR
collaboration. In our three examples, action steps resulting from the CBPR projects included the following:
1. Everett: regular meetings held with immigrants and police to enhance communication; health care providers
improved awareness how ICE affects their immigrant patients.
2. Somerville: regular monitoring of 911 activity; development of a citywide trauma response network.
3. Cambridge: recommendations developed for future interventions, including: encourage youth to drink water,
expand programming for Black girls to be physically active, limit TV consumption, and avoid using food to
comfort children when they are upset.
Third-Stage Considerations
• Determine how partners will be involved in both analysis and interpretation.
• Use an iterative process to assign meaning and generate consensus on that meaning.
• Utilize multiple dissemination strategies important to both community members and academics.
• Identify action steps.
CONCLUSION
In summary, there are clear steps in the CBPR process that should be considered (see Table 4.3). Engagement of the
community, defining the questions, conducting the research, and disseminating findings all take time and should be
addressed in a participatory manner. The CAB provides an excellent organizing strategy for working with community
partners, but the research team of community members and academics must map out communication and roles and
responsibilities and conduct the study in an environment of mutual respect.
Table 4.3 Summary of CBPR Steps
Stage 1
Community Engagement5–7
• Identify the community of interest.
• Conduct a needs assessment to understand the issues for potential research.
• Meet the stakeholders.
• Develop relationships.
• Assemble a CAB.
• Choose an area for research.
• Conduct a literature review.
• Hone the research questions and hypotheses.
Stage 2
Research Design, Roles, and Responsibilities
• Discuss methods.
• Assess feasibility.
• Define roles for community and for researcher.
• Provide education and skill development.
• Conduct study.
Stage 3
Analysis, Interpretation, and Dissemination
• Assess skills.
• Develop iterative processes.
• Work with CAB for interpretation and context.
• Determine dissemination modes.
• Develop processes that allow all to participate.
• Identify action steps based on evidence.
QUESTIONS AND ACTIVITIES
The purpose of this assignment is to help students put ideas about CBPR into practice. If you can actually assign your
students to work with local community partners, you can provide them with real-world experience. If that is not an
option, have them work in groups to develop a CBPR research proposal that addresses one of the community health
problems described below (i.e., childhood obesity, youth suicide, disparities in mortality among women living with
HIV, or disparities in cardiovascular disease among Black men).
Guidance
1. Describe your CBPR approach.
2. Who are your community partners and how will you engage them?
3. What is your research design?
• Sample
• Recruitment
• Methods
• Analysis (optional)
4. How will you disseminate findings? What will you do with the findings?
5. Identify potential challenges or limitations.
Scenario 1—Youth Suicide
Community Context
Setting: a community that is suburban with a population of about 80,000. The suburb is fairly wealthy, with average
home costs hovering around $500,000. It has a reasonable tax base and is home to an outstanding university, where you
work. The population is by and large White, and there is not much diversity economically, but the population is
changing as educated South Asians move into the community.
Description of a health problem: In 2010, one teenager who was attending the local high school committed suicide
via a pill overdose. Within several months, her best friend also left a note, and she too committed suicide, but she
jumped from a building in the downtown area. Parents of high school children are in an uproar, demanding that the
school do something to prevent another suicide. Teens are holding vigils for their lost friends.
Partners That Came to You With the Problem
Members of the school committee have convened a task force of parents, teachers, and members of the local
churches.
Health Question
What is happening in their community that could be leading to such a situation? They also want to understand if these
two suicides represent a “cluster” or “contagious” situation. They feel they need this information to inform their next
steps.
Scenario 2—Cardiovascular Disease in Men
Community Context
This is a city in the Midwest of about 1 million. There has been a real economic downturn here, and the
unemployment rate is upward of 10%. There is a large Black population in one area of the city, which is also an area
with high rates of poverty and even higher rates of unemployment.
Health Problem
The rates of stroke and heart attacks in the Black male population are alarming and far outstrip those for Whites in
the city. There are two hospitals, one catering to the small economically well-off population and one that is the
“poverty” hospital.
Partners That Came to You With the Problem
Providers at the poverty hospital have seen ever-increasing rates of young men coming in with strokes. They want to
do something to change the situation, but many men do not seek health care in part because they are uninsured and
unemployed.
Health Question
How can the health care system identify and reach out to men of color who are at risk for cardiovascular disease?
Scenario 3—Childhood Obesity
Community Context
This is a middle-income city on the West Coast with a population of approximately 100,000 persons. Approximately
30% of the population is under the age of 18, the majority of whom attend public schools across the city. The city is
racially and ethnically diverse. There is a large immigrant community from Mexico and other Central American
countries. There is also a growing Southeast Asian population.
Health Problem
In 2004, all of the city’s schools began measuring students’ BMI and reporting back to parents the results of the
assessment. Educational materials have been sent home with the reports, and school nurses are available to follow up
with parents who are interested in learning more about what they can do to improve their children’s health. Since 2004,
BMI scores have steadily increased among every racial and ethnic group in the city. BMI scores are particularly high in
several elementary schools.
Partners That Came to You With the Problem
The PE teacher in one school with particularly high BMI scores among students came to you with the data the school
has been collecting over the last few years. The PE teacher has been in touch with the school’s family council, PTA,
teachers, and administrators.
Potential Health
Questions
Why are BMI scores increasing at a higher rate among students in some schools? What underlies the increase in BMI
scores?
Questions
1. What are some of the factors to consider in engaging community partners in analysis of data?
2. How does the process of dissemination differ in a CBPR project from a non-CBPR process?
3. In what ways would community partners use data from a CBPR project to take action?
NOTES
1. Hacker K, Chu J, Leung C, Marra R, Pirie A, Brahimi M, English M, Beckmann J, Acevedo-Garcia D, Marlin RP. The impact of
Immigration and Customs Enforcement on immigrant health: perceptions of immigrants in Everett, Massachusetts, USA. Social Science &
Medicine. 2011 Aug;73(4):586–94.
2. Hacker K, Collins J, Gross-Young L, Almeida S, Burke N. Coping with youth suicide and overdose: one community’s efforts to
investigate, intervene, and prevent suicide contagion. Crisis. 2008;29(2):86–95.
3. U.S. Census Bureau. 2006–2008 American Community Survey 3-Year Estimates. 2008 [cited 2010 October 3]; Available from
http://factfinder2.census.gov.
4. Chomitz V, Arsenault L, Banks C, et al. H.E.L.P. Culminating Report. Cambridge, MA: Institute for Community Health; 2011.
5. Clinical and Translational Science Awards Consortium. Community Engagement Key Function Committee Task Force on the Principles of
Community Engagement. Principles of Community Engagement. 2nd ed. Rockville, MD: NIH; 2011.
6. Israel BA, Parker EA, Rowe Z, et al. Community-based participatory research: lessons learned from the Centers for Children’s
Environmental Health and Disease Prevention Research. Environmental Health Perspectives. 2005 Oct; 113(10):1463–71.
7. Minkler M, Wallerstein N., eds. Community-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass; 2003.
T
3
Methods for CBPR
“Does CBPR add value to health research, or is the very premise of involving communities in the conduct of research contradictory to the
tenets of ‘science’ defined traditionally as expert based and objective? Does CBPR truly shift the power relations between the observer
and observed? And if so, does this serve the public health agenda of improving the science on which that agenda is based?”1
he engagement of community partners in the research enterprise is critical for improving human health and
advancing science.2 CBPR is an approach in which community members are actively involved in every facet of the
research, shaping it, conducting it, interpreting results, and disseminating findings.3 This approach not only changes
traditional research paradigms but also has implications for the way we think about the scientific enterprise. When
research is intimately connected to the communities under study, it should enhance the value of research for those
communities. But will that mean that the results of the research can also be applied outside that community? As CBPR
researchers, we must employ rigorous methods that approach the problem in a systematic way and have the potential for
generalizability while also incorporating a participatory framework. Another way of thinking about the CBPR approach
is that it requires research methods that simultaneously meet standards of scientific merit and acceptability and
feasibility for community partners. To help the investigator better understand how methods are considered in CBPR,
this chapter will describe:
1. Advantages and challenges of CBPR in the research process
2. The research question
3. Conceptual models and theorizing
4. Choice of methods
5. Sampling
6. Design considerations
ADVANTAGES AND CHALLENGES OF CBPR IN THE RESEARCH PROCESS
The research process involves a series of steps that include theory, hypotheses, data collection, and analysis (see Figure
3.1).4 Theory helps us make sense of interrelated phenomena in the social world.4 Hypotheses are educated guesses
about relationships between two or more characteristics that are usually based on observation or the literature. They can
be proven true or false through the research process that includes data collection and analysis. While CBPR utilizes all
the same steps as any research process, community participation creates an added layer that poses both challenges and
opportunities for the research process. Perhaps one of the most important advantages of CBPR is its ability to enhance
certain dimensions of validity and authenticity. CBPR offers an opportunity for colearning—that is, the researcher
learns from the community as the community learns from the researcher.
Figure 3.1 The Research Circle
Source: Schutt, R. Investigating the Social World 6e, p. 42. Copyright © 2009 SAGE Publications.
In Schutt’s book Investigating the Social World,4 various facets of validity and authenticity are defined. Validity
refers to how accurately your research conclusions correspond to the real world. The three dimensions of validity
include measurement validity, generalizability, and causal validity. Measurement validity focuses on our ability to
actually measure what we think we are measuring. Generalizability exists when our conclusions hold true for the
community, population, or setting we specify. Sample generalizability refers to our ability to generalize from the
sample to the population, while external validity or cross-population validity refers to our ability to generalize our
findings from one group to other groups, populations, and settings. Causal validity refers to the state when your
conclusion that A leads to B is correct.4 The advantages and disadvantaged of achieving each form of validity in CBPR
will be addressed throughout this chapter.
Finally, authenticity refers to how well the resulting understanding of the social process produced by the research
reflects the various perspectives of participants in the setting.4 The CBPR approach increases authenticity, as
community participation assists with making the research more relevant to the community under study. This is one of
the most unique and important aspects of CBPR. It offers the researcher a window into a perspective that he or she
would normally find inaccessible. The colearning process inherent in CBPR enhances authenticity dramatically by
providing insight into the research process from multiple perspectives. In our Everett, Massachusetts, example, where
we studied the implications of Immigration and Customs Enforcement (ICE) on immigrant health, the advantages of the
CBPR process included the provision of a timely topic area that was particularly important to the community—ICE and
immigrant health—along with important input from the immigrant communities. Neither of these would have been
identified without the participation of community members.
The process of coeducating and colearning that occurs in CBPR has the advantage of providing a multidisciplinary
approach to problem solving. Not only are there opportunities for sharing information and knowledge across the
community/academic continuum, but there is joint learning that occurs throughout the experience that has implications
for future action, sustainability, and research. If the researcher is open to the ways the community might answer
pressing questions, it can result in better, more applicable approaches to generating knowledge.
But CBPR also poses methodological challenges. With increased relevancy, there may be questions about validity
and external generalizability, since the work is locally based. There may also be challenges to design and methods.
Community participation may influence choice of the research questions, sampling methods, and the recruitment of the
sample. The preferred methods of community partners may conflict with those of researchers. In this chapter, we will
discuss important elements of the research process and the unique methodological advantages and challenges of CBPR.
THE RESEARCH QUESTION
Working with community partners to hone a research question can be an invigorating and creative experience. Their
insights, provided from multiple perspectives, can reveal aspects of the problem that the researcher may never have
considered. However, a community’s questions may not always reflect what social scientists have identified as the key
unanswered questions. In general, community partners often have broad questions that reflect current problems in their
community, and their knowledge of existing literature may be lacking. They want to understand youth violence or
underperforming students or reasons for homelessness. In response, it is necessary to refine the questions such that the
researcher may need to broaden his or her questions and the community may need to narrow theirs. In Everett,
Massachusetts, the community had a particular concern: they were worried that immigrants in their community were
experiencing health problems due to immigration enforcement. When they approached the researcher, they had a broad
question that they wanted to explore: Does ICE impact immigrant health? The refinement process was collaborative and
multifaceted. First, the researchers provided a review and interpretation of the existing literature on the topic in the
initial meetings of the academic/community research team. Community partners may or may not be aware of the
literature on a topic. They are unlikely to be familiar with academic articles or possess knowledge to interpret them. Yet
understanding the state of the research and the new and emerging questions is an important task in the research process.
This is one of the skill sets that the researcher brings to the CBPR process. For Everett partners, the literature revealed
that little work had been done in this area. The limited prior literature was focused on experiences with a previously
repealed California law that limited immigrants’ access to health care. The literature review helped to identify a host of
new questions that the community/research partnership reviewed and considered. In addition, the team was able to
establish the need for this work at both the local and national levels. Questions that emerged from the literature
included: Does immigration enforcement lead to lower use of health services? Does fear of deportation result in
worsening of health conditions? Does deportation fear differ across immigrant groups?
In order to effectively work through the development and refinement of the research questions in CBPR, both
researchers and community partners need to be able to get their perspectives heard and incorporated. Question
refinement and hypothesis generation can take some time but will have a secondary effect of deepening collaboration.
In addition, the conceptual framework can help the CAB identify effective methods and short- and long-term outcomes.
CONCEPTUAL MODEL AND THEORIZING
A conceptual model or framework provides a visual representation of how a set of factors relate to one another and are
thought to impact or lead to a target outcome. It can be extremely helpful in connecting the research questions to theory
and further refining the hypotheses. As a next step, researchers in Everett took the information gleaned from the
literature along with perceptions of community members and drafted a conceptual model of the project. Building a
conceptual model was an excellent tool to help the partners identify the mechanism by which they believed that ICE
was having an impact on immigrant health. In turn, this model helped the research team further develop research
questions and hypotheses. In addition to the conceptual model, the researchers provided information on social theories.
In particular for this project, theories of social capital and segmented assimilation theories were presented.5–7 Using the
draft conceptual model as a starting point, the team had an opportunity to provide input on the mechanism behind the
impact of ICE on immigrant health. This exercise helped the community build its theory about the elements of the
problem it was exploring.
To demonstrate one of the values of CBPR, the two versions of the conceptual framework are shown below. The
version initially presented by researchers changed substantially as community members provided their contributions.
While researchers thought about the implications of increased fear on individual mental health and stress levels (Figure
3.2), community members pointed out that much of the problem was being driven by having or not having a driver’s
license. Immigrants who did not have driver’s licenses were at risk for being arrested by police when they were stopped
for even a minor traffic violation. If arrested, fingerprinting created high anxiety, as immigrants feared their information
would be shared with immigration authorities. This was incorporated into a final conceptual framework for the project
(Figure 3.3).
Figure 3.2 Conceptual Framework for the Impact of ICE Efforts on Immigrant Health
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
The visual aid along with the literature review previously mentioned helped the group further refine its set of research
questions and associated hypotheses. In CBPR, once the research process begins, new and emerging research questions
will surface, as the process is iterative and tends to move from broad to more specific. Here are some examples of
research questions that were informed by the literature review, the conceptual framework, and theory:
1. What were the perceptions of different immigrant communities about local police and the relationship to ICE?
Hypothesis 1: Immigrants did not distinguish between law enforcement and ICE.
Hypothesis 2: Undocumented immigrants were more likely to have their health more negatively impacted by
ICE than those that were documented.
2. How did the presence of ICE in the community impact access to health care?
Hypothesis: People exposed to high rates of deportation in their community were afraid to go out of their
houses and would therefore miss doctors’ appointments.
3. How did immigrants who came from countries where they had experienced trauma differ from immigrants who
did not experience trauma in their response to ICE?
Hypothesis: Immigrants from countries where they had experienced trauma would be more likely to have
physical symptoms of stress resulting from ICE compared to those who did not experience trauma.
Figure 3.3 Conceptual Framework for the Impact of ICE Efforts on Immigrant Health (Version 2)
Source: Reprinted from Social Science and Medicine 73 (2011) 586–594; Hacker, K., Chu, J., Leung, C., Marra, R., Pirie, A., Brahimi, M.,
English, M., Beckmann, J., Acevedo-Garcia, D., & Marlin, R. “The impact of Immigration and Customs Enforcement on immigrant health:
Perceptions of immigrants in Everett, Massachusetts, USA” page 588. Copyright 2011 with permission from Elsevier.
CHOICE OF METHODS
Once the research question is defined and the hypotheses delineated, it is time to match the methods to the question(s).
As with any research project, the methods used in CBPR should adequately address the question of interest and meet
the standards of rigor used in scientific investigation. However, when working in CBPR, balancing the needs and
desires of the community with the standard of rigorous science is fundamental to the premise of CBPR. The community
context is fluid and requires flexibility not typically seen in traditional research. Decisions about methods should be
made jointly. The acceptability of those methods—that is, whether the community considers methods appropriate for its
context—must also be assessed. In addition, feasibility, or whether the research can practically be carried out in the
community, will need to be determined by the community. Thus, determining the best research methods for the project
often requires a give and take between the researcher, who possesses knowledge of scientific inquiry and its design and
methods, and the community members, who possess knowledge of community context and what is possible for
political, historical, and practical reasons. With the help of community partners, novel and more applicable methods
may be identified that provide more appropriate strategies for generating knowledge than those proposed by the
researcher.
A discussion of the limitations and strengths of particular methods should be a group process. There are some pivotal
research concepts that can facilitate the discussion, particularly the concept and dimensions of validity and systematic
bias. Community members are often unfamiliar with these terms, and it is contingent upon researchers to translate them
to their community partners. These concepts are critical for understanding the strengths and limitations of the chosen
research methods and the impact on conclusions. The resulting strength of evidence will be dependent on eliminating as
much systematic bias as possible and maximizing validity. These fundamental research concepts can help community
partners understand the rationale for certain methods, what is involved in increasing research rigor (Figure 3.4), and
more importantly, what conclusions they will be able to draw based on the chosen methods and design.
To date, it is still rare for CBPR to utilize experimental designs,3 which limits the extent to which CBPR can
establish causal validity. Experimental designs require at least one group that receives some treatment and a control or
comparison group that does not receive the treatment. Participants are randomly assigned to the control group. Random
sampling is a sampling method that identifies subjects through chance. It can help to decrease sampling bias, which is
the “over or underrepresentation of some population characteristics in a sample due to the method used to select the
sample.”4 True experimental design is often not feasible in community settings for reasons that will be discussed later.
In addition, the need for highly controlled methods may not be readily accepted by community partners or necessary to
answer the question under study. Today, it is much more common to see nonexperimental methods utilized in CBPR
studies.
Figure 3.4 Increasing Research Rigor
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
Nonexperimental Methods
Nonexperimental methods include qualitative methods such as focus groups and interviews and quantitative methods
such as retrospective data review and surveys. These methods have the advantage of being easily adapted to a
participatory approach. With training, community partners can be involved in conducting observations, taking notes,
facilitating focus groups, or developing and conducting surveys. They can also be involved in the recruitment of
participants for any of these activities. Each of these methods also has potential utility for their future work, thus
building important community capacity. For example, community partners may utilize surveys as part of community
needs assessment and evaluation activities.
In our Everett example, community members wanted to learn more from those most affected by the issue under
study. Given the subject, formative and associational methods that would test the hypotheses were chosen. From the
onset, the research team was aware that the results of their work were likely to be limited to Everett and not have cross-
population generalizability. However, since little had been published on the subject, they also felt they would be adding
to the literature on the topic. The team agreed on a combination of methods that included both qualitative and
quantitative data collection. These included (1) focus groups, (2) a survey, and (3) interviews. These would allow for
triangulation of data in order to better understand the situation and plan for future interventions.
The partners chose to conduct a series of focus groups with the five dominant language groups in their community:
Portuguese (Brazilian), Spanish (Central American), Haitian (Creole), Arabic (Moroccan), and English (for those
immigrants that were bilingual in English). Secondarily, they wanted to understand the situation from the perspective of
local physicians, so an online survey methodology was chosen to access busy physicians. The third method allowed
them to learn more about the problem from key community stakeholders through a series of hour-long interviews.8
These three methods all lent themselves to a collaborative, participatory approach.
Focus groups are designed to get the opinion of a group.4 To begin, Everett team members identified the key
questions they felt were important and developed and piloted the moderator’s guide. The moderator’s guide included a
range of questions that were designed to test the hypotheses previously identified. For example, there was a series of
questions that asked about relationships with local law enforcement to try to tease out how immigrants felt about police
and whether they associated them with ICE. Once the guide was completed, the group decided to have community
partners facilitate the focus groups in the appropriate languages. Toward this goal, the researchers trained the
community leaders to facilitate the focus groups and serve as note takers. Community partners also had the ability to
recruit focus group participants. Through participants contacting other potential participants, more than 50 people were
recruited to participate. Focus groups were held in community settings to minimize anxiety of participants.8
The second method that Everett partners selected was an online survey for community health providers in order to
determine whether these providers were identifying an impact on their immigrant patients. Since there were no
available surveys that had been previously validated, the team developed its own questions. While CBPR allows for the
mutual development of community-relevant questions, this may sacrifice measurement validity—that is, ensuring that
“a measure measures what we think it measures.”4 These community-developed questions have not been field tested,
nor have they gone through validity testing. This is not an uncommon situation in CBPR. While community partners
provide excellent ideas for question development, they generally do not have extensive experience writing effective
survey questions, which may lead to confusing phrasing, problems with response categories, overlapping dimensions of
concepts, and other survey errors. Researchers can help their community partners with survey design and improve
measurement validity by building on existing instruments and refining and testing questions before release. The benefit
of survey methods along with the insight of community partners can work together to produce high-quality survey
instruments.
Evaluation Research
Another design strategy that is frequently seen in CBPR is the use of evaluation research to gauge the impact of
particular interventions or assess the needs of the community. Today, the incorporation of evaluative methods in
community health programming is widespread. Since the 1990s, more and more grant-funded programs require some
level of evaluation, that is, “Show us how you intend to measure the outcomes of your proposed program.” Whether it
is a new or existing intervention, community members are often very interested in determining what programs and
services work or might not work in their communities.
Evaluation research
is generally considered the “systematic
collection of information about the activities, characteristics, and outcomes of programs.”2 This might include needs
assessment and formative research as well as experimental intervention research. Much of what is done in CBPR could
fall within the definition of evaluation research. When evaluation researchers focus on the identification of the effects of
a program, they frequently utilize a pre-post design with or without a control group. This type of design can be
represented with a logic model, which is a descriptive model of how a program operates.4 Using a logic model with
community partners can also be a helpful exercise, particularly if doing intervention research. The logic model has been
used by program developers and evaluators to map out the premise behind the program and identify the measurable
indicators for evaluation.9 A template for logic models is included in Figure 3.5.
Figure 3.5 How to Read a Logic Model
Source: W.K. Kellogg Foundation Logic Model Development Guide, W.K. Kellogg Foundation Battlecreek MI Jan 2004 Item #1209 p.3
Available online http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx
Many community members will benefit from learning basic evaluation techniques and incorporating appropriate
measures into their work for process improvement. This process improvement in community programming resembles
the Plan Do Study Act process used throughout industry. Figure 3.6, shown previously in Chapter 1, demonstrates how
research can be used for process improvement. Community partners learn to question, study, and assess their results and
then adapt and adjust for improvement. An academic partner can assist communities with the development of a logic
model, framing of achievable short- and long-term goals, and identification of measures and measurement tools, as well
as choice of evaluation design.
Overall, the research design for any CBPR project should be appropriate to answer the question posed. In our Everett
example, some important points were identified. First, the background literature provided by the researcher helped
inform the community about existing evidence in the area. Second, the joint effort to build a conceptual framework
provided a starting point for determining the appropriate methods for the project. The process also allowed researchers
to better understand the current community political and cultural context and revealed previously unknown factors that
might influence immigrants’ response to ICE.
Figure 3.6 Research for Process Improvement
Source: Reproduced by permission from the Institute for Community Health, Cambridge, MA; 2011.
SAMPLING
The concept of sampling is of major import in the research process, yet it can be difficult to explain and manage in
CBPR. Appropriate sampling will help minimize sampling bias—that is, when characteristics of the sample do not
correlate to those of the population from which the sample is drawn. While one of the benefits of CBPR is that
community partners can reach out to the community and thus improve recruitment success and hopefully sample
generalizability, they may not consider the importance of keeping track of who refuses participation or exactly how
many people they approach. Yet if they do not understand the concept of sampling bias, they may end up recruiting a
group of people that does not reflect the characteristics of the population of interest and negates the validity of their
work by drawing inaccurate conclusions based on problematic data. Therefore, just as it is important for community
partners to explain the nuances of outreach and engagement to researchers, it is important for researchers to assist
community partners in understanding the value of sample generalizability so that critical mistakes can be avoided. In
Everett, community partners were so enthusiastic about conducting focus groups, they did not initially think about
documenting their recruitment process. Part of the researcher’s responsibility is to help build capacity for research in
their community partners. A researcher can explain important research concepts such as selection bias, validity, and
sampling. When partners understand why it is important to collect this data in a specific manner, they are more apt to
comply, and over time, these concepts will become part of the CBPR process as partners incorporate their meaning and
value.
DESIGN CONSIDERATIONS
Community and academic incentives to engage in CBPR are generally different. Communities are likely to be more
interested in the results of research activities as they pertain to local action rather than their implications for the field.
Their time frames for action may be shorter and their tolerance lower for tightly controlled studies. This is one of the
fundamental challenges in CBPR: How can research be meaningful to the community while also providing cross-
population generalizability outside the community? To some extent, the answer to this question lies in whether the
population under study and the community context itself was comparable to other populations in other communities.
While experimental designs will help achieve cross-population generalizability of study findings, they are not the only
factor to be considered. Today, CBPR researchers are attempting to better understand these contextual factors inherent
in communities and CBPR partnerships that shape the “nature of the research and the partnership”10 and ultimately help
disseminate effective approaches.
While the number of experimental trials conducted using CBPR is increasing, to date, the majority of CBPR studies
have largely been observational and quasi-experimental (studies that do not randomly assign participants but use
matched control groups instead).3, 11 As noted in the Agency for Healthcare Research and Quality (AHRQ) review of
2004,3 CBPR builds trust with communities and improves relevancy and recruitment, but the scientific rigor of CBPR is
still in question. One reason for this conclusion is the lack of experimental methods that pose threats to causal validity.
Why is it challenging to use randomization in CBPR? In part, community partners may be hesitant to consider
randomization, which excludes some community members from the receipt of an intervention while offering it to
others. They may feel that restricting service is inappropriate, especially for disadvantaged populations. For example, a
behavioral intervention that provides education and peer support to help teen mothers practice safe sex should be
offered to all regardless of whether the program is ultimately shown to “work.” The community may place more value
on obtaining additional services than on testing the value of those services. This may be frustrating to the researcher
who wants to identify scientific evidence for the uptake of new community interventions.
Another consideration is the feasibility of conducting a randomized experimental design in a community setting. The
CBPR partnership needs to assess whether this level of experimental research can actually be achieved. These studies
generally require more time and resources than other designs and are most often led by the researcher. They may not
lend themselves to a participatory approach. Most importantly, they require extensive control over all elements of the
study. Thus, undertaking a randomized experimental design using a CBPR approach will require the partnership to
negotiate roles and responsibilities, especially since the flexibility inherent in a CBPR approach may directly conflict
with the control needed to conduct the research. Rather than individual randomization, strategies such as cluster
randomization, in which clusters of social groups are randomized (churches, community clinics, schools), or delayed
interventions, in which the comparison group becomes the experimental group and receives the intervention in a
delayed fashion, may be more palatable for community partners.
While the challenges are numerous, experimental designs are possible. As CBPR partnerships evolve, they may be
more likely to consider a higher level of rigor in their research methodology based on their trust of the researcher and
their understanding of the value of experimental research designs. This may follow from earlier pilot work in which
lessons have been learned.
CONCLUSION
Refining the research question is an important place to start in CBPR. Once achieved, developing a consensus around
methods that are rigorous and simultaneously feasible and acceptable to community partners is critical. The researcher
who understands the inherent limitations of certain methods will not only achieve better long-term results but also is
more likely to build community partnerships that endure for the future.
As partnerships mature and community partners gain confidence with research methods, more complex designs may
be possible. Throughout the research process, there is an ongoing colearning process that is unique to a CBPR
approach. Both researchers and community partners build their capacity, and as partnerships deepen and endure, the
possibilities for future beneficial CBPR projects increase.
QUESTIONS AND ACTIVITIES
Activities
Have students discuss the steps to the research process as they differ in CBPR compared to traditional research.
1. Consider the following case:
Community members want to address violence in their community. They want to understand why young people in
one area of the city are involved in the majority of the drive-by shootings. They ask you as a researcher to help them
learn about the risk factors for violence in youth.
Describe the process for refining the research question in a way that utilizes the principles of CBPR.
2. A community group is interested in trying to identify whether a program for overweight children can actually
impact BMI. The program is a 10-week educational intervention delivered at a community youth development
site by youth leaders.
Using this case study, have students map out a logic model of the program and include appropriate measures.
3. You are approached by a community partner that runs a program for out-of-school youth designed to help them
get their high school certificates. The program does not seem to be having the desired impact. The program runs
for six sessions. Staff currently keep records of participation and high school completion.
Describe how you would approach the evaluation of this project and the methods you might suggest. Discuss the
relative merits of an experimental versus nonexperimental design within the context of CBPR.
Questions
1. Discuss the three dimensions of validity and give examples of how a CBPR project might strengthen or inhibit
the successful achievement of these dimensions.
2. Describe the challenges in conducting a randomized experiment using CBPR. What would be the concerns of the
community? Are there other strategies for randomization that might be more acceptable to community partners?
Describe them.
3. What are some strategies for discussing the advantages of more rigorous designs with community partners?
4. How can you as a researcher assist your community partners in understanding the evidence base related to a
question of interest?
NOTES
1. Minkler M, Wallerstein N., eds. Comunity-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass; 2003:241.
2. Clinical and Translational Science Awards Consortium. Community Engagement Key Function Committee Task Force on the Principles of
Community Engagement. Principles of Community Engagement. 2nd ed. Rockville, MD: NIH; 2011.
3. Viswanathan M, Ammerman A, Eng E, Gartlehner G, Lohr KN, Griffith D, Rhodes S, Samuel-Hodge C, Maty S, Lux, L, Webb L, Sutton
SF, Swinson T, Jackman A, Whitener L. Community-Based Participatory Research: Assessing the Evidence. Evidence Report/Technology
Assessment No. 99 (Prepared by RTI–University of North Carolina Evidence-based Practice Center). AHRQ Publication 04-E022-2. Rockville,
MD: Agency for Healthcare Research and Quality. July 2004.
4. Schutt RK. Investigating the Social World. 6th ed. Thousand Oaks, California: Pine Forge Press; 2009:38, 42, 51–53, 158, 345, 50, 411.
5. Kawachi I, Subramanian SV, Almeida-Filho N. A glossary for health inequalities. Journal of Epidemiology and Community Health. 2002
Sep;56(9):647–52.
6. Kim D, Subramanian SV, Kawachi I. Bonding versus bridging social capital and their associations with self rated health: a multilevel
analysis of 40 US communities. Journal of Epidemiology and Community Health. 2006 Feb;60(2):116–22.
7. Portes A, Fernandez-Kelly P, Haller W. Segmented assimilation on the ground: the new second generation in early adulthood. Ethnic and
Racial Studies. 2005 Nov;28(6):1000–40.
8. Hacker K, Chu J, Leung C, Marra R, Pirie A, Brahimi M, English M, Beckmann J, Acevedo-Garcia D, Marlin RP. The impact of
Immigration and Customs Enforcement on immigrant health: perceptions of immigrants in Everett, Massachusetts, USA. Social Science &
Medicine. 2011 Aug;73(4):586–94.
9. McLaughlin JA, Jordan GB. Logic models: a tool for telling your program’s performance story. Evaluation and Planning. 1999 Spring;22
(1):65–72.
10. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and
practice to improve health equity. American Journal of Public Health. 2010 Apr 1;100 (Suppl 1):S40–46.
11. De Las Nueces D, Hacker K, Digirolamo A, Hicks LS. A systematic review of community-based participatory research to enhance
clinical trials in racial and ethnic minority groups. Health Services Research. 2012 Jun;47 (3 pt 2):1363–86.
Appendix of Key Terms4
Taken from R. K. Schutt, Investigating the Social World.
Theory
A logically interrelated set of propositions about empirical reality (p. 38)
Hypothesis
A tentative statement about empirical reality, involving a relationship between two or more variables (p. 42)
Variable
A characteristic or property that can vary (take on different values or attributes) (p. 42)
Validity
The state that exists when statements or conclusions about empirical reality are correct (p. 50)
Measurement validity
Exists when a measure measures what we think it measures (p. 50)
Generalizability
Exists when a conclusion holds true for the population, group, setting, or event that we say it does, given the
conditions that we specify (p. 50)
Sample generalizability
Refers to the ability to generalize from a sample, or subset, of a larger population to that population itself (p. 51)
Cross-population generalizability (external validity)
Refers to the ability to generalize from findings about one group, population, or setting to other groups, populations,
or settings (p. 51)
Causal validity
Exists when a conclusion that A leads to our results in B is correct (p. 50)
Authenticity
When the understanding of a social process or social setting is one that reflects fairly the various perspectives of
participants in that setting (p. 50)
Sample
A subset of a population that is used to study the population as a whole (p. 149)
Program theory
A descriptive or prescriptive model of how a program operates and produces effects (p. 411)
Random sampling
A method of sampling that relies on a random, or chance, selection method so that every element of the sampling
frame has a known probability of being selected (p. 157)
Systematic bias
Overrepresentation or underrepresentation of some population characteristics in a sample due to the method used to
select the sample (p. 158)
Control group
A comparison group that receives no treatment (p. 223)
Quantitative methods
Methods such as surveys and experiments that record variation in social life in terms of quantities (p. 17)
Qualitative methods
Methods such as participant observation, intensive interviewing, and focus groups that are designed to capture social
life as participants experience it rather than in categories predetermined by the researcher (p. 17)
Evaluation research
Research that describes or identifies the impact of social policies and programs (p. 395)
Selection bias
When characteristics of the experimental and comparison groups differ or when the group under study has some
characteristics that biases their responses (in surveys or focus groups) (p. 238)
