USA: +1(669)289-8683 
Sign in
Unit 3 – Program Evaluation: Qualitative Research Design
INTRODUCTION
This unit focuses on qualitative evaluation design, data collection methods, and evaluating program effectiveness. Additionally, you will apply this knowledge to a real-world program evaluation.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
1. Understand and analyze qualitative program evaluation design.
2. Compare and contrast experimental and quasi-experimental designs.
3. Analyze pretest-posttest designs.
4. Communicate through writing that is concise, balanced, and logically organized.
U03S1] Studies – Multimedia and Readings (Complete the following):
• Framework for Program Development and Evaluation view the flow chart/transcript
• Writing an Action Research Dissertation: Part One view the media/transcript
• Writing an Action Research Dissertation: Part Two view the media/transcript
The Writing an Action Research Dissertation media pieces will help you to understand the academic writing standards for your doctoral program. You are expected to be proficient in this type of writing by the end of your program. By using the advice and guidance of the media, you can refine your academic writing and improve your success in this course and throughout your program.
• Read Chapter 5 – Program Evaluation and Performance Measurement text
o Pay attention to question 7 on page 221. The content this question addresses will be relevant for the first discussion in this unit.
• Read Moore and Tananis’s 2009 article, “Measuring Change in a Short-Term
Educational Program Using a Retrospective Pretest Design,” from American Journal of Evaluation, volume 30, issue 2, pages 189–202.
o Pay attention to the research design and data collection methods in this study. You will be analyzing them for two upcoming assignments, one in this unit and the
other in Unit 5.
[U03A1] Unit 3 Assignment 1 – Program Evaluation: Analysis of Study Design
Using what you have learned through the readings and discussions up to this point in the course, read and analyze the 2009 journal article “Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design” by Moore and Tananis. After you have finished your reading of the article, formalize your analysis by addressing the following:
• Identify the research design that was employed in the Moore and Tananis study.
• Explain whether the research design is experimental or quasi-experimental. Support your explanation by comparing and contrasting characteristics between the two types of designs.
◦ Make sure you clearly state which design (experimental or quasi-experimental) the Moore and Tananis study employs.
• Describe and analyze the data collection efforts detailed in the study.
◦ Discuss both quantitative and qualitative techniques.
• Justify any methodological recommendations for data collection methods for which you would advocate.
◦ Would you recommend using quantitative and qualitative data in program evaluation? Why or why not?
• Communicate through writing that applies current APA style and formatting and conforms to all assignment instructions.
Resources
Program Evaluation: Analysis of Study Design Scoring Guide.
Distinguished
1. Identifies a research design employed by a selected research study, and discusses the implications of using this design.
2. Compares and contrasts experimental and quasi-experimental designs, and notes which aspects of each design would be most suited for a specific study.
3. Analyzes the data collection efforts presented in a research study and uses supporting documentation from scholarly sources.
4. Justifies methodological recommendations for data collection with regard to program evaluation and notes assumptions and evidence upon which the justification is based.
5. Consistently communicates through scholarly writing that is concise, balanced, and logically organized, that flows with smooth transitions between ideas, and that demonstrates critical analysis of the literature.
6. Consistently communicates through writing that applies current APA style and formatting, is free of errors, and conforms to all assignment instructions.
https://projects.ncsu.edu/ffci/publications/2011/v16-n1-2011-spring/vaterlaus-higginbotham.php
The three qualitative evaluation methods identified by McDavid & Huse (2013)
The three qualitative evaluation methods identified by McDavid & Huse (2013) were qualitative program evaluation, inductive approach, and holistic approach.
According to the article, Qualitative Program Evaluation Methods indicated that “qualitative methods are commonly used in evaluations in order to explore specific facets of programs and give voice to participants”. The qualitative evaluation methods can provide in-depth information that can assist in enhancing the quality of a specific program (Qualitative Program Evaluation Methods).
According to McDavid & Huse (2013) defined inductive approach, as a method that “starts with ‘the data’ namely narratives, direct and indirect (unobtrusive) observations, interactions between stakeholders and the evaluator, documentary evidence, and other sources of information, and then constructs an understanding of the program” p200.
The Holistic Approach defined by McDavid & Huse (2003) is “discovering the themes in the data, weighting them, verifying them with stakeholders, and finally, preparing a document that reports the findings and conclusions” p200.
In the article, “Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design”, centered around the Pennsylvania Governor’s School International Studies having a summer program that gives gifted high school students a challenging introduction to the study of internal affairs. The program administrators wanted to understand the effect of the program on the student’s understanding of their knowledge surrounding the core issues (Moore & Tananis 2009). For the past four years the program administrators began using the retrospective pre-post design. The results from the pretest yielded that the students consistently overestimated their understanding of International Studies in the beginning. The program administrator found that using the “retrospective pretest items were more biases than pretest items in certain context” (Moore & Tannis 2009 p200). The authors concluded that the retrospective pretest perhaps could be the most effective way of capturing data that is accurate of pre intervention function than a pretest given before the program begins.
For this article, this learner would use the inductive approach for this article. This approach will allow the researcher and the program administrators to evaluate the program effectiveness through the data that was collected through the pretest, test that were administered throughout the course, and other data. This approach will allow for interviewing the participants throughout the five week program to determine if the students are learning and meeting the intended goal of the International Studies program. The program administrators can also observe the interactions between the participants.
For this particular study, it would be important to test the students prior to taking the course, and then conduct weekly testing, and a final to determine the comprehension of the material. There would be periodic interviews and/or questionnaires to establish how the students feel, how the material is being presented, and feelings towards varying factors regarding the material. Since this is a five week course, conducting initial interviews, mid-course interviews, and an exit interview will allow the program administrators to get real time information in conjunction with the weekly testing. All of the information would be gather to compare testing to the interviews and questionnaires to have a more solid information regarding the program effectiveness. With this data collected over a period of years, the program administrators would have a more solid understanding if the program is beneficial.
Reference:
McDavid, J. C., Huse, I., & Hawthorn, L. R. (2013). Program evaluation and performance measurement: An introduction to practice (2nded.). Thousand Oaks, CA: Sage Publications. ISBN: 9781412978316.
Moore, D., & Tananis, C. A. (2009). Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design. American Journal of Evaluation,30(2), 189-202. doi:10.1177/1098214009334506
Reference:
McDavid, J. C., Huse, I., & Hawthorn, L. R. (2013). Program evaluation and performance measurement: An introduction to practice (2nded.). Thousand Oaks, CA: Sage Publications. ISBN: 9781412978316.
Retrieved on January 8, 2018,
https://projects.ncsu.edu/ffci/publications/2011/v16-n1-2011-spring/vaterlaus-higginbotham.php
Article Qualitative Program Evaluation Methods.
I
2. Hacker K, Collins J, Gross-Young L, Almeida S, Burke N. Coping with youth suicide and overdose: one community’s efforts to
investigate, intervene, and prevent suicide contagion. Crisis. 2008;29(2):86–95.
3. U.S. Census Bureau. 2006–2008 American Community Survey 3-Year Estimates. 2008 [cited 2010 October 3]; Available from
http://factfinder2.census.gov.
4. Chomitz V, Arsenault L, Banks C, et al. H.E.L.P. Culminating Report. Cambridge, MA: Institute for Community Health; 2011.
5. Clinical and Translational Science Awards Consortium. Community Engagement Key Function Committee Task Force on the Principles of
Community Engagement. Principles of Community Engagement. 2nd ed. Rockville, MD: NIH; 2011.
6. Israel BA, Parker EA, Rowe Z, et al. Community-based participatory research: lessons learned from the Centers for Children’s
Environmental Health and Disease Prevention Research. Environmental Health Perspectives. 2005 Oct; 113(10):1463–71.
7. Minkler M, Wallerstein N., eds. Community-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass; 2003.
5
Translating Research Into Practice
View From Community
“Sustainability is building capacity and helping the community use their own data to help them continue work after the intervention has
finished.” (Community partner-conference participant)1
n this chapter, we will explore key questions from the community partner perspective about engaging in CBPR and
the value of this research to the community. This will include an overview of how a community partner should assess
its capacity for participation in CBPR. We will also explore the challenges inherent to implementing change based on
findings from CBPR. This will also include data-driven decision making and issues relevant to incorporation of
evidence into practice. My community partners were instrumental in supporting the writing of this chapter. Ms. Milagro
Grullon reviewed the chapter, and both she and Mr. Alex Pirie provided quotes. In addition, quotes from community
partners at a recent CBPR conference on building community capacity were also used.1 The coalitions that I worked
with in the given examples approved the use of their stories.
The following fictional example helps to illustrate a typical community response to a researcher request.
I am the executive director of a small organization dedicated to serving the Latino community in Community Y. We are currently fairly
small, with only two employees. All of us are immigrants and speak fluent Spanish. The mission of our organization is to provide help
with access to health care, city services, and other programs for the Latino immigrant community. A researcher from the local university
has contacted us and wants to get our help recruiting Latinos for a study on mental health issues. In the past, we had a bad experience with
a researcher. He asked for our help recruiting our constituents for a study on high blood pressure. We spent more than a month helping
with the recruitment, and while we did get paid for our efforts, our small staff could not do this work and keep up with the work of the
organization. Things were very tense. Then, when he completed the research, we never heard back from him about the results until we read
them in the newspaper. “Latino males who are overweight are at high risk for hypertension.” We never had a chance to participate in the
paper or get the results so we could work with the community to improve the situation. It was a surprise to us and to the people who had
agreed to participate. Overall, it felt like we got used, and we are not sure we want to do this again.
In the world of community research, this is not an uncommon situation. What went wrong? There was money shared,
and engagement seems to have taken place, but things really seem to have fallen down in the analysis/interpretation and
dissemination phase of the research. In this case, it is hard to tell whether the researcher was interested in a full CBPR
relationship, but the impact of this research has ultimately increased the divide between the community and researchers,
and, most importantly, while it may have contributed to the literature, it has provided little benefit to the local
community.
Many community partners may have negative impressions of research influenced by prior historical events such as
the Tuskegee experiment, especially in communities of color. They may have had negative experiences with past
research projects themselves and feel distrustful of new academic relationships. Additionally, they may be unfamiliar
with how research works or what their role in this work might entail.
“Communities have no motivation because these research projects go on yet there is no investment in the community after. Many
community members are not interested in being in a registry for research.” (Community partner-conference participant)1
“There is this mistrust of what the aims are for research and what this information will be used for.” (Community partner-conference
participant)1
Recognizing that these perspectives may linger long past any one individual research project is an important part of
the engagement process. A researcher needs to be aware of the history of research in the community. Community
partners need to be empowered to ask the hard questions of researchers so as not to repeat the same mistakes. They need
to understand the benefits and pitfalls of engaging in research. Gaining this insight is part of the engagement process
and is necessary for the development of a strong CBPR partnership. As a community begins to understand research and
its potential power, community members may initiate CBPR projects themselves, recognizing that they need data to
catalyze social change in policy and practice.
In our previous Everett example, Everett, one member of an immigrant advocacy group acknowledged the potential
benefit of engaging in research. As a member of a community advocacy group dedicated to working with Latinos in
Everett, he was very concerned about how recent immigration enforcement activity was affecting the Latino
community. Immigrant activists had many stories of how immigrants were being stopped and asked for their driver’s
licenses, and then, if they did not have a license, they were arrested. Many immigrants were in fear of their safety,
suspecting that this action would result in deportation. The immigrant advocacy community wanted to get more
information on this situation to convince the police to stop arresting people for not having a valid driver’s license. They
had made a connection with a local researcher that they already knew and trusted when a grant opportunity arose. They
asked her to help them get the data needed to actively change the situation. In this example, the Everett community
partners recognized that data collected through “research” might help them advocate for action. This
academic/community partnership was built on an existing trusting relationship.2 While not all CBPR partnerships will
be initiated by the community and all questions will not be solely originated in the community, using a CBPR approach
resets the researcher’s agenda such that the topic of study resonates with community priorities and is seen as beneficial
to the community. Community members should expect that their priorities will be considered and that their perspectives
will be valued in the research agenda and subsequent process.
ASSESSING COMMUNITY RESEARCH READINESS
“How do you justify this money being spent on research? Communities have real issues that money could help solve and yet this money is
being given to research—how do you prove the benefit and then share the money so that it benefits the community (immediately)?”
(Community member-conference participant)1
As noted previously (Chapter 2), it is important to assess the impact that engaging in research will have on the
community. The areas outlined below are important for community members to address prior to engaging in a CBPR
project.
ASSESSING THE PARTNERSHIP
The first area that community partners will need to assess is the nature of the CBPR partnership itself, the values, goals,
and priorities.3 What type of partnership is planned? How much power will the community members have in decision
making? Is the researcher intimidating? Is he or she skilled in participatory techniques? Does she or he speak in
layman’s terms and understand the community perspective? Is there a history of trust and respect?4 Is the researcher’s
agenda shared by the community?
“Today, decisions are made by prioritizing and based upon the needs in the community. From a program perspective we need research to
help implement the correct programs in our community. Prioritizing according to need can yield rich data to support programs and
interventions. Again, what makes a project appealing is the need and if the community has identified what is being presented as a priority
for them.” Milagro Grullon, Lawrence, MA
In order to learn more about the researchers, their goals, background, and expectations of a partnership, a face-to-face
meeting should be held, and community members should have a chance to ask these questions and determine if the
CBPR project is right for them.
Questions
outlined in Table 5.1 are a good place to start. In addition, the city of
Lawrence, Massachusetts, developed a “tools for research partnerships” document that can help community partners
build their formal agreements with researchers.5 As one partner noted related to the choice of an academic partner:
“Is this a person you can spend the next year or four years working with closely? Is she or he someone you can comfortably disagree with?
Are they sympathetic to the complexities and nuances of communities and willing to step outside of their traditional bubbled and siloed
turf to connect with the community?” Alex Pirie, Somerville, MA
Table 5.1 Questions for Community Partners
Questions for community partners to ask researcher prior to engaging in CBPR
1. What kind of partnership does the researcher have in mind? Is it really to be participatory?
2. How will decisions get made?
3. What are the research aims?
4. Who is the target population of interest?
5. How will the research be funded?
6. What will be our organization’s and/or my role in the project?
7. Will the time be compensated?
8. Who will own the data? What will happen to the data in the future after the project is completed?
9. What benefits will the project leave behind in the community (skills, programming, policy, infrastructure,
capacity building)?
10. What is the dissemination plan for this research?
11. Does the researcher have the skills and experience to collaborate?
12. Does he or she understand the complexity of the CBPR approach?
13. Does the researcher possess cultural humility?
Questions for community partners to ask themselves prior to engaging in CBPR
1. Does this study address an important problem relevant to my community and my constituents?
2. How does the research aim fit with the mission of my organization?
3. Do we have the capacity to participate? Space? Staff? Time?
4. What are our conflicting priorities?
5. What will be the impact of doing research on my organization’s ability to get its core work accomplished?
6. Will the results lead to action that will help my community?
TIME, RESOURCES, AND CAPACITY
As part of the development of a partnership pre–CBPR, community members should ask specific questions in three
critical areas to assess their own readiness for research: time, money, and capacity. That is, do they have the time to
participate without sacrificing their other responsibilities, are they going to get appropriate financial resources from the
researcher to support their work, and do they have the organizational capacity to participate in the research project?
Each of these issues can create problems if resources are inadequate for the project. For example, if the organization
and the staff do not have time available to participate in the CAB meetings or to actually conduct their roles in the
research, they will be unable to fulfill their responsibilities and the research will fail. As one community member said:
“The single largest obstacle to collaborations between academic and community partners is time. The relative time frames of academic and
community partners can create huge problems and it is essential that whichever of the two sides of the partnership is more experienced do
everything it can to educate the other on the discontinuity that exists.” Alex Pirie, Somerville, MA
In many organizations, particularly small ones, it is not feasible to just hire additional people to take on the new work
of a research project. So who will do the work? What are the implications for the organization? In addition, there is time
required to participate in CBPR partnership meetings and time required for the development of proposals and
dissemination products. Finally, sometimes, it can take several years before a research grant is funded. Any community
partner must consider where it will be in the future and whether it is able to plan that far in advance.
Where the money is concerned, budgets should reflect an equitable process between community and academic
partners. Community partners have not always been included in these decisions.
“When I first started working with academic institutions (12 years ago) I had no involvement with budgets. I was given an amount for
participating as a [representative of the] community without being asked how many staff would be involved or the hours we would spend
on the project. We realized shortly after the grant started that the demands on the staff and community were greater than the funds could
sustain.” Milagro Grullon, Lawrence, MA
As one prepares a grant proposal, both community and academic participants should assess whether there is adequate
support for a CBPR project. Financial support is needed for items ranging from community coinvestigators, liaisons,
space, meetings, and advisors to travel and dissemination. It should reflect the division of labor between academics and
the community.
“Depending on the involvement and time spent in the research, resources should be divided accordingly. The community is asked to
participate and do a lot of work (meetings, gather data, interviews, invite community members, go on the radio, etc. etc.). Without the
community, some of the research could not take place. Researchers need the community, and the community needs to get paid justly.”
Milagro Grullon, Lawrence, MA
Table 5.2 shows an example of a CBPR budget for a project that included focus groups and surveys. The dollars for
various items (stipends, food, etc.) can either sit with the academic or the community partner, depending in large part on
their ability to distribute the money. In some cases, community organizations may see a CBPR project as a way of
accessing dollars for needed infrastructure and programming. However, the research dollars are not flexible and cannot
be spent on nonresearch-related items. Regardless of how the ultimate budget looks, it should be negotiated up front,
and any further reductions that might be required if the grant is funded should also be discussed with the partnership
prior to starting the project.
Ms. Grullon recalled:
“The second time around, when an institution came to me for help or to develop a (pseudo) partnership, I started making clear demands. I
needed to see the budget and be included just as any researcher would be listed on the budget. I worked on a budget for the organization
and the community members that would be involved.” Milagro Grullon, Lawrence, MA
Table 5.2 Proposed CBPR Project Budget
Source: Reproduced by permission from Institute for Community Health, Inc. Cambridge, MA 2011 www.icommunityhealth.org
While differences in salary ranges can become a source of conflict, a bigger issue for many community partners is the
high rates of indirect costs that most academic institutions claim. This can be a source of great tension in CBPR.
“Indirect” rates are related to costs that include physical plant, research administration, and so forth, and they are
negotiated with the funding institution. As one community member said:
“When people congratulate us and say, ‘wow, you got a million bucks to study X, Y, or Z!,’ I say, well, no, we got about 45% of that, the
rest goes to mow lawns and trim ivy at the University of Wherever.” Alex Pirie, Somerville, MA
In general, these dollars do not flow directly to the principal investigator but, rather, are used by the institution to
support operations. While some community agencies have successfully negotiated an indirect rate with funders, this is
not always the case. Thus, how they support their operational needs (rent, secretarial support, utilities, and payroll)
comes into question. A community organization needs to discuss strategies for covering these costs with the researcher
either as an indirect rate, if allowed, or by putting these items into direct costs. Without these transparent negotiations,
the budget can become a major source of conflict.
The capacity of the community partner may include things like current staffing, financial health (cash flow, total
budget, grant management skills), and physical space. For example, can the organization accommodate the research
given its personnel issues (turnover rates, skill sets, etc.)? Are there procedures in place to deal with contracts? Is there
enough cash flow to conduct the work and then invoice for work done? How will the research impact the day-to-day
operations of the organization? Any community partner or member of a community organization should consider these
issues prior to agreeing to participate in a CBPR project.
It is also important to remember that research is relatively inflexible compared to program implementation. This
“rigidness” is needed to answer the research questions effectively. If the intervention keeps changing or the staff leaves
and new staff must be retrained, it poses a problem for the research. Data need to be collected on every participant in a
consistent manner. Records about why participants did not participate as well as how many did or did not participate
must be maintained. If a survey is done but no one knows what the total number of potential participants in the target
group was, it will be impossible to calculate the response rate. While these may be seemingly simple steps, in practice
they can be a major challenge. In the Everett project, surveys of community doctors were done, but the process of
figuring out exactly how many community doctors were serving Everett populations turned out to be much more
complicated than expected. Community partners wanted to send the survey out to as many providers as possible to
generate a high number of responses, but researchers first needed to figure out how many providers were serving the
Everett community. In other words, they needed to know the denominator so that they could understand whether the
final sample was representative of the total group of providers. A similar situation happened when community partners
wanted to survey immigrants. They held several forums and passed out surveys but were unable to keep track of who
refused to fill out the survey or even how many potential participants were at the forum. Researchers found themselves
counting attendees and trying to keep track of who got surveys, who filled them out, and who threw them in the waste-
basket. In the end, they were unable to determine what percentage of attendees actually did fill out the surveys.
Sampling concepts may be foreign to community groups, who may have little experience with obtaining responses in
a systematic way. The tension between action and rigor may create conflict with the organization’s culture and capacity.
After all, communities do not exist in a laboratory bubble. Unexpected events transpire that may have a deep impact on
a research project, ranging from turnover in local leadership to new and emerging social issues. Service demand may
interfere with scientific methodology. While not all research will require the same level of rigor, adherence to a protocol
really matters. For community partners, this may present a very different approach to conducting business. The research
will need to get done on schedule to achieve the aims set out in the proposal. CBPR is often a balancing act between the
researcher’s need for control to achieve scientifically valid results and the community’s desire for action. Successful
CBPR partnerships are able to navigate these challenges via mindful communication and planning.
DATA COLLECTION, OWNERSHIP, AND PROTECTION
“Data is not a four-letter word. Advancing the field and advancing the science [will only happen] because we have done good research and
then it will help.” (Community partner-conference participant)1
Data and their collection, security, use, and ownership is an area that requires some forethought in CBPR. Given that
CBPR represents a partnership for both community and academics, the data are the property of both. How they are
handled, analyzed, and disseminated is therefore important for both. As noted in previous chapters, some of these issues
can be sorted out and recorded in a memorandum of agreement. In our Somerville CBPR youth suicide example, the
researchers were helpful in translating real-time data into usable information for community partners. They provided the
analytic capacity to examine the youth risk behavior data and identified additional data sources that were useful in
monitoring suicide and overdose activity. These included death certificate data, 911 call data from the fire department,
and hospital data on use of the emergency room related to suicide intention and overdose. They were able to provide
these enhanced data to the community for real-time decision making. In addition, they were able to map data from the
911 call information onto maps of the community. This helped provide a contextual framework that allowed community
members to focus their attention on specific areas. The data were generally “deidentified,” meaning that information
that would identify a person was removed before reporting to the CAB. No one would be able to identify the
individuals, thus maintaining their privacy. The concept of data privacy is an important one in CBPR. Working with
community members who may know each other could put privacy at risk. In CBPR, it is particularly important to work
through these privacy issues prior to sharing data. In Somerville, the CAB agreed to a level of privacy that would not
allow reporting when numbers were small (< 5).6
The Somerville project is an example of using data for decision making in real time. While community members
bring a wealth of tacit knowledge to the table, they are not always equipped to evaluate their efforts. Thus, the ability to
use and analyze data in a more rigorous fashion may help to promote improvement. In CBPR, data can serve both as
means to a research project end and as quality-improvement information for community partners. As noted, the
translation of evidence to everyday practical solutions is a critical component of the CBPR process. The CBPR
researcher is, in effect, a purveyor of data, design, and methods that will facilitate this translation. Key to this translation
are the skills to communicate and the desire to utilize. The CBPR research process resembles the Plan Do Study Act
cycle in quality improvement.
To a community partner, the collection and management of these data thus have implications for the present project
but also, potentially, for future programmatic and policy decisions. Community partners should be very thoughtful
about how the data will be housed and about how decisions about present and future analysis will be handled. The
misuse of data can seriously undermine any future research efforts and potentially fracture a CBPR partnership. Setting
out strict standards about data use at the beginning of the project can be beneficial. Table 5.3 is an example of an
explicit policy regarding data confidentiality. This will be discussed at length in the next chapter.
Table 5.3 Data Ownership and Confidentiality7
There should be an explicit agreement between researchers and community and university partners with respect to
ownership of the research data. The nature of the agreement will be determined on a project-by-project basis, as the
community needs/requirements may differ (i.e., a school, a department of health, and a coalition may require different
agreements). This agreement should be made prior to the onset of the project and revisited during the project with the
project-specific working group.
Research projects will adhere to the human subjects review process standards and procedures set forth by the
appropriate institutional review board. In addition, research partners will adhere to standards of school departments and
community-based organizations when appropriate.
All investigators and research associates need to be certified for human subjects work.
All persons who work with data (including visiting scholars and graduate students) will be required to sign a pledge
of confidentiality for data use.
In general, we will not publicly report data with a cell size less than 5 subjects. Suppression of cell sizes < 5 in
reporting will be considered the general rule to protect the confidentiality of subjects. However, any agreement
requiring larger numbers for suppression will supersede this rule.
Source: Institute for Community Health, Cambridge, MA 2012.
DISSEMINATION OF FINDINGS
“The time delay between the conclusion of a research project and the publication of papers is always a problem. This is a period of time
when, because of the constraints of journal publication, there is a virtual embargo on the results except in the most general way, and it
drives the community side nuts. ‘Hey, we know this, we want to do something with/about it!’” Alex Pirie, Somerville, MA
Identifying the results of CBPR research is the first step in translating evidence into actionable efforts. As is part of
the CBPR process, the community is generally interested in using the results to make change. Change can be actualized
in many forms: new programs to address needs, changes in existing programs, policy change at the local and state
levels, and advocacy efforts. Findings make their way into these venues in multiple fashions. Program directors that
have access to the information can incorporate it into their day-to-day work. Advocates can promote the results with
their city councilors, aldermen, and legislatures to move an agenda forward; constituents can use the evidence to
improve their own situations. This is one of the most important value-added benefits of participating in CBPR efforts.
“Don’t save dissemination [or evidence] translation until the end. Systemic changes are key. Wait for results of qualitative + quantitative
research to see what worked and let the community know. Policy + systemic changes has built sustainability. Make it a priority to have
dialogue with community [and] key stakeholders.” (Community member-conference participant)1
In all research, the findings are not always what one might expect. Sometimes, results do not reveal the answers that
one is seeking, or they reveal unexpected information that may be particularly controversial. How will the partnership
deal with conflictual and potentially controversial results? Will community members want to censure the results? Will
they want to reframe the discussion? Community partners need to be prepared for these possibilities and discuss how
they will deal with results in advance of their delivery. In CBPR, the community partners are part of the analysis and
interpretation phase of the research, and this will help in the management of the dissemination process because of their
involvement in interpretation. However, from a community perspective, if you go into a project to prove something or
to demonstrate how well your program works, you may be disappointed. Community partners need to be realistic about
what the research can provide and discuss various possibilities for how they will deal with positive, negative, or neutral
results with their academic partners well in advance of fielding the research.
The definition of dissemination may be very different for academics and community partners. Community partners
want to get the information into the hands of policy makers or service providers in the form of reports, press coverage,
or community meetings. In our Everett example, the community partners requested that data be presented to the
community in a public forum at the end of the project so that community members could make recommendations for
action. They were adamant that this needed to happen quickly so as not to lose the momentum of the project. In
addition, given the focus of the work, the timing regarding the larger context of immigration enforcement was
opportune. The researchers would have to wait to work on publications for peer-reviewed journals until after this was
done. Together, they decided to present the data in a digestible format to an audience convened by the community
partners, who included local government as well as immigrants themselves. They also requested that community
members have an opportunity to consider recommendations for action based on the study. This engagement was built
into the forum agenda. Participants broke up into different language groups and worked together to answer key
questions and provide recommendations. The final product was a set of policy and practice recommendations that was
provided to leaders in the community. Many were then utilized to change police-immigrant relationships and address
health care access issues.2 Afterward, the researcher and community partners developed several papers for peer-
reviewed journals. The emphasis for dissemination was different for community and academics. Several other
dissemination options were also used that particularly spoke to advocacy, including a press release to local papers, a
letter to the Boston Globe, and several submissions to pertinent blogs. The dissemination plan for the project has since
taken multiple forms.
• Community forum with presentation
• Meetings with advocacy groups
• Meetings with police
• Presentations to local community advisory group
• Abstracts at national meetings
• Peer-reviewed publication(s)
• Press release(s)
• Letters to newspapers
• Blog submissions
TRANSLATION OF EVIDENCE INTO PRACTICE
“We always have to examine whether or not we are doing what we hope to be doing. But this is difficult because we get invested in what
we are doing and taking a critical examination is not always easy.” (Community member-conference participant)1
Dissemination of study results jump-starts the process of translation/adoption of evidence into practice. As people
hear about the ideas, they are likely to consider how to implement them. In addition, if the findings are strong, they may
influence how programs are run and, more importantly, whether programs continue to exist. Some of the decisions
about translation and sustainability may depend on the project topic. For example, if community members wanted to
test an intervention, there may already be tacit agreement to adopt the findings, when available, or to change or adapt
the program to improve it. The CBPR process is also making all kinds of new evidence available, including literature
reviews, information on evidence-based practices, and existing data for use in this CBPR process. There are often
unexpected outcomes of CBPR that may not have been the primary focus but are results of multidisciplinary approaches
that start with academic/community partnerships. Certainly, if adoption is contingent on money, translation and
sustainability might be harder to achieve. But the results of one study provide the foundation for the next. In Everett, the
community took the results and translated them directly into actionable steps. They launched police/immigrant
meetings. But they also used the data to obtain additional grants on racism and food insecurity and adolescent risk
behaviors. The data continue to have a life after the project. In Somerville, the community continued its vigilance by
monitoring 911 activity after the suicide/overdose crisis. They hired a mental health coordinator who is still in place. In
Cambridge, the work is ongoing, and the partners have written several other grants. While the project provides new data
for communities, it also shapes the researchers and their agendas. They benefit greatly from the colearning of the
partnership.
Overall, community partners should expect any CBPR researcher to be aware of the impact of a CBPR project on a
community at all different phases. This includes the impact on community partner capacity, mission, and time.
Researchers should recognize from the beginning that shared goals for a project need to be discussed and mutually
decided. Dissemination strategies take many forms, and communities perceive action—local action—as an end product.
A CBPR researcher should understand what the incentives for community participation are and simultaneously
acknowledge her or his own. Thus, as a final portion of this chapter, we offer a list of questions that a researcher should
consider prior to CBPR engagement.
Researcher Questions to Answer Before Getting Involved in CBPR
• Is my question emanating from community priorities?
• Will the results or findings of this study benefit the community?
• What am I really asking from the community partners? What level of responsibility will the community bear?
• How do I ensure that my partners understand the research and its limitations, the benefits and risks?
• Can I frame multiple dissemination products to serve both my goals and those of my community partners?
• Am I in it for the long haul? What is my commitment to the community?
A researcher who puts him- or herself in the community’s shoes and understands the community’s perspective will
go a long way toward supporting the CBPR process and improving the level of trust in community/academic
relationships.
“The strategy is to form it [the research] around a very clear, specific issue that everyone feels invested in. Then you can start building the
trust and relationship.” (Community member-conference participant)1
CONCLUSION
Community partners who wish to engage in CBPR should be aware of the benefits as well as the challenges of this
approach. They should be thoughtful about their own research readiness and aware of the realities—the responsibilities
and the potential outcomes—of engaging in a research project. They should seek a balanced partnership in which their
voice is heard and their viewpoints considered. Most importantly, they should consider how the outcomes of CBPR will
help their community, however that is defined. Knowledge is powerful. CBPR is an approach to gaining knowledge that
can empower communities. Thus, a well-designed CBPR study is more likely to encourage strong, trusting relationships
that cross academic/community boundaries and improve the translation of both evidence-based practice and practice-
based evidence into practical solutions for real-life problems.
QUESTIONS AND ACTIVITIES
Activities
Have students break up into small groups to discuss the following:
A community group had garnered a grant to develop and implement a program for African American men to improve
cardiovascular risk factors. The program was well liked by participants and by local politicians. But they had no idea if
the program made a difference in participants’ health. Initial data were promising with regard to satisfaction and
engagement of the participants, but there was no evidence that the outcomes (BP, cholesterol) were changing. There
was a great deal of investment in the program, and a publication that did not say “the program was working” would put
the program at risk financially. They engaged a local CBPR researcher to work with them and try to evaluate the impact
of the project.
What are the potential pitfalls in taking on this project?
Describe the community perspective and their incentives underlying this work.
How would you disseminate results?
What types of products might bring added value to the community?
Have students interview other CBPR researchers to ask them about their projects and how they approached the steps
or a particular step outlined in this chapter.
If possible, have students sit in on a meeting between CBPR researchers and their community partners and observe
the process. Then have them identify where they are in the research process and whether they adhered to CBPR
principles during the meeting. What went well and what was challenging?
Questions
1. What questions should a researcher ask at each step of the research process in CBPR?
2. What are the skills that a CBPR researcher will need to enter into CBPR?
3. You have gotten a grant to conduct a CBPR project. However, it ended up taking more time than you expected to
obtain the grant. During that time, the leader you were working with moved away. How would you re-engage the
community?
NOTES
1. Break-out sessions. Taking It to the Curbside: Engaging Communities to Create Sustainable Change for Health. Conference, Boston, MA;
April 2010.
2. Hacker K, Chu J, Leung C, Marra R, Pirie A, Brahimi M, English M, Beckmann J, Acevedo-Garcia D, Marlin RP. The impact of
Immigration and Customs Enforcement on immigrant health: perceptions of immigrants in Everett, Massachusetts, USA. Social Science &
Medicine. 2011 Aug;73(4):586–94.
3. Norris K, Brusuelas R, Jones L, Miranda J, Duru O, Mangione C. Partnering with community-based organizations: an academic
institution’s evolving perspective. Ethnicity & Disease. 2007;17(1):205.
4. Baker EA, Homan S, Schonhoff R, Kreuter M. Principles of practice for academic/practice/community research partnerships. American
Journal of Preventive Medicine. 1999 Apr;16(3 Suppl):86–93.
5. Lawrence Mayor’s Task Force. Tools for Research Partnerships in Lawrence, MA. Lawrence Mayor’s Task Force, Lawrence, MA 2006;
Available from: http://www.tuftsctsi.org/About-Us/CTSI-Components/Community-
Engagement/~/media/B35A1D1535DB422D90E1A47544743E4E.ashx.
6. Hacker K, Collins J, Gross-Young L, Almeida S, Burke N. Coping with youth suicide and overdose: one community’s efforts to
investigate, intervene, and prevent suicide contagion. Crisis. 2008;29(2):86–95.
7. Institute for Community Health. Policies. Data confidentiality/IRB policy. Cambridge, MA: 2002.
http://www.icommunityhealth.org/policies.html.
RESOURCES
Community Campus Parternships for Health.http://www.ccph.info/.
WRITING AN ACTION RESEARCH DISSERTATION: PART
ONE
INTRODUCTION
Writing an action research dissertation may seem a formidable task. However, action research offers exciting
prospects for expanding our understanding of the world around us and rich opportunities to be of service as
academics and as professionals.
Any well-written dissertation will have much in common with other well-written pieces of academic writing
including journal articles and research reports.
In Part One of this presentation, we’ll talk about some factors related to success as an academic writer, and
then in Part Two we’ll apply what we’ve discussed to the task of writing an action research dissertation.
At the end of the first presentation, you’ll be invited to explore links to academic writing tools. At the end of the
second presentation, you’ll be invited to explore links to academic journals that publish action research studies
and university centers that conduct and share the results of action research. When you’ve explored those links,
please share your ideas with your colleagues by responding to the discussion question in your courseroom.
ACADEMIC WRITING
First, let’s talk about academic writing generally.
Different kinds of writing are done differently and, when they’re completed, have different characteristics.For
example, writers of fiction or marketing materials are sometimes advised to, “Write the way we talk.” However,
that isn’t good advice for academic writing.
LEVEL OF REDUNDANCY
For one thing, spoken language contains a lot of redundancy, since the listener has to remember everything
that has been said. That isn’t the case with writing, of course, and academic writing has very little redundancy.
As a result, it can seem to be particularly dense.
ADJECTIVES AND ADVERBS
Another example is that we tend to use a lot of adjectives and adverbs when we talk, but strong academic
writing usually does not use a lot of them. A reason is that adjectives and adverbs tend to be imprecise. Words
such as “big,” “old,” “important” and so forth are vague. You will be strongest when you are most precise.
THIS DISSERTATION
A third example is that academic writers structure their sentences differently from the way they structure their
sentences when they speak. Research shows when we speak we tend to build our sentences out of a lot of
pairs of subjects and verbs — that is, our sentences tend to have a lot of clauses. For example, someone
might say “I told him that Jane thought that he needed a car.”
SPEECH
That sentence contains three clauses. The first clause has “I” as the subject and “told” as the verb, the second
clause has “Jane” as the subject and “thought” as the verb, and the third clause has “he” as the subject and
“needed” as the verb.
ACADEMIC WRITING
Experienced academic writers rarely write sentences that string together a sequence of subjects and verbs in
that way. Instead, they tend to create sentences that have one subject and one verb, with a lot of phrases
related to the subject and verb. “Poverty in the home, regardless of the level of education of the parents,
causes poor performance in middle school and increased dropout rates over time.” The first sentence, “I told
him that Jane thought that he needed a car,” is only 11 words long but has three pairs of subjects and verbs.
The second sentence, “Poverty in the home, regardless of the level of education of the parents, causes poor
performance in middle school and increased dropout rates over time,” is much longer, with 25 words, but has
only one subject and verb.
You don’t have to be an expert in grammar to succeed as an academic writer. In fact, probably very few, if any,
academic writers would be aware that they were making this shift in sentence structure. However, this example
illustrates that in learning to be an academic writer, you are learning to write in a new register that differs from
the ways in which you may be used to using language. One way to shepherd yourself into the kinds of
sentences and overall style used in your specific academic area is to read high-quality pieces of academic
writing in your area and pay attention to the ways in which language is used. Over time, you’ll become used to
these specific uses of language.
STRENGTH AND WEAKNESS
All of us have strengths and weaknesses, and in academic writing, as in other areas of life, our strengths and
weaknesses will end up to be systematic. This is a reason that if a teacher goes through your paper and does
line-by-line editing, marking all of your mistakes, research does not show that this will necessarily lead to
learning. What most often happens in those cases is that learners don’t take time to go through all of those
edits. Instead, being normal people, we tend to just go on to the next assignment.
On the other hand, if a teacher identifies categories of your strengths and weaknesses, that empowers you
going forward because then you will be able to work to improve in your weak areas, and protect your areas of
strength. Both of these are important, because as you make changes to improve your weaknesses, you may
accidentally lose some of your strengths without knowing it. Knowing what your strengths are will help you to
protect them. The links provided to you at the end of this presentation will give you the tools and skills you
need to accomplish this.
WRITING FEEDBACK
As a start, you’ll be introduced to Capella’s Personal Writing Assessment Tool and Writing Feedback Tool.
MEAL PLAN
You’ll also learn about two Writing Program resources that are particularly useful in dissertation writing:
Reverse Outlining and the MEAL PLAN.
Let me say a word about Reverse Outlining. This simply means that you write a draft and then outline what
you’ve written. This isn’t as easy as it sounds, because you have to outline what you actually wrote, not what
you wished you had written. As soon as you begin to reverse outline something you’ve written, you will trick
your mind into seeing your writing differently, and your mind will say, “Oh, I thought I had three major divisions
but I only have two,” or, “I have three divisions but I have 27 pages in the first one and only 2 pages in the
second one.”
This fact makes Reverse Outlining the best tool to use if your dissertation committee asks you to expand your
review of literature, which sometimes happens. If you just look over your review of literature and say, “Oh, ok, I
can write more about these two areas,” you’ll probably pick the two areas you already wrote too much about
because you’ll pick the areas you liked best or found easiest to research. If you use the Reverse Outlining
process, you’ll create a more balanced review of literature and save yourself a lot of revision time and
frustration.
CONCLUSION
Actually, this provides a good transition into the second part of this presentation, which is applying best
practices of academic writing to the specific challenges of writing an action research dissertation.
Here are the links to key academic writing resources for you to reflect on as you watch the second media
presentation:
Overview of the Writing Center:
http://media.capella.edu/CourseMedia/XX7010/WritingCenterOverview/wrapper.asp
Academic Integrity:
http://media.capella.edu/CourseMedia/DrPH8005/AcademicIntegrity/animation_wrapper.asp
Smarthinking:
http://media.capella.edu/CourseMedia/DrPH8005/Smarthinking/smarthinking_wrapper.asp
Peer Review:
http://media.capella.edu/CourseMedia/DrPH8005/PeerReview/animation_wrapper.asp
Licensed under a Creative Commons Attribution 3.0 License.
WRITING AN ACTION RESEARCH DISSERTATION: PART
TWO
INTRODUCTION
Welcome to the second part of our two-part media presentation. In the first part, we talked about key tools for
successful academic writing and gave an example of how these tools can make your dissertation writing
process faster and help you create a stronger dissertation document. This second part of our two-part media
presentation applies those key tools to the specific challenges of writing an action research dissertation.
PROCESS OF WRITING
The process of writing the research dissertation involves finding ways to help your readers understand both the
philosophy of research underlying the study and the methods employed in the study.
ACTION RESEARCH ROLES
It will be important to help your readers to understand the roles of the Principal Investigator and the participants
in your Action Research study. Rather than the “all-knowing” researcher who controls conditions and conducts
an experiment on or to someone or something else, Action Research assumes that the participants themselves
have insight and an important perspective and role.
ACTION RESEARCH METHODS
As a writer, you will want to help your reader understand that this idea of democratizing the research process
applies regardless of the research methodology used. Action Research involves methods that would be
familiar in both qualitative and quantitative types of research, and it would be very common to find mixed
methods in a particular study. Regardless of the methodology used, the participants can and will play an active
role at each step of the research process.
Help your readers to understand that the participants are active in the research process of planning, deciding,
and helping to shape the implementation of the study protocol and analysis of the findings. Also, help your
readers to understand that findings are interpreted or understood in part based on “meaningfulness” to the
participants engaged in the work.
An example might be a study on abusive behavior in hospitals among nursing and other staff. Members of the
study team could help refine the survey instruments, later take the survey themselves as part of the participant
pool, and help to interpret the findings.
IMPLICATIONS FOR WRITING…
In the next few minutes, we will discuss four elements of writing the action research dissertation. You will want
your dissertation to clearly explain the following:
1. How the research questions that you asked led you to decide to use action research.
2. How the research questions that you asked informed they way in which you organized and presented
your discussion of prior research.
3. How you balanced the two goals of being of service and avoiding risk to participants within the action
research paradigm.
4. The reporting of outcomes and analysis of outcomes.
ACTION RESEARCH DISSERTATIONS
Action Research dissertations are, by their nature, research studies in which the roles of the Principal
Investigator and the Participants are key. Rather than the “all-knowing” researcher who controls conditions and
conducts an experiment on or to someone or something else, this assumes that the participants themselves
have insight and an important perspective and role. Action Research studies privilege the process of reflection.
They seek to be of service.
Most dissertations, of any sort, are concerned with exploring the relationship between “x” and “y.” We
sometimes see dissertations that present research questions in two tiers.
THIS DISSERTATION WILL EXPLORE…
The top tier says something such as, “This dissertation will explore the relationship between x and y.” Then the
second tier says something such as, “This dissertation will explore this relationship by examining the following
three (or two or one or four) specific research questions.”
In many cases, the same top-tier question could lead to a variety of different dissertation studies. The lower-tier
question or questions, though, will be different for quantitative studies, in which we expect to see yes/no
questions, and qualitative studies, in which we expect to see “how” questions rather than “yes/no” ones. With
Action Research studies, either is possible. In addition, the research questions provide the answer to the
question, “Why did you choose Action Research?” When we choose Action Research, we do so because it
provides the best way to answer the questions that we’ve asked.
Remember that the research questions always drive the choice of methodology.
DECISION TO USE ACTION RESEARCH
As you read examples of Action Research in the journals and research reports from the links following this
presentation, look particularly at the research questions being asked. In each case, ask yourself how the
questions being asked lead to the selection of research methods and, within those, specific activities.
ORGANIZATION OF YOUR DISCUSSION
The research questions also drive your review of literature. A trick of the trade is to think of your review of
literature as focusing on your research topic and being made up of several sub-topics. Review each of those
topics as separate, but related, literature reviews.
SUGGESTION
Here is a suggestion. After you have agreed with your mentor on your research question but before you begin
your full review of literature, you may want to suggest a list of literature review sub-topics to your mentor. Your
mentor may agree with your topics, combine some of your sub-topics, or suggest new sub-topics. If you and
your mentor have an initial agreement on the sub-topics you will use in your literature review before you begin,
you can expect to cut significant time and frustration from your dissertation writing process.
Remember that, as a writer of an Action Research Dissertation, you will organize and write your literature
review to reflect the research questions that led you to choose an Action Research Methodology. There are
many ways to organize your review of literature. The worst way is usually chronological. You will end up with
some chronological ordering organically, in a natural way, but it’s a stronger choice to conceptualize the
organization of your review of literature by considering ideas. Let the organization of your review of literature
reflect the substance of the study you plan to conduct.
NOTICE PRIOR RESEARCH
As you read examples of Action Research in the journals and research reports from the links following this
presentation, notice how the discussion of prior research is organized and presented. Consider whether the
organizational structure that the writer used reflects the research questions and whether it works well in helping
you, as a reader, follow what is being presented.
ENGAGING IN ACTION RESEARCH
By engaging in Action Research, you accept both the opportunity to be of service and the challenge of
responsible involvement in a social research process.
RISK AND SERVICE
As you read the journal articles and research reports provided in the links at the end of this presentation, be
aware of the ways in which the studies explain the balance of service and risk that Action Research involves.
As you plan your own dissertation research, draw on your mentor, Capella’s Institutional Review Board, and
other resources to be sure that your study will, in the end, provide increased understanding in the ways that
you intend.
OUTCOMES AND RESULTS
Dissertations normally have a chapter devoted to reporting the results of the research and another chapter
discussing those results.
UNDERSTANDING THE WRITING PROCESS
Understanding the differences between these two chapters can cut significant amounts of time from your
writing process and result in a superior dissertation. With a quantitative dissertation, the chapter reporting
results may be comparatively easy to write, since it may focus on presenting numerical outcomes and the
results of statistical tests. On the other hand, learners who write qualitative dissertations or mixed methods
dissertations containing qualitative methods sometimes struggle because with qualitative methodologies it may
not be clear what should go into the chapter reporting the results, and what should go into the chapter
analyzing those results. A trick of the trade is to begin discussing this early on in the dissertation process with
your mentor. If you have parameters in mind as you shape and conduct the study, writing these chapters may
go more smoothly.
This is true for journal articles and research reports as well as for dissertations.
USING LANGUAGE TO SHOW DISTINCTION
As you read examples of Action Research journal articles and research reports, consider how easy or difficult
you find it to distinguish between sections that report study outcomes and sections that discuss and analyze
those outcomes. Be aware of whether and how the writer uses language to help you, as the reader, make that
distinction. If you are reading a study that employs mixed methods, be aware of how the reporting and
discussion of the multiple methodologies are organized and presented. Think about elements of the journal
article or research report that makes it easier or more difficult for you, as the reader, to follow the writer’s
presentation.
CONCLUSION
This concludes the media presentation on writing the Action Research Dissertation.
We have discussed what you should look for while reading examples of Action Research in the journals and
research reports from the links following this presentation. Those four elements include:
1. How the research questions asked led to the decision to use action research.
2. How the discussion of prior research is organized and presented.
3. How the study being reported balances service and risk within the Action Research paradigm.
4. How the writer separates reporting of outcomes and analysis of outcomes including how the writer uses
language to show the distinction.
Now, please look at the links to sources of academic writing related to action research. These links include
examples of (1) journals carried in the Capella library, and (2) university centers conducting and reporting
Action Research. The links to the academic writing tools that were introduced in the first part of this
presentation such as Reverse Outlining, the MEAL Plan, and the Writing Feedback Tool are repeated here so
you will be able to use them as you consider the material.
When you are ready, engage your fellow learners by responding to the discussion question in the courseroom.
Thank you for listening.
SCREEN 18
Overview of the Writing Center:
http://media.capella.edu/CourseMedia/XX7010/WritingCenterOverview/wrapper.asp
Academic Integrity:
http://media.capella.edu/CourseMedia/DrPH8005/AcademicIntegrity/animation_wrapper.asp
Smarthinking:
http://media.capella.edu/CourseMedia/DrPH8005/Smarthinking/smarthinking_wrapper.asp
Peer Review:
http://media.capella.edu/CourseMedia/DrPH8005/PeerReview/animation_wrapper.asp
SCREEN 19
Journals in the Capella library that publish action research articles:
• Action Research International
• Journal of Action Research
SCREEN 20
University research centers conducting and reporting action research:
Pepperdine University: Center for Collaborative Action Research
http://cadres.pepperdine.edu/ccar/
http://cadres.pepperdine.edu/ccar/define.html
The University of Illinois at Urbana-Champaign: East St. Louis Action Research Project
http://eslarp.illinois.edu/
http://eslarp.illinois.edu/view/about.aspx
Licensed under a Creative Commons Attribution 3.0 License.
1. Understand and analyze qualitative program evaluation design.
2. Compare and contrast experimental and quasi-experimental designs.
3. Analyze pretest-posttest designs.
4. Communicate through writing that is concise, balanced, and logically organized.
Unit 3 – Program Evaluation: Qualitative Research
Design
INTRODUCTION
This unit focuses on qualitative evaluation design, data collection methods, and evaluating program
effectiveness. Additionally, you will apply this knowledge to a real-world program evaluation.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
U03S1] Studies – Multimedia and Readings (Complete the following):
• Framework for Program Development and Evaluation view the flow chart/transcript
• Writing an Action Research Dissertation: Part One view the media/transcript
• Writing an Action Research Dissertation: Part Two view the media/transcript
The Writing an Action Research Dissertation media pieces will help you to understand the
academic writing standards for your doctoral program. You are expected to be proficient in this
type of writing by the end of your program. By using the advice and guidance of the media, you can
refine your academic writing and improve your success in this course and throughout your
program.
• Read Chapter 5 – Program Evaluation and Performance Measurement text
o Pay attention to question 7 on page 221. The content this question addresses will be
releant for the first discussion in this unit.
• Read Moore and Tananis’s 2009 article, “Measuring Change in a Short-Term
Educational Program Using a Retrospective Pretest Design,” from American Journal of
Evaluation, volume 30, issue 2, pages 189–202.
o Pay attention to the research design and data collection methods in this study. You
will be analyzing them for two upcoming assignments, one in this unit and the
other in Unit 5.
Constance
Highlight
Constance
Highlight
Constance
Highlight
Constance
Highlight
[U03A1] Unit 3 Assignment 1 – Program Evaluation: Analysis of Study Design
Using what you have learned through the readings and discussions up to this point in the course, read and analyze the 2009
journal article “Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design” by Moore
and Tananis. After you have finished your reading of the article, formalize your analysis by addressing the following:
• Identify the research design that was employed in the Moore and Tananis study.
• Explain whether the research design is experimental or quasi-experimental. Support your explanation by
comparing and contrasting characteristics between the two types of designs.
◦ Make sure you clearly state which design (experimental or quasi-experimental) the Moore and Tananis
study employs.
• Describe and analyze the data collection efforts detailed in the study.
◦ Discuss both quantitative and qualitative techniques.
• Justify any methodological recommendations for data collection methods for which you would advocate.
◦ Would you recommend using quantitative and qualitative data in program evaluation? Why or why not?
• Communicate through writing that applies current APA style and formatting and conforms to all assignment
instructions.
Resources
Program Evaluation: Analysis of Study Design Scoring Guide.
Distinguished
1. Identifies a research design employed by a selected research study, and discusses the implications of
using this design.
2. Compares and contrasts experimental and quasi-experimental designs, and notes which aspects of each
design would be most suited for a specific study.
3. Analyzes the data collection efforts presented in a research study and uses supporting documentation
from scholarly sources.
4. Justifies methodological recommendations for data collection with regard to program evaluation and
notes assumptions and evidence upon which the justification is based.
5. Consistently communicates through scholarly writing that is concise, balanced, and logically
organized, that flows with smooth transitions between ideas, and that demonstrates critical analysis of
the literature.
6. Consistently communicates through writing that applies current APA style and formatting, is free of
errors, and conforms to all assignment instructions.
APA Style and Format.
Writing an Action Research Dissertation: Part One | Transcript.
Writing an Action Research Dissertation: Part Two | Transcript.
Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest
Design
https://projects.ncsu.edu/ffci/publications/2011/v16-n1-2011-spring/vaterlaus-higginbotham.php
Constance
Highlight
Constance
Highlight
Constance
Highlight
Framework for Program Development and Evaluation
Reference: Comeau, J. (2011). Framework for program development and evaluation.Unpublished, Capella University, Minneapolis, MN.
L i c e n s e d u n d e r a C r e a t i v e C o m m o n s A t t r i b u t i o n 3 . 0 L i c e n s e .
