Statistical Analyses in Nursing

  

To prepare:

Review the articles presented in this week’s Learning Resources and analyze each study’s use of nonparametric tests.

Critically analyze each article, considering the following questions in your analysis:

What are the goals and purpose of the research study each article describes?

How are nonparametric tests used in each study? What are the results of their use?

Why are parametric methods (t tests and ANOVA) inappropriate for the statistical analysis of each study’s data?

What are the strengths and weaknesses of each study (e.g., study design, sampling, and measurement)?

How could the findings and recommendations of each study contribute to evidence-based practice in the health care field?

Reflect on the quantitative statistical analyses presented throughout this course in the research literature, the Learning Resources, media presentations, and those articles you reviewed for your abbreviated research proposal.

Ask yourself: Which method is most commonly used in research studies that pertain to my area of nursing practice, and why this might be so?

TO COMPLTE

Post 1-2 PAGES PAPER ON : A cohesive response addresses the following:

Critically analyze each article, including the items noted above.

Identify one statistical analysis method that you found recurring in many of the articles you used in your literature review for your research proposal. This method does not necessarily have to be nonparametric.

Based on your area of nursing practice ( family Nurse Practitioner) , which method of statistical analysis is most frequently used in the research literature? Why do you think other forms of statistical analysis are less frequently used? Provide a rationale for your response.

References

Gray, J.R., Grove, S.K., & Sutherland, S. (2017). Burns and Grove’s the practice of nursing research: Appraisal, synthesis, and generation of evidence (8th ed.). St. Louis, MO: Saunders Elsevier.

Chapter 25, “Using Statistics to Determine Differences”

Chapter 8, “Chi-Square and Nonparametric Tests”

See these attached articles :

Fisher, K., Orkin, F., & Frazer, C. (2010). Utilizing conjoint analysis to explicate health care decision making by emergency department nurses: A feasibility study. Applied Nursing Research, 23(1), 30–35. doi:10.1016/j.apnr.2008.03.

Tjia, J., Field, T., Garber, L., Donovan, J., Kanaan, A., Raebel, M., … Gurwitz, J. (2010). Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. American Journal of Managed Care, 16(7), 489–496.

Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. American Journal of Managed Care, 16(7) by Tjia, J., Field, T., Garber, L., Donovan, J., Kanaan, A., Raebel, M., & Gurwitz, J. Copyright 2010 by INTELLISPHERE, LLC. Reprinted by permission of INTELLISPHERE, LLC via the Copyright Clearance Center.

h 23 (2010) 30–35

www.elsevier.com/locate/apnr

Applied Nursing Researc

Utilizing conjoint analysis to explicate health care decision making by
emergency department nurses: a feasibility study

Kathleen Fisher, PhD, CRNPa,⁎, Fredrick Orkin, MD, MBA, Mscb,
Christine Frazer, MSN, CNSb

aCollege of Nursing and Health Professions, Drexel University, Philadelphia, PA 19102, USA
bPenn State University, Hersey Medical Center, Hersey, PA 17033, USA

Received 13 August 2007; revised 10 March 2008; accepted 22 March 2008

Abstract This descriptive study tests the feasibility of using clinical simulation to understand proxy decision

⁎ Corresponding
E-mail address: k

0897-1897/$ – see fro
doi:10.1016/j.apnr.200

making by emergency department nurses for individuals with intellectual disability (ID). Results from
a conjoint analysis used to identify decision-making patterns indicated that nurses relied on future
health status, functional status, and family input while making important health care decisions for their
clients. This information enhances our understanding of the complex array of services and supports that
nurses are expected to provide. As individuals with ID age and experience increased morbidity, the role
of the nurse and caregivers as critical health care decision makers is increasing.

© 2010 Elsevier Inc. All rights reserved.

1. Introduction

Recently, intellectual disability (ID) has emerged as the
preferred term to describe individuals who have significant
limitations in intellectual functioning and adaptive beha-
vior, the disability historically referred to as mental
retardation (Schalock, Luckasson, & Shogren, 2007).
According to the Diagnostic and Statistical Manual of
Mental Disorders, three criteria must be met to establish
this diagnosis, including impaired intellectual functioning
level (IQ) 70 or less; onset before the age of 18; and
significant limitations in two or more adaptive skill areas,
including communication, self-care, home living, social
and interpersonal skills, use of community resources, self-
direction, functional academic skills, work, leisure, health,
and safety (American Psychiatric Association, 1994).
Adaptive skills are needed to actively engage in commu-
nity living, and persons with ID by definition have
difficulty interacting with their environment. They are a
vulnerable population that places additional demands on
the health care system by virtue of their specific needs
(Fisher, Frazer, Hasson, & Orkin, 2007).

author. Tel.: +1 2157621208; fax: +1 2157621259.
athleen.mary.fisher@drexel.edu (K. Fisher).

nt matter © 2010 Elsevier Inc. All rights reserved.
8.03.004

2. Background

After the deinstitutionalization movement of the 1970s,
many persons with ID moved into community residential
agencies, such as group homes, where others routinely make
health care decisions for them for access to health care and
assistance with daily living. In a previous study of community
agency directors, proxy decision making was found to affect
the provision of appropriate health care services for indivi-
duals with ID and, in some situations, resulted in a delay or
even denial of health care. Disparities were particularly
evident when health care providers recommended less care for
individuals with ID when they perceived a lesser quality of life
as compared with that of individuals without ID (Fisher,
Haagen, & Orkin, 2005). The decision-making processes used
by proxies for persons with ID have not been well studied but
likely include assessment and synthesis of medical informa-
tion, personal beliefs and values, level of family involvement,
opinions of significant others including caregivers who know
the individual well, cognitive and functional status of the
individual, perceptions about the individual’s quality of life,
and institutional priorities and financial constraints. The
emphasis on individual variables probably differs from case
to case, such that decisions and the distribution of services
become unpredictable and disparate (Fisher et al., 2007).

mailto:kathleen.mary.fisher@drexel.edu

http://dx.doi.org/10.1016/j.apnr.2008.03.004

31K. Fisher et al. / Applied Nursing Research 23 (2010) 30–35

An estimated 7.5 million people in the United States have
an ID, representing approximately 3% of the U.S. population
(President’s Commission on Mental Retardation, 2002).
Health care issues associated with aging, chronic illness, and
end of life are new concerns for this vulnerable population.
In the past, those with ID did not live long enough to have
ongoing or chronic health problems (Fisher & Kettl, 2005).
This study’s focus on proxy health care decision making as it
relates to health promotion and access to care and services is
a critical, costly, and rising issue within the ID population.
Individuals with ID require assistance or supervision with
activities of daily living and health care decision making.
Many individuals with ID are aging, experiencing chronic
illness, outliving family caregivers, and can expect a return
to community residential support services.

3. Study purpose

The purpose of this pilot study was to test the feasibility of
conjoint analysis in studying the proxy decision-making
process among emergency department (ED) nurses and in
ascertaining their experiences with and perceptions of caring
for individuals with ID. The ED is a critical study site
because decisions made there may result in hospital
admission or discharge back into the community. Nurses
typically provide care for individuals with ID in the
community and acute care settings such as the ED, are
involved in health care decision making, and have an
influential role in determining health care outcomes.
Conjoint analysis is an innovative multivariate statistical
method that identifies, during an actual decision, the relative
“importance” of the factors in a decision and the ways
individual decision makers combine the factors in making
their decisions (Phillips, Johnson, & Maddala, 2002;
Phillips, Maddala, & Johnson, 2002). A clinical simulation
using conjoint analysis was developed with the assistance of
five nurses experienced in working with individuals with ID
and two ED nurse managers. The presenting clinical problem
described an individual with ID and a dental abscess.

4. Theoretical framework: Decision making

Decision making, also termed problem solving, informa-
tion processing, and judgment, has been studied extensively
during the past 30 years (Watson, 1994). Theories of
decision making exist in other disciplines and within
scientific and social science paradigms. Typically, these
decision-making models may not always apply to the real
world of decision making, particularly when attempting to
identify the optimal decision. The best option is not always
the one chosen (Noone, 2002).

Decision theory, which evolved from the field of cognitive
psychology, offers a model to examine the processes, out-
comes, and factors involved in decision making (Harbison,

2001). Most often, these theories view decision making as a
linear sequential process (Thompson, 1999). Utility theory,
one such theory, describes a management approach to
decision making under conditions of risk, although it has
not been widely used in nursing studies (Taylor, 2000). This
theory addresses one aspect of decision making for
individuals with ID and is explicated by conjoint analysis.

To understand a decision-making process, one might
merely ask an individual to explain how he or she made a
particular decision. However laudably simple that approach,
many individuals may be unable to verbalize precisely how
the decision was made, may overestimate and underestimate
the roles of given factors in the decision, or may offer a more
socially acceptable response. Also, such a simple approach
ignores the complexity inherent in any decision-making
process that involves the simultaneous evaluation and
combination of multiple factors, as in proxy health care
decision making for individuals with ID. These difficulties
may be avoided by studying decision making in the
controlled setting of a simulation in which the investigator
presents the decision maker with factors believed relevant to
a given decision. In such a simulation, a formal experimental
design dictates the groupings of factors presented simulta-
neously, such that it becomes possible to ascertain in an
unbiased manner the relative importance of individual
factors in decision making.

Decision making has been studied extensively in nursing
practice (Noone, 2002; Harbison, 2001). These studies offer
a foundation for understanding how nurses make decisions
with patients, but the context of a nurse–patient relationship
is different from a proxy relationship in which the person
receiving the care may have limited decision-making
capacity. Few studies have addressed proxy decision
making, and there is little knowledge to guide decisions,
particularly for a stigmatized population such as individuals
with ID (Fisher et al., 2005; Fisher et al., 2007).

5. Study sample

After receiving the institutional review board’s approval,
we undertook this study in spring 2004. We assembled a
convenience sample of 23 emergency department nurses
from two academic medical centers, located more than
100 miles apart. Each nurse gave informed consent before
participating.

6. Study design and instruments

Conjoint analysis is a measurement technique that uses
simulation coupled with a rigorous experimental design to
mathematically model decision processes at the level of the
individual decision maker (Green & Wind, 1975; Ryan &
Farrar, 2000). This multivariate statistical method is an
especially suitable analytic tool for studying proxy decision

Table 1
Hypothetical factors and factor levels for individual with ID having a minor
surgical procedure

Factors Factor levels

Mental competence Unable to make decisions
Legally incompetent

Functional status Ambulatory
Needing assistance
Bedfast

Likely future health status Unchanged
Improvement
Deterioration

Family input Absent
Approve
Disapprove

Extra cost to agency None
$1,000
$3,000

Person’s age (years) 7
30
62

32 K. Fisher et al. / Applied Nursing Research 23 (2010) 30–35

making because it explicates and describes decision making
and predicts outcomes of decisions made by proxies
(Phillips, Maddala, et al., 2002).

Conjoint analysis involves several steps, the first of which
is explicit specification of the decision to be modeled. Here,
the decision is approval of a minor surgical procedure (dental
extraction with anesthesia for a dental abscess) by the
designated health care decision maker for a person with ID.
The second step is selecting the factors believed relevant to
the decision. Using a literature review and interviews with
health care personnel who are faced with such decisions, we
identified a set of candidate factors for study: mental
competence, functional status, likely future health status,
person’s age, family input, and extra cost to agency (beyond
whatever health care insurance may be available). The third
step is assigning to each factor two or more plausible,
meaningful, and actionable factor levels to each of the factors
(Table 1). The factor levels and the factors are selected such
that decisions about each would be unlikely to be associated
with decisions about others (see Table 2).

Having thus developed the substrate for the simulation,
the fourth step is designing the scenarios in which each factor
is presented at the one-factor level (“full-profile” design). It
is not feasible to present all possible scenarios (i.e., 2 × 3 ×
3× 3 × 3 × 3 = 486) to the decision maker due to resultant

Table 2
Experimental design (fractional factorial design) for the first 6 scenarios among 2

Scenario Mental competence Functional status Likely future h

1 Unable to make decisions Bedfast Deterioration
2 Incompetent Ambulatory Deterioration
3 Unable to make decisions Ambulatory Deterioration
4 Incompetent Ambulatory Improvement
5 Unable to make decisions Bedfast Improvement
6 Unable to make decisions Bedfast Improvement

respondent fatigue that, in turn, would lead to decision
makers withdrawing from the study or oversimplifying their
decision making (e.g., decisions based solely on the “most
important” factor). To reduce the potential scenarios to a
manageable number, we used an experimental design
(fractional factorial design) that dictated the presentation of
22 scenarios, a subset of all possible combinations of factor
levels (Table 2); this highly favorable design requires that the
factors and factor levels be statistically independent (i.e., that
the underlying decision making relating to a given factor at a
given factor level is not influenced by that of other factor–
factor level combinations). Because of this design choice, the
analysis is limited to the role of each factor at each factor
level in decisions (“main effects”) and specifically cannot
explore potential influences (“interactions”) of factors at
given factor levels on one another. The description of each of
the 22 hypothetical scenarios, as dictated by the experimental
design, was presented on an index card (Table 2). The fifth
step is eliciting the decision makers’ preferences in relation
to the decision under study. The decision maker is asked to
rank order (most likely to least likely) or score (e.g., 1 to 100)
their likelihood of, in this case, approving the minor surgery
in each of the hypothetic scenarios. To reduce the intellectual
burden, we opted for rank ordering. In studies requiring more
than a half dozen factors and/or more factor levels than used
here, elicitation involves a large number of two-way
comparisons (e.g., Scenario A vs. Scenario B, Scenario A
vs. Scenario D); such a “discrete-choice” design seemed
excessively complicated for this application. With the data
collected, the final step is data analysis that is tailored to the
experimental design. Because the factors and factor levels
were chosen such that they are independent of each other in
the simulated decision, the analysis is analogous to an
analysis of variance with no interaction terms. For example,
a simpler decision involving two factors, each at three levels,
can be represented mathematically as follows: U(x) = B0 +
B1(X11) + B2(X12) + B3(X13) + B4(X21) + B5(X22) + B6(X23) +
Error, where U(x) is the overall perceive value (“utility”) of a
set of scenarios (xi through xk) composed of the two factors
(X1 and X2), B1 through B6 are the coefficients of factor
level (1 through 3) for each factor (1, 2), and B0 is the utility
when both factors are present at their first level. The
contribution of a factor at a given factor level (e.g., B1[X11])
to the overall utility is called the “partworth” and can be
computed in a dummy variable multiple regression analysis

2 hypothetical scenarios in conjoint analysis simulation

ealth status Family input Extra cost to agency, $ Person’s age

Disapprove 1,000 30
Disapprove 1,000 7
Approve None 30
Disapprove 3,000 30
Disapprove None 62
Approve 3,000 7

Fig. 1. Mean utility values for each factor at each factor level.

33K. Fisher et al. / Applied Nursing Research 23 (2010) 30–35

because the equation is composed of zeroes (in the absence
of a given factor at a given factor level) and ones (presence),
according to the factorial design. The overall perceive value
(utility), U(x), of a given scenario for a given decision maker
is its rank order. Thus, the regression analysis estimates each
decision maker’s set of utility values for each factor at each
factor level. In turn, the importance of a given factor for a
given decision maker is computed as that factor’s proportion
of the total utility (partworth).

7. Application to decision making for individuals
with ID

The conjoint analysis simulation required ED nurses to
place themselves in the role of decision maker for an
individual with ID, using a clinical scenario developed with
experienced ID nurses and the two ED clinical nurse man-
agers, based on experiences with individuals with ID who
frequented the ED for care. After completing a brief survey
that inquired about their age, gender, education, and years of
working experience, each nurse was asked to complete the
simulation task, requiring the rank ordering of 22 cards.

An example of one such card appears on the top row
of Table 2. Each nurse was read the following statements
by the nurse researcher and then handed the 22 cards for
rank ordering:

Like others, persons with ID have health care needs but may
not be capable of making decisions. Legal guardians,
including agencies overseeing residential homes, often make
health care decisions for the individual with ID. We are

studying the relative importance of different factors that may
influence these decisions. Assume that you are the designated
health-care decision maker for a person with ID who has a
dental abscess requiring a dental extraction and anesthesia
(i.e., a “minor” surgical procedure). The characteristics of
each of 22 such individuals are presented on these cards.
Please rank-order the cards so that the card describing the
individual for whom you would most likely approve the care
is first, the individual for whom you would least likely
approve the care is last, and the other cards are ranked in
between according to your likelihood of approving the care.

8. Data analysis

Conjoint analysis transformed each nurse’s set of
rankings into individual-factor utilities, from which we
computed the total utility of each care decision and the
percentage contribution of each factor to the care decisions
made by each nurse. To estimate the consistency with which
each nurse applied the utilities in their ranking decisions, we
correlated their actual rankings of a small subset of scenarios
not used to compute utilities with rankings predicted on the
basis of the derived utilities.

The importance of a given factor in decision making was
computed as the proportion of total utility in a given decision
scenario accounted for by the factor. Cluster analysis enabled
the identification of nurses whose decision-making patterns
were similar based on their factor utilities. Using con-
tingency tables with nonparametric tests (chi-square and
Fisher’s exact tests), we tried to explain the decision-making
patterns associated with the nurses’ characteristics. All

Fig. 2. Importance (percentage contribution) of each factor to the decision whether to approve a minor surgical procedure for a hypothetical person with ID by a
decision-making pattern. Whereas Group 1 and Group 2 are composed of 10 and 8 nurses, respectively, the other groups each consisted of 1 nurse. Error bars
denote 95% confidence intervals.

34 K. Fisher et al. / Applied Nursing Research 23 (2010) 30–35

statistical procedures were conducted with SPSS (Version
12, SPSS, Inc., Chicago, IL).

9. Results

Most of the nurses were women (95.7%), with an average
of 7 years of ED experience. Most were educated as diploma
nurses (43.5%), with others possessing bachelor of science in
nursing (30.4%), associate (21.7%), or master’s (4.3%)
degrees. Their mean age was 40 years, but ages ranged from
23 to 59 years. Each nurse took 20–25 minutes to complete
the rank ordering task; 2 nurses found the ranking task too
complicated. The 21 nurses who completed the task were
generally highly consistent in their rankings, with all but one
exhibiting a Pearson’s r of ≥.928 for the correlation between
their predicted and observed rankings. Using the proportion
of total utility as a surrogate for the importance of each factor
in the decision, the mean importance values for each factor
for the group of 21 nurses were likely future health status,
39%; family input, 19%; person’s age, 13%; extra cost to
agency, 12%; functional status, 10%; and mental compe-
tence, 6%. Underlying and accounting for these overall
group importance values were the participants’ utilities for
the individual-factor levels comprising the decisions (Fig. 1):
The decision to approve care was more likely if family
approval, improved future health status, and, to a lesser
extent, young age, no extra cost, and ambulatory functional
status were present. On the other hand, approval decisions
were least likely if there was deterioration in future health
and family disapproval and, to a lesser extent, if the patient
was bedfast and old and cost was high. The participants were
indifferent to mental competence, assistance needs,
unchanged future health status, absence of family input,
modest cost, and age between youth and being old.

Cluster analysis enabled identification of subgroups within
the group of 21 nurses, which exhibited discrete decision-
making patterns (Fig. 2). The largest subgroup (10 nurses)
relied largely on future health status (58% of total utility in
decision making), with lesser attention to family input, extra
cost to agency, and person’s age. Another subgroup (8 nurses)
relied moderately on future health status (25%) and family
input (31%), with lesser attention to functional status, extra
cost to agency, and person’s age. There were three other
decision-making patterns, each exhibited by one nurse:
Whereas one nurse relied heavily on mental competence
(43%) and person’s age (52%), another emphasized mental
competence (43%) and functional status (29%), and the third
used extra cost to agency (66%) supplemented by person’s
age (18%). Nurse’s work site, age, education, and years of
experience did not discriminate among these decision-
making patterns in this small pilot study sample.

10. Discussion

Conjoint analysis is feasible and useful for studying
complex health care decision making by proxies for those
with ID. Although this preference measurement technique
has been used almost 40 years in psychology and marketing
research (Green & Wind, 1975), it has been applied in a wide
array of health care applications only more recently
(Eberhart, Morin, Wulf, & Geldner, 2002; Orkin & Green-
how, 1978; Phillips, Johnson, et al., 2002; Phillips, Maddala,
et al., 2002; Ryan & Farrar, 2000). Rather than providing a
prescriptive or normative perspective on decision making,
the methodology reveals how decisions are actually being
made, in this situation by using a “real-world” clinical
simulation, that is, an individual with ID and a dental
abscess. In making health care decisions for individuals with

35K. Fisher et al. / Applied Nursing Research 23 (2010) 30–35

ID, nurses placed greatest weight on future health status,
particularly the likelihood of improvement. This was more
important than family input, age, extra cost, or current
functional status for the nurses as a group. Proxy decision
making is a complex issue that was not addressed uniformly
by all nurse respondents. There were subgroups with discrete
decision-making patterns that emphasized certain factors in
their decision. Of concern was the one nurse who placed
almost all weight on extra cost to agency and the age of the
individual in decision making and determining care for the
individual. Fortunately, this decision-making pattern was
expressed by only one individual. The appropriateness of
most proxy decisions was aligned with individual’s needs
and rights. Although conjoint analysis appears to be useful, it
is not known if the nurses responded as they might to an
actual ED patient or if there would be a difference in their
decision-making responses if they actually knew the
individual versus completing a simulation exercise.

11. Limitations

Real-world decision making may depart from what was
found in this study because simulation provides only an
approximation of reality, and conjoint analysis relies on an
additive utility model of decision making that arguably may
not capture the complexity of a particular decision. More-
over, given the multiple challenges in studying decision
making noted earlier and the absence of a gold-standard
methodology that might provide comparison results, it is not
possible to assess convergent, criterion, or discriminate
validity, even though the results reported herein appear to
have face and content validities. However, conjoint analysis
has become a mainstay approach in psychological and
marketing research because its results have been proven to be
robust, and more complicated models (e.g., alternatives to
additive linear model) have generally not been found better
or more informative. Although adequate for a feasibility
study, the sample size was insufficient to undertake a
meaningful explanation of the observed decision-making
patterns. Generalizability of our findings may be limited to
the two EDs studied. Study findings, however, suggest that
the simulation task was feasible and meaningful to this group
of nurses, supporting the use of conjoint analysis in future
research in proxy decision making.

12. Conclusions and implications

There is a gap in nursing knowledge related to proxy
decision making. This study demonstrates use of an
innovative method (conjoint analysis) to measure individual
variables in the decision-making process and describes how
study participants are currently making these decisions. We
concluded that the nurses used subsets of information in their
decision making and that almost all of the nurses (95%)

made their decisions on the basis of factors relating to the
individual with ID rather than on external issues (i.e., extra
cost to agency). Future health status was ranked most
important among studied factors by nurses in making health
care decisions for individuals with ID. Nurses in their role as
health educators and advocates for their clients need to know
what information proxy decision makers value. With this
knowledge, nurses can better serve their clients in institu-
tional and community settings, which should improve the
process and impact the recipient of the services. Further, we
conclude that the simulation task was feasible and mean-
ingful to this group of nurses, supporting the use of conjoint
analysis in future research.

References

American Psychiatric Association. (1994). Diagnostic and statistical
manual of mental disorders. (4th ed.) Washington, DC: American
Psychiatric Press, Inc.

Eberhart, L., Morin, A., Wulf, H., & Geldner, G. (2002). Patient preferences
for immediate postoperative recovery. British Journal of Anesthesia, 89,
760−761.

Fisher, K., Frazer, C., Hasson, K., & Orkin, F. (2007). A qualitative study of
emergency department nurses’ perceptions and experience in caring for
individuals with intellectual and developmental disabilities. Interna-
tional Journal of Nursing in Intellectual and Developmental Disabilities,
3(1), 1.

Fisher, K., Haagen, B., & Orkin, F. (2005). Acquiring medical services for
individuals with mental retardation in community based housing
facilities. Applied Nursing Research, 18(3), 155−159.

Fisher, K., & Kettl, P. (2005). Aging with mental retardation: Increasing
population of older adults with MR require health intervention and
prevention strategies. Geriatrics, 60(4), 26−30.

Green, P. E., & Wind, Y. (1975). New way to measure consumers’
judgments. Harvard Business Review, 53, 107−117.

Harbison, J. (2001). Clinical decision making in nursing: Theoretical
perspectives and their relevance to practice. Journal of Advanced
Nursing, 35(5), 126−133.

Noone, J. (2002). Concept analysis of decision making. Nursing Forum, 37
(3), 21−32.

Orkin, F. K., & Greenhow, D. E. (1978). A study of decision making: How
faculty define competence. Anesthesiology, 48, 267−271.

Phillips, K. A., Johnson, F. R., & Maddala, T. (2002). Measuring what
people value: A ED nurses decision making 15 comparison of “attitude”
and “preference” surveys. Health Services Research, 37, 1659−1679.

Phillips, K. A., Maddala, T., & Johnson, F. R. (2002). Measuring preference
for health care interventions using conjoint analysis: An application to
HIV testing. Health Services Research, 37, 1681−1705.

President’s Commission on Mental Retardation. (2002). Retrieved December
2002, from www.acf.dhhs.gov/programs/pcmr/mission.htm.

Ryan, M., & Farrar, S. (2000). Using conjoint analysis to elicit preferences
for health care. British Medical Journal, 320, 1530−1533.

Schalock, R. L., Luckasson, R. A., & Shogren, K. A. (2007). The renaming
of mental retardation: Understanding the change to the term intellectual
disability. Journal of Intellectual and Developmental Disabilities, 45(2),
116−124.

Taylor, C. (2000). Clinical problem solving in nursing: Insights from the
literature. Journal of Advanced Nursing, 31(4), 842−849.

Thompson, C. (1999). A conceptual treadmill: The need for “middle
ground” in clinical decision making theory. Journal of Advanced
Nursing, 30(5), 1222−1229.

Watson, S. (1994). An exploratory study into a methodology for the
examination of decision making by nurses in the clinical area. Journal of
Advanced Nursing, 20, 351−360.

http://www.acf.dhhs.gov/programs/pcmr/mission.htm

• Utilizing conjoint analysis to explicate health care decision making by emergency department nu…..

Introduction
Background
Study purpose
Theoretical framework: Decision making
Study sample
Study design and instruments
Application to decision making for individuals �with ID
Data analysis
Results
Discussion
Limitations
Conclusions and implications
References

VOL. 16, NO. 7 n THE AMERICAN JOURNAL OF MANAGED CARE n 489

n clinical n

© Managed Care &
Healthcare Communications, LLC

D
rug-induced injury is common. Independent risk factors for

adverse drug events in the ambulatory setting include ad-
vanced age, multiple comorbid conditions, and the use of

high-risk medications requiring close monitoring. For example, fail-
ure to appropriately monitor older patients receiving drug therapy ac-
counts for 36% of preventable adverse drug events in the ambulatory
setting.1

Efforts to improve monitoring within organizations are hampered by
a lack of comprehensive and specific guidelines for the laboratory moni-
toring of high-risk medications.2 Monitoring guidelines for selected
drugs exist in recommendations of organizations such as the American
Heart Association guidelines to manage heart failure,3 the American
Geriatrics Society Assessing Care of Vulnerable Elders medication
quality indicators,4 and the National Quality Forum–endorsed mea-
sures.5 A nationwide baseline monitoring assessment study6 developed
more comprehensive guidelines for medications used through 2000,
among which a subset of 14 were updated and adopted for an interven-
tion trial between 2002 and 2003 within a single health maintenance
organization.7,8

We sought to develop an updated and comprehensive list of drugs
requiring laboratory monitoring for an electronic medical record–em-
bedded clinical decision support intervention at a multispecialty group
practice for medications in clinical use during 2008. The intent of the
updated guidelines was to include drugs introduced to the market since
December 2000, the date of the literature review for the original drug
and laboratory monitoring recommendations published by Raebel et al6

in 2005, as well as to update laboratory test and frequency recommenda-
tions based on 2007 changes in monitoring recommendations.9 Herein,
we describe the development of guidelines to monitor high-risk medica-
tions in the ambulatory setting using a 2-step consensus-based approach,
including a national expert advisory panel and local leaders to select
candidate medications for monitoring and to determine the frequency of
laboratory monitoring. To estimate the potential effect of our guidelines
on actual practice, we determined the use frequency of the guideline
drugs and the prevalence for each of the recommended laboratory tests.

The specific objectives of this study
were (1) to develop recommendations
to guide the monitoring of high-risk
medications in the ambulatory set-
ting, (2) to assess the use prevalence

Development and Pilot Testing of Guidelines to Monitor
High-Risk Medications in the Ambulatory Setting

Jennifer Tjia, MD, MSCE; Terr y S. Field, DSc; Lawrence D. Garber, MD; Jennifer L. Donovan, PharmD;

Abir O. Kanaan, PharmD; Marsha A. Raebel, PharmD; Yanfang Zhao, MA; Jacquelyne C. Fuller, MPH;

Shawn J. Gagne, BA; Shira H. Fischer, AB; and Jerr y H. Gur witz, MD

Objectives: To develop guidelines to monitor
high-risk medications and to assess the preva-
lence of laboratory testing for these medications
among a multispecialty group practice.

Study Design: Safety intervention trial.

Methods: We developed guidelines for the
laboratory monitoring of high-risk medications
as part of a patient safety intervention trial. An
advisory committee of national experts and local
leaders used a 2-round Internet-based Delphi
process to select guideline medications based
on the importance of monitoring for efficacy,
safety, and drug–drug interactions. Test frequency
recommendations were developed by academic
pharmacists based on a literature review and
local interdisciplinary consensus. To estimate
the potential effect of the planned intervention,
we determined the prevalence of high-risk drug
dispensings and laboratory testing for guideline
medications between January 1, 2008, and July
31, 2008.

Results: Consensus on medications to include
in the guidelines was achieved in 2 rounds. Final
guidelines included 35 drugs or drug classes and
61 laboratory tests. The prevalence of monitor-
ing ranged from less than 50.0% to greater than
90.0%, with infrequently prescribed drugs having
a lower prevalence of recommended testing
(P <.001 for new dispensings and P <.01 for chronic dispensings, nonparametric test for trend). When more than 1 test was recommended for a selected medication, monitoring within a medication sometimes differed by greater than 50.0%.

Conclusions: Even among drugs for which there
is general consensus that laboratory monitor-
ing is important, the prevalence of monitoring
is highly variable. Furthermore, infrequently
prescribed medications are at higher risk for poor
monitoring.

(Am J Manag Care. 2010;16(7):489-496)

For author information and disclosures,
see end of text.

In this article
Take-Away Points / p490
www.ajmc.com
Full text and PDF

490 n www.ajmc.com n JULY 2010

n clinical n

of candidate medications for monitoring, and (3) to deter-
mine completion of recommended testing for medications
dispensed among the patient population.

METHODS
Study Setting and Population

This study was conducted in a large multispecialty group
practice that provides most medical care to members of a
closely associated health plan based in the New England
area of the United States. The group practice employs 250
outpatient clinicians at 30 ambulatory clinic sites. The
practice uses the EpicCare Ambulatory (Epic, Verona, WI)
electronic medical record system and provides medical care
to approximately 180,000 individuals. For this analysis, we
included patients if they received care from the multispe-
cialty group practice, were 18 years or older, and obtained
insurance coverage from the health plan between January
1, 2007, and July 31, 2008. Patients had to be continuously
enrolled during the observation period and not residing in
a long-term care facility. Data about medication exposure
were derived from the prescription drug claims of the health
plan. Data about laboratory test completion were derived
from the multispecialty group practice electronic medical
record.

Determination of High-Risk Drugs
and Laboratory Monitoring Guidelines

Laboratory monitoring guidelines for high-risk drugs were
developed using a sequential process adapted from an approach
created and tested for a study6 conducted within the Health
Maintenance Organization Research Network Center for Ed-
ucation and Research on Therapeutics. This approach used
an advisory committee of national experts and local health
plan leaders, including practicing clinicians and pharmacists
and experts in patient safety and geriatric pharmacotherapy.
The charge for this group was as follows: (1) to review a com-
prehensive initial list of medications requiring monitoring;
(2) to assess the importance of including monitoring recom-
mendations to evaluate efficacy, safety, and clinically relevant

drug–drug interactions; and (3) to deter-
mine the need to include infrequently
prescribed medications in the guidelines.
The initial list of high-risk drugs includ-
ed those commonly implicated in adverse
events among patients in the ambulatory
setting1 and those associated with adverse
events leading to emergency department
visits,10 as well as drugs with low moni-
toring rates,6,11 drugs included in national

quality guidelines,4 and drugs with black box warnings.12

We asked panel members to participate in an Internet-
based questionnaire administered in a 2-round modified
Delphi process13 between August and October 2008. Panel-
ists were asked whether electronic monitoring alerts should
be sent to primary care physicians, specialists, or both and
whether monitoring alerts should be generated for infre-
quently dispensed or effectively obsolete medications. Panel-
ists were also asked to rate the importance of monitoring each
candidate medication for efficacy, toxic effects, and drug–drug
interactions. Each question was answered based on a 5-point
Likert-type scale to evaluate agreement or disagreement with
statements concerning the importance of monitoring each
medication or medication class for the domain assessed. The
scale ranged from 1 (indicating “strongly agree”) to 5 (indi-
cating “strongly disagree”).

After the first round of the survey, we eliminated ques-
tions for which there was agreement and readministered
questions for which there was lack of consensus. Consistent
with other modified Delphi methods, consensus for a ques-
tion was defined by agreement on categorization by at least a
majority (>50.0%) of respondents.13 We then administered a
second questionnaire to participants. In this round, panelists
were reminded of their original responses to individual ques-
tions and were given the group’s aggregated response to the
questions in the first round. At this stage, each participant
was given the opportunity to revise his or her response to
increase consensus with the succeeding round. The results of
the Delphi process informed the selection of the final high-
risk drug list.

Determination of Laboratory Test
Monitoring Frequency

After selection of the final high-risk drug list, 2 research
pharmacists (JLD and AOK) reviewed the literature to de-
termine the appropriate frequency of laboratory monitoring
for each drug. The review included monitoring recommenda-
tions provided in manufacturer labeling information, nation-
ally available published guidelines, and clinical guidelines
from national organizations and initiatives (eg, American

Take-Away Points
This article adds to the existing literature and informs clinical decisions in the following
ways:

n By describing a process to develop guidelines for the laboratory monitoring of high-
risk medications within a multispecialty group practice.

n By estimating the potential effect of an intervention to improve laboratory testing of
high-risk medications by reporting the use frequency of high-risk medications requiring
monitoring in 2008 and the prevalence of test completion for high-risk drugs in 2008.

n By demonstrating that infrequently used medications have lower rates of recommend-
ed laboratory test monitoring.

VOL. 16, NO. 7 n THE AMERICAN JOURNAL OF MANAGED CARE n 491

Monitoring High-Risk Medications in the ambulatory Setting

completion. All analyses were conducted using commercially
available software (SAS 9.2; SAS Institute Inc, Cary, NC).
The study was approved by the institutional review boards of
the University of Massachusetts Medical School, Worcester,
and the participating group practice.

RESuLTS
Consensus Panel Survey

A panel of 3 pharmacists and 6 physicians participated
in the consensus survey. All physicians were certified by the
American Board of Internal Medicine, 2 had an added quali-
fication in geriatric medicine, and 3 had subspecialty board
certifications. All respondents completed the 2 rounds of the
survey. After the second round of the survey, consensus was
achieved for all 40 medications with respect to parameters
(ie, efficacy or toxic effects) for laboratory monitoring and for
alerts to physicians (ie, primary care physician vs specialist).
The consensus panel agreed that alerts for high-risk medi-
cations were indicated to monitor efficacy and toxic effects
alone or in the presence of significant drug–drug interactions.
The panel also agreed that alerts should be sent to special-
ists and primary care physicians and should be developed for
medications that were frequently and infrequently prescribed.
Because strict national prescribing policies already exist to
guide the use and monitoring of isotretinoin, we did not in-
clude this medication in the newly developed guidelines. The
resulting list of drugs included 39 drugs or drug classes.

Pilot Testing of Monitoring Recommendations
Once the medication list was established, we exam-

ined overall dispensings of the selected medications. The
chronic index dispensings ranged from 0 (procainamide and
ganciclovir) to 13,351 (statins), and new index dispens-
ings ranged from 0 (procainamide and ganciclovir) to 2463
(statins) during the observation period of January 1, 2008,
to July 31, 2008. Based on these results, 4 drugs with fewer
than 5 new or chronic users combined (ticlopidine, tolca-
pone, ganciclovir, and procainamide) were eliminated from
the final drug list, resulting in 35 drugs or drug classes in the
final list of monitoring guidelines. There were 61 separate
drug–laboratory test combinations for monitoring, with 15
requiring more than 1 laboratory test (eg, amiodarone–thy-
roid-stimulating hormone [TSH] and amiodarone–aspartate
aminotransferase [AST]) and 20 requiring a single test; for
the purpose of this study, we considered a basic metabolic
panel that includes sodium, potassium, chloride, bicarbonate,
blood urea nitrogen, and creatinine levels as a single test.

Overall, the rates of test completion ranged from 0.0% for
aspartate aminotransferase or alanine aminotransferase [ALT]

Heart Association guidelines3 and National Committee for
Quality Assurance–developed measures and guidelines5). Be-
cause manufacturers’ warnings often describe a nonspecific
frequency for monitoring such as “periodically,” we reviewed
other authoritative sources that pharmacists and clinicians
commonly use to guide decisions about the frequency of
monitoring. These sources included the following: (1) the
published literature; (2) the Micromedex Healthcare Series14
(Thomson Reuters [Healthcare] Inc, Greenwood Village,
CO) database, which sources the primary case report and ex-
perimental literature; (3) UpToDate Inc15 (Waltham, MA),
an online peer-reviewed reference database; and (4) the
Pharmacist’s Letter16 (Therapeutic Research Center, Stock-
ton, CA). The final list of laboratory monitoring tests and the
associated monitoring frequencies were reviewed by a local
panel of academic pharmacists, the clinical pharmacist for the
multispecialty group practice, and the medical director of the
multispecialty group practice to determine local acceptability
and congruence with local quality standards.

Statistical Analysis
We used drug dispensing claims between January 1, 2007,

and July 31, 2008, to identify the first dispensing of a high-risk
medication for a patient after January 1, 2008. New drug use
was defined as an initial dispensing on or after the index date
of January 1, 2008, and no drug dispensing in the 6 months
preceding the index date. Chronic (ongoing) drug use was de-
fined as a dispensing on or after January 1, 2008, with evidence
of drug dispensing in the 6 months before that date. Because
clinicians might rely on laboratory test results for up to 6
months before initiation of a drug or might order a test within
2 weeks after initiation of a drug, we defined test completion
for a new dispensing in 2008 as having occurred if there was at
least 1 associated monitoring test completed 180 days before
dispensing to 42 days after dispensing (test ordered <14 days after dispensing plus 28 days for test completion). Comple- tion of a test of serum drug levels (eg, serum carbamazepine) for new prescriptions was measured from the day of dispensing to 30 days after initial dispensing. Test completion for chronic medication use was defined as having occurred if there was at least 1 recommended test for the drug test pair that occurred up to 365 days before the index dispensing in 2008 through 42 days after the dispensing if the test was indicated annually (or 180 days before to 42 days after index dispensing if the test was indicated every 6 months). For each drug–laboratory test combination, the proportion of completed recommended tests was determined for all index dispensings in the observa- tion period. We used a nonparametric test for trend across or- dered groups by Cuzick17 to examine whether more frequently dispensed drugs had a higher prevalence of recommended test

492 n www.ajmc.com n JULY 2010

n clinical n

among new users of nefazodone, phenobarbital, and quinidine at
baseline to 96.8% for AST or ALT test among chronic users of
niacin (Table 1 and Table 2). Drug–laboratory test pair comple-
tion differed between new and chronic medication dispensings.
For new medication dispensings, less than 20.0% of the recom-
mended 61 drug–laboratory test pairs were monitored in the
majority (>75.0%) of patients. In contrast, close to 40.0% of the

recommended drug–laboratory test pairs for chronic dispensings
were monitored in greater than 75.0% of patients.

For drugs with multiple laboratory tests, the percentage
test completion often varied within the drug (Table 2). For
example, for new amiodarone use, 52.7% had an AST or ALT
test, while 74.6% had a TSH test; for azathioprine use, 28.0%
had a baseline AST or ALT test, while 72.0% had a com-

n Table 1. Prevalence of Laboratory Test Completion Among High-Risk Medications With a Single Laboratory Test
Indicated by Frequency of New Dispensings Between January 1, 2008, and July 31, 2008

Drug Name or Class

No. of Users

Test

Guideline
Recommendation

for Monitoring
Frequency

Completion of
Recommended

Laboratory Test, %

New ChronicNew Chronic

Statins 2463 13,351 AST or ALT Baseline and yearly 67.2 84.0

Angiotensin-converting
enzyme inhibitors

1736 8696 BMP Baseline and yearly 72.1 88.3

Diuretics

Thiazide 1198 5500 BMP Baseline and yearly 71.2 87.4

Loop 1031 2901 BMP Baseline and yearly 83.7 91.1

Warfarin 676 2268 INR Baseline, weekly
for first month,
and monthly

86.7 96.4

Thyroid replacement 636 4660 TSH Baseline and yearly 48.4 70.0

Potassium supplement 610 1610 K Baseline and yearly 87.7 90.0

Metformin 568 2056 Cr Baseline and yearly 44.7 60.4

Diuretics, potassium sparing 285 1383 BMP Baseline and yearly 71.5 87.4

Imidazole antifungalsa 243 20 AST or ALT Baseline and every
3 mo

28.8 60.0

Gemfibrozil 217 697 AST or ALT Baseline, 3 and
6 mo, and yearly

75.1 81.4

Allopurinol 198 821 Cr Baseline and yearly 85.9 90.9

Thiazolidinediones 120 537 AST or ALT Baseline, every
2 mo for first year,
and every 6 mo

73.3 64.8

Terbinafine 50 6 AST or ALT Baseline and every
2 mo

26.0 66.7

Niacin 49 95 AST or ALT Baseline and yearly 83.7 96.8

Angiotensin II receptor
blockers

32 69 BMP Baseline and yearly 62.5 82.6

Isoniazid 18 14 AST or ALT Baseline and every
2 mo

33.3 35.7

Rifampin 14 9 AST or ALT Baseline and every
2 mo

35.7 22.2

Theophylline 15 88 Theophylline
level

1 wk After initiation
and yearly

20.0 61.4

Nefazodone 2 15 AST or ALT Baseline, 3 and
6 mo, and yearly

0.0 26.7

ALT indicates alanine aminotransferase; AST, aspartate aminotransferase; BMP, basic metabolic panel; Cr, creatinine; INR, international normalized
ratio; K, potassium; TSH, thyroid-stimulating hormone.
aExcluding single-dose fluconazole.

VOL. 16, NO. 7 n THE AMERICAN JOURNAL OF MANAGED CARE n 493

Monitoring High-Risk Medications in the ambulatory Setting

n Table 2. Prevalence of Laboratory Test Completion Among High-Risk Medications With Multiple Laboratory
Tests Indicated by Frequency of New Dispensings Between January 1, 2008, and July 31, 2008

Drug Name or Class

No. of Users

Test

Guideline
Recommendation

for Monitoring Frequency

Completion of
Recommended

Laboratory Tests, %

New ChronicNew Chronic

Colchicine 289 255 Cr Baseline and yearly 79.6 84.3

CBC Baseline and yearly 74.1 78.4

Digoxin 202 1015 Cr Baseline and yearly 83.2 91.0

K Baseline and yearly 83.2 90.3

Digoxin level 5-7 d After dose changes
and yearly

31.7 63.5

Fenofibrate 147 190 CBC Baseline, monthly for 3 mo,
and every 6 mo for the
first year

55.8 70.0

AST or ALT Baseline and every 3-6 mo 71.4 86.3

Valproic acid 133 248 AST or ALT Baseline, every 2 mo for
6 months, and yearly

21.8 37.9

CBC Baseline and yearly 38.4 62.1

Valproic acid level 2-4 wk After initiation,
with changing clinical
status, and yearly

10.5 44.8

Cyclosporine 102 76 AST or ALT Baseline, monthly for 3 mo,
and yearly

38.2 48.7

Cr Baseline, every 2 wk for
3 mo, and yearly

59.8 34.2

Cyclosporine level Weekly for 2-3 mo
and monthly

0.0 18.4

Phenytoin 67 313 AST or ALT Baseline, monthly for 6 mo,
and yearly

31.3 46.3

Phenytoin level 2-4 wk After initiation
and yearly

37.3a 75.1

Methotrexate 65 213 AST or ALT Baseline and every 2-3 mo 60.0 83.6

CBC Baseline and monthly 72.3 66.2

Cr Baseline and every 2-3 mo 66.2 77.5

Amiodarone 55 79 AST or ALT Baseline and every 6 mo 52.7 59.5

TSH Baseline and every 3-6 mo 74.6 48.1

Carbamazepine 49 193 AST or ALT Baseline and yearly 38.8 57.5

CBC Baseline, monthly for 3 mo,
and yearly

57.1 72.5

Carbamazepine level 2-4 wk After initiation,
with changing clinical status,
and yearly

12.2 27.0

Lithium 36 125 CBC Baseline, 1 mo after
stabilized, and yearly

38.9 55.2

Cr Baseline, 1 mo after
stabilized, and yearly

41.7 60.8

TSH Baseline, 3 and 6 mo,
and yearly

11.1 39.2

Lithium level 2-4 wk After initiation,
with changing clinical status,
and yearly

16.7 39.2

(Continued)

494 n www.ajmc.com n JULY 2010

n clinical n

plete blood count (CBC). We also found that the frequency
of medication use was associated with the frequency of rec-
ommended test completion (Table 3). Infrequently dispensed
medications had a lower prevalence of test completion. Non-
parametric test for trend showed a significant trend toward
more frequently dispensed medications having a higher preva-
lence of appropriate testing (P <.001).

DISCuSSION
Herein, we describe the development of drug–laboratory

test monitoring guidelines based on manufacturers’ recom-
mendations and published guidelines. Recently published
guidelines to monitor drugs in the long-term care setting
included a similar list of medications and laboratory tests.18
To determine the potential effect of our guidelines on actual
practice, we determined the use frequency of the guideline
drugs and the prevalence of testing for each of the recom-
mended laboratory tests. We found that (1) certain high-risk

drugs had very low rates of use among our target population,
(2) rates of testing for high-risk drugs varied widely, and (3)
infrequently used drugs had lower testing rates. Our study adds
to the literature by identifying ways to inexpensively improve
patient safety and quality of care (describing a method for
formulating monitoring recommendations) and by provid-
ing final monitoring recommendations (detailed guidelines
are available from the authors) that are useful to practitioners
considering similar interventions at their local institutions.

It is important to point out that the level of evidence for
each recommended drug test pair varied and that this likely
explains some of the variation in testing among our sample.
For example, there is increasing controversy about the role
of evaluating renal function in patients taking metformin.19
Among our sample, we found that 45.0% to 60.0% of met-
formin users had completed a serum creatinine test. Similarly,
there is a difference in the role of digoxin-level monitoring
depending on indication, with digoxin serum drug levels of
less importance when prescribed for heart rate control and of

n Table 2. Prevalence of Laboratory Test Completion Among High-Risk Medications With Multiple Laboratory
Tests Indicated by Frequency of New Dispensings Between January 1, 2008, and July 31, 2008 (Continued)

Drug Name or Class

No. of Users

Test
Guideline
Recommendation
for Monitoring Frequency
Completion of
Recommended
Laboratory Tests, %
New ChronicNew Chronic

Primidone 27 60 CBC Baseline and every 6 mo 55.6 50.0

Phenobarbital level 2-4 wk After initiation, with
changing clinical status,
and every 6 mo

0.0 11.7

Primidone level 2-4 wk After initiation, with
changing clinical status, and
every 6 mo

0.0 11.7

Azathioprine 25 58 AST or ALT Baseline and every 3 mo 28.0 25.9

CBC Baseline, every week for
first month, every 2 wk for
second and third months,
then monthly or after
treatment changes

72.0 87.9

Methyldopa 12 44 AST or ALT Baseline and every 6-12 mo 16.7 63.6

CBC Baseline and every 6 mo 50.0 47.7

Phenobarbital 9 52 AST or ALT Baseline and every 6 mo 0.0 46.1

CBC Baseline and every 6 mo 22.2 69.2

Phenobarbital level 2-4 wk After initiation, with
changing clinical status,
and yearly

0.0 53.8

Quinidine 1 13 AST or ALT Baseline and yearly

0.0 92.3

Cr Baseline and yearly 100.0 92.3

K Baseline and yearly 100.0 92.3

Quinidine level 2-4 wk After initiation
and yearly

0.0 92.3

ALT indicates alanine aminotransferase; AST, aspartate aminotransferase; BMP, basic metabolic panel; CBC, complete blood count; Cr, creatinine;
K, potassium; TSH, thyroid-stimulating hormone.

VOL. 16, NO. 7 n THE AMERICAN JOURNAL OF MANAGED CARE n 495

Monitoring High-Risk Medications in the ambulatory Setting

greater importance when used for heart
failure.20 Among our sample, we found
digoxin levels completed for 30.0% to
60.0% of users. We did not discern the
indication for digoxin use in our study,
but previous investigations have shown
similarly low rates of digoxin-level test-
ing.21 Variance in the level of evidence
for testing may explain some of the dif-
ferences we observed in the prevalence
of completion among different tests
within the same drug.

We are concerned about the low
prevalence of testing for drug test pair recommendations that
are much less controversial such as evaluation of thyroid func-
tion among amiodarone users. Amiodarone-induced thyroid
dysfunction, including thyrotoxicosis and hypothyroidism,
can occur in up to 20% of users.22 For this drug test pair, we
found that 75.0% of new users and 50.0% of chronic users
had completed a TSH test. The reason for the lower rate of
testing among new drug users is unclear. Although a possible
explanation may be the initiation of drugs in the hospital, as
may be the case for amiodarone among using patients with
arrhythmias, the prevalence of hospitalizations did not differ
among those with versus without appropriate monitoring of
amiodarone in a prior study.23 Further investigation is neces-
sary to determine whether hospitalizations explain low testing
rates among our study population.

An important finding of our study was that patients using
infrequently prescribed drugs were less likely to complete rec-
ommended laboratory tests. This is consistent with literature
showing an association between patient volume and quality of
care24 but is not supported by at least 1 study25 examining patient
volume in association with process measures such as laboratory
testing in diabetes. The rationale supported by the human fac-
tors model of medical errors is that physicians will likely have
better familiarity of the literature for drugs that they prescribe
more frequently than for drugs that they do not prescribe.26

The implication of this finding can be considered in terms
of potential relative and absolute effects of an intervention
to improve monitoring. There is potential for significant ef-
fect from interventions aimed at improving infrequently used
medications because baseline rates of testing are low. This is
important for drugs such as phenobarbital, whose laboratory
monitoring is a Healthcare Effectiveness Data and Information
Set measurement for assessing health plan quality of care.27 In
contrast, the number of patients affected by infrequently pre-
scribed drugs is small from a population health perspective.
For example, the number of patients who failed to receive
monitoring of statins was 2136, more than 20 times greater

than the number of patients affected by failure to monitor the
7 infrequently prescribed drugs combined (ie, imidazole anti-
fungals, terbinafine, angiotensin II receptor blockers, isoniazid,
rifampin, theophylline, and nefazodone). Health plans consid-
ering the implementation of interventions to improve medi-
cation monitoring need to weigh the absolute and relative
potential effects of their program. Although the marginal cost
of targeting many medications in a health information tech-
nology–based intervention to improve monitoring is minimal
and allows for the inclusion of many medications, programs
using high-cost resources such as pharmacist time need to limit
target medications by weighing the relative overall effect care-
fully to optimally select target medications.

Other important drugs for inclusion in monitoring guide-
lines are those with a high likelihood of a serious adverse
outcome. Our guidelines incorporate several drugs commonly
implicated in adverse drug events leading to emergency depart-
ment visits,10 including hypoglycemics, warfarin, anticonvul-
sants, digoxin, theophylline, and lithium. Of these, we found
high rates for warfarin, which is often monitored through spe-
cial anticoagulation programs, but low-to-moderate rates for
digoxin, anticonvulsants, lithium, and theophylline.

Limitations of our study should be noted. First, our study
was conducted in a single multispecialty group practice. Sec-
ond, laboratory tests may have been ordered for another rea-
son (ie, not for high-risk medication monitoring), so that we
may have overestimated the prevalence of recommended test-
ing. Third, we were unable to confirm patient adherence to
drugs and were unable to identify patients who did not com-
plete tests because they were no longer using the medication.
Fourth, in many cases, despite the panel’s conclusions, limited
evidence was found to support testing frequency or impor-
tance, and that may explain the low prevalence of test order-
ing for some drugs. For many drugs, further study is necessary
to determine whether laboratory test monitoring improves
health outcomes. Furthermore, lower testing for new use may
also be explained by drug initiation in the hospital setting for

n Table 3. Completion of Recommended Laboratory Tests by Quartile of
Dispensing Frequency

Completion of Recommended Laboratory Tests,
Mean (SD), %

Quartile of Dispensing
Frequency

New Dispensingsa

Chronic Dispensingsb

1, Low 29.6 (33.7) 60.9 (25.5)

2 47.0 (22.4) 47.8 (23.9)

3 48.1 (27.4) 64.8 (18.5)

4, High 68.5 (17.6) 82.5 (11.7)

aP <.001, nonparametric test for trend. bP <.01, nonparametric test for trend.

496 n www.ajmc.com n JULY 2010

n clinical n

drugs such as amiodarone and antiepileptics. In these cases,
it is possible that baseline testing was obtained after initia-
tion in the hospital and was not repeated after discharge to
the ambulatory setting. However, similarly low rates of TSH
and aminotransferase testing have been reported for chronic
users of amiodarone23 and for determining serum levels of
antiepileptics.21 Fifth, we did not assess all guideline recom-
mendations such as monitoring after dose changes, drug–drug
interactions, changing renal function, and clinical status.

Our study updates the laboratory monitoring recommenda-
tions for high-risk drugs to include drugs used in ambulatory
practice until 2008. Our findings and guidelines are generalizable
to other institutions seeking to improve monitoring of high-risk
medications to enhance the safety for their patient population
after review and adaptation by local medical and pharmacy lead-
ership to ensure consistency with local standards. Health plans
considering similar interventions can use our findings to weigh
the potential benefit of targeting drugs of high-frequency and
low-frequency use in clinical practice, as our study demonstrates
potential for improvement in the laboratory monitoring of drug
types. Further research is necessary to understand factors con-
tributing to the low monitoring of high-risk drugs, including
physician factors resulting in lack of test ordering and patient
factors leading to incomplete performance of tests.

Acknowledgments
We acknowledge the contributions of the national expert advisory panel,

including Susan Andrade, ScD; Steven Simon, MD; Jerry H. Gurwitz, MD;
and Marsha A. Raebel, PharmD. Local experts included Josie Cambia-Kiely,
PharmD; Mojgan Hajji, PharmD; Michael Kelleher, MD; Leslie Harrold, MD;
Sarah McGee, MD; and Robert Yood, MD.

Author Affiliations: From the Division of Geriatric Medicine (JT, TSF,
SJG, SHF, JHG), University of Massachusetts, Worcester, MA; Meyers Pri-
mary Care Institute (JT, TSF, YZ, JCF, JHG), Worcester, MA; Fallon Clinic
(LDG), Worcester, MA; Massachusetts College of Pharmacy (JLD, AOK),
Worcester, MA; and Kaiser Permanente Colorado (MAR), Denver, CO.

Funding Source: This study was funded by grants R18 HS017203, R18
HS017817, and R18 HS017906 from the Agency for Healthcare Research
and Quality.

Author Disclosures: The authors (JT, TSF, LDG, JLD, AOK, MAR, YZ,
JCF, SJG, SHF, JHG) report no relationship or financial interest with any enti-
ty that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JT, TSF, LDG, JHG); ac-
quisition of data (JT, TSF, JCF); analysis and interpretation of data (JT, TSF,
LDG, JLD, AOK, MAR, YZ, JCF, SJG, SHF, JHG); drafting of the manuscript
(JT, TSF, JLD, AOK, MAR, SJG); critical revision of the manuscript for im-
portant intellectual content (JT, TSF, JLD, AOK, MAR, SJG, SHF, JHG);
statistical analysis (JT, YZ, JCF); obtaining funding (JT, JHG); and administra-
tive, technical, or logistic support (LDG, YZ, JCF, SJG, SHF).

Address correspondence to: Jennifer Tjia, MD, MSCE, Division of Ge-
riatric Medicine, University of Massachusetts, 377 Plantation St, Ste 315,
Worcester, MA 01605. E-mail: jennifer.tjia@umassmed.edu.

REFERENCES
1. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability
of adverse drug events among older persons in the ambulatory set-
ting. JAMA. 2003;289(9):1107-1116.
2. Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of elec-
tronic clinical reminders to improve medication laboratory monitoring.
J Am Med Inform Assoc. 2008;15(4):424-429.

3. Hunt SA, Abraham WT, Chin MH, et al; American College of Cardiol-
ogy, American Heart Association Task Force on Practice Guidelines,
American College of Chest Physicians, International Society for Heart
and Lung Transplantation, Heart Rhythm Society. ACC/AHA 2005
Guideline Update for the Diagnosis and Management of Chronic Heart
Failure in the Adult: a report of the American College of Cardiology/
American Heart Association Task Force on Practice Guidelines (Writ-
ing Committee to Update the 2001 Guidelines for the Evaluation and
Management of Heart Failure): developed in collaboration with the
American College of Chest Physicians and the International Society
for Heart and Lung Transplantation: endorsed by the Heart Rhythm
Society. Circulation. 2005;112(12):e154-e235.
4. Shrank WH, Polinski JM, Avorn J. Quality indicators for medication
use in vulnerable elders. J Am Geriatr Soc. 2007;55(suppl 2):
S373-S382.
5. National Committee for Quality Assurance. NQF-endorsed mea-
sures. 2008. http://www.ncqa.org/tabid/59/Default.aspx. Accessed
November 16, 2009.
6. Raebel MA, Lyons EE, Andrade SE, et al. Laboratory monitoring of
drugs at initiation of therapy in ambulatory care. J Gen Intern Med.
2005;20(12):1120-1126.
7. Raebel MA, Lyons EE, Chester EA, et al. Improving laboratory moni-
toring at initiation of drug therapy in ambulatory care: a randomized
trial. Arch Intern Med. 2005;165(20):2395-2401.
8. Raebel MA, Chester EA, Newsom EE, et al. Randomized trial to
improve laboratory safety monitoring of ongoing drug therapy in
ambulatory patients. Pharmacotherapy. 2006;26(5):619-626.
9. Dalbeth N, Stamp L. Allopurinol dosing in renal impairment:
walking the tightrope between adequate urate lowering and adverse
events. Semin Dial. 2007;20(5):391-395.
10. Budnitz DS, Pollock DA, Weidenbach KN, Mendelsohn AB, Schroed-
er TJ, Annest JL. National surveillance of emergency department visits
for outpatient adverse drug events. JAMA. 2006;296(15):1858-1866.
11. Feldstein AC, Smith DH, Perrin N, et al. Improved therapeutic moni-
toring with several interventions: a randomized trial. Arch Intern Med.
2006;166(17):1848-1854.
12. Wagner AK, Chan KA, Dashevsky I, et al. FDA drug prescribing
warnings: is the black box half empty or half full? Pharmacoepidemiol
Drug Saf. 2006;15(6):369-386.
13. Powell C. The Delphi technique: myths and realities. J Adv Nurs.
2003;41(4):376-382.
14. Micromedex Healthcare Series [online database]. Greenwood Vil-
lage, CO: Thomson Reuters (Healthcare) Inc; 2009.
15. UpToDate Inc [online database]. Waltham, MA: UpToDate Inc; 2009.
16. Pharmacist’s Letter. Stockton, CA: Therapeutic Research Center;
2009.
17. Cuzick J. A Wilcoxon-type test for trend. Stat Med. 1985;4(1):87-90.
18. Handler SM, Shirts BH, Perera S, Becich MJ, Castle NG, Hanlon JT.
Frequency of laboratory monitoring of chronic medications adminis-
tered to nursing facility residents: results of a national Internet-based
study. Consult Pharm. 2008;23(5):387-395.
19. Jones GC, Macklin JP, Alexander WD. Contraindications to the use
of metformin. BMJ. 2003;326(7379):4-5.
20. Aronson JK, Hardman M. ABC of monitoring drug therapy: digoxin.
BMJ. 1992;305(6862):1149-1152.
21. Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory
monitoring alerts within a computerized physician order entry system
for medication orders. Am J Manag Care. 2006;12(7):389-395.
22. Trip MD, Wiersinga W, Plomp TA. Incidence, predictability, and
pathogenesis of amiodarone-induced thyrotoxicosis and hypothyroid-
ism. Am J Med. 1991;91(5):507-511.
23. Raebel MA, Carroll NM, Simon SR, et al. Liver and thyroid monitor-
ing in ambulatory patients prescribed amiodarone in 10 HMOs.
J Manag Care Pharm. 2006;12(8):656-664.
24. Hewitt M, Petitti D, eds. Interpreting the Volume-Outcome Relation-
ship in the Context of Health Care Quality. Washington, DC: National
Academy Press; 2001.

25. Turchin A, Shubina M, Pendergrass ML. Relationship of physician
volume with process measures and outcomes in diabetes. Diabetes
Care. 2007;30(6):1442-1447.

26. Leape LL. A systems analysis approach to medical error. J Eval Clin
Pract. 1997;3(3):213-222.

27. National Committee for Quality Assurance. HEDIS 2009: Healthcare
Effectiveness Data & Information Set. Vol 1. Washington, DC: National
Committee for Quality Assurance; 2008. Narrative. n

Copyright of American Journal of Managed Care is the property of Intellisphere, LLC and its content may not

be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s express written

permission. However, users may print, download, or email articles for individual use.