NURS 80 WK 7 DISC

  

Logistic Regression in Nursing Practice

To prepare:

Review the three articles in this week’s Learning Resources and evaluate their use of logistic regression. Select one article that interests you to examine more closely in this Discussion

Critically analyze the article you selected considering the following questions:

What are the goals and purposes of the research study the article describes?

How is logistic regression used in the study? What are the results of its use?

What other quantitative and statistical methods could be used to address the research issue discussed in the article?

What are the strengths and weaknesses of the study?

How could the weaknesses of the study be remedied?

How could findings from this study contribute to evidence-based practice, the nursing profession, or society?

Post 4 paragraph that addresses the following:

1.In the first line of your posting, identify the article you examined, providing its correct APA citation.

2. Post your critical analysis of the article as outlined above.

3.Propose potential remedies to address the weaknesses of each study.

4.Analyze the importance of this study to evidence-based practice, the nursing profession, or society.

Provide 3 references

1.Gray, J.R., Grove, S.K., & Sutherland, S. (2017). Burns and Grove’s the practice of nursing research: Appraisal, synthesis, and generation of evidence (8th ed.). St. Louis, MO: Saunders Elsevier.

Chapter 24, “Using Statistics to Predict”

2. Statistics and Data Analysis for Nursing Research

Chapter 9, “Correlation and Simple Regression” (pp. 208–222)

3. Select One Article from below for your assignment discussion: 

Hoerster, K. D., Mayer, J. A., Gabbard, S., Kronick, R. G., Roesch, S. C., Malcarne, V. L., & Zuniga, M. L. (2011). Impact of individual-, environmental-, and policy-level factors on health care utilization among US farmworkers. American Journal of Public Health, 101(4), 685–692. doi:10.2105/AJPH.2009.190892

Tritica-Majnaric, L., Zekic-Susac, M., Sarlija, N., & Vitale, B. (2010). Prediction of influenza vaccination outcome by neural networks and logistic regression. Journal of Biomedical Informatics, 43(5), 774–781. doi:10.1016/j.jbi.2010.04.011

Xiao, Y., Griffin, M. P., Lake, D. E., & Moorman, J. R. (2010). Nearest-neighbor and logistic regression analyses of clinical and heart rate characteristics in the early diagnosis of neonatal sepsis. Medical Decision Making, 30(2), 258–266. doi:10.1177/0272989X09337791

Impact of Individual-, Environmental-, and Policy-Level
Factors on Health Care Utilization Among US Farmworkers
Katherine D. Hoerster, PhD, MPH, Joni A. Mayer, PhD, Susan Gabbard, PhD, Richard G. Kronick, PhD, Scott C. Roesch, PhD, Vanessa L. Malcarne, PhD,
and Maria L. Zuniga, PhD

US farmworkers face significant disease bur-
den1 and excessive mortality rates for some
diseases (e.g., certain cancers and tuberculosis)
and injuries.2 Disparities in health outcomes
likely stem from occupational exposures and
socioeconomic and political vulnerabilities. US
farmworkers are typically Hispanic with limited
education, income, and English proficiency.3

Approximately half are unauthorized to work in
the United States.3 Despite marked disease bur-
den, health care utilization appears to be low.1,4–9

For example, only approximately half of Califor-
nia farmworkers received medical care in the
previous year.6 This rate parallels that of health
care utilization for US Hispanics, of whom
approximately half made an ambulatory care
visit in the previous year, compared with 75.7%
of non-Hispanic Whites.10 Disparities in dental
care have a comparable pattern.6,8,11,12 However,
utilization of preventive health services is lower
for farmworkers5,7,13,14 than it is for both US
Hispanics and non-Hispanic Whites.15,16

Farmworkers face numerous barriers to
health care1,4,17: lack of insurance and knowledge
of how to use or obtain it,6,18 cost,5,6,12,13,18–20

lack of transportation,6,12,13,19–21 not knowing
how to access care,6,18,20,21 few services in the
area or limited hours,12,20,21 difficulty leaving
work,19 lack of time,5,13,19 language differ-
ences,6,8,18–20 and fear of the medical system,13

losing employment,6 and immigration officials.21

Few studies have examined correlates of health
care use among farmworkers. Those that have
are outdated or limited in representative-
ness.5,7,14,22,23 Thus, we systematically examined
correlates of US health care use in a nationally
representative sample of farmworkers, using re-
cently collected data. The sampling strategy and
application of postsampling weights enhance
generalizability. We selected correlates on the
basis of previous literature and the behavioral
model for vulnerable populations.24 The behav-
ioral model posits that predisposing, enabling,
and need characteristics influence health care

use.25 The ecological model, which specifies
several levels of influence on behavior (e.g.,
policy, environmental, intrapersonal),26 provided
the overall theoretical framework. To our
knowledge, we are the first to extensively exam-
ine multilevel correlates of farmworker health
care use. We sought to identify farmworkers at
greatest risk for low health care use and to
suggest areas for intervention at all 3 levels of
influence so that farmworker service provision
can be improved.

METHODS

The National Agricultural Workers Survey
(NAWS) sample, conducted annually in 39 US
states,27 provided the study’s primary data.
Because of fluctuations and regional differences
in population, the NAWS uses multistage sam-
pling and bases the sampling frame on crop labor
estimates.27 Employers are identified with simple
random sampling and whether they agree to

recruitment; their farmworkers are randomly
selected and asked to provide written informed
consent; and then consenting, eligible farm-
workers are interviewed.27 Eligible farmworkers
hold a variety of job titles (e.g., fieldworkers,
supervisors), but some (e.g., poultry or livestock
workers or workers with H-2A visas) are ex-
cluded from recruitment.27 We used data from
2006 (n =1519) and 2007 (n =1511) fiscal year
administrations. NAWS researchers contacted
5254 employers; 1456 were eligible, and 692
(47.53%) participated in recruitment. NAWS
researchers contacted 3379 workers, of whom
3099 (91.71%) participated (3030 provided
valid data). We imputed values for case partici-
pants with missing data on continuous variables
(age and income) with expectation maximization.
We used listwise deletion for categorical vari-
ables (all had < 5% of case participants missing) and eliminated 4 outliers. Although vulnerable farmworkers (e.g., low income, low education) had significantly more missing data, descriptive

Objectives. We examined individual-, environmental-, and policy-level corre-

lates of US farmworker health care utilization, guided by the behavioral model

for vulnerable populations and the ecological model.

Methods. The 2006 and 2007 administrations of the National Agricultural

Workers Survey (n = 2884) provided the primary data. Geographic information

systems, the 2005 Uniform Data System, and rurality and border proximity

indices provided environmental variables. To identify factors associated with

health care use, we performed logistic regression using weighted hierarchical

linear modeling.

Results. Approximately half (55.3%) of farmworkers utilized US health care in

the previous 2 years. Several factors were independently associated with use at

the individual level (gender, immigration and migrant status, English pro-

ficiency, transportation access, health status, and non-US health care utilization),

the environmental level (proximity to US–Mexico border), and the policy level

(insurance status and workplace payment structure). County Federally Qualified

Health Center resources were not independently associated.

Conclusions. We identified farmworkers at greatest risk for poor access. We

made recommendations for change to farmworker health care access at all 3

levels of influence, emphasizing Federally Qualified Health Center service

delivery. (Am J Public Health. 2011;101:685–692. doi:10.2105/AJPH.2009.

190892)

RESEARCH AND PRACTICE

April 2011, Vol 101, No. 4 | American Journal of Public Health Hoerster et al. | Peer Reviewed | Research and Practice | 685

and bivariate findings were comparable before
and after data cleaning. The final sample con-
sisted of 2884 farmworkers.

Measures

The NAWS is an approximately 60-minute
interviewer-administered survey. Location and
language are selected by the farmworker.

Outcome and individual-level factors. Re-
sponse to the question, ‘‘In the past 2 years, in
the United States, have you used any type of
health care services from doctors, nurses, den-
tists, clinics, or hospitals?’’ was the dichotomous
outcome variable. Categorical sociodemo-
graphic variables evaluated as potential corre-
lates were gender, marital status, country of
origin (US born vs non-US born), immigration
status (citizen, green card or other authoriza-
tion, and unauthorized status), English pro-
ficiency (speak and read well [proficient], speak
and read at least a little or somewhat [moder-
ate], and all others [limited]), and access to
transportation (US car or truck ownership vs
not). Because Latino or Hispanic ethnicity is
known to be associated with health care use,10

a variable reflecting race/ethnicity was included.
Participants were asked to categorize them-
selves into racial/ethnic categories, which the
NAWS research team had created. We then
combined responses to create a dichotomous
variable: Hispanic (Mexican, Mexican American,
Chicano, Puerto Rican, or other Latino/Hispanic)
versus not Hispanic.

Additional categorical variables were the
following: migrant status (nonmigrant, follow-
the-crop [FTC] migrant [2 farmwork loca-
tions > 75 miles apart], and shuttle migrant
[international shuttle or US homebase > 75
miles away but not FTC]), difficulty obtaining
health care (‡1 barrier vs none), need (diag-
nosis of chronic disease [i.e., heart disease,
diabetes, or asthma] vs none), and use of non-
US health care in the past 2 years. Age, annual
family income, and educational attainment (in
years) were continuous.

Environmental-level factors. To characterize
county rurality and US–Mexico border prox-
imity, we used the US Department of Agricul-
ture’s system (1 [urban]–9 [rural])28 and the
US–Mexico Border Health Commission’s defini-
tion (within 62 miles),29 respectively. The 2005
Uniform Data System, an annual survey admin-
istered to Federally Qualified Health Center

(FQHC) grantees,30 provided FQHC information.
FQHC information included grantee or delivery
site locations and Section 330 funds (i.e., federal
dollars distributed to FQHC grantees), full-time
equivalent (FTE) physicians, and total FTE staff.
Using geographic information systems,31 we
mapped grantees, delivery sites, and NAWS
growers.32 To obtain grantees’ counties and total
county FQHC delivery sites, we performed, re-
spectively, county–grantee and county–delivery
site geographic information systems spatial joins.
We aggregated FQHC resources to the county
level and incorporated farmworker population
(per 1000 farmworkers who performed agricul-
tural work in county; population estimates de-
rived from the 2007 census of agriculture). We
also used geographic information systems data to
calculate distance (Euclidean, in meters) from
employer to nearest FQHC delivery site.33 For all
variables except distance to nearest FQHC, we
merged data with the county in which interviews
were conducted.

Policy-level factors. Insurance status (insured
vs not), pay structure (salary, hourly, and piece
rate or combination piece and hourly), and
workers compensation (provided vs not) were
categorical variables.

Statistical Analysis

We applied postsampling weights to ac-
count for probability of sample inclusion.34

We used Stata version 9 (StataCorp LP, College
Station, TX)35 and SPSS version 13.0 (SPSS,
Inc, Chicago, IL)36 to calculate weighted in-
dividual or policy and nonweighted environ-
mental descriptive statistics, respectively. We
used Stata version 9 to assess weighted bivariate
associations. We included variables significantly
associated with health care use in bivariate
tests (P < .05) in a multivariate binary logistic regression analysis. Because of multicollinearity among health care resource and accessibility variables, we entered only 3 county-level vari- ables: total FTE staff, rurality, and border prox- imity. To account for clustering among farm- workers (level 1), within workplaces (level 2), and within counties (level 3), we used HLM version 6 (Scientific Software International, Lin- colnwood, IL).37 We entered individual and policy, distance to nearest FQHC, and county variables on levels 1, 2, and 3, respectively. We entered continuous variables grand-mean cen- tered. We performed dummy coding for variables

with 3 categories (we reran the model with new
reference groups to obtain all comparisons).

RESULTS

More than half (55.26%) of farmworkers
reported having used US health care during the
previous 2 years. Table 1 presents individual
variable descriptive data. The majority of
farmworkers were male, married, Hispanic,
foreign born, and in their 30s, with low
educational attainment and low annual family
income. Approximately half were unauthorized
or had limited English language proficiency.
Most farmworkers were nonmigrant, and few
had a chronic disease diagnosis. Fewer than
half had experienced at least 1 barrier to care or
did not own a vehicle in the United States. Less
than one fifth had used non-US health care
in the past 2 years. Table 2 presents environ-
mental variable descriptive data. NAWS
counties were more urban, and few were near
the US–Mexico border. Employers were closest
to 263 unique delivery sites (affiliated with 135
grantees). Nearly half (40.7%) of affiliated
grantees were migrant health centers. Table 3
presents policy variable descriptive data. The
majority reported uninsurance, workers com-
pensation coverage, and hourly payment.

Bivariate Associations With Health

Care Use

Table 1 presents bivariate associations be-
tween categorical individual variables and
health care use. Numerous characteristics were
associated with increased use: female, married,
non-Hispanic, US born, chronic disease diag-
nosed, owned a vehicle, and only used US
health care. US citizens were more likely to
have used health care than were those with
a green card or other authorization, who in
turn were more likely to have used health care
than were unauthorized farmworkers. The
highest rates of use were reported by English
proficient, followed by moderately proficient
farmworkers. Nonmigrant farmworkers used
more health care (FTC migrants had the second
highest rates). The relationship with barrier
endorsement was nonsignificant. Regarding
continuous variables, farmworkers who
used health care were significantly older
(mean = 36.47; SE = 0.59 vs mean = 32.40;
SE = 0.57; F = 24.66), with significantly higher

RESEARCH AND PRACTICE

686 | Research and Practice | Peer Reviewed | Hoerster et al. American Journal of Public Health | April 2011, Vol 101, No. 4

income (mean = 23 937.09; SE = 545.67 vs
mean =18 456.64; SE = 381.66; F = 67.74)
and education (mean = 8.58; SE = 0.15 vs
mean = 6.82; SE = 0.16; F = 61.00), P < .001.

Table 2 shows bivariate associations be-
tween continuous environmental variables
and health care use. Farmworkers who used
health care worked in counties with signifi-
cantly higher mean density of FQHC delivery
sites, Section 330 funds, FTE physicians, and
total FTE staff. Counterintuitively, farm-
workers who used health care had higher
mean distances to the nearest FQHC and
worked in more rural counties. The propor-
tion of farmworkers who used US health care
was significantly higher for nonborder
counties (55.89%; 95% confidence inter-
val[CI] = 52.81, 58.93 vs 33.74%; 95%
CI = 21.96, 47.95; F = 9.27; P = .002). Table 3
presents bivariate associations between policy
variables and health care use. Farmworkers with
insurance and workers compensation were
more likely to have used health care. Rates of
use were highest among salaried farmworkers
and lowest among those paid by piece or a
combination of hourly and piece pay.

Independent Multivariate Associations

With Health Care Use

We used median odds ratios (MORs) to
estimate clustering,38 which was moderate. The
range of level 2 MORs was 1.01 to 1.71. Cluster-
ing at level 3 was higher (MORs =1.73–2.26).
The addition of variables (especially at level 1)
substantially reduced the MORs. Results from the
multilevel population-average model are pre-
sented in Table 4. In multivariate tests, farm-
workers who were women, were moderately
proficient in English (vs limited), were nonmi-
grant (vs shuttle and FTC migrant), and had
a green card or other authorization (vs un-
authorized status) were more likely to have
used US health care in the previous 2 years, as
were those with a chronic disease diagnosis,
with US vehicle ownership, and who had not
sought non-US care in the past 2 years.
Working in a nonborder county was associ-
ated with higher US health care use. Insured
farmworkers and those paid by salary (vs
hourly and piece or combination) had higher
rates of use. Total FQHC full-time equivalent
staff and distance to nearest FQHC were not
independently associated.

TABLE 1—Individual-Level Characteristics and Their Bivariate Associations With US Health

Care Use: National Agricultural Workers Survey

Sample, United States, 2006–2007

Variable

Proportion, %

or Mean (SE)

Used Health Care During the

Previous 2 Y, % (95% CI)

Design-Based

Pearson F Test

Mean age, y 34.65 (0.42)

Mean income, US $ 21 484.97 (344.02)

Mean education, y 7.79 (0.12)

Gender 43.26***

Female 19.76 77.87 (70.84, 83.60)

Male 80.24 49.69 (46.33, 53.06)

Marital status 4.52*

Married 58.38 58.10 (54.49, 61.63)

Not married 41.62 51.27 (46.06, 56.44)

Race/ethnicity 73.55***

Non-Hispanic 20.32 82.71 (76.60, 87.48)

Hispanic 79.68 48.26 (45.04, 51.49)

Country of origin 75.97***

US born 26.16 80.55 (74.50, 85.45)

Foreign born 73.84 46.30 (43.11, 49.51)

Immigration status 77.69***

Citizen 28.51 79.56 (73.91, 84.25)

Green card or other authorization 20.57 65.49 (59.97, 70.63)

Unauthorized status 50.92 37.51 (33.87, 41.30)

English proficiency 74.46***

Proficienta 25.57 81.20 (74.99, 86.16)

Moderately proficientb 24.81 64.53 (59.58, 69.19)

Limited proficiencyc 49.62 37.25 (33.62, 41.02)

Migrant status 71.40***

Nonmigrant 70.87 66.39 (63.17, 69.48)

Follow-the-crop
d

4.35 36.82 (25.78, 49.44)

Shuttle migrant
e

24.78 26.65 (21.34, 32.73)

Health status 40.57***

Lifetime chronic disease diagnosis 8.13 87.54 (78.89, 92.96)

No chronic disease diagnosis 91.87 52.40 (49.25, 55.54)

Barriers to care 3.14

Endorsed none 55.80 57.71 (53.82, 61.51)

Endorsed ‡ 1 44.20 52.16 (47.36, 56.92)
Access to transportation in United States 102.50***

Owns car 54.73 69.41 (65.90, 72.71)

Does not own car 45.27 38.15 (33.54, 42.98)

Health care use outside United States 50.93***

No 81.80 60.49 (57.24, 63.65)

Yes 18.20 31.72 (25.40, 38.80)

Note. CI = confidence interval.
a
Defined as speaking and reading English well.

b
Defined as speaking and reading English at least a little or somewhat.

cDefined as all others who are not proficient or moderately proficient.
dDefined as a person who has 2 farmwork locations > 75 miles apart.
eDefined as a person who does an international shuttle to work or has a US homebase > 75 miles away but is not a follow-
the-crop worker.
*P < .05; ***P < .001.

RESEARCH AND PRACTICE

April 2011, Vol 101, No. 4 | American Journal of Public Health Hoerster et al. | Peer Reviewed | Research and Practice | 687

DISCUSSION

We characterized health care use in a rep-
resentative sample of US farmworkers. Just
over half had used US health care during the
previous 2 years, similar to previous studies
of farmworkers5–8 and US Hispanics.10 Rates
appear to be lower than were those for non-
Hispanic Whites.10 However, time frame and
methodological differences inhibit direct
comparison. Given the disproportionate

disease burden for farmworkers, the low rate of
use is of concern. We identified individual-,
environmental-, and policy-level correlates and
highlighted areas for intervention.

Consistent with previous studies of farm-
workers5,7,8 and Hispanics,39 women used sig-
nificantly more health care than did men. An-
other correlate was immigration status, with
unauthorized immigrants reporting less use. The
impact of immigration status on farmworkers’
health care use has not been studied previously,

but similar findings have been reported for US
Hispanics,40,41 perhaps as the result of poorer
labor protections42 or fear of immigration con-
sequences43; the latter is a barrier farmworkers
cited previously.21 Barriers to insurance for both
legal and illegal immigrants also may explain
the findings. Rates of having insurance for
Hispanics are lowest for unauthorized workers,
followed by those with a green card.41 Regardless
of immigration status, farmworkers who are
working in the United States should have access
to sound health care. Reducing immigrant
barriers to public-44 and employer-spon-
sored45 coverage and improving immigration
policy by providing more pathways to legal status
would likely improve farmworker health care
access. The public health care sector also should
enhance outreach to vulnerable immigrant
groups.

English proficiency was associated with
health care use, as in another study of farm-
workers.7 Farmworkers have reported language
as a barrier,6,8,18,19 but it may not be the strongest
impediment.46 Poor proficiency may affect
quality of care more than access. Improving
services for those with limited English language
proficiency would likely improve use as well as
the quality of that care. This potential improve-
ment may be especially true in rural areas, which
often lack language-tailored services.47,48 Cali-
fornia now requires that health plans, including

TABLE 2—Environmental-Level Characteristics and Their Bivariate Associations With US Health Care Use: National Agricultural Workers Survey

Sample, United States, 2006–2007

Variable Proportion, % or Mean (SD)

Used Health Care

During the Previous 2 Years, Mean (SE)

Did Not Use Health Care

During the Previous 2 Years, Mean (SE) Wald F Test

County FQHC sites
a

2.72 (7.82) 4.12 (0.70) 1.64 (0.52) 8.08**

County FQHC Section 330 funds
a
, US $ 773 846.54 (1 922 217.15) 1 023 556.00 (157 393.10) 469 375.00 (126 586.70) 7.53**

County FQHC FTE physicians
a

3.40 (8.77) 4.58 (0.59) 1.99 (0.30) 15.41***

County FQHC total FTE employeesa 40.77 (106.42) 50.07 (5.87) 24.25 (3.82) 13.61***

Nearest FQHC, metersb 16 797.58 (16 483.77) 16 997.03 (843.81) 14 479.53 (705.21) 5.24*

Ruralityc 3.58 (2.23) 2.99 (0.08) 2.73 (0.07) 6.19*

Proximity to US–Mexico borderc

‡ 62 miles 94.78
< 62 miles 5.22

Note. FQHC = Federally Qualified Health Center; FTE = full-time equivalent. All counties situated > 62 miles from the US–Mexico border were considered nonborder counties; all counties within 62
miles of the border were considered border counties.
a
2005 FQHC resource (per 1000 farmworkers in county) figures aggregated for counties from which farmworkers were sampled (n = 134).

b
Distance to nearest FQHC from employer using 2005 FQHC figures for employers from which farmworkers were sampled (n = 640).

c
Rurality and border figures for counties from which farmworkers were sampled (n = 134).

*P < .05; **P < .01; ***P < .001.

TABLE 3—Policy-Level Characteristics and Their Bivariate Associations With US Health Care

Use: National Agricultural Workers Survey Sample, United States, 2006–2007

Variable Proportion, %

Used Health Care During the

Previous 2 Years, % (95% CI)

Design-Based
Pearson F Test

Insurance status 107.21***

Insured 28.19 80.29 (75.51, 84.33)

Uninsured 71.81 45.43 (41.75, 49.16)

Workers compensation 34.65***

Provided by employer 70.94 61.75 (58.50, 64.90)

Not provided by employer 29.06 39.40 (33.06, 46.13)

Payment structure 20.70***

Salary 5.31 85.67 (78.11, 90.93)

Hourly 83.50 54.86 (51.47, 58.20)

Piece or combination hourly piece 11.19 43.80 (36.04, 51.89)

Note. CI = confidence interval.
***P < .001.

RESEARCH AND PRACTICE

688 | Research and Practice | Peer Reviewed | Hoerster et al. American Journal of Public Health | April 2011, Vol 101, No. 4

Medicaid, provide compensation for translation
services.49 Although this legislation represents
meaningful progress toward improving

patient–provider communication and quality of
care, the benefit would be limited to privately or
publicly insured individuals living in California.

Migrant farmworkers had lower rates of
health care use than did nonmigrants, as found
in another farmworker study.5 Migrant farm-
workers may not know where to go for care
when new to a community (a barrier farm-
workers cited previously).21 Employment or res-
idence instability may interfere with obtaining
insurance. Migrant farmworkers would benefit
from tailored outreach and services. Individual
FQHCs have attempted to tailor services to meet
the needs of migrant farmworkers. For example,
a Yuma County, Arizona, FQHC partnered
with other community-based organizations to
promote diabetes management among migrant
farmworkers, targeting migrant farmworker-spe-
cific barriers.50 Similar programming should be
disseminated across migrant farmworker-serving
FQHCs.

Access to transportation can improve health
care use, especially in nonurban settings,51 and
farmworkers have previously cited poor trans-
portation as a barrier.6,13,19,21 Indeed, we found
that vehicle ownership was associated with
health care use. Providing transportation to
services or to public transportation and using
more mobile clinics will likely improve health
care use for the many farmworkers lacking
transportation. Health status and use of non-US
health care, factors that we controlled for, also
were independently associated with the outcome.

To our knowledge, this was the first study
to comprehensively test the effect of public
health care resources on farmworker health
care use. We included only 4 environmental
variables in the multivariate model, 1 of which
(i.e., working in a non-border county) was
independently associated with use. Bivariate
associations suggested a positive impact of
FQHC resources, yet they had no independent
effect. Moreover, distance to nearest FQHC was
positively associated with use in bivariate tests.
These findings are surprising given that the
nearest FQHC was an average of approximately
10 miles from each grower and nearly half of
the nearest FQHC delivery sites were affili-
ated with migrant health centers. These find-
ings may suggest that some unmeasured
county characteristics accounted for the strong
bivariate effect. After all, FQHCs are dispro-
portionately located in areas of medical and
socioeconomic need (both of which affect
utilization). Findings also may suggest that
FQHCs are not adequately overcoming

TABLE 4—Multivariate Logistic Regression: Factors Independently Associated With Health

Care Use: National Agricultural Workers Survey Sample, United States, 2006–2007

Variable Coefficient OR (95% CI)

Individual-level factors

Age –0.01 0.99 (0.98, 1.01)

Income 0.00 1.00 (1.00, 1.00)

Educational attainment 0.01 1.01 (0.96, 1.05)

Female 1.18*** 3.24 (2.23, 4.73)

Married 0.22 1.24 (0.92, 1.69)

Non-Hispanic 0.41 1.51 (0.55, 4.16)

Born in United States 0.09 1.09 (0.39, 3.06)

Immigration status

Citizen vs unauthorized status 0.15 1.16 (0.55, 2.47)

Green card or other vs unauthorized status 0.48* 1.62 (1.09, 2.42)

Citizen vs green card or othera –0.33 0.72 (0.39, 1.32)

English language proficiency

Proficient vs limited 0.70 2.01 (0.68, 5.91)

Moderately proficient vs limited 0.54** 1.71 (1.20, 2.44)

Proficient vs moderately proficient
a

0.16 1.17 (0.64, 2.14)

Migrant status

Nonmigrant vs shuttle 0.81*** 2.26 (1.61, 3.16)

Follow-the-crop vs shuttle 0.06 1.06 (0.59, 1.91)

Nonmigrant vs follow-the-crop
a

0.75** 2.13 (1.33, 3.40)

Lifetime chronic disease diagnosis 1.66*** 5.25 (2.35, 11.71)

No barriers to care endorsedb

Owns car in United States 0.41** 1.50 (1.16, 1.95)

No health care use outside United States 0.50* 1.64 (1.12, 2.41)

Environmental-level factors

FQHC total FTE staff 0.00 1.00 (1.00, 1.00)

Proximity to nearest FQHC –0.00 1.00 (1.00, 1.00)

Rurality –0.05 0.96 (0.87, 1.05)

Nonborder county 1.08* 2.93 (1.25, 6.87)

Policy-level factors

Insurance and workers compensation status

Insured 0.84*** 2.32 (1.65, 3.26)

Has workers compensation 0.05 1.05 (0.81, 1.36)

Payment structure

Salary vs combination or piece 0.73* 2.08 (1.07, 4.08)

Hourly vs combination or piece 0.05 1.05 (0.69, 1.61)

Salary vs hourly
a

0.68* 1.98 (1.16, 3.39)

Notes. CI = confidence interval; FQHC = Federally Qualified Health Center; FTE = full-time equivalent; OR = odds ratio.
Participants were considered proficient in English if they spoke and read well, moderately proficient if they spoke and read at
least a little or somewhat, and limited if they did not fall into the first 2 categories. Participants were considered follow-the-
crop workers if they had 2 farmwork locations > 75 miles apart and shuttle workers if they performed an international shuttle
between their home and workplace or a shuttle > 75 miles between their US homebase and their workplace (and were not
considered follow-the-crop workers).
a
Dummy code comparison run in second multivariate (model estimates without robust SE).

bNot significant in bivariate tests of association.
*P < .05; **P < .01; ***P < .001.

RESEARCH AND PRACTICE

April 2011, Vol 101, No. 4 | American Journal of Public Health Hoerster et al. | Peer Reviewed | Research and Practice | 689

farmworkers’ personal barriers. Indeed, nearly
half of farmworkers who sought health care had
their last visit in a private setting (data not
shown). FQHCs are equipped to serve poor,
uninsured farmworkers; thus, to maximize their
utility to this population, they should better tailor
outreach and services to overcome personal
barriers, such as language and immigration status.

Farmworkers working near the US–Mexico
border had lower rates of US health care use,
as was the case in a study of women farm-
workers.23 Those working or living near the
border may seek care in Mexico,52 as shown
previously for border-dwelling Hispanics53 and
farmworkers.5,8,12 In fact, use of non-US care was
independently associated with decreased US
health care use in our study. Binational coverage,
in which entities (i.e., insurance, health care
system) on both sides of the US–Mexico border
share coverage and care, would capitalize on
such findings, likely improving access for farm-
workers. However, travel across the border is
inhibited for unauthorized farmworkers.
To explore this issue further, we reran the
multivariate model, stratifying by immigration
status group. Indeed, relationships between
nonborder county status and health care use
were significant and positive for citizens (odds
ratio [OR] = 9.05; P < .01) and those with green cards or other authorization (OR = 2.52; P < .01); the relationship was nonsignificant for unautho- rized farmworkers. Thus, although binational coverage holds promise as a solution for some border-dwelling farmworkers, its utility is limited to certain subpopulations.

Uninsurance has been cited as a barrier to
health care for farmworkers.6,18 It was associ-
ated with use in our study, consistent with studies
of women farmworkers14 and other popula-
tions.54,55 Only approximately one third of the
sample reported being insured, consistent with
rates for California farmworkers.8 Rates of health
care use among salaried farmworkers were
higher than were rates for those paid an hourly
or piece or combination wage. These findings are
likely related to health care cost, which has been
cited as a barrier in several farmworker stud-
ies.5,6,13,18,19 Payment structure also likely relates
to other barriers farmworkers previously
reported: fear of job loss,6 lack of time,5,13,19 and
need to stay at work to make money.19

Improving treatment of farmworkers not
paid with salary and insurance policies will

likely result in improved access to health care.
Mandating employer-sponsored coverage
would likely prove challenging, as would
a vast expansion of public benefits. Binational
coverage provides a promising alternative.
Insurance reform must be augmented with
programs targeting unauthorized immigrants,
who would likely be excluded from policy
changes. The FQHC system is well suited for
this task, assuming the suggested enhance-
ments are made.

Limitations

Our use of cross-sectional data limits our
interpretation. Only working farmworkers
were recruited, so those not at work because of
illness or injury were excluded, yielding a sam-
ple with unique characteristics relevant to
health care use. Similarly, participating em-
ployers were likely unique on relevant labor
practices. Because the NAWS was not designed
to measure all aspects of health care access and
use (e.g., regular source, perceived need), this
study’s characterization is incomplete. We ex-
amined only acculturation proxies (e.g., English
proficiency, which has significant limitations)56

and did not assess cultural determinants because
these factors were not assessed in the NAWS,
thereby limiting our understanding of critical
determinants.56,57 Cultural barriers and facilita-
tors of US farmworker health care use, from
perspectives of consumers and the workforce,58

should be studied further so that services can be
better tailored to population need. Because the
outcome’s 2-year time frame is lengthy, farm-
workers may not remember whether they used
health care. Still, study findings suggest validity
(e.g., need strongly associated with use).

We used Euclidean distance to nearest
FQHC to estimate geographic accessibility.
Future studies would benefit from examining
other proximity measures (e.g., to account for
geographic features).33 FQHC resources were
aggregated to the grantee’s county because of the
Uniform Data System structure. However, affili-
ated delivery sites may not be in the same
county. The measurement of FQHC impact
would be enhanced if distribution to delivery
sites was reported in the Uniform Data System. In
addition, we did not study resources from other
health care types (e.g., private physician offices,
hospitals, voucher programs). Future research
should explore their impact. The denominator

for health care resources was also imperfect;
FQHCs provide care to several non-farmworker
groups. A denominator derived from populations
living in poverty may have provided a superior
estimate. Finally, we did not account for the
endogeneity inherent in the relationship between
insurance and utilization.54 It was outside the
scope of our study to assess public policy’s impact
on health care use; studying its impact would
have implications for large-scale interventions to
improve access.

Conclusions

More research on farmworker health care
use is needed. Our understanding would be
enhanced with study of the usual source of
care, purpose, and volume of visits; additional
health care resources; and local, state, and
federal policy. We identified farmworker sub-
populations at risk for poor access as well as
numerous areas for intervention. To improve
access, outreach efforts should target farm-
workers at greatest risk for unmet need (i.e.,
those who are men, non-US citizens, migrant, or
who have limited English proficiency or trans-
portation). A plan for affordable health care
is needed. FQHCs provide low-cost services
regardless of immigration status. However,
differential community resources had no in-
dependent impact on utilization. Thus, al-
though increasing resources for the public
health care sector is needed, improvement in
how those resources are spent (e.g., enhanced
efforts to educate farmworkers about services,
providing tailored services, using more mobile
clinics) is needed as well.

Reform to the public health care system
alone will not resolve disparities in health care
utilization. Affordable health insurance is
needed, and the plan must address barriers
to insurance for farmworkers (e.g., immigration
status, inconsistent residence, income, and
employment). Binational coverage may assist
with these issues, especially for documented
farmworkers living near the US–Mexico bor-
der.

The farmworker population is large, with
approximately 3 million persons in the United
States occupying this position.59 Failing to pro-
vide them with sound health care will result in
continued disease burden, with implications for
both farmworkers and the US general popula-
tion.60 For example, tuberculosis burden is

RESEARCH AND PRACTICE

690 | Research and Practice | Peer Reviewed | Hoerster et al. American Journal of Public Health | April 2011, Vol 101, No. 4

significant for farmworkers2; providing primary,
secondary, and tertiary preventive care will not
only promote farmworker health but also protect
the general population from transmission. Addi-
tionally, farmworkers are a critical labor force
that assists with the production and distribution
of the US food supply. Protecting and promoting
farmworker health is, therefore, of great eco-
nomic and public health importance.

Farmworkers, FQHC providers and admin-
istrators, researchers, policymakers, advocacy
groups, and agricultural employers on both
sides of the US–Mexico border should continue
their efforts to move such changes forward.
These endeavors will likely improve farm-
workers’ health care use and in turn reduce
the observed disparities in disease burden and
mortality for this vulnerable population. j

About the Authors
At the time of this study, Katherine D. Hoerster was with
the San Diego State University/University of California,
San Diego Joint Doctoral Program in Clinical Psychology,
San Diego. Joni A. Mayer is with the Graduate School of
Public Health, San Diego State University, San Diego,
CA. Susan Gabbard is with JBS International, Burlingame,
CA. Richard G. Kronick and Maria L. Zuniga are with
the Department of Family and Preventive Medicine, Uni-
versity of California, San Diego. Scott C. Roesch and
Vanessa L. Malcarne are with the Department of Psychol-
ogy, San Diego State University.

Correspondence should be sent to Katherine D. Hoerster,
VA Puget Sound Healthcare System, Seattle Division,
1660 South Columbian Way (S-116), Seattle, WA
98108 (e-mail: Katherine.Hoerster@va.gov). Reprints
can be ordered at http://www.ajph.org by clicking the
‘‘Reprints/Eprints’’ link.

This article was accepted August 1, 2010.

Contributors
K. D. H. conceptualized, designed, and carried out all
aspects of the study and drafted the article. J. A. M.
assisted with conceptualization, design, analysis, and
revision of writing and supervised the work as doctoral
dissertation chair. S. G. provided assistance with con-
ceptualization, design, analysis, and revision of writing.
R. G. K. provided assistance with conceptualization, de-
sign, and revision of writing. S. C. R. provided statistical
consultation and assistance with design and revision of
writing. V. L. M. assisted with conceptualization, design,
and revision of writing. M. L. Z. assisted with conceptu-
alization, design, and revision of writing.

Acknowledgments
The US Department of Health and Human Services,
Health Resources and Services Administration (Bureau of
Primary Health Care, Office of Minority and Special
Populations) funded this work.

We are grateful to Marcia Gomez and Henry Lopez
for initiating this partnership. We thank the US De-
partment of Labor (especially Daniel Carroll) and JBS
International for their support and for granting access to

the National Agricultural Workers Survey (NAWS).
Harry Johnson and Andre Skupin from the San Diego
State University Geography Department provided guid-
ance and access to ArcInfo software. Access to the
Uniform Data System data was provided through col-
laboration with the National Center for Farmworker
Health. We are grateful to the farmworker participants
who provided their time and insights for the NAWS
and for the work they do on US farms every day.

Note. The funding was granted through K. Hoerster’s
collaboration with JBS International, an HRSA partner.
The funding body had no input into the study design, the
analysis, or the interpretation of the article’s findings.

Human Participant Protection
The institutional review boards of San Diego State
University and the University of California, San Diego
approved this study.

References
1. Villarejo D. The health of U.S. hired farm workers.
Annu Rev Public Health. 2003;24:175–193.

2. Mills PK, Beaumont JJ, Nasseri K. Proportionate
mortality among current and former members of the
United Farm Workers of America, AFL-CIO, in California
1973–2000. J Agromedicine. 2006;11(1):39–48.

3. Carroll D, Samardick RM, Bernard S, Gabbard S,
Hernandez T. Findings from the National Agricultural
Workers Survey (NAWS) 2001–2002: a demographic
and employment profile of United States farm workers.
Washington, DC: US Dept of Labor; 2005. Research
Report No. 9.

4. Arcury TA, Quandt SA. Delivery of health services
to migrant and seasonal farmworkers. Annu Rev Public
Health. 2007;28:345–363.

5. Littlefield C, Stout CL. A survey of Colorado’s
migrant farmworkers: access to health care. Int Migr Rev.
1987;21(3):688–708.

6. Rose D, Quade D. The Agricultural Worker Health
and Housing Program. Los Angeles: The California
Endowment; 2006.

7. Slesinger DP, Cautley E. Medical utilization patterns
of Hispanic migrant farmworkers in Wisconsin. Public
Health Rep. 1981;96(3):255–263.

8. Villarejo D, Lighthall D, Williams D, et al. Suffering in
Silence: A Report on the Health of California’s Agricultural
Workers. Woodland Hills, CA: The California Endow-
ment; 2000.

9. Feldman SR, Vallejos QM, Quandt SA, et al.
Health care utilization among migrant Latino farm-
workers: the case of skin disease. J Rural Health.
2009;25(1):98–103.

10. Ezzati-Rice TM, Rohde F. Variation in Ambulatory
Health Care Visits and Visits for General Checkup by
Demographic Characteristics and Insurance Status, U.S.
Civilian Noninstitutionalized Population Ages 18– 64,
2005. Rockville, MD: Agency for Healthcare Research
and Quality; 2008. Statistical Brief No. 201.

11. Manski RJ, Brown E. Dental Use, Expenses, Dental
Coverage, and Changes, 1996 and 2004. Rockville, MD:
Agency for Healthcare Research and Quality; 2007
MEPS Chartbook No. 17.

12. Quandt SA, Clark HM, Rao P, Arcury TA. Oral
health of children and adults in Latino migrant and

seasonal farmworker families. J Immigr Minor Health.
2007;9(3):229–235.

13. Goldsmith DF, Sisneros GC. Cancer prevention
strategies among California farmworkers: prelimi-
nary findings. J Rural Health. 1996;12(suppl 4):343–
348.

14. Palmer RC, Fernandez ME, Tortolero-Luna G,
Gonzales A, Mullen P. Correlates of mammography
screening among Hispanic women living in lower Rio
Grande Valley farmworker communities. Health Educ
Behav. 2005;32(4):488–503.

15. Soni A. Use of the Pap Test as a Cancer Screening Tool
Among Women Age 18– 64, U.S. Noninstitutionalized
Population, 2005. Rockville, MD: Agency for Healthcare
Research and Quality; 2007. Statistical Brief No. 173.

16. Soni A. Use of Breast Cancer Detection Exams Among
Women Age 40 and Over, U.S. Noninstitutionalized
Population, 2005: Rockville, MD: Agency for Healthcare
Research and Quality; 2007. Statistical Brief No. 170.

17. Hansen E, Donohoe M. Health issues of migrant and
seasonal farmworkers. J Health Care Poor Underserved.
2003;14(2):153–164.

18. Goertz HD, Calderon VJ, Goodwin S. Understand-
ing health needs of migrant workers in America’s
heartland. Urol Nurs. 2007;27(5):429–436; discussion
437–439.

19. Lantz PM, Dupuis L, Reding D, Krauska M, Lappe K.
Peer discussions of cancer among Hispanic migrant farm
workers. Public Health Rep. 1994;109(4):512–520.

20. Lukes SM, Simon B. Dental services for migrant and
seasonal farmworkers in US community/migrant health
centers. J Rural Health. 2006;22(3):269–272.

21. Perilla JL, Wilson AH, Wold JL, Spencer L. Listening
to migrant voices: focus groups on health issues in south
Georgia. J Community Health Nurs. 1998;15(4):251–
263.

22. Hoerster KD, Beddawi S, Peddecord KM, Ayala GX.
Healthcare use among California farmworkers: predis-
posing and enabling factors. J Immigr Minor Health.
2010;12(4):506–512.

23. Palmer RC, Fernandez ME, Tortolero-Luna G,
Gonzales A, Mullen P. Acculturation and mammog-
raphy screening among Hispanic women living in
farmworker communities. Cancer Control. 2005;
12(suppl 2):21–27.

24. Gelberg L, Andersen RM, Leake BD. The behav-
ioral model for vulnerable populations: application to
medical care use and outcomes for homeless people.
Health Serv Res. 2000;34(6):1273–1302.

25. Andersen RM. Revisiting the behavioral model and
access to medical care: does it matter? J Health Soc Behav.
1995;36(1):1–10.

26. US Dept of Health and Human Services. Theory at
a Glance: A Guide for Health Promotion Practice. Bethesda,
MD: National Cancer Institute; September 2005.

27. US Dept of Labor. The National Agricultural Workers
Survey. Available at: http://www.doleta.gov/agworker/
report9/introduction.cfm#method. Accessed May 1,
2008.

28. US Dept of Agriculture. Measuring Rurality: Rural-
Urban Continuum Codes. Available at: http://www.
ers.usda.gov/briefing/rurality/ruralurbcon. Accessed
March 15, 2009.

RESEARCH AND PRACTICE

April 2011, Vol 101, No. 4 | American Journal of Public Health Hoerster et al. | Peer Reviewed | Research and Practice | 691

29. US-Mexico Border Health Commission. Healthy
Border 2010: An Agenda for Improving Health on the
United States-Mexico Border. Executive Summary. El Paso,
TX; 2003.

30. Health Resources and Services Administration.
Health Center Program: Uniform Data System (UDS).
Available at: http://bphc.hrsa.gov/uds. Accessed August
15, 2008.

31. ESRI. ArcGIS ArcInfo (Version 9.3.1). Redlands, CA;
2009.

32. ESRI. ArcGIS: ArcInfo (Version 9.3.1) Data and
Maps. Redlands, CA; 2009.

33. ESRI. An Overview of the Distance Toolset. Available
at: http://webhelp.esri.com/arcgisdesktop/9.3/index.
cfm?TopicName=An_overview_of_the_Distance_
toolset. Accessed April 19, 2009.

34. US Dept of Labor. Statistical Methods of the National
Agricultural Workers Survey. Available at: http://www.
doleta.gov/agworker/pdf/1205_0453_Supporting_
Statement_PartB32210 . Accessed May 10, 2009.

35. StataCorp. Stata Statistical Software, version 9.
College Station, TX: StataCorp LP; 2005.

36. SPSS. Statistical Package for the Social Sciences,
version 13.0. Upper Saddle River, NJ: Prentice Hall; 2005.

37. Raudenbush SW, Bryk AS, Congdon RHLM. 6:
Hierarchical linear and nonlinear modeling, version 6.0.
[Computer software]. Lincolnwood, IL: Scientific Soft-
ware International; 2004.

38. Merlo J, Chaix B, Ohlsson H, et al. A brief conceptual
tutorial of multilevel analysis in social epidemiology:
using measures of clustering in multilevel logistic re-
gression to investigate contextual phenomena. J Epide-
miol Community Health. 2006;60(4):290–297.

39. Cabassa LJ, Zayas LH, Hansen MC. Latino adults’
access to mental health care: a review of epidemio-
logical studies. Adm Policy Ment Health. 2006;33(3):
316–330.

40. Callahan ST, Hickson GB, Cooper WO. Health care
access of Hispanic young adults in the United States.
J Adolesc Health. 2006;39(5):627–633.

41. Ortega AN, Fang H, Perez VH, et al. Health care
access, use of services, and experiences among undocu-
mented Mexicans and other Latinos. Arch Intern Med.
2007;167(21):2354–2360.

42. Passel J, Capps R, Fix M. Undocumented Immigrants:
Facts and Figures. Washington, DC: Urban Institute
Immigration Studies Program; 2004.

43. Berk ML, Schur CL. The effect of fear on access to
care among undocumented Latino immigrants. J Immigr
Health. 2001;3(3):151–156.

44. Fremstad S, Cox L. Covering New Americans: A Review
of Federal and State Policies Related to Immigrants’ Eligibility
and Access to Publicly Funded Health Insurance. Washing-
ton, DC: The Henry J. Kaiser Family Foundation Com-
mission on Medicaid and the Uninsured; 2004.

45. Buchmueller TC, Lo Sasso AT, Lurie I, Dolfin S.
Immigrants and employer-sponsored health insurance.
Health Serv Res. 2007;42(1 pt 1):286–310.

46. Timmins CL. The impact of language barriers on
the health care of Latinos in the United States: a review of
the literature and guidelines for practice. J Midwifery
Womens Health. 2002;47(2):80–96.

47. Casey MM, Blewett LA, Call KT. Providing health
care to Latino immigrants: community-based efforts in

the rural midwest. Am J Public Health. 2004;94(10):
1709–1711.

48. Torres M, Parra-Medina D, Bellinger J, Johnson AO,
Probst JC. Rural hospitals and Spanish-speaking patients
with limited English proficiency. J Healthc Manag. 2008;
53(2):107–119, discussion 119–120.

49. California Dept of Managed Health Care. Language
Assistance (SB 853). Available at: http://www.hmohelp.
ca.gov/healthplans/gen/gen_langassist.aspx. Accessed
December 15, 2009.

50. Rural Assistance Center. Rural Health Outreach
Diabetes Education Program. Available at: http://www.
raconline.org/success/success_details.php?success_id
=505. Accessed December 15, 2009.

51. Probst JC, Laditka SB, Wang JY, Johnson AO. Effects
of residence and race on burden of travel for care: cross
sectional analysis of the 2001 US National Household
Travel Survey. BMC Health Serv Res. 2007;7:40.

52. Wallace SP, Mendez-Luck C, Castaneda X. Heading
south: why Mexican immigrants in California seek health
services in Mexico. Med Care. 2009;47(6):662–669.

53. Ortiz L, Arizmendi L, Cornelius LJ. Access to health
care among Latinos of Mexican descent in colonias in two
Texas counties. J Rural Health. 2004;20(3):246–252.

54. Buchmueller TC, Grumbach K, Kronick R, Kahn JG.
The effect of health insurance on medical care utilization
and implications for insurance expansion: a review of
the literature. Med Care Res Rev. 2005;62(1):3–30.

55. Hargraves JL. Trends in health insurance coverage
and access among Black, Latino and White Americans,
2001–2003. Track Rep. 2004;11:1–6.

56. Lara M, Gamboa C, Kahramanian MI, Morales LS,
Bautista DE. Acculturation and Latino health in the
United States: a review of the literature and its sociopo-
litical context. Annu Rev Public Health. 2005;26:367–
397.

57. Page JB. The concept of culture: a core issue in
health disparities. J Urban Health. 2005;82(2 suppl. 3):
iii35–iii43.

58. Barron F, Hunter A, Mayo R, Willoughby D.
Acculturation and adherence: issues for health care
providers working with clients of Mexican origin.
J Transcult Nurs. 2004;15(4):331–337.

59. Larson A. Migrant and Seasonal Farmworker Enu-
meration Profiles Study. Vashon Island, WA: Migrant
Health Program, Bureau of Primary Health Care, Health
Resources and Services Administration; 2000.

60. Mechanic D, Tanner J. Vulnerable people, groups,
and populations: societal view. Health Aff (Millwood).
2007;26(5):1220–1230.

RESEARCH AND PRACTICE

692 | Research and Practice | Peer Reviewed | Hoerster et al. American Journal of Public Health | April 2011, Vol 101, No. 4

Copyright of American Journal of Public Health is the property of American Public Health Association and its

content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s

express written permission. However, users may print, download, or email articles for individual use.

Journal of Biomedical Informatics 43 (2010) 774–781

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / y j b i n

Prediction of influenza vaccination outcome by neural networks
and logistic regression

Ljiljana Trtica-Majnaric a,*, Marijana Zekic-Susac b, Natasa Sarlija b, Branko Vitale c

a Department of Family Medicine, Medical School Osijek, University of Osijek, Osijek, Croatia
b Faculty of Economics, University of Osijek, Osijek, Croatia
c Institute Rudjer Bošković, Zagreb, Croatia

a r t i c l e i n f o a b s t r a c t

Article history:
Received 10 December 2009
Available online 6 May 2010

Keywords:
Influenza vaccination
Outcome
Prediction
Neural networks
Multilayer perceptron
Radial-basis function
Probabilistic network
Logistic regression

1532-0464/$ – see front matter � 2010 Elsevier Inc. A
doi:10.1016/j.jbi.2010.04.011

* Corresponding author. Address: Department o
School Osijek, University of Osijek, Strossmayerova 1
31 376353.

E-mail address: ljiljana.majnaric@hi.t-com.hr (L. T

The major challenge in influenza vaccination is to predict vaccine efficacy. The purpose of this study was
to design a model to enable successful prediction of the outcome of influenza vaccination based on real
historical medical data. A non-linear neural network approach was used, and its performance compared
to logistic regression. The three neural network algorithms were tested: multilayer perceptron, radial
basis and probabilistic in conjunction with parameter optimization and regularization techniques in
order to create an influenza vaccination model that could be used for prediction purposes in the medical
practice of primary health care physicians, where the vaccine is usually dispensed. The selection of input
variables was based on a model of the vaccine strain which has frequently been changed and on which a
poor influenza vaccine response is expected. The performance of models was measured by the average hit
rate of negative and positive vaccine outcome. In order to test the generalization ability of the models, a
10-fold cross-validation procedure revealed that the model obtained by multilayer perceptron produced
the highest average hit rate among neural network algorithms, and also outperformed the logistic regres-
sion model with regard to sensitivity and specificity. Sensitivity analysis was performed on the best
model and the importance of input variables was discussed. Further research should focus on improving
the performance of the model by combining neural networks with other intelligent methods in this field.

� 2010 Elsevier Inc. All rights reserved.

1. Introduction

Prevention and control of influenza epidemics is a major chal-
lenge for public health care services. The current approach is
annual vaccination with a trivalent inactivated influenza vaccine
(against the A/H1N1, A/H3N2 and B influenza virus strains) [1].
Although this approach is generally safe and effective in preventing
influenza, there is a need for influenza vaccines with improved effi-
cacy in the elderly [2]. This need is based on the observation that
available vaccines are less effective in the elderly than in the youn-
ger population [2,3]. Factors responsible for these differences have
been identified and include: older age, previous exposures to influ-
enza viruses, pre-existing antibody titres and chronic aging dis-
eases [4,5].

Several new vaccine preparations and vaccination approaches
are currently being pursued in order to improve the efficacy of
influenza vaccines in the elderly [6,7]. Decision on the introduction
of new vaccines into national vaccination programs is, however,

ll rights reserved.

f Family Medicine, Medical
05, Osijek, Croatia. Fax: +385

rtica-Majnaric).

connected with the demand for cost-effectiveness analyses and
development of an influenza vaccination action plan [8]. The main
challenge is to predict which individual will most likely adequately
respond to conventional influenza vaccines and which individual
will not.

Experience from clinical medicine suggests the use of a biology
systems approach within the context of artificial intelligence,
when obtaining models of prediction [9]. This assumption is based
on the observation that chronic aging diseases are characterized by
multiple factors, interacting with each other in a non-linear
manner. This is the reason for the huge variability in disease
expression and severity among individuals [10]. The artificial
neural networks (ANN) has been shown to be a suitable com-
puter-based method which can incorporate non-linear effects
and interactions between multiple variables in a valid probability
model [11].

Neural networks (NNs) as one of the artificial intelligence meth-
ods has been successfully used for classification, prediction and
association in different fields, including general purpose, as well
as some specific fields, such as diagnosis of disease [12]. Together
with genetic algorithms, clustering algorithms, decision trees and
other methods, NNs are widely used in Data Mining methodology
for revealing hidden non-linear relationships among data [13].

http://dx.doi.org/10.1016/j.jbi.2010.04.011

mailto:ljiljana.majnaric@hi.t-com.hr

http://dx.doi.org/10.1016/j.jbi.2010.04.011

http://www.sciencedirect.com/science/journal/15320464

http://www.elsevier.com/locate/yjbin

L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781 775

Previous research showed that, as a non-parametric method, NNs
has remarkable information processing characteristics, pertinent
mainly to non-linearity, high parallelism, fault and noise tolerance,
learning and generalization capabilities [14]. Some of the other
advantages of NNs are the ease of optimization, cost-effectiveness
and flexible non-linear modeling of large datasets, and accuracy for
predictive inference showing that NNs could serve as a valuable
decision support tool in different areas, including medicine [15].
Three NN algorithms were tested in this paper: multilayer percep-
tron (MLP), radial-basis function network (RBFN) and probabilistic
network (PNN). MLP is a general purpose feedforward network,
and one of the most frequently used NN algorithms. In order to
optimize the error function it uses the classical backpropagation
algorithm based on deterministic gradient descent algorithm orig-
inally developed by Paul Werbos in 1974, extended by Rumelhart,
Hinton, Williams (in [16]). RBFN is based on a clustering procedure
for computing distances among each input vector and the center,
represented by the radial unit. The ability of RBFN with one hidden
layer to approximate any non-linear function has been demon-
strated by Park and Sandberg (in [17]). The PNN algorithm was
tested due to its fast learning and efficiency in classification. It is
a stochastic-based NN developed by Specht (in [16]). The architec-
ture of the PNN is based on Bayes’classifier, using the Parzen win-
dow estimator to estimate the probability distributions of the class
samples [18].

Logistic regression (LR) modeling is widely used for analyzing
multivariate data involving dichotomous responses dealt with in
this paper. It provides a powerful technique analogous to multiple
regression and ANOVA for continuous responses. Since the likeli-
hood function of mutually independent variables Y 1; . . . ; Y n with
outcomes measured on a binary scale is a member of the exponen-
tial family with ðlogð p11�p1Þ; . . . ; logð

pn
1�pn
ÞÞ as a canonical parameter

(pj is a probability that Y j becomes 1), the assumption of the logis-
tic regression model is a linear relationship between a canonical
parameter and the vector of explanatory variables xj (dummy vari-
ables for factor levels and measured values of covariates):

log
pj

1 � pj

� �
¼ xsj b ð1Þ

This linear relationship between the logarithm of odds and the vec-
tor of explanatory variables results in a non-linear relationship be-
tween the probability of Y j equals 1 and the vector of explanatory
variables:

pj ¼ expðxsj bÞ=ð1 þ expðx
s
j bÞÞ ð2Þ

Detailed description of the logistic regression can be found in
Harrel [19].

Three NN algorithms, as well as logistic regression were used in
order to provide the influenza vaccination probability model that
could be used for prediction purposes in the practice of primary
health care physicians, where the vaccine is usually dispensed.

2. Materials

The study was based on original data collected in primary
health care in Croatia. A total number of 90 patients, out of 150
individuals requiring the influenza vaccine during 2003/2004, gave
their consent and were enrolled in the study. There were 35 male
and 55 female patients, 50–89 years old (median 69), all suffering
from multiple chronic medical conditions. Study protocol was ap-
proved by the local ethics committee.

The commercially licensed trivalent inactivated split vaccine
was used for the vaccination, manufactured by Solway, the Nether-
lands, containing the following influenza virus strains: A/H1N1/
New-Caledonia/20/99-like, A/H3H2/Moscow/10/99-like and B/Hong

Kong-330/2001-like. Specific antibody production was measured
by the Hemagglutination Inhibition (HI) test, a standard microtitre
technique. At least a fourfold increase in antibody titre was used
for expression of the specific antibody induction. For calculation
of geometric mean titres (GMT), a titre of <1:10 was arbitrarily set at 5. The influenza B vaccine strain was also tested on the B/Sicuan 379/99 strain, contained in the vaccine in the recent past, for a het- erologous reaction [1].

The target attribute in our study was the vaccination reaction to
the influenza vaccine virus strain B/Hong Kong. The output variable
used in NN models was binary, expressed in the form of two cate-
gories, where 0 value denoted the category of negative vaccine
reaction, or less than the fourfold increase in antibody titre, while
the value of 1 denoted the positive vaccine reaction, or the fourfold
and more increase in antibody titre.

Available input space included a large set of variables, such as:
the physician’s diagnoses of the main groups of chronic diseases,
anthropometric measures, hematological and biochemical labora-
tory tests. Blood tests were chosen on the basis of two criteria:
(a) to determine the main age-related pathogenetic changes and
(b) to be available in a real health care system setting. Based on
these criteria, we performed blood tests to determine: (1) inflam-
mation, (2) nutritional status, (3) metabolic status, (4) chronic re-
nal impairment, (5) latent infections, (6) humoral immunity and
(7) the neuroendocrine status. Blood samples were collected from
subjects three times prior to the vaccination and once 4 weeks
after the vaccination (for paired serum measuring). Hematological
analyses were carried out on fresh blood samples, while sera for
biochemical analyses and serological tests were separated by cen-
trifugation and stored at �40 �C until assayed.

Due to the large dimension of initial input space, it was neces-
sary to reduce the number of input variables before NN and LR
modeling. Non-linear Data Mining algorithms were used for this
purpose, resulting in a final set of 26 input variables [20]. Conse-
quently, the results obtained by NN and LR modeling could be
biased by the choice of the preprocessing method. Future research
should be focused on the use of other preprocessing methods in
modeling, and use of more datasets.

The total 26 input variables used in the NN and LR models can
be divided into three groups: (1) data related to previous exposure
to influenza viruses (the number of previous vaccinations and pre-
existing antibody titres for all four influenza virus vaccine strains,
measured in the study), (2) the set of medical data, and (3) age
(implicating age-related changes in the immune system). All avail-
able input variables and their descriptive statistics (mean and stan-
dard deviation for continuous variables, frequencies for categorical
variables) are presented in Table 1.

The whole sample consisted of 60 patients with negative vac-
cine outcome, and 30 patients with positive vaccine outcome.

Many authors, such as Liu and Tourassi, emphasize that model
selection should be performed on the basis of generalization error
[21,22]. Some of the well-known methods for testing the general-
ization ability of models are n-fold cross-validation, jackknifing,
bootstrapping and round robin technique [13,22]. All of them
have the purpose of reducing the small-sample estimation bias
and variance contributions [22,23]. Cross-validation was used in
this paper because it produces no statistical bias of the result
since each tested sample is not a member of the training set.
According to Witten and Frank, extensive tests on numerous
datasets have shown that 10 is a sufficient value for n in the n-
fold cross-validation [13]. Therefore, a 10-fold cross-validation
procedure (or leave k cases out, where k = 1/10 of the total sam-
ple) was performed according to a slightly modified description of
Masters including the following steps: (1) the in-sample data
were divided into 10 equally-sized independent subsamples, (2)
each NN model estimated 10 times, each time using a different

Table 1
Input variables and their descriptive statistics.

Variable No. Variable code Variable description Descriptive statistics

1 VACC The number of previous vaccinations 0 = 39.79%
0 = vaccinated for the first time 1 = 20.43%
previously vaccinated: 1 = once, 2 = two or three times 2 = 13.98%
3 = four or more times 3 = 25.81%

2 H1N1_1 Pre-existing antibody titre on the influenza virus A/H1N1 strain Mean = 11.08
stdev = 22.38

3 H3N2_1 Pre-existing antibody titre on the influenza virus A/H3N2 strain Mean = 69.68
stdev = 63.54

4 KONG_1 Pre-existing antibody titre on the influenza virus B/Hong Kong strain Mean = 43.44
stdev = 99.90

5 SICM_1 Pre-existing antibody titre on the influenza virus B/Sicuan strain Mean = 30.32
stdev = 44.64

6 GLU Fasting blood glucose Mean = 6.52
stdev = 2.10

7 SKINFOLD Triceps skinfold thickness (indicating malnutrition) Mean = 33.37
stdev = 7.38

8 AGE Age Mean = 67.66
stdev = 7.96

9 PSYCH Neuropsychiatric diseases (anxiety/depression, Parkinson’s disease,
cognitive impairments) (0 = no, 1 = yes)

0 = 40.86%
1 = 59.13%

10 HPA Helicobacter pylori specific antibodies type IgA (indicating chronic gastritis) Mean = 32.61
stdev = 51.39

11 HPG Helicobacter pylori specific antibodies type IgG (indicating chronic gastritis) Mean = 66.53
stdev = 63.42

12 EO Eosinophils % in White Blood Cell differential (indicating humoral immunity) Mean = 3.85
stdev = 2.65

13 MO Monocytes % in White Blood Cell differential (indicating immune cell activation) Mean = 8.10
stdev = 2.26

14 LY Lymphocytes % in White Blood Cell differential (indicating lymphopenia) Mean = 35.45
stdev = 8.99

15 MCV Mean Cell Volume (indicating vitamin B12 deficiency) Mean = 91.03
stdev = 5.03

16 ALB Serum albumin (indicating inflammation/malnutrition) Mean = 46.11
stdev = 3.13

17 CRCLEA Creatinine clearance(indicating chronic renal impairment) Mean = 1.69
stdev = 0.45

18 HOMCYS Amino acid homocysteine (indicating nutritional status/chronic renal impairment) Mean = 12.35
stdev = 3.81

19 BETA Beta-globulins in serum proteins electrophoresis (indicating low-grade chronic inflammation) Mean = 8.44
stdev = 0.94

20 GAMA Gamma-globulins in serum proteins electrophoresis (indicating low-grade
chronic inflammation/chronic humoral immune reaction)

Mean = 12.47
stdev = 2.29

21 VITB12 Vitamin B12 (indicating vitamin B12 deficiency/the nutritional status) Mean = 284.33
stdev = 158.79

22 PRL Hormone prolactin (indicating hyperprolactinemia) Mean = 124.57
stdev = 120.39

23 TSH TSH (thyroid-stimulating hormone) (indicating thyroid gland hormone hypofunction) mean = 2.04
stdev = 2.61

24 FT3 Free triiodothyronine (thyroid gland hormone)(indicating thyroid gland hormone hypofunction) Mean = 5.46
stdev = 0.53

25 FT4 Free thyroxine (thyroid gland hormone) (indicating thyroid gland hormone hypofunction) Mean = 14.01
stdev = 2.21

26 IGE Immunoglobulin type E (indicating impaired/decreased humoral immunity) Mean = 135.91
stdev = 245.59

27 Output Vaccine response (0 = negative, less then fourfold increase in antibody titre)
(1 = positive, fourfold and more increase in antibody titre)

0 = 32.26%
1 = 67.74%

776 L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781

set of 9 subsamples for training, and tested on 1 sample that was
left out of training, (3) 10 different results were obtained for each
model, (4) an average of 10 obtained results, i.e. average error
was computed [16]. The generalization ability in our study was
measured by the average error, and the model with the lowest
average error was selected as the best model.

The 10-fold CV procedure was performed on each of the three
NN models: MLP, RBFN and PNN, as well as on the LR model. The
size of the subsamples is presented in Table 2.

The purpose of the study was to design a computer-based neu-
ral network (NN) model that will enable successful prediction of
the outcome of influenza vaccine efficacy based on data related
to influenza viruses and influenza vaccination, in combination with
historical medical data. The creation of the NN and LR models in
this study was based on the immune response to the influenza

virus strain B whose content in the vaccine was recently changed
(new influenza virus vaccine strain) and on which a poor antibody
response is expected [23].

3. Methods

3.1. Neural network methodology

Three NN algorithms were tested: MLP, radial basis and proba-
bilistic. The output layer of all three NN models consisted of one
neuron (valued as 1 for the positive response, and 0 for the nega-
tive response). One hidden layer was used in all NN models in
our experiments. With regard to the number of hidden units, the
method of pruning was used which eliminates weights which are
lower than the threshold (0.05 in our experiments) input and

Table 2
Sampling in the 10-fold cross-validation procedure.

Sample Total

Number of patients Proportion (%)

Train 80 90
Test 10 10
Total 90 100

L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781 777

hidden units at the end of the training process in order to produce
smaller and faster networks with equivalent performance. The ini-
tial number of hidden units was set to 15 in MLP networks, and to
one-half of the size of the training sample in RBFN and to the size
of the training sample in the PNN. Overtraining was avoided by a
split-sample process which alternatively trains and tests the net-
work (using a separate test sample) until the performance of the
network on the test sample does not improve for n number of iter-
ations. The maximum number of training epochs was set to 600.
The generalization ability of all three NN models was determined
by a 10-fold cross-validation procedure described in Section 2.

The level of output sensitivity to each input variable in NN mod-
els was computed by sensitivity analysis. Sensitivity analysis stud-
ies how the variation in the output of a mathematical model can be
apportioned to different sources of variation in the input of a mod-
el. It is a technique that systematically changes model parameters
to determine the effects of such changes [25]. The basic principle is
that experimenting with a wide range of values gives insight into
the behavior of a system in extreme situations, and can lead to
identification of parameters whose specific value can significantly
influence the behavior of the model.

There are various approaches to sensitivity analysis. We used a
common OAT (one-factor-at-a-time) approach which changes the
values of one input variable to see what effect it produces on the out-
put, while all other variables are fixed to their central or baseline va-
lue [26]. The level of importance of each input variable is computed
by a sensitivity index, which represents the relative sensitivity of the
output to the changes of an input. Among the number of developed
sensitivity indices, we used the importance index which can be
interpreted to show the relative importance of an input variable to
the output. A higher value of importance index means higher sensi-
tivity of the output to changes of that particular input.

3.2. Logistic regression methodology

The aim of LR modeling in this study was to estimate the risk of
reaction to influenza vaccine and to extract variables which are
found to be important in risk prediction. The LR model used the
same initial set of input variables as the NN model (see Table 1)
with the forward selection procedure (selection criteria was
p < 0.05). At the output, a binary variable was used with one cate- gory representing a patient with a negative influenza vaccine out- come (0) and the other representing a patient with a positive influenza vaccine outcome (1). The SAS software was used to con- duct the procedure, with standard overall fit measures such as like- lihood ratio and score, as well as c statistics which measure discriminative power of logistic equation.

3.3. Evaluating model performance

The performance of the NN and LR models was measured by the
hit rate of negative vaccine outcome (i.e. the ‘‘negative hit rate” –
hit0), hit rate of positive vaccine outcome (i.e. the ‘‘positive hit rate”
– hit1), and the average hit rate (ave hit) computed by:

hit0 ¼
c0
t0
; hit1 ¼

c1
t1
; ave hit ¼

hit0 þ hit1
2

ð3Þ

where c0 is the number of patients accurately predicted to have
negative vaccine outcome (i.e. the number of true negatives), t0 is
the number of patients with actual (target) negative vaccine out-
come, c1 is the number of patients accurately predicted to have po-
sitive vaccine outcome (i.e. the number of true positives), and t1 is
the number of patients with actual positive vaccine outcome. The
above performance measures were computed on test samples for
all 10 NN and LR models. In order to test the generalization ability
of the models, the average hit rate of all ten samples was also com-
puted and used as the measure of model performance, as well as the
model selection criterion. The positive hit rate is equivalent to the
term of model sensitivity, while the negative rate is equivalent to
the model specificity, which is important when investigating the
ability of the model to accurately recognize positive and negative
outcomes. The computation of sensitivity and specificity also ex-
plains the proportion of false negatives or false positives that the
models produce. For this purpose, their sensitivity and specificity
ratios were computed in each of the 10 test samples used in the
10-fold cross-validation procedure, and type I and type II errors
were calculated. The sensitivity is computed according to:

sensitivity ¼
c1

ðc1 þ d0Þ
ð4Þ

where c1 is the number of true positives and d0 is the number of
false negatives [26]. The specificity is computed according to:

specificity ¼
c0

ðc0 þ d1Þ
ð5Þ

where c0 is the number of true negatives and d1 is the number of
false positives. The sensitivity is equivalent to the positive hit rate
hit0, while the specificity is equivalent to the negative hit rate hit1
(see Eq. (3)). The false positive rate (i.e. type I error) is computed
as a = 1 � specificity, while the false negative rate (i.e. type II error)
is computed as b = 1 � sensitivity. Comparison of false positive and
false negative rates explains the tendency of a model to misclassify
positive patients into the group of negatives, or negative patients
into the group of positive patients [26]. The model with a high sen-
sitivity could be used to screen for disease, since it has a tendency to
misclassify more negative patients into the group of positive
patients. The model with a high specificity could be used to confirm
the test results, since it is more specific in recognizing the actual
positive patients. The ideal situation would be that both the sensi-
tivity and specificity of a model have high values [26].

4. Results

Among the three tested NN algorithms (MLP, RBFN and PNN),
the best performance with regard to the average hit rate was ob-
tained by the MLP algorithm. The results of the best NN model
and LR models of the 10-fold cross-validation procedure are pre-
sented in Table 3.

It can be seen from Table 3 that the MLP hit rates across 10 sam-
ples in the 10-fold CV procedure varied, with the average hit rate of
72.52%. Following the modeling strategy described in Section 3, the
best model is the model with the highest average hit rate out of the
10 samples. Therefore, the procedure showed that the NN model
was more successful than the LR model. In order to analyze the sta-
tistical significance of the difference in the average hit results of
the two models, non-parametric Mann–Whitney test was con-
ducted since it is more appropriate in this case due to the small-
sample size [27]. The results of the test showed that the difference
between the average hit rates of the NN model and the LR model
was significant at the level 10% (p = 0.0587).

Due to the fact that the difference in performance (i.e. the aver-
age hit rate) was statistically significant, it can be concluded that
the NN model was more accurate in predicting vaccine outcome

Table 3
Neural network and logistic regression results of the 10-fold cross-validation procedure.

Test sample in CV
procedure

Results of NN model Results of LR model

Ave. hit
rate (%)

Positive (1) hit
rate (%)

Negative (0) hit
rate (%)

Ave. hit
rate (%)
Positive (1) hit
rate (%)
Negative (0) hit
rate (%)

LR model fitting measures

1 50.00 33.33 66.67 83.33 100.00 66.67 Likelihood ratio: 22.72, p = 0.0009, Score:
18.56, p = 0.005, c = 0.808

2 50.00 100.00 0.00 52.50 80.00 25.00 Likelihood ratio: 18.33, p = 0.0004, Score:
15.67, p = 0.0013, c = 0.782

3 28.57 57.14 0.00 42.50 60.00 25.00 Likelihood ratio: 36.72, p < 0.0001, Score: 28.93, p = 0.0001, c = 0.876

4 83.33 66.67 100.00 35.71 71.43 0.00 Likelihood ratio: 21.20, p = 0.0003, Score:
17.40, p = 0.0016, c = 0.797

5 83.33 100.00 66.67 65.00 80.00 50.00 Likelihood ratio: 36.83, p < .0001, Score: 28.94, p = 0.0001, c = 0.872

6 83.33 66.67 100.00 53.57 57.14 50.00 Likelihood ratio: 21.44, p < .0001, Score: 18.76, p = 0.0003, c = 0.791

7 91.67 100.00 83.33 60.71 71.43 50.00 Likelihood ratio: 19.34, p = 0.0002, Score:
16.56, p = 0.0009,c = 0.795

8 75.00 83.33 66.67 66.67 66.67 66.67 Likelihood ratio: 23.27, p = 0.0003, Score:
20.11, p = 0.0012, c = 0.799

9 90.00 100.00 80.00 58.33 50.00 66.67 Likelihood ratio: 28.31, p = 0.0002, Score:
22.42, p = 0.0021, c = 0.843

10 90.00 80.00 100.00 33.33 66.67 0.00 Likelihood ratio: 27.01, p = 0.0001,Score:
21.80, p = 0.0013,0.842

Ave. hit rate across
10 samples

72.52 78.71 66.33 55.17 70.33 40.00

778 L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781

than the LR model. When the hit rate of each output category in the
NN model is observed separately, it can be seen that the average
hit rate was higher for the positive group of patients (78.71%) than
for the negative group of patients (66.33%) indicating that this
algorithm recognizes positive patients more easily.

Table 4 presents the average sensitivity and specificity, as well
as type I and type II errors of the NN model, while these ratios ob-
tained by the LR model are presented in Table 5. It can be seen from
Table 4 that the specificity of the NN model is 0.79, while its sen-
sitivity is 0.66. The false positive rate (i.e. type I error) of the NN
model is 0.21, while the false negative rate (i.e. type II error) is
0.24. It reveals that the NN model is more sensitive than specific,
tending to misclassify more patients who actually had negative
vaccine outcome into the category of positive vaccine outcome.
However, the difference between type I and type II errors is small,
implying that the NN model is able to balance between sensitivity
and specificity.

Although the LR model is also more sensitive than specific (see
Table 5), both its sensitivity and specificity ratios are smaller than
the same ratios of the NN model. The LR model’s false positive rate
(0.30) is twice as small as its false negative rate (0.60).

In order to further investigate the importance of each input var-
iable to the output, sensitivity analysis was performed on each NN

Table 4
The average sensitivity and specificity of the NN model.

Vaccine outcome predicted by the NN models Actual vaccine outcome

1 (positive) 0 (negative)

1 (positive) 0.79 0.24
0 (negative) 0.21 0.66

Table 5
The average sensitivity and specificity of the LR model.

Vaccine outcome predicted by the NN models Actual vaccine outcome
1 (positive) 0 (negative)

1 (positive) 0.70 0.60
0 (negative) 0.30 0.40

model in the 10-fold cross-validation procedure, and the average
sensitivity ratio of the influenza vaccine outcome to input variables
is presented in Fig. 1.

The analysis shows that the VACC is the variable with the high-
est influence on the vaccine outcome, closely followed by the HPG.
The average sensitivity ratio is also high for variables PSYCH, BETA,
HPA, EO, VITB12 and CRCLEA (sensitivity ratio higher than 1.1),
while the ratio of other input variables is around 1.0. The variables
with the lowest ratios (less than 1.0) are: AGE, H1N1_1, LY and
HOMCYS.

In the process of LR modeling, a forward selection procedure
was used to select input variables important for the model. By
using such selection criteria, the LR extracted a total set of nine in-
put variables: EO, VITB12, GLU, AGE, VACC, HPG, BETA, LY and IGE.
Although the procedure of selecting input variables differs from
the one used in NN modeling, some of the highly ranked variables
identified by the NNs were also extracted by the LR models. If the
rank of input variables is compared across NN and LR models, it can
be seen that almost all variables extracted by the LR also had very
high importance for the NNs (such as VACC, HPG, EO, VITB12, BETA,
GLU), while some of the variables (such as AGE, IGE, and LY) were
found to be significant for the LR model, although not very impor-
tant for the NN model.

5. Discussion

The results clearly indicate that both types of data, those related
to previous influenza viruses exposure and those describing the
health status of examinees, influence outcome values of performed
predictive models and could be used as efficient predictors of the
influenza vaccine efficacy (Fig. 1). The reason for the low influence
of the variable AGE in the NN models could be found in the sample
structure, since most of the patients in the observed sample were
elderly (mean age 67.67 years) (Table 1, Fig. 1). Furthermore, the
variable AGE can hardly be separated from the factors related to
previous antigenic exposure, and from factors related to chronic
health disorders, both known to be age-dependent.

With regard to the influenza vaccine component, the content of
which was recently changed (new influenza virus strain), such as

0
0.2
0.4
0.6
0.8
1

1.2
1.4
1.6
1.8
2

V
A
C
C

H
P
G

P
S
Y
C
H

B
E
T
A

H
P
A

E
O

V
IT
B
12

C
R
C
LE
A

P
R
L

G
LU

A
LB M
O

IG
E

F
T
3

S
K
IN
F
O
LD F
T
4

M
C
V

G
A
M
A

K
O
N
G
_1

T
S
H

S
IC
M
_1

H
3N
2_
1

A
G
E

H
1N
1_
1

LY

H
O
M
C
Y
S

Input variable

A
ve

. s
en

si
tiv

ity
r

at
io

Ave.ratio

Fig. 1. Sensitivity ratios of the output variable towards the input variables.

L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781 779

the component B/Hong Kong, the most important information
could be the number of previous vaccinations (variable VACC)
(Fig. 1). In this connection, we analyzed the postvaccination anti-
body titres in relation to the number of previous vaccinations for
all three influenza virus strains contained in the vaccine prepara-
tion used in the study (A/H1N1, A/H3N2 and B/Hong Kong). Indeed,
a statistically significant decrease was only found for the compo-
nent B/Hong Kong and only for individuals frequently vaccinated
(previously vaccinated four or more times) (The results are not pre-
sented in the study). This may be a consequence of the negative
impact of heterologous immune reaction (to influenza B virus
strains from vaccine preparations used for vaccination in the re-
cent past, before the last vaccine change) [23].

The results also show that for building a model of prediction the
health status of examinees must be described with regard to many
aspects (Fig. 1). Hematological and biochemical laboratory tests are
preferable, in comparison to clinical diagnoses of diseases (only
one diagnosis, that of neuropsychiatric disease, was selected as
being relevant for prediction of low antibody response to influenza
vaccination) (Table 3 and Fig. 1).

Some of the relevant variables, those with the highest influence
on the vaccine outcome, are likely to indicate chronic clinical con-
ditions which, in elderly people, could affect the antibody immune
response. These variables are: (1) PSYCH, indicating neuropsychi-
atric diseases, (2) HPA (and HPG), indicating Helicobacter pylori po-
sitive chronic gastritis, and (3) CRLEA, indicating chronic renal
impairment. Other variables, shown in this study to be necessary
for the modeling, may represent common intermediate mecha-
nisms, associated with these main clinical conditions (Fig. 1).

In accordance with these explanations, low influenza vaccina-
tion antibody response was found in the elderly with dementia/
depression [28]. Evidence suggests that, in elderly people, both dis-
orders, anxiety/depression and neurodegenerative/cognitive
impairments, are closely related and frequently occur together
[29]. Mechanisms such as neuroendocrine disorders, including also
hyperprolactinemia and thyroid gland hormone hypofunction (in
our results indicated by the variables PRL, FT3, FT4 and TSH),
may link these disorders with impaired immune response to influ-
enza vaccination [30–32]. Another proposed mechanism may be
chronic activation of immunocomponent cells, during the course
of neurodegenerative processes [33]. This can lead to a deficiency
of vitamin B12 and, in turn, to impaired immune reaction, because
this vitamin is necessary for proliferation of immunocompetent
cells [34,35]. These assumptions, in our results, are likely to be rep-
resented by the variables: VITB12, MCV, MO and LY (Table 1,
Fig. 1). The above explanation, based on existing knowledge on
the issue, may be the reason for differences in variable selection,
between the two used methods of prediction: the LR and the NN.
Namely, because of the mutual interdependence between variables

VITB12 and LY, they could gain different statistical power in the
two models, based on the different performance due to linearity
and dependence among the data.

Chronic gastritis is a frequent disorder in older people. In the
majority of cases, it is caused by H. pylori infection. This disorder
can be described as chronic inflammatory reaction in the gastric
submucosa. Chronic inflammation and mechanisms such as in-
creased oxidative stress and cytokine production may contribute
to enhanced lymphocyte apoptosis and lymphopenia and, conse-
quently, to impaired immune reaction [36]. In addition, because
of intensive activation of B-lymphocytes in gastric submucosa,
the disorder may lead to impairment of the specific humoral im-
mune response [37]. In our results all these mechanisms, linking
chronic gastritis with insufficient antibody production to influenza
vaccination, are likely to be implicated by variables such as: BETA,
EO, ALB, MO, IGE, GAMA and LY (Table 1 and Fig. 1). Close func-
tional association between these variables may be the reason for
complementariness among some of them, such as, e.g., the paired
variables EO and IGE, indicating the same pathogenetic disorder
– impaired humoral immunity. This may also provide a reasonable
explanation for differences in selection of these variables in two
different models of prediction: the LR and the NN.

Some other mechanisms can be proposed, linking chronic gas-
tritis with low antibody response to influenza vaccination. Such
mechanisms include vitamin B12 deficiency, probably due to mal-
absorption, and thyroid gland hormone hypofunction, due to evi-
dence showing that chronic gastritis and chronic autoimmune
thyroiditis, both common aging diseases, usually appear in co-
morbidity [38,39].

In general, B-vitamin deficiency and mild hyperhomocystein-
emia, closely related metabolic disorders, may be markers of im-
paired cell cycling [40]. This may have the greatest negative
impact on the function of cells with a high cell turn-over, such as
immunocomponent cells [35,41].

Chronic renal impairment, a clinical condition in our results
negatively influencing antibody response to influenza vaccination,
has frequently been found to be associated with numerous disor-
ders. Many of them are intermediate mechanisms, already men-
tioned above, including: inflammation, lymphopenia, increased
mononuclear leukocyte activity, B-vitamin deficiency, hyperhomo-
cysteinemia, as well as multiple neuroendocrine disorders [42–46].
Low-grade inflammation, common in patients with decreased re-
nal function, is also associated with strong protein malnutrition,
which in our results is most likely indicated by variables ALB and
SKINFOLD (Table 1 and Fig. 1). Inflammation and protein malnutri-
tion are mechanisms which both may affect the immune system
function [47]. Moreover, metabolic disorders, including hyperhom-
ocysteinemia and impaired glucose metabolism, are usually associ-
ated with inflammation/malnutrition, forming a unique syndrome

780 L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781

[43]. This fact, in our results, is likely to be supported by the vari-
able GLU, indicating fasting blood glucose concentration (Table 1
and Fig. 1).

Hence, for influenza vaccine strains which are frequently chan-
ged (such as the B/Hong Kong vaccine strain), the poorest antibody
response after vaccination can be expected in individuals previ-
ously vaccinated several times and who, in addition, are burdened
by health disorders, with great impact on the immune reaction.

6. Conclusion

The aim of the study was to design an intelligent computer-
based neural network model that will enable successful prediction
of the outcome of influenza vaccine efficacy. The model is based on
the results of vaccination by the influenza vaccine strain, the con-
tent of which was recently changed (in this case the B/Hong Kong
vaccine strain) and on which, therefore, a poor antibody response
was expected. The results were compared with a standard logistic
regression approach. The input space consisted of 26 input vari-
ables, comprising both variables related to previous influenza virus
exposure and previous vaccinations and variables describing many
aspects of the health status of a group of high-risk older patients
vaccinated against influenza.

Multilayer perceptron, radial-basis function, probabilistic neural
network and logistic regression were used to predict the outcome of
influenza vaccination, based on a set of previously selected input
variables. Due to the small-sample size, it was necessary to perform
a 10-fold cross-validation procedure in order to estimate the gener-
alization ability of the model. The procedure showed that the mul-
tilayer perceptron algorithm had the highest average performance
obtained on 10 samples, and therefore can be proposed as the mod-
el that generalizes better. The sensitivity and specificity ratios of the
NN model were also higher than those ratios of the LR model. The
neural network model was also able to balance between the false
positive and false negative hit rates, showing that it was able to
determine important features necessary to correctly classify
patients with negative vaccine outcome and those with positive
vaccine outcome.

The sensitivity analysis showed that the following predictor
variables have the greatest importance for the output: VACC (the
number of previous vaccinations), HPG (H. pylori specific antibodies
type IgG), PSYCH (neuropsychiatric disease), BETA (beta-globulins
in serum proteins electrophoresis), HPA (H. pylori specific antibod-
ies type IgA), EO (eosinophils % in White Blood Cell differential),
VITB12 (vitamin B12), and CRCLEA (creatinine clearance). Although
the logistic regression approach extracted a more narrow set of
predictors, it produced a significantly lower average hit rate com-
pared to the neural network approach, implying that the multi-
layer structure of neural networks is more capable of
understanding the interconnections among input variables and
the output in order to successfully predict the vaccine output. Re-
sults obtained by both methods clearly indicate that the health sta-
tus of the examined patients, for the purpose of prediction, must be
determined by many aspects, and by a sufficiently large set of vari-
ables. This is due to the nature of chronic health disorders, charac-
terized by many factors, each having only small predictive power.

Since this is a preliminary study, improvement in model perfor-
mance can be expected by increasing the number of patients in-
cluded in the dataset, and by using other intelligent methods,
such as genetic algorithms, support vector machines, and other
classifiers which could combine with neural networks in order to
build a more successful model. The potential of the methodology
in this field is evident, and benefit from this research in primary
health care can be anticipated, as well as in more global strategic
planning of influenza vaccination.

References

[1] Keitel WA, Cate TR, Couch RB. Efficacy of sequential annual vaccination with
inactivated influenza virus vaccine. Am J Epidemiol 1988;127:353–64.

[2] Gross PA, Hermogenes AW, Sacks HS, Lau J, Levandowski RA. The efficacy of
influenza vaccine in elderly persons. A meta-analysis and review of the
literature. Ann Intern Med 1995;123:518–27.

[3] Goodwin K, Viboud C, Simonsen L. Antibody response to influenza vaccination
in the elderly: a quantitative review. Vaccine 2006;24(8):1159–69.

[4] Remarque EJ. Influenza vaccination in elderly people. Exp Gerontol
1999;34(3):445–52.

[5] Beyer WE, Palache AM, Sprenger MJ, Hendriksen E, Tukker JJ, Darioli R, et al.
Effects of repeated annual influenza vaccination on vaccine sero-response in
young and elderly adults. Vaccine 1996;14(14):1331–9.

[6] Tosh PK, Poland GA. Emerging vaccines for influenza. Expert Opin Emerg Drugs
2008;13(1):21–40.

[7] Tripp RA, Tompkins SM. Recombinant vaccines for influenza virus. Curr Opin
Investig Drugs 2008;9(8):836–45.

[8] Kim SY, Goldie SJ. Cost-effectiveness analyses of vaccination programmes: a
focused review of modelling approaches. Pharmacoeconomics
2008;26(3):191–215.

[9] Cruz JA, Wishart DS. Application of Machine Learning in cancer prediction and
prognosis. Review. Cancer Informatics 2006;2:59–78.

[10] Buchanan AV, Weiss KM, Fullerton SM. Dissecting complex disease: the quest
for the philosopher’s stone? Int J Epidemiol 2006;35:562–71.

[11] Linder R, Konig IR, Weimar C, Diener HC, Poppl SJ, Ziegler A. Two models for
outcome prediction – a comparison of logistic regression and neural networks.
Methods Inf Med 2006;45(5):536–40.

[12] Soucek B, the IRIS Group. Neural and intelligent systems integration. New
York: John Wiley & Sons; 1991.

[13] Witten H, Frank E. Data mining, practical machine learning tools and
techniques with java implementations. San Francisco: Morgan Kaufmann
Publishers; 2000.

[14] Basheer IA, Hajmeer M. Artificial neural networks: fundamentals, computing,
design, and application. J Microbiol Methods 2000;43:3–31.

[15] Lisboa PJ, Taktak AFG. The use of artificial neural networks in decision support
in cancer: a systematic review. Neural Netw 2006;19:408–15.

[16] Masters T. Advanced algorithms for neural networks, A C++ sourcebook. New
York: John Wiley & Sons; 1995.

[17] Karayiannis NB, Weigun GM. Growing radial basis neural networks: merging
supervised and unsupervised learning with network growth techniques. IEEE
Trans Neural Netw 1997;8:1492–505.

[18] Patterson DW. Artificial neural networks. New York: Prentice Hall; 1995.
[19] Harrel Jr FE. Regression modeling strategies with applications to linear models,

logistic regression and survival analysis. Berlin: Springer; 2001.
[20] Majnaric TLJ, Vitale B, Martinis M. Non-linear approach in establishing

influenza vaccine priority groups. Elsevier editorial system for vaccine™,
manuscript draft, available from http://ees.elsevier.com/jvac/default.asp.

[21] Liu Y. Unbiased estimate of generalization error and model selection in neural
network. Neural Netw 1995;8:215–9.

[22] Tourassi GD, Floyd CE. The effect of data sampling on the performance
evaluation of artificial neural networks in medical diagnosis. Med Decis
Making 1997;17:186–92.

[23] Hilden J. Neural networks and the roles of cross validation. Med Decis Making
1998;18:122–3.

[25] Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, et al. Global
sensitivity analysis. The Primer, Chicester: John Wiley & Sons; 2008.

[26] Simon D, Boring III JR. Sensitivity, specificity, and predictive value. In: Walker
HK, Hall WD, Hurst JW, editors. Clinical methods: the history, physical, and
laboratory examinations. 3rd ed. Stoneham, MA: Butterworths; 1990. p. 49–54.

[27] Sheskin DJ. Handbook of parametric and nonparametric statistical
procedures. Washington, DC: CRC Press; 1997.

[28] Vedhara K, Cox NK, Wilcock GK, Perks P, Hunt M, Anderson S. Chronic stress in
elderly carers of dementia patients and antibody response to influenza
vaccination. J Psychosom Res 2003;55(6):569–72.

[29] Magri F, Cravello L, Barili L, Sarra S, Cinchetti N, Salmoiraghi F, et al. Stress and
dementia: the role of the hypothalamopituitaryadrenal axis. Aging Clin Exp
Res 2006;18(2):167–70.

[30] Carta MG, Loviselli A, Hardoy MC, Massa S, Cadeddu M, Sardu C, et al. The link
between thyroid autoimmunity (antithyroid peroxidase autoantibodies) with
anxiety and mood disorders in the community: a field of interest for public
health in the future. BMC Psychiatry 2004;18:4–25.

[31] Sonino N, Navarrini C, Ruini C, Fallo F, Boscaro M, Fava GA. Life events in the
pathogenesis of hyperprolactinemia. Eur J Endocrinol 2004;151(1):61–5.

[32] Dorshkind K, Horseman ND. The roles of prolactin, growth hormone, insulin-
like growth factor-I, and thyroid hormones in lymphocyte development and
function: insights from genetic models of hormone and hormone receptor
deficiency. Endocr Rev 2000;21(3):292–312.

[33] Widner B, Leblhuber F, Frick B, Laich A, Artner-Dworzak E, Fuchs D. Moderate
hyperhomocysteinemia and immune activation in Parkinson’s disease. J
Neural Transm 2002;109(12):1445–52.

[34] Scott TM, Tucker KL, Bhadelia A, Benjamin B, Patz S, Bhadelia R, et al.
Homocysteine and B vitamins relate to brain volume and white-matter
changes in geriatric patients with psychiatric disorders. Am J Geriatr
Psychiatry 2004;12(6):631–8.

http://ees.elsevier.com/jvac/default.asp

L. Trtica-Majnaric et al. / Journal of Biomedical Informatics 43 (2010) 774–781 781

[35] Schroecksnadel K, Frick B, Wirleitner B, Winkler C, Schennach H, Fuchs D.
Moderate hyperhomocysteinemia and immune activation. Curr Pharm
Biotechnol 2004;5:107–18.

[36] Peek RM, Miller GG, Tham Kyi T, Perez-Perez GI, Zhao X, Atherton JC.
Heightened inflammatory response and cytokine expression in vivo to cagA+

Helicobacter pylori strains. Lab Invest 1995;71(12):760–70.
[37] Hussell T, Isaacson PG, Crabtree JE, Dogan A, Spencer J. Immunoglobulin

specificity of low grade B cell gastrointestinal lymphoma of mucosa-associated
lymphoid tissue (MALT) type. Am J Pathol 1993;142(1):285–92.

[38] Sipponen P, Laxen F, Huotari K, Harkonen M. Prevalence of low vitamin B12 and
high homocysteine in serum in an elderly male population: association with
atrophic gastritis and Helicobacter pylori infection. Scand J Gastroenterol
2003;12:1209–16.

[39] Centanni M, Marignani M, Gargano L, Corleto VD, Casini A, Delle Fave G, et al.
Atrophic body gastritis in patients with autoimmune thyroid disease: an
underdiagnosed association. Arch Intern Med 1999;159(15):1726–30.

[40] Jaenisch R, Bird A. Epigenetic regulation of gene expression: how the genome
integrates intrinsic and environmental signals. Nat Genet Suppl 2003;33:
245–54.

[41] Zhang C, Cai Y, Adachi MT, Oshiro S, Aso T, Kaufman RJ, et al. Homocysteine
induces programmed cell death in human vascular endothelial cells

through activation of the unfolded protein response. J Biol Chem 2001;276:
35867–74.

[42] van Guldener C, Robinson K. Homocysteine and renal disease. Semin Thromb
Hemost 2000;26(3):313–24.

[43] Pecoits-Filho R, Lindholm B, Stenvinkel P. The malnutrition, inflammation and
atherosclerosis (MIA) syndrome – the heart of the matter. Nephrol Dial
Transplant 2002;17(Suppl. 11):28–31.

[44] Luger A, Lang I, Kovarik J, Stummvoll HK, Templ H. Abnormalities in the
hypothalamic–pituitary–adrenocortical axis in patients with chronic renal
failure. Am J Kidney Dis 1987;9(1):51–4.

[45] Kutlay S, Atli T, Koseogullari O, Nergizoglu G, Duman N, Gullu S. Thyroid
disorders in hemodialysis patients in an iodine-deficient community. Artif
Organs 2005;29(4):329–32.

[46] Fernandez-Fresnedo G, Ramos MA, Gonzalez-Pardo MC, de Francisco AL,
Lopez-Hoyos M, Arias M. B lymphopenia in uremia is related to an accelerated
in vitro apoptosis and dysregulation of Bcl-2. Nephrol Dial Transplant
2000;15(4):502–10.

[47] Hara M, Tanaka K, Hirota Y. Immune response to influenza vaccine in healthy
adults and the elderly: association with nutritional status. Vaccine
2005;23(12):1457–63.

• Prediction of influenza vaccination outcome by neural networks and logistic regression

Introduction
Materials
Methods
Neural network methodology
Logistic regression methodology
Evaluating model performance
Results
Discussion
Conclusion
References

Nearest-Neighbor and Logistic Regression
Analyses of Clinical and Heart Rate

Characteristics in the Early Diagnosis
of Neonatal Sepsis

Yuping Xiao, MS, M. Pamela Griffin, MD, Douglas E. Lake, PhD, J. Randall Moorman, MD

Objectives. To test the hypothesis that nearest-neighbor
analysis adds to logistic regression in the early diagnosis of
late-onset neonatal sepsis. Design. The authors tested meth-
ods to make the early diagnosis of neonatal sepsis using con-
tinuous physiological monitoring of heart rate characteristics
and intermittent measurements of laboratory values. First, the
hypothesis that nearest-neighbor analysis makes reasonable
predictions about neonatal sepsis with performance compara-
ble to an existing logistic regression model was tested. The
most parsimonious model was systematically developed by
excluding the least efficacious clinical data. Second, the
authors tested the hypothesis that a combined nearest-neigh-
bor and logistic regression model gives an outcome prediction
that is more plausible than either model alone. Training and
test data sets of heart rate characteristics and laboratory tes

t

results over a 4-y period were used to create and test

predictive models. Measurements. Nearest-neighbor, regres-
sion, and combination models were evaluated for discrimina-
tion using receiver-operating characteristic areas and for fit
using the Wald statistic. Results. Both nearest-neighbor and
regression models using heart rate characteristics and avail-
able laboratory test results were significantly associated with
imminent sepsis, and each kind of model added independent
information to the other. The best predictive strategy
employed both kinds of models. Conclusion. The authors
propose nearest-neighbor analysis in addition to regression in
the early diagnosis of subacute, potentially catastrophic ill-
nesses such as neonatal sepsis, and they recommend it as an
approach to the general problem of predicting a clinical event
from a multivariable data set. Key words: nearest-neighbor;
logistic regression; heart rate characteristics; sepsis. (Med
Decis Making 2010;30:258–266)

The early diagnosis of sepsis in the neonatalintensive care unit should be an excellent appli-
cation for data-mining approaches. First, we wished

to tested the hypothesis that nearest-neighbor analy-
ses make predictions on neonatal sepsis that are com-
parable to the existing regression models that relate
heart rate characteristic (HRC) index and laboratory
tests to neonatal sepsis. Second, we wished to test
the hypothesis that the combined model of nearest-
neighbor and regression, not necessarily using the
same clinical data, is more accurate than either
model by itself.

BACKGROUND

Neonatal sepsis is a major cause of morbidity and
mortality in premature infants hospitalized in neo-
natal intensive care units (NICUs).1 It is difficult to
diagnose in its earliest and most treatable stages
because clinical signs and laboratory test abnormali-
ties are subtle and nonspecific,2,3 and thus, it com-
monly presents in advanced stages as systemic
inflammatory response syndrome.4,5 We regard

Received 1 December 2005 from the Departments of Internal Medicine
(YX, DEL, JRM) and Statistics (DEL) and Pediatrics (MPG) and the Car-
diovascular Research Center (JRM), University of Virginia Health System,
Charlottesville. We thank W. E. King for suggesting nearest-neighbor
analysis to us. Supported by NIGMS-64640; American Heart Associa-
tion, Mid-Atlantic Affiliate; Children’s Medical Center Research Fund,
University of Virginia; Virginia’s Center for Innovative Technology; and
Medical Predictive Science Corporation, Charlottesville, Virginia. Pre-
sented in part at the Pediatric Academic Societies annual meeting, May
2005. Medical Predictive Science Corporation of Charlottesville, Virginia,
has a license to market technology related to heart rate characteristics
monitoring of newborn infants and supplied partial funding for this study.
Drs. Griffin, Lake, and Moorman have an equity share in this company.
Revision accepted for publication 15 February 2007.

Address correspondence to Douglas E. Lake, PhD, Box 800158,
UVAHS, Charlottesville, VA 22908; e-mail: dlake@virginia.edu.

DOI: 10.1177/0272989X0933779

1

258 • MEDICAL DECISION MAKING/MAR–APR 20

10

CLINICAL PREDICTION MODELS

neonatal sepsis as an example of many subacute ill-
nesses with subclinical phases during which treat-
ment should be highly effective in preventing
potential catastrophe.

To aid in the early diagnosis of neonatal sepsis,
we developed HRC monitoring6�9 based on the
observation that reduced variability and transient
decelerations of heart rate occur in the hours to days
prior to the clinical presentation.10 These abnormal-
ities can be quantified using novel time-series
measures11�13 incorporated into a predictive model
based on logistic regression. The resulting HRC
index has a highly significant association with sep-
sis in internal and external validation studies.

We know, though, that physicians rely primarily
on experience and not regression equations to diag-
nose illness and employ pattern recognition to distill
complex presenting features into a short list of possi-
ble diseases. This invaluable exercise is not usually
formalized, and most diagnostic test results arrive
independently and without contextual interpreta-
tion. Thus, the common clinical discourse of ‘‘when-
ever I see x and y, I think of z’’ is not codified for
universal use. Formal approaches to statistical pat-
tern recognition have been developed in other
fields,14 and data mining is a recently coined term for
these strategies, which often exploit computer pro-
cessing of large databases. Moreover, in general, to
achieve optimal prediction of the outcome, we need
to know the probability distribution of all the vari-
ables. As this is impossible in many practical pro-
blems, decision rules that do not need knowledge of
distributions are favored. Nearest-neighbor analysis,
described below, is one such strategy. It takes advan-
tage of a large number of known samples in the train-
ing set to compensate for the lack of distributions,
and it is reliable for a mixture of continuous and dis-
crete variables.15 Specific to clinical medicine is the
idea that a single abnormal finding could supersede
many normal findings in diagnosing or predicting ill-
ness, and all clinicians recall patients in whom dire
diagnoses were made based on a single crucial abnor-
mal finding in a sea of normal ones.

The current study was motivated by the recent
finding that laboratory test results add independent
information to HRC monitoring in the diagnosis of
neonatal sepsis.16 Our analysis has been based on
regression, which recognizes only monotonic rela-
tionships between laboratory values and risk. This
reasoning fails for tests such as white blood cell
count (WBC) or body temperature, which might be
abnormally high or low in illness. Thus, for the new
study, we have used a pattern-recognition technique

called nearest-neighbor analysis.14 The principle is
simple. For a patient with a set of findings, one finds
the most similar infants in their experience and lists
their diagnoses and outcomes. Nearest-neighbor
analysis has been widely used in pattern recognition
studies of many kinds but has been relatively under-
used in clinical medicine. Haddad and coworkers1

7

used this approach to detect coronary artery disease
using patterns of perfusion scintigraphy in 100
patients in whom the presence or absence of disease
was established by angiography.17 Qu and Gotman18

developed a patient-specific seizure detection algo-
rithm based on electroencephalogram waveforms in
the presence and absence of seizures.18 Most
recently, Lutz and coworkers19 were able to forecast
effects of psychotherapy based on reference
responses of 203 clients.19

A particular strength of nearest-neighbor analysis
is independence from assumptions about normal
levels of test results or about relationships among
test results. The results arise entirely from experi-
ence, mimicking at least part of a physician’s
thought process. Because sepsis elicits a complex
systemic inflammatory response syndrome with
dysfunction of multiple organs, it seems sensible to
consider as many simultaneous processes as possi-
ble. On the other hand, however, some laboratory
tests are taken much less frequently than the others,
making the database smaller the more processes we
take into consideration at the same time. Last but
not least, although many variables contribute to
a model, some may be more closely related to the
model outcome than the others, and before the roles
of all variables are understood fully, including as
many variables as possible in 1 model may turn out
to be impractical and potentially problematic. Thus,
we need a model, or a combination of multiple mod-
els, that can include many important variables and
also deal with the real-world problem of intermit-
tent sampling of laboratory measures.

Research Question

Does nearest-neighbor analysis add to existing
logistic regression methods for early diagnosis of
neonatal sepsis?

METHODS

Study Population

We studied all admissions to the University of
Virginia NICU from July 1999 to July 2003 of

CLINICAL PREDICTION MODELS 259

INFORMATICS MONITORING

patients who were 7 or more days of age. The clini-
cal research protocol was approved by the Human
Investigations Committee of the University of Virgi-
nia. Laboratory test results were available from an
electronic archive. The analysis was limited to the
time that HRC data were available, or 92% of the
total time. Infants were followed prospectively to
identify cases of sepsis, but health care personnel
were not aware of the result of the HRC monitoring.
We defined sepsis to be present when a physician
suspected the diagnosis, obtained a blood culture
that grew bacteria not ordinarily considered to be
a contaminant, and initiated antibiotic therapy of

5

or more days’ duration. This definition is consistent
with the diagnosis of proven sepsis used by the
Centers for Disease Control.20 Sepsis was diagnosed
in the usual course of care; that is, no cultures were
done for study purposes.

Data Sets

Both the nearest-neighbor and logistic regres-
sion analyses call for a training data set and a test
data set. Each point represents a summary of the
past 12 h, and points were calculated every 6 h.
The HRC index was available at each point, and
laboratory test results were intermittently avail-
able. Thus, each data point in the test set could be
evaluated for HRC values, and many could be eval-
uated for 1 or more individual laboratory test
results when they were available. In our analysis,
the duration of a test result was 12 h. For this
work, HRC and laboratory test data obtained from
1999 to 2003 in the University of Virginia NICU
were split randomly and nearly evenly into a train-
ing set of patients and a test set of patients. A total
of 676 patients (with more than 70,000 records)
were included in this study, and among them, 32

6

patients were in the training set. All were obtained
after 7 days of age, and we neglected data during
the 7 days after a positive blood culture. The sep-
sis event was defined to occur over a 24-h period
beginning 6 h before the positive blood culture
and ending 18 h after. We justify this selection
based on our prior work using regression modeling
that shows this is the epoch in which most of the
diagnostic test results are available.8,16 Birth
weight and days of age were included in all of the
models, as we reasoned that clinicians were
always aware of these parameters. Cases of sepsis
were individually reviewed for data accuracy.
Health care personnel were blinded to the results
of the HRC monitoring.

Nearest-Neighbor Models

Conceptually, each point from the test set was
placed among the points of the training set in a mul-
tidimensional space, and its nearest neighbors were
identified. For each of the 36,000 points in the train-
ing set, the distance from each point in the 35,000-
point test set was calculated. The distances were
calculated for HRC values, days of age, and birth
weight for all of the data, and the process was
repeated with additional contributions from each
laboratory test individually and in several combina-
tions. Self-records were excluded.

A very important aspect of implementing accu-
rate nearest-neighbor models is the selection of the
distance metric used to measure similarity. The
Mahalanobis distance is a robust choice for measur-
ing similarity.15 The Mahalanobis distance between
1 records xi, xj is calculated by

di, j = A xi � xj
� ��� ��,

where xi = ½xi1, xi2, � � � , xin�T is a vector of available
clinical data features of a record and A is an n×n
matrix that transforms each feature vector so that
xikðk ¼ 1, 2, � � � , nÞ have the same scale and are
uncorrelated. The transformation matrix A can be
estimated from the training data set by

A=VS−12,

where S is a diagonal matrix consisting of eigen-
values of the covariance matrix of the data set and V
is the orthogonal matrix of corresponding eigenvec-
tors. The important results of this transformation are
that variables with large numerical values (such as
WBC) do not dwarf those with small values (such as
the ratio of immature to total white blood cell forms
[I:T ratio]), and variables that are highly correlated
(such as pCO2; pH, and HCO3Þ are uncoupled in the
analysis. Mahalanobis distance can be viewed as
a simple Euclidean distance following that linear
transformation.

For a new test record xi, distances from all
records in the training data set to it are sorted in
ascending order. The new record is assigned a value
between 0 and 1 postulated to represent the chance
of illness in the next 24 h. If there are ks records that
occurred within 24 h of sepsis among the ki nearest
neighbors in the training set, then the probability of
the new record also within 24 h of illness is given by

pi = ks
ki
:

260 • MEDICAL DECISION MAKING/MAR–APR 2010

XIAO AND OTHERS

For this work, ks was selected to be 10 to ensure a rea-
sonable degree of statistical accuracy of this probabil-
ity. Results were similar for values between 5 and 10.
There were 220 records near sepsis from 88 episodes
of sepsis in the training set. Were they randomly dis-
tributed, the size of the neighborhood would be 10/
220, or less than 5% of the data set. Thus, the strategy
allows effective localization of infants with similar
clinical features within the large data set.

Each combination of HRC and laboratory values
can be thought of as a predictive model and evalu-
ated using standard metrics such as ROC area and
Wald statistic.

Logistic Regression Models

We used previously described techniques and
adjusted for repeated measures.6�8,16,21

Combining Nearest-Neighbor and Logistic
Regression Models

Individual nearest-neighbor models with HRC and
laboratory values were combined so that if 1 or more
laboratory values were available, the maximum of the
predictions was made in the final prediction of the
combined model. If no laboratory values were avail-
able, the final prediction adopted the outcome of the
single model with only HRC and other basic variables
(such as birth weight and days of age).

Models combining probability estimates from the
nearest-neighbor and logistic regression analyses
were based on bivariable logistic regression. The sta-
tistical significance of added information was calcu-
lated using the Wald chi-square test.

HRC Index and Laboratory Results

The HRC index has been previously described.6,8

Briefly, it is an internally and externally validated mul-
tivariable regression–based measure that is propor-
tional to the risk of acute illness in infants in the
NICU. In this analysis, we used individual measure-
ments of the standard deviation, sample entropy,12,13,2

2

and sample asymmetry (R1 and R2 measures),11 the
major elements of the HRC index. Laboratory test
results were obtained from an electronic archive.

Study Design

The 1st goal was to test the hypothesis that near-
est-neighbor analyses alone make effective predic-
tions on neonatal sepsis. All models included days

of age and birth weight, as this information is always
available to the clinician. We systematically tested
nearest-neighbor models that added 1 laboratory test
at a time and all combinations of 2 laboratory results.
We then added 4-dimensional HRC data (s, 2-sample
asymmetry statistics, and sample entropy—the com-
ponents of the regression-based HRC index) and
repeated the procedure. Receiver-operating charac-
teristic area (ROC) and Wald statistic were used to
measure model performance.

The 2nd goal was to test the hypothesis that
nearest-neighbor and logistic regression models
add independent information to each other and
that combined models improve prediction of neo-
natal sepsis. The approach was to prepare multiple
nearest-neighbor models and regression models, eval-
uate the predictive performances of each model, and
compare them.

RESULTS

Table 1 shows the demographic features and rates
of sepsis in the training and overall data sets. Figure
1 shows the method for calculating the proportion of
sick neighbors. As an example, a 3-dimensional
space is shown, with each axis representing a mea-
surement modality—here, birth weight, day of age,
and WBC. The complete analysis included higher
dimensions. The filled points are measurements
from past infants who were within 24 h of the diag-
nosis of sepsis. Two populations can be identified

Table 1 Patient Population and Laboratory
Test Results

Training Set All Infants

n 327 676
#HRC 36,309 71,25

4

Birth weight (g)a 1454 (973,2620) 1581 (974,2700)
<1500 g 166 (51%) 317 (47%) Gestational age (wk)a 31 (27, 36) 31 (27, 36) Episodes of sepsis 88 (67 infants) 163 (120) Laboratory test results

White blood
cell count

6723 13,014

I:T ratio 5616 10,887
Glucose 24,592 48,312
Platelet count 8275 16,144
pH, pCO2 25,865 50,464
HCO3 20,828 40,630

Note: #HRC=number of 6-h heart rate characteristic (HRC) records; I:T
ratio= ratio of immature to total white blood cell forms.
a. Given as median (25th, 75th percentiles).

CLINICAL PREDICTION MODELS 261

INFORMATICS MONITORING

with different WBC values, as expected from clinical
observation that septic infants might have abnor-
mally high or low values.

Table 2 shows modeling results. The major find-
ing is that many nearest-neighbor and logistic
regression models were significantly associated with
upcoming sepsis, with ROC areas as high as 0.71.
The nearest-neighbor results validate the general
approach of using the similarity of present data to
past cases to estimate the risk of imminent illness.
Most interestingly, a combined model consisting of
the maximum (or the worst) prediction of several
performed better, with ROC area 0.85, validating the
clinicians’ reasoning that a single clearly abnormal
result trumps a number of other normal results.

To determine an optimal model, we tested all 256
possible combinations of HRC index and laboratory
tests. The best predictive performance was returned
for the combination of HRC index, WBC, I:T ratio,
and HCO3, with ROC area 0.86. This is the model
that was further evaluated in Figures 4 and 5.

The column of Table 2 titled ‘‘Both’’ shows the
results of combined models using both nearest-
neighbor analysis and logistic regression. The
important result is that the 2 kinds of models often
added independent information to each other, as
shown in the rightmost columns. This validates the
approach of using multiple predictive models, each
of them incorporating multiple variables such as
laboratory test results and HRC.

Figure 2 shows fit (Wald statistic) as a function of
discrimination (ROC area) for bivariable regression
models relating sepsis to probability estimates from
logistic regression and nearest-neighbor models
using the same predictor variables. Strategic combi-
nation of multiple models led to improved fit and
discrimination.

New data point
healthy
neighbor

septic
neighbor

neighborhood

WB
C

Day of age

B
ir

th
w

ei
gh

t

Figure 1 Method of nearest-neighbor analysis. The plot is a styl-

ized representation of a 3-dimensional space in which each point

is a white blood cell count (WBC) value measured on a known
day of age in an infant of known birth weight. Filled points

occurred in infants within 24 h of a positive blood culture

obtained for signs of sepsis. There are 2 clusters of filled points,
indicating sepsis occurring in infants with low or high WBC

values. The test point is at the center of the shaded sphere. Con-

ceptually, the sphere is enlarged until it contains 10 filled points,

and the proportion of filled to total points is calculated. This
probability measure is the nearest-neighbor analysis result of the

likelihood of imminent sepsis.

Table 2 Modeling Results

N-n ROC N-n Wald stat LR ROC N-n Wald stat Both: ROC Both: Wald stat N-n add? LR add?

HRC 0.71 134 0.74 105 0.74 137 * *
WBC 0.57 12 0.53 3.1 0.58 15 *
I:T ratio 0.67 58 0.69 60 0.70 74 * *
Platelet count 0.61 33 0.62 55 0.63 58 *
Glucose 0.51 0.4 0.54 3.5 0.54 3.8
pCO2 0.56 5.5 0.58 5.8 0.58 6.9 *
pH 0.57 6.3 0.58 6.2 0.59 9.6 *
HCO3 0.53 0.9 0.58 9.0 0.60 15 * *
Combined model 0.85 317 0.87 311 0.87 472 * *
Optimal model 0.86 358 0.87 319 0.88 480 * *

Note: N-n=nearest-neighbor analysis; ROC= receiver-operating characteristic area; Wald stat=Wald statistic; LR= logistic regression; both=bivariable
regression model using results of nearest-neighbor analysis and multivariable logistic regression models; HRC=heart rate characteristics; WBC=white
blood cell count; I:T ratio= ratio of immature to total white blood cell forms; N-n add *=nearest-neighbor model added significant information to logis-
tic regression model (P < 0.05); LR add *= logistic regression model added significant information to nearest-neighbor model (P < 0.05).

262 • MEDICAL DECISION MAKING/MAR–APR 2010

XIAO AND OTHERS

Figure 3 shows the proportion of time that each
model was applicable (i.e., the appropriate laboratory
tests were available) as well as the proportion of time
that each model contributed the highest prediction
value in the final model. The important result is that
all models contributed to the final prediction proba-
bility. Because the variables were selected for their
relevance to neonatal sepsis, the result is unlikely to
be due to outliers from irrelevant data.

As noted above, the optimal model used HRC,
WBC, I:T ratio, and HCO3, and Figure 4 shows an
evaluation of its performance. The smooth line shows
the output of the predictive model using coefficients
determined from the training set. The circles are the
observed results from the test set, and the boxes
describe the 95% confidence limits determined by
bootstrap. There is good agreement, with a sharp
increase in predicted and observed probability in the
top 10%. We defined this as a high-risk group,16 and
we similarly defined low-risk (lowest 70%) and inter-
mediate-risk (70th-90th percentile) groups.

Figure 5 shows the time dependence of the
change in risk stratified by these groupings. Initially,

there is a very large distinction between the low-
and high-risk groups. For example, there is a 20-fold
increase in the relative risk of sepsis in the

high-risk

group compared with the low-risk group, from 0.28-
fold to 5.5-fold increase in the average risk. Seven
days after a measurement, there is still a more than
3-fold increase in the high-risk group compared
with the low-risk group, from 0.70-fold to 2.3-fold
increase in the average risk.

DISCUSSION

We studied the use of predictive models based on
nearest-neighbor analysis and on logistic regression
in the early diagnosis of late-onset neonatal sepsis.
Our major finding was that combining nearest-
neighbor and logistic regression models, each based
on multiple variables, led to improved prediction.
We incorporated the reasoning of clinicians that les-
sons learned from past cases were valuable.

Neonatal sepsis seems to be a particularly apt
clinical problem to use nearest-neighbor analysis
in creating predictive models. These infants are con-
tinuously monitored and have frequent laboratory
testing, but because no single test has extremely
high predictive performance, physicians are almost
always uncertain of the diagnosis until signs of
severe illness are present. The analytical strategy
presented here should be useful in quantifying the
experience of other physicians with other patients
in an observational database. The prediction result,

0.5 0.6 0.7 0.8 0.9 1
0

250

500

Discrimination (ROC area)

F
it

(
W

al
d

s
ta

ti
st

ic
)

platelets
pCO2

pH
HCO

3

glucose I:T ratio

HRC index

combined model
optimal model

WBC

Figure 2 Combination of multiple nearest-neighbor models lead
to improved prediction of illness. The fit of the predictive models,

calculated as the Wald chi-square statistic of the data given the

model, is plotted as a function of the area under the receiver-
operating characteristic area. Of the models using a single vari-

able (in addition to day of age and birth weight), heart rate char-

acteristic index had the best performance and ratio of immature

to total white blood cell forms the next best. Either strategy of
model combination had better performance.

0.0 0.2 0.4 0.6 0.8 1.0

Proportion of time used / available

HRC index
pH

pCO2
glucose

HCO3
platelets

WBC
I:T ratio

Figure 3 Availability and utilization of predictive variables. The

filled section of each bar is the proportion of the total time for
which the model incorporating the specified variable led to the

highest predicted probability of illness, and the open section

is the proportion of time for which the variable was available.

By the study design, heart rate characteristic was available all of
the time.

CLINICAL PREDICTION MODELS 263

INFORMATICS MONITORING

of course, requires further contextual interpretation
by the physician.

Nearest-neighbor analysis might find general utility
in other clinical situations in which the stakes are high
and the data plentiful but unsorted. The combination
of nearest-neighbor and logistic regression is especially
appealing in multivariate applications in which there
are both linear and nonlinear associations. Logistic
regression may have superior performance in handling
the linear processes, and nearest-neighbor may be
more effective in treating the nonlinear components.
The challenge to the algorithm developer is optimal
handling of continuous and intermittent data with
unequal magnitude and variation and unknown corre-
lations. To adapt to the general situations that labora-
tory measures were taken at irregular times when there
might be a need, we developed a highly flexible model
that combines individual models associated with
those laboratory tests but makes the final estimate
based on what is available. Because nearest-neighbor
rules are based on the assumption of independently
distributed variables,15 we used Mahalanobis dis-
tance measure to deal with the problem of different
kinds of data with unequal magnitude and unknown
correlations.

Limitations of the Study

HRC data were not available for online inspection
by physicians, and results are likely to be different
when they are. In our hospital, real-time HRC moni-
toring has been in use since September 2003 and has

resulted in diagnosis and treatment of bacterial sep-
sis with no or only symptoms.9 C-reactive protein
(CRP) and newer tests for systemic inflammation are
not in routine use at our hospital but are likely to
add useful diagnostic information.23 Finally, diag-
nosis of the presence or absence of neonatal sepsis is
often uncertain. Blood cultures have notoriously
poor diagnostic accuracy, especially in this set-
ting, where only very small blood samples can be
spared.24 Moreover, neurodevelopmental abnormal-
ities are the same in infants with clinical sepsis
regardless of the blood culture result.25 As a result,
the most recent guidelines for diagnosis of blood-
stream infection in neonates lean heavily on clinical
and laboratory findings other than blood cultures,26

with a preference for multivariable analysis.27

The major limitation of any nearest-neighbor
analysis is the database itself. If populated with
incorrect or irrelevant data, there is obviously a dete-
rioration of predictive performance. A more specific
limitation is the nature of the outcome itself, the
clinical and laboratory diagnosis of neonatal sepsis.
Because blood cultures are a tarnished gold stan-
dard, there is irreducible uncertainty in the precise
diagnosis of an infant with obvious clinical signs of
illness but negative blood cultures. There is increas-
ing awareness that many such infants have systemic
inflammation and are vulnerable to identical neuro-
developmental impairment as infants with the
same clinical illness but positive blood cultures.25

Furthermore, the diagnosis of neonatal sepsis is rela-
tively rare: an episode per 6 to 12 infant-months, or
<1% of the time.1 As a result, most neighbors are

5
10

F
o

ld
-i

n
cr

ea
se

in
r

is
k

0 25 50 75 100
Model prediction, as percentile

Figure 4 Internal validation of the nearest-neighbor model of

selected laboratory tests. The plot shows predicted (smooth line)

and observed (circles) rates of sepsis based on the percentile of
the predicted probability. The boxes show the 95% confidence

limits of the observed rates.

0 1 2 3 4 5 6 7
0

1
2
3
4
5
6
7
F
o
ld
-i
n
cr
ea
se
in
r
is
k

Time (days)

high-risk

intermediate

low

Figure 5 Relative risks of sepsis in low-, intermediate-, and high-

risk groups based on the model of selected laboratory tests.

264 • MEDICAL DECISION MAKING/MAR–APR 2010

XIAO AND OTHERS

‘‘well’’ no matter how abnormal the HRC and clini-
cal data. Our training set of 36,000 6-h records held
only 220 occurring within 24 h of sepsis. To guard
against including inappropriately large search spaces,
we found the 10 nearest ‘‘sick’’ neighbors and calcu-
lated their proportional frequency. Were the abnormal
records scattered randomly, our search space would
include less than 5% of the total data set.

In the nearest-neighbor analysis, we used the
Mahalanobis distance metric instead of the more
straight-forward Euclidean distance because it cor-
rects for problems of analyzing correlated data with
different scales. This advantage outweighs any theo-
retical limitation, but we note that the Mahalanobis
distance was designed for multivariate normal data.

We have used only logistic regression and nearest-
neighbor analysis to explore how data mining might
aid physicians in the early diagnosis of neonatal sep-
sis. There are many other kinds of analysis that might
be added or substituted, including neural and Bayes-
ian networks and decision tree analysis. There are
many other possible variables to measure, including
continuous ones such as O2 saturation monitoring
and intermittent ones such CRP and other new labo-
ratory markers of systemic inflammation. There are
other disease processes for which this approach
would be useful in addition to systemic inflammatory
response syndrome. Our targets are subacute, poten-
tially catastrophic illnesses or complications. Exam-
ples include acute exacerbations of chronic asthma
or chronic obstructive pulmonary disease, pneumo-
nia, urinary tract infection and sepsis after brain or
spinal cord injury, cancer recurrence, exacerbation of
inflammatory bowel disease, severe hypoglycemia in
insulin-requiring diabetes mellitus, complications of
nursing home care such as bed sores, relapse of con-
gestive heart failure, new or recurrent infection in
HIV disease, community outbreaks of influenza or
other infection, or effects of bioterrorism agents. For
each setting, an observational database that houses
continuous and intermittent measures can be devel-
oped and predictive models derived.

Future Work

We foresee challenges in the implementation of
nearest-neighbor analysis. First, because the major
target events are infrequent compared with the test-
ing, we can expect many false-positive results. We
find this acceptable because the finding of a recent
increase in risk of the target illnesses need result in
only no-invasive or minimally invasive testing such
as imaging or blood sampling. This seems warranted

by the potentially catastrophic outcome of late diag-
nosis and late treatment. The goal is an early detec-
tion system for increased risk, not a substitute for
a physician. Second, as with all predictive models,
there is danger of overfitting. Here, we have guarded
against this by developing and validating predic-
tive algorithms in separate populations and by
appropriate adjustments when repeated measures
are employed. Third, not all clinical problems are
suitable for informatics-monitoring approaches.
Truly sudden catastrophes with either no prodrome
or ones that are too short to allow intervention are
unlikely to be amenable to predictive algorithms. It
is possible, for example, that arterial thrombosis
leading to acute myocardial infarction or stroke has
no prodrome. Finally, the issue of liability must be
prospectively addressed. Consider a situation in
which a monitored patient’s risk for, say, sepsis
in the setting of spinal cord injury increases more
than 3-fold in the middle of the night. Is one negli-
gent to ignore this until office hours?

CONCLUSION

Nearest-neighbor analysis adds to existing logistic
regression methods for early diagnosis of neonatal
sepsis. Nearest-neighbor analysis is a novel approach
to predicting imminent illness, and both logistic
regression and nearest-neighbor analysis models
based on HRC and laboratory test results contribute
independent information. The best predictions
employed both kinds of models and were driven by
the single most abnormal finding. This approach to
predicting illness may prove useful in other clinical
situations.

REFERENCES

1. Stoll BJ, Hansen N, Fanaroff AA, et al. Late-onset sepsis in very

low birth weight neonates: the experience of the NICHD Neonatal

Research Network. Pediatrics. 2002;110(2 pt 1):285–91.

2. Fanaroff AA, Korones SB, Wright LL, et al. Incidence, present-

ing features, risk factors and significance of late onset septicemia

in very low birth weight infants. The National Institute of Child

Health and Human Development Neonatal Research Network.

Pediatr Infect Dis J. 1998;17(7):593–8.

3. Escobar GJ. The neonatal ‘‘sepsis work-up’’: personal reflec-

tions on the development of an evidence-based approach toward

newborn infections in a managed care organization. Pediatrics.

1999;103(1 suppl E):360–73.

4. Bone RC. Important new findings in sepsis. JAMA. 1997;

278(3):249.

5. Brilli RJ, Goldstein B. Pediatric sepsis definitions: past, pres-

ent, and future. Pediatr Crit Care Med. 2005;6(3):S6–S8.

CLINICAL PREDICTION MODELS 265

INFORMATICS MONITORING

6. Griffin MP, O’Shea TM, Bissonette EA, Harrell FE Jr, Lake DE,

Moorman JR. Abnormal heart rate characteristics preceding neo-

natal sepsis and sepsis-like illness. Pediatr Res. 2003;53:920–6.

7. Griffin MP, O’Shea TM, Bissonette EA, Harrell FE Jr, Lake DE,

Moorman JR. Abnormal heart rate characteristics are associated

with neonatal mortality. Pediatr Res. 2004;55:782–8.

8. Griffin MP, Lake DE, Bissonette EA, Harrell FE Jr, O’Shea TM,

Moorman JR. Heart rate characteristics: novel physiomarkers to pre-

dict neonatal infection and death. Pediatrics. 2005;116(5):1070–4.

9. Moorman JR, Lake DE, Griffin MP. Heart rate characteristics

monitoring in neonatal sepsis. IEEE Trans Biomed Eng. 2006;

53(1):126–32.

10. Griffin MP, Moorman JR. Toward the early diagnosis of neo-

natal sepsis and sepsis-like illness using novel heart rate analysis.

Pediatrics. 2001;107:97–104.

11. Kovatchev BP, Farhy LS, Cao H, Griffin MP, Lake DE, Moor-

man JR. Sample asymmetry analysis of heart rate characteristics

with application to neonatal sepsis and systemic inflammatory

response syndrome. Pediatr Res. 2003;54(6):892–8.

12. Lake DE, Richman JS, Griffin MP, Moorman JR. Sample

entropy analysis of neonatal heart rate variability. Am J Physiol.

2002;283:R789–97.

13. Richman JS, Moorman JR. Physiological time series analysis

using approximate entropy and sample entropy. Am J Physiol.

2000;278:H2039–49.

14. Fukunaga K. Introduction to Statistical Pattern Recognition.

San Diego, CA: Academic Press; 1990.

15. Devijver PA, Kittler J. Pattern Recognition: A Statistical

Approach. London: Prentice-Hall; 1982.

16. Griffin MP, Lake DE, Moorman JR. Heart rate characteristics

and laboratory tests in neonatal sepsis. Pediatrics. 2005;115:

937–41.

17. Haddad M, Adlassnig KP, Porenta G. Feasibility analysis of

a case-based reasoning system for automated detection of coro-

nary heart disease from myocardial scintigrams. Artif Intell Med.

1997;9(1):61–78.

18. Qu H, Gotman J. A patient-specific algorithm for the detection

of seizure onset in long-term EEG monitoring: possible use as

a warning device. IEEE Trans Biomed Eng. 1997;44(2):115–22.

19. Lutz W, Leach C, Barkham M, et al. Predicting change for

individual psychotherapy clients on the basis of their nearest

neighbors. J Consult Clin Psychol. 2005;73(5):904–13.

20. Garner JS, Jarvis WR, Emori TG, Horan TC, Hughes JM. CDC

definitions for nosocomial infections, 1988. Am J Infect Control.

1988;16(3):128–40.

21. Harrell FE Jr. Regression Modeling Strategies: With Applica-

tions to Linear Models, Logistic Regression and Survival Analy-

sis. Berlin: Springer; 2001.

22. Richman JS, Lake DE, Moorman JR. Sample entropy. Methods

Enzymol. 2004;384:172–84.

23. Malik A, Hui CP, Pennie RA, Kirpalani H. Beyond the com-

plete blood cell count and C-reactive protein: a systematic review

of modern diagnostic tests for neonatal sepsis. Arch Pediatr Ado-

lesc Med. 2003;157(6):511–6.

24. Jawaheer G, Neal TJ, Shaw NJ. Blood culture volume and

detection of coagulase negative staphylococcal septicaemia in

neonates. Arch Dis Child Fetal Neonatal Ed. 1997;76(1):F57–58.

25. Stoll BJ, Hansen NI, Adams-Chapman I, et al. Neurodevelop-

mental and growth impairment among extremely low-birth-weight

infants with neonatal infection. JAMA. 2004;292(19):2357–65.

26. Haque KN. Definitions of bloodstream infection in the new-

born. Pediatr Crit Care Med. 2005;6(3):S45–9.

27. Escobar GJ. What have we learned from observational studies

on neonatal sepsis? Pediatr Crit Care Med. 2005;6(3):S138–45.

266 • MEDICAL DECISION MAKING/MAR–APR 2010

XIAO AND OTHERS

<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams false /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /ACaslon-Bold /ACaslon-BoldItalic /ACaslon-Italic /ACaslon-Ornaments /ACaslon-Regular /ACaslon-Semibold /ACaslon-SemiboldItalic /AdobeCorpID-Acrobat /AdobeCorpID-Adobe /AdobeCorpID-Bullet /AdobeCorpID-MinionBd /AdobeCorpID-MinionBdIt /AdobeCorpID-MinionRg /AdobeCorpID-MinionRgIt /AdobeCorpID-MinionSb /AdobeCorpID-MinionSbIt /AdobeCorpID-MyriadBd /AdobeCorpID-MyriadBdIt /AdobeCorpID-MyriadBdScn /AdobeCorpID-MyriadBdScnIt /AdobeCorpID-MyriadBl /AdobeCorpID-MyriadBlIt /AdobeCorpID-MyriadLt /AdobeCorpID-MyriadLtIt /AdobeCorpID-MyriadPkg /AdobeCorpID-MyriadRg /AdobeCorpID-MyriadRgIt /AdobeCorpID-MyriadRgScn /AdobeCorpID-MyriadRgScnIt /AdobeCorpID-MyriadSb /AdobeCorpID-MyriadSbIt /AdobeCorpID-MyriadSbScn /AdobeCorpID-MyriadSbScnIt /AdobeCorpID-PScript /AGaramond-BoldScaps /AGaramond-Italic /AGaramond-Regular /AGaramond-RomanScaps /AGaramond-Semibold /AGaramond-SemiboldItalic /AGar-Special /AkzidenzGroteskBE-Bold /AkzidenzGroteskBE-BoldEx /AkzidenzGroteskBE-BoldExIt /AkzidenzGroteskBE-BoldIt /AkzidenzGroteskBE-Ex /AkzidenzGroteskBE-It /AkzidenzGroteskBE-Light /AkzidenzGroteskBE-LightEx /AkzidenzGroteskBE-LightOsF /AkzidenzGroteskBE-Md /AkzidenzGroteskBE-MdEx /AkzidenzGroteskBE-MdIt /AkzidenzGroteskBE-Regular /AkzidenzGroteskBE-Super /AlbertusMT /AlbertusMT-Italic /AlbertusMT-Light /Aldine401BT-BoldA /Aldine401BT-BoldItalicA /Aldine401BT-ItalicA /Aldine401BT-RomanA /Aldine401BTSPL-RomanA /Aldine721BT-Bold /Aldine721BT-BoldItalic /Aldine721BT-Italic /Aldine721BT-Light /Aldine721BT-LightItalic /Aldine721BT-Roman /Aldus-Italic /Aldus-ItalicOsF /Aldus-Roman /Aldus-RomanSC /AlternateGothicNo2BT-Regular /AmazoneBT-Regular /AmericanTypewriter-Bold /AmericanTypewriter-BoldA /AmericanTypewriter-BoldCond /AmericanTypewriter-BoldCondA /AmericanTypewriter-Cond /AmericanTypewriter-CondA /AmericanTypewriter-Light /AmericanTypewriter-LightA /AmericanTypewriter-LightCond /AmericanTypewriter-LightCondA /AmericanTypewriter-Medium /AmericanTypewriter-MediumA /Anna /AntiqueOlive-Bold /AntiqueOlive-Compact /AntiqueOlive-Italic /AntiqueOlive-Roman /Arcadia /Arcadia-A /Arkona-Medium /Arkona-Regular /ArrusBT-Black /ArrusBT-BlackItalic /ArrusBT-Bold /ArrusBT-BoldItalic /ArrusBT-Italic /ArrusBT-Roman /AssemblyLightSSK /AuroraBT-BoldCondensed /AuroraBT-RomanCondensed /AuroraOpti-Condensed /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-Demi /AvantGarde-DemiOblique /Avenir-Black /Avenir-BlackOblique /Avenir-Book /Avenir-BookOblique /Avenir-Heavy /Avenir-HeavyOblique /Avenir-Light /Avenir-LightOblique /Avenir-Medium /Avenir-MediumOblique /Avenir-Oblique /Avenir-Roman /BaileySansITC-Bold /BaileySansITC-BoldItalic /BaileySansITC-Book /BaileySansITC-BookItalic /BakerSignetBT-Roman /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /Baskerville-Bold /BaskervilleBook-Italic /BaskervilleBook-MedItalic /BaskervilleBook-Medium /BaskervilleBook-Regular /BaskervilleBT-Bold /BaskervilleBT-BoldItalic /BaskervilleBT-Italic /BaskervilleBT-Roman /BaskervilleMT /BaskervilleMT-Bold /BaskervilleMT-BoldItalic /BaskervilleMT-Italic /BaskervilleMT-SemiBold /BaskervilleMT-SemiBoldItalic /BaskervilleNo2BT-Bold /BaskervilleNo2BT-BoldItalic /BaskervilleNo2BT-Italic /BaskervilleNo2BT-Roman /Baskerville-Normal-Italic /BauerBodoni-Black /BauerBodoni-BlackCond /BauerBodoni-BlackItalic /BauerBodoni-Bold /BauerBodoni-BoldCond /BauerBodoni-BoldItalic /BauerBodoni-BoldItalicOsF /BauerBodoni-BoldOsF /BauerBodoni-Italic /BauerBodoni-ItalicOsF /BauerBodoni-Roman /BauerBodoni-RomanSC /Bauhaus-Bold /Bauhaus-Demi /Bauhaus-Heavy /BauhausITCbyBT-Bold /BauhausITCbyBT-Heavy /BauhausITCbyBT-Light /BauhausITCbyBT-Medium /Bauhaus-Light /Bauhaus-Medium /BellCentennial-Address /BellGothic-Black /BellGothic-Bold /Bell-GothicBoldItalicBT /BellGothicBT-Bold /BellGothicBT-Roman /BellGothic-Light /Bembo /Bembo-Bold /Bembo-BoldExpert /Bembo-BoldItalic /Bembo-BoldItalicExpert /Bembo-Expert /Bembo-ExtraBoldItalic /Bembo-Italic /Bembo-ItalicExpert /Bembo-Semibold /Bembo-SemiboldItalic /Benguiat-Bold /Benguiat-BoldItalic /Benguiat-Book /Benguiat-BookItalic /BenguiatGothicITCbyBT-Bold /BenguiatGothicITCbyBT-BoldItal /BenguiatGothicITCbyBT-Book /BenguiatGothicITCbyBT-BookItal /BenguiatITCbyBT-Bold /BenguiatITCbyBT-BoldItalic /BenguiatITCbyBT-Book /BenguiatITCbyBT-BookItalic /Benguiat-Medium /Benguiat-MediumItalic /Berkeley-Black /Berkeley-BlackItalic /Berkeley-Bold /Berkeley-BoldItalic /Berkeley-Book /Berkeley-BookItalic /Berkeley-Italic /Berkeley-Medium /Berling-Bold /Berling-BoldItalic /Berling-Italic /Berling-Roman /BernhardBoldCondensedBT-Regular /BernhardFashionBT-Regular /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /BernhardTangoBT-Regular /BlockBE-Condensed /BlockBE-ExtraCn /BlockBE-ExtraCnIt /BlockBE-Heavy /BlockBE-Italic /BlockBE-Regular /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /Bodoni-Poster /Bodoni-PosterCompressed /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /BremenBT-Black /BremenBT-Bold /BroadwayBT-Regular /CaflischScript-Bold /CaflischScript-Regular /Caliban /CarminaBT-Bold /CarminaBT-BoldItalic /CarminaBT-Light /CarminaBT-LightItalic /CarminaBT-Medium /CarminaBT-MediumItalic /Carta /Caslon224ITCbyBT-Bold /Caslon224ITCbyBT-BoldItalic /Caslon224ITCbyBT-Book /Caslon224ITCbyBT-BookItalic /Caslon540BT-Italic /Caslon540BT-Roman /CaslonBT-Bold /CaslonBT-BoldItalic /CaslonOpenFace /CaslonTwoTwentyFour-Black /CaslonTwoTwentyFour-BlackIt /CaslonTwoTwentyFour-Bold /CaslonTwoTwentyFour-BoldIt /CaslonTwoTwentyFour-Book /CaslonTwoTwentyFour-BookIt /CaslonTwoTwentyFour-Medium /CaslonTwoTwentyFour-MediumIt /CastleT-Bold /CastleT-Book /Caxton-Bold /Caxton-BoldItalic /Caxton-Book /Caxton-BookItalic /CaxtonBT-Bold /CaxtonBT-BoldItalic /CaxtonBT-Book /CaxtonBT-BookItalic /Caxton-Light /Caxton-LightItalic /CelestiaAntiqua-Ornaments /Centennial-BlackItalicOsF /Centennial-BlackOsF /Centennial-BoldItalicOsF /Centennial-BoldOsF /Centennial-ItalicOsF /Centennial-LightItalicOsF /Centennial-LightSC /Centennial-RomanSC /Century-Bold /Century-BoldItalic /Century-Book /Century-BookItalic /CenturyExpandedBT-Bold /CenturyExpandedBT-BoldItalic /CenturyExpandedBT-Italic /CenturyExpandedBT-Roman /Century-HandtooledBold /Century-HandtooledBoldItalic /Century-Light /Century-LightItalic /CenturyOldStyle-Bold /CenturyOldStyle-Italic /CenturyOldStyle-Regular /CenturySchoolbookBT-Bold /CenturySchoolbookBT-BoldCond /CenturySchoolbookBT-BoldItalic /CenturySchoolbookBT-Italic /CenturySchoolbookBT-Roman /Century-Ultra /Century-UltraItalic /CharterBT-Black /CharterBT-BlackItalic /CharterBT-Bold /CharterBT-BoldItalic /CharterBT-Italic /CharterBT-Roman /CheltenhamBT-Bold /CheltenhamBT-BoldCondItalic /CheltenhamBT-BoldExtraCondensed /CheltenhamBT-BoldHeadline /CheltenhamBT-BoldItalic /CheltenhamBT-BoldItalicHeadline /CheltenhamBT-Italic /CheltenhamBT-Roman /Cheltenham-HandtooledBdIt /Cheltenham-HandtooledBold /CheltenhamITCbyBT-Bold /CheltenhamITCbyBT-BoldItalic /CheltenhamITCbyBT-Book /CheltenhamITCbyBT-BookItalic /Christiana-Bold /Christiana-BoldItalic /Christiana-Italic /Christiana-Medium /Christiana-MediumItalic /Christiana-Regular /Christiana-RegularExpert /Christiana-RegularSC /Clarendon /Clarendon-Bold /Clarendon-Light /ClassicalGaramondBT-Bold /ClassicalGaramondBT-BoldItalic /ClassicalGaramondBT-Italic /ClassicalGaramondBT-Roman /CMR10 /CMR8 /CMSY10 /CMSY8 /CMTI10 /CommonBullets /ConduitITC-Bold /ConduitITC-BoldItalic /ConduitITC-Light /ConduitITC-LightItalic /ConduitITC-Medium /ConduitITC-MediumItalic /CooperBlack /CooperBlack-Italic /CooperBT-Bold /CooperBT-BoldItalic /CooperBT-Light /CooperBT-LightItalic /CopperplateGothicBT-Bold /CopperplateGothicBT-BoldCond /CopperplateGothicBT-Heavy /CopperplateGothicBT-Roman /CopperplateGothicBT-RomanCond /Copperplate-ThirtyThreeBC /Copperplate-ThirtyTwoBC /Coronet-Regular /Courier /Courier-Bold /Courier-BoldOblique /Courier-Oblique /Critter /CS-Special-font /DellaRobbiaBT-Bold /DellaRobbiaBT-Roman /Della-RobbiaItalicBT /Della-RobbiaSCaps /Del-NormalSmallCaps /Delphin-IA /Delphin-IIA /Delta-Bold /Delta-BoldItalic /Delta-Book /Delta-BookItalic /Delta-Light /Delta-LightItalic /Delta-Medium /Delta-MediumItalic /Delta-Outline /DextorD /DextorOutD /DidotLH-OrnamentsOne /DidotLH-OrnamentsTwo /DINEngschrift /DINEngschrift-Alternate /DINMittelschrift /DINMittelschrift-Alternate /DINNeuzeitGrotesk-BoldCond /DINNeuzeitGrotesk-Light /Dom-CasItalic /DomCasual /DomCasual-Bold /Dom-CasualBT /Ehrhard-Italic /Ehrhard-Regular /EhrhardSemi-Italic /EhrhardtMT /EhrhardtMT-Italic /EhrhardtMT-SemiBold /EhrhardtMT-SemiBoldItalic /EhrharSemi /ELANGO-IB-A03 /ELANGO-IB-A75 /ELANGO-IB-A99 /ElectraLH-Bold /ElectraLH-BoldCursive /ElectraLH-Cursive /ElectraLH-Regular /ElGreco /EnglischeSchT-Bold /EnglischeSchT-Regu /ErasContour /ErasITCbyBT-Bold /ErasITCbyBT-Book /ErasITCbyBT-Demi /ErasITCbyBT-Light /ErasITCbyBT-Medium /ErasITCbyBT-Ultra /Euclid /Euclid-Bold /Euclid-BoldItalic /EuclidExtra /EuclidExtra-Bold /EuclidFraktur /EuclidFraktur-Bold /Euclid-Italic /EuclidMathOne /EuclidMathOne-Bold /EuclidMathTwo /EuclidMathTwo-Bold /EuclidSymbol /EuclidSymbol-Bold /EuclidSymbol-BoldItalic /EuclidSymbol-Italic /EUEX10 /EUFB10 /EUFB5 /EUFB7 /EUFM10 /EUFM5 /EUFM7 /EURB10 /EURB5 /EURB7 /EURM10 /EURM5 /EURM7 /EuropeanPi-Four /EuropeanPi-One /EuropeanPi-Three /EuropeanPi-Two /EuroSans-Bold /EuroSans-BoldItalic /EuroSans-Italic /EuroSans-Regular /EuroSerif-Bold /EuroSerif-BoldItalic /EuroSerif-Italic /EuroSerif-Regular /Eurostile /Eurostile-Bold /Eurostile-BoldCondensed /Eurostile-BoldExtendedTwo /Eurostile-BoldOblique /Eurostile-Condensed /Eurostile-Demi /Eurostile-DemiOblique /Eurostile-ExtendedTwo /EurostileLTStd-Demi /EurostileLTStd-DemiOblique /Eurostile-Oblique /EUSB10 /EUSB5 /EUSB7 /EUSM10 /EUSM5 /EUSM7 /ExPonto-Regular /FairfieldLH-Bold /FairfieldLH-BoldItalic /FairfieldLH-BoldSC /FairfieldLH-CaptionBold /FairfieldLH-CaptionHeavy /FairfieldLH-CaptionLight /FairfieldLH-CaptionMedium /FairfieldLH-Heavy /FairfieldLH-HeavyItalic /FairfieldLH-HeavySC /FairfieldLH-Light /FairfieldLH-LightItalic /FairfieldLH-LightSC /FairfieldLH-Medium /FairfieldLH-MediumItalic /FairfieldLH-MediumSC /FairfieldLH-SwBoldItalicOsF /FairfieldLH-SwHeavyItalicOsF /FairfieldLH-SwLightItalicOsF /FairfieldLH-SwMediumItalicOsF /Fences /Fenice-Bold /Fenice-BoldOblique /FeniceITCbyBT-Bold /FeniceITCbyBT-BoldItalic /FeniceITCbyBT-Regular /FeniceITCbyBT-RegularItalic /Fenice-Light /Fenice-LightOblique /Fenice-Regular /Fenice-RegularOblique /Fenice-Ultra /Fenice-UltraOblique /FlashD-Ligh /Flood /Folio-Bold /Folio-BoldCondensed /Folio-ExtraBold /Folio-Light /Folio-Medium /FontanaNDAaOsF /FontanaNDAaOsF-Italic /FontanaNDCcOsF-Semibold /FontanaNDCcOsF-SemiboldIta /FontanaNDEeOsF /FontanaNDEeOsF-Bold /FontanaNDEeOsF-BoldItalic /FontanaNDEeOsF-Light /FontanaNDEeOsF-Semibold /FormalScript421BT-Regular /Formata-Bold /Formata-MediumCondensed /ForteMT /FournierMT-Ornaments /FrakturBT-Regular /FrankfurterHigD /FranklinGothic-Book /FranklinGothic-BookItal /FranklinGothic-BookOblique /FranklinGothic-Condensed /FranklinGothic-Demi /FranklinGothic-DemiItal /FranklinGothic-DemiOblique /FranklinGothic-Heavy /FranklinGothic-HeavyItal /FranklinGothic-HeavyOblique /FranklinGothicITCbyBT-BookItal /FranklinGothicITCbyBT-Demi /FranklinGothicITCbyBT-DemiItal /FranklinGothicITCbyBT-Heavy /FranklinGothicITCbyBT-HeavyItal /FranklinGothic-Medium /FranklinGothic-MediumItal /FranklinGothic-Roman /Freeform721BT-Bold /Freeform721BT-BoldItalic /Freeform721BT-Italic /Freeform721BT-Roman /FreestyleScrD /FreestyleScript /Freestylescript /FrizQuadrataITCbyBT-Bold /FrizQuadrataITCbyBT-Roman /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura /FuturaBlackBT-Regular /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldCondensed /FuturaBT-BoldCondensedItalic /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-ExtraBlack /FuturaBT-ExtraBlackCondensed /FuturaBT-ExtraBlackCondItalic /FuturaBT-ExtraBlackItalic /FuturaBT-Heavy /FuturaBT-HeavyItalic /FuturaBT-Light /FuturaBT-LightCondensed /FuturaBT-LightItalic /FuturaBT-Medium /FuturaBT-MediumCondensed /FuturaBT-MediumItalic /Futura-CondensedLight /Futura-CondensedLightOblique /Futura-ExtraBold /Futura-ExtraBoldOblique /Futura-Heavy /Futura-HeavyOblique /Futura-Light /Futura-LightOblique /Futura-Oblique /Futura-Thin /Galliard-Black /Galliard-BlackItalic /Galliard-Bold /Galliard-BoldItalic /Galliard-Italic /GalliardITCbyBT-Bold /GalliardITCbyBT-BoldItalic /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Galliard-Roman /Galliard-Ultra /Galliard-UltraItalic /Garamond-Antiqua /GaramondBE-Bold /GaramondBE-BoldExpert /GaramondBE-BoldOsF /GaramondBE-CnExpert /GaramondBE-Condensed /GaramondBE-CondensedSC /GaramondBE-Italic /GaramondBE-ItalicExpert /GaramondBE-ItalicOsF /GaramondBE-Medium /GaramondBE-MediumCn /GaramondBE-MediumCnExpert /GaramondBE-MediumCnOsF /GaramondBE-MediumExpert /GaramondBE-MediumItalic /GaramondBE-MediumItalicExpert /GaramondBE-MediumItalicOsF /GaramondBE-MediumSC /GaramondBE-Regular /GaramondBE-RegularExpert /GaramondBE-RegularSC /GaramondBE-SwashItalic /Garamond-Bold /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BoldItalic /Garamond-Book /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-BookItalic /Garamond-Halbfett /Garamond-HandtooledBold /Garamond-HandtooledBoldItalic /GaramondITCbyBT-Bold /GaramondITCbyBT-BoldCondensed /GaramondITCbyBT-BoldCondItalic /GaramondITCbyBT-BoldItalic /GaramondITCbyBT-BoldNarrow /GaramondITCbyBT-BoldNarrowItal /GaramondITCbyBT-Book /GaramondITCbyBT-BookCondensed /GaramondITCbyBT-BookCondItalic /GaramondITCbyBT-BookItalic /GaramondITCbyBT-BookNarrow /GaramondITCbyBT-BookNarrowItal /GaramondITCbyBT-Light /GaramondITCbyBT-LightCondensed /GaramondITCbyBT-LightCondItalic /GaramondITCbyBT-LightItalic /GaramondITCbyBT-LightNarrow /GaramondITCbyBT-LightNarrowItal /GaramondITCbyBT-Ultra /GaramondITCbyBT-UltraCondensed /GaramondITCbyBT-UltraCondItalic /GaramondITCbyBT-UltraItalic /Garamond-Kursiv /Garamond-KursivHalbfett /Garamond-Light /Garamond-LightCondensed /Garamond-LightCondensedItalic /Garamond-LightItalic /GaramondNo4CyrTCY-Ligh /GaramondNo4CyrTCY-LighItal /GaramondThree /GaramondThree-Bold /GaramondThree-BoldItalic /GaramondThree-BoldItalicOsF /GaramondThree-BoldSC /GaramondThree-Italic /GaramondThree-ItalicOsF /GaramondThree-SC /GaramondThreeSMSIISpl-Italic /GaramondThreeSMSitalicSpl-Italic /GaramondThreeSMSspl /GaramondThreespl /GaramondThreeSpl-Bold /GaramondThreeSpl-Italic /Garamond-Ultra /Garamond-UltraCondensed /Garamond-UltraCondensedItalic /Garamond-UltraItalic /GarthGraphic /GarthGraphic-Black /GarthGraphic-Bold /GarthGraphic-BoldCondensed /GarthGraphic-BoldItalic /GarthGraphic-Condensed /GarthGraphic-ExtraBold /GarthGraphic-Italic /Geometric231BT-HeavyC /GeometricSlab712BT-BoldA /GeometricSlab712BT-ExtraBoldA /GeometricSlab712BT-LightA /GeometricSlab712BT-LightItalicA /GeometricSlab712BT-MediumA /GeometricSlab712BT-MediumItalA /Giddyup /Giddyup-Thangs /GillSans /GillSans-Bold /GillSans-BoldCondensed /GillSans-BoldExtraCondensed /GillSans-BoldItalic /GillSans-Condensed /GillSans-ExtraBold /GillSans-ExtraBoldDisplay /GillSans-Italic /GillSans-Light /GillSans-LightItalic /GillSans-LightShadowed /GillSans-Shadowed /GillSans-UltraBold /GillSans-UltraBoldCondensed /Gill-Special /Giovanni-Bold /Giovanni-BoldItalic /Giovanni-Book /Giovanni-BookItalic /Glypha /Glypha-Bold /Glypha-BoldOblique /Glypha-Oblique /Gothic-Thirteen /Goudy /Goudy-Bold /Goudy-BoldItalic /GoudyCatalogueBT-Regular /Goudy-ExtraBold /GoudyHandtooledBT-Regular /GoudyHeavyfaceBT-Regular /GoudyHeavyfaceBT-RegularCond /Goudy-Italic /GoudyOldStyleBT-Bold /GoudyOldStyleBT-BoldItalic /GoudyOldStyleBT-ExtraBold /GoudyOldStyleBT-Italic /GoudyOldStyleBT-Roman /GoudySans-Black /GoudySans-BlackItalic /GoudySans-Bold /GoudySans-BoldItalic /GoudySans-Book /GoudySans-BookItalic /GoudySansITCbyBT-Black /GoudySansITCbyBT-BlackItalic /GoudySansITCbyBT-Bold /GoudySansITCbyBT-BoldItalic /GoudySansITCbyBT-Light /GoudySansITCbyBT-LightItalic /GoudySansITCbyBT-Medium /GoudySansITCbyBT-MediumItalic /GoudySans-Medium /GoudySans-MediumItalic /Granjon /Granjon-Bold /Granjon-BoldOsF /Granjon-Italic /Granjon-ItalicOsF /Granjon-SC /GreymantleMVB-Ornaments /Helvetica /Helvetica-Black /Helvetica-BlackOblique /Helvetica-Black-SemiBold /Helvetica-Bold /Helvetica-BoldOblique /Helvetica-Compressed /Helvetica-Condensed /Helvetica-Condensed-Black /Helvetica-Condensed-BlackObl /Helvetica-Condensed-Bold /Helvetica-Condensed-BoldObl /Helvetica-Condensed-Light /Helvetica-Condensed-Light-Light /Helvetica-Condensed-LightObl /Helvetica-Condensed-Oblique /Helvetica-Condensed-Thin /Helvetica-ExtraCompressed /Helvetica-Fraction /Helvetica-FractionBold /HelveticaInserat-Roman /HelveticaInserat-Roman-SemiBold /Helvetica-Light /Helvetica-LightOblique /Helvetica-Narrow /Helvetica-Narrow-Bold /Helvetica-Narrow-BoldOblique /Helvetica-Narrow-Oblique /HelveticaNeue-Black /HelveticaNeue-BlackCond /HelveticaNeue-BlackCondObl /HelveticaNeue-BlackExt /HelveticaNeue-BlackExtObl /HelveticaNeue-BlackItalic /HelveticaNeue-Bold /HelveticaNeue-BoldCond /HelveticaNeue-BoldCondObl /HelveticaNeue-BoldExt /HelveticaNeue-BoldExtObl /HelveticaNeue-BoldItalic /HelveticaNeue-Condensed /HelveticaNeue-CondensedObl /HelveticaNeue-ExtBlackCond /HelveticaNeue-ExtBlackCondObl /HelveticaNeue-Extended /HelveticaNeue-ExtendedObl /HelveticaNeue-Heavy /HelveticaNeue-HeavyCond /HelveticaNeue-HeavyCondObl /HelveticaNeue-HeavyExt /HelveticaNeue-HeavyExtObl /HelveticaNeue-HeavyItalic /HelveticaNeue-Italic /HelveticaNeue-Light /HelveticaNeue-LightCond /HelveticaNeue-LightCondObl /HelveticaNeue-LightExt /HelveticaNeue-LightExtObl /HelveticaNeue-LightItalic /HelveticaNeueLTStd-Md /HelveticaNeueLTStd-MdIt /HelveticaNeue-Medium /HelveticaNeue-MediumCond /HelveticaNeue-MediumCondObl /HelveticaNeue-MediumExt /HelveticaNeue-MediumExtObl /HelveticaNeue-MediumItalic /HelveticaNeue-Roman /HelveticaNeue-Thin /HelveticaNeue-ThinCond /HelveticaNeue-ThinCondObl /HelveticaNeue-ThinItalic /HelveticaNeue-UltraLigCond /HelveticaNeue-UltraLigCondObl /HelveticaNeue-UltraLigExt /HelveticaNeue-UltraLigExtObl /HelveticaNeue-UltraLight /HelveticaNeue-UltraLightItal /Helvetica-Oblique /Helvetica-UltraCompressed /HelvExtCompressed /HelvLight /HelvUltCompressed /Humanist521BT-Bold /Humanist521BT-BoldCondensed /Humanist521BT-BoldItalic /Humanist521BT-ExtraBold /Humanist521BT-Italic /Humanist521BT-Light /Humanist521BT-LightItalic /Humanist521BT-Roman /Humanist521BT-RomanCondensed /Humanist521BT-UltraBold /Humanist521BT-XtraBoldCondensed /Humanist531BT-BlackA /Humanist531BT-BoldA /Humanist531BT-RomanA /Humanist531BT-UltraBlackA /Humanist777BT-BlackB /Humanist777BT-BlackCondensedB /Humanist777BT-BlackItalicB /Humanist777BT-BoldB /Humanist777BT-BoldCondensedB /Humanist777BT-BoldItalicB /Humanist777BT-ExtraBlackB /Humanist777BT-ExtraBlackCondB /Humanist777BT-ItalicB /Humanist777BT-LightB /Humanist777BT-LightCondensedB /Humanist777BT-LightItalicB /Humanist777BT-RomanB /Humanist777BT-RomanCondensedB /Humanist970BT-BoldC /Humanist970BT-RomanC /HumanistSlabserif712BT-Black /HumanistSlabserif712BT-Bold /HumanistSlabserif712BT-Italic /HumanistSlabserif712BT-Roman /ICMEX10 /ICMMI8 /ICMSY8 /ICMTT8 /Iglesia-Light /ILASY8 /ILCMSS8 /ILCMSSB8 /ILCMSSI8 /Imago-Book /Imago-BookItalic /Imago-ExtraBold /Imago-ExtraBoldItalic /Imago-Light /Imago-LightItalic /Imago-Medium /Imago-MediumItalic /Industria-Inline /Industria-InlineA /Industria-Solid /Industria-SolidA /Insignia /Insignia-A /IPAExtras /IPAHighLow /IPAKiel /IPAKielSeven /IPAsans /ITCGaramondMM /ITCGaramondMM-It /JAKEOpti-Regular /JansonText-Bold /JansonText-BoldItalic /JansonText-Italic /JansonText-Roman /JansonText-RomanSC /JoannaMT /JoannaMT-Bold /JoannaMT-BoldItalic /JoannaMT-Italic /Juniper /KabelITCbyBT-Book /KabelITCbyBT-Demi /KabelITCbyBT-Medium /KabelITCbyBT-Ultra /Kaufmann /Kaufmann-Bold /KeplMM-Or2 /KisBT-Italic /KisBT-Roman /KlangMT /Kuenstler480BT-Black /Kuenstler480BT-Bold /Kuenstler480BT-BoldItalic /Kuenstler480BT-Italic /Kuenstler480BT-Roman /KunstlerschreibschD-Bold /KunstlerschreibschD-Medi /Lapidary333BT-Black /Lapidary333BT-Bold /Lapidary333BT-BoldItalic /Lapidary333BT-Italic /Lapidary333BT-Roman /LASY10 /LASY5 /LASY6 /LASY7 /LASY8 /LASY9 /LASYB10 /LatinMT-Condensed /LCIRCLE10 /LCIRCLEW10 /LCMSS8 /LCMSSB8 /LCMSSI8 /LDecorationPi-One /LDecorationPi-Two /Leawood-Black /Leawood-BlackItalic /Leawood-Bold /Leawood-BoldItalic /Leawood-Book /Leawood-BookItalic /Leawood-Medium /Leawood-MediumItalic /LegacySans-Bold /LegacySans-BoldItalic /LegacySans-Book /LegacySans-BookItalic /LegacySans-Medium /LegacySans-MediumItalic /LegacySans-Ultra /LegacySerif-Bold /LegacySerif-BoldItalic /LegacySerif-Book /LegacySerif-BookItalic /LegacySerif-Medium /LegacySerif-MediumItalic /LegacySerif-Ultra /LetterGothic /LetterGothic-Bold /LetterGothic-BoldSlanted /LetterGothic-Slanted /Life-Bold /Life-Italic /Life-Roman /LINE10 /LINEW10 /Linotext /Lithos-Black /LithosBold /Lithos-Bold /Lithos-Regular /LOGO10 /LOGO8 /LOGO9 /LOGOBF10 /LOGOSL10 /LOMD-Normal /LubalinGraph-Book /LubalinGraph-BookOblique /LubalinGraph-Demi /LubalinGraph-DemiOblique /LucidaHandwritingItalic /LucidaMath-Symbol /LucidaSansTypewriter /LucidaSansTypewriter-Bd /LucidaSansTypewriter-BdObl /LucidaSansTypewriter-Obl /LucidaTypewriter /LucidaTypewriter-Bold /LucidaTypewriter-BoldObl /LucidaTypewriter-Obl /LydianBT-Bold /LydianBT-BoldItalic /LydianBT-Italic /LydianBT-Roman /LydianCursiveBT-Regular /Machine /Machine-Bold /Marigold /MathematicalPi-Five /MathematicalPi-Four /MathematicalPi-One /MathematicalPi-Six /MathematicalPi-Three /MathematicalPi-Two /MatrixScriptBold /MatrixScriptBoldLin /MatrixScriptBook /MatrixScriptBookLin /MatrixScriptRegular /MatrixScriptRegularLin /Melior /Melior-Bold /Melior-BoldItalic /Melior-Italic /MercuriusCT-Black /MercuriusCT-BlackItalic /MercuriusCT-Light /MercuriusCT-LightItalic /MercuriusCT-Medium /MercuriusCT-MediumItalic /MercuriusMT-BoldScript /Meridien-Bold /Meridien-BoldItalic /Meridien-Italic /Meridien-Medium /Meridien-MediumItalic /Meridien-Roman /Minion-Black /Minion-Bold /Minion-BoldCondensed /Minion-BoldCondensedItalic /Minion-BoldItalic /Minion-Condensed /Minion-CondensedItalic /Minion-DisplayItalic /Minion-DisplayRegular /MinionExp-Italic /MinionExp-Semibold /MinionExp-SemiboldItalic /Minion-Italic /Minion-Ornaments /Minion-Regular /Minion-Semibold /Minion-SemiboldItalic /MonaLisa-Recut /MrsEavesAllPetiteCaps /MrsEavesAllSmallCaps /MrsEavesBold /MrsEavesFractions /MrsEavesItalic /MrsEavesPetiteCaps /MrsEavesRoman /MrsEavesRomanLining /MrsEavesSmallCaps /MSAM10 /MSAM10A /MSAM5 /MSAM6 /MSAM7 /MSAM8 /MSAM9 /MSBM10 /MSBM10A /MSBM5 /MSBM6 /MSBM7 /MSBM8 /MSBM9 /MTEX /MTEXB /MTEXH /MTGU /MTGUB /MTMI /MTMIB /MTMIH /MTMS /MTMSB /MTMUB /MTMUH /MTSY /MTSYB /MTSYH /MTSYN /MusicalSymbols-Normal /Myriad-Bold /Myriad-BoldItalic /Myriad-CnBold /Myriad-CnBoldItalic /Myriad-CnItalic /Myriad-CnSemibold /Myriad-CnSemiboldItalic /Myriad-Condensed /Myriad-Italic /MyriadMM /MyriadMM-It /Myriad-Roman /Myriad-Sketch /Myriad-Tilt /NeuzeitS-Book /NeuzeitS-BookHeavy /NewBaskerville-Bold /NewBaskerville-BoldItalic /NewBaskerville-Italic /NewBaskervilleITCbyBT-Bold /NewBaskervilleITCbyBT-BoldItal /NewBaskervilleITCbyBT-Italic /NewBaskervilleITCbyBT-Roman /NewBaskerville-Roman /NewCaledonia /NewCaledonia-Black /NewCaledonia-BlackItalic /NewCaledonia-Bold /NewCaledonia-BoldItalic /NewCaledonia-BoldItalicOsF /NewCaledonia-BoldSC /NewCaledonia-Italic /NewCaledonia-ItalicOsF /NewCaledonia-SC /NewCaledonia-SemiBold /NewCaledonia-SemiBoldItalic /NewCenturySchlbk-Bold /NewCenturySchlbk-BoldItalic /NewCenturySchlbk-Italic /NewCenturySchlbk-Roman /NewsGothic /NewsGothic-Bold /NewsGothic-BoldOblique /NewsGothicBT-Bold /NewsGothicBT-BoldCondensed /NewsGothicBT-BoldCondItalic /NewsGothicBT-BoldExtraCondensed /NewsGothicBT-BoldItalic /NewsGothicBT-Demi /NewsGothicBT-DemiItalic /NewsGothicBT-ExtraCondensed /NewsGothicBT-Italic /NewsGothicBT-ItalicCondensed /NewsGothicBT-Light /NewsGothicBT-LightItalic /NewsGothicBT-Roman /NewsGothicBT-RomanCondensed /NewsGothic-Oblique /New-Symbol /NovareseITCbyBT-Bold /NovareseITCbyBT-BoldItalic /NovareseITCbyBT-Book /NovareseITCbyBT-BookItalic /Nueva-BoldExtended /Nueva-Roman /NuptialScript /OceanSansMM /OceanSansMM-It /OfficinaSans-Bold /OfficinaSans-BoldItalic /OfficinaSans-Book /OfficinaSans-BookItalic /OfficinaSerif-Bold /OfficinaSerif-BoldItalic /OfficinaSerif-Book /OfficinaSerif-BookItalic /OnyxMT /Optima /Optima-Bold /Optima-BoldItalic /Optima-BoldOblique /Optima-ExtraBlack /Optima-ExtraBlackItalic /Optima-Italic /Optima-Oblique /OSPIRE-Plain /OttaIA /Otta-wa /Ottawa-BoldA /OttawaPSMT /Oxford /Palatino-Bold /Palatino-BoldItalic /Palatino-Italic /Palatino-Roman /Parisian /Perpetua /Perpetua-Bold /Perpetua-BoldItalic /Perpetua-Italic /PhotinaMT /PhotinaMT-Bold /PhotinaMT-BoldItalic /PhotinaMT-Italic /PhotinaMT-SemiBold /PhotinaMT-SemiBoldItalic /PhotinaMT-UltraBold /PhotinaMT-UltraBoldItalic /Plantin /Plantin-Bold /Plantin-BoldItalic /Plantin-Italic /Plantin-Light /Plantin-LightItalic /Plantin-Semibold /Plantin-SemiboldItalic /Poetica-ChanceryI /Poetica-SuppLowercaseEndI /PopplLaudatio-Italic /PopplLaudatio-Medium /PopplLaudatio-MediumItalic /PopplLaudatio-Regular /ProseAntique-Bold /ProseAntique-Normal /QuaySansEF-Black /QuaySansEF-BlackItalic /QuaySansEF-Book /QuaySansEF-BookItalic /QuaySansEF-Medium /QuaySansEF-MediumItalic /Quorum-Black /Quorum-Bold /Quorum-Book /Quorum-Light /Quorum-Medium /Raleigh /Raleigh-Bold /Raleigh-DemiBold /Raleigh-Medium /Revival565BT-Bold /Revival565BT-BoldItalic /Revival565BT-Italic /Revival565BT-Roman /Ribbon131BT-Bold /Ribbon131BT-Regular /RMTMI /Rockwell /Rockwell-Bold /Rockwell-BoldItalic /Rockwell-Italic /Rockwell-Light /Rockwell-LightItalic /RotisSansSerif /RotisSansSerif-Bold /RotisSansSerif-ExtraBold /RotisSansSerif-Italic /RotisSansSerif-Light /RotisSansSerif-LightItalic /RotisSemiSans /RotisSemiSans-Bold /RotisSemiSans-ExtraBold /RotisSemiSans-Italic /RotisSemiSans-Light /RotisSemiSans-LightItalic /RotisSemiSerif /RotisSemiSerif-Bold /RotisSerif /RotisSerif-Bold /RotisSerif-Italic /RunicMT-Condensed /Sabon-Bold /Sabon-BoldItalic /Sabon-Italic /Sabon-Roman /SackersGothicLight /SackersGothicLightAlt /SackersItalianScript /SackersItalianScriptAlt /Sam /Sanvito-Light /SanvitoMM /Sanvito-Roman /Semitica /Semitica-Italic /SIVAMATH /Siva-Special /SMS-SPELA /Souvenir-Demi /Souvenir-DemiItalic /SouvenirITCbyBT-Demi /SouvenirITCbyBT-DemiItalic /SouvenirITCbyBT-Light /SouvenirITCbyBT-LightItalic /Souvenir-Light /Souvenir-LightItalic /SpecialAA /Special-Gali /Sp-Sym /StempelGaramond-Bold /StempelGaramond-BoldItalic /StempelGaramond-Italic /StempelGaramond-Roman /StoneSans /StoneSans-Bold /StoneSans-BoldItalic /StoneSans-Italic /StoneSans-PhoneticAlternate /StoneSans-PhoneticIPA /StoneSans-Semibold /StoneSans-SemiboldItalic /StoneSerif /StoneSerif-Italic /StoneSerif-PhoneticAlternate /StoneSerif-PhoneticIPA /StoneSerif-Semibold /StoneSerif-SemiboldItalic /Swiss721BT-Black /Swiss721BT-BlackCondensed /Swiss721BT-BlackCondensedItalic /Swiss721BT-BlackExtended /Swiss721BT-BlackItalic /Swiss721BT-BlackOutline /Swiss721BT-BlackRounded /Swiss721BT-Bold /Swiss721BT-BoldCondensed /Swiss721BT-BoldCondensedItalic /Swiss721BT-BoldCondensedOutline /Swiss721BT-BoldExtended /Swiss721BT-BoldItalic /Swiss721BT-BoldOutline /Swiss721BT-BoldRounded /Swiss721BT-Heavy /Swiss721BT-HeavyItalic /Swiss721BT-Italic /Swiss721BT-ItalicCondensed /Swiss721BT-Light /Swiss721BT-LightCondensed /Swiss721BT-LightCondensedItalic /Swiss721BT-LightExtended /Swiss721BT-LightItalic /Swiss721BT-Medium /Swiss721BT-MediumItalic /Swiss721BT-Roman /Swiss721BT-RomanCondensed /Swiss721BT-RomanExtended /Swiss721BT-Thin /Swiss721BT-ThinItalic /Swiss921BT-RegularA /Symbol /Syntax-Black /Syntax-Bold /Syntax-Italic /Syntax-Roman /Syntax-UltraBlack /Tekton /Times-Bold /Times-BoldA /Times-BoldItalic /Times-BoldOblique /Times-Italic /Times-NewRoman /Times-NewRomanBold /TimesNewRomanPS-BoldItalicMT /TimesNewRomanPS-ItalicMT /TimesNewRomanPSMT /Times-Oblique /Times-PhoneticAlternate /Times-PhoneticIPA /Times-Roman /Times-RomanSmallCaps /Times-Sc /Times-SCB /Times-special /TimesTenGreekP-Upright /TradeGothic /TradeGothic-Bold /TradeGothic-BoldCondTwenty /TradeGothic-BoldCondTwentyObl /TradeGothic-BoldOblique /TradeGothic-BoldTwo /TradeGothic-BoldTwoOblique /TradeGothic-CondEighteen /TradeGothic-CondEighteenObl /TradeGothicLH-BoldExtended /TradeGothicLH-Extended /TradeGothic-Light /TradeGothic-LightOblique /TradeGothic-Oblique /Trajan-Bold /TrajanPro-Bold /TrajanPro-Regular /Trajan-Regular /Transitional521BT-BoldA /Transitional521BT-CursiveA /Transitional521BT-RomanA /Transitional551BT-MediumB /Transitional551BT-MediumItalicB /Univers /Universal-GreekwithMathPi /Universal-NewswithCommPi /Univers-BlackExt /Univers-BlackExtObl /Univers-Bold /Univers-BoldExt /Univers-BoldExtObl /Univers-BoldOblique /Univers-Condensed /Univers-CondensedBold /Univers-CondensedBoldOblique /Univers-CondensedOblique /Univers-Extended /Univers-ExtendedObl /Univers-ExtraBlackExt /Univers-ExtraBlackExtObl /Univers-Light /Univers-LightOblique /UniversLTStd-Black /UniversLTStd-BlackObl /Univers-Oblique /Utopia-Black /Utopia-BlackOsF /Utopia-Bold /Utopia-BoldItalic /Utopia-Italic /Utopia-Ornaments /Utopia-Regular /Utopia-Semibold /Utopia-SemiboldItalic /VAGRounded-Black /VAGRounded-Bold /VAGRounded-Light /VAGRounded-Thin /Viva-BoldExtraExtended /Viva-Regular /Weidemann-Black /Weidemann-BlackItalic /Weidemann-Bold /Weidemann-BoldItalic /Weidemann-Book /Weidemann-BookItalic /Weidemann-Medium /Weidemann-MediumItalic /WindsorBT-Elongated /WindsorBT-Light /WindsorBT-LightCondensed /WindsorBT-Roman /Wingdings-Regular /WNCYB10 /WNCYI10 /WNCYR10 /WNCYSC10 /WNCYSS10 /WoodtypeOrnaments-One /WoodtypeOrnaments-Two /ZapfCalligraphic801BT-Bold /ZapfCalligraphic801BT-BoldItal /ZapfCalligraphic801BT-Italic /ZapfCalligraphic801BT-Roman /ZapfChanceryITCbyBT-Bold /ZapfChanceryITCbyBT-Demi /ZapfChanceryITCbyBT-Medium /ZapfChanceryITCbyBT-MediumItal /ZapfChancery-MediumItalic /ZapfDingbats /ZapfDingbatsITCbyBT-Regular /ZapfElliptical711BT-Bold /ZapfElliptical711BT-BoldItalic /ZapfElliptical711BT-Italic /ZapfElliptical711BT-Roman /ZapfHumanist601BT-Bold /ZapfHumanist601BT-BoldItalic /ZapfHumanist601BT-Demi /ZapfHumanist601BT-DemiItalic /ZapfHumanist601BT-Italic /ZapfHumanist601BT-Roman /ZapfHumanist601BT-Ultra /ZapfHumanist601BT-UltraItalic /ZurichBT-Black /ZurichBT-BlackExtended /ZurichBT-BlackItalic /ZurichBT-Bold /ZurichBT-BoldCondensed /ZurichBT-BoldCondensedItalic /ZurichBT-BoldExtended /ZurichBT-BoldExtraCondensed /ZurichBT-BoldItalic /ZurichBT-ExtraBlack /ZurichBT-ExtraCondensed /ZurichBT-Italic /ZurichBT-ItalicCondensed /ZurichBT-Light /ZurichBT-LightCondensed /ZurichBT-LightCondensedItalic /ZurichBT-LightExtraCondensed /ZurichBT-LightItalic /ZurichBT-Roman /ZurichBT-RomanCondensed /ZurichBT-RomanExtended /ZurichBT-UltraBlackExtended ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages true /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >>
/ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >>
/JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >>
/JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >>
/AntiAliasGrayImages false
/CropGrayImages true
/GrayImageMinResolution 150
/GrayImageMinResolutionPolicy /OK
/DownsampleGrayImages true
/GrayImageDownsampleType /Bicubic
/GrayImageResolution 300
/GrayImageDepth -1
/GrayImageMinDownsampleDepth 2
/GrayImageDownsampleThreshold 1.50000
/EncodeGrayImages true
/GrayImageFilter /DCTEncode
/AutoFilterGrayImages true
/GrayImageAutoFilterStrategy /JPEG
/GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >>
/GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >>
/JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >>
/JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >>
/AntiAliasMonoImages false
/CropMonoImages true
/MonoImageMinResolution 1200
/MonoImageMinResolutionPolicy /OK
/DownsampleMonoImages true
/MonoImageDownsampleType /Bicubic
/MonoImageResolution 1200
/MonoImageDepth -1
/MonoImageDownsampleThreshold 1.50000
/EncodeMonoImages true
/MonoImageFilter /CCITTFaxEncode
/MonoImageDict << /K -1 >>
/AllowPSXObjects false
/CheckCompliance [
/None
]
/PDFX1aCheck false
/PDFX3Check false
/PDFXCompliantPDFOnly false
/PDFXNoTrimBoxError true
/PDFXTrimBoxToMediaBoxOffset [
0.00000
0.00000
0.00000
0.00000
]
/PDFXSetBleedBoxToMediaBox false
/PDFXBleedBoxToTrimBoxOffset [
0.00000
0.00000
0.00000
0.00000
]
/PDFXOutputIntentProfile (U.S. Web Coated \050SWOP\051 v2)
/PDFXOutputConditionIdentifier ()
/PDFXOutputCondition ()
/PDFXRegistryName (http://www.color.org)
/PDFXTrapped /Unknown
/CreateJDFFile false
/SyntheticBoldness 1.000000
/Description << /FRA
/JPN
/DEU
/PTB
/DAN
/NLD
/ESP
/SUO
/ITA
/NOR
/SVE
/ENU (Use these settings for creating PDF files for submission to The Sheridan Press. These settings configured for Acrobat v6.0 08/06/03.)
>>
>> setdistillerparams
<< /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice