Application 2 – Annotated Bibliography

Application 2 – Annotated Bibliography

As part of your doctoral seminar for this set of weeks, you are participating in a seminar-style discussion about the weekly topics. Recall that you were asked to address 5 of the Required Resources and at least 5 additional resources from the Walden Library and to incorporate them into your posting. As a related exercise, submit an annotated bibliography of the 10 resources you referred to this week. For each entry, be sure to address the following as a minimum:

· Include the full APA citation

· Discuss the scope of the resource

· Discuss the purpose and philosophical approach

· Discuss the underlying assumptions

· If referring to a research reporting article, present the methodology

· Relate the resource to the body of resources you have consulted in this course

· Discuss any evident limitations and opportunities for further inquiry

忝�

contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 113

d o i : 1 0 . 1 1 4 5 / 1 5 6 2 1 6 4 . 1 5 6 2 1 9 4

by achita (mi) muthitacharoen and Khawaja a. saeed

M s . P e r e z w a s g i v i n g a P o w e r P o i n t P r e s e n ta t i o n
to her potential clients in the hope of landing a
big contract. She was presenting a new advertising
campaign for a mutual fund company and had
spent three months with her team on perfecting the
proposal. Everything seemed to be going well when
suddenly a small window screen popped up informing
her that an error had occurred and asked if she would
wish to send an error report. She clicked the send
button and the application on her laptop shut down,
disrupting the flow of her presentation and making
her look unprofessional.

This story entails an example of a
user’s experience and response to a new
method for collecting information on
software application errors. To main-
tain a certain level of quality and ensure
customer satisfaction, software firms
spend approximately 50% to 75% of
the total software development cost on
debugging, testing, and verification ac-
tivities. Despite such efforts, it is not un-
common for a software application to
contain errors after the final version is
released. To better manage the software
development process in the long run
firms are involving the users in software
improvement initiatives by soliciting er-
ror information, while they are using the
software application. The information
collected through an error reporting
system (ERS) plays an important role in
uncovering bugs and prioritizing future
development work. Considering that
about 20% of bugs cause 80% of the er-
rors,10 gathering information on appli-
cation errors can substantially improve
software firms’ productivity and im-
prove the quality of their products. High
quality software applications can ben-
efit the software users individually and
also help improve the image of the soft-
ware community as a whole. Thus, un-
derstanding the emerging error report-
ing systems (ERS) and why users adopt
them are important issues that require
examination. Such an analysis can help
the software companies in learning how
to design better ERS and educate the us-
ers about ERS and its utilities.

error reporting systems
Traditional error reporting methods
follow a passive approach, wherein the
users are required to directly contact
the software company or to transmit
error information via a third party ven-
dor (such as CERT/CC, SecurityForcus).
CERT/CC acts as a mediator to collect
such information and delivers it to the
software companies. In general, soft-
ware firms are given a limited time to
fix errors and provide patches to its
users before CERT/CC releases the er-
ror information to the public domain.
Such a situation occasionally pressures

examining
user
involvement
in continuous
software
development
(a case of error
reporting system)

114 c o m m u n i c at i o n s o f t h e a c m | s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9

contributed articles

for Windows and .NET connected lo-
gos. It is important to recognize that
ERS is not the cause of an application
error. It is an outcome of the error and
a mechanism that allows the user to
notify Microsoft regarding the applica-
tion faults, kernel fault, or unrespon-
sive applications. Participation in the
error reporting activities does not im-
mediately solve the error. The infor-
mation sent through the ERS varies,
depending on the embedded function
in the ERS. However, Microsoft claims
that ERS enables it to solve the prob-
lems more efficiently and effectively
in the long run.7,10 While using ERS
can greatly benefit the software com-
munity as a whole, little has been done
to investigate user perceptions and be-
havior towards the ERS.

examining error
reporting behavior
Prior studies on continuous software
development have examined the prac-
tice of disclosing error information by
third-party organizations (for example,
CERT/CC, Security Focus) and its im-
pact on the speed of patch release by
software vendors.5 Some recent studies
have examined the difference in speed
of patch release across open source
vendors vs. closed source vendors.1
A critical issue related to the speed of
patch release is the quick accumula-
tion of information about the error or
deficiency that can lead to the develop-
ment of the patch. While users are the
main source of this information, it is
unclear why they report errors through

the software companies to release their
patches prematurely.

A more proactive approach involves
the use of ERS. It is a pioneer system
that enables the users of software ap-
plications to send error related infor-
mation, when they are using a software
application. ERS, also known as Online
Crashing Analysis, is a relatively new
functionality integrated into many sys-
tem platforms (such as Windows XP
and Vista, Mozilla, and Mac).11 Differ-
ent platforms use different techniques,
which vary in the degree of automation
of the ERS. For instance, the ERS used
by the GNOME platform requests its us-
ers to fully describe the incidents. Oth-
er systems have a higher degree of au-
tomation, where the ERS automatically
collects the information and prompts
the user to send the error report.11

Although ERS can increase the po-
tential for companies to under invest
in quality control processes, anecdotal
evidence suggests that information
collected through the ERS has been put
to good use. Windows XP SP1 team, for
instance, was able to fix approximately
29% of all Windows XP errors based
on information collected through the
ERS.12 Further, ERS has been instru-
mental in fixing problems found in
beta version of Windows Vista and MS
office 2007.12 The release of Windows
Vista and Office 2007 was delayed be-
cause Microsoft took the inputs from
beta users seriously.7,12 The growing
popularity of ERS is also visible in new
computer products such as Dell desk-
top and laptop computers. Dell Com-
puter Inc. is using a system similar
to ERS to collect software incompat-
ibilities information and information
about hardware problems.

ERS properties in Windows XP and
Vista can be configured by accessing
system properties, clicking on the ad-
vanced tab (XP)/ advance system set-
ting (Vista), and then selecting error-
reporting tab (XP)/ system startup and
recovery settings (Vista). dwwin.exe
(an operating system utility) creates
a crash report, when an error occurs.
Subsequently the utility triggers a pop-
up window, prompting the users to
send the error report. Windows error
reporting system is a part of the Win-
dows Quality Online Service (Winqual)
program, which focuses on hardware
and software that bears the designed

the ERS. ERS, unlike other avenues to
collect error information, is a proactive
system and its appearance is triggered
when the software in use experiences a
bug or a malfunction. Understanding
user perceptions regarding ERS can
help the designers in developing more
usable ERS and better promote the ERS
in the user community, consequently
increasing the likelihood of getting
better user participation.

In this study, we categorized fac-
tors that influence ERS usage intention
into system-design group and human-
computer-interaction (HCI) group.
Organizing the factors into these two
categories allows us to examine the
differences across the two groups and
later provide meaningful guidelines on
the effective promotion and design of
the ERS (See Figure 1). We elaborate on
the selected factors next.

System-design factors capture de-
sign decisions that can be controlled
by ERS designers. ERS initiates interac-
tion with the users by sending a pop-up
window, indicating that a problem has
occurred in the software application.
Users consequently have the freedom
to choose whether or not to send the
error report. While interacting with
the ERS, the users may expect to be in-
formed about the type of data that will
be submitted (Data Transparency), how
the data will be processed, accessed,
and transmitted (Process Transparen-
cy), and expect feedback on the value of
their inputs (Feedback Transparency).
ERS designers control the decision on
whether to provide such information

figure 1: ers critical factors

User’s Intention
to Use ERS

Human-Computer Interaction
(HCI ) Issues
Motivation

• Expectation in Better Product
User’s task

• Role Clarity
• Value Fit or Value Compatibility

Synchronization
• Work Interruption
• Work Compatibility

System Design Issues

• Data Transparency
• Process Transparency
• Feedback Transparency



contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 115

to the users. Further, the proactive na-
ture of ERS increases the importance
of these transparencies.8,9 It is likely an
ERS that is configured to be transpar-
ent along these factors increases the
user’s sense of control.9 Having limited
or no transparency can undermine us-
er’s trust in and response to the ERS. In
short, adequate transparency in data,
process, and feedback can help pro-
mote user’s ERS usage intentions.

While design elements are important
in explaining how users respond to the
ERS, HCI literature indicates that user’s
cognitive processes also play a criti-
cal role. ERS is similar to a solicitation
system. The user’s decision to respond
to the solicitation can be influenced by
their reaction to a request for free infor-
mation, how the request is made, why
the request is made, and their assess-
ment of the benefits of responding to
the request. We categorized issues relat-
ed to the HCI group into a) user’s moti-
vation,4 b) user’s task,2,4 and c) synchro-
nization. User’s motivation is driven by
the likely benefit of using the ERS. The
interaction between a user and the ERS
is short and apparently does not pro-
vide the users with any instant benefits.
However, in the long run, the user can
expect software applications with fewer
bugs. Such an expectation may serve
as a more realistic view of the benefits
from using ERS. In term of user’s tasks,
ERS users may question their role and
appropriateness of the approach for re-
questing information. Value fit ties to
an individual’s belief that soliciting in-
formation on errors is appropriate and

they should comply with the request.
Similarly, if the users are clear about
their role as a contributor to the overall
application development process, their
likelihood of responding to the ERS can
be enhanced. Thus, role clarity and val-
ue fit are both important consideration.
HCI literature also highlights synchro-
nization between user’s task and tech-
nology as an important driver. Work on
examining timing of occurrence and
compatibility issues have been brought
forward by studies on the assessment of
advertising pop-up window.2,3 Like the
advertising pop-up window, ERS win-
dow can interrupt the user’s on-going
activities and is likely to receive a nega-
tive response from the users.

The review here brings forward eight
factors, categorized into either system-
design group or HCI group (See Figure
1). We examine the influence of these
factors on user’s intention to use the
ERS. We collect ERS user’s perception
survey from 317 respondents. Nine (9)
samples were later removed from the
data set since they reported no previ-
ous exposure to the ERS, rendering a
final sample of 308. Considering the
sample, caution should be taken in ap-
plying the results to computer users
outside the US. The sample includes
180 males (57%) and 137 females (43%)
with an average age of 25 years. The
respondents use the computer for 29
hours per week on average and have an
average computer usage experience of
11 years. Majority of respondents (86%)
have high speed Internet access. Sixty
two percent (62%) of the respondents

report that using MS Internet explorer
generated the error reports, followed
by Microsoft office applications (10%
of the respondents). The rest of the
sample (28%) report experiencing the
error report window while using other
applications (such as, Media Players,
Netscape, and Windows).

drivers of ers usage intentions
Descriptive statistics reveals that users
in general rated system-design factors
lower than factors in the HCI group
(Table 1). Users seem to be clear about
their roles in interacting with the ERS
but find the ERS process ambiguous.
They also reported that ERS does not
provide sufficient feedback. Further, us-
ers expect to get an improved software
application in the future and perceive
ERS solicitation as appropriate but also
state that the ERS disrupts their work.
Table 1 summarizes the impact (total
effect)a of system-design and HCI fac-
tors on user’s ERS usage intentions.b Of
the proposed eight factors, seven fac-
tors show a significant total effect on
user’s ERS usage intentions.

The analysis of total effect indicated
that feedback transparency is the only
factor that does not influence ERS us-
age intention. According to the respon-
dents, ERS does not provide sufficient
feedback (low mean value) and con-
sequently they may not consider get-
ting feedback as an important factor
in making the decision to use the ERS.
This result may be attributed to the
proactive nature of the ERS. Users gen-
erally do not initiate the communica-
tion process with the ERS and may not
consider getting feedback a realistic
expectation. Total effect also provides
information about the relative impor-
tance of factors in driving user’s ERS
usage intentions. The results show that
users consider process transparency
more important than data transpar-
ency. Existing ERS enables the users to
view the data, but in its current form
this information is difficult to under-
stand. Data needs to be presented in a
fashion that is comprehensible to the

a Total effect (a sum of direct and indirect effect) is a
measure of overall influence of a variable on another. A
higher absolute value of total effect depicts a stronger
influence.

b SEM (structural equation modeling) was used to compute
and test the total effect of user’s perception on their
intention to send the ERS.

table 1: relative importance of each contributing factor on user’s intention to send
an error report

Group factors mean*
(std dev)

total
effect

ranking of
importance
by factors

System-Design Issues Data Transparency 3.58 (1.78) 0.10** 7

Process Transparency 2.56 (1.53) 0.23** 4

Feedback Transparency 2.87 (1.70) 0.02 8

HCI Issues Work Compatibility 3.29 (1.53) 0.19** 5

Value Compatibility 4.56 (1.56) 0.47** 1

Role Clarity 4.93 (1.45) 0.30** 2

Work Interruption*** 4.76 (1.73) -0.25** 3

Better Product Expectation 4.67 (1.39) 0.16** 6

* 1(low) – 7 (high) scale was used to measure the factors
** indicates factors with significant total effect on intention to send error report
*** All of the proposed variables, except feedback transparency, were found to promote ERS usage intentions. For instance,
a higher degree of data clarity (high data transparency) increases the user’s intention to use the ERS. On the contrary, if
users perceive the ERS to cause work interruption, it actually lowers their intention to use the ERS.

116 c o m m u n i c at i o n s o f t h e a c m | s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9

contributed articles

the application in forming their overall
perception about time commitment in
reporting the error.

developing and designing error
reporting systems
The study offers interesting insights
that can guide the promotion and re-
configuration of the ERS. First and
foremost, we believe that ERS needs
to be better promoted as a system that
assists in improving the quality of soft-
ware applications. If the users perceive
it as a system that benefits the overall
user community, they will be more
inclined to use the ERS. Emphasiz-
ing the role of the user in software im-
provement through a message on ERS
pop-up windows can also improve the
chances of getting better responses.
Further, providing concise information
about data collected by the ERS and its
processing can alleviate user concerns
and improve participation.

Considering that users highlighted
work interruption and time commit-
ment as important issues, we believe
that delineating the error report from
the overall process of application shut-
down and restart is critical. Providing
the choice to the users to automatically
send the error report at the time of in-
stallation or aggregate the errors and
report the errors all at once at their con-
venience are interesting options that
designers can explore. Recent practice
of similar technology also shows that
ERS designers should consider relocat-
ing ERS windows on the user’s screen.
Dell Computer Inc., for example, em-
ploys similar technology and the ERS
window appears on the right bottom

users. Further, user’s lack of control
over what information is transmitted
is another contingency that influences
the important of data transparency.

Assessment of factors in the HCI
category shows that user’s perception
regarding the appropriateness of col-
lecting information (value compat-
ibility) through the ERS is the most
important consideration, followed by
user’s perceived clarity of their role in
interacting with the ERS. User’s percep-
tion regarding work interruption nega-
tively impacts intention to use the ERS
and ranks third in terms of importance.
Work interruption is the only factor that
negatively impacts ERS usage intention.
Thus, users perceive ERS as a disruptive
system that interrupts them and such a
characteristic has a detrimental effect
on user’s intention to respond to the
ERS. Other factors, such as work com-
patibility and expectation of getting a
better product in the future were found
less important. Overall, factors in the
HCI group play a relatively more impor-
tant role in influencing the user’s inten-
tion than factors related to the system
design group. So, tweaking the design
elements of the ERS can help, but pro-
moting ERS as an appropriate system
and educating the users on their role in
interacting with ERS constitutes a more
promising approach.

analysis of ers usage
Table 2 shows a cross-tab analysis of
user’s frequency of sending an error
report and the frequency of their ex-
posure to the ERS. A majority of the
respondents (53.6%) stated that they
encounter the error report window oc-
casionally, 21.1% receive the window
very frequently, and 7.6% receive the
window all the time. Respondents were
almost uniformly divided in terms of
the frequency of which they send an
error report. Assessment of the user’s
exposure and response rate to the ERS
indicates that the user responds to the
ERS more often, if the ERS window ap-
pears more frequently. For instance,
19% of the users who rarely receive the
error report said that they send the er-
ror report either very frequently or all
the time. The percentage increases for
the group of users who occasionally re-
ceive the ERS to 40%. The percentage
continues to increase to 64% and 80%
for the group of users who received the

ERS very frequently and all the time.
Overall, the analysis shows that us-
ers that receive the error report more
frequently are more likely to respond
to the request. It can be argued that,
over time, users develop automatic re-
sponse patterns or reflexive behavior in
their interaction with the ERS. They will
adhere to a course of action once they
make a decision. In clicking the send
button, these users may not reflectively
assess the issues in the HCI and system
design domain but rather interact with
the ERS in the form of habit. Further
examination of reflexive behavior in the
context of ERS offers an interesting av-
enue for future research in this area.

Qualitative responses to an open
ended question that was included in
the survey shows that majority of the
respondents are more likely to use ERS
if their applications continue to expe-
rience problems. This result is consis-
tent with our previous discussion on
the relationship between frequency
of ERS exposure and users’ response
rate to the ERS. Assurance from the
software manufacturers that the prob-
lems will be resolved was the second
most reported issue. So, users’ expec-
tation regarding feedback (feedback
transparency) in general may be not
important, but they are interested in
knowing how their inputs help in im-
proving the software application (spe-
cific type of feedback). The third most
cited issue was time constraint. While
participating in the ERS may take only
a few seconds, users might take into
consideration the overall process of
responding to the ERS, shutting down
of the application, and restarting of

table 2: cross-tab analysis of users responses to the ers

frequency of receiving
a error report

frequency of sending error report

Never Rarely Occasionally Very
Frequently

All the
time

Total

Never 9 0 0 0 0 9 (2.9%)

Rarely 12 19 7 3 6 47
(14.8%)

Occasionally 27 23 52 33 35 170
(53.6%)

Very frequently 6 10 8 36 7 67
(21.1%)

All the time 3 2 0 1 18 24
(7.6%)

Total 57
(18.0%)

54
(17.0%)

67
(21.2%)

73
(23.0%)

66
(20.8%)

317
(100%)

Note: Nine (9) responses were removed from the sample set and they are not used in the analysis due to their lack of
exposure to the ERS, rendering a final sample of 308.

contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 117

corner of the user’s screen. Further,
ERS designers should consider ways
to minimize the redundancy of the er-
ror reports. Reoccurrence of errors can
undermine user’s trust in the utility of
the ERS. ERS should be configured to
recognize errors and limit the genera-
tion of an error report, if the user has
already responded to the same error on
the same application. We summarized

our recommendations and guidelines
for ERS users, ERS designers, and soft-
ware firms in Table 3.

Our study offers a preliminary ex-
amination of ERS from a user’s percep-
tive and there are many other aspects
of ERS that call for further research. For
instance, prior ERS usage experience,
user’s disposition towards the software
firm, whether/how the system-design
factors effect the HCI factors, and the
significance of the software applications
to users’ job performance are promising
topics in this area. Formally examining
the influence of the qualitative factors is
another interesting issue that needs to
be assessed. Examining the variation in
the user’s error reporting behavior and
its drivers across open source vs. closed
source software applications also re-
quires attention. Error reporting behav-
ior may as well be impacted by the differ-
ence in cultural association of users. For
example, users from different countries
may have different perceptions regard-
ing the ERS and ERS behavior. Finally,
it will be interesting to assess the condi-
tions in which cognitive process drive us-
ers interaction with the ERS as opposed
to reflexive responses. We hope that the
avenues for future research outlined in
the study provide an impetus to further
research on the ERS.

It is important to note that the recom-
mendations from this study are based
on data collected from the users of cur-
rent ERS technologies. Future studies
can pursue alternative perspectives. For

example, understanding the impact of
ERS on quality control processes within
the software firms is an interesting area.
Overall, ERS is a beneficial technol-
ogy that can help in improving software
quality, but its implementation can also
encourage companies to compromise
on the comprehensiveness of software
testing. Further, as with any other infor-
mation system, understanding develop-
ment issues and the designer’s view is
important. Thus, future works on costs
and benefits associated with ERS con-
figuration and promotion are encour-
aged. Research on these topics along
with the results outlined in this study
can provide a more complete picture
and more comprehensive guidelines on
prioritizing promotion and reconfigu-
ration initiatives related to the ERS. Like
other information systems, the success
of ERS will be defined by its user accep-
tance and how software firms use/abuse
it to their short-term and/or long term
advantages.

References
1. Arora, A. Krishnan, R., Telang, R., and Yang, Y. An

empirical analysis of software vendors’ patching
behavior: Impact of vulnerability disclosure. In
Proceedings of the 27th International Conference on
Information Systems, (Milwaukee, WI), (2006). 307-
322.

2. Cho, C.H. and Cheon, J.J.Why do people avoid
advertising on the Internet. J. of Advertising 33, 4,
(2004), 89-97.

3. Edwards, S. M., Lee, J. H., and Li, H. Forced exposure
and psychological reactance: Antecedents and
consequences of the perceived intrusiveness of pop-up
ads. J. of Advertising 31, 3, (Fall 2002), 83-35.

4. Karat, J. Karat, C.M. and Ulelson, J. Affordances,
motivations, and the design of user interfaces, Comm.
ACM 43, 8, (2000), 49-51.

5. Kannan K. and Telang, R. Market for software
vulnerabilities? Think again, Management Sciences 51,
5, (2005), 726-740.

6. Lohr, S. Microsoft software updates go on sale to
business, The New York Times, (Dec. 1, 2006).

7. Microsoft Corps; http://msdn.microsoft.com/library/
default.asp?url=/library/en-us/debug/base/windows_
error_reporting.asp, (2006).

8. Nambisan, S. Designing virtual customer
environments for new product development: Toward a
theory. Academy of Management Review 27, 3, (2002),
392-413

9. Porra, J. Colonial System. Information Systems
Research 10, 1, (1999), 38-69.

10. Sayer, P. Microsoft Still Bugged by Software
Problems: Office XP’s error reporting technology is
helping the software giant clean its code, but more
work is needed; http://www.pcworld.com/news/
article/0,aid,105599,00.asp (2002).

11. Wikipedia, Crash Reporter; http://en.wikipedia.org/wiki/
Crash_reporter, (2006).

12. Wikipedia, Windows Error Reporting; http://en.
wikipedia.org/wiki/Windows_Error_Reporting, (2007).

Achita (Mi) Muthitacharoen (achita.muthita@wichita.
edu) is an associate professor in MIS at Wichita State
University, Wichita, KS.

Khawaja A. Saeed (khawaja.saeed@wichita.edu) is an
associate professor of Information Systems in the W.
Frank Barton School of Business at the Wichita State
University, Wichita, KS.

table 3: Guidelines to promote user’s participation in the ers

GuideLines

ers designers
can experiment
with the idea of:

• Reducing the redundant pop-up windows if the system finds that the same error has
already been reported.

• Relocating the ERS window on the user’s screen to a location that is less interruptive.
• Informing users about the purpose of the ERS (i.e. include a statement in the

installation phase or use media to publicize).
• Explaining to the users, on the pop-up window screen, what information will be

transmitted.
• Providing the users with an option to send the error report automatically.
• Aggregating the error reports which can be submitted all at once at the user’s

convenience.
• Segregating the application shut down and initiation from the reporting of the error.

software
manufacturers
should
consider:

• Building an image of the ERS as an appropriate and commonly used approach to
solicit information.

• Promoting the importance of the ERS as a system that can help improve the quality of
the software application.

• Publicizing the role of the users as participants in the software development process.
• Reassuring the users that the problem will be fixed in a timely manner.
• Working together with companies who adopt the software applications to provide

users with tangible incentives to participate in the ERS.

ers users
should
understand
that:

• Higher usage of the ERS can help in improving software quality.
• Using the ERS can improve software development and improvement processes at the

software firms.
• ERS does not cause the error, but is an outcome of it.
• ERS can be disabled in almost all platforms.

research method
We adopted survey method to test
system-design and HCI factors. Questions
in the survey were adopted from previous
studies and new questions were developed
for some factors.

The development and refinement of
the questionnaire was conduced in
three stages. The first stage involved a
thorough assessment of the literature to
understand the conceptual structure of
the factors. In the second stage two pilot
studies were conducted. In the first pilot
study subjects were informed about the
objectives of the study. Then, the subjects
completed the questionnaire. A picture
of the error report and a brief description
of the project were presented on the first
page of the questionnaire. The subjects
provided feedback on the instrument and
issues that they thought were important
in the context of the ERS at the end of the
questionnaire. After making modifications
based on the results of the first pilot study,
the second pilot study was conducted to
further refine the instrument.

Later, a final survey was distributed to
collect the data. Three hundred and
seventeen (317) responses were collected
from graduate and undergraduate
students at two universities in the US.

SOFTWARE PROCESS IMPROVEMENT AND PRACTICE
Softw. Process Improve. Pract. 2009; 14: 65 – 83

Published online 24 January 2008 in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/spip.376

Continuous Process
Improvement at a Large
Software Organization

Research Section
Viviane Malheiros1,2,†* Fábio Rilston Paim1 and
Manoel Mendonça3
1 Serviço Federal de Processamento de Dados (SERPRO) São Paulo, SP,
Brazil
2 USP, Universidade São Paulo, ICMC, São Paulo, SP, Brazil
3 NUPERC, Salvador University (Unifacs) Salvador, BA, Brazil

Designing software processes for large enterprises is a hard task, but evolving this process can
be even harder. As large companies increasingly invest in software process improvement, they
have to face typical problems regarding software process evolution within such environments in
order to guarantee their software process survival. This article describes experiences and lessons
learned at Serpro, the largest government owned software company in Latin America, while
defining its organizational standard software process and fostering its continuous evolution.

KEY WORDS: software process improvement; SPI practice; case study

1. INTRODUCTION

Several studies argue that the quality of a soft-
ware system is highly influenced by the quality of
the process used to develop it (Fiorini et al. 2001,
IEEE Software Engineering Coordinating Commit-
tee 2001). Good software processes should help pro-
duce better, cheaper software faster. In fact, when
either a defined process is not available or when
such a process is defined but not used the chances
are that a software project will fail. Within such a
scenario, software process improvement becomes
an important challenge to modern organizations
(McFeeley 1996). In recent years, maturity models
such as CMMI (Capability Maturity Model Integration)
(SEI 2002) have played an important role in helping

∗ Correspondence to: Viviane Malheiros, Serviço Federal de
Processamento de Dados (SERPRO) São Paulo, SP, Brazil
† E-mail: viviane.malheiros@serpro.gov.br

old-style software companies to achieve higher lev-
els of maturity, in terms of both their development
process and product quality. According to Software
Engineering Institute (SEI), to be considered mature
an organization needs not only to define and use a
software process but also evolve it continuously. In
order to pursue such a goal, CMMI Maturity level
3 (named Defined) establishes the definition of an
organizational standard software process as well as
the basis for a wide software process improvement
(SPI) program by means of the following process
areas (PA): Organizational Process Definition (OPD)
and Organizational Process Focus (OPF). Both process
areas consolidate a set of practices related to contin-
uous evolution of an organizational development
process, which highlight the need to understand
and deal with weaknesses and strengths identified
throughout the process execution.

Process improvement can be harder and more
error-prone than defining an initial version, if such
a software process is to run in a large environment.
Several challenges must be faced in this context:

Research Section V. Malheiros, F. R. Paim and M. Mendonça

(i) identify procedures that can be established and
easily automated to promote effective treatment of
improvements, reverting them into process opti-
mization, without renouncing a thorough analy-
sis of its contents, (ii) incorporate improvements
into brand new software process releases while
maintaining the integrity of all process assets,
(iii) disseminate SPI efforts through several soft-
ware development units in an homogeneous way,
(iv) ensure fast absorption of changes and a stan-
dard process that fulfills each individual unit needs,
and (v) keep successive process releases respecting
best software management and development prac-
tices, without losing track of the reality of each
organizational unit.

This article describes experiences and lessons
learned by Serpro1, a very large software company
owned by the Brazilian Government, in defining
its organizational standard software process. The
resulting process is now used by more than
3000 developers in 40 development areas, widely
spread throughout the country. We will describe
the set of solutions adopted to overcome the
challenges mentioned above, which have enabled
the successful companywide adoption of Serpro’s
standard software process.

The rest of this article is organized as follows:
Section 2 presents related work within the SPI
field that we consider meaningful for understand-
ing some ideas discussed here. Section 3 briefly
describes Serpro’s profile as a software engineering
organization. Section 4 presents Serpro’s approach
to software process improvement, highlighting the
SPI program defined and conducted in the organiza-
tion, the alternatives selected for standard software
process management and the automated SPI control
adopted companywide. Section 5 analyzes SPI data
collected within the organization. Lastly, Section 6
presents our conclusions and future works.

2. SOFTWARE PROCESS IMPROVEMENT

Osterweil (1997) established an interesting analogy
between software (applications) and software pro-
cesses: ‘Processes and applications are both executed,
they both address requirements that need to be under-
stood, both benefit from being modeled by a variety of

1 SERPRO is an acronym for ‘Serviço Federal de Processamento de
Dados’ or Data Processing Federal Service.

sorts of models, both must evolve guided by measurement,
and so forth.’

In fact, software quality evolved historically from
a limited perspective that associated quality strictly
to final product quality (i.e. software free of errors)
to a novel one where process quality influences
software quality. As well as being error-free, soft-
ware should satisfy customer’s requirements and
meet cost and schedule goals. Powered by this
vision, software quality assurance has become well
accepted in government, industrial, and academic
communities. Practices for planning, engineering,
and managing software development and mainte-
nance have been shown to enhance organizations’
ability to meet goals for cost, schedule, functional-
ity, and product quality (SEI 1993). In this sense,
many works relate process quality to product qual-
ity (ISO 1991, SEI 1993, 2002, Gremba and Myers
1997, ISO 1998, Florac and Carleton 1999, Jacobson
et al. 1999, Henninger 2000).

Systematic approaches to manage software pro-
cess improvement have been proposed (Basili 1985,
Gremba and Myers 1997, INTRO 2002) and sev-
eral different process-related aspects have been
researched and published to date. Successful works
focus on compiling and organizing SPI best prac-
tices that, when implemented under certain matu-
rity criteria, are likely to promote high quality soft-
ware process construction, such as the CMM (Capa-
bility Maturity Model) (SEI 1993) and CMMI (SEI
2002) models. Others are structured as a flexible soft-
ware development process platform – for instance,
the Rational Unified Process (Kruchten 1999) – that
can be customized by enterprises to meet their orga-
nizational needs. Furthermore, some researchers
emphasize the importance of both reuse centered
and knowledge management approaches to soft-
ware development (Henninger 1999, IEEE Soft-
ware Engineering Coordinating Committee 2001,
Rus et al. 2001, Oliveira et al. 2004, Shull et al.
2004). The Software Engineering Body of Knowl-
edge – SWEBOK (IEEE Software Engineering Coor-
dinating Committee 2001) is of particular note in this
group as a guide to relevant software engineering
knowledge existing in literature.

Most of the above-mentioned approaches con-
verge to general knowledge management principles
in support of reuse by means of:

• adopting standard processes, with a few adap-
tations, to clarify ‘how to’ knowledge so that it

66 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

becomes easy to identify an organizational ‘way
of work’

• defining a standard way to store information.
Templates (standard, predefined artifacts) are
indicated to collect data and together enable
information reuse among software projects,
especially new ones. Such reuse may occur
in both end product and intermediate work
products.

Although applying development methods, pro-
cesses, and tools can lead to better software product
quality, inevitably it also tends to cause weaknesses
to appear which are related to either: (i) process
inadequacy for the internal reality or (ii) high costs
regarding task execution, not always aggregating
earned value to the final product or even (iii) process
assets that spread defects, failures, errors, or mis-
interpretations or a combination of two or more of
these factors.

In order to deal with and convert identified weak-
nesses into process improvement process mainte-
nance becomes necessary, just as with software.
Unintended effects that emerge in the course
of maintenance operations can be complex and
present widely spread symptoms throughout the
organization. Hence, systematic and specialized
approaches are needed to manage the SPI adop-
tion life cycle (Gremba and Myers 1997, INTRO
2002). Such approaches generally follow a Process
Improvement Model, which proposes a number of
steps toward improving software process quality.
Among these models are PDCA (Florac and Car-
leton 1999), CMMI (SEI 2002), IDEAL (Gremba and
Myers 1997), QIP (Basili 1985) and INTRO – IDEAL-
Based New Technology Rollout (INTRO 2002). On the
basis of these approaches several works report expe-
riences on: (i) the definition and adoption (Sutton
et al. 1995, Humphrey 1997, Brownsword et al. 2000,
Cockburn 2000, Morisio et al. 2000); and (ii) control
and improvement of software processes (Basili
et al. 1992, Tdvet and Gollofello 1995, Caputo 1998,
Wiegers 1998, Florac and Carleton 1999, Florac et al.
2000, Johnson and Brodman 2000, Keeni 2000, Pitter-
man 2000, Conradi and Fuggetta 2002, Goldenson
and Gibson 2003).

Some characteristics are common to all previous
approaches:

• they all claim process improvement must be
treated as a project, which means there must
be an assigned project manager; some project

planning should be done, resources should
be allocated, progress oversight should take
place across project phases, among other related
activities

• a continuous improvement cycle is adopted,
based on a gradual and evolving process

• all approaches incorporate within their pro-
cesses (or practices) a self-assessment activity
which is responsible for identifying improve-
ments to be considered in a next release

Besides having all the above characteristics,
Serpro’s Software Process Improvement approach
should be geared to a very large organization
that aims to have CMMI compliant software
development units. Some of the implications of
these requirements are discussed below.

2.1. The CMMI Reference Model

CMMI – Capability Maturity Model Integration – is a
reference model for software process improvement
that can be used by organizations to guide process
improvement efforts across a project, a division,
or an entire organization. CMMI helps integrate
traditionally separate organizational functions, sets
process improvement goals and priorities, provides
guidance for quality processes, and provides a point
of reference for appraising current processes (SEI
2002).

CMMI from its beginning has been a joint effort
of government and industry as well as the SEI.
It was built from the integration of other capa-
bility maturity models: Capability Maturity Model
for Software (SW-CMM), Systems Engineering Capa-
bility Model (SECM) and Integrated Product Devel-
opment Capability Maturity Model (IPD-CMM), into
a single improvement framework that accommo-
dates best practices from previous models, for use
by organizations pursuing enterprise-wide process
improvement.

Some of the benefits reported from adopt-
ing CMMI include: (a) improved project bud-
get and schedule prediction, (b) software prod-
uct cycle enhancement, (c) productivity growth,
(d) final quality improvement, in terms of num-
ber of defects found, (e) team motivation extended,
and (f) Return of Investment (ROI) can be calculated
using a variety of costs and aggregated benefits
(Haley 1996, Trienekens et al. 2001, Goldenson and
Gibson 2003, Trienekens et al. 2005).

DOI: 10.1002/spip 67

Research Section V. Malheiros, F. R. Paim and M. Mendonça

A mature software organization possesses an
organization-wide ability to manage software
development and maintenance processes. The soft-
ware process is accurately communicated to both
existing staff and new employees, and work activi-
ties are carried out according to the planned process.
The processes mandated are fit for use (Humphrey
1990) and consistent with the way the work actu-
ally gets done. These defined processes are updated
when necessary, and improvements are developed
through controlled pilot-tests and/or cost-benefit
analyses. Roles and responsibilities within the
defined process are clear throughout the project
and across the organization.

The CMMI process areas can be grouped into
four categories: (a) Process Management, (b) Project
Management, (c) Engineering, and (d) Support. The
Process Management categories group process
areas that contain cross-project activities related to
defining, planning, resourcing, deploying, imple-
menting, monitoring, controlling, appraising, mea-
suring, and improving processes. Although process
areas often interact and have an effect on each other,
regardless of the defined category, focus on the Pro-
cess Management group brings a useful overview of
the process improvement according to CMMI. The
basic Process Management process areas provide
the organization with a basic capacity to document
and share best practices, organizational process
assets, and learning across the organization (SEI
2002).

Formal SPI management start-up and a standard
software process establishment should occur with
the implementation of both OPD and OPF process
areas. Organizational training needs are also dealt
with across projects and support groups, such as
Organizational Training (OT) practices that are
incorporated into the company.

Process improvement evolution comes along with
the evolution of organization maturity itself. Higher
maturity levels require advanced Process Manage-
ment process areas, such as OPP (Organizational
Process Performance) and OID (Organizational Inno-
vation and Deployment Process), which provide the
organization with a stronger capability to achieve
its quantitative objectives for quality and process
performance (SEI 2002).

Reference models such as CMMI reflect their
authors’ concern with common organizational
issues regarding: (i) the set up of internal direc-
tives (ii) standard process definition, evolution,

and adjustment (iii) overall tracking of organi-
zational strategic goals (iv) process performance
and (v) innovation. Nevertheless, CMMI does not
attempt to prescribe specific life cycle models, orga-
nization structure models; or even the best tech-
niques or tools to be used in order to reach a higher
level of maturity. These issues should be conducted
in accordance with the specific business context and
objectives of a given organization (Paulk et al. 1995).

Few works refer to the additional effort required
by corporations when software development is per-
formed across several development units. These
organizations have to internally structure them-
selves to successfully incorporate best practices
proposed by a model. Such an internal arrangement
must guarantee that internalization takes place
across the entire institution and does not remain
restricted to ‘knowledge islands’ located in some of
its component units.

2.2. SPI Challenges in Large Organizations

Since the advent of SEI’s CMM, thousands of
software companies worldwide have applied effort
and money toward establishing their own defined
software process, powered by the promise of
systematic use of best technical and management
practices to successfully complete their projects
(Paim and Tavares 2002).

However, software process instantiation is a chal-
lenging task. Even more so in a large company
whose development process spans over several
engineering units countrywide. One example of
such an environment is found at Serpro, a soft-
ware company that develops solutions for several
government institutions by means of indepen-
dent development units distributed geographically
throughout Brazil. In such an environment, a soft-
ware process has to be conceived as both a straight-
forward set of tasks and as an evolving product to
the community involved. Additionally, one cannot
expect software process solutions to be successful in
large enterprises without being supported by a reli-
able SPI program from management to operational
levels.

Such an SPI program has to deal with issues
that often arise when applying development cycle
and standardization approaches across a number of
productive units that most frequently share project
phases:

68 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

1. How to make sure that the organizational stan-
dard process satisfies needs of each individual
unit within the corporation.

2. How to assure that the standard process holds
enough information to secure a minimal core
identity and consistent communication among
units, facilitating also team members’ changes
without unexpected shortcomings.

3. How to easily disseminate knowledge about
software process among thousands of develop-
ers geographically spread over several places
and with different cultures.

4. How to assess process internalization and
maturity in each individual organization.

5. How to increase the probability of a particular
organization success being repeated by another
organization.

6. How to offer proper conditions, as well as
efficient support to continuous software pro-
cess improvement, taking into consideration
variables such as: (a) development teams geo-
graphically distant from each other, (b) high
costs involved with implanting a software pro-
cess and promoting official SEI assessments
(CBA-IPI 2001, SEI 2002), (c) large numbers of
developers with highly diverse profiles.

7. And, ultimately, are practitioners working in
new, better ways (the bottom line in process
improvement) (Wiegers 1998)?

Section 3 describes Serpro’s SPI approach to deal
with the above-mentioned issues.

3. SOFTWARE PROCESS IMPROVEMENT
AT SERPRO

Serpro is the largest information technology and
communications (ITC) services company in Latin
America. It provides ITC services for the Brazilian
Government (SERPRO 2006) and operates in two
different business segments:

1. Public Finance – offering services for the Trea-
sury, its Secretariats, and other related depart-
ments

2. Federal Public Administration – providing
facilities to implement the set of structural-
ized and integrative actions conducted by the
Brazilian Planning, Budget and Management
Ministry and extended to other government
institutions that require customized ITC ser-
vices.

Over the past 10 years Serpro has diversified
its lines of business; however, most of its work
remains in the software development field. Among
its more than 10,000 employees, at least 30% are
dedicated to this business area. The company’s
main office is located in Brazil’s capital city, Brası́lia,
and there are ten other regional offices spread
across the country. Internally, Serpro is divided
into Management Units (MUs). These management
units are in turn organized into interest areas,
some of which are subdivided into business areas
(e.g. Tax Administration, Human Resources, National
Patrimony Administration, Foreign Trade Commerce),
and some according to technical services (data
centers, multiservice networks, client support, among
others). Each MU is managed by a Superintendency
and may have several branches across the country
(up to one for each regional office).

Apart from rare exceptions, each MU has its
own development teams geographically distributed
and software projects are carried out among teams
using an integrated approach. Each regional branch
of a MU is called a ‘Development Unit’ and is
administrated by a Senior Manager, supported by
a team of project leaders. There are currently more
than 40 development units at Serpro.

3.1. Serpro Solution Development Process
(PSDS)

Software development for the Brazilian Federal
Government demands a robust and efficient soft-
ware development and maintenance process, as
well as fresh, specialized business knowledge due
to its diversity. It also has delivery dates and restric-
tions often determined by law and long system
lifespan requirements (Malheiros et al. 2002).

Serpro solutions use a multitude of technologies
and are built on a variety of platforms, using a
variety of operational systems, programming lan-
guages (e.g. Java, Visual Basic, C, Natural, Cobol,
Python), development paradigms, among others.
Additionally, factors such as the geographical dis-
persion, wide-open business opportunities, and the
large number of software engineers accentuate both
software process improvement and development
management complexity.

The above scenario led Serpro to define a standard
process for solution development and maintenance

DOI: 10.1002/spip 69

Research Section V. Malheiros, F. R. Paim and M. Mendonça

Figure 1. PSDS main website

in 2001 – called PSDS2 – part of a large software
development modernization program started in
2000. The CMM-SW model was used as an initial
reference and since 2005 CMMI has become the
PSDS reference model for process improvement.

The main goal of PSDS is to offer development
teams a standard and dynamic software process. A
process that encourages the capture and evaluation
of the best software development practices from
local team members in order to make them available
for use by all development units nationwide
(Malheiros et al. 2002). PSDS works to answer the
following common questions: ‘who does what’,
‘when’, and ‘how’ in organizational environments
and processes. Its structure is built upon macro-
activities, i.e. a related set of activities that may
refer to either Software Engineering tasks or Project
Management (Malheiros et al. 2002).

Conceived as a web tool (Figure 1), the PSDS
site is freely inspired in both RUP (RUP 2005) and
PMBoK (Project Management Body of Knowledge, PMI

2 PSDS is an acronym for Processo Serpro de Desenvolvimento de
Soluções or Serpro’s Solution Development Process.

2004) elements, adapted to best practices defined in
CMM/CMMI models.

As software, PSDS evolves continuously and
incorporates defined procedures to add improve-
ments to its core processes. Its constant evolution,
as well as institutionalization practices across devel-
opment units are planned and tracked to conclusion
according to a specifically designed organizational
program. Further information on PSDS can be found
in Malheiros et al. 2002, and Paim and Tavares 2002.

3.2. Process Improvement Program

In 2000 Serpro began a large software development
modernization program, named PSMDS3 , which
aimed to: (i) establish general policies for software
development (ii) define a standard software pro-
cess that could be used by all Serpro’s development
units and (iii) deploy it through the entire company.
The program came from the realization that soft-
ware development success depends fundamentally

3 PSMDS stands for Programa SERPRO de Melhoria do Desenvolvi-
mento de Soluções or Serpro’s Solution Development Improve-
ment Program.

70 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

upon a mature organizational culture, as well as on
investments in discipline, methods, policies, capa-
bility and procedures, and ultimately on a proper
software development process.

From the beginning, constant software improve-
ment was a concern and CMM-SW was elected
the standard process reference model for process
improvement within the organization. In 2001, the
first PSDS version appeared forged upon RUP ele-
ments and incorporating best practices from CMM-
SW as well as successful internal practices collected
from across the organization and organized and
documented in a centralized form for wider use.

From its debut to its present version, PSMDS has
undergone several changes in order to upgrade it
and further align it to Serpro strategic objectives.
In its most recent version, PSMDS has been embed-
ded into the organizational structure as a Strategic
Alignment Unit (UAE in Serpro terminology) and
it is under the supervision of an assigned Admin-
istrative Director, which reflects the relevance of
software process improvement to the company.

PSMDS assures training, resources and funds for
PSDS incorporation across units (more than 40) and
keeps the process aligned to best practices adopted
worldwide. Some of the program goals are: (a) treat
process improvement corporatively, conferring it
the necessary independence to formulate policies
and procedures, while standardizing practices to be
used at all development units; (b) define strategies
to consolidate Serpro among top-level enterprises in
the market, recognized for delivering high quality
products and services.

The program works on two main fronts: con-
tinuous software process evolution and software
process institutionalization through development
units. To coup with these goals, the following
directives were established: (a) maintain and evolve
PSDS, having CMM/CMMI as reference models,
(b) monitor PSDS use across units directly involved
with software solution creation, (c) map aspects
of simplicity and practice from local procedures,
when defining PSDS standards and procedures,
aiming at enhancing the quality of delivered solu-
tions, (d) strengthen software quality groups with
adequate process usage inductors, (e) continuously
pursue improvement in client service satisfaction
rating, (f) define requirements for process support
tools, and (g) seek integration between PSDS initia-
tives and other internal processes and projects.

3.3. Organizational Structure of the Process
Improvement Program

Owing to its size and complexity, Serpro has
carefully crafted an organizational structure for
software process improvement. The structure is
organized in three layers. The first layer is at the
top-level administrative structure of the company.
It has a unit responsible for coordinating the process
improvement program at the corporate level, called
Serpro Strategic Alignment Unit (UAE). The second
level covers the business unit level. It is composed
of Management Units (UG, in Portuguese) which
are responsible for certain business areas. They
are usually distributed and composed of several
development units. The units compose the third
and last level of the structure, the unit level. This
layer is composed of 40 software development units.
Figure 2 shows the organizational structure and the
roles specifically defined to implement the Process
Improvement Program.

At the top corporate level, the Process Coor-
dination Group (CPSDS) was created to support
Serpro SPI program (PSMDS). The PSMDS entails
two projects: the Software Process Improvement
Strategic Project (PSMPS), equivalent to Software
Engineering Process Group (SEPG) for CMMI; and
the Process and Product Quality Assurance Corpo-
rative Project (PSGQS), which is responsible for the
coordination of all Serpro quality assurance consul-
tants (named GQS Consultants). All three groups
(CPSDS, PSMPS and PSGQS) integrate Serpro’s
Strategic Alignment Unit (UAE) and are formally
embedded into the company’s administrative struc-
ture.

Moving to the supportive groups at the business
level, these groups are not part of the formal
company structure as they are created by means
of designations. In other words, such groups are
composed of personnel from several development
areas within the company. They directly involve
personnel from development units in maintaining
the process – Specialist Groups (or simply ‘GE’).
Motivation to create these groups was derived from
the idea of increasing developer’s participation in
the software development process definition and
evolution, giving SPI tasks arrangement a cross
company configuration.

The Specialist Groups (GE) work as process
assets maintainers and there is one group for
each PSDS macro-activity. Serpro currently has

DOI: 10.1002/spip 71

Research Section V. Malheiros, F. R. Paim and M. Mendonça

Figure 2. Organizational structure in support of the SPI program

12 specialist groups (e.g. Project Management, Soft-
ware Testing, Requirements), each responsible for
upgrading the macro-activity content, whenever
changes are applied to the reference model or sug-
gested in improvement proposals sent by process
users and approved by PSMPS. They also provide
technical support to PSMPS on process improve-
ment proposals analysis, which involves estimating
‘cost and benefit’ for each change as well as the
required time for implementation. The main focus
of a GE is process evolution. All group members
are only partially dedicated to SPI activities and
otherwise act as system analysts, project leaders,
and testers, among other roles in software develop-
ment.

With the same principle of increasing developer’s
participation, the unit layer has the Software Engi-
neering Process Groups (Local SEPG). The Local
SEPG is responsible for process institutionaliza-
tion in the unit. They are the ‘process guardians’
in the units. Among their attributions are: (a) offer
support to on-going software projects under its
jurisdiction so as to better use PSDS, (b) guarantee
that software process adaptation to projects pro-
ceeds according to PSDS guidelines, (c) perform

periodic assessments to verify that PSDS applica-
tion in the units conforms to its defined procedures
and periodicity, (d) identify process improvement
opportunities, (e) encourage PSDS changes adop-
tion, and (f) provide first-level support to the unit
layer.

The Senior Management defines the local SEPG
structure. Some units have decided to form groups
with part-time dedication to process activities, sim-
ilar to the GE approach. In other cases, managers
have preferred to allocate one local SEPG coordi-
nator full-time to foster PSDS institutionalization.
Cultural issues, along with human resources profile,
have been taken into consideration when choos-
ing one approach or the other. However, differ-
ences among internal local SEPG structures do not
affect PSDS internalization and software develop-
ment standardization – advantages and disadvan-
tages observed will be discussed in the Section
‘Result Analysis’. Besides its core activities, the
Local SEPG collaborates with process evolution by
identifying and submitting process improvement
proposals and preanalyzing those created in the
unit.

The PSMPS group is responsible for facilitat-
ing the software development process definition,

72 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

Figure 3. State diagram illustrating PMP analysis flow

maintenance, and improvement. Constant process
improvement is assured by means of process
improvement proposals (PMP) that can be fired
directly from PSDS site and are managed according
to a collaborative analysis flow that pervades local
SEPGs, GEs, and PSMPS as well. Figure 3 presents
the state diagram for this flow.

The PSGQS, UG-GQS, and Unit-GQS (the chain-
of-command in the middle in Figure 2) constitute
the Process and Product Quality Assurance Groups
in the organization. GQS stands for Software
Quality Assurance in Portuguese. The Local GQS
(at Unit level) acts within the development units
and is responsible for nonconformance findings
in software projects, addressing the practices of
CMMI’s Process and Product Quality Assurance
area. The other GQS levels were created to deal
with specific issues of large organizations, such
as working with various development units at
different maturity levels. The UG-GQS acts at the
business level and the PSGQS controls global GQS
activities. Both groups supervise Local GQS work to
ensure alignment among GQS from different units
and to give the corporate hierarchy the quality
issues visibility it needs.

Not shown in Figure 2 is the regional structure
which is orthogonal to the business level. The 40
development units are organized as 8 business
units, but are spread over 10 local offices, from

here on referred to as Regional. Each Regional con-
ceals a diversity of development units as projections
of different Superintendencies, arranged according
to business segment. Even though the development
units may not work on the same projects, the phys-
ical proximity facilitates knowledge sharing among
them. For this reason, Serpro also has Regional Com-
mittees (COMREG). They serve as ‘ombudsman’
and process-related subject coordinators. Similar to
local SEPGs, COMREGs focus on process institu-
tionalization. However, they focus on evaluating
regional needs for infrastructure, training and con-
sulting, and promoting knowledge sharing among
local units.

As the standard process is set in motion through
more or less mature units, improvement opportuni-
ties are identified and proposed by developers. The
GQS and COMREG groups also contribute with
process improvement when PMP are generated as
a result of their activities. A GQS Consultant, for
example, while working with development teams,
may notice conceptual shortcomings in the pro-
cess that stimulate recurrent deviations. In such
case, he/she will generate a PMP to report the
inconsistency and propose a possible solution.
The entire PMP life cycle, from its creation to
approval/rejection is controlled through the GM-
PSDS tool, which is discussed in Section 4.

DOI: 10.1002/spip 73

Research Section V. Malheiros, F. R. Paim and M. Mendonça

3.4. Institutionalizing the Process in a Software
Development Unit

Serpro has established a strategy to promote effec-
tive process use across its units. This approach
follows a series of steps, whose chaining is illus-
trated in Figure 4. The next paragraphs will briefly
describe each step. A complete description of the
approach, along with lessons learned from the
whole institutionalization process, will be avail-
able in a future work, as the present article’s focus
is on PSDS evolution rather than its deployment
throughout the organization.

The goal of the Plan PSDS Deployment phase is to
establish a Software Process Improvement Plan for
the target unit. Guidelines on how to instantiate the
plan is described in Organization Process Management
(GPO) macro-activity. The plan must be tracked
through the entire PSDS deployment project and is
revised when necessary.

When beginning an improvement process, one
needs to bear in mind that obtaining qualification
means simply gaining recognition that the target
unit has improved its software processes qual-
ity. Hence, the real motivation should rely on the
improvement itself, not on the qualification sta-
tus. Therefore, the next step to achieve this goal
is to Obtain Commitment from all stakeholders.
Formally, such a commitment is part of the organi-
zational policies established for each macro-activity.
One must guarantee that managerial staff, as well

Figure 4. PSDS institutionalization steps

as work teams, do know them and work according
to such policies.

When implementing PSDS, special attention is
given to team preparation, which involves three
important cornerstones: training, mentoring, and
internal support. The first step in achieving prepa-
ration is Capacitate People. Development teams
must be trained and have a minimum PSDS train-
ing grade, according to each collaborator role in
the projects (the minimum grade is available from
PSDS Website). Besides training, two other forms of
capacitating are proposed: (a) Process mentoring,
a coaching event where a mentor offers technical
support for a group of team members and/or spe-
cialists, provides support on solving problems, and
provides guidelines and general orientation about
PSDS. GE members are usually process mentors.
(b) First-level support, provided by the local SEPG,
aims to clarify team members’ doubts on adopting
PSDS and offers process explanation applied to the
reality of projects.

Once deployment planning is set, having acquired
agreement from all stakeholders and fulfilled nec-
essary preconditions (infrastructure and people
ready), comes the time for implanting PSDS, follow-
ing the SPI plan strategy in order to assure PSDS
will be disseminated, properly applied, maintained,
supported, controlled, and periodically assessed.

During this phase, pilot software projects as
a start-up are strongly recommended, one pilot-
project per team. This approach reduces process
deployment risks as it restricts the maintenance
scope and avoids impacts to slip through on-going
projects. Local SEPG members must discuss all
doubts brought up during piloting and Frequently
Asked Questions (FAQ) list may be produced
and distributed. Whenever the local SEPG is
not comfortable with answering a question, it is
forwarded to PSMPS supervisors (Figure 4).

Implantation activities must be tracked to com-
pletion. This is the goal of the Track Progress phase.
In support of progress oversight, it is important to
perform appraisals on preset milestones to map
the organization progress and status. With that
in mind, two steps are carried out: Prepare for
Appraisal and Execute Serpro Internal Appraisal.
Appraisals verify the development unit adherence
to CMMI practices concerning a certain maturity
level by means of measuring PSDS use on recently
developed projects. Full-use of PSDS is always
expected as a result of appraisal events. For the

74 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

last and most important event, Serpro’s Internal
Appraisal is mandatory once the unit reaches full-
adherence to CMMI practices pertaining to a certain
maturity level (e.g. level 2 – Managed). The unit is
formally qualified as an organization with such a level
of maturity from the corporate level point of view.
This appraisal also provides senior managers with
recommendations on possible improvement oppor-
tunities to be included and treated as part of a
future Action Plan. Serpro’s Internal Appraisal is
a 5-day event, organized by PSMPS and involving
six appraisers (two of whom are from target unit
collaborators). The current appraisal method used
is in compliance with SEI SCAMPI (SCAMPI 2001)
method.

Although SEI official appraisals have been con-
ducted successfully in four of Serpro’s units, the
Serpro Internal Appraisals do not have to involve
external organizations. At this point, Serpro has
internalized its own appraisal practices, and pro-
moting internal appraisals aims to: (a) motivate
work teams to implant and continuously improve
PSDS, (b) increase the unit knowledge level on the
best software development practices by means of
training and practical experience, (c) generate an
indicator that enables tracking each unit evolution,
as well as to analyze such evolution in the context of
the entire corporation, and (d) promote the gener-
ation of improvement proposals (PMP), gathering
process improvement needs identified during the
appraisal.

When performed as internal events, maturity
appraisals are costly and emotionally stressful for
the assessed units. For this reason, the company has
adopted some preliminary steps to prepare units
for an appraisal. Institutionalization mentoring is
one of these steps. The objective of this event is
to evaluate the unit evolution status and the local
software process (or some of its parts) adherence to
CMMI practices. This initiative also tries to match
the characteristics of the unit development and
maintenance life cycle against the PSDS directives.
The final goal of the initiative is to elaborate a
final report listing the strengths and weaknesses
identified and an action plan rich in improvement
suggestions that will serve as a basis for upgrading
the PSDS deployment plan designed for the unit.
In addition, an inter-unit mentoring initiative
prepares local appraisers to participate in the
Internal Appraisal Team.

4. PROCESS IMPROVEMENT:
ACHIEVEMENTS AND PERSPECTIVES AT
SERPRO

Over the past 6 years Serpro’s SPI approach has
steadily changed the way software is developed
within the company. Development teams have
incorporated better software engineering and man-
agerial practices that have led their units to achieve
significant results in terms of client satisfaction,
product quality, and software project control. The
increasing amount of units qualified in SW-CMM
and CMMI level 2 has come in response to this
and forged the new face of software development
within the corporation. More than simply market
fitting, Serpro’s experiences have fostered a cul-
tural change that goes beyond the adherence to
standards to redefine the way developers create
software products. Furthermore, Serpro’s SPI pro-
gram pursues both the homogeneous evolution of
unit’s maturity across UGs and a continuous adher-
ence of qualified units to higher CMMI maturity
levels, without losing track of corporate goals. In
this section, we briefly present some achievements
and perspectives of SPI in Serpro according to data
collected from software process institutionalization
throughout the company.

4.1. PSDS Evolution

PSDS was released in 2001. Since then, it has
been through several upgrades constantly adjusting
its procedures to the development teams’ actual
needs. PSDS version 1.0 was built over the Unified
Software Development Process (USDP, Jacobson
et al. 1999) and RUP frameworks with a free
adaptation to SW-CMM practices. Over time PSDS
has undergone a number of changes to incorporate
user suggestions, to adjust to SW-CMM sunset
and new platform specificities (e.g. Web) and,
more recently, to correct gaps regarding CMMI
compliance. All these process ‘maintenances’ were
performed so as to maximize process efficiency and
effectiveness.

In version 6.2, PSDS has guided 12 Serpro units
through their successful journey to SW-CMM or
CMMI level 2 compliant4. For more than 2 years

4 We use the term ‘compliant’ to indicate that the unit has gone to
a formal Serpro Internal Appraisal, not necessarily a SEI official
appraisal.

DOI: 10.1002/spip 75

Research Section V. Malheiros, F. R. Paim and M. Mendonça

PSDS has also provided CMMI level 3 support to
development units pursuing such levels of maturity.
Three of our units are to be submitted to level 3
appraisals by the end of this year.

One point that stands out in PSDS evolution is the
search for a balance between CMMI compliance and
the fulfillment of overall units’ requirements, with-
out prevalence of either corporate or individual unit
interests. In a large company like Serpro, this is dif-
ficult to achieve. The balance between corporate or
unit interests can only be achieved by intense corpo-
rate involvement and a well structured continuous
SPI program. The hierarchical SPI organizational
structure depicted in Figure 2 has allowed Serpro
to continuously evolve the process while adopting
PSDS as its companywide standard. All this was
done despite the geographical distribution of the
company and the multitude of clients it serves.

In Serpro’s approach the process is constantly
revised and whenever needed either specific pro-
cedures or new technical orientations are intro-
duced to comply with clients’ and units’ particular
way of working, while PSDS core activities are
preserved as independent and adherent to best
development practices as possible. However, this
has a cost. Defining software processes under such
premises requires more effort and a greater amount
of resources when compared to other strategies
that privilege the improvement of prominent matu-
rity islands within the company. Such an approach
could upgrade certain units faster to higher levels
of maturity, but it would also make the company
drift from its commitment to forging a corporate
standard of product quality that all its clients can
recognize and benefit from.

4.2. Process Maturity Improvement in
Development Units

Serpro had its first unit qualified as CMM-SW level
2 by a SEI partner in 2002. This effort to implement
the CMM recommended processes took 2 years. At
that time, PSDS was on version 2.1 and was used
only in 20% of Serpro’s units. Since then, PSDS
adoption has reached 100% of the units. Twelve of
those units are now evaluated as level 2 compliant
(four of them officially by SEI).

Since units mature differently, large companies
will breed several maturity generations. Figure 5
illustrates Serpro maturity evolution over the past

Figure 5. Maturity evolution of the development units

5 years. Bars represent the amount of units qual-
ified each year while the ascending curve shows
cumulative evolution toward the goal of all 44 units
qualified for at least level 2.

On average at Serpro CMMI level 2 internaliza-
tion cycles require 2 years. The fluctuations in the
bars illustrate this fact. Moreover, this variation
reflects Serpro’s alternated focus between process
evolution and process institutionalization. In 2004,
for example, the company prioritized efforts to cre-
ate and customize CMMI level 3 process areas that
were being used extensively in 2005 by the CMM
level 2 qualified units. On the other hand, during
2005 the strategic objectives were revisited and insti-
tutionalization of the PSDS through units became a
priority again. The focus was on supporting institu-
tionalization ensuing in both (a) obtaining return
on the spread out institutionalization initiatives
applied during 2003 and 2004, resulting in five new
units qualified that year; and (b) preparing a new
generation to be qualified in 2007 (Serpro decided to
restructure the company internally creating many
development units).

In 2006 Serpro decided once again to redirect its
qualification focus to CMMI (process evolution), as
a consequence only two units achieved qualifica-
tion. At this point, third generation units pursuing
levels 2 and 3 were still internalizing new proce-
dures.

Some new qualifications are expected for 2007. As
new units become qualified more people become
able to disseminate PSDS practices, so despite the
focus on process evolution, Serpro’s SPI program
(PSMDS) has led to many hours of training in
nearly all PSDS macro-activities, and it has provided
mentoring and pre-qualification events for units
with good Process and Product Quality Assurance
(PPQA) results.

76 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

As the company matured, many of its level 2 units
have begun parallel efforts to reach higher maturity
levels. This has required considerable resources as
the transition from level 2 to a level 3 maturity
level is quite costly. The company was faced with
the realization that candidates for level 3 could
consume most of the SPI budget. To deal with
this issue, Serpro decided to pursue CMMI level 3
qualification for a small group of units. Three units
were chosen to start the process. These units have
been trained in the level 3 disciplines and will be
supported by external consultants to consolidate
the CMMI level 3 practices during 2007.

4.3. Tool Support for a Large SPI Plan

Tool support is fundamental in SPI. The larger
the company the more important tools are. Tools
facilitate the implementation of the standard prac-
tices at the same time as they help with the large
amount of information generated by the organiza-
tion. Complex enterprises such as Serpro have so
many improvement requirements to fulfill that com-
mercial off-the-shelf (COTS) tools alone cannot do
the job. Serpro adopts a hybrid approach anchored
on both: (a) well-known market products; and
(b) corporate tools developed internally to satisfy
the SPI program (PSMDS) needs. The well-known
products in the market include the following tools:

Tool name Function in the SPI program

Rational
Suite

Manages software requirements (RequisitePro)
and project baselines (ClearCase). Also offers
support for software test planning, running,
and environment building (TestManager).

Mantis Used as both bug tracking tool and peer review
manager.

MsProject Controls project schedule and works as a
repository of some project measures.

Serpro’s internal tools include the following:

Tool name Function in the SPI program

Revisa Automates the full PPQA review cycle, from
planning to nonconformance resolution.
Enables Corporate PPQA Area to manage
PPQA agents’ performance and units’
adherence to PSDS in a monthly basis. Collects
PPQA measures and presents a variety of
reports in support of PPQA tracking.

Tool name Function in the SPI program

SGI Controls team members’ effort allocation both
in process and project activities. Its daily
records are the input to general project and
process management at higher administrative
levels.

ADOTE Generates budget simulations and final
proposals for end clients. Manages budget use
across departments.

Solicita Manages project lifecycle. Among other
functionalities, registers and allocates client
demands to project leaders; allows for task
assignment to team members; controls tasks
flow.

GPA Data Sheet that collects and oversees main
project measures (effort, cost, timeline and size
control). Automates Function Point
calculations, project effort distribution
throughout phases and Earned Value
projections. Records and tracks actions taken to
solve project issues. Works as a repository of
measure rationale and data.

SISTED Records training event results.
GM-PSDS Registers and administrates process

improvement proposals (PMP) and process
support requests (PS). Manages PMP/PS
workflow and provides fully automated
support for CMMI process areas OPD and
OPF. A powerful SQL interface enables data
retrieval to aid in PSDS evolution and new
release generation.

Serpro’s internal tools have many features in com-
mon: data workflow management; data and user
access control; historic data recording and extrac-
tion; wide access through intranet; integration with
corporate e-mail for alert services; and a central-
ized database. All these features were customized
to meet company specificities in a way that would
be hard to achieve with COTS applications.

It is beyond the scope of this article to describe all
these tool functionalities in detail. We will refer to
features and results obtained from using GM-PSDS,
the tool most directly related to software process
improvement. Other tool results and lessons learned
will be published elsewhere.

4.4. Process Improvement with GM-PSDS

GM-PSDS is a process improvement support
tool developed on the ARS – Remedy (BMC.ARS-
Remedy 2006) platform to implement the pro-
cedures defined by the PSDS macro-activity for

DOI: 10.1002/spip 77

Research Section V. Malheiros, F. R. Paim and M. Mendonça

Change Management and User Support. It repre-
sents a unique entry point for improvement pro-
posals (PMP) registering and resolution, as well as
for process support requests (PS) by final users.
The tool provides fully automated support for the
GPO activities (including its subactivities) shown
in Table 1, managing the entire PMP/PS life cycle.
GM-PSDS also provides indirect results to aid exe-
cution of the following activities: Control Release
Generation and Oversight Process Evolution.

Through GM-PSDS, all process improvement-
related data is stored in a relational database and can
be retrieved by means of predefined or customized
SQL queries. GM-PSDS main interface is shown
in Figure 6. It provides users with answers to
the following common questions: (a) What is the

Table 1. Activities and subactivities supported directly by
GM-PSDS
Activity: Manage Process Improvement Proposals (PMP)

PSDS Subactivity Tool-related Facility

1 Elaborate
improvement proposal

2 Execute preliminary
analysis (and)

3 Send proposal for
analysis

4 Analyze improvement
proposal

Set PMP status to ‘under
analysis by PSMPS’

5 Request technical
analysis to Specialist
Group

6 Elaborate technical
report

Specialist Group registers
technical report

7 Approve
improvement proposal

Activity: Provide Process Support

PSDS Subactivity Tool-related Facility

1 Register support request Register PMP/PS
2 Provide first-level support

(and)
Register Local SEPG action
(either solution or
transferring to PSMPS)

3 Transfer PS to second level
support

4 Request technical analysis to
Specialist Group

5 Elaborate technical report Specialist Group registers
technical report

6 Provide second level support Register support request
solution

7 Generate PMP –
8 Register Generic Solution –

present PMP status? (b) To whom was the PMP
sent? (c) What was the means of transference (e-
mail, telephone, express mail)? (d) Has the PMP
already been analyzed? By whom? (e) When was
it created? (f) How long has it been there without
a solution? (g) How many PMP have been created,
approved, implanted (by Regional and by Unit)?

GM-PSDS has been in use since November 2004.
After more than 2 years, one can now evaluate its
actual contribution to improvement proposal analy-
sis and tracking regarding PSDS. The tool usage has
brought relevant benefits to PSDS improvement
control. As each Process Improvement Proposal
(PMP) can only be recognized if registered into the
tool database, no process change can be processed
unless there is a corresponding and approved PMP
that justifies the change. Such a practice facilitates
PSDS configuration management as a positive col-
lateral effect.

On the other hand, the users usually demand
fast response from the PSDS supporting service
(PS). For this reason, the support facility cannot
be exclusively controlled through the GM-PSDS
tool. Part of it is provided by other means such
as consulting, mentoring, and informal support
(personal or by phone). All these forms of support
are useful and productive in their own way.
However, in order to consistently disseminate
support considerations among users and thus
identify improvement points for PSDS, support
specialists are strongly advised to register on the
tool database each support they provide to others.
Additionally, including a support request into
the tool enables its future incorporation into the
corporative knowledge base.

PMP/PS handling is tracked weekly and a
summary report is generated and addressed to each
Specialist Group supervisor, containing a complete
list of open submissions. Moreover, once a month
an executive report is produced for the PSMPS
Coordinator to display consolidated PMP/PS data.
Every 6 months, measures are also consolidated to
enable analysis of the efficiency of improvement
efforts throughout the company.

The amount of PMP/PS opened in 2 years of tool
operation reflects the high level of user commitment
to the improvement program. In all 2512 requests
have been submitted, averaging more than four
submissions per working day. It was seen that 76%
of all requests were PMP (1909) and 24% were PS
(603); 32% of all requests were submitted during the

78 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

Figure 6. Main GM-PSDS interface, showing PMP details

first year while 68% were submitted after that. This
number shows an increasing generation of PMP.
This was expected, since new units with different
characteristics are adhering to the PSDS procedures.
Among all the PMP analyzed, 69% were approved
for incorporation into PSDS and 31% were rejected.
This high percentage of proposal endorsement
reflects the pertinence as well as the quality of
submissions. The focus of PMP submissions to GM-
PSDS concentrate on the following issues: activities,
subactivities, and PSDS procedures (48%); PSDS
artifacts (33%); and development tools applied
(12%). The remaining 7% refer to a variety of other
issues.

Building upon a similar approach proposed by
Natali and Falbo (2002), GM-PSDS has a feature
that enables users to mark a PMP for inclusion
into the Corporative Knowledge Base or even at
the PSDS FAQ (Frequently Asked Questions). Some

indicators, however, demonstrate that this feature
has not been widely used. Only five submissions
(0.1% of the Submissions Total) were marked to
be included into the knowledge base and only 15
others (0.6% of the Total) were ticked for inclusion
into the FAQ area. These indicators are important
for improving knowledge management within the
organization. The low values obtained point to a
need to reinforce the importance of such indicators
for specialist groups.

Figure 7 illustrates that at present macro-activities
related to CMMI level 3 are responsible for 38%
of PMP submissions, while other level 2 macro-
activities are responsible for 62% of the total number
of submissions. Considering the quantity of units
implementing level 3 macro-activities (4 units, out
of 44 development units), such a percentage seems
elevated. The PSMPS Group attributes this high
amount of level 3 PMP to two main reasons:

DOI: 10.1002/spip 79

Research Section V. Malheiros, F. R. Paim and M. Mendonça

Legend:

REQ – Requirements Management, NEG – Business Modeling, GP – Project Management (includes
Risk Management and Integrated Product Management also), GAF – Supplier Agreement Management,
GQS – Process and Product Quality Assurance, GCS – Change Management,
PSDS – PMP related to the process as a whole, RP – Peer Reviews (basically Verification Practices),
TES – Product and Acceptance Test (Basically Validation Practices), PTR – Training Program,
GPO – Organizational Process Management, EPS – Software Product Engineering (analysis, design,
implementation, deployment)

PMP distributed by Process Area

GCS
9%

GP
20%

GQS
11%

REQ
14%

MA
6%

GAF
1%

NEG
1%

ADR
1%

EPS
7%

GPO
3%

PSDS
9%

PTR
6%

RP
5%

TES
7%

CMMI 3
38%

Figure 7. PMP distributed by CMMI process areas

(i) level 3 processes are younger and therefore more
susceptible to improvement; (ii) units implementing
level 3 practices are exactly those that have been
using PSDS for a longer time (more than 4 years)
and are more aware of the significance of process
improvement. As the process gains maturity, this
difference tends to decrease. During the first year,
the level 3 process areas were responsible for 44%
of the PMP amount. As can be seen, this number
has now decreased to 38%.

From the mapping of PMP distribution per
macro-activity, it is possible to identify macro-
activities requiring revision and working groups
that are in most demand. It is also possible to infer,
with a certain amount of precision, those macro-
activities that are being used most frequently by
development teams.

An indication that preference for continuous
process improvement increases proportionally, as
the development team’s maturity grows, is that the
business unit that contributed the most with process
improvement is also the one that holds more CMM
level 2 qualified units, corresponding to 55% of all
submissions.

As for the type of change, 13% of PMP were clas-
sified as corrective. Such a correction percentage
was considered high by the PSMPS. Neverthe-
less, given the complexity of PSDS, with over

4000 files and more than 50 roles, this percent-
age indicates that some actions regarding inten-
sifying PSDS maintenance and deployment tests
should be carried out. This percentage has also
motivated the adoption of a XML/HTML based
tool for collaborative process modeling, The tool,
called Atabaque, is publicly available at http://
sourceforge.net/projects/atabaque/.

Some issues remain and need to be considered in
order to assure faster, more efficient process support
through GM-PSDS. They include:

(a) Time reserved for PMP analysis activities is
not always sufficient. Although incorporating
members of development teams into specialist
groups brings a practical vision into improve-
ment proposal analysis, it causes, on the other
hand, problems related to giving priority to
executing process-related activities.

(b) Some specialist groups have not been able
to involve all their members effectively. These
members are often committed to client solution
development and are subordinated to a project
manager different from the specialist group
coordinator.

With respect to support submitted requests (PS),
we observed that 30% of them were analyzed and
answered locally in a first-level support mode. This

80 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

number indicates that the knowledge is dissem-
inated through development teams and demon-
strates that the support workload is well distributed.
On the other hand, it indicates an opportunity to
continue enforcing support distribution. Therefore,
the PSMPS group decided to recycle the local SEPGs
with new training.

The general satisfaction level with PMP/PS can
be considered very good. The average rating for
answers’ quality was four out of a maximum of
five. However, the satisfaction levels with the given
answers are greater than the satisfaction levels
related to time spent to process the requests. This
shows that specialist groups are well-prepared to
deal with support demands, but could act faster on
these requests.

The rates given to the ‘time defined’ to implement
an approved PMP are slightly lower. Justification
comes from the fact that changes to PSDS become
available only in the next release. The general
directive to pack a number of changes in a
single release aims to issue a proper treatment to
PSDS improvement. PSDS deployments are done
based on well-controlled releases to avoid constant
disruptions in the developers’ daily business.

5. CONCLUSION

This article presented the continuous process
improvement approach used at Serpro, a very large
software company owned by the Brazilian Gov-
ernment. The presented SPI approach influences
more than 3000 developers in 40 development areas
widely spread throughout the country. This size of
this enterprise creates many challenges to define,
deploy, and evolve its standard software process.

As large companies worldwide increasingly
invest in software process improvement, they also
have to deal with typical issues regarding soft-
ware process evolution within such environments
in order to guarantee the survival of their software
process.

Serpro’s SPI program pursues: (i) the homoge-
neous evolution of unit’s maturity across its UGs
(ii) the continuous adherence of qualified units to
higher CMMI maturity levels and (iii) doing all this
without losing track of corporate goals. As a con-
sequence, the company as a whole benefits from
a steady, companywide visible standard. This can

generate IT solutions of similar quality for a multi-
tude of clients, despite their inherent diversity.

After 6 years of SPI experiences, the many lessons
learned were gathered at Serpro. There are impor-
tant differences between qualifying a unit with a
high maturity levels and spreading this maturity to
several company’s units. Many publications point to
best practices and experiences in SPI applied within
one software development unit, but not many refer
to approaches to structure an SPI program to deal
with several development units and their asso-
ciated organizational problems. This article does
exactly this. It points out that institutionalizing an
SPI program throughout several development units
assuring a minimum quality level and a standard
process to all of them, is a hard task and demands a
well defined organizational strategy. This strategy
is discussed in detail in Section 3.

Some of the data gathered is presented in Section
4. This data was gathered through GM-PSDS, a
SPI supporting tool that helps the organization
to manage the improvement proposals. It grants
users with answers to the following common
questions: (a) What is the present status of an
improvement proposal? (b) Who was this proposal
sent to? (c) What was the means of transference (e-
mail, telephone, express mail)? (d) Has the proposal
already been analyzed? By whom? (e) When was it
created? (f) How long has it been there without
a solution? (g) How many proposals have been
created, approved, and implanted by Regional and
Unit? The tool also fosters developers’ participation
in SPI. The results presented have been useful to
better understand and evolve the software process,
its adoption, and improvement activities. The data
is also potentially useful for researchers and other
companies dealing with SPI or CMMI. In future
works, we plan to data mine it with respect to
causal and associative relationships between key
process variables.

Knowledge management issues can also be
analyzed from GM-PSDS data. The data can be used
to explore issues such as: (a) training needs, (b) who
knows what in the organization, or (c) which units
are more active in process improvement (e.g.
generating more proposals or requests).

The continuous process improvement approach
presented here discusses the SPI perspectives and
achievements at Serpro. It can serve as a reference
to large software companies that deal with the
same challenge of having many development

DOI: 10.1002/spip 81

Research Section V. Malheiros, F. R. Paim and M. Mendonça

units geographically spread producing high quality
software. However, we also hope to motivate
researchers to address some of problems related
to doing SPI at company level.

REFERENCES

Basili V. 1985. Quantitative evaluation of software
engineering methodology. Proceedings of the 1° Pan Pacific
Computer Conference, Melbourne, 379 – 398.

Basili V, Caldiera G, MacGarry F, Pajerski R, Page G,
Waligora S. 1992. The software engineering laboratory:
an operational software experience factory. Proceedings of
the 14th International Conference on Software Engineering,
Melborne, 370 – 381.

BMC.ARS-Remedy. 2006. http://www.remedy.com/
solutions/servicemgmt/itsm.html, Last access: March
2006.

Brownsword L, Oberndorf T, Sledge CA. 2000. Develop-
ing new processes for COTS-based systems. IEEE Software
17(4): 48 – 55.

Caputo K. 1998. CMM Implementation Guide: Choreograph-
ing Software Process Improvement. Addison-Wesley Pub-
lishing Company: Reading, MA.

CBA-IPI. 2001. CMM-Based Appraisal for Internal
Process Improvement (CBA-IPI). version 1.2, Method
Description. Technical Report CMU/SEI-2001-TR-033.
Software Engineering Institute’, Carnegie Mellon
University, Pittsburgh, PA.

Cockburn A. 2000. Selecting a project’s methodology.
IEEE Software 17(4): 64 – 71.

Conradi E, Fuggetta A. 2002. Improving software process
improvement. IEEE Software 19(4): 92 – 99.

Fiorini ST, Leite J, Lucena CJP. 2001. Process reuse
architecture. Proceedings of the 13th International Conference
on Advanced Information Systems Engineering. Springer-
Verlag: London.

Florac WA, Carleton AD. 1999. Measuring the Software
Process: Statistical Process Control for Software Process
Improvement. Addison-Wesley Longman Publishing Co.,
Inc. Boston, MA, 1999.

Florac WA, Carleton AD, Barnard JR. 2000. Statistical
process control: analysing a space shuttle onboard
software process. IEEE Software 17(4): 97 – 106.

Goldenson D, Gibson D. 2003. Demonstrating the impact
and benefits of CMMI : an update and preliminary
results. Special Report CMU/SEI-2003-SR-009. Carnegie
Mellon Software Engineering Institute: Pittsburgh.

Gremba J, Myers C. 1997. The IDEAL Model: A Practical
Guide for Improvement (Versão 1.1). Bridge, issue three.
SEI – Software Engineering Institute, CMU – Carnegie
Mellon University: http://www.sei.cmu.edu/ideal/
ideal.bridge.html., Last access: 03/08/2005.

Haley T. 1996. Software process improvement at
raytheon. IEEE Software 13(6): 33 – 41.

Henninger S. 1999. Using software process to
support learning software organizations. Proceedings
of the Workshop on Learning Software Organizations,
Kaiserslaurten.

Henninger S. 2000. Software Process as a Mean to
Support Learning Software Organizations, University of
Nebraska – Lincoln, Annual NASA Software Engineering
Workshop.

Humphrey WS. 1990. Managing the Software Process.
Addison-Wesley Publishing, Company: Reading, MA.

Humphrey WS. 1997. Introduction to the Software Personal
Software Process. Addison-Wesley Publishing Company:
Reading, MA.

IEEE Software Engineering Coordinating Committee.
2001. SWEBOK: Guide to the Engineering Body of Knowl-
edge – Trial Version 1.00, http://www.swebok.org, Last
access: June 2004.

INTRO – IDEAL-Based New Technology Rollout – Bib-
lioteca do Processo. 2002. (requer usuário cadastrado –
cadastramento gratuito), http://www.sei.cmu.edu/
intro/process/, Last access: 07/08/2005.

ISO – International Organization for Standardization. ISO
9000-3. 1991. Quality Management and Quality Assurance
Standards. Part 3. Guidelines for the application of ISO
9001 to the development, supply and maintenance of
software.

ISO – International Organization for Standardization.
ISO/IEC 15504. 1998. Software Process Assessment.

Johnson DL, Brodman JG. 2000. Applying CMM project
planning practices to diverse environments. IEEE Software
17(4): 40 – 47.

Jacobson I, Booch G, Rumbaugh J. 1999. The Unified
Software Development Process. Addison Wesley Longman
Publishing Co., Inc., Boston, MA.

Keeni G. 2000. The evolution of quality processes at Tata
Consultancy Services. IEEE Software 17(4): 79 – 88.

Kruchten P. 1999. The Rational Unified Process. Addison-
Wesley: Reading, MA.

Malheiros V, Mendonça M, Farias L. 2002. Uma
abordagem de Gerência de Projetos de Software, sob

82 DOI: 10.1002/spip

Research Section Process Improvement at a Large Software Organization

o enfoque da Gestão do Conhecimento. Segunda Jornada
Ibero-americana de Engenharia de Software e Engenharia de
Conhecimento, Salvador. Anais da JIISIC 2002, V. 1, 1 – 6, (in
Portuguese) Salvador, Brazil.

McFeeley R. 1996. IDEAL: A User’s Guide for Software Pro-
cess Improvement, CMU/SEI-96-HB-001, ADA305472. Soft-
ware Engineering Institute, Carnegie Mellon University:
Pittsburgh, PA, http://www.sei.cmu.edu/publications/
documents/96.reports/96.hb.001.html.

Morisio M, Tully C, Ezran M. 2000. Diversity in reuse
process. IEEE Software 17(4): 56 – 63.

Natali ACC, Falbo RA. 2002. Knowledge management in
software engineering environments. Proceedings of the 16th
Brazilian Symposium on Software Engineering, Gramado.

Oliveira KM, Zlot F, Rocha AR, Travassos GH, Gallota C,
Menezes C. 2004. Domain-oriented software develop-
ment environment. Journal of Systems and Software, Estados
Unidos 172(2): 145 – 161.

Osterweil J. 1997. Software processes are software too
(revised). Proceedings of the 1997 International Conference
on Software Engineering. ACM Press: Boston, MA.

Paim FRS, Tavares HC. 2002. Implementing a living
software process. Proceedings of the First ICSE Workshop on
Software Quality (WoSQ), Orlando, (Maio).

Paulk M, Webber C, Curtis B, Chrissis M. 1995. The
Capability Maturity Model: Guidelines for Improving the
Software Process. Addison-Wesley: New York.

Pitterman B. 2000. Telcordia technologies: the journey to
high maturity. IEEE Software 17(4): 86 – 96.

PMI – Project Management Institute. 2004. A Guide to the
Project Management Body of Knowledge (PMBOK Guide).
Project Management Institute: http://www.pmi.org/,
Last access: July 2005.

RUP – IBM Rational Unified Process. 2005. Rational
Unified Process, http://www-306.ibm.com/software/
awdtools/rup/, Last access: July 2005.

Rus I, Lindvall M, Sinha S. 2001. Knowledge Manage-
ment in Software Engineering, http://citeseer.ist.psu.
edu/rus01knowledge.html, Last access: April 2006.

SCAMPI. 2001. Standard CMMI Appraisal Method for
Process Improvement (SCAMPI), Version 1.1: Method
Definition Document. HandBook CMU/SEI-2001-HB-
001, December.

SEI – Software Engineering Institute. 1993. Key Practices
of the Capability Maturity Model version 1.1. Technical
Report CMU/SEI-93-TR-025, February.

SEI – Software Engineering Institute. 2002. Capability
Maturity Model Integration (CMMI) for Software
Engineering, version 1.1, Staged Representation,
CMU/SEI-2002-TR029, March.

SERPRO – Serviço Federal de Processamento de Dados.
2006. http://www.serpro.gov.br, Last access: January
2006 (in Portuguese).

Shull F, Mendonça M, Basili V, Carver J, Maldonado J,
Fabbri S, Travassos G, Ferreira M. 2004. Knowledge-
sharing Issues in Software Engineering, Empirical
Software Engineering, 9, 111 137. Academic Publishers:
Netherlands.

Sutton SM, Heimbigner D, Osterweil LJ. 1995. Appl/a:
a language for software process programming. ACM
Transactions on Software Engineering and Methodology 4(3):
221 – 286.

Tdvet J, Gollofello JS. 1995. Evaluating the effectiveness
of process improvements on software development
cycle time via system dynamics modelling. Proceedings
Nineteenth Annual International Computer Software and
Applications Conference, COMPSAC 95, Computer Society,
Washington, DC, USA.

Trienekens JJM, Kusters R, Solingen R. 2001. Product
focused software process improvement: concepts and
experiences from industry. Software Quality Journal,
Springer Netherlands, 9(4): 269 – 281.

Trienekens JJM, Kusters RJ, Balla K. 2005. Software pro-
cess improvement, quality assurance and measurement.
13th IEEE International Workshop on Software Technology
and Engineering Practice, Budapest, Hungary.

Wiegers KE. 1998. Read my lips: new models. IEEE
Software 15(5): 10 – 13.

DOI: 10.1002/spip 83

contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 143

d o i : 1 0 . 1 1 4 5 / 1 5 6 2 1 6 4 . 1 5 6 2 2 0 0

by dinesh batra

F r u s t r a t i o n w i t h t h e b u r e a u c r a t i c n a t u r e o F

the disciplined approach has led to the call for agile
development.3 The new approach is defined by the
Agile Manifesto (http://agilemanifesto.org/), which
values individuals and interactions over processes
and tools, working software over comprehensive
documentation, customer collaboration over contract
negotiation, and agility in responding to change
over following a prescribed plan. Agile development
does not focus on process improvement; instead
it focuses on customer satisfaction and employee
empowerment. This is evident from reading the stated
values and principles of the Agile Manifesto, which
include fairly extreme positions such as “welcome
changing requirements, even late in development”
and “the best architectures, requirements, and
designs emerge from self-organizing teams.”

An interesting issue arising from the call for agile
development is its role in distributed development,
which usually translates to offshore development.

A recent study7 indicates that agile
practices can reduce temporal, geo-
graphical, and socio-cultural distances
in distributed development projects.
The study researched agile develop-
ment between teams located in the U.S.
and Ireland, and while it reported that
overall communication was improved,
it also noted problems related to geo-
graphical, temporal, and even lan-
guage distances. Although there are
other reported successes of distributed
agile development,9 the projects are
generally small, the team members are
likely familiar with each other, and the
participants are largely experts or high
caliber developers.

This raises a research, as well as a
practical, question: can we extend the
use of agile practices from small proj-
ects to medium and large projects that
involve a significant outsourcing com-
ponent? To address this question, we
must drop constraints such as small size
projects, and expert developers belong-
ing to the same company, and examine
problems arising from geographical,
temporal, and cultural distances. Ac-
cordingly, agile practices may need to
be modified.

In this article, the key issues of soft-
ware projects with an outsourced com-
ponent are first identified. These issues
are then used as a background to evalu-
ate how standard agile practices stand
up when applied to larger projects. This
evaluation is followed by recommen-
dations for modified agile practices for
outsourced software projects.

Key issues of outsourced
software Projects
Although outsourcing is not equivalent
to offshoring a software development
project, the high availability of skilled
labor and the economic advantages of
considerably lower wages have made
the two equivalent for all practical pur-
poses. Outsourcing seems to have a sta-
ble future;4,5 several studies (www.acm.
org/globalizationreport/) have shown
that it improves productivity and does
not necessarily lead to loss of jobs for
the local region of the outsourcing cli-

modified agile
Practices for
outsourced
software
Projects

144 c o m m u n i c at i o n s o f t h e a c m | s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9

contributed articles

namic, global, customer-focused, and
competitive environment. Pure struc-
ture approaches are not suitable for
dynamic environments.3 The nature of
contemporary software development
requires flexible and organic instead of
overly hierarchical, mechanistic struc-
tures. The Agile Manifesto and its prin-
ciples, therefore, are certainly relevant
in this environment; however, to date
agile development has been reported
only for limited size projects and small
teams.3 Beck2 states: “Size clearly mat-
ters. You probably couldn’t run an XP
project with a hundred programmers.
Not fifty. Nor twenty, probably. Ten is
definitely doable.” Agile development
needs to be examined in the context of
larger projects. Just as organizations
cannot keep growing in an entrepre-
neurial mode and need formal coordi-
nation and control, agile development
may need some degree of formality im-
posed for larger projects.

Cultural distance. Compared to dis-
cipline-based development, agile de-
velopment denotes a significant shift
in culture. There are no prescribed
processes or tools; instead, values and
principles govern agile development.
Perhaps, its most important charac-
teristic is the organizational and team
culture as guided by a manifesto. Orga-
nizational culture is embedded in, and
affected by, the larger national culture.8
Agile values, for instance, can fit in the
less bureaucratic, more flexible Ameri-
can culture, but not necessarily in the
more hierarchical and structured cul-
tures found in Asia, Latin America,
and certain parts of Europe. Work that

ent. A severe recession has, indeed, re-
cently led to massive layoffs; however,
these layoffs have primarily resulted
from a weak economy, not because of
outsourcing. The primary motive for
offshoring is to keep costs down, but
there are several other noteworthy ben-
efits, such as the ability to undertake
larger projects, to expand markets,11
and to achieve agility in development
and operation.5 However, offshoring
does not always lead to cost savings or
more successful projects.

When a software development proj-
ect is outsourced, generally three par-
ties are involved (Figure 1): the cus-
tomer, the outsourcing client, and the
outsourcing vendor. Consider a typical
scenario: a customer in the US needs
an information system, and a software
division in the customer’s same com-
pany or another software development
company in the U.S. is asked to develop
the system. The outsourcing client is
the primary software development unit
and may outsource a portion of the proj-
ect to a vendor in another country, for
example, India. For competitive advan-
tage, the outsourcing client may want
to retain core activities such as analysis
and testing while outsourcing commod-
ity activities like programming. This
arrangement provides for growth and
agility while controlling costs for the
outsourcing client. Several outsourcing
vendors may be involved, although an
analysis involving multiple vendors is
beyond the scope of the paper.

The relationship between the out-
sourcing client and the outsourcing
vendor is likely to be based on a con-
tract.4, 5 Thus, the outsourcing client
may not have a strong influence on the
internal practices of the vendor. Per-
suading the vendor to follow certain
development practices may be difficult.
On the other hand, if the outsourcing
vendor is a partner or collaborator in
some respect, the client may have suf-
ficient influence to initiate practices
considered novel at the vendor site.

In adopting agile practices for out-
sourced projects, we need to consider
factors such as the size and nature of
the project, the cultural distance be-
tween the outsourcing client and the
outsourcing vendor, along with the tem-
poral, and the geographical distances.

Size and nature of the project. Today’s
software projects are developed in a dy-

takes place over long distances will of-
ten involve communication engaging
different national cultures.10 Business
alliances and collaborations have been
found to fail because of such differenc-
es.8 Organization literature has report-
ed interesting “Catch-22” scenarios
arising as a result of cultural clashes.

Temporal distance. It is obvious that
temporal distance is also likely to miti-
gate communication, which is a critical
ingredient of agile practice. When it is 9
am, or time to start work in New York, it
is around 6:30 pm, or time to leave the
office, in New Delhi. Even when collabo-
rating parties are in similar time zones
(such as those of the U.S. and Mexico),
studies have shown that different ori-
entations to time and schedules may
lead to misunderstandings.

Geographical distance: This factor
can greatly affect communication.
Surely, the widespread use of email and
other communication means have en-
hanced virtual communication. How-
ever, these mechanisms still largely
facilitate “business” communication,
not “organizational” communication.
Business communication includes for-
malized and planned messages, codi-
fied in letters, memos, reports, Web
sites, and advertising campaigns. Or-
ganizational communication includes
business communication as well as in-
formal, day-to-day interactions among
organizational members.8 Agile prac-
tices, as currently prescribed, rely on
organizational, not business commu-
nication. Although technologies such
as videoconferencing and online chat
address the geographical distance

figure 1: Parties and dynamics in outsourced software development

Contracts
project and meets

requirements

Provides
software product

Outsources routine
activities (e.g.
programming)

Delivers
working code

Customer
(Ex: US)

Outsourcing Client
(Ex: US)

May retain core project
activities such as

requirements, analysis, design,
testing, and integration

Outsourcing Vendor
(Ex: India)

contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 145

problem, it is questionable if the re-
sulting engagement satisfies the level
expected in agile development.

Here, we explained the context for
the various factors involved in evaluat-
ing agile practices in distributed set-
tings. This contextual knowledge can
be employed to scrutinize the agile
values and principles in order to assess
which ones hold up for larger projects
in a distributed environment.

interaction of agile Values with
outsourced development
The Agile Manifesto lists four fun-
damental values that should govern
software development. Table 1 evalu-
ates these values as applied in an out-
sourced project. Each value is quali-
fied by the advantages it provides to
outsourced development, and how the
value may be mitigated in outsourced
development.

Individuals and interactions vs. pro-
cesses and tools. In co-located settings,
agile practices improve organizational
communication by facilitating infor-
mal interactions. In distributed devel-
opment, the informal communication
process is considerably broken down.
For example, the assumption of tacit
knowledge is challenged, and the pair
programming practice is likely to be
drastically affected. This value may not
be welcome however, in regions where
the national culture is characterized by
high power distance, where the values
tend toward hierarchy and structure.

Working software vs. comprehensive
documentation. The basic premise of
outsourcing is that it allows a project

to be done faster and cheaper. Thus,
the goal of providing working software
is likely to be achieved more quickly.
However, documentation, too, is im-
portant in an outsourced project, giv-
en the realities and constraints of any
agency agreement. As the project be-
comes bigger, one cannot rely on tacit
knowledge alone.

Customer collaboration vs. contract
negotiation. Transactional cost theory
favors contractual agreements for out-
sourcing non-core activities of software
development, especially given the low
monitoring costs in today’s global en-
vironment. Although some companies
open their own offices at offshore sites
and do not need a contract, the remote
unit may not be large enough to pro-
vide the level of support needed given
that the software activities that are typi-
cally outsourced (such as coding) need
a large amount of personnel resources.

Responding to change vs. following a
plan. The client-vendor contract can-
not be renegotiated on an ad-hoc basis.
If the customer proposes changes, es-
pecially at the later stages of a project,
these changes will have to be passed on
to the vendor. The vendor’s response
will depend on various factors such
as the relationship with the outsourc-
ing client, to what extent the agile cul-
ture is prevalent, the costs involved in
responding to the changes and how
these are recovered, and how much fu-
ture business is expected. Whether one
applies the agency theory, or just plain
common sense here, it seems unlikely
the vendor will absorb the costs arising
from significant changes.

interaction of agile Principles
with outsourced development
The Agile Manifesto lists 12 principles
that should govern software develop-
ment. The advantages, and the prob-
lems, found in subscribing to each
principle in an outsourcing environ-
ment are listed in Table 2.

Outsourced development is general-
ly governed by client-vendor contracts.
Since organizational culture cannot be
captured in a contract, a fundamental
problem arises when employing some
agile principles. For example, a con-
tract cannot “welcome changing re-
quirements” however, it can spell out
the financial implications of the change
requirements. Face-to-face commu-
nication is ruled out with outsourced
development because of geographical
distances; if high quality virtual meet-
ings are a requirement, then someone
needs to pay for installing additional
equipment and providing the band
width. The practice of pair program-
ming may be difficult to include in a
contract. Pair programming is practi-
cally impossible between program-
mers at the client and the vendor end.
The 40-hour week is a mirage in vendor
companies; anecdotal evidence sug-
gests that most developers in countries
like India end up working 50-60 hours
per week, which can lead to fatigue and
burnout. Further, high turnover is a
given, as developers continually scout
for better pay and opportunities in a
growing sector of the economy.

modified agile Practices
for outsourced Projects
The previous analysis indicates the use
of agile practices in outsourced soft-
ware projects is not a simple, straight-
forward issue. Nevertheless, agility is
paramount in today’s fast-paced, dy-
namic, and global environment, and
the Agile Manifesto is a good starting
point to analyze software practices.
Successful organizations today are cus-
tomer- and employee-focused, less hi-
erarchical and more horizontal, more
integrated with vendors,1 responsive to
change, and willing to adapt.

Yet, agile values and principles can
be extreme and incongruous. In 1946,
Herbert Simon12 challenged the tenets
of classical organizational theory pro-
posed by Fayol, Gulick, and others by
pointing out the inherent contradic-

table 1. support for current agile Values in outsourcing environment

agile manifesto
Values

advantages in an
outsourcing environment

Problems in an
outsourcing environment

Individuals and interactions
vs. processes and tools.

there is some evidence
that organizational
communication may
improve.

various distances may impede
communication and interactions.

both outsourcing and outsourced parties are
likely to employ business (formal), instead
of organizational (formal and informal),
communication.

likely to conflict with the cultural values of
the vendor party.

Working software
vs. comprehensive
documentation.

software can be developed
more quickly.

Documentation is essential for avoiding
ambiguities arising because of lack of tacit
knowledge.

Customer collaboration vs.
contract negotiation.

the normal outsourcing practice is through
contract negotiation.

responding to change vs.
following a plan.

more resources may be
available to handle changes.

vendor may not welcome changes unless
there is additional compensation.

Contract between client and vendor parties
may not be easily renegotiated.

146 c o m m u n i c at i o n s o f t h e a c m | s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9

contributed articles

Certainly, “disruptive technologies”
may eventually overcome such bar-
riers,6 but, as an example, high qual-
ity videoconferencing on workstations
has not yet fully matured.

Modified Agile Practices. The principle,
“welcome changing requirements, even
late in development” is acceptable if the
outsourcing client is ready to absorb the
increased costs, and not try to pass it on
to the vendor. Even then, the stipulation,
“even late in development,” needs to be
eliminated. Late stage changes are not
feasible in large projects, which need a
higher degree of accountability. The prin-
ciple gives the customer carte blanche
without the financial obligations.

The principle, “business people and
developers must work together daily
throughout the project,” is generally in-
terpreted as the customer being located

tions. The recently proposed Agile Man-
ifesto has not been challenged so far,
although some inconsistencies are evi-
dent. For example, if a software division
decides to accept changes proposed
late in the project, then it will have to
ask the employees to work harder than
the espoused 40-hour week. If the em-
ployees continue to work at the routine
pace, then the project will get delayed,
and this may not be acceptable to the
customer. Similarly, if the division de-
mands excellence from its employees,
it will have to decide what to do with its
below average employees, which goes
against the agile principle that individ-
uals are valued more than processes.

When the agile practices are sub-
jected to the realities of outsourced de-
velopment, the contradictions are es-
pecially evident, as suggested in Tables
1 and 2 in the previous section. Some
of the values and principles are highly
feasible, a few are infeasible, and the
remainder need to be adjusted. Here,
we consider each of these three catego-
ries and makes recommendations in
the outsourcing context. Since the prin-
ciples capture the essence of the values,
Table 2 is used to guide this discussion.

Feasible Agile Practices. The four
principles termed as feasible in Table
2 are clearly viable.

Infeasible Agile Practices. The princi-
ples, “continuous attention to technical
excellence and good design enhances
agility” and “build projects around mo-
tivated individuals” are somewhat con-
fusing at the outset. Software projects
certainly need excellent design and
technology skills, but they also need ex-
cellent requirements gathering, analy-
sis, communication, team work, and
other skills. Ideally, one would build
projects around motivated individuals,
but the “motivated” and “excellent”
definitions are subjective and can be
misinterpreted in organizations. For
example, these definitions can be used
to fire the bottom 10% of the personnel
each year. A better practice would be to
build motivated and capable individu-
als by providing training and develop-
ment programs.

In an outsourced project, the client
cannot control personnel excellence at
the vendor end, unless the vendor has
been acquired by the client. The client
would obviously examine the capabili-
ties and references of the vendor. If the

project is awarded to a vendor with an
established record of technical excel-
lence, then the client need not micro-
manage the vendor. Eventually, the
collaboration may lead to permeable
structures and trusting relationships1
and the vendor may be more open to
instituting agile practices.

If the client is developing its own
facility in an offshored area, there is
some leeway in the selection of the top
tier personnel. However, large scale
hiring will be difficult with such an ex-
acting constraint.

The principle, “the most efficient
and effective method of conveying in-
formation to and within a development
team is face-to-face conversation” when
referring to communication between
the client and the vendor in outsourced
development is practically impossible.

table 2. evaluation of current agile Principles in outsourcing environment

agile manifesto
Principle

advantages in an
outsourcing environment
Problems in an
outsourcing environment

our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software.

Feasible because more
resources are available.

Welcome changing requirements, even
late in development. Agile processes
harness change for the customer’s
competitive advantage.

the lowering of costs
because of outsourcing may
provide some flexibility in
managing changes.

vendor may not accept late
changes. vendor may not
agree to fixed price contract.

Deliver working software frequently,
from a couple of weeks to a couple of
months, with a preference to the shorter
timescale.

Feasible because more
resources are available.

business people and developers must
work together daily throughout the
project.

temporal and geographic
distances may make it more
difficult to work with the
vendor.

build projects around motivated
individuals. Give them the environment
and support they need, and trust them to
get the job done.

Difficult to control motivation
at the vendor’s site.

Approximately 49.99% of
designers are below average.

the most efficient and effective method
of conveying information to, and within,
a development team is face-to-face
conversation.

practically impossible
between the outsourcing
client and vendor.

Working software is the primary measure
of progress.

Feasible.

Agile processes promote sustainable
development. the sponsors, developers,
and users should be able to maintain a
constant pace indefinitely.

somewhat feasible. because of temporal
distance, fatigue is inevitable
if practices like pair
programming are instituted.

Continuous attention to technical
excellence and good design enhances
agility.

Approximately 49.99% of
designers are below average.

simplicity – the art of maximizing the
amount of work not done – is essential.

Feasible.

the best architectures, requirements,
and designs emerge from self-organizing
teams.

Cultural disparities between
the outsourcing client and
vendor may inhibit self-
organization.

At regular intervals, the team reflects on
how to become more effective, then tunes
and adjusts its behavior accordingly.

organizational culture may
hamper this practice.

contributed articles

s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9 | c o m m u n i c at i o n s o f t h e a c m 147

at the developer’s premises. This takes
a broader meaning when three parties –
customer, outsourcing client, and out-
sourcing vendor – are involved. If agile
practices are applied literally, one may
come up with a communication con-
figuration as shown in Figure 2.

The recommended structure shown
in Figure 3 is an altered configuration
that maintains reasonable communi-
cation and feedback, but reduces the
communication links required. The
full lines indicate regular and defined
communication links, while the dotted
lines indicate informal and as-needed
links. The customer representative is
physically located at the outsourcing
client in both configurations.

The vendor’s team would likely re-
semble a programming team under a
group leader, although there may also
be design and other personnel. To fa-
cilitate communication while main-
taining control, the outsourcing client
needs to send an ambassador to the
vendor’s site. Typically, the ambassa-
dor would be someone who has spent
a reasonable period of his or her life
in the vendor country. The ambassa-
dor should be in daily communication
with the programming leader, pro-

grammers, and designers at the vendor
end. Similarly, there must be a primary
contact on the outsourcing client side.
This role is filled by the team coordi-
nator or project leader, who forms the
bridge between the customer and the
outsourcing vendor.

The Agile Manifesto also puts stress
on working software, so there is the
need for an integration group. This
group, consisting mainly of program-
mers, continually test, evaluate, and
integrate software so that the program-
ming group at the vendor’s end gets
daily feedback.

The customer, the ambassador, the
coordinator, the programming group
leader, and some members from the
client’s design team, and integration
team should form an agile team. The ag-
ile team’s purpose is to communicate
on a daily basis through state-of-the-art
videoconferencing in order to continu-
ally evaluate and monitor the project.
This group can be the hub for daily
planning, coordinating, communicat-
ing, monitoring, and control. However,
this should not be the only network of
communication, and informal commu-
nication should be encouraged among
individuals, and between teams.

In light of the above discussion, the
principle, “the best architectures, re-
quirements, and designs emerge from
self-organizing teams,” may need to be
modified. Although the notion of a self-
organizing team is sound, there is the
need to incorporate accountability into
agile development, especially if the size
of the project is medium or large. This
is done through the roles like the ones
described in Figure 3. Large projects
will need some degree of hierarchical
structure to ensure accountability.

The principle, “at regular intervals,
the team reflects on how to become
more effective, then tunes and adjusts
its behavior accordingly,” requires an
adaptive mindset, and given the cultur-
al distances, may not be possible at the
vendor end. The notion of self-organiz-
ing teams and adjusting behavior may
be too much for organizational cultures
built on high power distance (such as
accepting power inequality) and high
uncertainty avoidance. Organizational
learning and change can take time.

Finally, the principle, “agile process-
es promote sustainable development;
the sponsors, developers, and users
should be able to maintain a constant
pace indefinitely,” can be controlled at
the client’s end only. Unless the vendor
company is owned by the client, the is-
sue of constant pace should be left to
the former. Given the agile practice of
“welcoming changing requirements,
even late in development,” this prac-
tice is easier said than done.

conclusion
It is evident that agile practices will
need to be modified for outsourced
software projects that tend to be typi-
cally of medium or large size. To eval-
uate the claims made in the paper,
researchers need to conduct empiri-
cal research based on case studies,
surveys, and action research. Such re-
search could lead to a hybrid approach
that harmonizes agility and discipline,
and adapts it based on the context and
the environment.

References
1. Ashkenas, R. N. The boundaryless organization:

Breaking the chains of organizational structure (2nd
Ed.). Jossey-Bass, 2002. San Francisco, CA.

2. Beck, K. Extreme programming explained: Embrace
change (2nd Ed.). Addison-Wesley, Reading, MA, 2004.

3. Boehm, B. W., and Turner, R. Balancing agility and
discipline: A guide for the perplexed. Addison-Wesley,
Boston, 2003.

figure 2.

figure 3.

148 c o m m u n i c at i o n s o f t h e a c m | s e p t e m b e r 2 0 0 9 | v o l . 5 2 | n o . 9

contributed articles

4. Brown, D., and Wilson, S. The black book of
outsourcing: How to manage the changes, challenges,
and opportunities. John Wiley & Sons, Hoboken, N.J.,
2005.

5. Cohen, L. and Young, A. Multisourcing: Moving beyond
outsourcing to achieve growth and agility. Harvard
Business School Press, Boston, 2006.

6. Hammer, M. and Champy, J. Reengineering the
Corporation: A manifesto for Business Revolution.
Harper Business, New York, 1993.

7. Holmstrom, H., Fitzgerald, B., Ågerfalk, P. J., and
Conchúir, E. O. Agile practices reduce distance in
global software development. Information Systems
Management, 23, 3, (2006), 7-18.

8. Keyton, J. Communication & organizational culture:
A key to understanding work experiences. Sage
Publications, Thousand Oaks, CA, 2005.

9. Kircher, M., Jain, P., Corsaro, A., and Levine, D.
Distributed extreme programming. XP2001
Conference, Sardinia, Italy, 2001.

10. Olson, J. S., and Olson, G. M. Culture surprises in
remote software development teams. ACM Queue 1,
9, (2003), 52-59.

11. Sahay, S. Global software alliances: The challenge of
standardization. Scandinavian Journal of Information
Systems, 15, (2003), 3-21.

12. Simon, H.A. The proverbs of administration Public
Administration Review 6, 1, Winter 1946, 53-67.

Dinesh Batra (Dinesh.Batra@business.fiu.edu) is a Knight-
Ridder Research Professor in the College of Business
Administration at the Florida International University. He
is a co-author of the textbook Object-Oriented Systems
Analysis and Design, Pearson Prentice-Hall.

COMMUNICATIONS OF THE ACM October 2007/Vol. 50, No. 10 79

Web services are rapidly becoming the
defacto distributed enterprise comput-
ing technology for realizing and man-
aging multi-party business processes
[5]. By now, most enterprises have
gained some early experience in devel-
oping internal Web services in a first
attempt to leverage their legacy assets
and to accelerate development and
integration of their enterprise applica-
tions. However, most of today’s enter-
prises are still far from benefiting from
the full-fledged potential of Web ser-
vice technology embodied in its ability
to mix-and-match internal and exter-
nal services alike.

In fact, it is common practice for
most enterprises to develop their Web
services by simply placing a thin
SOAP/WSDL/UDDI layer on top of
existing enterprise applications or soft-

ware components. This is in no way suf-
ficient for implementing and managing
commercial-strength, service-enabled
enterprise applications. While simple
(internal) Web services may be con-
structed that way, a development life
cycle methodology is of crucial impor-
tance to design, construct, monitor, and
manage enterprise SOA applications
supporting existent business process
ecosystems that are highly complex and

BY MICHAEL P. PAPAZOGLOU AND WILLEM-JAN VAN DEN HEUVEL

An innovative
roadmap brings together
the worlds of business
processes and Web services,
harnessing their power to
construct industrial-strength
business applications.

BUSINESS PROCESS DEVELOPMENT

Life Cycle
Methodology

80 October 2007/Vol. 50, No. 10 COMMUNICATIONS OF THE ACM

agile in nature. This article describes the insights of a
methodology and roadmap that assists service
providers and service aggregators in assembling multi-
party business processes, such as invoicing and inven-
tory control, from Web services.

METHODOLOGY ROADMAP
The service-oriented business process development
methodology is based on a roadmap that comprises
one preparatory phase to plan development, and
five distinct phases that concentrate on business
processes: analysis and design (A&D), construc-
tion and testing, provisioning, deployment, and
execution and monitoring (see Figure 1).

The stepwise transition through these phases
tends to be incremental and iterative in nature and
should accommodate revisions in situations where
the scope cannot be completely defined a priori.
This allows an approach that is one of continuous
invention, discovery, and implementation with each
iteration, forcing the development team to drive the
software development artifacts closer to completion
in a predictable and repeatable manner. This
approach considers multiple realization scenarios for
business processes and Web services that take into
account both technical and business concerns.

The basic functioning of the methodology is
exemplified by means of an order management
process that involves a client, a seller, and a trusted
third party. This process is organized as follows. On
receiving a purchase order from a client, five tasks
are executed concurrently at the seller’s site: check-
ing the credit worthiness of the customer, deter-
mining whether or not an ordered part is available
in the product inventory, calculating the final price
for the order and billing the customer, selecting a
shipper, and scheduling the production and ship-
ment for the order. While some of the processing
can proceed concurrently, there are control and data
dependencies between these tasks. For instance, the
customer’s creditworthiness must be ascertained
before accepting the order, the shipping price is
required to finalize the price calculation, and the
shipping date is required for the complete fulfillment
schedule.

DESIGN GUIDELINES
Business processes developed on the basis of existing
or newly coded services must rely on sound service
design principles to ensure autonomous, self-con-
tained and modular business processes with clearly
defined boundaries and discrete service endpoints.
Two key principles serve as the foundation for ser-
vice- and business-process design: service coupling
and cohesion.

Service coupling. Service coupling refers to the
degree of interdependence between business processes
or Web services. The design objective is to minimize
coupling, that is, to make business processes as self-
contained as possible by ensuring they need virtually
no knowledge or service of any other business
processes, thus increasing their maintainability and
reuse potential. Similarly, services that are orches-
trated inside a process should be loosely coupled to
each other, avoiding interdependencies at the level of
data or control logic. It is useful to distinguish
between three ways to achieve service and process
coupling:

1. Representational coupling: Business processes
should not depend on specific representational or
implementation details and assumptions underlying
business processes. That is, business processes do not
need to know the scripting language that was used to
compose their underlying services. Representational
coupling is useful for supporting:

Figure 1. Business
process development life

cycle methodology.

Services that are orchestrated inside a process should be
loosely coupled to each other, avoiding interdependencies at
the level of data or control logic.

• Interchangeable/replaceable services: Existing services
may be swapped with new service implementa-
tions—or ones supplied by a different provider
offering better performance or pricing—without
disrupting the overall business process functional-
ity;

• Multiple service versions: Different versions of a ser-
vice may work best in parts of a business processes
depending on the application’s needs. For example,
a purchase order service may provide different
detail levels; it may provide, for example, internal
or external requisition details such as internal or
external accounts, depending on the ordering
options.

2. Identity coupling: Connection channels between
services should be unaware of who is providing
the service. It is not desirable to keep track of the
targets (recipients) of service messages, especially
when they are likely to change or when discover-
ing the best service provider is not a trivial matter.

3. Communication protocol coupling: The number of
messages exchanged
between a sender and
addressee in order to
accomplish a certain
goal should be mini-
mal, given the com-
munication model,
such as one-way,
request/response, or
solicit/response. For
example, a one-way
form of communica-
tion, where a service
endpoint receives a
message without hav-
ing to send an
acknowledgment places the lowest possible
demands on the service performing the operation.
The service that performs such an operation
assumes nothing about the consequences of send-
ing the message.
Service cohesion defines the degree of strength of

functional relatedness between operations in a service
or business process. The service aggregator should
strive to offer cohesive (autonomous) processes whose
services and service operations are strongly and gen-
uinely related to one another. The guidelines by
which to increase cohesion are as follows:
1. Functional service cohesion: A functionally cohesive

business process should perform one and only one
problem-related task and contain only services nec-
essary for that purpose. For an order management
business process, such services include get-product-

price, check-product-availability, and check-credit-
worthiness

2. Communicational service cohesion: A communica-
tionally cohesive business process is one whose
activities and services use the same set of input and
output messages. Such business processes are
cleanly decoupled from other processes as their
activities are hardly related to activities in other
processes.

3. Logical service cohesion: A logically cohesive business
process is a one that performs a set of independent
but logically similar functions (alternatives) that are
tied together by means of control flows, such as
mode of payment.

PHASE 1. THE PLANNING PHASE
Planning sets the scene for all ensuing phases by ana-
lyzing the business case for all feasible combinations
of development approaches and realization strategies
(see Figure 2). Fundamentally, business case analysis
embraces two activities: gap analysis and scenario

analysis.
Gap analysis com-

mences with comparing
planned services with avail-
able software service imple-
mentations that may be
assembled within the enclo-
sures of a newly developed
business process. Four cate-
gories of existing service
resources are discerned:
legacy systems, commercial
off-the-shelf (COTS) com-
ponents, enterprise resource
planning (ERP) packages,

and existing Web services. We collectively refer to these
categories as the Web services landscape. Gap analysis
matches high-level descriptions of new services that
collectively make up a business process against the
available resources in the Web service landscape.

Scenario analysis. Gap analysis is intertwined with
a scenario analysis, which considers costs, risks, bene-
fits, and return of investment of the following options
to develop new business processes:

1. Green-field development: Follows a classical devel-
opment model covering specification down to devel-
opment. With this option we assume that the business
processes must be developed from scratch without any
reuse of existing process chunks or service specifica-
tions.

2. Top-down development: Usually a blueprint of
business use cases provides the skeleton specification
for business services. The top-down process decom-

COMMUNICATIONS OF THE ACM October 2007/Vol. 50, No. 10 81

Figure 2. Scenario analysis.

poses the business domain into its functional areas
and subsystems, including its flow or process decom-
position into processes, sub-processes and high-level
business use cases.

3. Bottom-up development: Starts from existing
enterprise applications and repositories and trans-
forms them to new services by wrapping.

4. Out-of-the-middle development: Combines top-
down and bottom-up development by allowing ser-
vice aggregators to select those fragments of enterprise
resources that best satisfy new business process
requirements. Fragments are then retrofitted using
services.

If gap analysis results in high functional fit, this
points toward a bottom-down approaches for the
remainder of the development process, whereas
green-field or top-down development approach are
best in cases of low functional fit. Out-of-the-middle
development may be preferred in those cases where
legacy assets have considerable overlap with new ser-
vice requirements and lack substantial business value.

One of the issues with current top-down and bot-
tom-up development is that they are rather ambigu-
ous regarding which business processes an enterprise
should start from and how these business processes
can be combined to form business scenarios. To
address this problem, service development solutions
must target specific focal points and common prac-
tices within the enterprise, such as those specified by
its corresponding sector reference models. Reference
models—a typical example of which is RosettaNet—
address a common large proportion of the basi

“plumbing” for a specific sector, from a process, oper-
ational function, integration, and data point of view.
Such a verticalized development model presents the
ideal architecture for supporting service development
and will be used throughout this article to explicate
elements of the services development methodology.

Each development option we have discussed thus
far comes equipped with one or more of the four real-
ization strategies:

• Reuse existing (internal) Web services or business
process logic;

• Develop new Web services or business processes

logic from scratch;
• Purchase/outsource Web services or (parts of )

business processes;
• Revamp existing (COTS) components or existing

(ERP/legacy) systems in Web services or business
processes.

During scenario analysis, estimates for existing and
expected operational costs, integration costs, service
and process customization costs, service and process
provisioning costs, and architecture costs are assessed.
Architecture costs are associated with acquiring arti-
facts for realizing the target architecture including
hardware, software (OS), training, and network
bandwidth. Service acquisition costs are based on a
one-off or annuity-based payment for implementa-
tion and usage of the software and are different from
service provisioning costs. For complex business-
related Web services, service providers may operate
different charging alternatives, incurring various types
of costs for the service aggregator, including usage
costs, subscription costs, leasing costs, and free ser-
vices.

Risk analysis weighs benefits and costs in the pro-
visioning scenarios and verifies feasibility before a par-
ticular option is chosen. During risk analysis, both
the impact and probability of an event that may occur
is estimated. This is expressed in terms of return on
investment (ROI), which is calculated for each devel-
opment option as the ratio of net benefits over costs.
After calculating the ROI for each scenario and assess-
ing the technical quality of available service resources,
the most appropriate design option may be selected.

Essentially, the planning phase plots a strategy for
realizing new business processes, given the risk that is
involved, and adopts the best-fitting process model.
High-risk projects may require a “chicken-little”
approach advocating gradual migration, whereas low-
risk projects allow a “big-bang” strategy.

PHASE 2. SERVICE AND PROCESS ANALYSIS AND DESIGN
Service analysis aims at identifying, conceptualizing,
and rationalizing business processes as a set of inter-
acting Web services. This step is logically succeeded
by service design, during which conceptual processes

82 October 2007/Vol. 50, No. 10 COMMUNICATIONS OF THE ACM

One of the issues with current top-down and bottom-up development
is that they are rather ambiguous regarding which business processes an
enterprise should start from and how these business processes can be
combined to form business scenarios.

and services are transformed into a set of related,
platform-agnostic interfaces.

Service analysis and design. Service interfaces are
identified in the problem domain by carving out and
grouping service capabilities based on service coupling
and cohesion criteria. For example, consider the ship-
ping service. By applying the dominant cohesion cri-
terion, namely functional cohesion, this service is
decomposed into service operations like “select ship-
per” and “check product
availability.

In case reference models
are available (such as the
XML common business
library, xCBL, or Roset-
taNet), business functions
can be derived on their
basis. For example, the
order management busi-
ness process in Figure 2
could be derived on the
basis of RosettaNet’s part-
ner interface process (PIP),
“manage purchase order”
(PIP3A4).

Service specification.
Here, we will briefly examine the process of writing an
interface specification for a Web service. We adopt the
standard, XML-based Web service definition lan-
guage (WSDL) [3] for this purpose. This process basi-
cally comprises four steps: describing the service
interface, specifying operation parameters, designat-
ing the messaging and transport protocol, and finally
fusing port types, bindings, and actual location (URI)
of the Web services. These steps themselves are rather
trivial and are already described in-depth in the liter-
ature [1].

Identifying processes. The objective of this step is
to identify services that may be aggregated into a busi-
ness process whose interface has a high viscosity.
Again, we can apply the design principles of coupling
and cohesion to achieve this. For example, the order
management process has low communication proto-
col coupling with the material requirements process.
In case of top-down development, reference models
are given that standardize processes and correlated ser-
vices and serve as a point of reference for new services.

Process identification could, for instance, start with
comparing the abstract business process layout with
RosettaNet’s “order management” partner interface
process. This PIP encompasses four complementary
process segments supporting the entire chain of activ-
ities from purchase order creation to tracking-and-
tracing, each of which is further decomposed into

multiple individual processes. A quick-scan of this
PIP reveals that two of the activities, “quote and order
entry” and “returns and finance” could be combined
into the process “order management.”

Specifying processes. Once business processes are
extracted and their boundaries are clearly demarcated,
they need to be described in the abstract in four steps:

1. Determine style of service composition.
The service designer may choose from two basic

types of service composi-
tion: orchestration and
choreography. Orchestra-
tion describes how Web
services can interact with
each other at the message
level, including the busi-
ness logic and execution
order of the interactions
from the perspective and
under control of a single
endpoint. (For an exam-
ple of this type of process
flow, see Figure 3, where

the business process flow is designed from the vantage
point of a supplier.) Orchestration refers to an exe-
cutable business process that may result in a long-
lived, transactional, multi-step process model. With
orchestration, the business process interactions are
always controlled from the (private) perspective of one
of the business parties involved in the process.

Choreography is typically associated with the pub-
lic (globally visible) message exchanges, rules of inter-
action and agreements that occur between multiple
business process endpoints, rather than a specific busi-
ness process that is executed by a single party. Chore-
ography is more collaborative in nature than

orchestration.

It is described from the perspective of
all parties (common view), and defines the comple-
mentary observable behavior between participants in
business process collaboration.

Orchestration is the preferred composition style for
internal business processes with tightly coupled inter-
nal actors, while choreography is typically chosen for
developing cross-organizational business processes
between loosely coupled partners. Orchestration and
choreography may also be applied simultaneously. For
example, in Figure 3 the client and supplier are shown
to integrate their business processes. This figure
assumes that the respective business analysts at both
companies have agreed upon both the processes as
well as the service level agreements (SLAs) involved
for the process collaboration. Using a graphical user
interface (GUI) and a tool that can serve as a basis for
the collaboration, the client and supplier agree upon

COMMUNICATIONS OF THE ACM October 2007/Vol. 50, No. 10 83

Figure 3. Combining
choreography and

orchestration.

their interactions and generate a choreography, such
as a WS-CDL (choreography description language)
representation [4]. This representation can then be
used to generate an orchestration, such as a business
process execution language for Web services (BPEL)
workflow template for both the manufacturer and
supplier. BPEL is the standard industry specification
designed specifically for Web services based orchestra-
tion [2]. The two BPEL workflow templates reflect
the business agreement.

2. Determine objectives and derive the business
process structure.

The private services of the client and supplier can
then be scripted using BPEL. This language enforces
the representational coupling requirement. In partic-
ular, BPEL is a platform-agnostic, block-structured
workflow-like language to define business processes
that may orchestrate one or more Web services.

A key part of the BPEL document is the definition
of the sequence of business activities required to han-
dle an incoming purchase order request. The internal
process flow of the supplier (see Figure 3, right side)
includes an initial request from a client, followed by
invocations of a scheduling service, credit service, and
billing service in parallel, and ultimately a response to
the client from the seller sending a invoice.

3. Describe business activity responsibilities
(roles).

The third step during business process design is to
identify responsibilities associated with business
process activities and the roles that are responsible for
performing them. Roles may invoke, receive, and
reply to business activities. The result of this phase
actually constitutes the foundation for implementing
business policies, notably role-based access control
and security policies. For example, the supplier may
play the role of billing, credit checking, and schedul-
ing agency.

4. Identify non-functional business process con-
cerns.

A service development methodology must go far
beyond ensuring “simple” functional correctness, and
deal with non-functional process design concerns
including performance, payment model, security

model, and transactional behavior. Here, we will
briefly mention typical business-related, non-func-
tional concerns that are captured in an SLA.

SLAs provide a proven vehicle for not only captur-
ing non-functional requirements but also for moni-
toring and enforcing them. SLAs are special legal
agreements that encapsulate multiple concerns, and
symmetrically fuse the perspective of service supplier
and customer.

Mutual agreements can be partially realized by
specifying transactional quality of service (QoS) akin
to business interactions that contain business-related
atomicity properties including payment atomicity,
goods atomicity, (certified) delivery atomicity, and
conversation atomicity [6]. For example, the purchase
order business process adopts a multi-level atomicity
protocol signifying that after the customer is billed,
payment follows (payment atomicity); that producing
the order and shipment results in the goods being
delivered (goods atomicity) in the correct manner
(certified delivery atomicity). Moreover, the terms of
any negotiation sequence resulting in a purchase need
to be transcribed and kept in storage for future refer-
ence (conversation atomicity).

Another important ingredient of an SLA is security
policies to protect multi-party collaborations. Know-
ing that a new business process adopts a Web services
security standard such as WS-security is insufficient
information to guarantee successful composition.
The client must know if the services in the business
process actually require WS-security, what kind of
security tokens they are capable of processing, and
which ones they prefer. Moreover, the client must
determine if the service should communicate using
signed messages. If so, it must determine what token
type must be used for the digital signatures. Finally,
the client must decide on when to encrypt the mes-
sages, which algorithm to use, and how to exchange a
shared key with the service. For example, the purchase
order service in the order management process may
indicate it only accepts username tokens that are
based on signed messages using an X.509 certificate
that is cryptographically endorsed by a third party.

84 October 2007/Vol. 50, No. 10 COMMUNICATIONS OF THE ACM

Service provisioning is central to operating revenue-generating
Web services between organizations. The provisioning requirements for
Web services impose serious implications for the business process
development life cycle methodology.

PHASE 3. THE CONSTRUCTION AND TESTING PHASE
Once the service and process specifications have
reached a steady state, they need to be transformed
into service implementations, and subsequently val-
idated and verified against business service and
process specifications. Business process realization
typically adopts a verticalized development model.
This activity encompasses two broad steps: code Web
services and code business processes.

Code Web services. Web services that were specified
in WSDL may be programmed with any preferred pro-
gramming language as long as the language is capable
of supporting WSDL’s protocol bindings, including
XML-RPC and SOAP.

Code business processes. Once the Web services
have been codified, the flow logic that is comprised in
the BPEL specification is compiled and injected into a
BPEL execution engine. As a preparatory step, the busi-
ness process is revamped as a coarse-grained asynchro-
nous operation, which is, like any other service, defined
in WSDL.

PHASE 4. THE PROVISIONING PHASE
Service provisioning is central to operating revenue-
generating Web services between organizations. The
provisioning requirements for Web services impose
serious implications for the business process develop-
ment life cycle methodology. Tasks executed during
this phase involve making choices for service gover-
nance, service certification, service enrolment, service
auditing, metering, billing, and managing operations
that control the behavior of a service during its use. As
such, they entail a complex mixture of technical and
business aspects for supporting service client activities.

PHASE 5. THE DEPLOYMENT PHASE
The tasks associated with the deployment phase of the
Web service development life cycle include the fol-
lowing three steps:

• Publish the service interface. The publish operation
is an act of service registration or service advertise-
ment, such as using the Universal Description,
Discovery and Integration (UDDI) protocol. It
acts just like a contract between the service registry
and the service provider.

• Deploy the Web service and business process. The run-
time code for the service and process and any
deployment meta-data that is required to run it are
deployed during this step.

• Publish service implementation details. The service
implementation definition contains the definition
of the network-accessible endpoint(s) where the
Web service can be invoked.

PHASE 6. THE EXECUTION AND MONITORING PHASE
During this phase, the business processes and sup-
porting Web services are fully deployed and made
operational. By doing so, a service requestor can find
the service definition and invoke all defined service
operations. Also, the progress of running business
processes can be monitored. For the time being only
reactive monitoring mechanisms are in place.

CONCLUSION
Currently, enterprise applications constructed with
Web services tend to be developed in a rather ad hoc
fashion. This unavoidably leads to processes and
enterprise systems with nonexistent architectures, or
fragile ones that are hard to maintain, reuse, and
extend. Even more importantly, these practices do
not take into account important business and orga-
nizational considerations, such as the leasing costs of
Web services and organizational issues, making them
unsuitable for constructing industrial-strength busi-
ness applications.

The methodology presented in this article reflects
an attempt in defining a foundation of development
principles for Web services based on which real-world
business processes can be assembled into business sce-
narios. Currently, we are enriching the methodology
with patterns, (design and coding) templates that are
derived from empirical material. Next, we aim to
develop an integrated toolset supporting the phases of
the methodology.

References
1. Alanso, G., Casati, F., Kuno, H., and Machiraju, V. Web Services: Con-

cepts, Architectures and Applications. Springer, Heidelberg, 2004.
2. Andrews, T. et al. Business Process Execution Language for Web Ser-

vices, Version 1.1.
3. Chinnici, R., Gudgin, M., Moreau, J.J., Schlimmer, J., and Weerarana,

S. Web Srvices Description Language (WSDL), Version 2.0, (Mar.
2004); w3c.org.

4. Kavantas et al. Web Service Choreography Description Language, Ver-
sion 1 working draft (Aug., 2004); w3c.org.

5. Papazoglou, M.P. and Georgakapoulos, G. Introduction to the special
section on service-oriented computing. Commun. ACM 46, 10 (Oct.
2003), 24–29.

6. Papazoglou, M.P. Web Services: Principle and Technology. Prentice Hall,
July 2007.

Michael P. Papazoglou (mikep@uvt.nl) is a professor at
Tilburg University in the Netherlands.
Willem-Jan van den Heuvel (wjheuvel@uvt.nl) is an associate
professor at Tilburg University in the Netherlands.

Permission to make digital or hard copies of all or part of this work for personal or class-
room use is granted without fee provided that copies are not made or distributed for
profit or commercial advantage and that copies bear this notice and the full citation on
the first page. To copy otherwise, to republish, to post on servers or to redistribute to
lists, requires prior specific permission and/or a fee.

COMMUNICATIONS OF THE ACM October 2007/Vol. 50, No. 10 85

14 I E E E S o f t w a r E P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y 0 7 4 0 – 7 4 5 9 / 0 9 / $ 2 5 . 0 0 © 2 0 0 9 I E E E

E d i t o r : S t a n R i f k i n n M a s t e r S y s t e m s n s r @ m a s t e r – s y s t e m s . c o m

viewpoints

Software Development
Effort Estimation:
Formal Models or Expert Judgment?

Magne Jørgensen and Barry Boehm

Which is better for estimating software project resources: formal models, as instantiated in estimation
tools, or expert judgment? Two luminaries, Magne Jørgensen and Barry Boehm, debate this question
here. Outside this article, they’re colleagues with a strong inclination to combine methods. But for this
debate, they’re taking opposite sides and trying to help software project managers figure out when, and
under what conditions, each method would be best.

While it might be less argumentative—and certainly less controversial—to agree that the use of both
methods might be best, Magne is making a stronger point: Perhaps our refinement of and reliance on the
formal models that we find in tools is wrong headed, and we should allocate our scarce resources toward
judgment-based methods. (For a primer on both methods, see the sidebar.) — Stan Rifkin

M
agne: Formal software development
effort estimation models have been
around for more than 40 years. They’re
the subject of more than a thousand re-
search studies and experience reports.
They’re described and promoted in

many software engineering textbooks and guide-
lines. They’re supported by user-friendly tools and

advisory services from consultancy companies. In
spite of this massive effort and promotion, available
empirical evidence shows that formal estimation
models aren’t in much use.1 Formal effort estima-
tion models have had more than sufficient oppor-

tunities to become a success story; it’s now time to
move on and focus industrial estimation process
improvement work and scientific research on judg-
ment-based effort estimation methods. This is a
much more promising way forward for several rea-
sons. Here are three.

First, 10 out of the 16 studies reviewed in “Esti-
mation of Software Development Work Effort: Evi-
dence on Expert Judgment and Formal Models” re-
port that using (typically simple and unstructured)
judgment-based effort estimation methods led to
more accurate effort estimates than using sophis-
ticated formal models.2 The introduction of more
structure and more supporting elements, such as
structured group processes and experience-based
estimation checklists, could even further improve
judgment-based effort estimates. See, for example,
Principles of Forecasting: A Handbook for Re-
searchers and Practitioners for general forecasting
evidence on the benefits of several types of expert-
judgment process structures and support.3

Second, so far, little work has been done on im-
proving judgment-based effort estimation processes
compared with that on formal effort estimation

M arch/April 20 0 9 I E E E S o f t w a r E 15

ViEwpointS

models. As an illustration, Martin Shep-
perd and I found that only 15 percent of the
journal papers on software development
effort estimation analyzed judgment-based
effort estimation.4 Unlike most papers on
formal effort estimation models, the great
majority of these papers didn’t aim at im-
proving judgment-based effort estimation
processes.

Third, most of the software industry, as
far as I’ve experienced, is much more will-
ing to accept, understand, and properly use
judgment-based estimation methods. I have,
for example, repeatedly observed that soft-

ware projects officially applying a formal
estimation model actually use the model as
a disguise for expert estimation. This low
acceptance and “models as judgment in dis-
guise” attitude appear in several forecasting
disciplines and shouldn’t be surprising. An
important reason for the rejection of for-
mal models might be that experts’ highly
specific knowledge—for example, about
the developers who are supposed to do the
work—frequently can’t be included prop-
erly as model input. It’s understandably
difficult to trust an estimation method un-
able to make use of strongly relevant infor-

mation. It’s hardly possible to unite highly
specific knowledge with the need to estab-
lish general relationships in formal models;
that is, this limitation isn’t a question of im-
proved models.

In short, there are very good reasons to
claim that future estimation process im-
provement and research initiatives should
aim at better judgment-based effort esti-
mation processes and not at better formal
models.

Although formal effort estimation
models might seem primarily objective
and repeatable, I believe they aren’t. All

The Difference between Model- and Expert-Judgment-Based Estimation

table 1
A comparison of the processes for expert-based and model-based estimation

Estimation activity Expert judgment Estimation models

Understand the
estimation problem.

Judgment- and analysis-based processes possibly
structured by templates for work breakdown.

Judgment- and analysis-based processes possibly
structured by the need for model input if the model
has already been chosen at this stage.

Agree on decisions and
assumptions relevant
for estimation.

Judgment- and analysis-based processes possibly
structured by checklists or guidelines.

Judgment- and analysis-based processes possibly
structured by checklists or guidelines. Several assumptions
implicitly made when selecting a model.

Collect information
relevant for the
estimation.

Judgment- and analysis-based processes possibly
structured by checklists or guidelines. The set of
variables considered relevant is judgment based. The
collected information includes subjective assessments.

Judgment- and analysis-based processes structured by the
model’s need for input data. The set of variables considered
relevant is predetermined. The collected information
typically includes subjective assessments.

Evaluate the
importance (weight-
ing) of different pieces
of information.

Judgment-based processes. Analysis-based processes based on statistical analysis
of historical data.

Quantify the effort
on the basis of the
information.

Judgment-based processes. Analysis-based processes based on a formalized,
repeatable process or formula.

Review the effort
estimate.

Judgment- and analysis-based processes possibly
structured by checklists, guidelines, and expert
judgment.

Judgment- and analysis-based processes possibly
structured by checklists, guidelines, and expert judgment,
some times leading to judgment-based updates of the
effort estimate.

The essential difference between formal-model-based and
expert-judgment-based effort estimation is the quantifica-
tion step—that is, the final step that transforms the input
into the effort estimate. Formal effort estimation models,
such as Cocomo and function-point-based models, are
based on a mechanical quantification step such as a
formula. On the other hand, judgment-based estima-
tion methods, such as work-breakdown structure-based
methods, are based on a judgment-based quantification
step—for example, what the expert believes is the most
likely use of effort to complete an activity. Judgment-
based estimation processes range from pure “gut feel-

ings” to structured, historical data and checklist-based
estimation processes. That is, there are many other ways
of improving judgment-based processes than selecting the
best expert.

Some estimation methods, such as the Planning Poker
method (www.planningpoker.com/detail.html), can be
implemented as judgment- or model-based, depending
on whether the final step is mechanically based on the
velocity (productivity) of previous iterations. This illustrates
that the difference between models and expert judgment
isn’t always clear. Table 1 summarizes typical similarities
and differences of the processes involved.

16 I E E E S o f t w a r E w w w . c o m p u t e r . o r g / s o f t w a r e

ViEwpointS

meaningful estimation models require
judgment to produce the input to the
models. This might include judgment of
complexity, team skill, and the custom-
ers’ requirements for the system. This
means that software companies benefit
from a stronger focus on better judgment-
based processes—even when they choose
to apply estimation models, for example,
in combination with expert judgment.

Barry: First, let me say that I think Magne’s
published analyses are well done and im-
portant to understand. However, I don’t
think that a 10/16 = 62.5 superiority per-
centage for expert judgment in empirical
studies is conclusive, particularly because
there are uncertainties in whether the re-
sults of empirical studies are representative
of results in project practice. If most people
are using expert judgment methods in prac-
tice, the results aren’t encouraging. The
last few years of Standish Group surveys
indicate that only about 30 percent of the
projects surveyed successfully deliver their
software within their estimates.5 T. Capers
Jones’ analysis of 50 completed projects in-
dicated that only four (8 percent) were esti-
mated within 10 percent of their actuals.6
However, these data do support Magne’s
call for more research into better methods
of judgment-based estimation.

One candidate explanation of this dis-
crepancy is that the expert estimates pro-
duced in empirical studies aren’t represen-
tative of estimates produced in practice.
Some nonrepresentativeness occurs be-
cause not everyone is an expert, nobody
is an expert in all domains, and empirical
studies based on estimates and data from
completed projects aren’t representative
of estimates made when they’re needed,
at the beginning of projects. And it leaves
open the question of what to do if no suit-
able experts are available.

A well-documented problem with the
accuracy of estimates, particularly early in
the life cycle, is the Cone of Uncertainty,
which indicates that early estimates might
be off by factors of two to four, simply be-
cause not enough is known about either
the requirements or the solutions (for ex-
ample, the feasibility of using off-the-shelf
components).7,8 When faced with such
uncertainties, many organizations highly
value the ability to perform extensive sen-
sitivity, risk, and trade-off analyses. The

number of cases involved in such analy-
ses goes well beyond the available time of
experts to perform, whereas parametric
models can run them rapidly, with little
human effort.

When comparing expert-judgment ver-
sus parametric-model estimates, another
factor might cause parametric models to
appear less accurate. The organization’s
counting rules used for software size,
cost, and schedule might differ from those
used to calibrate the models. For size, this
might involve differences between logi-
cal and physical lines of code, counting of
nonexecutable statements, and counting of
support software such as deliverable tools
and test drivers. For cost and schedule, it
might involve differing phases counted
(programming versus full development
cycle), work hours counted (training, per-
sonal email, and overtime), and activi-
ties counted (configuration management,
quality assurance, data preparation, and
hardware and software integration). For
Cocomo, my colleagues and I found fac-
tor-of-three differences between an evalu-
ation study’s data and data conforming to
Cocomo’s published definitions.

In situations involving high cost and
schedule uncertainty, it’s a good idea to
draw upon as many sources of insight
as possible. Parametric models contain a
good deal of information about which fac-
tors cause software costs to go up or down,
and by how much. This information is
based on extensive analyses of real-project
data and feedback from real-project users.
Their absolute estimates might differ from

your actuals owing to the causes I just
mentioned, but their relative estimates of
the effects of various cost drivers are gen-
erally fairly accurate. As a bottom line, I
agree with Magne’s recommendation for
more research on better methods of expert-
judgment estimation. But I would strongly
disagree with any recommendations to
stop people from performing research on
which factors affect software costs and by
how much, or to discourage people from
using insights provided by parametric
models.

Magne: Barry and I agree that the accu-
racy of judgment-based effort estimation
isn’t exactly amazing. Judgment-based es-
timates are far from perfect. For example,
they frequently involve a high degree of
wishful thinking and inconsistency. The
surprising observation is therefore that es-
timation models haven’t managed to pro-
duce more accurate effort estimates! This
observation is even more surprising when
you consider that models seem to out-
perform expert judgment in most other
disciplines.9

Barry suggests that poor model accu-
racy can be a result of size-counting rules
that differ from those used to calibrate the
model. I share his concern about the lack
of standardized size measurement and
agree that there’s an improvement poten-
tial related to the practical use of models.
His suggestion does, however, also nicely
illustrate how complex and time consum-
ing it can be to use effort estimation mod-
els properly. An organization must there-
fore decide whether it should prioritize its
limited resources on training and monitor-
ing people’s size-counting processes, or on
better methods and support for judgment-
based effort estimation.

Estimation models have inherent prob-
lems that make the latter choice more
worthwhile and that explain the empirical
results in favor of the expert judgment. In
particular, the relation between effort and
size in software development contexts isn’t
stable. The results reported in “On the
Problem of the Software Cost Function,”10
based on an evaluation of 12 data sets, il-
lustrates how much the functional relation
between effort and size varies from data
set to data set. This lack of relation sta-
bility is a problem not only between data
sets but also within data from the same

Judgment-based
estimates are far from

perfect. For example, they
frequently involve a high

degree of wishful thinking
and inconsistency.

M arch/April 20 0 9 I E E E S o f t w a r E 17

ViEwpointS

organization. Essential effort relation-
ships might, for example, change over time
owing to learning or technology change.
In situations with unstable relationships,
expert judgment has been shown to be
preferable.11

In addition, as mentioned before, soft-
ware development situations frequently
contain highly specific, highly important
information. Take, for example, the infor-
mation that there’s a serious conflict be-
tween the project leader Peter and his best
programmer Paul. This conflict will likely
have a serious impact on the project’s pro-
ductivity. No effort estimation model, as
far as I know, integrates this kind of rela-
tion mechanically. A model might choose
to integrate this type of highly specific in-
formation through expert judgment—for
example, through expert judgment of how
much such conflicts will likely impact over-
all productivity. Then, however, dominant
model variables would be judgment-based
and subject to biases and problems similar
to those of a purely expert-judgment-based
process. Consequently, it isn’t easy to see
how the models would be able to perform
much better than experts in such situa-
tions. Several studies—for example, “On
Knowing When to Switch from Quantita-
tive to Judgemental Forecasts”12—suggest
that expert judgment can have great ad-
vantages in situations with highly specific
information that’s not mechanically inte-
grated, or integrated at all, in the model.

Even if models should tend to be less
accurate than expert judgment, software
organizations might, according to Barry,
benefit from using them as a means to
sensitivity, risk, and trade-off analyses.
I’m not so sure, and I’ve seen no empirical
evidence to support the usefulness of this.
Simulation of how a single variable affects
effort may require an ability to model all
the underlying relationships that’s even
better than that needed for effort estima-
tion. That’s why simulation models typi-
cally are much more complex than estima-
tion models.

The representativeness of the 16 em-
pirical studies comparing estimation mod-
els and judgment-based effort estimation
might not be optimal. And, of course, these
studies don’t prove that judgment-based
estimation models are on average better.
These studies do, on the other hand, pro-
vide the best available knowledge about

this issue and should therefore be impor-
tant input when deciding where to put es-
timation improvement effort. I hope, how-
ever, that there soon will be more studies
shedding light on when models and judg-
ment-based estimation processes are likely
to be more accurate.3

Barry: I’ll try to respond to Magne’s main
points in his second-round contribution.

First, Magne argues that properly using
estimation models requires more invest-
ment in data collection, model calibration,
education, and training. I agree. It corre-
lates with a recent Chinese survey of 153
people in 116 organizations involved in
process improvement, in which this invest-
ment was the highest-ranked (58 percent)
of the “barriers or difficulties of applying
cost models.”13

Often, though, organizations find these
investments worthwhile. Recently, my col-
leagues and I administered a survey to 16
acquisition managers at a software ac-
quisition cost and risk workshop and to
28 cost estimation professionals at a cost
estimation forum. They answered sev-
eral questions of the form “How much
value do current parametric cost models
contribute when you perform the follow-
ing functions, as compared to human-
judgment estimates?” on a scale of 1 (much
less) through 3 (about equal) to 5 (much
more).

The top-ranked relative values of para-
metric cost models (with the average re-
sponses for the two groups) were

preparing cost proposals (4.3, 4.4), n
evaluating cost proposals (3.75, 4.0), n
performing risk or sensitivity analyses n
(3.6, 3.8), and
making cost-schedule-performance n
trade-offs (3.45, 3.8).

The top-ranked relative values of human-
judgment estimates were

performing make-buy-outsource analy- n
ses (2.35, 2.7),
investing in productivity improvements n
(2.35, 2.8),
allocating project budgets (2.5, 3.05), n
and
making software evolution or phaseout n
decisions (2.7, 2.9).

You can see not only that the results
vary with the population being sampled
but also that the results vary even more
strongly by the cost estimation function
you’re trying to perform. So, at least in
these sampled populations, making a one-
size-fits-all decision on using models ver-
sus experts in all situations doesn’t appear
to be a good idea.

Second, Magne states that

The relation between effort and size in
software development contexts isn’t
stable.

Again, I would agree, and go further in say-
ing that this phenomenon has been well
known for quite a while and is the main
reason that parametric cost models have
incorporated additional productivity-driver
parameters to explain the variation. Chap-
ter 29 of Software Engineering Economics
shows the significant difference between the
original Basic Cocomo model with a size
parameter and a single-mode parameter
(estimating the 63 projects in the database
within a factor of 1.3 of the actuals only 29
percent of the time), and the Intermediate
Cocomo model with an added 15 produc-
tivity-driver parameters (within a factor of
1.3, 78 percent of the time).7 This provides
clear evidence that adding more parameters
to a simple model can increase its accuracy.

Several purported evaluations of the ac-
curacy of “Cocomo” didn’t have cost driver
data and could only evaluate the less-
accurate Basic Cocomo, in which case it
wasn’t surprising that what they called

Making a one-size-
fits-all decision on using
models versus experts
in all situations doesn’t

appear to be a good idea.

18 I E E E S o f t w a r E w w w . c o m p u t e r . o r g / s o f t w a r e

ViEwpointS

Cocomo didn’t produce very accurate
results.

Third, Magne says that

Software development situations
frequently contain highly specific,
highly important information [such
as personnel compatibility].

Such information is often incorporated in
the leading parametric models. For exam-
ple, both the Cocomo family and SEER14
productivity-driver parameters Analyst
Capability and Programmer Capability
rate team rather than average individual
capability, including such factors as raw
ability, efficiency and thoroughness, and
ability to communicate and cooperate.
It’s true, though, that parametric mod-
els never have enough calibration data
to accurately estimate the effect on the
productivity of 99th-percentile teams.
It’s also true that well-calibrated para-
metric models are biased toward the re-
sults of organizations that collect and are
willing to share their

productivity data.

These tend to be better-performing orga-
nizations, although highly capable agile
teams can be even more productive with-
out needing to collect and analyze data.

Finally, according to Magne,

Judgment-based estimates …
frequently involve a high degree of
wishful thinking.

I agree. A major advantage of a paramet-
ric model is that it doesn’t modify its esti-
mates when customers, managers, or mar-
keters apply pressure.

The only way you can get a paramet-
ric model to reduce its estimates is by go-
ing on record and accepting accountability
for making a visible change in the project’s
estimated size or in its productivity-driver
parameter ratings. Thus, using a calibrated
parametric model enables negotiation of
the price of a software development con-
tract to be driven by verifiable adjustments
of project size (by reducing functionality) or
productivity-driver parameters (for exam-
ple, by restaffing with more-capable peo-
ple), rather than by a contest of wills be-
tween self-described experts.

Barry: Because Magne has been gracious
enough to start first in Rounds 1 and

2 and let me have the last word in each
round, and because I think we’ve covered
a lot of the significant ground in those
rounds, I propose that we each make clos-
ing statements, that I’ll go first, and that
he can have the last word.

Rather than continue to discuss which
is better of two estimation methods that
I think are both complementary and use-
ful, I’d like to focus on what you should do
with the estimates that you’ve produced or
received. Unfortunately, for the many or-
ganizations that consistently overrun, the
usual practice is to discard them as hav-
ing served their purpose and to avoid fu-
ture embarrassment when the estimates
are overrun. However, by doing so, such
organizations lose any chance of learn-
ing through experience. They also remain
vulnerable to overly optimistic expert- or
model-based estimators.

The organizations I’ve seen that have
had the most success at meeting cost and
schedule commitments use a mix of para-
metric models and expert judgment. But
more important, they save the estimates
and use them in a closed-loop feedback
process to compare estimate inputs and
outputs with actual experiences. Any
mismatches not only can be corrected
during the project but also can be used to
calibrate the inputs for future project es-
timates. For example, consistently over-
optimistic expert estimators or model us-
ers can be warned or excluded, domain
experts can be calibrated for their opti-
mism or pessimism about how others
will succeed in their domain, or model

parameters can be recalibrated to evolv-
ing experience.

I used to think that closed-loop feed-
back and recalibration would enable orga-
nizations and models to become increas-
ingly perfect estimators. But I don’t any
more. The software field continues to re-
invent and rebaseline itself too rapidly to
enable this to happen. As a bottom-line
statement, we need to recognize that peo-
ple and models will never be perfect soft-
ware cost and schedule estimators, and
build this into our project plans.

A good way to do this is to use incre-
mental development and to use the pro-
cess called either timeboxing or cost and
schedule as an independent variable. This
involves customers prioritizing their de-
sired features, and developers architecting
their systems to make borderline-priority
features easy to add or drop. Then, as each
increment goes on and finds itself ahead of
or behind its estimated cost and schedule,
it can add or drop borderline features to
make the increment fit its budget or sched-
ule. This enables the customers to receive
their most valuable features in each incre-
ment. However, it also recognizes that if
estimation can’t be perfect, then the num-
ber of such features can’t be an indepen-
dent variable along with cost and schedule.
But again, it requires using the estimates
to monitor progress with respect to plans,
and applying closed-loop feedback pro-
cesses within each increment, across incre-
ments, and across projects to keep on top
of changing situations.

Magne: My main claim in this discussion
is that organizations’ process improvement
work and research initiatives should focus
on better judgment-based effort estimation
processes, not on introducing or improv-
ing formal estimation models. The claim is
backed up with results from a comprehen-
sive review of relevant empirical estima-
tion studies and explanations based on re-
sults from other forecasting disciplines. As
far as I can see, none of Barry’s arguments
provides evidence against this claim.

Barry’s arguments are instead related to
the observation that some software profes-
sionals find estimation models useful, that
model use can be improved, that we should
combine models and expert judgment, and
that more frequent feedback through incre-
mental models improves estimates. These

Well-calibrated
parametric models

are biased toward the
results of organizations

that collect and are
willing to share their

productivity data.

M arch/April 20 0 9 I E E E S o f t w a r E 19

ViEwpointS

claims are easy to agree on but don’t easily
translate to an organization’s decision on
whether to improve model- or judgment-
based estimation processes.

The current strong belief in models
among some software professionals and
many software engineering researchers
might be a result of myths about models
and expert estimation, possibly illustrated
by Barry’s claim that

use of a calibrated parametric model
enables negotiation of the price of a
software development contract to be
driven by objective adjustment of the
project’s size or productivity-driver
parameters, rather than by a contest of
wills between self-described experts.

The fi rst myth here is that models are
objective. As I stated earlier, they clearly
aren’t. Essential input to the models is
based on expert judgment and can conse-
quently be impacted by outside pressure.

The second myth is that judgment-
based effort estimation is a black-box
process leading to a contest of wills be-
tween self-described experts—that is, a
process that’s diffi cult to improve. This
isn’t true, either. There are many ways to
improve judgment-based effort estimation
and avoid contests of wills. Barry’s own
Wideband Delphi process and the guide-
lines suggested in “Practical Guidelines for
Expert-Judgment-Based Software Effort
Estimation”15 are examples of such ways.

A third myth, especially common
among software engineering researchers,
is that more advanced estimation models
will likely lead to substantially more ac-
curate effort estimates. For example, that
the intermediate version of the Cocomo
models or a neural-network-based esti-
mation model will likely be more accurate
than much simpler models. In software
engineering, as in most other similar fore-
casting disciplines, the perhaps most sta-
ble result is that simple models typically
perform just as well as more advanced
models and that more sophisticated mod-
els are vulnerable to overfi tting to out-
dated or unrepresentative data sets. The
improvement potential of the model side
will consequently likely be quite low.1,3

Although we clearly shouldn’t reject es-
timation models as providing no value, we
shouldn’t ignore available empirical evi-

dence, either. If the software engineering
discipline wants to become evidence based,
it should focus on improving those pro-
cesses for which the evidence suggests that
improvements will more likely occur. In
most effort estimation contexts, but hardly
all, the available evidence suggests a much
stronger focus on improving judgment-
based effort estimation processes.

Acknowledgments
We’re indebted to Steve McConnell, Mike
Cohn, and the anonymous reviewers for their
thorough, candid feedback. We also thank
Hakan Erdogmus, IEEE Software’s editor in
chief, for suggesting this format for the dis-
cussion of this controversy.

References
1. M. Jørgensen, “A Review of Studies on Expert

Estimation of Software Development Effort,”
J. Systems and Software, vol. 70, nos. 1–2,
2004, pp. 37–60.

2. M. Jørgensen, “Estimation of Software Devel-
opment Work Effort: Evidence on Expert Judg-
ment and Formal Models,” Int’l J. Forecasting,
vol. 23, no. 3, 2007, pp. 449–462.

3. J.S. Armstrong, ed., Principles of Forecasting:
A Handbook for Researchers and Practitio-
ners, Kluwer Academic Publishers, 2001.

4. M. Jørgensen and M. Shepperd, “A Systematic
Review of Software Development Cost Esti-
mation Studies,” IEEE Trans. Software Eng.,
vol. 33, no. 1, 2007, pp. 33–53.

5. J. Johnson, My Life Is Failure, Standish Group
Int’l, 2006.

6. T.C. Jones, Estimating Software Costs, Mc-
Graw-Hill, 1998.

7. B. Boehm, Software Engineering Economics,
Prentice Hall, 1981.

8. S. McConnell, Software Project Survival
Guide, Microsoft Press, 1998.

9. R.M. Dawes, D. Faust, and P.E. Meehl, “Clini-

cal versus Actuarial Judgment,” Science, vol.
243, no. 4899, 1989, pp. 1668–1674.

10. J.J. Dolado, “On the Problem of the Software
Cost Function,” Information and Software
Technology, vol. 43, no. 1, 2001, pp. 61–72.

11. R.G. Webby and M.J. O’Connor, “Judgement-
al and Statistical Time Series Forecasting: A
Review of the Literature,” Int’l J. Forecasting,
vol. 12, no. 1, 1996, pp. 91–118.

12. N.R. Sanders and L.P. Ritzman, “On Know-
ing When to Switch from Quantitative to
Judgemental Forecasts,” Int’l J. Operations &
Production Management, vol. 11, no. 6, 1991,
pp. 27–37.

13. D. Yang et al., “A Survey of Software Cost Es-
timation in the Chinese Software Industry,”
Proc. 2nd IEEE-ACM Int’l Symp. Empirical
Software Eng. and Measurement, ACM Press,
2008, pp. 253–262.

14. D. Galorath and M. Evans, Software Sizing,
Estimation, and Risk Management, Auer-
bach, 2006.

15. M. Jørgensen, “Practical Guidelines for
Expert-Judgment-Based Software Effort Esti-
mation,” IEEE Software, vol. 22, no. 3, 2005,
pp. 57–63.

Magne Jørgensen is a professor at the Simula Research
Laboratory. His research focuses on improving judgment-based
effort estimation. He has worked as a software developer and
project manager for several years and trained estimators to
use estimation models and expert-judgment-based estimation
processes. Contact him at magnej@simula.no.

Barry Boehm is a professor in the University of Southern
California’s Computer Science and Industrial and Systems Engi-
neering Departments. He led the development of the Cocomo
and Cocomo II parametric estimation models, the Wideband
Delphi method for expert-judgment estimation, the Bayesian
calibration method for combining expert judgment and data
analysis into a parametric estimation model, and the Agile
Cocomo II method and tool for combining analogy-based and
parametric-model-based estimation. Contact him at boehm@
usc.edu.

Stan Rifkin is the founder and principal of Master
Systems, an advisory service for organizations for which
computing is strategic. He was an associate editor in chief of
IEEE Software during an early stage of this debate. Contact him
at sr@master-systems.com.

One myth is that more
advanced estimation

models will likely
lead to substantially
more accurate effort

estimates.

Questions?
Comments?

IEEE Software wants
to hear from you!

software@computer.org
Email

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Information and Software Technology 53 (2011) 276–290

Contents lists available at ScienceDirect

Information and Software Technology

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / i n f s o f

Agile methods rapidly replacing traditional methods at Nokia: A survey
of opinions on agile transformation

Maarit Laanti a,⇑, Outi Salo a, Pekka Abrahamsson b
a Nokia Corporation, P.O. Box 407, 00045 Nokia Group, Finland
b University of Helsinki, P.O. Box 68, 00014 University of Helsinki, Finland

a r t i c l e i n f o

Article history:
Received 25 July 2009
Received in revised form 21 November 2010
Accepted 28 November 2010
Available online 3 December 2010

Keywords:
Software engineering
Agile software development
Software processes
Agile deployment

0950-5849/$ – see front matter � 2010 Elsevier B.V. A
doi:10.1016/j.infsof.2010.11.010

⇑ Corresponding author. Tel.: +358 71808000; fax:
E-mail addresses: maarit.laanti@nokia.com (M. Laa

Salo), Pekka.Abrahamsson@cs.helsinki.fi (P. Abrahams

a b s t r a c t

Context: Many organizations have started to deploy agile methods, but so far there exist only a few stud-
ies on organization-wide transformations. Are agile methods here to stay? Some claim that agile software
development methods are in the mainstream adoption phase in the software industry, while others hope
that those are a passing fad. The assumption here is that if agile would not provide real improvement,
adopters would be eager at first but turn pessimistic after putting it into practice.
Objective: Despite the growing amount of anecdotal evidence on the success of agile methods across a
wide range of different real-life development settings, scientific studies remain scarce. Even less is known
about the perception of the impacts of agile transformation when it is deployed in a very large software
development environment, and whether agile methods are here to stay. This study aims to fill that gap by
providing evidence from a large-scale agile transformation within Nokia. While we have yet to confirm
these findings with solid quantitative data, we believe that the perception of the impacts already pin-
points the direction of the impacts of large-scale agile transformation.
Method: The data were collected using a questionnaire. The population of the study contains more than
1000 respondents in seven different countries in Europe, North America, and Asia.
Results: The results reveal that most respondents agree on all accounts with the generally claimed ben-
efits of agile methods. These benefits include higher satisfaction, a feeling of effectiveness, increased
quality and transparency, increased autonomy and happiness, and earlier detection of defects. Finally,
60% of respondents would not like to return to the old way of working.
Conclusion: While the perception of the impact of agile methods is predominantly positive, several chal-
lenge areas were discovered. However, based on this study, agile methods are here to stay.

1. Introduction

Since the emergence of agile software engineering (SE) methods
in the mid 1990s, the software industry has grown increasingly
interested in them. Nowadays, a wide range of organizations claim
to be agile or report interest in the adoption of agile methods.
According to a Forrester study where IT decision-makers repre-
senting each a different company were interviewed during Sep-
tember 2006, around 17% of enterprises already used agile
methods, while more than half of the remaining companies were
interested in adopting them [1]. This implies that the benefits of
agile methods have been recognized in the IT software industry.
In addition, the existing numerous publications on agile methods
on both scientific and non-scientific forums mainly seem to report
positive impacts of agile adoption. However, adoption of agile

ll rights reserved.

+358 718068810.
nti), Outi.Salo@nokia.com (O.
son).

methods has proven to be a challenging task [2]. It has been
claimed that agile adopters are often unaware of what agile adop-
tion really means, and how broad a change is actually required [1].

While the benefits of agile methods have been widely acknowl-
edged, the strength of empirical evidence is still claimed to be
fairly low [1,3,4]. One reason for this is that companies rarely have
comparable data to explain the impact of agile methods before and
after adoption [1]. From a scientific viewpoint, the vast majority of
the agile-related research currently addresses qualitative case
studies of individual development projects and lacks broader
scope. In addition, a need has been identified for ‘‘more and better
empirical studies of agile software development within a common
research agenda’’ ([3] p. 833). Currently, it is still rare to come
across scientific studies presenting quantitative evidence on agile
methods to provide more generalizable results (e.g., [1,5,6]). In
addition, one challenge identified among the current agile software
engineering research is that the reported studies tend to focus on
the few most familiar agile methods, namely Extreme Program-
ming (XP) [7] and Scrum [8]. In fact, recent studies reveal an almost

http://dx.doi.org/10.1016/j.infsof.2010.11.010

mailto:maarit.laanti@nokia.com

mailto:Outi.Salo@nokia.com

mailto:Pekka.Abrahamsson@cs.helsinki.fi

http://dx.doi.org/10.1016/j.infsof.2010.11.010

http://www.sciencedirect.com/science/journal/09505849

http://www.elsevier.com/locate/infsof

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 277

exclusive focus on XP among agile-related SE studies (76%) while
proposing an urgent need to broaden the focus of studies to include
other agile methods [3].

Thus, the need to expand the current quantitative scientifically
valid research regarding various aspects of agile methods is evi-
dent. This study aims to provide empirical evidence from a survey
conducted in 2008 for a large product development organization
(Nokia Corporation) during the course of its agile transformation.
The main agile model used can be defined as scaled-up Scrum [8]
as defined in [9]. In total, the survey data include more than
1000 responses regarding different aspects of agile development
from the viewpoint of various organizational stakeholders. The
goal of this paper is to provide empirical evidence as to whether
a linkage can be found between a person’s attitude toward agility
and the following two factors: (1) experience (time) in applying
agile methods in practice and (2) previous experience (time) in
applying traditional (non-agile) methods. In addition, this study
provides empirical evidence regarding the areas perceived as being
the main challenges and benefits of agile methods. To provide
broader perspective, this was done by comparing both the most
receptive and most resistant segments of the population.

Organizations that are still contemplating agile transformation
will find the presented results useful from at least two perspec-
tives. First, we study how practitioner’s experience influences
their opinions towards agile adoption. Secondly, we identify the
most challenging areas that are likely to be faced when attempt-
ing to transform a large organization toward higher degrees of
agility.

From the industrial viewpoint, these results indicate that once
the respondents have experience on agile methods, these are per-
ceived to provide added value. This claim is supported by the find-
ing that those with lengthy experience using both agile and
traditional methods have positive attitudes towards agility. The ac-
tual experience in using agile methods seems to play a more
important role than the earlier prejudices. In the study, three espe-
cially interesting areas of agile development were identified: (1)
deployment of agile methods, (2) requirements management and
planning in a flexible and iterative manner and (3) visibility and
transparency. While deployment itself was perceived as the most
challenging task, the areas of requirements management and
transparency were perceived as among the most beneficial,
yet also among the most challenging, areas of agile development
throughout the survey population.

This paper is structured as follows: Section 2 contains back-
ground information. Section 3 discusses research aims, context,
and data collection and analysis methods. Section 4 contains re-
sults with representative statistics, research questions, qualitative
analysis, and limitations. Section 5 covers discussion of the main
results, and lessons learned.

2. Related literature

The fundamentals of agile methods, i.e., the values and princi-
ples behind them (http://agilemanifesto.org/), define the underly-
ing aspiration of agility. The values stated in the Agile Manifesto
highlight the values of communication and collaboration, respon-
siveness, and focusing on the implementation of working software.
In agile principles, several more detailed ambitions for agile meth-
ods have been set. Based on this information, a software organiza-
tion could be interested, for example, in: increased return on
investment (ROI) for their efforts in the deployment of agile meth-
ods; increased ability to deliver software early and continuously;
better

responsiveness to change

s in product content; closer collab-
oration and communication within the organization and with the
customer; higher motivation and increased self-organization of

development teams; a sustainable and steady working pace; con-
tinuously increasing work productivity; and higher quality of
end-products.

Schwaber et al. [1], among many others, argue that agile meth-
ods yield several benefits. These include reduced time-to-market,
increased quality, reduced waste, better predictability, and better
morale. Among other suggested benefits are, for instance, an in-
creased ability to respond to dynamic market changes [10]. It
should be noted that different agile methods aim to tackle different
aspects of software development. For example, XP [7] focuses on
the implementation of software, Test Driven Development [11] fo-
cuses on unit testing, and Scrum [8] focuses on the management of
product planning and implementation. A large amount of predom-
inantly positive anecdotal evidence indicates benefits associated
with agile methods, including quality of the end-product, produc-
tivity, and increased transparency and collaboration. Most (76%) of
the publications seem to have focused solely on XP while commu-
nicating an urgent need to broaden the focus of studies to include
other agile methods [3]. Meanwhile, sound empirical evidence on
the actual benefits and challenges of agile methods remains rare
[3].

Many studies report that the adoption of individual agile prac-
tices, or certain fundamentals of agile software development, has
been undertaken to complement the organization’s existing pro-
cesses [12]. Often, the adoption of existing agile methods may re-
quire their radical modification to fit the context at hand [13].
Thus, it is no surprise that several case studies report challenges
in the adoption of agile software development methods [2]. It
has been reported that agile adopters are often not aware of what
agile adoption really means, and how broad a change is actually re-
quired [1]. It has been stated that just having shorter iterations
resulting in better quality may not be sufficient for large compa-
nies producing complex software; a more holistic view on agility
may be needed [14].

Various success and failure factors have been proposed to be
significant in the agile adoption phase ([5,15]). For example, it
has been suggested [15] that for various reasons, agile methods
seem to polarize people and stakeholder groups into opponents
and supporters having very different standpoints regarding the
usefulness of agile methods for the organization. To overcome this,
a fundamental change of philosophy and the development of new
behaviors are claimed to be required across the organization [15].
A recent survey study proposes that the appreciation of agile meth-
ods seems to increase once they have been adopted and applied in
practice (5), which indicates that while there is likely to be resis-
tance among the agile adopters in the organization, time and expe-
rience in applying the methods will have a positive effect on this
resistance.

It has been claimed that organizations that have failed in plan-
ning agile adoption in systematic manner will generate sporadic
results across the organization [15]. Kettunen and Laanti have
proposed a framework [14] for understanding the multidimen-
sional nature of agility that could be used as a basis for software
process improvement (SPI). Not all problems in software projects
can be solved by software process methodologies, but during agile
transformation an organization can be seen as a force field con-
sisting of forces for and against agility [14]. Kettunen and Laanti
have described this force field as Goals (what we try to achieve
with agility), Means (what we need to put in place in order to
be agile) and Enablers (what needs to be in place before agility
can happen). They also present the outline for an agility evalua-
tion grid containing several hundred of items collected from liter-
ature and categorized into Goals, Means, and Enablers. Due to the
lack of other suitable frameworks, this ‘‘Agility evaluation grid
outline’’ [14] was adopted as the basis for the qualitative analysis
in this study.

http://agilemanifesto.org/

278 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

3. Research approach

In this section, the research aims, context, data collection, and
analysis methods are presented.

3.1. Research aims

The original reason why the organization started to adopt agile
methods is scoped out from this study: i.e. the initial set-up is that
the organization in some way or other following agile methods and
we want to reflect the reality via the practitioners’ opinions, atti-
tudes, and experience on agile methods. It is of major importance
to study and understand the underlying factors affecting the satis-
faction of stakeholders towards agility in organizations adopting
agile methods, because the level of success in software process
improvement can be characterized in terms of personnel satisfac-
tion and whether the new process is actually used [16]. Currently,
agile methods are subject of much discussion yet limited empirical
evidence exists to support their usefulness [3]. Thus, it is important
to evaluate how sound agile methods really prove to be. The
assumption here is that if agile methods provide no real improve-
ment, adopters would initially be eager but would then turn pessi-
mistic after experimenting with agile methods in practice.
Alternatively, if following agile methods would prove to be very
difficult in practice, the followers might first be quite optimistic
and then become realistic, seeing it as something laborious but
necessary. The second assumption is that past experience with tra-
ditional methods might affect attitudes and opinions negatively.
Also, we wanted to know if the old saying, ‘‘You can not teach an
old dog a new trick,’’ applies to agile adoption.

In this study, we first studied how strongly a person’s back-
ground – i.e., the length of the person’s experience with agile and
traditional methods – affects the perceptions of and satisfaction
with, agile methods. From a broader perspective, this provides us
a way to find out whether agility is making a permanent entrance
or if those methods will fade away. Thus the first research aim of
this study is to determine respondents’ opinions and attitudes to-
ward agility and how they change as the person gains more and
more experience in using agile methods. This leads us to the fol-
lowing two research questions that guide the quantitative analysis
of the study:

(R1) How does the length of practitioners’ experience using
agile methods affect their opinions toward agile methods?
(R2) How does the length of practitioners’ experience in using
traditional methods affect their opinions toward agile methods?

The research method is one-way ANOVA (analysis of variance)
[17]. In the analysis of the opinion statements, we use the original
ordinal data with range 0.00–120.00 from the survey tool. As one-

Fig. 1. Respondents’ background in non-agile

way ANOVA assumes Homogeneity of Variance, Levene’s test was
used for homogeneity [17]. A post-hoc analysis is done with Tukey
HSD [17].

The second research aim of this study is to take a closer look at
qualitative findings to reveal respondents’ perceptions of new,
emerging challenges and benefits of agile development. In order
to provide the view on this topic, respondents are first categorized
as having negative, neutral, or positive attitude towards the adop-
tion of agile methods. Then, the goal is to detect any significant dif-
ferences in underlying perceptions between the two extreme
groups, i.e., the negative and positive groups of respondents.

3.2. Research context

The sample of the study contains more than 1000 respondents
from seven different countries in Europe, North America, and Asia.
90% of the respondents represented Research and Development
(R&D), while the rest represented mainly Marketing, Design, or
Process Development organizations. The total response rate was
33% from the population under the study. Fig. 1 shows the
respondents’ background in agile and non-agile development as
percentages.

The deployment method and main agile model applied among
the survey population was scaled-up Scrum, as defined in more de-
tail in [9]. Traditional methods in this case refer to waterfall-type
of product lines, as defined in [18]. Ninety percentage of respon-
dents were able to identify their agile role. Distribution of different
agile roles can be seen in Fig. 2.

3.3. Data collection and analysis

This research was conducted as a questionnaire-based survey.
Personal interviewing, telephone interviewing, and mail data col-
lection have earlier been proposed as suitable data collection
methods for such questionnaire-based surveys [19,20]. However,
in this case a computer assisted web-based survey was conducted,
because it is faster and easier to fill than a paper survey and also
the handling of the responses is of lesser work. We also had proper
tools to do a web-based survey at hand. Nardi [21] proposes using
web-based surveys to eliminate human errors caused by manual
data handling, and to enable a higher response rate while also
increasing speed and decreasing cost. A low response rate – rarely
over 30% – has been identified as one of the major problems of di-
rect mail data collection [19]. The questionnaire was conducted in
March 2008 and it was open for two weeks in April 2008.

The questionnaire consisted of multiple sections: (1) back-
ground (e.g. organizational unit), (2) agile role and experience,
(3) support, site distribution, and collaboration in agile develop-
ment, (4) current use and usefulness of agile methods, (5) effects
of agile methods, and (6) opinions. A questionnaire form was

and agile development as percentages.

Table 1
Use of sliding scale values in data analysis.

Respondent’s selection
on sliding scale (1–7)

Corresponding
integer #a

Corresponding selection range
values generated by the systemb

1.0–1.49 1 0.00–9.99
1.5–2.49 2 10.00–29.99
2.5–3.49 3 30.00–49.99
3.5–4.49 4 50.00–69.99
4.5–5.49 5 70.00–89.99
5.5–6.49 6 90.00–109.99
6.5–7.0 7 110.00–120.00

a Used in graphs.
b Used in ANOVA calculations.

Fig. 2. Respondents’ agile role as percentages (of those 90% who reported they had an agile role).

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 279

automatically generated based on the respondents’ answers to the
following questions: (1) respondent’s organizational unit, (2)
respondent’s role in agile development, and (3) respondent’s expe-
rience in applying agile methods. Thus, depending on how a
respondent replied to the previous questions the system generated
different sections to be answered. The following scenarios form the
case sensitive questionnaire:

– ALL respondents: Sections 1 and 6.
– Respondents from R&D, marketing, design, or process organiza-

tions: Section 2.
– Respondents with identifiable agile role and experience in agile

development: Section 3.
– Agile team members with experience: Section 4.
– Program and product owners: Section 5.

This study focuses on Sections 1, 2 and 6.
Among other questions, the background section focused on

respondents’ experience both in non-agile (i.e., traditional SE)
methods and agile methods. Collecting data about respondents’
experience with traditional methods provided the ability to evalu-
ate the linkage between earlier experience and attitude and opin-
ions regarding agile methods. The following scales were used to
collect the data regarding experience with traditional software
engineering methods:

1. <1 year. 2. 1–2 years. 3. 3–5 years. 4. 5–10 years. 5. 10 years.

Gathering information about respondents’ experience in apply-
ing agile methods, on the other hand, provided the ability to eval-
uate whether respondents’ opinions and attitude toward agile
methods could be considered to be the result of actual experiences
or rather to be mere assumptions. The agile experience level of the
respondent was determined using the following scale:

1. Not at all.
2. 1–3 months.
3. 4–6 months.
4. 7–11 months.
5. 1–2 years.
6. 2 years.

In the opinions section, nine statements were presented (Table
2; also Appendix A). A seven-step sliding scale (ranging from
1 = totally disagree to 7 = totally agree, the middle position being
‘‘neutral’’) was used to collect respondents’ opinions, see Appendix

A. Table 1 aims at clarifying how the two decimal precise selections
of responses on sliding scale were converted into integer (auto-
matic conversion by survey tool used) and range values (system
conversion). These values have then been used to illustrate the re-
sults of the analysis throughout the paper. The more precise value
range (0.00–120.00) have been used for ANOVA calculations and
illustrations while the integer range (1–7) has been used for other
graphs.

In addition, both open questions and multi-choice questions
were included in the questionnaire, providing possibilities for both
qualitative and quantitative data analysis. In the multi-choice
questions the satisfaction or opinion of the respondent, or the
experienced use and usefulness of specific agile methods and
whether one would like to go back to the old methods were asked.

Kitchenham [22] points out one limitation of surveys being
their ability to only confirm association instead of causality. In
addition, Miles and Huberman [23] have proposed linking qualita-
tive and quantitative data to provide richer detail and deeper
understanding for the research. Taking these issues into consider-
ation, the analysis includes both quantitative and qualitative data,
the latter collected through free-form text entry fields in the
survey.

In the qualitative analysis first all data was checked against the
categories, and these were cross-checked and the unclear cases
were discussed i.e. whether the comments were interpreted in
the right way and the true meaning of the comment was captured
or diluted in the process. Few of these cases led to changing the
category. Finally, we checked that the different interpretation of
the open comments would not have an impact on the final results
(i.e. if a change in the interpretation of a comment that had multi-
ple interpretations would have changed the result or the ordering
of the categories). When we concluded that different interpreta-
tions could not affect the order of the categories or the conclusions,
we accepted the results and that the process was correct and prob-
ability of any errors small enough.

The large number of responses allowed not only descriptive
data analysis (i.e., frequencies and cross tabulation) but also more

Table 2
Opinions statements and descriptive statistics.

Statements N Mean Variance Standard deviation

1. Agile development increases the effectiveness of development 751 4.97 1.652 1.285
2. Agile development increases the quality of the product 751 4.70 1.854 1.362
3. Agile development increases the transparency of development 751 5.13 1.917 1.385
4. Agile development increases collaboration 751 5.04 1.769 1.330
5. Agile development makes work more fun 751 4.61 2.259 1.503
6. Agile development makes work more organized/planned 751 4.50 2.341 1.530
7. Agile development increases the autonomy of development teams 751 4.86 1.901 1.379
8. Agile development enables the earlier detection of bugs/errors/defects 751 4.77 2.005 1.416
9. Agile development makes work less hectica 751 3.64 2.318 1.522

a The original question in the survey was phrased as ‘‘agile development makes work more hectic’’. In the analysis, the question and scales are reversed so that the
descriptives are better comparable with other questions.

280 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

sophisticated quantitative data analysis. Clustering used in this
study [24] provides an analysis method that seeks to build up a
number of clusters. It is a widely used method in biology, medicine,
and market research (especially segmenting). Cluster techniques
are used to create taxonomies, identify patterns of associations,
and distinguish sub-groups in samples. Clustering analysis empha-
sizes diversity rather than central tendency. In clustering analysis
the distance between data points is measured, and the data points
closer to each other are marked as belonging to the same cluster.
Cluster analysis is especially suitable for studying the differences
in a sample. Even though general linkages exist, cluster analysis
can be used to demonstrate patterns that do not conform to the
majority in the sample [24].

In this study, cluster analysis was used to discover small differ-
ences between the groups, primarily consisting of two different
attitude shifts. First there is analysis of the (negative) impact of
the length of experience with traditional methods on opinions to-
wards agility and, secondly, there is analysis of the (positive) im-
pact of actual experience with agility on the attitude and
opinions detected. Then we drill behind the differences by analyz-
ing different respondent groups using the attitude question ‘‘would
you go back’’. Finally, a cluster analysis of answers to the same atti-
tude question, ‘‘would you go back’’ was used to determine the
opinions of those people who belong to both of these groups, i.e.,
who have long experience with both non-agile and agile methods.

4. Results of the study

In this section, we present research data and analysis to evalu-
ate findings in general (Section 4.1), answer the research-questions
set for the study (Section 4.2), examine qualitative data for finding
underlying factors (Section 4.3), analyze quantitative data to see if
the agile methods will prevail (Section 4.4) and list limitations of
the study (Section 4.5).

The focus of this study is investigation of the impact of agile and
non-agile experience to opinions and toward agile methods. We
also analyzed other possible factors that could have impact on
respondents’ opinions (such as respondent’s agile role and
location) but decided to scope out all additional explaining factors
from this study in order to keep it into reasonable length.

4.1. Representative statistics

The perceptions of the respondents regarding agile develop-
ment were captured in the ‘‘opinions’’ section of the survey. Firstly,
nine specific statements (Table 2) were presented to evaluate the
respondents’ view on the impact of agile methods in the organiza-
tion. The statements were formulated based on the underlying fun-
damentals and claimed benefits of agile methods (discussed in
Section 2). Secondly, an attitude question was presented to evalu-

ate the respondents’ overall attitude toward the use of agility in the
organization. In the following sub-sections the analysis of the
respondents’ opinions (4.1.1) is presented.

It should be noted that even though the ‘‘opinions’’ section of
the questionnaire was available for the entire survey population,
not all respondents replied to all of the questions.

4.1.1. Opinions on the impact of agile development
In the Opinions section, nine statements were presented (Table

2). A sliding scale from 1 to 7 was used to collect the responses:
1 = totally disagree, 4 = neutral and 7 = totally agree. The data dis-
tribution to each of the questions was symmetric and normal.

In Table 2, descriptive statistics regarding the nine statements
are presented. As can be seen, responses to only one of the state-
ments were below neutral. In other words, on the average the
respondents thought that agile makes work more hectic. All other
statements received positive (i.e., above neutral) responses on the
average. The most positive effects of agile development, within the
limits of the above statements, were the ‘‘increase of transparency’’
and ‘‘increase of collaboration.’’ The rest of the statements also
gained results above neutral (4), i.e., closer to the value of ‘‘some-
what agree’’ (5).

4.1.2. Attitude towards the use and deployment of agile methods
In the Opinions section, the following attitude question was

asked:

1. Would you go back to the old way of working?

This attitude question had the following options: 1 = Yes, 2 = I
see no difference, 3 = No.

Fig. 3 shows the overall distribution of responses to the ques-
tion and how the responses were distributed between respondents
without agile experience (Binary agile experience = 0) and with
agile experience (Binary agile experience = 1). The overall attitude
towards agile methods could be considered as positive, as 60%
wanted to stay in an agile mode of working, while only 9% wanted
to go back to traditional working methods. However, as many as
31% of respondents did not see any difference or did not have a
clear opinion on this.

In the analysis the respondents were divided in three groups
based on the attitude question as follows: Yes = negative group, I
see no difference = neutral group, No = positive group. The large
portion of neutral answers (31%) is partially explained (by 12%,
see Fig. 3) by the large number of people not yet having experience
with agile methods (21% of all respondents).

4.2. Answering the research questions

The research-questions set for the study were:

Fig. 3. Overall distribution of responses to the question, ‘‘Would you go back to the old way of working?’’, categorized by agile experience.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 281

In this section, we present the analysis of the research data
against the above questions. Note that the values 1–7 have been
used only in the graphs and the calculations have been done with
the using the more accurate data values from 0.00 to 120.00 as de-
fined in Section 3.3 and Table 1.

4.2.1. Analysis of variance between agile experience and opinions on
agile development

To shed light on the first research question, the nine opinions
statements were further analyzed to find whether the length of
practical experience with agile methods has a positive impact on
respondents’ opinions.

ANOVA assumes the homogeneity of variances i.e. the variance
of data in groups should be the same. In order to check this
assumption we used Levene’s test which tests the null hypothesis
that population variances for groups are equal. The null hypothesis
can be rejected if the p-value is less than a certain threshold; in this
study we use a threshold of 0.05 (See Table 3).

For certain statements, ‘‘effectiveness of development’’, ‘‘quality of
product’’, ‘‘more fun’’, ‘‘more hectic’’, ‘‘more organized/planned’’ and
‘‘autonomy of development teams’’ the p-value is less than 0.05,
hence there is a significant difference in population variances
and we cannot use ANOVA in these cases. For other statements,

Table 4
Means of opinions per group/agile experience.

Means of opinions of groups Not at all 1–3 months

Transparency 57.0 66.1
Collaboration 19.4 19.2

Table 3
The homogeneity of variances of opinions/agile experience.

Levene statistic p-Value

Effectiveness of development 2.792 0.017
Quality of Product 4.137 0.001
Transparency 0.425 0.832
Collaboration 0.684 0.636
More fun 4.196 0.001
More hectic 4.882 0.000
More organized/planned 3.126 0.008
Autonomy of development teams 3.256 0.007
Earlier detection of errors 0.939 0.455

the p-value is greater than 0.05, hence we cannot reject the null
hypothesis. For these statements, we can say that the differences
in variances might have occurred by chance. ANOVA is used for
these statements only.

The ANOVA results (Appendix B) indicate a significant differ-
ence between the experience with agility and the opinions ‘‘Agile
increases the transparency of development’’ (F(5, 698) = 10.79,
p < 0.05) and ‘‘Agile development increases collaboration’’ (F(5, 694) = 4.425, p < 0.05). The respective means of opinions for these groups are presented in Table 4. As seen in Table 4, the means of opinions tend to become more positive as agile expertise grows.

The Tukey HSD was used for post hoc analysis. The opinions
transparency and collaboration were divided into two groups by
Tukey HSD. See Appendix B.

Fig. 4 presents a plot on agile experience means on the ‘‘Agile in-
creases the transparency of development’’ opinion (scale 0.00–
120.00).

As a result, a statistically significant positive shift was found in
two of the nine opinion statements, namely collaboration and
transparency.

4.2.2. Analysis of variance between non-agile experience and opinions
on agile development

Nine opinions statements were studied to reveal whether the
length of experience with non-agile methods (i.e., with traditional
software development) seems to have a negative impact on opin-
ions towards agile methods. Levene’s test for homogeneity reveals
that (Table 5) variances for opinions for effectiveness, collabora-
tion, more organized/planned work, autonomy, and earlier detec-
tion of bugs are likely to have occurred by chance. The analysis
of variance (Appendix C) indicates a significant difference between
the groups and the opinions on ‘‘Agile development increases the
effectiveness of development’’ (F(4, 683) = 6.791, p < 0.05), ‘‘Agile development increases collaboration’’ (F(4, 671) = 7.811, p < 0.05), ‘‘Agile makes work more organized/planned’’ (F(4, 670) = 7.707, p < 0.05) and ‘‘Agile development enables the earlier detection of er- rors’’ (F = 4, 670) = 7.510, p < 0.05).

Fig. 5 presents a plot on experience with traditional methods
and opinions means of ‘‘Agile development increases transparency’’,
(scale 0.00–120.00).

4–6 months 7–11 months 1–2 years >2 years

67.8 68.8 73.6 72.9
21.3 19.7 17.8 21.3

Table 5
Homogeneity of variances of opinions variables/non-agile experience.

Effectiveness of
development

Transparency Collaboration More organized/
planned

Autonomy of
development teams

Earlier detection of errors

Levene statistic 0.894 3.631 1.099 0.972 1.124 2.144
p-Value 0.467 0.006 0.356 0.422 0.344 0.074

Fig. 5. Means plot of transparency attitude per experience on traditional methods (x represents non-agile development background, y represents respondents’ opinion means
with 95% Confidence Interval with scale 0.00–120.00).

Fig. 4. Effect of agile experience on the opinion, ‘‘Agile development increases transparency’’ (x represents agile background, y represents respondents’ opinion means with 95%
Confidence Interval, with scale 0.00–120.00).

Table 6
Means of opinions grouped by non-agile experience.

Means of
opinions

<1 year 1–2 years 3–5 years 5–10 years >10 years

Effectiveness 70.4 69.0 64.1 60.6 61.0
Collaboration 17.1 18.2 19.7 18.9 21.9
More organized/

planned
66.3 62.7 59.1 53.3 54.4

Earlier detection
of errors

69.0 67.2 57.9 58.4 59.0

282 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

Table 6 represents the means of opinions within each group. We
can see the trend towards a more negative attitude as non-agile
experience increases, except for collaboration, where the trend is
the opposite.

The post hoc analysis with Tukey HSD (Appendix C) revealed
that opinions for effectiveness, collaboration, more organized/
planned, and earlier detection of errors could be divided into two
groups based on non-agile experience. People with more than 3–
5 years of experience in traditional development seemed to think
more negatively about agile.

Fig. 6. Means of all opinions questions classified by agile experience (0 = no agile experience, 1 = agile experience) (x represents opinions; y represents opinion means with
95% confidence interval with scale 0.00–120.00. Figure on left: respondents having no agile background only. Figure on right: respondents having agile background only.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 283

Based on the analysis above, it can be concluded that long expe-
rience with non-agile development negatively affects certain opin-
ions of agile development (i.e., increased effectiveness, work being
more organized/planned, and earlier detection of errors), and pos-
itively on the opinion of increased collaboration. Fig. 6 shows plots
of means for all opinions. It can be seen that in general when agile
experience grows, practitioners’’ opinions sharpen and get more
positive towards agility.

4.3. Perceived benefits and challenges of agile methods: qualitative
analysis

As the parametric research and clustering analysis do not pro-
vide a comprehensive explanation of opinions towards agility, we
decided to also analyze the collected qualitative data. The goal
was to find the perceived benefits and challenges reported by the
respondents and to see whether significant differences could be
detected among respondents with positive and negative attitudes
towards agile methods.

Two open questions were selected for further analysis: (1)
‘‘What do you consider the main benefits of agile methods in y

our

organization?’’ and (2) ‘‘What do you consider the main challenges
of agile methods in your organization?’’ Firstly, respondents were
grouped based on their answer to the attitude question, ‘‘Would
you like to go back to the old way of working’’. This question
was selected for grouping as it was considered to distinguish the
positive group (‘‘No’’ responses) and the negative group (‘‘Yes’’ re-
sponses) from the survey population. Neutral responses were not
included in the analysis. 16% of the negative group did not report
any challenges and 26% did not list any benefits of agile adoption.

20% of the positive group did not answer either of these free-form
open questions.

No suitable frameworks were found for classification of the
qualitative data of the study. Thus we based the classification to
our own earlier work and formed the initial groups from the Key
Areas presented in Table 7 on ‘‘agility evaluation grid outline’’
[14]. Then we went through the qualitative responses in several
increments to find a set of classes that would fit in grouping them
all. The following increments were gone through: (1) classification
of negative responses of the negative group, (2) classification of
positive responses of negative group, (3) classification of positive
findings of positive group, and (4) classification of negative find-
ings of positive group. During each increment, new findings were
added to the already identified classes, and if no suitable classes
existed, new classes were created or existing ones were modified.
Table 7 presents the distribution of findings in each key area for the
positive and negative groups of respondents.

Somewhat surprisingly, the single largest key area among the
negative group was that no benefits or challenges were identified
at all (key area 12) or that benefits were unidentified; as for exam-
ple, is the case with the following comment: ‘‘There is not any, if you
count out momentary increase of work effort which eventually
increases workers stress level!!!!!!!!!!!!!!!!!!’’ this comment is an
extreme example – most respondents just wrote that there are
no improvements. When leaving this particular key area out of
the scope, the following Top 3 challenges (Table 8) and benefits
(Table 9) could be identified among the two groups.

In the following, the key areas common for both of the control
groups – i.e., deployment of agile methods, requirements engineer-
ing, and visibility and transparency — are discussed in more detail.

Table 7
Agile benefits and challenges perceived by positive and negative respondents.

Key area Negative group Positive group

+Benefits (#61)
(%)

�Challenges (#42)
(%)

+Benefits (#347)
(%)

� Challenges (#349)
(%)

(1) Requirements management/planning/changing requirements/
flexibility

13 17 16 9

(2) (Product) integration – 5 – 1
(3) Resourcing/effort management – 14 – –
(4) Visibility/transparency 20 10 16 2
(5) Deployment/tailoring/process improvement 2 24 8 21
(6) Self-organization/cross-functional teams/teamwork 8 10 11 3
(7) Competence – 2 – 7
(8) Management 3 7 2 6
(9) Quality/testing/tools/continuous integration 8 7 7 7

(10) Team size/distribution – 2 – –
(11) Other 9 2 5 4
(12) No benefits or challenges or I do not know 22 – 3 –
(13) Frequent delivery/speed/responsiveness to change 11 – 6 2
(14) Productivity/focus/efficiency/predictability – – 13 1
(15) People/communication/motivation/collaboration 5 – 12 3
(16) Dependencies/co-operation/work distribution/subcontracting – – – 9
(17) Error fixing/maintenance/ad hoc work – – – 5

284 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

4.3.1. Deployment of agile methods
Clearly, the largest hindrance found by both the negative group

and positive group of respondents was the deployment of agile
methods itself (key area 5). Respondents in the positive group
saw problems with, for example, rolling the deployment out to
all teams simultaneously as is also the case in this typical case:
‘‘Rolling it out to people who are not familiar with it and ensuring that
it is followed correctly.’’ The proper usage (adaptation) and deploy-
ment of agile methods on a large scale was also much commented
upon. One of the negative group’s comments concerned, for in-
stance, defining the right balance and level of agility: ‘‘We must re-
sist the temptation to go too far with the agile idea at the cost of
predictable and consistent deliverables to our product programs’’
and ‘‘If one team does agile, then all teams must do (and respect) agile
in order for it to be effective.’’ These comments are also typical to the
negative group concerned with deployment, i.e. that we would go
too extreme with agility.

4.3.2. Requirements management / iterative planning
The second largest group of challenges was the same in both of

the compared groups: namely, managing and planning require-

Table 8
Top three perceived challenges of agile development.

Top three perceived challenges

Positive group Negative group

1. Deployment of agile methods 1. Deployment of agile methods
2. Requirements management/iterative

planning
2. Requirements management/

iterative planning
3. Dependencies, co-operation, work

distribution, subcontracting
3. Resourcing/effort
management

Table 9
Top three perceived benefits of agile development. (Note that visibility/transparency
and requirements management/iterative planning got equal number of votes.).

Top three perceived benefits

Positive group Negative group

1. Visibility and transparency 1. Visibility and transparency
2. Requirements management/

iterative planning
2. Requirements management/

iterative planning
3. Productivity/focus/efficiency/

predictability
3. Frequent delivery/speed/

responsiveness to change

ments in a more flexible and iterative manner (key area 1). The
negative group interpreted agility as missing plans and planning
as is also the case in this typical comment: ‘‘Software is now created
as fast as possible without any planning or design. Major decisions are
made off the cuff in the hallway, based on whatever the project man-
ager happens to think at the moment.’’ The positive group saw prod-
uct management, frequent changes, and the new way of managing
requirements in prioritized backlogs as a challenge: ‘‘Not clear how
to get new tasks prioritized on backlog(s). Planning is always hard, be-
cause of surprises.’’ The comment is typical to the group, represent-
ing a true concern on how we get the deployment done correctly.

However, planning was considered beneficial by both groups
too. The negative group found beneficial the flexibility to take in
new requirements any time. The attitude behind most comments
is visible also in this example: ‘‘The only idea around agile develop-
ment that I see as positive is the idea of flexibility. Beyond that, I think
the agile concepts do not mix well with the need to provide predictable
and consistent delivery dates and content information to device.’’ The
positive group saw agile planning and priorities as increasing pre-
dictability – and typically as simply better way to do planning, as is
case also in this comment: ‘‘Product plans are more realistic and thus
is easier to plan other things related to products’’.

4.3.3. Visibility and transparency
Visibility and transparency (key area 4) was found as another

top benefit of agile development by both groups of respondents.
Most respondents from both groups gave no specific explanations
as to why visibility and transparency was perceived as a benefit.
Two respondents from the negative group, however, named daily
scrum meetings as the particular reason for visibility/transpar-
ency: ‘‘Visibility into work and reduced slack in developers time due
to daily scrums’’. From the positive group, a few respondents gave
fast error handling and cross-team communication as explanations
as in this comment: ‘‘Visibility inside the team: everybody in the team
knows the priorities and why we do what we do’’. Some respondents
from the positive group identified increased visibility especially
on progress and problems.

Visibility was listed as one of the major problems in agility by
the negative group. This comment is clearly reflecting the typical
concern that we will lose visibility with agility: ‘‘To get the big pic-
ture what is happening at program level’’. Here, visibility and trans-
parency were defined as information sharing, and (traditional)
program/project tracking.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 285

4.4. Cluster analysis of attitude question ‘‘Would you like to go back to
the old way of working?’’

We wanted to dive deeper to the question whether agile meth-
ods are here to stay or not. Assuming opinions and attitudes would
really give an answer, we selected the attitude question ‘‘would you
like to go back’’ as a basis of further analysis. Obviously, the same
person may have long experience with both non-agile and agile
development. These may be typical opinion leaders, so it might
be valuable to know what especially this group thinks. For this
analysis, clustering analysis was used (Table 10).

In this study, the K-means cluster analysis was used due to the
large number of data points. The K-means clustering analysis uses
simple Euclidean distance analysis to compute the clusters and re-
quires the user to specify the number of clusters. Hierarchical anal-
ysis with the first 100 respondents gave 2 cluster centers only,
leaving all those who would like to go back to traditional methods
as outliers. That simply did not fit our purpose of finding out how
people with both extensive agile and extensive non-agile experi-
ence think. Thus, several sets of numbers for clustering were tried
out. As stated by Rapkin and Luke [24], group differences become
evident only after a sufficient number of clusters is identified. Rap-
king and Luke also suggest the generation of two or more clustering
solutions to determine whether cluster structures in a sample gen-
eralize across specific measures of interest, and they claim that the
proper number of clusters can be determined empirically. As we
were interested to find out who would be those who would like
most to go back to the old methods, several cluster numbers were
tried out until it was noticed that eight cluster centers were needed
to find a cluster with a positive answer to this question. Also, large
numbers of clusters (like 16 and 32) were tried out, but they pro-
vided no additional information compared to the eight-cluster case.

Groups 5, 6 and 7 belong to the group of people we are inter-
ested in, having lengthy experience with both agile and non-agile
development. Groups 5 and 6 answered that they would like to

Table 10
Quick cluster results for experiences and agile favoring.

Cluster

1 2 3

Final cluster centers
How much experience do

you have using agile in
practice?

Not at all 1–2 years 7–11 months

How much experience do
you have using non-agile
development?

1–2 years 1–2 years 1–2 years

Would you go back to the
old way of working?

I see no difference No I see no difference

Number of cases per cluster 62 84 20

Fig. 7. Analysis result presented as a bubble graph. Note! The bubbles represent found clu
cluster.

stay agile, while group 7 would like to go back to the traditional
way of working. When comparing the sizes of these groups, we
found that the size of group 7 was roughly 1/3 the size of group
5 – i.e., more people having lengthy experience with both agile
and non-agile development methods favor agile than have a nega-
tive attitude toward it. See Fig. 7 for a graphical representation of
cluster analysis and this result. Could it simply be that people with
an extensive development background do not easily ‘‘buy’’ any
new development method, but once they have experienced agile
development, the majority of them will have a positive attitude?

Fig. 8 illustrates the number of people with extensive experi-
ence in both agile and non-agile methods if they would like to stay
in the agile mode of working.

Because the sub-group of those who would like to go back is
quite small, small variations in the background may not be visible
in the cluster analysis. Thus, a cross tabulation was done of all the
answers of those who would like to go back to old methods, in
order to verify the results from the clustering analysis. The cross
tabulation is presented in Table 11.

In Table 11 it can be seen that the backgrounds of the respon-
dents are quite evenly distributed. Thus there need to be other
explaining factors why these people would like to go back to old
methods than the respondent’s background in agile and traditional
methods.

To conclude, clustering analysis revealed that majority of the
respondents with lengthy experience in both agile and non-agile
(traditional) development have a positive attitude towards agility.
It is highly plausible that original negative attitudes towards the
new working mode are replaced by actual positive experience,
once the subjects have been exposed to agile development meth-
ods. It could be stated that, based on the positivity of the study
group and the positive shift in attitudes based on their experience
and willingness to stick to an agile way of working, the future
usage of agile methods looks promising. From a practical view-
point, this is an important finding, as it is known that persons

4 5 6 7 8

Not at all 7–11 months 1–2 years 7–11 months 4–6 months

5–10 years 5–10 years >10 years 5–10 years <1 year

I see no difference No No Yes No

137 143 84 59 70

ster centers, and their size is relative to number of respondent’s belonging that that

Table 11
Cross tabulation of experience (agile/non-agile) when respondent would not like to use agile methods.

How much experience do you have with non-agile development?

<1 year 1–2 years 3–5 years 5–10 years >10 years Total

Experience from agile in practice � experience from non-agile development (Cross tabulationa)
How much experience do you have with agile in practice? Not at all 1 0 1 7 3 12

1–3 months 2 3 0 1 1 7
4–6 months 2 1 2 6 2 13
7–11 months 1 2 9 6 1 19
1–2 years 2 1 1 1 1 6
>2 years 0 0 1 1 0 2
Total 8 7 14 22 8 59

a Would you go back to the old way of working? = Yes.

Fig. 8. Cluster Responses to ‘‘Would you like to go back,’’ paneled by non-agile expertise, and separated by agile experience. This graph is showing only answers from
respondents having long expertise in both in agile and non-agile development.

286 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

having lengthy experience are more likely to have recognized and
unrecognized leading positions.

4.5. Limitations

This study is scoped only into respondent’s opinions and their
agile and non-agile background. The reported benefits and prob-
lems with agile methods are scoped out from this study. Also we
do not know how much the success in agile deployment at the pro-
gram and team level actually impacted the results – this would
have required a comparative study over a lengthy time period.

The survey was done in one organization only, so it might be
biased by the organizational culture and the specific interpretation
of agility; so there remains a question for external validity [25], i.e.
how largely these results can be generalized. It could well be e.g.
that the resistance to agile methods has some relation to the speed
of the transition, and the deployment method itself. This can only
be verified by making similar studies in other organizations.

External reliability deals with the issue of whether or not inde-
pendent researchers can replicate studies in the same or similar
settings [26]. There is a question how much the results were af-
fected by the response rate – i.e. would it be possible that only
those respondents who are more satisfied with agile methods an-
swered the survey. Some smaller studies show similar results
[27,28], so this is likely out of the question.

The impact of applicability of agile model to respondents’ role is
also scoped out from the study, even though agile impact to some
roles have definitely been bigger than to some other roles.

The classification decisions made in qualitative analysis were
double-checked for validation and in cases of possible multiple
interpretations discussed thoroughly. This adds to the reliability
of the study [25] – the results should not be biased by the interpre-
tation of the individual researcher. We also double-checked that
ranks of the classes were correct and not affected the final inter-
pretation made – i.e. if the classes were chosen differently in the
questionable cases the order would still have remained the same
(in few cases the interpretation of the comment impacted the class
or classes chosen).

Construct validity [25] is a concern of obtaining the right mea-
sures for the concept being studied. We did a separate pilot survey
with a team of roughly one hundred respondents before the survey
was sent to a wider distribution.

5. Conclusions

The motivation of this study was to provide empirical evidence
of the perceived usefulness of agile methods in practice, especially
from the viewpoint of agile transformation in a large-scale indus-
trial setting. Firstly, the focus set was to study whether agile meth-
ods will provide any real improvement from the viewpoint of
adopters. Thus, in this study it is firstly studied how a person’s
background – i.e., the length of a person’s experience with agile
and traditional methods – correlates with satisfaction and percep-
tions of agile methods. From a broader perspective, this provides us
with a way to find out whether agility will make a permanent en-
trance or if it shall just fade away. As a result, it can be claimed that

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 287

the longer experience with agile methods in practice positively af-
fects opinions regarding their usefulness. This finding is in line
with earlier reported results [29].

As an answer to the first research question ‘‘How does the length
of practitioners’ experience using agile methods affect their opinions’’
we can conclude, that although the opinions were all very positive
towards the agile methods we could see that in general the opin-
ions changed more positive as the experience grew and that this
change was large enough regarding transparency and collaboration
to be statistically significant. Likewise we could answer to the sec-
ond research question ‘‘How does the length of practitioners’ experi-
ence in using traditional methods in affect their opinions’’ that if you
had more than 3 years experience on traditional methods it af-
fected somewhat negatively attitude towards agile methods, and
that this difference was significant with opinions on effectiveness,
increased collaboration, work being more organized/planned and
agile enabling the earlier detection of errors.

Although a linkage was found between longer experience with
traditional methods and a negative attitude towards agile meth-
ods, further investigation of ‘‘would you like to go back’’ attitude
with clustering analysis revealed that actual experience with agile
methods overrides any possible prejudices. Overall, actual experi-
ence with agile methods seems to play a more important role than
does earlier experience with traditional methods. From the indus-
trial viewpoint, these results indicate that once the respondents
have experience on agile methods, these are perceived to provide
added value.

The second research goal of this study was to contribute to the
scientific and industrial body of knowledge on the perceived chal-
lenges and benefits of agile development. As a result, three espe-
cially challenging areas of agile development were identified: (1)
deployment of agile methods, (2) requirements management and
planning in a flexible and iterative manner, and (3) visibility and
transparency. Thus, our qualitative results confirm earlier proposi-
tions [15] that agile deployment and adaptation of agile methods
to fit the organization are seen as a challenge. In this study, this
was the biggest challenge reported by several survey respondents.
Interestingly, although the deployment of agile methods was con-
sidered the largest challenge, the perceived usefulness of agile
methods still grew with agile experience among the survey
respondents. The areas of requirements management and visibility
and transparency were perceived as among the most beneficial,
yet also among the most challenging, areas of agile development
among the survey respondents. Thus, it could be claimed that the
above mentioned three areas in particular should be carefully con-
templated by any organization planning, or in transition towards,
agile methods.

As in some other reported cases [14], it could be identified that
respondents fall into groups for and against agility. In this study
60% of respondents could be considered advocates for agile meth-
ods (would like to stay in an agile mode of working) and 9% as
opponents (would like to go back to a traditional way of working).
The rest of the respondents (31%) were neutral. Among the small
opponent group, a negative shift was found with non-agile experi-
ence in practice. This, again, indicates that agile methods prove to
be useful in practice, once carefully adopted and adapted.

Agile transformation is sometimes described as a development
culture change and the organizational culture has been mentioned
as one explanation to the encountered difficulties with this trans-
formation [30]. An organizational culture however is never fixed.
In this study we can see how the viewpoints of agile advocates
and opponents are different, even when the conclusion is the
same; e.g. that the agile deployment is a challenge. Some opinions
like effectiveness and quality might have been widely discussed in
the organization and this could be the reason why there was no
significant difference between the opinions of more and less expe-

rienced respondents. Other opinions such as transparency and col-
laboration (were the significant difference was found) may be less
discussed and thus less understood – or simply not yet practiced
widely and properly enough. This study differs from the preceding
studies (such as Dybå and Dingsøyr [31] or Petersen and Wohlin
[32]) because of the large number of respondents going through
an agile deployment transition at the time when the survey was
conducted.

This study supports the findings of Dybå and Dingsøyr [31]
about the benefits of (customer) collaboration and increased quality.
Some benefits listed by Dybå and Dingsøyr [31] like work processes
for handling effects, learning in pair programming, thinking ahead for
management, focusing on current work for engineers and estimation,
and increased standardization of the job were not questioned in our
survey. Dybå and Dingsøyr [31] list the lack of attention to design
and architectural issues as limitations of agile methods. These did
not emerge as agile transformation challenges in this qualitative
study. There may be various reasons to this: firstly, it could be that
mostly architects are concerned about these subjects and the per-
centage of architects among the respondents is small. Or it may be
that the model deployed takes architectural concerns better into
account. Or it may be that the other concerns at this state of
deployment were simply bigger.

Petersen and Wohlin [32] mention better knowledge transfer due
to better communication and frequent feedback as well as early feed-
back via discussions with customers as the main benefits of agile
methods. These benefits have been categorized under visibility
and transparency in our qualitative study. Petersen and Wohlin
[32] also mention lacking focus to architectural design but then
further list agile development not scaling well and communication
between teams as the biggest challenges in incremental and agile
development. The concern about scaling agility came up in the
positive group of the qualitative part of this study as well: we
categorized this under the deployment of agile methods. Com-
munication between teams was seen as a third largest problem
by the positive group but also in a wider context including organi-
zational challenges, work distribution, subcontracting, and co-
operation in general.

The survey was done in one company, where agile deployment
has been executed in a certain way, so the results may not be gen-
erally applicable elsewhere. It would be interesting to evaluate
how much the organizational culture and the deployment ap-
proach, for instance, affect the results. In addition, it could be ana-
lyzed whether other factors such as location, role, or personality of
respondents affect their opinions. For instance, one typical
assumption is that introverted people who like working alone op-
pose agile methods, which emphasize co-operation and teamwork.
However, this cannot be verified in our study, as post-interviewing
is not an option due to the anonymity of the study. It would be also
interesting to study whether the Technology Adoption Lifecycle
[33] affects people’s ability to adopt agile practices – i.e., can the
respondents wanting to go back to traditional methods be charac-
terized as ‘‘Skeptics’’?

At the time of the survey, a large proportion of the population
had no experience with agile development. It would be interesting
to repeat the survey after some time to see how their opinions
have developed as their experience with and understanding of
agile methods have increased. It is also unclear how much the po-
sitive attitude towards agility has affected people’s decision to
start using agile methods – i.e., what came first, the positive atti-
tude or the experience? The positive attitude detected among peo-
ple who have used the agile method longest can partially be
explained by the fact that people who think positively about agile
are the first ones to start using it. However, the survey shows con-
stant increasing satisfaction towards agile methods over the time
that people use those methods. Would this be the case, if only peo-

288 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

ple having a positive attitude in the first place would start to use
these methods? Typically, individuals cannot make the decision
to use agile methods themselves; the whole team needs to start
using these methods simultaneously. This means that even though
people are forced to follow company policy in using agile methods,
they still develop a more positive attitude towards agile methods
after a certain amount of exposure to them.

This survey revealed some new aspects of agility and experi-
ence and it also raises some questions for future research. It would
be extremely interesting and useful, e.g. to study what level of rela-
tion exists with programs or teams that start and succeed in the
agile deployment i.e. does the actual level success of agile deploy-
ment (e.g. measured as number of agile practices followed or the
personal satisfaction how in that team works or the results they
get) have any relation to how the team’s or project member’s atti-
tudes develop. This type of study would require a close follow-up
of team or project attitudes over a longer period of time, and pref-
erably from the beginning of the agile deployment.

This survey changed minds across Nokia how agile methods are
seen. Before the survey we spoke about agile adoption – after the
survey we started to speak with more active tone about agile
deployment – the idea being that once people get first-hand expe-
rience with agile methods they start to understand these better.
The fact that agile methods received such a positive feedback
was a surprise to many. For anyone imposing loud criticism to-
wards agile methods his or her actual experience with agile meth-
ods is then discussed.

1. Agile development increases the effectiveness of development (Li
2. Agile development increases the quality of the product (Li

ne)

3. Agile development increases the transparency of development (Li
4. Agile development increases collaboration (Line)
5. Agile development makes work more fun (Line)
6. Agile development makes work more hectic (Line)
7. Agile development makes work more organized/planned (Line)
8. Agile development increases the autonomy of development team
9. Agile development enables the earlier detection of bugs/errors/de
10. How important you consider agile and iterative development in

(Multi Choice Qu

estion)

Choices:

1. Very important
2. Important
3. Neutral
4. Not very important
5. Not important at all

11. How satisfied you are with agile and iterative development with
work? (Multi Choice Question)
Choices:

1. Very satisfied
2. Satisfied
3. Neither satisfied nor dissatisfied
4. Dissatisfied
5. Very dissatisfied

12. Would you go back to the old way of working? (Multi Choice Qu
Choices:

1. Yes
2. I see no difference
3. No

13. What do you consider as the main benefits of agile methods in y
organization? (Fre

e Answer)

14. What do you consider as the main challenges of agile methods i
organization? (Free Answer)

15. If there was one thing you could change, what would it be? (Fre

It would be interesting to conduct another study later to see if
the perceived challenges and perceived benefits will change over
time. The organization may be able to overcome some of the
deployment challenges and the understanding of the benefits
may depend on how many agile practices were actually deployed.
This would be an interesting hypothesis to test in the future. Also
the relation between agile methods and the organizational culture
is widely still not understood or studied properly.

Acknowledgements

Maarit Laanti would like to thank Dr. Judy Sheard from the
Monash University Australia, and Professor Lauri Malmi for the
Quantitative Methods course at Helsinki University of Technology.
Without that course this paper would not exist. We would also like
to thank Danielle Arvanitis for proof reading the manuscript and
Abhishek Tripathi, from University of Helsinki for his valuable
comments related to the statistical methods used.

Appendix A. Opinions questions in the survey

A.1. Opinions on agile development

Please, respond based on your current experiences on agile
development.

ne)

Answering Areas:

Line
ne)

s (Line)
fects (Line)
the future?

in your own

estion)
our

n your

e Answer)

Appendix B. ANOVA and Tukey HSD results for agile experience and opinions

ANOVA

Sum of squares df Mean square F p-Value

Agile development increases the transparency of development (X)
Between groups 21288.962 5 4257.792 10.792 .000
Within groups 275383.476 698 394.532
Total 296672.438 703

Agile development increases collaboration (X)

Between groups 8580.965 5 1716.193 4.425 .001
Within groups 269181.828 694 387.870
Total 277762.793 699

Agile development enables the earlier detection of bugs/errors/defects (X)
Between groups 5069.697 5 1013.939 2.303 .043
Within groups 304178.937 691 440.201
Total 309248.634 696

Agile development increases collaboration (X) Agile development increases the transparency (X)
Tukey HSDa Tukey HSDb

Length of agile experience N Subset for
alpha = 0.05

1 2 1 2

Not at all 153 59.0576 Not at all 153 56.9689
1–3 months 90 64.2907 64.2907 1–3 months 91 66.1159
7–11 months 196 65.1451 65.1451 4–6 months 137 67.8189
4–6 months 135 65.6467 65.6467 7–11 months 197 68.8189
1–2 years 29 66.4721 66.4721 >2 years 29 72.9183
>2 years 97 70.6624 1–2 years 97 73.5965
Sig. .164 .317 Sig. 1.000 .162

Means for groups in homogeneous subsets are displayed.

a Uses Harmonic Mean Sample Size = 80.055.
b Uses Harmonic Mean Sample Size = 80.329.

Appendix C. Tukey HSD for opinions grouped by non-agile expertise, homogeneous subsets

ANOVA
Sum of squares df Mean square F p-Value

Agile development increases the effectiveness of development (X)
Between groups 9925.163 4 2481.291 6.791 .000
Within groups 249536.583 683 365.354
Total 259461.747 687

Agile development increases collaboration (X)
Between groups 11701.943 4 2925.486 7.811 .000
Within groups 251306.902 671 374.526
Total 263008.846 675

Agile development makes work more organized/planned (X)
Between groups 15012.716 4 3753.179 7.707 .000
Within groups 326262.434 670 486.959
Total 341275.150 674

Agile development increases the autonomy of development teams (X)
Between groups 2687.577 4 671.894 1.577 .179
Within groups 283247.177 665 425.936
Total 285934.754 669

Agile development enables the earlier detection of bugs/errors/defects (X)
Between groups 12736.691 4 3184.173 7.510 .000
Within groups 284077.272 670 423.996
Total 296813.963 674

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 289

Appendix C (continued)

Agile development increases the effectiveness
of development (X)

Agile development increases collaboration (X)

Tukey HSD Tukey HSD

Non-agile experience N Subset for alpha = 0.05 Non-agile experience N Subset for alpha = 0.05

1 2 1 2

5–10 years 230 60.6142 >10 years 127 59.9668
>10 years 129 60.9547 5–10 years 225 61.7078
3–5 years 143 64.1315 64.1315 3–5 years 141 65.9684 65.9684
1–2 years 93 68.9910 1–2 years 92 68.9129
<1 year 93 70.3806 <1 year 91 72.1008 Sig. .599 .078 Sig. .113 .100

Means for groups in homogeneous subsets are displayed.

Agile development makes work more organized/planned (X) Agile development enables the earlier
detection of bugs/errors/defects (X)

Tukey HSD Tukey HSD
Non-agile experience N Subset for alpha = 0.05 Non-agile experience N Subset for alpha = 0.05
1 2 1 2

5–10 years 222 53.2573 >10 years 143 57.9318
>10 years 126 54.4899 5–10 years 221 58.4302
3–5 years 143 59.1130 59.1130 3–5 years 127 59.0463
1–2 years 92 62.6925 1–2 years 92 67.2321
<1 year 92 66.2714 <1 year 92 68.9928 Sig. .235 .086 Sig. .993 .963

Means for groups in homogeneous subsets are displayed.

290 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

References

[1] K. Schwaber, G. Laganza, D. D’Silva, The Truth about Agile Processes: Frank
Answers to Frequently Asked Questions, Forrester Report, 2007.

[2] H. Svensson, M. Höst, Introducing an agile process in a software maintenance
and evolution organization, in: Proceedings of 9th European Conference of
Maintenance and Reengineering, 2005.

[3] T. Dybå, T. Dingsøyr, Empirical studies of agile software development: a
systematic review, Information and Software Technology 50 (2008) 833–859.

[4] T. Grant, P. Burris, Z. Reiss-Davis, Inquiry Insights: Agile Development,
Forrester Report, 2008.

[5] T. Chow, D. Cao, A survey study of critical success factors in agile software
projects, Journal of Systems and Software 81 (2008) 961–971.

[6] A. Sillitti, M. Ceschi, B. Russo, G. Succi, Managing uncertainty in requirements:
a survey in documentation-driven and agile companies, in: Proceedings of
11th IEEE International Symposium on Software Metrics, 2005.

[7] K. Beck, Extreme Programming Explained: Embrace Change, Addison Wesley
Longman, Inc., 2000.

[8] K. Schwaber, M. Beedle, Agile Software Development with Scrum, Prentice-
Hall, Inc., 2002.

[9] M. Laanti, Implementing program model with agile principles in a large
software development organization, in: Proceedings of 32nd Annual IEEE
International Conference on Computer Software and Applications (Compsac
‘08), 2008.

[10] M. Lycett, R. Macredie, C. Patel, R. Paul, Migrating agile methods to
standardized development practice, Computer 36 (2003) 79–85.

[11] K. Beck, Test-Driven Development by Example, Addison-Wesley, 2003.
[12] P. Manhart, K. Schneider, Breaking the ice for agile development of embedded

software: an industry experience report, in: Proceedings of the 26th
International Conference on Software Engineering (ICSE 2004), 2004.

[13] J. Grenning, Launching XP at a process-intensive company, IEEE Software 22
(1) (2001) 3–9.

[14] P. Kettunen, M. Laanti, Combining agile software projects and large-scale
organizational agility, Software Process: Improvement and Practice 13 (2008)
183–193.

[15] D. Norton, Five reasons organizations fail to adopt agile methods, Gartner 9
(2008).

[16] P. Abrahamsson, Measuring the success of software process improvement: the
dimensions, in: Proceedings of European Software Process Improvement
Conference (EuroSPI 2000), 2000.

[17] A. Field, Discovering Statistics using SPSS, third ed., Sage Publications Ltd.,
2009. ISBN: 978-1-84787-906-6.

[18] F. van der Linden, J. Bosch, E. Kamsties, K. Känsälä, H. Obbink, Software Product
Family Evaluation, Lecture Notes in Computer Science, Springer Link, 2004.

[19] P. Alreck, R. Settle, The Survey Research Handbook: Guidelines and Strategies
for Conducting a Survey, IRWIN Professional Publishing, 1995.

[20] A. Oppenheim, Questionnaire Design, Interviewing and Attitude Measurement,
Continuum International Publishing Group, 1992.

[21] P. Nardi, Doing Survey Research: a Guide to Quantitative Methods, Pearson
Education, Inc., 2003.

[22] B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El-Emam, J.
Rosenberg, Preliminary Guidelines for Empirical Research in Software
Engineering, National Research, 2001.

[23] M. Miles, A. Huberman, Qualitative Data Analysis; An Expanded Sourcebook,
second ed., SAGE Publications, 1994.

[24] B. Rapkin, D. Luke, Cluster analysis in community research: epistemology and
practise, American Journal of Community Psychology 21 (1993) 247–277.

[25] K. Petersen, C. Wohlin, D. Baca, The waterfall model in large-scale
development, 10th International Conference on Product-Focused Software
Process Improvement, 2009. ISBN: 978-3-642-02151-0.

[26] W. Wiersma, S. Jurs, Research Methods in Education, eighth ed., Pearson
Education Inc., 2005.

[27] B. Tessem, F. Mauer, Job Satisfaction and Motivation in a Large Agile Team.
Lecture Notes in Computer Science, 2007, vol. 4536/2007, 54-61, doi:10.1007/
978-3-5.40-73101-6_8.

[28] G. Melnik, F. Mauer, Comparative analysis of job satisfaction in agile and non-
agile software development teams, Lecture Notes in Computer Science, 2006,
vol. 4044/2006, 32-42, doi:10.1007/11774129_4.

[29] O. Salo, P. Abrahamsson, Agile methods in European embedded software
development organizations: a survey on the actual use and usefulness of
Extreme Programming and Scrum, IET Software 2 (2008) 58–64.

[30] T. Dingsøyr, T. Dybå, N.B. Moe, Agile Software Development: Current Research
and Future Directions, Springer, 2010.

[31] T. Dingsøyr, T. Dybå, Empirical studies of agile softw are development: a
systematic review, Information and Software Technology (2008).

[32] K. Petersen, C. Wohlin, A comparison of issues and advantages in agile and
incremental development between state of art and an industrial case, Journal
of Systems and Software (2009).

[33] J.M. Bohlen, G.M. Beal, The Diffusion Process, Agriculture Extension Service,
Iowa State College, 1957.

Agile methods rapidly replacing traditional methods at Nokia: A survey of opinions on agile transformation

Introduction
Related literature
Research approach
Research aims
Research context
Data collection and analysis
Results of the study
Representative statistics
Opinions on the impact of agile development
Attitude towards the use and deployment of agile methods
Answering the research questions
Analysis of variance between agile experience and opinions on agile development
Analysis of variance between non-agile experience and opinions on agile development
Perceived benefits and challenges of agile methods: qualitative analysis
Deployment of agile methods
Requirements management / iterative planning
Visibility and transparency
Cluster analysis of attitude question “Would you like to go back to the old way of working?”
Limitations
Conclusions
Acknowledgements
Opinions questions in the survey
Opinions on agile development
Tukey HSD for opinions grouped by non-agile expertise, homogeneous subsets
References
Tukey HSD for opinions grouped by non-agile expertise, homogeneous subsets

SOFTWARE PROCESS IMPROVEMENT AND PRACTICE
Softw. Process Improve. Pract. 2009; 14: 85 – 105

Published online 12 June 2008 in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/spip.387

Rationale Modeling for
Software Process Evolution

Research Section

Alexis Ocampo* ,† and Jürgen Münch
Fraunhofer Institute for Experimental Software Engineering
Fraunhofer-Platz 1, 67663 Kaiserslautern, Germany

Evolving a software process without a retrospective on its evolution and, in consequence,
without an appropriate understanding, can lead to important problems for a software
development organization. Two examples of such problems are inefficient performance as a
consequence of the arbitrary introduction of changes or difficulty in demonstrating compliance
to a given standard. Capturing information on the rationale underlying process changes
provides a means for better understanding process evolution. This article presents two studies
aimed at understanding and identifying information needs for describing the rationale for
process evolution. Additionally, it presents an approach for incrementally evolving processes
supported by this rationale. This approach is derived from the results of the studies and a
survey of related work. An application of the approach during the evolution of a reference
process for developing service-oriented applications is presented together with experience and
open questions for future research work.

KEY WORDS: sofware process evolution; rationale; software process models; process changes; resource description framework; process
management

1. INTRODUCTION

Software process models support software engi-
neers in systematically performing the engineering
processes needed to develop and maintain software
products. As these models are enacted, suggestions
for process adjustment or refinement arise, which,
in turn, demands evolution of the models. Usu-
ally, certain events such as the introduction of a
new software development technology (e.g. new
testing support tools or platform), new/updated
standards/guidelines for software development or
process engineering, or new/updated best prac-
tices emerging from community experience gener-
ate situations that must be resolved, i.e. issues, by

∗ Correspondence to: Alexis Ocampo, Fraunhofer Institute for
Experimental Software Engineering Fraunhofer-Platz 1, 67663
Kaiserslautern, Germany
† E-mail: ocampo@iese.fhg.de

changing the engineering processes and the process
models underlying them. Changing these models
in organizations is typically a complex and expen-
sive task (Nejmeh and Riddle 2006). In many cases,
because of budget and time constraints, arbitrary
decisions are made, and process models are evolved
without storing or keeping track of the justification
behind such changes (Armbrust et al. 2005). This, in
turn, frequently results in inconsistencies or ambi-
guity being introduced into the process models.

The justification for a decision has been defined
as rationale by researchers, who have done extensive
investigation on capturing, organizing, and analyz-
ing product design decisions (Dutoit et al. 2006).
Rationale modeling is considered to be critical in
two areas: knowledge capture and decision making.
By making rationale information explicit, decision
elements such as criteria, priorities, and arguments
can improve the quality of software development
decisions. Additionally, when a change in a prod-
uct’s functions or an addition, respectively removal,

Research Section A. Ocampo and J. Münch

is considered or implemented, rationale models
enable developers to track those decisions that
should be revisited and those alternatives that
have already been evaluated. Dutoit et al. (2006)
provide a description of potential uses of design
rationale and design rationale methods and argue
that these can also apply to any other kind of
rationale-based software engineering activity apart
from software design. Drawing a parallel to pro-
cess evolution, some of these potential uses are the
following: it supports reworking of software pro-
cess model/standards; it supports identification of
the potential impact area of a process change; it
encourages self-reflection when making decisions;
and it supports the identification and analysis of
nonsystematic and rushed decisions. Additionally,
we have observed during the execution of a pro-
cess evolution project (Armbrust et al. 2005) that
information about the rationale of the process evo-
lution can help process engineers or the person(s)
responsible for the process in the organization to
effectively and efficiently tailor process models or
update them. What matters to real-world organiza-
tions is how fit their process is regarding current and
future needs. Tailoring a reference process model or
a software process engineering standard can be sup-
ported by knowing about the issues, alternatives,
arguments, and criteria that justified the definition
of the current reference process model/standard.
The process engineer can define a tailoring strat-
egy that finds a balanced compromise for ade-
quate process changes, taking into account both
the rationale behind the current reference process
model/standard and the current and future needs of
his/her organization. Equally, updating a reference
process model/standard whose history is known
can lead to changes to process models/standards
that reflect actual practices and are oriented towards
satisfying current and future needs of development
organizations.

The main contribution of this article consists of
transferring the concepts defined by the design
rationale community to the process modeling com-
munity, especially to the problem of process evo-
lution. It is important to notice that large amounts
of rationale information could become a straight-
jacket for the process of evolving a software pro-
cess. Therefore, we followed an experience-based,
bottom-up approach for defining the concepts, the
classification of issues, the approach, and a tool (our
specific contributions) to be used by organizations

for collecting appropriate amounts of information
about the rationale underlying process changes and
for evolving the process in a systematic way. We
have defined a roadmap for moving iteratively
forward towards such concepts, classification of
issues, approach, and tool. It can be summarized
in the following three main steps: The first step is
to understand the nature of process changes. This
means understanding what is needed for describ-
ing a change and the reason for it. Additionally, we
assume that by having a predefined classification
of the most common situations for process changes,
the task of collecting the information related to the
rationale can be simplified and become more suit-
able for use in real process evolution projects. Once
this is understood, in a second step, a structured con-
ceptual model of rationale can be produced together
with an approach that provides guidance on how
to perform systematic process evolution supported
by the developed conceptual model. In a final third
step, the conceptual model and the approach can
be applied in process evolution projects, supported
by a tool. The experience acquired in trial projects
is used for fine-tuning the conceptual model, the
approach, and the tool before using them in new
projects. So far, we have derived our current results
from the experience acquired in the context of
traditional, plan-driven (in Boehm’s sense) pro-
cesses, i.e. development is performed following
a requirements/design/build paradigm with stan-
dard, well-defined processes (Boehm and Turner
2004).

On the basis of the definition of software engi-
neering validation models provided by Zelkowitz
and Wallace (1998), our contributions are the prod-
uct of three different case studies performed in the
context of a large and a small organization: the
European Space Agency (ESA) and the Adaptive
Services Grid (ASG) project, respectively. The first
study was concerned with the problem of capturing
the justification of changes to aerospace engineering
standards (Armbrust et al. 2005). The second study
aimed at understanding the most common issues
behind process changes (Ocampo and Münch 2006).
Both studies were performed in the context of the
ESA. The results of these studies and the results of
a search on rationale literature provided the input
for a new conceptual model tested in the second
organization (ASG project), which constitutes our
third case study.

86 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

Table 1. Comparison of some common rationale concepts

Approach Rationale

Kunz and Rittel (1970) Issue Position Argument
MacLean et al. (1991) Question Option Argument Criterion
Lee (1990) Issue Alternative Claim Criteria Procedure
Bruegge and Dutoit (2004) Issue Alternative Argumentation Criteria Resolution
Chung et al. (1999) Goal Alternative Claim Criteria
Sauer (2002) Argument prototypes Assessment function
Ramaesh and Dhar (1994) Issue Position Argument Assumption Decision

This article presents the progress obtained from
the research roadmap as follows: Section 2 provides
short descriptions of related work, where concepts
for understanding process changes and rationale
were developed. Section 3 presents the context
and enhanced results of our previous studies
performed for understanding rationale information
needs. Section 4 presents the conceptual model
and approach developed incrementally with the
findings of the case studies. Section 5 discusses the
application of the conceptual model and approach
during the evolution of a process for developing
service-oriented applications in the context of
the ASG project. Section 6 summarizes the most
relevant findings with respect to the conceptual
model and approach as well as new research
questions to be addressed in the future.

2. RELATED WORK

This section discusses how research work from the
design domain and the process modeling domain
has addressed the problem of understanding the
nature of decisions, changes, and their rationale.

Historically, much research about rationale has
focused on design (Dutoit et al. 2006). Rationale
models represent the reasoning that leads to a
system, including its functionality and its imple-
mentation (Bruegge and Dutoit 2004). Kunz and
Rittel (1970) developed the IBIS issue model for
design rationale, which has been used as a basis
for other work on rationale modeling such as Lee
(1990), MacLean et al. (1991), Bruegge and Dutoit
(2004), and Chung et al. (1999). This model con-
tains three types of nodes: issues, positions, and
arguments. Issues are used to state the problem.
A position is a solution suggested by a developer.
Arguments either support or contradict a position.
The assumption made for issue modeling is that

the design of a system/component is performed
as follows: first, an issue is identified; then sev-
eral alternatives to resolve the issue are proposed;
at the same time, such alternatives are evaluated
against relevant project criteria; finally, a decision is
made by selecting the alternative that best matches
the criteria and resolves the issue. The ideas of
Kunz and Rittel were implemented by Conklin and
Burgess-Yakemovic in a tool called gIBIS (Conklin
and Burgess-Yakemovic 1991).

Table 1 presents our classification of some of the
approaches for representing rationale information.

This comparison identifies possible commonali-
ties among the approaches. For example, an issue
for Kunz and Rittel (1970) is similar to a question
or a goal for MacLean et al. (1991) and Chung et al.
(1999), respectively. Kuntz and Rittel’s issue mod-
eling is the basis for most of the approaches. The
MacLean et al. (1991), Bruegge and Dutoit (2004),
and Chung et al. (1999) approaches propose that
the rationale should be the driver for design activ-
ities. This way, rationale-related activities become
a part of the software development process and
not merely a parallel, additional activity. These
approaches extend the issue model with concepts
such as criteria/criterion/assumption, and resolu-
tion/procedure. Criteria are desirable qualities that
the selection of a proposal should satisfy. For exam-
ple, during requirements analysis, criteria include
usability or performance, while during project man-
agement criteria include timeframe, costs, or risks.
Finally, a decision is the resolution of an issue.
It contains the selected proposal and the justifica-
tion of the selection. At the same time, a resolution
could trigger more issues. Ramaesh and Dhar (1994)
developed a conceptual model and a prototype that
relates the use of knowledge about design ratio-
nale to the use of knowledge in design tasks. They
extended the IBIS model (Kunz and Rittel 1970)

DOI: 10.1002/spip 87

Research Section A. Ocampo and J. Münch

with concepts necessary for representing knowl-
edge components involved in system design (e.g.
data flow, data storage).

A common drawback of all the previous
approaches is that these are prescriptive approaches
that call for discussions before creating or updating
a design, something that may not be necessary in
cases where changes are trivial.

Sauer (2002) presents a procedure for extract-
ing rationale information from the project log and
for justifying project decisions. The concepts used
are taken from the Event-Based Design Rationale
Model (EDRM) developed for the Minimally Inva-
sive Long-Term Organizational Support (MILOS)
environment (Maurer et al. 2000). Sauer claims that
by following his procedure, rationale information
can be generated semiautomatically, thereby over-
coming one of the major obstacles to capturing
rationale, namely the costs and effort required
for its elicitation. The concepts are similar to the
ones presented by the IBIS issue model, with
the major difference being that the information is
derived from an analysis of a project trace that
contains events and their interdependencies. How-
ever, Sauer’s approach does not capture information
about the positions/alternatives/options that were
not taken into account, which is valuable informa-
tion for future decisions. In other words, semiauto-
matic generation of possible strategies for making
a decision is very helpful, but this must be com-
plemented with the information captured from the
experts who actually make the decision (e.g. actual
discussion, criteria for selecting the best alternative).

In general, several design rationale concepts and
approaches that provide a structured basis for sys-
tematically performing and documenting changes
to a software product are available. Almost all of
the approaches offer tool support (often in the form
of hyper-linked information) and put an emphasis
on it. Most criticism regarding these approaches is
related to the costs of collecting and maintaining
such rationale information (Lee 1997, Buckingham
Shum et al. 2006). Since this prior work is highly
relevant to our work, and since we are aware of
its pros and cons (Bruegge and Dutoit 2004), we
surveyed how process modeling approaches per-
form and document changes, with a focus on their
similarities to the design rationale domain and on
determining whether such approaches adequately
address the criticism regarding cost.

We found that research work has been directed
toward developing approaches (Nguyen and Con-
radi 1996, Ahmed 2004, Bhuta et al. 2005, Nejmeh
and Riddle 2006, Madhavji), and process support
environments (PSEs)1 (Jaccheri and Conradi 1993,
Kaba and Derniame 1995, Koskinen and Martiin
1998, Maurer et al. 2000, Alloui et al. 2001, Green-
wood et al. 2001, Seet et al. 2003, Weber et al. 2005,
Cunin et al. 2001), for controlling software process
evolution. Some researchers propose encapsulat-
ing information about the process, its context, and
its changes. Examples are process components pro-
posed by Bhuta et al. (2005), dependency and change
structures proposed by Madhavji, the change pat-
terns proposed by Nguyen and Conradi (1996),
update taxonomies proposed by Ahmed (2004), and
the execution logs, change logs, and cases proposed
by Weber et al. (2005).

Other investigators have proposed conceptual
frameworks as in the case of Nejmeh and Riddle
(2006).

AU of the approaches recognize the need for
concepts and tools that can be used for collecting
information about process model changes that
help evolve the process in a systematic way.
However, most of them have not considered
rationale information as an important part of their
frameworks. Those approaches that do consider
concepts similar to the ones from design rationale
provide little evidence (i.e. collected data) on
reasons that trigger process evolution in a given
context (Madhavji, Nguyen and Conradi 1996,
Weber et al. 2005). Their research was mostly
oriented toward developing structures and patterns
to apply in PSEs that evolved both the instance and
the model of the process.

One alternative approach called software process
redesign (SPR) (Scacchi 2000) is concerned with
the identification, application, and refinement of
new ways to dramatically transform or improve
a process. The main inputs for SPR are the so-
called SPR heuristics and knowledge about the
use of a process. SPR heuristics can be classified
as being of two types: domain-independent SPR
heuristics, which can be applied to a large set
of processes, and domain-specific SPR heuristics,
which can be applied only to certain processes

1 PSEs are defined as human-oriented systems intended to serve
the interaction of computerized tools and humans (Seet et al.
2003).

88 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

in certain circumstances. The knowledge used to
derive the heuristics is obtained from narrative
reports on how to dramatically improve the life
cycle or prevent defects. Such narrative reports
are extracted from case studies, lessons learned,
best practices, and experience reports. It is here
that rationale concepts can contribute to the SPR
approach because they can be used for structuring
and therefore improving the sources of knowledge
needed for elicitation of redesign heuristics.

In summary, there has been a lot of research in
the design rationale domain targeted at providing
support to software engineers for making decisions
about changes to the software product. There has
also been a lot of research in the process modeling
domain concerning the evolution of processes, with
a focus on how to make changes to processes being
enacted, and how to reflect such changes at the
process model level. However, less emphasis has
been put on the reasons for changing a process
model and on how these can be used as input
for future decisions that change a process. This
motivates our research goal and contribution, which
consist of creating a clearer and more systematic
link between the domains of design rationale
and software process evolution. The next section
presents the results of a first step toward our goal,
namely understanding rationale information needs.
This has been realized by performing a study of the
concepts needed for expressing the rationale for
process changes and a study of the most common
issues for changing a process.

3. STUDIES FOR UNDERSTANDING
RATIONALE INFORMATION NEEDS

Two studies that were performed in the context
of a project focused on the evolution of aerospace
engineering standards are presented in this section.
The first study was performed with the objective
of identifying which information should be part of
the description of a rationale for process changes.
Additionally, a second study was performed, with
the objective of identifying those most common
issues that triggered changes to the process.

3.1. Context

The European Cooperation for Space Standardiza-
tion (ECSS) is an initiative established to develop

a coherent, single set of easy-to-use standards for
all European aerospace-related engineering activ-
ities, covering all areas, including engineering,
quality assurance, and project management. For
the ESA, the relevant standards applicable for
developing software are the following: ECSS-E-
40B Space Engineering – Software (ECSS-E-40 Part
1B Space Engineering: Software – Part 1: Princi-
ples and Requirements 2003) (mostly based on
the ISO 12207 standard : Information Technol-
ogy – Software Life Cycle Processes International
Organization for Standardization), and ECSS-Q80-B
Space Product Assurance – Software (2003). Organi-
zations or projects that are part of ESA are required
to develop and use specific tailoring(s) of the ECSS
standards suited to their work. This is a particularly
complex task because it requires detailed under-
standing of the whole standard, something that an
average software developer or project manager usu-
ally does not have (Ponz and Spada 2006). At the
ESA Space Operations Center ESOC (the ESA orga-
nization where this project took place), this tailoring
was called the Software Engineering and Manage-
ment Guide (SEMG) (Jones et al. 2002) and was used
for all their major projects.

After some years of experience with the ECSS
standards, they were revised by ESA, and a new
version was published. This also meant that the
SEMG had to be revised, in order to be compliant
with the revised ECSS standard. This compliance
had to be proven by means of traceability of every
ECSS requirement to its implementation and by
providing a tailoring justification for every tailored
requirement.

Another important task was to improve the
usability of the SEMG. For the purposes of this
project, process engineers (i.e. the authors of this
article) considered that the ease of use of a document
is positively influenced by improving (i) internal
consistency, i.e. avoiding contradictions between
parts of the document, (ii) external consistency, i.e.
avoiding contradictions with other documents and
assuring that links to external sources are correct,
and (iii) conciseness, e.g. ensuring that indexed
tables of contents allow people to find important
things quickly, that different concepts are explained
and marked clearly, and that the document is not
larger than necessary.

DOI: 10.1002/spip 89

Research Section A. Ocampo and J. Münch

3.2. First Study

The setting of this project allowed the definition of
research goals oriented towards obtaining insights
about the rationale for process changes. This section
describes the specific research goal defined for the
study, the way the study was performed, and its
results.

3.2.1. Goal and Operation
The goal of the first study was to understand which
type of information was needed for describing the
rationale for changes to a process model/standard.
In order to instrument this goal, process engineers
maintained detailed meta-information tables on
a per-section basis that allowed introducing the
description of the change and the reason for that
change. This table was to satisfy the demands of a
wide variety of different stakeholders who wanted
to keep track of the changes performed to the SEMG
and their justifications (Armbrust et al. 2005).

Table 2 shows a meta-information table. The
table contained not only the section’s unique
identifier (field invariant ID) but also a change
log and a list of traceability relationships. The
fields for storing the traceability relationships (ECSS
Coverage section) corresponded to the analog fields
in a database where this information was stored. The
ECSS ID column corresponded to the requirements
identifier in the ECSS standard, and the compliance
column was used for selecting one out of the
following: tailored, tailored out, and compliant.
The justification column captured the rationale for
the changes. In case the compliance changed, the

Table 2. Table for collecting the rationale for changes to the
SEMG

Meta-information

Invariant ID
Change log

version to this version>
Reviewer’s comments

changes and rationale>

ECSS coverage

ECSS ID Compliance Justification

justification provided the rationale for the new
value. Therefore, if a requirement was compliant,
i.e. was found implemented in the SEMG, but a
decision was made to tailor it out, i.e. to remove
its implementation from the SEMG, the description
provided in the justification column supported such
a decision and the changes performed to the SEMG.
The opposite case can also be taken as an example.
In case an ECSS requirement was not found in
the SEMG, i.e. was tailored out, the rationale for
implementing it in the SEMG, i.e. for making
it compliant, was described in this justification
column.

The SEMG was modified iteratively and incre-
mentally, resulting in the SETG (Tailoring of
ECSS Software Engineering Standards for Ground
Segments in ESA) as follows: Process engineers
changed the SEMG and delivered a new version for
review. Afterwards, reviewers discussed changes
performed to the SEMG and accepted or rejected
such changes. The reviewers documented their deci-
sions and sent comments and suggestions to the
process engineers. Process engineers reworked the
SEMG on the basis of the comments and sugges-
tions. This iterative process allowed updating the
SEMG in a controlled way and enabled a continual
review of the accomplishment of the tasks.

Two versions of the SEMG resulted from the
editing – reviewing iterations.

3.2.2. Study Results
The collected rationale for changes was validated by
means of reviews performed by the project partner
members of the review committee and additional
process engineers. They studied each rationale and
provided feedback accordingly.

The validated rationale stored in a database was
used as a basis for automatically generating a part
of the standard called ‘Part D Traceability Against
ECSS’ (Tailoring of ECSS Software Engineering
Standards for Ground Segments in ESA). This docu-
ment presents tables with traceability relationships
between the SETG (old SEMG) and ECSS standards
as well as the justification for changes to affected
parts of the SETG standard.

The process engineers documented observations
with respect to the use of the meta-information
tables and their structure as a means for describing
the rationale. Below are some of these observations:

The tables were used by process engineers for
describing what changed and why. Sometimes the

90 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

information about what changed was too detailed,
sometimes too abstract. This might be due to the
fact that the structure provided did not contemplate
a difference between finely granular changes (e.g.
grammar errors or misspellings) and larger ones
(e.g. wrong control flow). Some SEMG changes
collected, such as the correction of misspellings or
grammar errors, or format changes, were not helpful
for recognizing important decisions. However, the
opposite was also observed, i.e. sets of changes
that helped to identify important decisions. One
example was a set of changes to the product
flow, where we observed a clear attempt to
separate system and software engineering activities,
something that was needed in the organization.
For example, many activities (such as system
criticality analysis, system requirements review,
system partitioning) that were performed by system
engineering personnel and their produced artifacts
were irrelevant for the software engineers and were
still part of the SEMG. This generated a lot of
frustration, because software engineers had to show
evidence of artifacts they did not need or use.

The lack of structure of text fields (change log
and justification) used in the meta-information table
influenced the understandability of the collected
information. The ESA reviewers commented on
confusing justifications that contained what was
performed instead of information on why. In other
cases, the change log contained information about
the change and about the reason in the same place.
These findings motivated the need to change the
conceptual model and try to use it in a new project.
The resulting conceptual model and its application
will be presented in Sections 4 and 5.

3.3. Second Study

Once we collected the information about the SEMG
changes, we used it as the basis for a second study
of the most important and common issues that were
resolved by each change. This section presents the
goal of the study together with its operation and
results.

3.3.1. Goal and Operation
The goal of the second study was to analyze the
changes performed to the SEMG for the purpose of
characterizing the issue types that triggered changes
in the context of the evolution of ESA standards.

We accomplished this by querying the database
that contained information on changes to the
SEMG and by manually analyzing each change’s
justification. A process ID uniquely identified the
changed entity during both iterations. Each record
provided information about the date of the change,
the entity’s name, and the change justification. First,
we assigned each justification a particular situation
or problem faced during the SEMG evolution. Then,
we proceeded to review in internal meetings the set
of situations and problems and grouped them into
a list of most common issues faced while doing the
SEMG evolution. A group of three process experts
outside the project reviewed the list of issues in two
iterations and provided feedback. Their comments
were used for refining the list.

3.3.2. Study Results
The list of issues was validated by means of reviews
performed by three external process experts who
provided feedback. The following is the refinement
of the list presented in Ocampo and Münch (2006)
and an explanation of the issues:

1. Process model is inaccurate
(a) Process engineers found that the pre-

scribed control flow among activities dif-
fered from the one followed in real
projects;

(b) Process engineers found that the pre-
scribed product flow differed from the one
observed in real projects.

2. Process model lacks precision
(a) Process engineers found activity descrip-

tions that could be understood in two or
more possible senses or ways;

(b) Process engineers found examples that
could be understood in two or more
possible senses or ways;

(c) Process engineers found names that did not
reflect the meaning of the process element
as commonly understood by practitioners.
A process element can be any of the
concepts used to model processes, e.g.
activity, tool, role, or artifact.

3. Process model is not concise
(a) Process engineers found activity descrip-

tions that contained superfluous or unnec-
essary statements;

(b) Process engineers found examples that
contained superfluous or unnecessary
statements;

DOI: 10.1002/spip 91

Research Section A. Ocampo and J. Münch

Figure 1. No. of occurrences per issue

4. The activity is noncompliant
• Process engineers found cases where activi-

ties did not fulfill the requirements stated in
the ECSS standards.

5. Process model is inconsistent
• Process engineers found examples that were

incorrectly referenced.

Figure 1 reflects the number of changes caused
by the issues listed above when changing the SEMG
during the first and second iterations. The list of
issues derived from analyzing the database with
the information about the evolution of process
standards provides an initial insight into the type
of changes performed in the context of this type
of project. It can be observed that the issues that
generated the highest number of occurrences such
as ‘process model is inaccurate’ (1) and ‘process
model lacks precision’ (2) reflect the distance that
existed between the process model and the actual
understanding of the process by its practitioners.
A more detailed description of the results can be
found in Ocampo and Münch (2006).

Unfortunately, the analysis performed for extract-
ing the list of most common issues was not repeated
afterwards by process engineers not belonging to
the project. This constitutes a threat of validity to

be further investigated. However, this list of issues
was taken into account for the definition of the con-
ceptual model presented in Section 4 and reused
during its application described in Section 5.

4. RATIONALE CONCEPTS AND
APPROACH

Although the required tracking of changes was
supported during the first case study, the project
gave clear indications that advanced rationale
management could be beneficial. This motivated
us to design a new version of the conceptual model
and approach.

According to researchers in the design rationale
community (Fischer et al. 1991, Bratthall et al. 2000,
Burge and Brown 2004, Dutoit et al. 2006), main-
tainability and controlled evolution of a system are
dependant on the understanding of what is cur-
rently present, as changes in design are affected
by prior design. This means that in the case of
software process evolution, one should be able
to describe the relationships between an existing
process model and its previous version(s). Such
relationships denote differences between versions
due to distinguishable modifications. The purpose
of such modifications can be distinguished if one
can understand the rationale behind them. Figure 2

92 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

Figure 2. Rationale support for organizational process evolution

shows one scenario where rationale knowledge sup-
ports organizational process evolution. Here, an
organizational process must be followed by dif-
ferent projects inside the organization. However,
owing to the special contextual characteristics and
needs, each project changes the organizational pro-
cess. In this example, project 1 evolves the process
and produces two new versions, while project 2 pro-
duces three new versions. At certain points in time,
the organization’s management or roles responsible
for the process take a look at the changes performed
by each project with the purpose of understanding
what is new, what was not suitable, and what could
be relevant for other projects. Such questions can be
answered better if the reasons for changes R (ratio-
nale) between project process versions are known.
Furthermore, the rationale available can be used as
input for changing the organizational process.

Rationale knowledge provides the decision-
making elements that lead to the process knowledge
at both the project and organizational level (Dutoit
et al. 2006). It also augments the process knowl-
edge, i.e. the work required to develop the system,
including knowledge about roles, resources, tasks,
and work products.

According to Kneuper (2002), the knowledge
representation schemes must fulfill the following
characteristics:

• Expressiveness: refers to the ability to represent
the desired knowledge at the appropriate levels
of granularity.

• Effectiveness: concerns the means that a rep-
resentation scheme provides for inferring new
knowledge from old.

• Efficiency: means that the user must be able to
find the knowledge needed within a reasonable
time frame, and be able to find and use it with
acceptable effort.

• Explicitness: allows different levels of detail or
abstraction.

• Accessibility: is accessible from different techni-
cal environments.

• Modifiability: is easy to improve and adapt to
changing environments.

• Understandability: is easy to understand and
use by users.

• Searchability: refers to appropriate search and
selection mechanisms.

One lesson learned from the first study on the
evolution of SEMG standards (Armbrust et al. 2005)
was that the information provided in the respective
change logs by the process engineers who changed
the standards was not sufficient for understanding
the evolution.

Our conceptual model lacked expressiveness,
effectiveness, efficiency, explicitness, and under-
standability. However, the technology that we chose
supported us in providing appropriate accessibility,
modifiability, and search and selection mechanisms
(Armbrust et al. 2005).

Before the study, we were convinced that one
major value of such meta-information tables is to
document information to be used in the future. This

DOI: 10.1002/spip 93

Research Section A. Ocampo and J. Münch

was not the case, because our meta-information
tables lacked the structure for describing why and
where a process was changed, what the changes
were, what triggered them, what the alternatives
were, and which one was selected as the final
decision. We arrived at the conclusion that the
lack of a structure of and guidelines for the meta-
information tables were the main factors leading to
this confusing information. This led us to develop a
second version of our conceptual model targeted at
understanding the information needs for capturing
the rationale behind process changes (Figure 3). We
started with a small set of concepts that will be
refined with time. The reason for keeping the model
as simple as possible comes from the criticism
regarding the high costs of capturing rationale
information (Bruegge and Dutoit 2004, Burge and
Brown 2004, Dutoit et al. 2006). We wanted to avoid
these high costs and find those appropriate concepts
needed to describe the rationale of changes.

We took the most common concepts used in
the design rationale domain (Table 1) as a basis,
customized them to our needs, and connected
them to four entities that were relevant for us,
i.e. event, changes, process element, and version
(the non-shadowed classes in Figure 3). An event is
considered to be the trigger of issues. Events can be
external or internal. Examples are:

• External
– New/updated process engineering technol-

ogy (e.g. a new process modeling technique)
– New/updated regulatory constraints

• Internal
– Responses to failures to pass internal or

external appraisals, assessments or reviews
(e.g. changes needed to address results from
a process assessment based on the ISO
15504 standard (International Organization
for Standardization. ISO/IEC 15504 : 2003))

– New/updated best practices emerging from
lessons learned in just-completed projects
(e.g. a new best practice approach to han-
dling design reviews)

Changes are the result of the decision captured
in the resolution. They are performed on process
elements and are produced in a given version.
Some examples of changes performed on process
elements are: activity x has been inserted; artifact y
has been deleted; activity x has been moved to be a
sub-activity of activity z. The version is composed
of a set of process elements.

Issues are situations that arise as a consequence
of an event which need to be solved and are
related to a (part of a) process. An issue contains a
question, a description, a status (open, closed), and a
discussion. The discussion is intended for capturing
the emails, memos, letters, etc. where the issue was
treated by process engineers. Additionally, an issue
can be categorized by a type. This type must be
selected from a classification of issues that needs
to be developed or customized for an organization.
The classification presented as a result of the second
case study in Section 3.3 can be used as a basis,

Figure 3. Rationale model (UML static structure diagram)

94 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

but it should be refined continually on the basis of
experience gained from process evolution projects.

Alternatives are proposals for resolving the issue.
Alternatives can be captured with subject (short
description) or long descriptions. Alternatives
are evaluated and assessed by process engineers
regarding their impact and viability.

Finally, a resolution chooses an alternative whose
implementation causes changes to the process
elements. At the same time, one resolution could
lead to opening more issues. Note that a resolution
has a subject (short description), a long description,
and a justification. The justification is intended for
capturing a summary of the analysis of the different
alternatives, the final decision, and the proposed
changes. Table 3 shows an example taken from a
real project, where we illustrate the concepts.

The approach we propose for evolving a process
based on rationale is illustrated in Figure 4.

This approach is inspired by the existing rationale
management approaches presented in Section 2

Table 3. Example of a rationale for process changes

Concept Value

Event 1 – name: Process review;
– type: Internal;
– description: Review performed by process

engineers based on interviews with
practitioners.

Issue 1 – type: Process model is inaccurate;
– description: The processes that describe the

engineering of flight software are performed
by another team of engineers.

Alternative 1 – description: Remove all processes concerned
with the engineering of flight software;

– assessment: Positive – It shall be clear that
these processes are performed by another
special team of engineers and therefore
products that are to be checked during the
reviews shall not be produced at this team.

Alternative 2 – description: Declare this process optional;
– assessment: Negative – The engineers at this

team do not follow these processes. This is
not their area of competence.

Resolution 1 – description: The review board agreed
to remove these processes;

– justification: Process engineers agreed that
alternative 1 was more appropriate
because it reflects better what is being done
at the team and avoids confusion during the

reviews.
Changes – Existent processes related to the engineering

of flight software must be removed;
– The products generated by such processes

must be removed along with their templates.

and traditional change management processes that
can be found in several software development
standards, e.g. (IEEE Std-828-2005). Despite its
similarity to existing approaches, what is new in the
approach of Figure 4 is that it attempts to explicitly
describe the steps.

First, an internal or external event that has taken
place is communicated to the process engineer in
charge of the process via mail, memo, official doc-
ument, or by the most convenient artifact. In the
activity ‘identify issues’, the process engineer iden-
tifies the issue(s) he or she is facing due to the
event and documents them using the previously
described concepts. The process engineer proceeds
to ‘identify conflicting issues’. He does this by
querying and visualizing previous rationale infor-
mation in order to find open issues that conflict
with this new one. He could also identify issues
treated before and how they were resolved. He also
analyzes dependencies on issues, and the impact
of the issues on the process model. The process
engineer ‘proposes alternatives’ for discussing with
other involved roles (e.g. quality assurer, project
manager, quality manager, trainer, software devel-
oper) to resolve the issue(s). The process engineer
then ‘resolves the issue(s)’ by selecting the most
appropriate alternative(s) and defining a strategy
of changes to be realized. Then, either the process
engineer or the role assigned inside the organization
‘extracts the process model version’ to be modified
and ‘updates the process model’ using the reso-
lution information as input. Once the changes are
finalized, the process engineer ‘stores a new process
model version’ and closes the issue(s).

5. APPLICATION OF CONCEPTS
AND APPROACH

The conceptual model and the approach have been
applied for the purpose of characterization in a
real environment. This has given us insights into
the extent to which process engineers used the
terminology to capture rationale, and the extent
to which such an approach can be introduced
successfully in industry.

5.1. Context

The environment where we applied our concep-
tual model corresponds to the ASG project (ASG:

DOI: 10.1002/spip 95

Research Section A. Ocampo and J. Münch

Figure 4. Approach (product flow – Spearmint notation (Becker-Kornstaedt et al. 1999))

Adaptive Services Grid). The ASG was intended
to develop a software infrastructure that enables
the design, implementation, and use of appli-
cations based on adaptive services, namely the
ASG platform. Although the basic concepts of
service-oriented architectures and web services
have become very popular in the last few years,
much confusion remains concerning strategies and
processes suitable for engineering adaptive web ser-
vices (Bayer et al. 2006). Owing to this problem, we
were in charge of defining, establishing, evaluating,
and systematically evolving the development pro-
cess applied in the project to develop the platform.
Development activities within the ASG project were
performed, for instance, by several teams from
different companies, universities, and research insti-
tutes. Development teams ranged from two-person

teams consisting of a PhD student and a master
student to ten professional programmers. Develop-
ment teams were not co-located and team members
spoke different native languages.

Software processes were described in terms of
activities (e.g. plan project or design module),
artifacts (e.g. project plan, module design), roles
(e.g. project manager, designer), and assets (e.g.
plan template, design guidelines). The difference
between artifacts and assets is that artifacts are
tangible results whereas assets are resources sup-
porting process performance (Nejmeh and Riddle
2006). The resulting process models include both
textual descriptions and diagrams that illustrate the
relationships between the entities of the model in
a graphical way (e.g. workflows and role-specific
views).

96 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

5.2. Study Goals and Operation

The goal of this study was to analyze the conceptual
model and the approach for the purpose of eval-
uation in the context of the evolution of the ASG
reference process model.

The ASG reference process model was developed
mainly in three iterations. This means that there
are three versions of the model. At certain project
milestones, we interviewed developers about the
current process model version. Such interviews
were taken as a basis for performing changes
to the process model. Mapped to the previously
presented conceptual rationale model (Section 4),
such interviews can be seen as the event that
triggered issues to improve the process. We found
these interviews very useful for eliciting issues
and alternatives, since developers usually provided
information about possible solutions to the problem
as observed in Table 4.

We discussed the interviews, decided on the
changes, and documented their rationale. Then we
proceeded to perform the approved changes.

In general, the main idea behind the system-
atic evolution approach followed was to start with
commonly accepted process knowledge, refine it
with information gathered from the development

cycles, and therewith improve the textual descrip-
tions and diagrams of the process according to the
real project needs. We extended a tool-assisted way
of editing and reviewing the process description
(Armbrust et al. 2005) with a means for editing
and visualizing rationale (Figure 5). To achieve
this, we created a persistent connection of standard
word processor documents containing the process
description to a model of the documents in a rela-
tional database, which allowed us to keep rationale
meta-information as well as automate advanced
consistency checks. Our tool called ‘Rationale-
driven Evolution and Management Information
System’ (REMIS) (Ocampo and Münch 2007) also
allowed us to visualize rationale information in the
form of graphs before editing the process.

Our solution relied on the fact that modern
word processing programs increasingly support the
Extensible Markup Language (XML) as a document
format (Merz 2004). As an open format, XML can be
processed using a variety of widely available tools,
including high-level libraries that can be invoked
from most modern programming languages. Using
the interpreted, object-oriented Python program-
ming language (Lutz 2001), we developed a parser
that was able to navigate through the XML versions

Table 4. Excerpt of interview with software developers

Interview 3: software developer 1

Interview conducted in – Tromsø, Wednesday, July 13, 2005, 11 : 00 AM – 12 : 00 noon
Interviewed person – User 1
Organization – Organization 1
ASG subsystem – Service Composer (C-2)
Role in prototype development – Subsystem Manager and Integrator
Team size – Two persons (incl. Subsystem Manager)
Team experience – More that 10 years programming experience with several languages

(object-oriented, imperative, and functional)
ASG development process – The ASG process was followed. The ASG process description on ASG Wiki was

known and used. Software developer 1 is the author of many of the linked pages
for assets and tool descriptions

Process change/improvement proposals – Owing to the short-termed synchronization points defined for the prototype
development, the steps defined within the subsystem-related activities seem
to be unnecessarily complex

– The process description should explain how the platform, the applications, and the
underlying services are developed/integrated

– Role-specific views on the overall process should be provided in order to improve
readability

– The activities Integration and Integration Test should be merged, since they are
concurrently performed by the same persons with the aid of the same
supporting tools

– The process description is too verbose

DOI: 10.1002/spip 97

Research Section A. Ocampo and J. Münch

Figure 5. Tool support for process evolution supported by rationale

of the ASG process model description, identify-
ing the section headings and rationale information
tables, and moving information to and from the
database as necessary. This functionality allowed
us to update the database automatically after a
set of changes, and to check the data for consis-
tency before doing any further editing. We used
the Resource Description Framework (RDF) as a
basis for representing rationale information and
process evolution information (Klyne and Carroll
2004). In brief, RDF was originally designed for rep-
resenting metadata about web resources, such as
the title, author, modification date of a web page,
and copyright. However, it is possible to generalize
the concept of ‘Web Resource’ and say that RDF can
be used to represent ‘things’ that are identifiable.
RDF’s conceptual model allows describing ‘things’
by using statements and models such as nodes and
arcs in a graph. We use the RDF/XML (Manola and
Miller 2004) syntax for storing and querying RDF
graphs in the database. We see rationale as metadata
about processes. Such metadata can be queried for

describing the evolution of processes. The visual-
ization tools developed so far allow us to display
rationale information in the form of graphs. This
approach is similar to the one proposed by Scacchi
(2000) for representing, querying, and visualizing
process model information and redesign heuristics.
One key difference between the two approaches lies
in the way information is captured. In REMIS, the
conceptual model structures the knowledge infor-
mation to be captured, whereas Scacchi’s approach
does not provide a structure for it and references dif-
ferent kinds of knowledge, such as narrative reports
or case studies. These are analyzed and used for
deriving SPR heuristics. Another key difference is
that Scacchi’s approach uses inference mechanisms
for deriving the SPRs, while REMIS does not use
them yet for suggesting resolutions to issues that
happened in the past.

5.3. Study Results

We observed that attaching the rationale and
change information directly to each of the process

98 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

document sections, and collecting it automatically
from there, facilitated the whole evolving activ-
ity, in the context of this project. Our conceptual
model played an important role, because it allowed
structuring the reasoning behind a decision and
motivated self-reflection about the need for chang-
ing the process. Equally, the structure of the con-
ceptual model allowed reusing this information in
a straightforward way, before performing future
changes. This improves the quality of the deci-
sions and support traceability from the changed
process entity to the issue as well as knowledge
transfer.

Table 5 shows an example taken from a real
project, where we illustrate the concepts.

We also noted that rationale visualization can be
useful for answering different types of questions

relevant for process managers or quality assurers
before designing the strategy to resolve issues.
Examples of such questions are:

• What was the event that triggered certain issues?
• What is the type of issue that causes most

changes to process elements?
• Which are the still open issues that may conflict

with the resolution of a new issue?
• What was the resolution that caused certain

changes to a process element?
• How many processes were affected by a

resolution?
• Which are the rejected alternatives for a closed

issue? Why?

Figure 6 shows a generated graph-like visualiza-
tion for answering the question ‘How many ASG
process elements were affected by resolution 1?’ (for

Table 5. Example of rationale for process changes in the ASG project

Concept Value

Event 1 – name: Process review;
– type: Internal;
– description: Review of the ASG life cycle process performed by process engineers based on interviews with
developers.

Issue 1 – type: Process description lacks precision;
– description: The ASG platform offers capabilities such as the integration of external services and provision of
services to external applications. Some groups of developers are in charge of external services, while others
implement applications to exploit the power of the platform. Additionally, they are geographically distributed. The
current ASG life cycle process description guides developers on platform development. However, the ASG life
cycle process description does not explain how applications and external services can be developed and integrated
into the platform;
– question: What is the best strategy for transforming the actual process description into one suitable for platform,
service, and application engineering?

Alternative 1 – description: Create one reference process model that covers all three disciplines (platform, service, and application
engineering), but clearly mark those processes that belong to each one;
– assessment: Positive – Reworking cost of the process description due to redundancy and inconsistency between
the different disciplines can be lowered.

Alternative 2 – description: Create a process description for each engineering discipline, i.e. one for process, one for application,
and one for platform engineering;
– assessment: Negative – The risk of process inconsistencies, and/or redundancies because of maintenance of three
similar but at the same time different disciplines can lead to high reworking costs, and low quality products.

Resolution 1 – description: The review board agreed to create one reference process model that covers all disciplines;
– justification: Process engineers agreed on alternative 1; process engineers agreed that alternative 2 was less
appropriate because the risk of inconsistencies and redundancies was too high. Process engineers agreed that
alternative 1 was more appropriate not only because of minimizing the process reworking risks but also because
one reference model can lead to a better understanding of the interactions between service, platform, and
application engineering.

Changes – existent processes were renamed with a prefix according to the discipline they belong to;
– the existent activities’ purposes were modified in order to differentiate that a process belongs to a certain
discipline;
– the product flow was modified in order to appreciate the separation among disciplines more clearly;
– the involvement of roles was modified accordingly;
– new activities were introduced;
– new roles were created.

DOI: 10.1002/spip 99

Research Section A. Ocampo and J. Münch

Figure 6. Impact of a resolution (partial view of a graph generated by the REMIS tool)

a description of resolution 1, see Table 5). It shows
the specific name of the processes belonging to three
different disciplines (platform, service, and appli-
cation engineering) present in the ASG reference
process, which were affected by several changes
performed because of such a resolution. The alter-
natives, issue, type of issue, and event could also
be displayed by using our tool to generate a larger
graph, which also includes these concepts and their
relationships.

Our tool provided us with functionalities for flex-
ibly querying rationale information and visualizing
the results of such queries. Figure 7 shows an exam-
ple of such a capability with a simple visualization
that helped us to answer the question ‘Which are
the rejected alternatives for an issue?’ Having the
possibility to get this visualization can be useful
for avoiding inconsistencies and rework in cases
where process engineers face an issue they believe
to have solved in the past. Process engineers can
have better means to analyze why other alterna-
tives were rejected at the time and assess if this
still continues to be the case. Further research is
needed into defining common queries and their
most appropriate visualization.

Some observations regarding the collection of
rationale gained in relevance. For example, we
found that sometimes it was difficult to differentiate

between the descriptions of the alternatives and the
description of the resolution. We also felt there was
redundancy when documenting the justification of
the resolution. This led us to the discussion of the
need for an additional, lighter rationale approach,
because sometimes special types of issues do not
require the formulation of alternatives, but rather a
direct resolution (Figure 8).

In those cases, the process engineer should be
able to document light rationale information after
performing the changes. This can be done by
using a tool that helps a process engineer to
detect changes between two process model versions
automatically, such as in the approach presented in
Soto and Münch (2006). The comparison tool shows
a summary of changes done on the work version,
which serves as input for the process engineer
for going through each change and documenting
a light rationale consisting of short descriptions
of the event, the issue, and the resolution. An
example of how to collect this rationale information
and connect it to the changed process elements
after changes have been performed is presented in
Ocampo and Soto (2007). Before we start to follow
this ‘light’ variation of the rationale approach, we
might further investigate differentiating between
those issues that need a very well documented
rationale and those that do not. This can be helpful

100 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

Figure 7. Issues, their alternatives and assessment (graph generated by the REMIS tool)

Figure 8. Lighter collection of rationale information

for minimizing the risk of over-expenditure and
addressing the criticism associated with capturing
rationale information (Burge and Brown 2004,
Dutoit et al. 2006).

6. SUMMARY AND OUTLOOK

Processes may be changed more consistently if
the information about the process, its context, and

DOI: 10.1002/spip 101

Research Section A. Ocampo and J. Münch

the rationale of its evolution is captured. Existing
approaches recognize the need for concepts and
tools that can be used for collecting information
about process model changes that could help evolve
the process in a systematic way. We observed that
most of the approaches did not consider rationale
information as an important part of their work
(Madhavji, Nguyen and Conradi 1996, Weber et al.
2005). This is one possible reason for the small
amount of evidence available on the rationale of
process evolution. With a predefined classification
of the issues that trigger process model changes,
the task of collecting the information related to such
rationale becomes simpler and more suitable for use
in real process evolution projects.

In this article, we have presented a classification of
issues based on a study performed on data collected
in change logs during the evolution of software
engineering process standards from the aerospace
engineering domain. More research work has to be
done on describing this initial list of issues more
precisely, so that they are as orthogonal as possible.
More empirical data is needed for that purpose.
We will analyze in a postmortem study the actual
type of issues documented during the evolution
of the ASG process, in order to contribute to the
significance of the actual list of issues presented in
this article. It is important to note that the studies
used as a basis for the work have been performed
mainly by the authors. This intrinsically biases the
results. However, the application in the ASG project
was supported by the partners involved, which
provided feedback to the conceptual model and
method and contributed to diminishing this bias.

We have also presented a conceptual model
that can be used as a basis for collecting the
rationale for changes. This conceptual model is
the result of combining lessons learned from
evolving aerospace engineering standards and an
extensive literature survey on design rationales
and software process evolution. Additionally, the
approach that uses the conceptual model and a
brief explanation of the tool support developed
were introduced. The observations obtained from
applying the approach during the evolution of
a reference process for developing a software
platform for adaptive services motivated us to
design an additional light rationale approach to
be applied in upcoming projects. This variation of
the approach will again be helpful for evaluating the
conceptual model and enhance it if necessary. We

also decided to define a common set of queries to
be displayed through visualization that empowers
rationale information. We observed that having
the means to visualize this information provides
process engineers with powerful analysis tools
that can be helpful when designing the strategy
to evolve a process model. As mentioned before,
not all process model changes are of value for
systematic process evolution. Therefore, we need
to continue experimenting (during the evolution of
processes used in industrial settings) on how to
capture and visualize such information in order
to more effectively identify those changes (and
their rationale) that impact the performance of
practitioners and really make a difference on their
software processes.

Furthermore, agile development practices and
techniques can be used as input for enriching our
contributions. According to Boehm and Turner
(2004), agile practices are characterized by the
following attributes: iterative (several cycles), incre-
mental (the entire product is not delivered at once),
self-organizing (teams determine the best way to
handle work), and emergence (processes, princi-
ples, work structures are recognized during the
project rather than predetermined). In the mean-
time, it can be assumed that in the context of new
methodologies like agile (and also open source),
which focus on the working code, the likelihood
of developers spending effort on carefully evolv-
ing their processes is low. However, this remains
an open issue for research in future projects. For
example, it can be investigated whether our contri-
butions can be used for enhancing self-organization
and emergence based on the principle of active
communication and discussion behind a rationale.
Other questions to investigate could be: Can our
conceptual model and approach become an impor-
tant tool for effective face-to-face communication in
agile practices? Could simple schems for capturing
and discussing agile practice rationale with good
tool support enhance the chances of acceptance?

Although agile practitioners emphasize the infor-
mality of their practices, such practices also evolve.
We assume that rationale can add value by self-
reflecting on the real reasons for such evolution.

Finally, we plan to relate the approach to popular
standards such as CMMI (CMMi Product Team
2006) or ISO 15504 (International Organization for
Standardization. ISO/IEC 15504 : 2003 2007).

102 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

ACKNOWLEDGEMENTS

We would like to thank Dr William E. Riddle
for his valuable comments on the list of issues
and discussion on process evolution. We thank
Michael Jones and Mariella Spada from the ESOC
for their reviews, support, and collaboration during
the ESOC project. We also would like to thank all
ASG project members who participated in the ASG
process interviews, and especially Fabio Bella from
Fraunhofer IESE, for eliciting and documenting
the information. Additionally, we would like to
thank Sonnhild Namingha from Fraunhofer IESE
for editing the English version of this article.
Finally, we thank the anonymous reviewers who
have contributed to the improvement of this article.
This work was partially supported by the German
Federal Ministry of Education and Research (V-
Bench Project, No.01| SE 11 A).

REFERENCES

Ahmed NM. 2004. Evolution of software processes and of
their models: a multiple strategy. Journal of Research and
Practice in Information Technology 36(1): 9 – 22.

Alloui I, Cı̂mpan S, Oquendo F. 2001. Monitoring
software process interactions: a logic-based approach.
In Proceedings of the 8th European Workshop on Software
Process Technology, Lecture Notes in Computer Science, vol.
2077, Ambriola V (ed.). Springer-Verlag: London, 39 – 46.

Armbrust O, Ocampo A, Soto M. 2005. Tracing process
model evolution: a semi-formal process modeling
approach. In ECMDA Traceability Workshop (ECMDA-TW)
2005 – Proceedings, Oldevik J, Aagedal J (eds): Trondheim,
SINTEF ICT, 2005 (SINTEF Report STF90 AO5133) ISBN:
82-14-03813-8, 57 – 66.

ASG: Adaptive Services Grid. Integrated Project
Supported By the European Commission, Available
at: http://asg-platform.org/cgi-bin/twiki/view/Public
Last checked 2008-03-05.

Bayer J, Bella F, Ocampo A. 2006. Characterization of
semantic grid engineering. In Workshop on Future
Research Challenges for Software and Services, Margaria T,
Fernandez-Villacanas JL, Banti M. (eds). FRCSS 2006:
Vienna, Austria, 112 – 124.

Becker-Kornstaedt U, Hamann D, Kempkens R, Rösch P,
Verlage M, Webby R, Zettel J. 1999. Support for the
process engineer: the spearmint approach to software
process definition and process guidance. Proceedings of
the Eleventh Conference on Advanced Information Systems

Engineering (CAISE ‘99), Lecture Notes in Computer Science.
Springer-Verlag: Berlin, New York, 119 – 133.

Bhuta J, Boehm B, Meyers S. 2005. Process Elements:
Components of Software Process Architectures, Software
Process Workshop, China.

Boehm BW, Turner R. 2004. Balancing agility and
discipline. A Guide for the Perplexed. Addison-Wesley:
Boston, MA.

Bratthall L, Johansson E, Regnell B. 2000. Is a design
rationale vital when predicting change impact? A
controlled experiment on software architecture evolution.
In Proceedings of the Second international Conference on
Product Focused Software Process Improvement, Lecture Notes
in Computer Science, vol. 1840, Bomarius F, Oivo M (eds).
Springer-Verlag: London, 126 – 139.

Bruegge B, Dutoit AH. 2004. Rationale management.
Object-oriented Software Engineering, Using UML, Patterns,
and Java, 2nd edn. Pearson Education: Upper Saddle
River, NJ.

Buckingham Shum SJ, Selvin AM, Sierhuis M, Conklin J,
Haley CB, Nuseibeh B. 2006. Hypermedia support
for argumentation-based rationale: 15 Years on from
gIBIS and QOC. Rationale management in software
engineering: concepts and techniques. In Rationale
Management in Software Engineering, Dutoit A, McCall R,
Mistrı́k I, Paech B (eds). Springer-Verlag: Berlin, 111 – 129.

Burge J, Brown DC. 2004. An integrated approach for
software design checking using rationale. In Design
Computing and Cognition ‘04, Gero J (ed.). Kluwer
Academic Publishers: Netherlands, 557 – 576.

Chung L, Nixon BA, Yu E, Mylopoulos J. 1999. Non-
functional Requirements in Software Engineering. Kluver
Academic: Boston, MA.

CMMi Product Team. 2006. CMMI for Development,
Version 1.2, Improving Processes for Better Products.
Software Engineering Institute, Carnegie Mellon
University, Pittsburg, USA: CMU/SEI-2006-TR-008.

Conklin J, Burgess-Yakemovic KC. 1991. A process-
oriented approach to design rationale. Human-Computer
Interaction 6: 357 – 391.

Cunin P, Greenwood RM, Francou L, Robertson I,
Warboys B. 2001. The PIE methodology – concept and
application. In Proceedings of the 8th European Workshop
on Software Process Technology, Lecture Notes in Computer
Science, vol. 2077, Ambriola V (ed.). Springer-Verlag:
London, 3 – 26.

Dutoit A, McCall R, Mistrı́k I, Paech B. 2006. Rationale
management in software engineering: concepts and
techniques. In Rationale Management in Software

DOI: 10.1002/spip 103

Research Section A. Ocampo and J. Münch

Engineering, Dutoit A, McCall R, Mistrı́k I, Paech B (eds).
Springer-Verlag: Berlin, 1 – 43.

ECSS-Q-80B Space Product Assurance. 2003. Standard
available at http://www.ecss.nl. Last checked 2007-06-
02.

ECSS-E-40 Part 1B Space Engineering: Software – Part 1:
Principles and Requirements. 2003. Standard available at
http://www.ecss.nl, Last checked 2007-06-02.

European Cooperation for Space Standardization (ECSS).
Available at http://www.ecss.nl. Last checked 2008-03-
05.

Fischer G, Lemke AC, Mccall R, Morch AI. 1991. Making
argumentation serve design. Human-Computer Interaction
63&4: 393 – 419.

Greenwood RM, Balasubramaniam D, Kirby GN,
Mayes K, Morrison R, Seet W, Warboys B, Zirintsis E.
2001. Reflection and reification in process system evo-
lution: experience and opportunity. In Proceedings of the
8th European Workshop on Software Process Technology, Lec-
ture Notes in Computer Science, vol. 2077, Ambriola V (ed.).
Springer-Verlag: London, 27 – 38.

IEEE Standard for Software Configuration Management
Plans, IEEE Std 828 – 2005 (Revision of IEEE Std 828 -1998),
New York, USA, 2005, pages 0-1-19.

ISO/IEC 15504-5:2006(E): Information technology – Process
assessment – Part 5: An exemplar process assessment
model. International Organization for Standardization
(ISO). Genf, 2006.

ISO/IEC 12207:1995(E): Information technology – Software
Life Cycle Processes. International Organization for Stan-
dardization (ISO). Genf, 1995.

Jaccheri LM, Conradi R. 1993. Techniques for process
model evolution in EPOS. IEEE Transactions on Software
Engineering (Special Issue on Process Model Evolution 19(12):
1145 – 1156.

Jones M, Gomez E, Mantineo A. 2002. Mortensen U.K.
Introducing ECSS Software-Engineering Standards
within ESA. Practical approaches for space- and ground-
segment software. ESA bulletin 111 – august. Available at:
http://www.esa.int/esapub/bulletin/bullet111/
chapter21 bul111 .

Kaba BA, Derniame J. 1995. Transients change processes
in process centered environments. In Proceedings of the
4th European Workshop on Software Process Technology,
Lecture Notes in Computer Science, vol. 913, Schäfer W
(ed.). Springer-Verlag: London, 255 – 259.

Klyne G, Carroll J (eds). 2004. World Wide Web
Consortuim (W3C). Resource Description Framework

(RDF): Concepts and Abstract Syntax W3C Recom-
mendation 10 February, Available at: http://www.
w3.org/TR/rdf-concepts/.

Kneuper R. 2002. Supporting software processes using
knowledge management. Handbook of Software Engineering
and Knowledge Engineering, Vol. II. World Scientific
Publishing Co Singapore.

Koskinen M, Martiin P. 1998. Developing a customizable
process modeling environment: lessons learnt and future
prospects. In Proceedings of the 6th European Workshop
on Software Process Technology (September 16 – 18, 1998),
Lecture Notes in Computer Science, vol. 1487, Gruhn V (ed.).
Springer-Verlag: London, 13 – 27.

Kunz W, Rittel H. 1970. Issues as Elements of Information
Systems, Working Paper No. 131. Institut für Grundlagen
der Planung, Universität Stuttgart: Germany.

Lee J. 1990. A qualitative decision management system.
In Artificial Intelligence at MIT: Expanding Frontiers, Vol.
1, Winston PH, Shellard S (eds). MIT Press: Cambridge,
MA, 104 – 133.

Lee J. 1997. Design rationale systems: understanding
the issues. IEEE Expert: Intelligent Systems and their
Applications 12(3): 78 – 85.

Lutz M. 2001. Programming Python, 2nd edn. O’Reilly &
Associates: Sebastopol, CA.

MacLean A, Young RM, Belloti V, Moran T. 1991.
Questions, options, and criteria: elements of design space
analysis. Human-Computer Interaction 6: 201 – 250.

Madhavji N. 1992. Environment evolution: the Prism
model of changes. IEEE Transactions on Software
Engineering 18(5): 380 – 392.

Manola F, Miller E (eds). 2004. World Wide Web
Consortuim (W3C). RDF Primer W3C Recommendation,
Available at: http://www.w3.org/TR/rdf-primer/.

Maurer F, Dellen B, Bendeck F, Goldmann S, Holz H,
Kötting B, Schaaf M. 2000. Merging project planning
and web-enabled dynamic workflow technologies. IEEE
Internet Computing 4(3): 65 – 74.

Merz D. 2004. XML for Word Processors, IBM Developer
Works: 25 February, Available at: http://www-
128.ibm.com/developerworks/library/x-matters33/.

Nejmeh BA, Riddle WE. 2006. The PERFECT approach
to experience-based process evolution. In Advances in
Computers, Zelkowitz M (ed.). Amsterdam, Academic
Press.

Nguyen MN, Conradi R. 1996. Towards a rigorous
approach for managing process evolution. Software

104 DOI: 10.1002/spip

Research Section Rationale Modeling for Software Process Evolution

Process Technology: 5th European Workshop, EWSPT ‘96,
Nancy.

Ocampo A, Münch J. 2006. Process evolution supported
by rationale: an empirical investigation of process
changes. In Software Process Change: International
Software Process Workshop and International Workshop on
Software Process Simulation and Modeling, SPW/ProSim
2006 – Proceedings, Qing W, Wang Q, Pfahl D, Raffo DM,
Wernick P. (eds). Springer-Verlag: Berlin, 334 – 341.

Ocampo A, Münch J. 2007. The REMIS approach for
rationale-driven process model evolution. Proceedings
of the International Conference on Software Processes.
Minneapolis.

Ocampo A, Soto M. 2007. Connecting the rationale for
changes to the evolution of a process. Proceedings: PROFES
2007 (Product Focused Software Development and Process
Improvement) Riga, Latvia.

Ponz D, Spada M. 2006. Three Years of ECSS Software
Standards: An Appraisal and Outlook. OPS-G Forum,
Available at: http://esamultimedia.esa.int/multimedia/
esoc/opsg/2006-01-20-OPS-G-Forum sw stnds.ppt. Last
checked 2007-06-02.

Ramaesh B, Dhar V. 1994. Representing and maintaining
process knowledge for large-scale systems development.
IEEE Expert, IEEE 9(2): 54 – 59.

Sauer T. 2002. Project history and decision dependencies.
Diploma Thesis. University of Kaiserslautern.

Scacchi W. 2000. Understanding software process
redesign using modeling, analysis and simulation.
Software Process Improvement and Practice 5: 183 – 195.

Seet W, Warboys B, Oquendo F. 2003. A compliant
environment for enacting evolvable process models. In
Proceedings of the 9th European Workshop on Software Process
Technology, Lecture Notes in Computer Science, vol. 2786,
Oquendo F (ed.), Springer-Verlag: Berlin 154 – 163.

Soto M, Münch J. 2006. Process model difference analysis
for supporting process evolution. Proceedings of the
13th European Conference in Software Process Improvement,
EuroSPI 2006. Springer LNCS 4257 Berlin.

Tailoring of ECSS Software Engineering Stan-
dards for Ground Segments in ESA. Available
at ftp://ftp.estec.esa.nl/pub/wm/wme/bssc/, Last
checked 2008-03-05.

Tailoring of ECSS Software Engineering Standards
for Ground Segments in ESA. Part D Trace-
ability Against ECSS, Available at: ftp://ftp.estec.
esa.nl/pub/wm/wme/bssc/SETG-D1.0 . Last chec-
ked 2008-03-05.

Weber B, Reichert M, Rinderle S, Wild W. 2005. Towards
a framework for the agile mining of business processes.
Proceedings Workshop on Business Process Intelligence (BPI),
in Conjunction with BPM 2005, Nancy.

Zelkowitz M, Wallace DR. 1998. Experimental models for
validating technology. Computer 31(5): 23 – 31.

DOI: 10.1002/spip 105

Mingers & Walsham/Toward Ethical Information Systems

THEORY AND REVIEW

TOWARD ETHICAL INFORMATION SYSTEMS:
THE CONTRIBUTION OF DISCOURSE ETHICS1

By: John Mingers
Kent Business School
University of Kent
Canterbury CT7 2PE
UNITED KINGDOM
j.mingers@kent.ac.uk

Geoff Walsham
Judge Business School
University of Cambridge
Trumpington Street
Cambridge CB2 1AG
UNITED KINGDOM
g.walsham@jbs.cam.ac.uk

Abstract

Ethics is important in the Information Systems field as illus-
trated by the direct effect of the Sarbanes-Oxley Act on the
work of IS professionals. There is a substantial literature on
ethical issues surrounding computing and information tech-
nology in the contemporary world, but much of this work is
not published nor widely cited in the mainstream IS literature.
The purpose of this paper is to offer one contribution to an
increased emphasis on ethics in the IS field. The distinctive
contribution is a focus on Habermas’s discourse ethics. After
outlining some traditional theories of ethics and morality, the
literature on IS and ethics is reviewed, and then the paper
details the development of discourse ethics. Discourse ethics

1Lynne Markus was the accepting senior editor for this paper. Matthew Lee
served as the associate editor.

is different from other approaches to ethics as it is grounded
in actual debates between those affected by decisions and
proposals. Recognizing that the theory could be considered
rather abstract, the paper discusses the need to pragmatize
discourse ethics for the IS field through, for example, the use
of existing techniques such as soft systems methodology. In
addition, the practical potential of the theory is illustrated
through a discussion of its application to specific IS topic
areas including Web 2.0, open source software, the digital
divide, and the UK biometric identity card scheme. The final
section summarizes ways in which the paper could be used in
IS research, teaching, and practice.

Keywords: Ethics and IS, ethical theories, Habermas,
discourse ethics, deliberative democracy, soft systems
methodology

Introduction

Consideration of the ethical aspects of business goes all the
way back to Adam Smith (2002 (orig. 1759)), who based his
work on strong moral foundations (Werhane and Freeman
1999). However, it can be argued that ethical issues in
business have become particularly prominent over the last
decade or so with highly publicized breaches of moral
legitimacy and trust such as in the Enron and WorldCom
scandals. More recently, the turmoil in the world’s financial
markets caused partly by the sub-prime fiasco brings ethical
issues to the forefront, such as executive reward systems
encouraging irresponsibly risky behavior.

But what is the relevance of ethics to practitioners and
academics in the field of information systems? A good illus-
tration of the importance of ethical issues in the IS field is
provided by the Sarbanes-Oxley Act (SOX) of 2002 in the

MIS Quarterly Vol. 34 No. 4 pp. 833-854/December 2010 833

Mingers & Walsham/Toward Ethical Information Systems

United States. This Act was brought in to improve corporate
governance and ethical business practices through legislation
in areas such as increased accountability and strengthened
financial control. However, many of the provisions of the Act
have a direct effect on the work of IS professionals.
Damianides (2005) noted that IS practitioners face great
challenges to meet raised expectations to provide accurate,
visible, and timely information, while ensuring the protection,
privacy, and security of their organizations’ information
assets. Hall et al. (2007) discussed the specific issue of IT
outsourcing in the context of SOX, and argued that large-scale
outsourcing may reduce oversight, weaken financial controls,
and reduce the accuracy and clarity of financial reports.
Chang et al. (2008) described the development of an auditing
information system that complies with the requirements of
SOX. It is clear that IS professionals need to understand the
ethical imperatives of SOX in order to carry out their work.

However, the relevance of ethics to the IS field is not
restricted to Sarbanes-Oxley. A wide range of ethical issues
are important to the practice of IS and thus to IS academics
also. These include codes of ethics for IS practitioners, issues
of privacy and security, combating of cybercrime, intellectual
property disputes, free and open software, hacking, and the
digital divide as a form of social exclusion. Issues such as
these are discussed in existing literature (for example, Himma
and Tavani 2008; Tavani 2007; van den Hoven and Weckert
2008), but much of this work is not published nor widely cited
in the mainstream IS literature. Indeed it can be argued that
the core IS field, based on publications in journals such as
MIS Quarterly, is underrepresentative of ethics and IS,
bearing in mind the importance of this subfield.

The purpose of this paper is to offer one contribution to an
increased focus on ethics in the mainstream IS literature.
However, the paper also aims to make an original contribution
to the literature on ethics and IT generally. This is achieved
by focusing on a relatively recent approach, namely that of
Jürgen Habermas’s discourse ethics (Habermas 1992b,
1993b). This approach stems from Habermas’s earlier work
on critical theory (Habermas 1984, 1987), but it is an
interesting development. It draws on traditional Kantian
ethical theory but it brings in other ethical approaches as well
as innovations of direct practical relevance. We will argue in
the paper that discourse ethics is a distinctive approach to
ethical theory and moral practice which has high relevance to
the field of information systems.

The rest of the paper is organized as follows. We start with
a relatively brief review of traditional theories of ethics and
morality as a common basis for understanding the later parts
of the paper, and we then review the existing literature on
ethics and IS. The major section which follows then explains

the key ideas of discourse ethics and its development into a
wider theory of deliberative democracy. The paper then
discusses the potential for applying discourse ethics within
business and information systems. However, it is recognized
that the theory could be considered too abstract and idealized
to be directly used in the IS field, and thus the next section
focuses on pragmatizing discourse ethics through, for ex-
ample, the use of existing soft and critical methodologies. In
order to further demonstrate the potential of the theoretical
approach, this section also discusses some IS application
areas including Web 2.0, open source software, the digital
divide, and the UK identity card scheme. The concluding sec-
tion summarizes the contribution of the paper and discusses
how it could be used in IS teaching, research, and practice.

Ethics and Morality

In common language, ethics and morality tend to have similar
meanings but within philosophy a distinction is drawn,
although not always strictly followed, in that morals or
morality refers to particular beliefs or norms while ethics
refers to the science or system of morals, or to a particular
ethical code (LaFollette 2007; Singer 1994; Ulrich 2008).
Ethics itself is often divided into categories, for example
meta-ethics, which deals with the most general nature of
ethical theories; normative ethics that concerns ways in which
moral conclusions should be reached; and applied ethics that
considers applications in particular contexts.

Within this context, discourse ethics can be considered as an
example of normative ethics in that it proposes procedures for
deciding on moral norms. In this section we will introduce
three general types of ethical approaches—consequentialism,
deontology, and virtue ethics and communitarianism—
although each has a degree of variety within it (Donaldson
and Werhane 1999; Pojman 1995). There are other ap-
proaches, for example the ethics of care (Gilligan 1990), but
there is general agreement (Baron et al. 1997; LaFollette
2007) that these are currently the main approaches and they
certainly cover virtually all areas of business ethics.

Consequentialism (Teleology)

One of the fundamental distinctions within ethics is whether
an act is judged in terms of intrinsic rightness or in terms of
the consequences that it has. Consequentialism holds that
correct actions are ones that maximize the overall good or
minimize the overall harm. Stemming originally from David
Hume and Adam Smith, the position was developed as
utilitarianism by Jeremy Bentham (1948 (orig. 1789)) and
John Stuart Mill (2002 (orig. 1861)). Bentham particularly

834 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

was a social reformer who wanted to move away from tradi-
tional duties and religious codes toward actions that could
genuinely help improve people’s lives, not because this was
fair or just, but simply because it would improve human
happiness. There are thus two aspects to utilitarianism: that
it is the consequences of an act that count; and that the act is
judged in terms of the degree of goodness that results.

For Bentham, goodness meant the degree of pleasure or pain
that resulted from an act and he even developed a complex
hedonic calculus to measure this. However, this provides a
rather basic view of the good life and Mill developed a more
sophisticated version that distinguished between the lower,
sensuous pleasures of the body and the higher ones of intel-
lectuality, creativity, and spirituality. Not all utilitarians
equate goodness with pleasure. Some consider things such as
knowledge, moral maturity, and friendship, while in modern
economics people’s actual, and differing, preferences can be
transformed into a measure of utility which is then to be maxi-
mized. There is also a distinction between rule utilitarians
and act utilitarians. The latter judge the individual actions of
a particular person or group while the former analyze the
results of adopting particular sets of rules on the general good.

Consequentialism seems a very obvious approach, and in
many ways accords with our commonsense (and indeed
rational decision making) approach to deciding what to do:
evaluate alternative possibilities in terms of which will have
the best consequences. However, it has many limitations
(Pettit 1997; Ross 1930; Vallentyne 2007). First is simply the
difficulty of actually predicting the consequences of an action,
particularly far into the future. In our complex modern world,
outcomes are usually the result of many, unpredictable factors
and so is it reasonable to judge an act in terms that the actor
could not have foreseen? The small boy kicks a ball into the
road, causing a car to swerve, killing a pedestrian. Is the boy
guilty of murder? Second are the problems of agreeing and
being able to measure appropriate forms of good or utility.
Third are questions of justice: utilitarianism tries to maximize
the greatest good for the greatest number and thereby risks
injustice for the minority; and it may lead to condoning
actions that by most standards would be considered wrong if
they are thought to result in a greater good—the end justifies
the means.

In terms of ethics applied in business, this approach licences
the instrumentalist view that business is primarily concerned
with making money for its stockholders (Friedman 1962;
Jensen 2002) and the more recent theories of competitive
advantage (Porter 1985; Prahalad and Hammond 2002).
Perhaps Bowen (1953) was the first in recent times to argue
systematically that businesses, because of their great power
and influence, were obliged to be socially responsible.

Deontology

Deontology (from the Greek meaning duty) shifts the judg-
ment from the consequences of an act to the act in itself.
Actions are to be seen as morally right or wrong, just or
unjust, in themselves regardless of their consequences. The
end never justifies the means. We shall consider two ap-
proaches: Kantian ethics based on the individual, and con-
tractarian ethics based on general social procedures
(McNaughton and Rawling 2007).

Kant’s (1991 (orig. 1785)) aim is to provide a general and
universal justification for moral action that is independent of
consequences or human desire. He argues that imperatives,
statements that we should or ought to do certain things, are of
two types: hypothetical and categorical. Hypothetical
imperatives are conditional, dependent on some particular
circumstances or requirement: “If you want to earn money,
get a job.” Categorical imperatives are not contingent or
qualified but apply in themselves without reservation. They
are acts which one knows intuitively to be right over and
above one’s personal inclinations on the basis of reason and
rationality. In fact, Kant suggests that there is only one
genuine categorical imperative to which all more specific
maxims of action must conform and that is

Act only on that maxim through which you can at
the same time will that it should become a universal
law (Kant 1991 (orig. 1785), p. 97).

The categorical imperative has two fundamental aspects: that
maxims for action should be based on a concern for other
people rather than purely ourselves; and that they should be
universal, that is, apply to everyone. The underlying argu-
ment for this is that most actions are done to achieve a
purpose: they are means to an end, and it is the end that is
valued. However, people may value different ends or objec-
tives differently, so can there be a universal end? Kant’s
answer was that there could be: human beings in themselves.
It is rational human beings who make value judgments and so
we need to treat other humans as equal to ourselves, as ends
and not means. This leads to a second formulation of the
categorical imperative:

Act in such a way that you always treat humanity,
whether in your own person or in the person of any
other, never simply as a means, but always at the
same time as an end (Kant 1991 (orig. 1785), p.
106).

Taken together, these two formulations can be used as a test
of maxims of action: does a maxim treat human beings as
ends in themselves? And can it be universalized to all
humans?

MIS Quarterly Vol. 34 No. 4/December 2010 835

Mingers & Walsham/Toward Ethical Information Systems

Kant’s approach has been one of the cornerstones of ethical
theory and accords well with our ideas of duty and respon-
sibility toward other people. Its foundation lies in reason and
rationality rather than tradition or religion. Criticisms of the
approach are (Baron 1997; Hursthouse 2007): first, what
justifies this view of rationality as the ultimate foundation of
moral behavior? Could we not equally appeal to religion,
community, or feelings such as care and concern? Second, we
may well find that there are many situations in which different
norms or duties may conflict with each other and the cate-
gorical imperative gives no guidance as to how we should
choose one over the other. Generally, what grounds do we
have for thinking that morals can be totally universal espe-
cially across widely differing cultures and belief systems?
Third, presaging Habermas whom we will discuss later, the
approach is fundamentally individualistic or monological, set
in terms of the subjective decisions of the individual agent.

The second deontological approach moves away from indi-
vidual acts toward sets of rules that could govern society. We
will examine two: the human or natural rights approach and
Rawlsian theory of justice. John Locke (1980 (orig. 1689))
argued that people were born with certain natural rights and
that everyone possessed these equally. Government then
consists of a social contract between people which maintains
and protects these rights, and individual and organizational
actions can be judged in terms of maintaining or abrogating
such rights. This approach was very influential in its time as
the basis for the American Constitution and more recently in
terms of the Universal Declaration of Human Rights.

John Rawls (1971) approached the idea of a social contract
from a different and novel direction. He conducted a thought
experiment in which he asked what principles of justice
rational people would choose if they had no knowledge of
their own personal characteristics or the position they would
hold in such a society. Thus, if one was behind a “veil of
ignorance” as to one’s gender, abilities or disabilities, age,
position in society, and so on, would one not choose a set of
rules that were as fair as possible to all? This, according to
Rawls, should generate a universal agreement about what
would constitute the values of a truly fair society. Rawls’
approach can be criticized from a libertarian perspective that
justice should not be concerned with the redistribution of
resources (Nozick 1974) and also from a Habermasian per-
spective that it presumes too much commonality of agreement
across communities (Habermas 1996).

Within business ethics there are theories of corporate
agency—that is, conceptualizing corporations as morally
responsible agents—that are based on human rights (Matten
and Crane 2005) and Rawlsian social contract (Donaldson and
Dunfee 1995). There are also stakeholder theories which
recognize that an organization depends for its successful

operations on a range of different groups or stakeholders and
therefore owes some duties to them. Theorists have drawn on
a range of ethical positions including Kantianism (Bowie
1999), Rawlsianism (Phillips 2003), and extreme liber-
tarianism (Freeman and Phillips 2002).

Virtue Ethics and Communitarianism

The third major approach to ethics has a very long history
dating back to Aristotle’s (2000) idea of the virtuous life, and
a modern renaissance in MacIntyre’s (1985) communi-
tarianism (Hursthouse 2007; Slote 1997).

Whereas consequentialism sees actions in terms of their
calculated outcomes and deontology sees actions in terms of
a duty to behave properly, Aristotle was concerned with
people developing ways of behaving that would naturally lead
to the well-being of both the individual and the community,
what he called a state of eudaimonia. This involves the
development of the whole person, their emotions, personality,
and moral habits, so that they “naturally” behave virtuously.
Examples of such characteristics are honesty, courage, tem-
perance, fairness, and patience. Aristotle also held to the
principle of the Golden Mean, that is, that each of the virtues
was in the middle between two extremes. Courage is between
rashness and cowardice; patience between anger and careless-
ness; and temperance between licentiousness and insensi-
bility. There is one virtue that underlies the others and that is
what Aristotle called phronesis, which can be translated as
prudence, wisdom, or judgment. It is the ability to success-
fully balance different and perhaps conflicting elements
together in a way that one only learns through experience.

These ideas of what constitutes a virtuous and good life have
been taken up by MacIntrye (1985) and Taylor (1989) as a
reaction against Rawls and the deontological tradition. In
particular, they objected to the individualistic and ahistorical
nature of human nature assumed by Rawls. MacIntyre argues
that we only become human beings through our development
and socialization within a particular community, and that we
therefore gain our ethical codes and judgments from that
community. Different communities, whether they be cultural,
ethnic, or religious, generate their own ethical practices and
standards and it is never possible to go beyond all traditions
to a universal eternal viewpoint. The “good life” must always
be relative to a particular context or community. The Aristo-
telian view has been utilized within business ethics by
Solomon (1992). While it is easy to accept that the com-
munities we grow up in will have particular ethical stances
and practices that affect us as individuals, if the communi-
tarian approach is taken strongly then it involves a relativism
that is perhaps unhelpful in today’s globalized multicultural
societies.

836 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

Ethics and Information Systems

Despite the massive effects that developments in information
and communication technologies (ICTs) are having on the
world society, there has not been a huge literature on ethics
within the mainstream of information systems journals.
However, there has been a considerable amount of work in
the areas of computer ethics and information ethics which we
consider to be of great relevance for IS.

In an overview of ethics and IS, Smith and Hasnas (1999)
reviewed literature during the 1990s in terms of the main
business ethics positions mentioned above (stockholder,
stakeholder, and social contract) to see what insights they
gave to those confronting ethical issues. They concluded that
while the number of ethical quandaries was growing signi-
ficantly, there was somewhat of a theoretical vacuum as to
how to deal with them: “Whether as managers, IS profes-
sionals or academic researchers, we ignore these ethical
dilemmas and their theoretical assessment at the risk of our
own community’s credibility” (p. 125). Prior et al. (2002)
surveyed IS professionals in the United Kingdom and found
a high level of ethical awareness but also identified many
practical problems that organizations need to address.

Within IS research there has been little written in the
literature, as Walsham (2006) notes, although in many ways
IS research is little different to other forms of research within
the social sciences. The issues mainly concern possible
effects on human subjects who are involved in the research.
Generally accepted ethical principles (Beauchamp and
Childress 1994) are non-maleficence (not harmful), benef-
icence (providing some benefit), autonomy (respecting the
individual in terms of gaining informed consent, confiden-
tiality, no deception), and justice (fair to all especially
minorities).

One research area where there has been particular interest in
ethics is that of critical IS research (Cecez-Kecmanovic et al.
2008), much of which is Habermasian and therefore of
particular interest to this paper. Critical research must inevi-
tably have a strong connection to morality since it is con-
cerned with revealing the effects of IS/IT on people in society
and reciprocally the effects of society and its interests on IS
research. Its foundation is based on justice and emancipation.
Stahl (2008b), in overviewing the area, draws two related
distinctions: that between ethics and morality, and that be-
tween the German and French critical traditions. The German
tradition is strongly based on Kant and is exemplified by
Habermas. It assumes that there are rational ways of deter-
mining moral practices and that people then have a duty to
uphold them. In the French tradition, an example being
Foucault, ethics is seen in terms of visions of the good life
while morality is rules that constrain individual behavior.

Moving to IS development, Myers and Miller (1996) con-
sidered some fundamental dilemmas in areas such as privacy
and information access from an Aristotelian perspective while
Walsham (1993) discussed ethical issues from the point of
view of the individual analyst as a moral agent, examining the
extent to which methodologies, including soft systems meth-
odology (SSM), can support this. This individualist view-
point has also been explored in the context of a postmodern
approach to ethics (Chaterjee et al. 2009). Smith and Keil
(2003) have developed a model to explain the extent to which
developers are reluctant to report bad news on a software
project. One of the few genuine attempts to create a design
methodology that embodied ethical principles was that of
Enid Mumford (Hirschheim and Klein 1994), who actually
used the acronym ETHICS for her approach. Introna (2002)
argues, rather extremely perhaps, that in fact codes,
frameworks, and moral arguments are now of little use. He
suggests that as individuals we tend to behave morally when
we feel obligations to others and that this is stronger when
they are concrete others that we name and see face to face.
However, ICTs are increasingly disembodying our inter-
actions and generating a virtual reality for us to live in which
actual names, faces, and persons become ever more
attenuated. Culnan and Williams (2009) have considered
privacy breaches in two recent cases (ChoicePoint and TJX).

There has been much more discussion of ethical issues within
computing and IT, especially in the light of IT developments,
the information age, and globalization. The discussion can
usefully be divided into foundational or meta-ethical theories
(the level at which discourse ethics would be seen to work)
and applications to particular problematic issues (Himma and
Tavani 2008).

Computer and information ethics (which we will generally
use as interchangeable terms) can arguably (Bynum 2008) be
traced to the seminal work of Norbert Weiner (1950), one of
the founders of cybernetics. He recognized that the develop-
ments within automated control systems and computing
would have profound effects on human society and tried to
deal practically with the problems that would be thrown up so
as to ensure the flourishing of human beings. The next major
step was Moor’s paper, “What Is Computer Ethics” (1985),
which established the domain of computer ethics as more than
simply another example of applied ethics. He argued that
computers were essentially a malleable, universally applicable
tool so that the potential applications, and consequent ethical
issues, were both novel and almost limitless. During the
1990s, the attention moved to professional ethics (Gotterbarn
1991; Oz 1992; Walsham 1996), trying to define codes of
ethics for the developers of computer systems in their every-
day practice, a particular example being the ACM Code of
Ethics adopted in 1992.

MIS Quarterly Vol. 34 No. 4/December 2010 837

Mingers & Walsham/Toward Ethical Information Systems

The most profound development (Tavani 2001), though, has
been the work of Luciano Floridi in proposing both a philo-
sophical theory of information (Floridi 2002b, 2005b) and a
theory of information ethics based on it (Floridi 1999, Floridi
2005a; Floridi and Sanders 2002). Traditionally, as we have
seen, it is human subjects that are the targets of ethical
theories in that it is humans who both act and are acted on. In
recent years there have been moves to widen the scope of
ethics to include living creatures (bioethics) and the natural
environment (land ethics). These theories argue that it is not
just people that have intrinsic rights but also animals and
nature. Floridi generalizes this dramatically to suggest that
information itself, what he calls the “infosphere,” also has an
intrinsic worth, and that anything that harms or diminishes
information is evil.

His theory is both complex and sophisticated and we can only
characterize it briefly in order to contrast it later with
discourse ethics. Floridi’s fundamental position is one of
“informational structural realism” (Floridi 2008), which is
committed to the existence of a mind-independent reality that
constrains our knowledge of it. Reality consists of structural
objects (not necessarily physical or observable) that are
“informational.”2 Drawing on computer science, and espe-
cially object-oriented programming, these informational
objects can be conceptualized as a combination of a data
structure (relative to the context in which the object operates)
and a set of behaviors or processes. In principle, therefore,
Floridi’s theory is a “theory of everything” in that everything
is an informational structure.

Moving to the ethical implications of this position, Floridi
(2002a) argues that, at least minimally, information objects
can be both “moral agents” (i.e., act in ways that may be
judged morally) and “moral patients” (that is, entitled to a
degree of moral value or respect). We have responsibilities
to informational objects that allow them to survive and
flourish by ensuring that we do not increase entropy. By
entropy, Floridi does not mean the traditional physics or even
information theory concept, but “any kind of destruction,
corruption, pollution and depletion of informational objects.”
(Floridi 2005a, p. 26). Thus Floridi generalizes the sphere of
morality from human beings to everything that is informa-
tional, which for Floridi is everything. It has been applied to
issues such as privacy, vandalism, and biogenetics (Floridi
1999). Clearly such an ambitious and original theory has
generated much debate, not least how competing moral claims
across such a widened target area can be resolved. Some of
these will be discussed in the section on discourse ethics, but
the interested reader is referred to the 2008 special issue of

Ethics and Information Technology (10:2-3) devoted to
Floridi’s work.

Moving to applications of ethics, the list of issues that have
been debated is long (De George 2003; Langford 1995;
Weckert and Adeny 1997): privacy and personal information,
intellectual property, globalization, challenges of the Internet,
freedom and censorship, the digital divide, and Internet
research ethics to name but a few. We will illustrate some of
these here and then discuss other particular examples (Web
2.0, open source software, the digital divide, privacy and
identity cards) in more detail later to illustrate the potential of
discourse ethics.

The first issue concerns globalization, cultural ethical
diversity, and the impact of the Internet (Capurro 2008). One
of the major ethical debates is that of pluralism versus
universalism (Ess 2008a). To what extent are moral issues
tied inextricably to particular cultures rather than being able
to be agreed on universally? Ess (2006; see also Hiruta 2006)
advocates what he calls ethical pluralism, rooted in Plato and
especially Aristotle, that tries to generate shared ethical norms
while at the same time recognizing intrinsically different
ethical traditions. Martinsons and Ma (2009) and Davison et
al. (2009) have carried out interesting surveys into the atti-
tudes to ethics among Chinese managers and IT professionals.
An element of this is the importance of dialogue between
traditions, which connects to discourse ethics.

Initially, the development of the Internet was seen optimis-
tically as offering the possibility of unfettered communication
across the world and realizing Marshall McLuhan’s idea of a
Global Village. However, as it has in fact developed, partly
in response to the cultural battles between globalization and
fundamentalism, such optimism may be misplaced (Rogerson
2004). First, it is clear that some cultures do not embrace the
values of open communication inherent in the Internet,
leading to clashes over censorship and democracy (witness
the current battle between Google and the Chinese state).
Second, the Internet can foster the development of com-
munities, or indeed individuals, that are increasingly insular
and isolated from humanity in general (Sunstein 2008;
Thelwall 2009). Third, the digital divide is exacerbating
rather than diminishing the dislocation between the developed
and developing worlds.

The second issue that we will discuss, and again a funda-
mental one, is the question of the ethical or moral aspects of
the design of IS artifacts (van den Hoven 2008). There are
two aspects: the extent to which IT systems inevitably embed
particular values which have a moral impact without that
being either deliberate or even recognized; and designing
systems that will positively embody certain values or help us2This position is quite similar to that of critical realism (Mingers 2004).

838 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

avoid moral dilemmas. The first can be illustrated by the
work of Brey (2000) on “disclosive ethics.” Brey argues that
complex technologies bring with them new moral problems
precisely because of their complexity and opaqueness for non-
experts. Because of this, technologies that appear morally
neutral may in fact embody significant normative implications
and hence there is a need for revealing and disclosing such
characteristics. Introna (2005) gives an example concerning
facial recognition systems. In such cases, one has to begin by
taking a particular value, such as privacy, and then using it
like a searchlight to identify potential problems, and then,
taking a wider view, see how the software could be rede-
signed to overcome the problems.

The second aspect can be exemplified by what has become
known as value sensitive design (VSD) (Friedman et al.
2008). This is the culmination of several strands within IS
design such as social informatics, computer-supported cooper-
ative work, and participatory design where the aim is not
simply to design an IT system that performs a task effectively,
but to design it in such a way that it explicitly embodies
particular important values in the manner in which it operates.
Examples of values that have been employed are privacy,
autonomy, universal usability, trust, and cooperation. The
developed approach employs three strategies: conceptual
investigations, where different stakeholders debate the rela-
tive importance of particular values (an obvious application
for discourse ethics); empirical investigations of the actual
context of use; and technical investigations of the extent to
which available technology could in fact support or hinder
particular values.

Discourse Ethics and Deliberative
Democracy

The literature reviewed in the previous section is substantial
in its treatment of ethical issues related to computers and
information systems. However, discourse ethics does not
receive any significant attention in this literature and we turn
now to this topic, which is the main focus of the paper.
Before describing discourse ethics in detail, some historical
background is necessary to set the approach in context.

There is a tendency nowadays to see ethics as in some ways
antithetical to business and economics, but it has not always
been so. As noted briefly in the “Introduction,” one of the
founders of modern economics, Adam Smith, whose work is
often invoked in support of the separation of market econom-
ics from ethical considerations, was actually not of that view
at all (Ulrich 2008). His first major work, The Theory of

Moral Sentiments (Smith 2002 (orig. 1759)), was a discourse
on the fundamentally moral nature of human action and this
work underpinned his more famous treatise on the economic
system —The Wealth of Nations (Smith 2008 (orig. 1776)).
His theory of moral action, which was essentially psycho-
logical, was very relevant to discourse ethics as we shall see.
He argued that we behave in moral ways that may not always
be purely in our own self-interest because of the way we want
to be seen by others. In doing that, we try to take on the
perspective or role of the other to see how our actions would
appear from their perspective. Smith termed this the
“impartial spectator” and suggested that we act in ways we
feel would gain praise from the impartial spectator even
though no actual praise would occur. “To disturb {another’s}
happiness merely because it stands in the way of our own …
is what no impartial spectator can go along with” (quoted in
Ulrich 2008, p. 51).

In the latter book, Smith mapped out the workings of an
economic system through individuals acting on their own and
in their own manner, but this was seen as an essentially moral
system; indeed it was seen from a Christian viewpoint as the
acting out of the plans of God. “{Human subjects} ought to
let themselves be guided by these ends {of God}. In this way
the market—and not the weak moral power of man—is
interpreted as the site of morality” (Ulrich 2008, p. 152).

After Smith, driven by the positivism of the 19th and early 20th

centuries, economic theory became stripped of any ethical or
moral dimension in the name of value-freedom. To the extent
that economic ethics has resurfaced it is generally in two
forms: what may be termed applied ethics or normative
ethics. The former treats economics and business as an area
for ethics to be “applied to” as a corrective to the effects of
unmitigated market forces. The latter regards self-interested
economic activity as necessary for the continuation of the
economic system and therefore intrinsically moral.

There has developed a “third way” between the two, known
as integrative economics (Ulrich 2008), that is actually based
on taking discourse ethics, as developed by both Apel (1980)
and Habermas, as fundamental to an ethical economic system.
This in turn is part of an active debate within Germany (and
therefore largely in German) about the nature of business
ethics, strands of which are also driven by a fundamentally
discursive approach (Preuss 1999). This provides a back-
ground against which to consider the potential contribution of
discourse ethics to information systems. Discourse ethics
itself is intertwined with Habermas’s more general social
theory—the theory of communicative action—so we will
begin with that.

MIS Quarterly Vol. 34 No. 4/December 2010 839

Mingers & Walsham/Toward Ethical Information Systems

Theory of Communicative Action

This will be a brief overview as it is already well described
elsewhere (Klein and Huynh 2004; Mingers 1992). The
theory of communicative action (TCA) (Habermas 1984,
1987), which developed out of the earlier theory of
knowledge-constitutive interests (Habermas 1978), argues
that the most fundamental characteristic of human beings as
a species is our ability to jointly coordinate our actions
through language and communication; and further that the
ability to communicate is grounded on the capacity to under-
stand each other. Thus the primary function of communica-
tion is the construction of understanding and then agreement
about shared activities. Humans do, of course, engage in other
activity: for example purposive instrumental action in solving
a problem or reaching a goal, or strategic action where
communication is used to achieve personal ends through some
form of deception or control. But even in this latter case,
understanding is a necessary prior condition.

Habermas, therefore, sees communication oriented toward
reaching agreement as the primary, and most common, form
of communication, and proposes that the principal means of
reaching agreement is through rational discussion and
debate—the “force of the better argument”—as opposed to
the application of power, or the dogmas of tradition or
religion. Habermas elucidates the nature of a “rational”
argument or discourse in terms of two concepts: (1) that
contentions or utterances rest on particular validity claims that
may be challenged and defended, and (2) that the process of
debate should aspire to being an “ideal speech situation.”

Whenever we actually say something, make an utterance, we
are at least implicitly making claims that may be contentious.
These validity claims are of three types, and each one points
to or refers to an aspect of the world, or rather analytically
different worlds. These three are

• Truth: concerning facts or possible states of affairs
about the material world

• Rightness: concerning valid norms of behavior in our
social world

• Sincerity (truthfulness): concerning my personal world
of feelings and intentions

In our everyday discussions and debates, disagreements and
misunderstandings develop and these lead to one or more of
the validity claims to be challenged. It is then up to the
speaker to defend the claim(s) and possibly challenge the op-
ponents. The discussion is now at a meta level to the original
conversation. In order to achieve a valid (i.e., rational) out-

come the discussion should occur in such a way that it is the
arguments themselves that win the day rather than distorting
aspects of the people involved or the social/political situation.
Such an ideal speech situation (that can only ever be a
regulative idea at which to aim) should ensure (Habermas
1990, p. 86)

• All potential speakers are allowed equal participation in
a discourse

• Everyone is allowed to
— Question any claims or assertions made by anyone
— Introduce any assertion or claim into the discourse
— Express their own attitudes, desires, or needs

• No one should be prevented by internal or external, overt
or covert coercion from exercising the above rights

Habermas argues that these are not merely conventions, but
inescapable presuppositions of rational argument itself. Thus
someone engaging in an argument without accepting the
above is either behaving strategically (deception) or is com-
mitting a performative contradiction (hypocrisy).

Habermasian critical social theory (CST) has a long, if some-
what marginalized, history within IS although his work on
discourse ethics has not yet been taken up. Perhaps the first
to draw attention to the potential of critical theory was
Mingers (1980), who contrasted it with soft systems meth-
odology (SSM). The case for CST being a foundational
philosophy for IS research, in distinction to positivism or
interpretivism, was made by Lyytinen and Klein (1985), Klein
and Lyytinen (1985), and Hirschheim and Klein (1989). This
led on to the development of CST-based research methodo-
logies (Cecez-Kecmanovic 2001; Hirschheim and Klein 1994;
Laughlin 1987; Lyytinen 1992).

There have also been a range of empirical studies carried out
from a CST perspective focusing either on the design of IS
systems or their effects. For example, in terms of IS design
and planning, Cordoba (2007; see also Cordoba and Midgely
2006) developed a critically based methodology for partici-
pative IS planning in a Columbian university; Sheffield
(2004) designed a system for GSS-enabled meetings based on
the ideal speech situation; and Ngwenyama and Lyytinen
(1997) made the case for CST as a basis for computer sup-
ported group working, a position criticized by Sharrock and
Button (1997).

In terms of the usage and effects of IS, Ngwenyama and Lee
(1997) studied the use of email as a form of communication;
Pozzebon et al. (2006) explained the prevalence of IT fads
and fashions as a result of the continual pressure for rhetorical

840 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

closure in IT negotiations; and Cecez-Kecmanovic et al.
(1999) studied web-based teaching and learning systems.

Discourse Ethics

Discourse ethics, which is somewhat badly named as we will
see, stems almost directly from TCA through considering
actions in general rather than just communications. It is
clearly Kantian in thrust, although with a very significant
reorientation, but also sweeps in, to some extent, utilitarian
and communitarian concerns.

Beginning with the traditional ethical question of how should
we act, Habermas (1993b) recognizes that such questions
occur in different contexts. We may begin with basic prag-
matic or purposive questions about the best ways to achieve
particular ends. How to earn some money? How to fix the
car? These questions often concern problems in the material
world and they may be quite complex. Their resolution may
well require information, expertise, and resources. Many of
the problems that occur within a business context are often
seen like this and in that domain they would be classed as
“hard” rather than “soft.” In terms of ethical theory, this
relates to the consequentialist approach in which actions are
judged in terms of their effects and consequences but only in
the self-interests of the actor(s) concerned.

The question might, however, be rather deeper. What if the
goals or ends to be achieved are themselves in question, or if
the means to be used raise ethical or moral issues? Here we
are concerned with the core values and the self-understanding
of a person or a community. What kind of person am I, or
what kind of group are we, that we should have these parti-
cular values and behaviors? These questions concern what
Taylor (1989) called strong preferences, to do with our being
and way of life, rather than simply weak preferences such as
tastes in food and clothes. Habermas calls these types of
questions ethical questions in contrast to the pragmatic ques-
tions discussed above and moral questions discussed below.

Within the pragmatic domain, efficacy is the test: Does the
action work? Does it have the desired effect? But within the
ethical domain, goodness or virtue is at issue. Does the action
accord with and develop the actor’s own existential identity
and self-understanding? This clearly picks up on the Aristote-
lian and communitarian positions that emphasize the impor-
tance of developing the good life within one’s community.
Although the pragmatic and the ethical have very different
concerns—the efficacious and the good—they are similar in
that they are both oriented toward the self-interests of parti-
cular individuals or groups: the question is, what is effective
or good for us? It is when one goes beyond that perspective

to consider what might be good for all that one moves into the
domain of moral questions. And this is really the focus of
discourse ethics.3

We should not expect a generally valid answer when
we ask what is good for me, or good for us, or good
for them; we must rather ask: what is equally good
for all? This “moral point of view” constitutes a
sharp but narrow spotlight, which selects from the
mass of evaluative questions those action-related
conflicts which can be resolved with reference to a
generalizable interest; these are questions of justice
(Habermas 1992a, p. 248).

So, while discourse itself applies to all three domains, the
main thrust of discourse ethics is actually moral questions,
that is, those that concern justice for all; those that transcend
the interests of any particular individual, group, nation, or
culture but that should apply equally for all people. His
approach is clearly Kantian in that he is interested in that
which is universalizable but he effects a major transition away
from the subjective thoughts or will of the individual agent (a
monological focus) toward a process of argumentation and
debate between actually existing people (a dialogical focus).
This separates discourse ethics from other approaches as
Habermas does not see this as just an analytical procedure or
thought experiment; he intends that such debates, especially
within society as a whole, should actually occur. We can see
now how discourse ethics is intimately related to TCA: the
three domains—the pragmatic, the ethical, and the moral—
correspond with the three worlds, and the whole approach is
embedded within the processes of communicative action.

How should we judge whether an action-norm is univer-
salizable? Kant’s categorical imperative is an exercise
conducted from a particular person’s viewpoint: What do
they think would be suitable for all? We need to go beyond
that and test whether such a maxim or norm can also be
accepted by all of those affected. This leads to a reformu-
lation of the categorical imperative in what Habermas calls
the discourse principle:

Only those norms can claim to be valid that meet (or
could meet) with the approval of all affected in their
capacity as participants in a practical discourse
(Habermas 1992b, p. 66).

This is a general statement about what would constitute a
valid norm and has two essential parts: that the norm must be
agreed upon or approved by all those affected, and that this

3
Indeed, Habermas accepts that it should really have been called “a discourse

theory of morality” rather than ethics (Habermas 1993a).

MIS Quarterly Vol. 34 No. 4/December 2010 841

Mingers & Walsham/Toward Ethical Information Systems

must occur through an actual process of discourse. This is
analogous to the truth of descriptive statements (Habermas
1999a). A statement is true if what it claims about the world
is in fact the case. This is a definition but it does not tell us
how to find true statements. Equally, a moral is right if all
affected have participated in a fair discussion and agreed to
it.4 But the discourse principle does not specify what such
norms might be, nor what might be the process of discourse.
The latter point is developed through a further universaliza-
tion principle, which outlines how such norms might be
arrived at:

A norm is valid when the foreseeable consequences
and side-effects of its general observance for the
interests and value-orientations of each individual
could be jointly accepted by all concerned without
coercion (Habermas 1999a, p. 42).

The point of this process is to try to generate a common will
and not just an accommodation of interests. That is, the
participants should become convinced that it is genuinely the
best way for all of them to resolve their common differences.
To this end, (1) the mention of interests and value-
orientations refers to the participants’ concerns within the
pragmatic and ethical domains respectively; (2) participants
should try and genuinely take on the perspectives and roles of
the other, and be prepared to modify their own; and (3) agree-
ment should be based, as always, on force of argument rather
than force of power.

Toward Deliberative Democracy

Habermas has always had as one of his primary concerns
politics and the nature of the state. In the 1960s, he argued
against increasing instrumentality and technocracy in
Towards a Rational Society (1971) and, in the 1970s,
analyzed the developing crisis in Western societies in
Legitimation Crisis (1976). During the 1990s, he developed
his communicative and moral theories into a powerful model
of the nature of democratic society within the post-national
and multi-cultural age (Habermas 1996, 1999b, 2001). This
has generated considerable debate within politics and legal
circles (Dryzek 2002; O’Flynn 2006; Parkinson 2006).

Societies are governed by laws and laws embody, in part,
norms of expected behavior. There is, therefore, an intimate
connection between morality, with its concern for rightness

and justice for all, and the law and its need for legitimacy.
The law also ultimately rests on the discourse principle, which
defines valid norms, but there are significant differences
between morality and law. Morality, as we have seen, is a
domain drawn narrowly to include only those norms that can
gain universal acceptance and it thereby excludes the ethical
domain of individual or community values and conceptions of
the good, and the pragmatic domain of goals and self-interest.
The law cannot do that, however. It must operate in the real
world and be able to regulate all three domains together.
Moreover, and perhaps partly because of this, the law is posi-
tive as well as normative: it can take action and apply
coercion and sanctions as well as claiming validity, whereas
the moral domain rests on individuals and their consciences
for its enactment.

These relations are illustrated in Figure 1 (Habermas 1996).
At the top is the discourse principle which then splits in
two—the moral principle and the democracy principle—
although, as can be seen, these are at different levels. The
democracy principle governs those norms that can be legally
embodied and gain the assent of all citizens through a legally
constituted legislative process. Such laws have to deal with
questions that arise in all three domains: the pragmatic, the
ethical, and the moral. Each domain involves different
reference groups and different discursive procedures. Moral
questions are governed by considerations of fairness for all
and ultimately relate to the world community. Moral norms
can be justified through the universalization principle, but
there also needs to be discourse about their application to
particular situations, the application principle. Ethical ques-
tions concern issues of self-understanding of particular com-
munities or forms of life and are highly relevant to the multi-
cultural societies that exist nowadays. Pragmatic questions
involve bargaining and negotiating fair compromises between
competing interests.

Morality and the law are thus distinct but complementary.
Morality is a domain where people agree to take on duties and
particular forms of behavior because they reach consensus
through debate that the norms are universally applicable. The
law should enshrine these norms but will also have to include
many more specific norms to deal with ethical conflicts
between different communities and pragmatic conflicts
between different interests. Habermas (1996, p. 164) envis-
ages stages through which such debates may occur. Initially,
proposals or programs for action are brought forward and
these are evaluated in generally technical terms, based on
information, knowledge, and technical expertise, an example
of the classic decisionistic approach of evaluating different
means for accepted ends. Often, however, the ends, that is the
values and interests themselves, are seen to compete and
discourse now needs to change to another level. There are
now three possibilities.

4
For Habermas, both truth and rightness are discursively vindicated but there

is a significant difference. For truth, discourse merely recognizes or signifies
that a statement is (believed to be) true with respect to an objective world.
For morality, discourse actually justifies or creates the norm as a norm within
the social world (Habermas 1999a).

842 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

Figure 1. Relationships between Principles of Discourse

First, the issues may involve moral questions, that is questions
that need to be solved in the interests of all, for example
social policies such as tax, health provision or education.

Second, they could involve ethical questions that may differ
between different communities and may not be generalizable
such as immigration policies, abortion, or the treatment of the
environment and animals.

Third, the problem may not be resolved either through general
assent or the strength of a particular value because of the
range of different communities and interests involved. In
these cases, one has to turn to bargaining rather than dis-
course. The parties involved need to come to a negotiated
agreement or accommodation rather than attain a consensus.
This is not a rational discourse (in Habermas’s terms) since
the parties involved will be acting strategically and may well
employ power, and because the parties may agree for different
reasons, whereas with a moral consensus the parties will agree
for the same reasons. Nevertheless, rationality and the dis-
course principle can be applied to the process of negotiation
if not its actual content.

Deliberative democracy can be seen to weave together a
whole variety of different forms of discourse and communi-

cation involving rational choice and the balancing of interests,
ethical debates about forms of community, moral discussion
of a just society, and political and legal argumentation. This
complexity occurs not just in the traditional institutions of
politics and the law, but increasingly in what Habermas
(1996, ch. 8) refers to as the voluntary associations of civil
society. The whole third sector of community and voluntary
groups, pressure groups, nongovernment organizations, trade
associations, and lobbyists, underpinned by the explosion of
communication technologies, now occupy the space between
the everyday communicative lifeworld, the economy and the
state. They sense and respond to issues and concerns that
arise within the public sphere and channel them into the sluice
gates of the politico-legal center. Bohman (2008) discusses
the extent to which this transformation of the public sphere
(Habermas 1989) can be facilitated by the Internet.

Applying Discourse Ethics in Business
and Information Systems

In terms of business generally, discourse ethics has been
advocated in two main ways: concerning the role of corpora-

Discourse Principle
Only those norms can claim to be valid that meet (or
could meet) with the approval of all affected in their
capacity as participants in a practical discourse

Democracy Principle
Only those statutes can claim legitimacy that can meet
with the assent of all citizens in a discursive process of
legislation that in turn has been legally constituted

Pragmatic
All involved social groups
Negotiating fair compromises
between competing interests

Ethical
“Our” community or form of life
Express authentic self-
understanding

Moral Principle
The community of world citizens
Equal consideration given to the
interests of all

Universalization Principle
Principle of argumentation
for justifying universal
norms

Application Principle
Principle of argumentation
for applying norms in
particular situations

MIS Quarterly Vol. 34 No. 4/December 2010 843

Mingers & Walsham/Toward Ethical Information Systems

tions as a whole within society, drawing on the later theory of
deliberative democracy; and also at the level of communi-
cations within organizations.

Reed (1999a, 1999b) has used discourse ethics as the basis of
a normative stakeholder theory of the firm, arguing that the
distinctions between legitimacy, morality, and ethicality
provide a more sophisticated and comprehensive approach to
dealing with the normative bases of stakeholder claims, and
that the underlying communicative theory goes beyond the
abstract notions of a Rawlsian veil of ignorance toward actual
debate and discourse and a recognition of the realities of
compromise and bargaining. Smith (2004), in part developing
from Reed’s work, argues that, increasingly, companies will
not be able to achieve their long-term strategic aims by acting
in a purely instrumental, pragmatic manner, but need to
become engaged within the moral and communicative spheres
of society as a whole. In a similar vein, Palazzo and Scherer
(2006; see also Scherer and Palazzo 2007) argue that corpora-
tions need to become politicized in the sense that they need to
become genuinely political agents within an increasing
globalized, “postnational” (Habermas 2001), world: “These
phenomena need to be embedded in a new concept of the
business firm as an economic and a political actor in market
societies” (Scherer and Palazzo 2007, p. 1115).

Moving to communicative action as such, Meisenbach (2006)
has attempted to operationalize Habermas’s universalization
principle to guide those conversations within an organization
that have a moral dimension (i.e., that potentially affect all
those within the community). Discourse ethics has also been
suggested as a basis for theorizing moral principles in deci-
sion making in organizations (Beschorner 2006; de Graaf
2006) and as a basis for ethical auditing (Garcia-Marza 2005).

If we stand back and ask ourselves what is it exactly that
discourse ethics has to offer for both business in general and
information systems in particular, then we would suggest
there are three major contributions.

• Practical discourse. Discourse ethics is unlike all other
ethical theories in that it requires actual discussion and
debate among those who may be affected by a norm or
proposal and accepts the outcome as that which is
morally correct, assuming of course that the debate was
sound. Discourse ethics is, therefore, entirely procedural;
it does not specify moral behaviors but only methods for
agreeing upon them. In this, it would seem to have the
potential for bringing about ongoing, practical resolutions
of moral and ethical concerns. It also links directly into
the shift that has occurred within IS and management
science more generally toward soft approaches such as
soft systems methodology (SSM) (Checkland and

Holwell 1998; Mingers 1980) and cognitive mapping
(Bryson et al. 2004; Eden 1995) whose purpose is struc-
turing complex problems through exploration and debate
(Rosenhead and Mingers 2001).

• Universalization. Discourse ethics distinguishes moral
issues that concern everyone involved in a particular
situation from ethical and pragmatic ones that are relative
to particular individuals or groups. It therefore pushes us
to consider, and involve, as wide a range of stakeholders
as possible in decisions and system designs. This too
links into several management science approaches that
stress the importance of boundary decisions such as
Churchman’s (1979) ethics of systems design, Ulrich’s
(2000) critical systems heuristics, Midgley’s (1997,
2000) boundary critique, and Mingers’ (1997, 2006)
critical pluralism.

• The just, the good, and the practical. Discourse ethics
is both more comprehensive, and in a particular sense
more practical, than other ethical theories in recognizing
that in the real world there are different types of issues,
and different perspectives from which to approach them.
As well as questions of justice, discourse ethics incor-
porates, to some extent, the concerns of utilitarians and
consequentialists in accepting pragmatic questions that
need to be settled through bargaining and even the exer-
cise of strategic action (the practical). It also recognizes
the concerns of communitarians in accepting that some
(ethical) questions may well not generate universal, but
only local, agreement and yet can still be the subject of
rational discourse (the good). It proposes that law is a
practice, characterized in terms of legitimacy, which has
to deal with issues in all three of these domains, and we
would argue that business (and the public sector) is
similar in that ultimately long-term effectiveness also
requires an acknowledgment of the good and the just as
well as the practical.

If these are the strengths of discourse ethics, it has to be
accepted that, as it stands, it can appear too abstract and
idealized to be directly or practically utilized within business.
So we need to consider if it can be pragmatized without
becoming entirely emasculated.

Pragmatizing Discourse Ethics for the
Information Systems Field

In talking of pragmatizing discourse ethics we mean it in the
everyday sense of making discourse ethics more useful, but
we also allude philosophically to the American pragmatists

844 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

such as Peirce (1878), who originally developed the idea of a
discourse theory of truth—the community of inquirers—
which underpinned Habermas’s (1978) epistemology. In this
section, we consider how the three potential strengths of
discourse ethics discussed above can be turned into pragmatic
approaches drawing in some cases on existing techniques and
methodologies. We also provide a number of examples of IS
application areas to illustrate the potential of discourse ethics.
These are not full case examples, which would each require
a further paper in themselves, but rather show the potentially
wide scope and relevance of the theory of discourse ethics to
the IS field. A summary of the key points in this section is
provided in Table 1.

The Discourse Process

As we have seen, discourse ethics presumes that real debates
will happen within an ideal speech situation. This is never
fully realized, although it can stand as a regulative ideal
toward which actual debates can aspire. What it leads into,
though, are methods and techniques to help realize an ideal
speech situation to the greatest extent. This is very much the
domain of SSM, other soft methods such as cognitive map-
ping, and indeed group support systems (GSS). All of these
methods share some general characteristics: recognition that
participants will have different views and stances on a
particular issue; an aim of resolving the issue by exploring
these viewpoints through discussion and debate; using a
variety of devices or transitional objects to surface and help
participants understand each others’ views; and trying to
involve a range of stakeholders and to ensure equal partici-
pation either through human or computer facilitation. We can
also point to a more IS-specific approach called joint appli-
cation development (JAD) which is a facilitated group
technique for determining systems requirements (Duggan and
Thachenkary 2004; Liou and Chen 1993).

The methods just discussed tend to be used with relatively
small groups, generally within an organizational setting, but
there are also a variety of methods that have been developed
to work with large groups (perhaps up to 2,000) of ordinary
citizens or representatives (Bryson and Anderson 2000; White
2002). Examples are nominal group technique, team syn-
tegrity (Beer 1994), open space technology (Owen 1992),
search conferencing (Emery and Purser 1996), and decision
support systems (Mathieson 2007).

All these methods represent a pragmatization of discourse
ethics in that they cannot guarantee the requirements of an
ideal speech situation, but they do in large measure have a
commonality of purpose with it. Although they have not been
developed with moral questions in mind, they are generally
used for resolving specific organizational problems or for

developing common visions, but they certainly could be used
for moral issues and, in at least one case (SSM), ethicality is
already one of the criteria that can be called upon. They also
do not guarantee consensus (i.e., agreement for the same
reasons) as would be required for pure discourse ethics but
they do encourage participants to genuinely see the world
from another’s perspective and come to an understanding
rather than merely a bargain.

In terms of direct relevance of these ideas to the IS field, some
literature has already examined the potential for the Internet
to facilitate improved discourse and debate. Heng and de
Moor (2003) described an Internet-based tool for collabora-
tive authoring that was used by an environmental group as a
new way in which to create a more equal exchange of ideas
among various stakeholders in the debate on genetically
modified food. The authors referred to Habermasian theory
of communicative action, but noted barriers to communication
in the form of entrenched power structures and limitations to
human rationality and responsibility. Unerman and Bennett
(2004) examined a web forum initiated by Shell as a means to
facilitate dialogue between stakeholders inside and outside the
organization on issues such as social and environmental
responsibility. The article referred specifically to Haber-
masian discourse ethics as a theoretical basis, but concluded
empirically that the web forum was not used by many of the
stakeholders as part of an honest attempt at reaching mutual
understanding regarding Shell’s ethical responsibilities. Van
Es et al. (2004) compared the merits of face-to-face versus
Internet-based negotiations and concluded that, although
Internet negotiation has serious restrictions, it can enhance
reflection and play down emotion, which the authors argue
are important qualities when handling complex and delicate
ethical issues. Yetim (2006) uses discourse ethical concepts
to construct a framework for evaluating genres, that is specific
communicational artifacts such as an application form, and
their role within communication. He also uses Habermas’s
more general discourse theory to develop tools for supporting
team and group discussions (Yetim 2009).

This earlier literature focused on technologies such as web
forums and collaborative authoring tools, but what about the
potential for newer technologies, sometimes captured under
the Web 2.0 label, such as wikis, blogs and social networking
tools (McAfee 2006; McKinsey Quarterly 2007)? These
newer bottom-up collaborative applications emphasize con-
tinuous intersubjective exchange aimed at generating under-
standing of the position of others and, in the case of Wiki-
pedia, for example, some achieved consensus. The extent to
which their characteristics provide better opportunities for
dialogue that seeks to arrive at genuine consensus through
engaged debate of a wide range of stakeholders is thought to
be a prime candidate for an exploration through the theoretical
basis of discourse ethics.

MIS Quarterly Vol. 34 No. 4/December 2010 845

Mingers & Walsham/Toward Ethical Information Systems

Table 1. Pragmatizing Discourse Ethics for the Information Systems Field

Approach of
Discourse Ethics

Existing Techniques and Methodologies
to Support This Approach Example IS Application Areas

The discourse
process

• Small groups—SSM, GSS, JAD
• Larger groups–nominal group technique,

team integrity, etc.

• Internet to facilitate improved discourse and debate.
• Web 2.0 applications: wikis, blogs, social networking.
• Open source software movement: accommodations

or genuine consensus?

Universalization • Critical systems heuristics
• Questioning of boundary judgments

• Digital divide: who participates?
• Digital inclusion projects: need for discourse process

between stakeholder groups

Legitimacy and
effectiveness

• SSM CATWOE
• Efficacy, ethicality, and equity

• Biometric identity cards: will they work/ are they
acceptable to particular groups; are they fair and just?

• Role of academic IS departments in discourse
process

A further IS topic area where one could argue a prima facie
case for the application of discourse ethics is the open source
software movement, which can be considered to be a rela-
tively bottom-up and consensus approach to the development
of software. For example, Lee and Cole (2003) described the
case of the Linux kernel development project that involved
thousands of talented volunteers, dispersed across organi-
zational and geographical boundaries, in collaborating via the
Internet to produce an innovative product of high quality. It
could be argued that the “ideal” discourse conditions do not
apply since contributors are not equal parties to the debate,
with an inner core of developers having strong rights of
control on implemented changes in the Linux kernel. Never-
theless, the discourse process between the stakeholders is
clearly more open and inclusive than that in closed proprietary
applications, and it would be interesting to use the formal
concepts of discourse ethics to investigate the degree to which
the discourse surrounding the Linux kernel development
conforms to an ideal type.

The “purity” of the open source model can be considered to
be comprised by the way in which it is now involves many
commercial and governmental actors as well as individual
software developers (Fitzgerald 2006). Von Hippel and von
Krogh (2003) talk of a “private–collective” model that
contains elements of both the private investment and collec-
tive action models and can thus offer society the best of both
worlds. This would certainly be challenged by proponents of
the “free software movement” (Elliott and Scacchi 2008), who
embrace an ideology and set of work practices that maintain
a clear distinction between collective work based around free
discourse and software work based around commercial
interests. This is not the place to examine such ideological
differences further, but rather we wish to argue that such
differences could indeed be critically examined under the lens
of discourse ethics, for example in differentiating between

accommodations and genuine consensus as discussed earlier
in the paper.

Universalization

A second area to be considered is the discourse ethics
requirement that for a norm to be moral it must be acceptable
to all those affected. This clearly raises major practical issues
for this would never be possible in theory let alone in practice
(Haas and Deetz 2000), although “universal” does not mean
all human beings, only those affected by a particular issue and
so could even in principle be quite restricted. It can be seen
as involving a tension between ethical contextualism and
moral universalism (Ulrich 2006). The more that discourse
aims for moral universalism, the less it will be able to justify;
the more that it accepts a narrow context, the less justified the
results will be. This could be read as a disabling contradiction
within discourse ethics, but we would prefer to see it as a
creative tension that can lead to better, and more just,
decisions.

It is also useful here to contrast discourse ethics with Floridi’s
information ethics described earlier (Stahl 2008a). There are
certainly some similarities: both are generally within the
Enlightenment tradition, which values rationality and reason
as ways of determining moral action; both share, at least in
part, a deontological approach of duties and responsibilities
toward moral patients or subjects; both recognize the impor-
tance of information, language, and communication, at least
within the human world; and both are concerned with the uni-
versality of their propositions. However, we would highlight
two major differences. The first is the extent of the univer-
salization and the second is the unique emphasis of discourse
ethics on discourse.

846 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

We might already regard Habermas as being ambitious in
aiming for moral norms that apply to all those affected, but
Floridi wants to enlarge it to all informational entities both as
agents and patients. This clearly raises significant difficulties
(Ess 2008b): to what extent is it acceptable to treat nonhuman
entities (including especially artificial agents) as either worthy
of moral treatment, or capable of it? How does one dis-
tinguish between the potential moral status of different
informational objects? Should we not recognize that it would
be the designers of artificial agents that have the moral
responsibility, not the agents themselves? In what way can
nonhuman objects articulate moral claims? If we humans are
to do it for them, are we not then in fact doing it from an
anthropocentric viewpoint?

The second difference is that Floridi’s approach has within it
no ways of arbitrating between different claims even between
humans let alone between humans and nonhuman objects.
This, as we have seen, is where discourse ethics scores over
rival ethical approaches. It is procedural, not substantive, and
once people can be drawn into a process of discourse, at
whatever level, then they have already begun to commit them-
selves to the game of rational argumentation with some hope
that it may lead to a way forward.

Again, this can be assisted by already existing methodologies.
In essence, the problem is twofold: boundaries (i.e., which
stakeholders, interpreted as anyone who could be affected,
must be included?) and representation (i.e., who can stand as
representatives for the many?). Churchman (1968) was one
of the first to say that in order to properly evaluate our designs
we have to consider the whole system of which they are a
part. This meant drawing the boundaries expansively or
“sweeping in” as many aspects of the situation as possible,
but the problem is always, where do you stop? If we assume
that the organization concerned is basically benevolent (i.e.,
it wishes to generate a just result), then we can follow Reed
(1999a) in suggesting that we need to include stakeholders
who will be affected economically, politically in terms of
equality, and ethically in terms of their self-identity.

However, it seems more likely that organizations will often
not wish to include all stakeholders, and there may indeed be
situations of outright conflict as for example between com-
panies and pressure groups, or between planners and citizens
(Palazzo and Scherer, 2006). Here we can turn to critical
systems heuristics and boundary critique. Critical systems
heuristics (Ulrich, 1994) developed from a combination of
Churchman and Habermas’s work and helps to challenge the
boundary judgments that are often made by experts or those
in power against the interests of those who are affected but
powerless. It consists of twelve critical questions, first asked

in the “is” mode and then asked in the “ought to be” mode,
that aim to reveal the partiality of the judgments that have
been made over which facts and values are relevant to the
design. This forces those in power to justify their boundary
judgments and ultimately perhaps to change them. Midgley
(2000; Cordoba and Midgely 2006) and Mingers (2006) have
also developed methods for challenging and questioning
boundary judgments.

With respect to the second aspect, representation, this is
clearly necessary in many cases, although the Internet does
offer the potential for mass participation, but is a complex
area. Representatives can be chosen in many ways—
elections, random sampling, ad hoc, convenience, etc.— and
each has its own benefits and problems. Parkinson (2006)
provides a comprehensive review within the context of
practical real-world attempts at deliberative democracy.

The concern with universalization in discourse ethics can be
related directly to debates in the IS field, and a good illus-
tration of this concerns the so-called “digital divide”
(Warschauer 2003). This involves lack of access to digital
technologies but also the social, cultural, educational, and
linguistic contexts that affect whether people are able to
utilize technologies effectively even if they have access.
Thompson (2004) provides an interesting case study of the
World Bank’s Global Development Gateway that aims to
provide a global forum for debate about development issues.
However, Thompson argues that the forum excludes the views
of many bottom-up and local development groups and can be
seen as a form of land-grab by the Bank in defining the
meaning of development and thus, ironically, a form of digital
divide in itself. The focus on universalization in discourse
ethics relates directly to Thompson’s concerns about who
participates on the Development Gateway, who are the silent
voices, and how could they be better represented.

One response to concerns such as these is to undertake
“digital inclusion” projects, designed to provide access to
particular targeted groups, but also to provide various types of
support for learning and capacity building. For example,
Madon et al. (2009) describe three such projects, namely the
Akshaya telecenter project in the state of Kerala in India
(Madon 2005), a community-based ICT project in a town in
a rural area of South Africa (Phahlamohlaka, et al. 2008) and
the efforts of various agencies on telecenter projects in the
mega-city of São Paulo in Brazil (Macadar and Reinhard
2006). The cases demonstrated a wide variety of experience
in the three cases and a complex mix of success and failure of
the digital inclusion projects, for example in whether they
were sustainable over time or whether they could be scaled to
include larger numbers of people and more communities.

MIS Quarterly Vol. 34 No. 4/December 2010 847

Mingers & Walsham/Toward Ethical Information Systems

What is the potential contribution of discourse ethics to this
important topic area? Madon et al. derived four processes of
institutionalization which they argued to be of key relevance
to digital inclusion projects in developing countries, and it is
interesting to note that two of these are strongly related to the
discourse processes between stakeholder groups. One of
these was “getting symbolic acceptance by the community,”
a process involving extensive participation, debate, and
eventual consensus within the communities that were targets
for the digital inclusion project. A second involved “enrolling
government support” and was concerned with linking of the
community and local government through discourse in order
to try to arrive at a community/government consensus on
action. While discourse ethics is not the only theoretical
schema of relevance to digital inclusion debates, we are
suggesting that it offers a potential new way to look at the
discourse processes that were argued to be of key importance
in the literature cited above.

Legitimacy and Effectiveness

A major strength of discourse ethics is its recognition of a
plurality of types of issues. Consequentialism tends to focus
on economic aspects; communitarianism and virtue ethics is
valuable but has no standpoint outside of different cultures or
religions to attempt to deal with the major divergences that
exist in today’s world; and Rawls (1991) also finds it difficult
to deal with this problem, simply assuming that different
traditions will have enough in common to generate agreement.
Discourse ethics accepts that all three types of issue exist and
can be resolved in different ways. In principle, the distinc-
tions are clear: pragmatic issues assume agreement among
those involved about goals and values; ethical issues accept
that there are genuine differences between individuals and
groups but that these can be tolerated; moral issues are those
that genuinely require the agreement of all affected.

However, in practice things are not so straightforward.
Habermas tends to assume that an issue or question will be
either pragmatic, or ethical, or moral. Or, he sees a process in
which issues that begin as pragmatic then become problema-
tized as ethical or moral. However, we think it more likely
that complex issues may well involve aspects of all three, or
that the three offer different perspectives or lenses on a
complex issue, possibly held by different stakeholders. For
instance, let us take the environment as an issue. We can see
that this must in part be a moral issue since the health of the
planet concerns all human beings. But we can also see that
peoples’ reactions to it in terms of becoming vegetarian or
going carbon neutral could be ethical issues on which indi-
viduals or groups could differ. Finally, some aspects such as

making airplane engines more efficient or improving alter-
native energy sources could be seen as purely pragmatic. But,
there could be debate over even this with some groups, such
as former President Bush’s government, trying to maintain
that the whole question is a pragmatic one that will ultimately
be solved by technology.

This means that the distinctions which seem clear in theory
are not in practice and may well require considerable practical
debate to sort them out. Habermas (1993a) and Apel (2001)
do recognize that, even when norms have been agreed, there
may be a practical problem in deciding whether or not a
particular norm applies in a given situation and they accept
the need for further debate in what they call discourses of
application (rather than justification).

Here again, SSM can be of use, not least because it contains
within it concepts that are clearly relevant to discourse ethics,
namely CATWOE5 and the 3(5)E’s.6 CATWOE is used as a
checklist and contains reference to three groups of stake-
holders: customers, who may be beneficiaries or victims of
the system; actors who carry out the system’s activities; and
the owners of a system. The 5E’s are used to monitor and
control the system’s activities and can easily be aligned with
discourse ethics:

• E1 Efficacy: does the system work and do what it is
supposed to? (Pragmatic)

• E4 Ethicality: is the system compatible with the values
of stakeholders? (Ethical)

• E6 Equity: is the system fair and just for all affected?
(Moral)

• E3 Effectiveness: does the system meet the owners’
aspirations in the long term? This has to take into
account all the other criteria in the same way that
legitimacy does for law.

There is an implied ordering between these: the overall goal
is effectiveness, the long-term success and sustainability of
the system’s owner and this obviously requires that the
designed systems actually work. However, this is not enough,
for efficacy must be subordinate to ethicality: the system
must not contravene the authentic values of those who use and
are affected by the system. And ethicality, in turn, is subordi-
nate to morality: individual or group visions of the good life
cannot override what is fair and just for all (Habermas 1992b).

5
Customers, Actors, Transformation, Weltanschauung, Owner, Environment.

6
Efficacy, Efficiency, Effectiveness, sometimes augmented by Ethicality and

Aesthetics. We have added in Equity.

848 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

In any real-world situation there will be a complex interplay
of pragmatic, ethical, and moral issues, and probably differing
viewpoints about them. It is tempting for managers to stick
with the efficacy question and concentrate on developing
systems that at least work, for that is often difficult enough
(witness the many, public IS failures). But, we would argue,
long-term effectiveness and sustainability require that respon-
sible managers engage with the moral and ethical issues as
well. Discourse ethics provides a rigorously justified proce-
dural framework for this task, although one that needs to be
adopted in a pragmatic fashion, aided by well-tested method-
ologies for shaping and facilitating discourse.

A specific IS example involving issues of efficacy, ethicality,
and equity is the proposed introduction of biometric identity
cards in the United Kingdom (Whitley et al. 2007). There are
major pragmatic concerns as to whether these will “work” in
technical terms. Whitley and Hosein (2008) note that, by
choosing a high-tech solution, drawing on the state of the art
in biometric technologies, the scheme is also high-risk. How-
ever, there are also important ethical concerns as to whether
the identity cards are acceptable to a wide range of individuals
and stakeholder groups, for example by increasing the possi-
bilities for identity theft. The scheme is targeted to reduce
identity theft, by providing a unique individual identifier but,
ironically, this could have the effect of increasing it, because
the identity thief could obtain all the necessary information in
one central place.

In addition, from the perspective of discourse ethics, a third
question is whether the identity card project will be fair and
just for all those affected. There are major concerns that the
system involves a centralized register that could be used to
analyze and target specific individuals or groups in a discrim-
inatory way. Generally, is the project a step too far in central
government surveillance and the invasion of personal privacy
(Whitley 2009)? We are not trying to answer these questions
here, but merely to argue that discourse ethics offers a
potentially valuable theoretical framework in investigating
complex socio-technical systems such as the UK identity card
project from the interlinked perspectives of efficacy, ethi-
cality, and equity.

The identity card project can also be used to illustrate a
further theme of relevance to the IS community and to
discourse ethics, namely the role of academic IS departments
in policy debates about such projects. Extensive work,
including that referenced above, has been carried out by staff
at the London School of Economics, and they have clashed
with the UK government on a number of occasions over the
issue (Whitley et al. 2007). If we adopt the principles of the
need for discourse between stakeholder groups, the impor-

tance of universalization, and the pursuit of the just, the good,
and the practical, then the participation of the LSE in the
debate should surely be welcomed. Yet Whitley et al. argue
that a moral of their story is that universities should only
contemplate undertaking such policy research if their govern-
ing body is willing to stand full-square behind its academics,
and to resist all forms of political pressure.

Conclusions

We have argued in this paper that ethics is important for the
practice of IS in the contemporary world, and thus is also
highly relevant for IS research and teaching. The distinctive
contribution of this paper is the first analysis of the potential
of discourse ethics for IS in the mainstream IS literature. We
have discussed how the theory could be applied, or pragmat-
ized, to a range of IS topic areas including Web 2.0 applica-
tions, open source software, the digital divide, and the UK
biometric identity card scheme. A detailed analysis of each
of these areas through the lens of discourse ethics would
require at least a full paper for each area, and this is beyond
the scope of the current paper. However, we believe that we
have provided a good starting point for such analyses in the
future through our description of the key elements of dis-
course ethics and some indicative material on the distinctive
contribution which it has to offer to such topic areas.

If ethical issues are important for the IS field as a whole, then
they are also important in terms of IS teaching. We have
noted earlier in the paper that much of the literature on ethical
issues with respect to IT/IS is located outside the mainstream
IS journals. We have reviewed some of this literature in the
current paper and we hope, therefore, that this can provide a
valuable starting point for IS teaching on ethical issues. In
addition, we have articulated the particular contribution of
discourse ethics, and we have argued that it has high potential
as a way of exploring contemporary IS issues. Our paper
could be used as a way into the literature on discourse ethics
which would be readily accessible to IS students, since the
paper is oriented specifically to the IS field.

Finally, what about IS practice? We started the paper with a
reference to the Sarbanes-Oxley Act as an example of the
contemporary relevance of ethics for IS practitioners. Al-
though discourse ethics could be considered somewhat
abstract and idealized, we have discussed in the paper how it
could be pragmatized for practical application. For example,
we have shown how existing techniques and methodologies
such as soft systems methodology, critical system heuristics,
and the three dimensions of efficacy, ethicality, and equity

MIS Quarterly Vol. 34 No. 4/December 2010 849

Mingers & Walsham/Toward Ethical Information Systems

can be used to support practical analyses of particular ethical
topic areas. We believe that our paper provides one way
forward for the investigation of ethical issues in IS practice
and, generally, provides a contribution to raising the profile
of ethics in the IS field as a whole.

References

Apel, K. O. 1980. Towards a Transformation of Philosophy,
London: Routledge Kegan & Paul.

Apel, K. O. 2001. The Response of Discourse Ethics, Leuven,
Belgium: Peeters.

Aristotle. 2000. Nicomachean Ethics, Cambridge, UK: Cambridge
University Press.

Baron, M. 1997. “Kantian Ethics,” in Three methods of Ethics: A
Debate, M. Baron, P. Pettit, and M. Slote (eds.), Oxford, UK:
Blackwell, pp. 3-91.

Baron, M., Pettit, P., and Slote, M. 1997. Three Methods of Ethics:
A Debate, Oxford, UK: Blackwell.

Beauchamp, T., and Childress, J. 1994. Principles of Biomedical
Ethics, Oxford, UK: Oxford University Press.

Beer, S. 1994. Beyond Dispute: The Invention of Team Syntegrity,
Chichester, UK: Wiley.

Bentham, J. 1948 (orig. 1789). Introduction to the Principles of
Morals and Legislation, Oxford, UK: Oxford University Press.

Beschorner, T. 2006. “Ethical Theory and Business Practices: The
Case of Discourse Ethics,” Journal of Business Ethics (66), pp.
127-139.

Bohman, J. 2008. “The Transformation of the Public Sphere:
Political Authority, Communicative Freedom and Internet
Oolitics,” in Information Technology and Moral Philosophy,
J. Hoven and J. Weckert (eds.), Cambridge, UK: Cambridge
University Press, pp. 67-92.

Bowen, H. 1953. Social Responsibilities of the Businessman, New
York: Harper.

Bowie, N. 1999. Business Ethics: A Kantian Perspective, Oxford,
UK: Blackwell.

Brey, P. 2000. “Method in Computer Ethics: Towards a Multi-
Level Interdisciplinary Approach,” Ethics and Information
Technology (2), pp. 125-129.

Bryson, J., Ackermann, F., Eden, C., and Finn, D. 2004. Visible
Thinking: Unlocking Causal Mapping for Practical Business
Results, Chichester, UK: Wiley.

Bryson, J., and Anderson, S. 2000. “Applying Large-Group Inter-
action Methods in the Planning and Implementation of Major
Change Efforts,” Public Administration Review (60:2), pp.
143-162.

Bynum, T. 2008. “Milestones in the History of Information and
Computer Ethics,” in The Handbook of Information and
Computer Ethics, K. Himma and H. Tavani (eds.), Hoboken, NJ:
Wiley, pp. 25-48.

Capurro, R. 2008. “Intercultural Information Ethics,” in The
Handbook of Information and Computer Ethics, K. Himma and
H. Tavani (eds.), Hoboken, NJ: Wiley, pp. 639-665.

Cecez-Kecmanovic, D. 2001. “Doing Critical IS Research: The
Question of Methodology,” in Qualitative Research in IS: Issues
and Trends, E. Trauth (ed.), Hershey, PA: Idea Group
Publishing, pp. 158-180.

Cecez-Kecmanovic, D., Klein, H., and Brooke, C. 2008. “Ex-
ploring the Critical Agenda in Information Systems Research,”
Information Systems Journal (18), pp. 123-135.

Cecez-Kecmanovic, D., Webb, C., and Taylor, P. 1999. “Being
Flexible by Being WISE: Two Case Studies of Web-Based
Teaching and Learning,” in Managing Information Technology
Resources in Organizations in the Next Millenium, M.
Khosrowpour (ed.), Hershey, PA: Idea Group Publishing, pp.
730-734.

Chang, S., Wu, C.-C., and Chang, I.-C. 2008. “The Development
of a Computer Auditing System Sufficient for Sarbanes-Oxley
Section 404: A Study on the Purchasing and Expenditure Cycle
of the ERP System,” Information Systems Management (25:3),
pp. 211-229.

Chaterjee, S., Sarker, S., and Fuller, M. 2009. “Ethical Information
Systems Development: A Baumanian Postmodernist Perspec-
tive,” Journal of the Association for Information Systems (10:11),
pp. 787-815.

Checkland, P., and Holwell, S. 1998. Information, Systems and
Information Systems: Making Sense of the Field, Chichester,
UK: Wiley.

Churchman, C. W. 1968. Challenge to Reason, New York:
McGraw Hill.

Churchman, C. W. 1979. The Systems Approach and its Enemies,
New York: Basic Books.

Cordoba, J.-R. 2007. “Developing Inclusion and Critical Reflection
in Information Systems Planning,” Organization (14:6), pp.
909-927.

Cordoba, J.-R., and Midgely, G. 2006. “Broadening the Boun-
daries: An Application of Critical Systems Thinking to IS
Planning in Columbia,” Journal Operational Research Society
(57), pp. 1064-1080.

Culnan, M. J., and Williams, C. C. 2009. “How Ethics Can
Enhance Organizational Privacy: Lessons from the ChoicePoint
and TJX Data Breaches,” MIS Quarterly (33:4), pp. 673-687.

Damianides, M. 2005. “Sarbanes-Oxley and IT Governance: New
Guidelines on IT Control and Compliance,” Information Systems
Management (22:1), pp. 77-85.

Davison, R., Martinsons, M., Ou, C., Murata, K., Drummond, D.,
Li, Y., and Lo, H. 2009. “The Ethics of IT Professionals in
Japan and China,” Journal of the Association for Information
Systems (10:11), pp. 834-859.

De George, R. 2003. The Ethics of Information Technology and
Business, Oxford, UK: Blackwell.

de Graaf, G. 2006. “Discourse and Descriptive Business Ethics,”
Business Ethics: A European Review (15:3), pp. 246-258.

Donaldson, T., and Dunfee, T. W. 1995. “Integrative Social
Contracts Theory: A Communitarian Conception of Economic
Ethics,” Economics and Philosophy (11:1), pp. 85-112.

Donaldson, T., and Werhane, P. (eds.). 1999. Ethical Issues in
Business: A Philosophical Approach, Upper Saddle River, NJ:
Prentice Hall.

850 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

Dryzek, J. 2002. Deliberative Democracy and Beyond: Liberals,
Critics, Contestations, Oxford, UK: Oxford University Press.

Duggan, E. W., and Thachenkary, C. S. 2004. “Integrating
Nominal Group Technique and Joint Application Development
for Improved Systems Requirements Determination,” Infor-
mation & Management (41:4), pp. 399-411.

Eden, C. 1995. “On Evaluating the Performance of ‘Wide-Band’
GDSS’s,” European Journal of Operational Research (81), pp.
302-311.

Elliott, M., and Scacchi, W. 2008. “Mobilization of Software
Developers: The Free Ssoftware Movement,”“ Information
Technology & People (21), pp. 4-33.

Emery, M., and Purser, S. 1996. The Search Conference, San
Francisco: Jossey Bass.

Ess, C. 2006. “Ethical Pluralism and Global Information Ethics,”
Ethics and Information Technology (8), pp. 215-226.

Ess, C. 2008a. “Culture and Global Networks: Hope for a Global
Ethics?,” in Information Technology and Moral Philosophy,
J. Hoven and J. Weckert (eds.), Cambridge, UK: Cambridge
University Press, pp. 195-225.

Ess, C. 2008b. “Luciano Floridi’s Philosophy of Information and
Information Ethics: Critical Reflections and the State of the Art,”
Ethics and Information Technology (10:2), pp. 89-96.

Fitzgerald, B. 2006. “The Transformation of Open Source
Software,” MIS Quarterly (30:3), pp. 587-598.

Floridi, L. 1999. “Information Ethics: On the Philosophical
Foundation of Computer Ethics,” Ethics and Information
Technology (1), pp. 37-56.

Floridi, L. 2002a. “On the Intrinsic Value of Information Objects
and the Infosphere,” Ethics and Information Technology (4), pp.
287-304.

Floridi, L. 2002b. “What Is the Philosophy of Information?,”
Metaphilosophy (33:1-2), pp. 123-145.

Floridi, L. 2005a. “Information Ethics, its Nature and Scope,”
SIGCAS Computers & Society (35:2), pp. 3-3.

Floridi, L. 2005b. “Is Semantic Information Meaningful Data?,”
Philosophy and Phenomenological Research (70:2), pp. 351-370.

Floridi, L. 2008. “A Defence of Informational Structural Realism,”
Synthese (161:2), pp. 219-253.

Floridi, L., and Sanders, J. 2002. “Mapping the Foundational
Debate in Computer Ethics,” Ethics and Information Technology
(4), pp. 1-9.

Freeman, R., and Phillips, R. 2002. “Stakeholder Theory: A Liber-
tarian Defence,” Business Ethics Quarterly (12:3), pp. 331-349.

Friedman, B., Kahn Jr., P., and Borning, A. 2008. “Value Sensitive
Design and Information Systems,” in The Handbook of Infor-
mation and Computer Ethics, K. Himma and H. Tavani (eds.),
Hoboken, NJ: Wiley, pp. 69-102.

Friedman, M. 1962. Capitalism and Freedom, Chicago: University
of Chicago Press.

Garcia-Marza, D. 2005. “Trust and Dialogue: Theoretical
Approaches to Ethics Auditing,” Journal of Business Ethics (57),
pp. 209-219.

Gilligan, C. 1990. In a Different Voice: Psychological Theory and
Women’s Development, Boston: Harvard University Press.

Gotterbarn, D. 1991. “Computer Ethics: Responsibility Regained,”
National Forum: The Phi Beta Kappa Journal (71), pp. 26-31.

Haas, T., and Deetz, S. 2000. “Between the Generalized and the
Concrete Other: Approaching Organizational Ethics from
Feminist Perspectives,” in Rethinking Organizational and
Managerial Communication from Feminist Perspectives,
P. Buzzanell (ed.), Thousand Oaks, CA: Sage, pp. 24-46.

Habermas, J. 1971. Towards a Rational Society, London:
Heinemann.

Habermas, J. 1976. Legitimation Crisis, London: Heinemann.
Habermas, J. 1978. Knowledge and Human Interests, London:

Heinemann.
Habermas, J. 1984. The Theory of Communicative Action Vol. 1:

Reason and the Rationalization of Society, London: Heinemann.
Habermas, J. 1987. The Theory of Communicative Action Vol. 2:

Lifeworld and System: A Critique of Functionalist Reason,
Oxford, UK: Polity Press.

Habermas, J. 1989. The Structural Transformation of the Public
Sphere, Cambridge, MA: MIT Press.

Habermas, J. 1990. Moral Consciousness and Communicative
Action, Cambridge, UK: Polity Press.

Habermas, J. 1992a. “Discourse Ethics, Law and Sittlichkeit,” in
Autonomy and Solidarity : Interviews with Jürgen Habermas,
P. Dews (ed.), London: Verso, pp. 245-271.

Habermas, J. 1992b. “Discourse Ethics: Notes on a Programme of
Philosophical Justification,” in Moral Consciousness and
Communicative Action, J. Habermas (ed.), Cambridge, UK:
Polity Press, pp. 43-115.

Habermas, J. 1993a. Justification and Application, Cambridge,
UK: Polity Press.

Habermas, J. 1993b. “On the Pragmatic, the Ethical, and the Moral
Employments of Practical Reason,” in Justification and Applica-
tion, J. Habermas (ed.), Cambridge, UK: Polity Press, pp. 1-17.

Habermas, J. 1996. Between Facts and Norms, Cambridge, UK:
Polity Press.

Habermas, J. 1999a. “A Genealogical Analysis of the Cognitive
Content of Morality,” in The Inclusion of the Other, Cambridge,
UK: Polity Press, pp. 3-48.

Habermas, J. 1999b. The Inclusion of the Other, Cambridge, UK:
Polity Press.

Habermas, J. 2001. The Postnational Constellation: Political
Essays, Cambridge, UK: Polity Press.

Hall, J., Liedtka, S., Gupta, P., Liedtka, J., and Tompkins, S. 2007.
“The Sarbanes-Oxley Act: Implications for Large-Scale IT
Outsourcing,” Communications of the ACM (50:3), pp. 95-100.

Heng, M., and de Moor, A. 2003. “From Habermas’s Communi-
cative Theory to Practice on the Internet,” Information Systems
Journal (13), pp. 331-352.

Himma, K., and Tavani, H. (eds.). 2008. The Handbook of Infor-
mation and Computer Ethics, Hoboken, NJ: Wiley.

Hirschheim, R., and Klein, H. K. 1989. “Four Paradigms of Infor-
mation Systems Development,” Communications of the ACM
(32:10), pp. 1199-1216.

Hirschheim, R., and Klein, H. K. 1994. “Realizing Emancipatory
Principles in Information Systems Development: The Case for
ETHICS,” MIS Quarterly (18:1), pp. 83-109.

MIS Quarterly Vol. 34 No. 4/December 2010 851

Mingers & Walsham/Toward Ethical Information Systems

Hiruta, K. 2006. “What Pluralism, Why Pluralism, and How? A
Response to Charles Ess,” Ethics and Information Technology
(8), pp. 227-236.

Hursthouse, R. 2007. “Virtue Theory,” in Ethics in Practice,
H. LaFollette (ed.), Oxford, UK: Blackwell, pp. 45-61.

Introna, L. 2002. “The (Im)possibility of Ethics in the Information
Age,” Information and Organization (12), pp. 71-84.

Introna, L. 2005. “Disclosive Ethics and Information Technology:
Disclosing Facial Recognition Systems,” Ethics and Information
Technology (7), pp. 75-86.

Jensen, M. 2002. “Value Maximization, Stakeholder Theory and
the Corporate Objective Function,” Business Ethics Quarterly
(12:2), pp. 235-256.

Kant, I. 1991 (orig. 1785). Groundwork of the Metaphysics of
Metaphysics of Morals, London: Routledge.

Klein, H., and Huynh, M. 2004. “The Critical Social Theory of
Jurgen Habermas and its Implications for IS Research,” in Social
Theory and Philosophy for Information Systems, J. Mingers and
L. Willcocks (eds.), Chichester, UK: Wiley, pp. 157-237.

Klein, K., and Lyytinen, K. 1985. “The Poverty of Scientism in
Information Systems,” in Research Methods in Information
Systems, E. Mumford, R. Hirschheim, G. Fitzgerald, and T.
Wood-Harper (eds.), Amsterdam: North Holland, pp. 131-162.

LaFollette, H. (ed.). 2007. Ethics in Practice, Oxford, UK:
Blackwell.

Langford, D. 1995. Practical Computer Ethics, London: McGraw-
Hill.

Laughlin, R. 1987. “Accounting Systems in Organizational Con-
texts: A Case for Critical Theory,” Accounting, Organizations
and Society (12:5), pp. 479-502.

Lee, G., and Cole, R. 2003. “From a Firm-Based to a Community-
Based Model of Knowledge Creation: The Case of the Linux
Kernel Development,” Organization Science (14:6), pp. 633-649.

Liou, Y., and Chen, M. 1993. “Using Group Support Systems and
Joint Application Development for Requirements Specification,”
Journal of Management Information Systems (10:3), pp. 25-41.

Locke, J. 1980 (orig. 1689). Second Treatise of Government
Cambridge, UK: Hackett Publishing Co.

Lyytinen, K. 1992. “Information Systems and Critical Theory,” in
Critical Management Studies, M. Alvesson and H. Willmott
(eds.), London: Sage, pp. 159-180.

Lyytinen, K., and Klein, H. 1985. “The Critical Theory of Jurgen
Habermas as a Basis for a Theory of Information Systems,” in
Research Methods in Information Systems, E. Mumford, R.
Hirschheim, G. Fitzgerald and T. Wood-Harper (eds.), Amster-
dam: North Holland, pp. 219-236.

Macadar, M., and Reinhard, N. 2006. “Governance and Manage-
ment in the São Paulo Telecentre Network,” Information
Technology for Development (12), pp. 241-246.

MacIntyre, A. 1985. After Virtue, London: Duckworth.
Madon, S. 2005. “Governance Lessons from the Experience of

Telecentres in Kerala,” European Journal of Information Systems
(14), pp. 401-416.

Madon, S., Reinhard, N., and Roode, D. 2009. “Digital Inclusion
Projects in Developing Countries: Processes of Institutiona-

lization,” Information Technology for Development (15:2), pp.
95-107.

Martinsons, M., and Ma, D. 2009. “Sub-Cultural Differences in
Information Ethics across China: Focus On Chinese Manage-
ment Generation Gaps,” Journal of the Association for Infor-
mation Systems (10:11), pp. 816-833.

Mathieson, K. 2007. “Towards a Decision Science of Ethical
Decision Support,” Journal of Business Ethics (76), pp. 269-292.

Matten, D., and Crane, A. 2005. “Corporate Citizenship: Toward
an Extended Theoretical Conceptualization,” Academy of
Management Journal (30), pp. 166-179.

McAfee, A. 2006. “Enterprise 2.0: The Dawn of Emergent Collab-
oration,” MIT Sloan Management Review (47:3), pp. 21-28.

McKinsey Quarterly. 2007. “How Businesses Are Using Web 2.0:
A McKinsey Global Survey,” March (available online at
http://www.mckinseyquarterly.com/How_businesses_are_usin
g_Web_20_A_McKinsey_Global_Survey_1913).

McNaughton, D., and Rawling, P. 2007. “Deontology,” in Ethics
in Practice, H. LaFollette (ed.), Oxford, UK: Blackwell, pp.
31-44.

Meisenbach, R. 2006. “Habermas’s Discourse Ethics and Principle
of Universalization as a Moral Framework for Organization
Communication,” Management Communication Quarterly (20:1),
pp. 39-62.

Midgley, G. 2000. Systemic Intervention: Philosophy, Methodo-
logy, and Practice, New York: Kluwer Academic/Plenum
Publishers.

Mill, J. S. 2002 (orig. 1861). Utilitarianism, New York: Hackett
Publishing Co.

Mingers, J. 1980. “Towards an Appropriate Social Theory for
Applied Systems Thinking: Critical Theory and Soft Systems
Methodology,” Journal Applied Systems Analysis (7), pp. 41-50.

Mingers, J. 1992. “Recent Developments in Critical Management
Science,” Journal of the Operational Research Society ( 43:1),
pp. 1-10.

Mingers, J. 1997. “Towards Critical Pluralism,” in Multi-
methodology: Theory and Practice of Combining Management
Science Methodologies, J. Mingers and A. Gill (eds.), Chichester,
UK: Wiley, pp. 407-440.

Mingers, J. 2004. “Re-establishing the Real: Critical Realism and
Information Systems Research,” in Social Theory and Philo-
sophical for Information Systems, J. Mingers and L. Willcocks
(eds.), London: Wiley, pp. 372-406.

Mingers, J. 2006. Realising Systems Thinking: Knowledge and
Action in Management Science, New York: Springer.

Moor, J. 1985. “What Is Computer Ethics?,” Metaphilosophy
(16:4), pp. 266-275.

Mumford, E. 1995. Effective Systems Design and Requirements
Analysis: The ETHICS Approach ,London: Macmillan.

Myers, M., and Miller, L. 1996. “Ethical Dilemmas in the Use of
Information Technology: An Aristotelian Perspective,” Ethics
and Behavior (6:2), pp. 153-160.

Ngwenyama, O., and Lee, A. 1997. “Communication Richness in
Electronic Mail: Critical Social Theory and the Contextuality of
Meaning,” MIS Quarterly (21:2), pp. 145-167.

852 MIS Quarterly Vol. 34 No. 4/December 2010

Mingers & Walsham/Toward Ethical Information Systems

Ngwenyama, O., and Lyytinen, K. 1997. “Groupware Environ-
ments as Action Constitutive Resources: A Social Action Frame-
work for Analyzing Groupware Technologies,” Computer Sup-
ported Cooperative Work: The Journal of Collaborative
Computing (6), pp. 71-93.

Nozick, R. 1974. Anarchy, State and Utopia, New York: Basic
Books.

O’Flynn, I. 2006. Deliberative Democracy and Divided Societies,
Edinburgh: Edinburgh University Press.

Owen, H. 1992. Open Space Technology: A User’s Guide,
Potomac, MD: Abbott Publishing.

Oz, E. 1992. “Ethical Standards for Information Systems Profes-
sionals: A Case for a Unified Code,” MIS Quarterly (16:4), pp.
423-433.

Palazzo, G., and Scherer, A. 2006. “Corporate Legitimacy as
Deliberation: A Communicative Framework,” Journal of
Business Ethics (66), pp. 71-88.

Parkinson, J. 2006. Deliberating in the Real World: Problems of
Legitimacy in Deliberative Democracy, Oxford, UK: Oxford
University Press.

Peirce, C. 1878. “How to Make Our Ideas Clear,” Popular Science
Monthly (12), pp. 286-302.

Pettit, P. 1997. “The Consequentialist Perspective,” in Three
Methods of Ethics: A Debate, M. Baron, P. Pettit, and M. Slote
(eds.), Oxford, UK: Blackwell, pp. 92-174.

Phahlamohlaka, J., Braun, J., Romijn, H., and Roode, D. (eds.).
2008. Community-Driven Projects: Reflections on a Success
Story: A Case of Science Education and Information Technology
in South Africa , Pretoria, South Africa: Van Schaik.

Phillips, R. 2003. “Stakeholder Legitimacy,” Business Ethics
Quarterly (13:1), pp. 25-41.

Pojman, L. 1995. Ethics: Discovering Right and Wrong, Belmont,
CA: Wadsworth.

Porter, M. 1985. Competitive Advantage: Creating and Sustaining
Superior Advantage, New York: Free Press.

Pozzebon, M., Titah, R., and Pinsonneault, A. 2006. “Combining
Social Shaping of Technology and Communicative Action
Theory for Understanding Rhetorical Closure in IT,” Information
Technology and People (19:3), pp. 244-271.

Prahalad, C., and Hammond, A. 2002. “Serving the World’s Poor,
Profitably,” Harvard Business Review (80:9), pp. 48-58.

Preuss, L. 1999. “Ethical Theories in German Business Ethics
Research,” Journal of Business Ethics (18), pp. 407-419.

Prior, M., Rogerson, S., and Fairweather, B. 2002. “The Ethical
Attitudes of Information Systems Professionals: Outcomes of an
Initial Survey,” Telematics and Informatics (19), pp. 21-36.

Rawls, J. 1971. A Theory of Justice, Boston: Harvard University
Press.

Reed, D. 1999a. “Stakeholder Management Theory: A Critical
Theory Perspective,” Business Ethics Quarterly (9:3), pp.
453-483.

Reed, D. 1999b. “Three Realms of Corporate Responsibility:
Distinguishing Legitimacy, Morality and Ethics,” Journal of
Business Ethics (21), pp. 23-35.

Rogerson, S. 2004. “The Virtual World: A Tension between
Global Reach and Local Sensitivity,” International Journal of
Information Ethics (2), pp. 1-7.

Rosenhead, J., and Mingers, J. (eds.). 2001. Rational Analysis for
a Problematic World Revisited, Chichester, UK: Wiley.

Ross, W. 1930. The Right and the Good, Oxford, UK: Oxford
University Press.

Scherer, A., and Palazzo, G. 2007. “Towards a Political Concep-
tion of Corporate Responsibility: Business and Society Seen
from a Habermasian Perspective,” Academy of Management
Review (32:4), pp. 1096-1120.

Sharrock, W., and Button, G. 1997. “On the Relevance of
Habermas’ Theory of Communicative Action for CSCW,” Com-
puter Supported Cooperative Work: The Journal of Collabora-
tive Computing (6), pp. 369-389.

Sheffield, J. 2004. “The Design of GSS-Enabled Interventions: A
Habermasian Perspective,” Group Decision and Negotiation
(13:5), pp. 415-435.

Singer, P. (ed.). 1994. Ethics, Oxford, UK: Oxford University
Press.

Slote, M. 1997. “Virtue Ethics,” in Three Methods of Ethics: A
Debate, M. Baron, P. Pettit and M. Slote (eds.), Oxford, UK:
Blackwell, pp. 175-238.

Smith, A. 2002 (orig. 1759). The Theory of Moral Sentiments,
Cambridge, UK: Cambridge University Press.

Smith, A. 2008 (orig. 1776). An Inquiry into the Nature and
Causes of the Wealth of Nations, Oxford, UK: Oxford
Paperbacks.

Smith, H. J., and Hasnas, J. 1999. “Ethics and Information
Systems: The Corporate Domain,” MIS Quarterly (23:1), pp.
109-127.

Smith, H. J., and Keil, M. 2003. “The Reluctance to Report Bad
News on Troubled Software Projects: A Theoretical Model,”
Information Systems Journal (13), pp. 69-95.

Smith, J. D. 2004. “A Precis of a Communicative Theory of the
Firm,” Business Ethics: A European Review (13:4), pp. 317-331.

Solomon, R. 1992. “Corporate Roles, Personal Virtues: An
Aristotelian Approach to Business Ethics,” Business Ethics
Quarterly (2:3), pp. 317-340.

Stahl, B. 2008a. “Discourses on Information Ethics: The Claim to
Universality,” Ethics and Information Technology (10), pp.
97-108.

Stahl, B. 2008b. “The Ethical Nature of Critical Research in
Information Systems,” Information Systems Journal (18), pp.
137-163.

Sunstein, C. 2008. “Democracy and the Internet,” in Information
Technology and Moral Philosophy, J. Hoven and J. Weckert
(eds.), Cambridge, UK: Cambridge University Press, pp. 93-110.

Tavani, H. 2001. “The State of Computer Ethics as a Philosophical
Field of Enquiry: Some Contemporary Perspectives, Future
Projections, and Current Resources,” Ethics and Information
Technology (3), pp. 97-108.

Tavani, H. 2007. Ethics and Technology: Ethical Issues in an Age
of Information and Communication Technology, Hoboken, NJ:
Wiley.

MIS Quarterly Vol. 34 No. 4/December 2010 853

Mingers & Walsham/Toward Ethical Information Systems

Taylor, C. 1989. The Sources of the Self: The Making of the
Modern Identity, Cambridge, MA: Harvard University Press.

Thelwall, M. 2009. “Homophily in MySpace,” Journal of the
American Society for Information Science and Technology (60:2),
pp. 219-231.

Thompson, M. 2004. “Discourse, Development and the Digital
Divide: ICT and the World Bank,” Review of African Political
Economy (31:99), pp. 103-123.

Ulrich, P. 2008. Integrative Economic Ethics: Foundations of a
Civilized Market Economy, Cambridge, UK: Cambridge
University Press.

Ulrich, W. 1994. Critical Heuristics of Social Planning: A New
Approach to Practical Philosophy, Chichester, UK: Wiley.

Ulrich, W. 2000. “Reflective Practice in the Civil Society: The
Contribution of Critically Systemic Thinking,” Reflective
Practice (1:2), pp. 247-268.

Ulrich, W. 2006. “Critical Pragmatism: A New Approach to
Professional and Business Ethics,” in Interdisciplinary Yearbook
of Disciplinary Ethics, L. Zsolnai (ed.), Oxford, UK: Peter Lang
Publishing Group, pp. 53-85.

Unerman, J., and Bennett, M. 2004. “Increased Stakeholder Dia-
logue and the Internet: Towards Greater Corporate Accounta-
bility or Reinforcing Capitalist Hegemony?,” Accounting,
Organizations and Society (29), pp. 685-707.

Vallentyne, P. 2007. “Consequentialism,” in Ethics in Practice,
H. LaFollette (ed.), Oxford, UK: Blackwell, pp. 22-30.

van den Hoven, J. 2008. “Moral Methodology and Information
Technology,” in The Handbook of Information and Computer
Ethics, K. Himma and H. Tavani (eds.), Hoboken, NJ: Wiley,
pp. 49-68.

van den Hoven, J., and Weckert, J. (eds.). 2008. Information
Technology and Moral Philosophy, Cambridge, UK: Cambridge
University Press.

van Es, R., French, W., and Stellmaszek, F. 2004. “Resolving
Conflict Over Ethical Issues: Face-to-Face Vversus Internet
Negotiations,” Journal of Business Ethics (53), pp. 165-172.

Von Hippel, E., and von Krogh, G. 2003. “”Open Source Software
and the ‘Private–Collective’ Innovation Model: Issues for
Organization Science,”“ Organization Science (14), pp. 209-223.

Walsham, G. 1993. “Ethical Issues in Information Systems Devel-
opment: The Analyst as Moral Agent,” in Human, Organiza-
tional, and Social Dimensions of Information Systems Develop-
ment, D. Avison, J. Kendall and J. DeGross (eds.), Amsterdam:
North Holland, pp. 281-294.

Walsham, G. 1996. “Ethical Theory, Codes of Ethics and IS
Practice,” Information Systems Journal (6), pp. 69-81.

Walsham, G. 2006. “Doing Interpretive Research,” European
Journal Information Systems (15), pp. 320-330.

Warschauer, M. 2003. “Dissecting the Digital Divide: A Case
Study of Egypt,” The Information Society (19:4), pp. 297-304.

Weckert, J., and Adeny, D. 1997. Computer and Information
Ethics, London: Greenwood Press.

Weiner, N. 1950. The Human Use of Human Beings, New York:
Houghton Mifflin.

Werhane, P., and Freeman, R. 1999. “Business Ethics: The State
of the Art,” International Journal of Management Reviews (1:1),
pp. 1-16.

White, L. 2002. “Size Matters: Large Group Methods and the
Process of Operational Research,” Journal of the Operational
Research Society (53), pp. 149-160.

Whitley, E. 2009. “Perceptions of Government Technology,
Surveillance and Privacy: The UK Identity Cards Scheme,” in
New Directions in Privacy and Surveillance, B. Goold and D.
Neyland (eds.), Cullompton, UK: Willan Publishers, pp.
133-156.

Whitley, E., and Hosein, I. 2008. “Departmental Influences on
Policy Design,” Communications of the ACM (51), pp. 98-100.

Whitley, E., Hosein, I., Angell, I., and Davies, S. 2007. “Reflec-
tions on the Academic Policy Analysis Process and the UK
Identity Cards Scheme,” The Information Society (23), pp. 51-58.

Yetim, F. 2006. “Acting with Genres: Discursive–Ethical Con-
cepts for Reflecting on and Legitimating Genres,” European
Journal of Information Systems (15), pp. 54-69.

Yetim, F. 2009. “Critical Examination of Information: A Discur-
sive Approach and its Implementations,” Informing Science (12),
pp. 1-21.

About the Authors

John Mingers is Professor of Operational Research and Information
Systems at Kent Business School and Director of Research. His
research interests include the development of theory concerning the
nature of information, meaning, and knowledge; the use of systems
methodologies in problem situations, particularly the mixing of
different methods within an intervention (multimethodology); the
development of critical realism as an underpinning philosophy for
information systems; and ethics in IS. He has published many books
and papers including Social Theory and Philosophy for Information
Systems (with Leslie Willcocks) and Realizing Systems Thinking:
Knowledge and Action in Management Science.

Geoff Walsham is Professor of Management Studies (Information
Systems) at Judge Business School, University of Cambridge. In
addition to Cambridge, he has held academic posts at the University
of Lancaster in the United Kingdom where he was Professor of
Information Management, the University of Nairobi in Kenya, and
Mindanao State University in the Philippines. His teaching and
research is focused on the question: Are we making a better world
with information and communication technologies? He was one the
early pioneers of interpretive approaches to research on information
systems.

854 MIS Quarterly Vol. 34 No. 4/December 2010

its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s

express written permission. However, users may print, download, or email articles for individual use.

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/4536338

Utilizing Usability Evaluating Model in
Applying CMM to Improve the Quality of
Software Maintenance Process

Article in Computer and Information Science · July 201

DOI: 10.5539/cis.v3n3p180 · Source: DOAJ

CITATIONS

READS

2 authors:

Some of the authors of this publication are also working on these related projects:

Curriculum Design and Review System Model for the Higher Learning Institution View project

Developing and evaluating a knowledge audit model in requirement elicitation process View project

Amir Mohamed Talib

Imam Muhammad bin Saud Islamic University

41 PUBLICATIONS 167 CITATIONS

SEE PROFILE

Rusli Abdullah

Universiti Putra Malaysia

95 PUBLICATIONS 222 CITATIONS

SEE PROFILE

All content following this page was uploaded by Amir Mohamed Talib on 31 May 2014.

The user has requested enhancement of the downloaded file.

https://www.researchgate.net/publication/45363382_Utilizing_Usability_Evaluating_Model_in_Applying_CMM_to_Improve_the_Quality_of_Software_Maintenance_Process?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_2&_esc=publicationCoverPdf

https://www.researchgate.net/project/Curriculum-Design-and-Review-System-Model-for-the-Higher-Learning-Institution?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_9&_esc=publicationCoverPdf

https://www.researchgate.net/project/Developing-and-evaluating-a-knowledge-audit-model-in-requirement-elicitation-process?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_9&_esc=publicationCoverPdf

https://www.researchgate.net/?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_1&_esc=publicationCoverPdf

https://www.researchgate.net/profile/Amir_Talib?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_4&_esc=publicationCoverPdf

https://www.researchgate.net/institution/Imam_Muhammad_bin_Saud_Islamic_University?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_6&_esc=publicationCoverPdf

https://www.researchgate.net/profile/Rusli_Abdullah2?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_4&_esc=publicationCoverPdf

https://www.researchgate.net/institution/Universiti_Putra_Malaysia?enrichId=rgreq-d5d50707a22dce78bf34145a1ea7086b-XXX&enrichSource=Y292ZXJQYWdlOzQ1MzYzMzgyO0FTOjEwMjg3NzU3ODEzNzYwMkAxNDAxNTM5MzM0NDMy&el=1_x_6&_esc=publicationCoverPdf

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 180

Utilizing Usability Evaluating Model in Applying CMM to Improve
the Quality of Software Maintenance Process

Amir Mohamed Talib

Faculty of Computer Science & IT, University Putra Malaysia

43400 UPM, Serdang, Selangor, Malaysia
E-mail:ganawa53@yahoo.com

Rusli Abdullah
Faculty of Computer Science & IT, Information System Department,

University Putra Malaysia
43400 UPM, Serdang, Selangor, Malaysia

E-mail: rusli@fsktm.upm.edu.my
Abstract
Maintenance plays an important role in the life cycle of a software product. It is estimated that there are more
than 100 billion lines of code in production in the world. As much as 80% of it is unstructured, patched and not
well documented. Maintenance can alleviate these problems. The purpose of this paper is to explore the use of
the Capability Maturity Model (CMM) to improve the quality of software maintenance process (SMP). The
architecture of the CMM model has been retained almost as is while its content, which was specific to the
development process, has been either modified or extended to take into account the characteristics specified to
the maintenance function, these characteristics were then organized into key process areas (KPAs) of CMM
model. This paper applied the definition of (ISO 9241-11, 1998) that examines effectiveness, efficiency, and
satisfaction. The emphasis will be given to the SMP activities
Keywords: Software maintenance, CMM, Software life cycle, Software maintenance process,

Usability

evaluating model, Software development
1. Introduction
Within the budget allocated to software in major corporations, the maintenance process is often taking the greatest
share of software corporate resources: for many of these organizations, between 50% and 70% of the software
budget is allocated to the maintenance process (Arthur, (1988)); (Swanson and Beath, (1989)); (Sharpe, (1991)). It
is felt that software maintenance has not received it proportionate share of management attention and that it has
suffered from lack of planning, thus resulting typically in crisis management. Within this context, software
maintenance is perceived as expensive and ineffective.
The literature search has not come up with diagnostic techniques to evaluate the quality of the maintenance process
of a given organization, nor to identify an improvement path (Arthur, (1988)); (Swanson and Beath, (1989));
(Sharpe, (1991)).
Evaluation models must support the following three management objectives:

At the operational level they provide a detailed analysis and evaluation of a business process and of its key
process.

At the tactical level, they identify the strengths and weaknesses of each process as well as a progression
path, should there be a decision taken within a continuous improvement process program. They also
provide a map to develop an action plan to address the strengths and weaknesses within the set of
organizational priorities and allocation of resources.

At the strategic level, they provide to the senior executives the relative positioning of their organizations
within their competitive environment. Based on this evaluation, priorities are then set, which lead to the
allocation of scarce corporate resources to meet the corporate objectives.

2. Literature Review
2.1 Software Maintenance Process
(Pigoski, 2002) advice is to focus on a few key characteristics when measuring the software and not measure
everything. It is better to use a few measures that are well understood rather than many measures that are not.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 18

In an interview with (Swanson, 2006), he claims that not only software systems should be maintained but also
the user knowledge of particular systems, (Kajko-Mattsson, 2001).
(Pfleeger, 2001) described a model of SM that includes measured feedback. It is an interesting approach for
measuring the impact of a proposed change to decide what the risks are and eight various options and depicts the
activities executed when a change is requested. The arrows with text coming from the rectangles pointing down,
represents the measurements with information that can be used for decisions about when and how to make a
change.
According to (Pigoski, 2007), over 50% of the software maintainer’s time is spent (in order to perform
corrections and enhancements) on understanding the program and that productivity can be improved vastly if this
process is improved.
Further (Pigoski, 2007) demonstrates that corrective SM represents only 20% of the total maintenance effort.
The remaining 80% is for enhancements, i.e. the adaptive and perfective categories of maintenance. The opinion
is that maintenance costs are too high because of the concept that maintenance is only “fixing bugs or mistakes”,
but that is the small part of SM (20%).
(Pigoski, 2007) also implies that another reason for the high SM cost is that the needed items are seldom
included initially in the development phase because of schedules or monetary constraints but are deferred until
the operation-maintenance phase.
(Vaidyanathan, 2002) says that new development approaches (prototyping, incremental and spiral models) still
need maintenance. Development continues after delivery of software and it is still subject to corrective and
preventive maintenance and may be subject to perfective and adaptive maintenance. Systems are being fielded
more quickly and the systems are modified quickly to accommodate user requirements.
2.2 CMM to improve the quality of Software Maintenance Process
(Diane, 2006) describes how CMMI can contribute to measurable improvements. It also may be useful for
executives and senior managers who are faced with decisions about the allocation of scarce resources for
improvement efforts.
In this study many of the organizations described that have achieved improvements in product quality and
customer satisfaction also have achieved higher productivity, cost performance, and schedule performance.
Better quality may not always be free, but it can occur with better project performance as a result of disciplined
process improvement.
Brief by (Nicole McFarland, 2006) describe the overview of the Capability Maturity Model Integration (CMMI)
infrastructure and concepts. CMMI is a process-improvement model for the enterprise that provides a set of best
practices that address productivity, performance, cost, and stakeholder satisfaction when producing products or
services. CMMI provides the practitioner with common terminology, common training and an integrated
appraisal method. CMMI will tell you what to do, but not how to do it or who should do it.
(Carol, D and Barbara, E., 2002) studies the mutual effect of increased process maturity and an organization’s
maturity in their use of Function Point Analysis (FPA). As a company moves to a higher maturity level
according to the Capability Maturity Model® Integration SM (CMMISM), its measurement maturity should also
increase. The tie between the CMMI’s process areas and FPA is not well understood, yet there is a direct
connection that can be made between the model and FPA.
(Raffo, 1999) have provides a brief background on alternative quantitative approaches, describes the model
currently being utilized by the SBMS Melbourne site, and provides an example of its use in analyzing a potential
process change.
In order to justify and obtain management support for proposed process improvements, process analysis tools
that support quantitative methods and address process level issues are needed.
(Viktor, C and Frank N, 2004) introduces the IT Service Capability Maturity Model (IT Service CMM®) and
gives an overview of the model itself, and describes how to use it to improve the maturity of an IT service
organization. Also describes the contents of the IT Service CMM, the main ways in which you can apply it to
improve an organization, and the background you need to understand the IT Service CMM specification.
2.3 Usability Evaluating Model
(Virzi, 1992) described usability according to five basic parameters; 1) easiness and speed of learning of system
use 2) efficiency to use 3) easiness to remember system use after certain period of time 4) reduced numbers of

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 182

user errors and easy recovery from them and 5) subjective satisfaction of users. Many authors further the study
of research in usability based on these 5 characteristics.
(Faulkner, 2005) further suggested that 15 users can uncover 90-97% of usability problems. (Carstens D. S.,
and Patterson P., 2005) mentioned that there should be specific usability guidelines for specific type of web
application.
(Kostaras N. and Xenos M, 2004) suggested the heuristic evaluation method which is to be carried out by
usability experts as it is easy to use, fast, relatively cheap and it can be employed in systems that are completed
and fully operational. Heuristic evaluation is mainly based on rules of thumb by evaluators. Usability specialists
will judge the compliance between user interface and usability principles called Heuristics either based on their
own point of view or observation.
3. Overview of CMM model
The CMM was originally developed to assist the U.S. Department of Defense (DoD) in software acquisition. The
rationale was to include likely contractor performance as a factor in contract awards. This model for determining
likely contract performance also became a guide or framework for software process improvement. It used to
judge the maturity of the software processes of an organization and to identify the key practices that are required
to increase the maturity of these processes. It describes the principles and practices underlying software process
maturity and is intended to help software organizations improve the maturity of their software processes in terms
of an evolutionary path from ad hoc, chaotic processes to mature, disciplined software processes (Herbsleb,
1994).
3.1 Benefits of CMM
There is now substantial evidence of the business benefits of CMM-based software and a growing understanding
of the factors that contribute to a successful improvement effort. In the past years, numerous empirical studies of
organizations using CMM have been conducted by SEI and others. The most broadly supported claim is that
CMM-based SPI has substantial business benefits for those moving from the initial through the defined levels
(Herbsleb, 1997). Extensive studies have showed that organizations implementing CMM have achieved
tremendous benefits in terms of productivity, calendar time, quality, and business value
The CMM represents a “common sense engineering” approach to software process improvement. The maturity
levels, key process areas, common features, and key practices have been extensively discussed and reviewed
within the software community. It represents a broad consensus of the software community and is a useful tool
for guiding software process improvement efforts. The CMM provides a conceptual structure for improving the
management and development of information systems products in a disciplined and consistent way. It can
improve the likelihood with which a software organization can achieve its cost, quality, and productivity goals.
However, CMM does not address all the issues of information systems development. Rigorously following the
practices and procedures in CMM sometimes would make information systems development process too rigid
and structured. Moreover, the overall model is too large and resource demanding for small businesses. The
complexity also makes it difficult to tailor the model for specific project.
CMM cannot possibly fit the needs of all organizations equally and cannot be interpreted literally. An
organization needs to use common sense when applying the CMM structure and enhance its own practices such
that they meet the CMM goals” (Johnson and Brodman, 2000). It’s important for an organization to rigorously
follow the key practices and procedures in CMM to achieve higher level of capability maturity level therefore
increase its market competitive advantage. But at the same time, practitioners need to make their own judgments
to flexibly implement the model to achieve the best result.
3.2 Quality in CMM model
Quality of software maintenance varies in different organizations at different CMM levels. There are many
factors influencing the maintenance process, which in turn affect the final product quality. In an immature
organization, there is no objective basis for judging product quality or for solving product or process problems.
Therefore, product quality is difficult to predict. Activities intended to enhance quality such as reviews and
testing are often curtailed or eliminated when projects fall behind schedule. When hard deadlines are imposed,
product functionality and quality are often compromised to meet the schedule. In a mature organization,
managers monitor the quality of the software products and the process that produced them. There is an objective,
quantitative basis for judging product quality and analyzing problems with the product and process.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 18

3.3 The KPAs of the CMM
Figure 1 (Note 1) lists the KPAs for each maturity level in the CMM. The CMM categorizes the overall
company process maturity into five levels of maturity. The maturity levels, 1 to 5, indicate the overall
effectiveness of the company’s software engineering practices. Each increasing level is based on achieving the
attributes of the previous low levels. The five levels are describing as follows (Paulk, 1993):

• Level 1: Initial – The software process is characterized as ad hoc, and occasionally even chaotic. Few
processes are defined, and success depends on individual effort.

• Level 2: Repeatable – The project management processes are established to track cost, schedule, and
functionality. The necessary process discipline is in place to repeat earlier successes on project with
similar applications.

• Level 3: Defined – The software processes for both management and engineering activities is
documented, standardized, and integrated into a standard software process for organization. All
projects use an approved, tailored version of the organization’s standard software process for
developing and maintaining software.

• Level 4: Managed – Detailed measures of the software process and product quality are collected.
Both the software process and products are quantitatively understood and controlled.

• Level 5: Optimizing – Continuous process improvement is enabled by quantitative feedback from the
process and from piloting innovative ideas and technologies.

Each maturity level has associated KPAs that describe the software engineering attributes that must be present to
satisfy that particular level. As the maturity level increases, quality and productivity increase and risk of
unsuccessful and unpredictable projects decreases.

By definition, KPAs are expressed at a single maturity level. There are, however, relationships between the key
process areas, and improvements in a specific management or technical area need not be restricted to a single
key process area. Figure 2 (Note 1) illustrates these relationships. Organizations may work on higher level KPAs
before they have achieved lower level maturity levels, and attention must continue to be focused on lower level
KPAs even when KPAs at higher maturity levels have been achieved.
The KPAs are categorized in Figure 2 (Note 1) into three broad categories: Management, Organizational, and
Engineering processes. The Management process category contains the project management activities as they
evolve from planning and tracking at Level 2, to managing according to a defined software process at Level 3, to
quantitative management at Level 4, to innovative management in a constantly changing environment at Level 5.
The Organizational process category contains the cross-project responsibilities as the organization matures,
beginning with a focus on process issues at Level 3, continuing to a quantitative understanding of the process at
Level 4, and culminating with the management of change in an environment of continuous process improvement
at Level 5. The Engineering process category contains the technical activities, such as requirements analysis,
design, code, and test, which are performed at all levels, but that evolve toward an engineering discipline at
Level 3, statistical process control at Level 4, and continuous measured improvement at Level 5.
Note that at Levels 4 and 5 there are KPAs that span these process categories. This helps identify potential new
KPAs for CMM v2 as Levels 4 and 5 become better understood.
4. Methodology
4.1 Utilized Usability Model
Usability is a multidimensional construct and can be assessed using various criteria. This project applies the
definition of (ISO 9241-11, 1998) that examines effectiveness, efficiency, and satisfaction. (ISO 9241-11, 1998)
defines usability and explains how to identify the information which is necessary to take into account when
specifying or evaluating usability of a visual display terminal in terms of measures of user performance and
satisfaction. Guidance is given on how to describe the context of use of the product (hardware, software or
service) and the relevant measures of usability in an explicit way. The guidance is given in the form of general
principles and techniques, rather than in the form of requirements to use specific methods.
The guidance in (ISO 9241-11, 1998) can be used in procurement, design, development, evaluation, and
communication of information about usability. (ISO 9241-11, 1998) includes guidance on how the usability of a

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 18

product can be specified and evaluated. It applies both to products intended for general application and products
being acquired for or being developed within a specific organization.
(ISO 9241-11, 1998) also explains how measures of user performance and satisfaction can be used to measure
how any component of a work system affects the whole work system in use. The guidance includes procedures
for measuring usability but does not detail all the activities to be undertaken. Specification of detailed user-based
methods of measurement is beyond the scope of (ISO 9241-11, 1998).
(ISO 9241-11, 1998) is considered as one of the most widely adopted and cited definitions of usability is that of
the International Organization for Standardization, which identifies usability with the ability to use a product for
its intended purposes: ‘the extent to which a product can be used by specified users to achieve specified goals
with effectiveness, efficiency and satisfaction in a specified context of use (ISO 9241-11, 1998).
According to the benefits and importance of (ISO 9241-11, 1998), this project proposes an evaluation model for
assessing usability of SMP. As reflected in the definition, three central criteria for usability are the effectiveness,
efficiency and satisfaction with which users can achieve specified goals.
4.1.1 Effectiveness
The first criterion, effectiveness, suggests that specified goals are to be achieved with completeness (ISO
9241-11, 1998). Effectiveness can be understood as “how good a system is at doing what it is supposed to do”
(Preece, Rogers and Sharp, 1992) and is related to the “utility” of the system (Grudin, 1992), and “to the extent
to which the system provides the right kind of functionality so that users can do what they need or want to do”
(Preece, Rogres, and Sharp, 1992). Effectiveness can also be defined as the accuracy and completeness with
which users achieve certain goals.
Indicators of effectiveness include quality of solution and error rates. Moreover, quality of solution is used as the
primary indicator of effectiveness, i.e. a measure of the outcome of the user’s interaction with the system. In this
project, effectiveness is to evaluate if the system as a whole can provide information and functionality effectively
and is measured by how many answers are correct.
4.1.2 Efficiency
The second criterion, efficiency, suggests that specified goals are to be achieved with as little expenditure of
resources as possible (ISO 9241-11, 1998). But in another way, measures of efficiency relate the level of
effectiveness achieved to the expenditure of resources (Bevan, 1995).
According to (Bevan, 1995), resources may be ‘mental or physical effort, which can be used to give measures of
human efficiency, or time, which can be used to give a measure of temporal efficiency, or financial cost, which
can be used to give a measure of economic efficiency’.
Moreover efficiency, which is the relation between (1) the accuracy and completeness with which users achieve
certain goals and (2) the resources expended in achieving them. Indicators of efficiency include task completion
time and learning time. In this study, we use task completion time as the primary indicator of efficiency. In
addition, efficiency is likewise to evaluate if the system as a whole can be used to retrieve information efficiently
and is measured.
4.1.3 Satisfaction
The third criterion, satisfaction, suggests that users should feel comfortable with, and have positive attitudes
towards the use of the system (ISO 9241-11, 1998). In this sense, satisfaction relates to concepts such as ease of
use, user satisfaction and usefulness (Davis, 1989) and (Mathieson and Keil, 1998). The ‘specified context of use’
includes users, tasks, equipment and the physical environment, where ‘task’ is defined in terms of activities
required to achieve a goal (ISO 9241-11, 1998).
Satisfaction is the users’ comfort with and positive attitudes towards the use of the system. Users’ satisfaction
can be measured by attitude rating scales. And can measure by use preference as the primary indicator of
satisfaction.
Finally, Satisfaction looks into the areas of ease of use, organization of information, and visual attractiveness,
and error handling and is measured by questionnaires. Ease of use is to evaluate user’s perception on the ease of
use of the system. Organization of information is to evaluate if the system’s structure, layout, and organization
meets the user’s satisfaction. Labeling is to evaluate from user’s perception if the system provides clear labeling
and if terminology used are easy to understand. Visual attractiveness evaluates the site’s design to see if it is
visually attractive. Contents evaluate the authority and accuracy of information provide.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 18

From previous studies, we found interlocking relationships among effectiveness, efficiency, and satisfaction. The
longer it took to perform a task, the greater the number of steps involved. The greater the number of steps
involved in completing a task, the lower the satisfaction. The more time spent on completing a task, the lower
the satisfaction. Further, incorrect answers entailed more steps, while correct answers entailed fewer steps. This
means that when we knew how to locate the answer, it take fewer steps. Although there are interlocking
relationships, effectiveness, efficiency, and satisfaction are three distinct criteria and should be measured
separately. One cannot replace the other.
For that a usability evaluating model has been developed in order to identify a comprehensive set of usability
components needed to evaluate various aspects of is how the

CMM KPAs

improve the quality of SMP shown in
Figure 3 (Note 1)
4.2 Respondents
The respondents of this study are two companies in Malaysia and these companies were already familiar with the
SMP and its applications and the companies respondents was chosen to fill the questionnaire of this study.
4.3 Procedure
In the beginning, the respondents was received a short, scripted verbal orientation explaining the purpose of the
usability evaluating. Then they were asked to complete a short background questionnaire to collect their
demographic characteristics. The respondents were asked to perform a set of information via a questionnaire
sheet (Note 3) about how to improve the quality of SMP using the CMM KAPs. The tasks were written on a
sheet of paper that included a space where respondents were asked to indicate their answers. Once the tasks were
completed, respondents were asked to complete a short respondent satisfaction questionnaire to collect and test
their own perceptions towards SM.
4.4 Tasks
Respondents were complete three tasks:
1. They completed a background/experience questionnaire that including name, gender, age, education level,
Major/Department, and years of experience.
2. They performed tasks using the questionnaire’s sheet.
3. There was also a post-survey questionnaire that specifically examines CMM KAPs techniques. After
completing a task, the respondents were asked to rank satisfaction and to write down comments.
4.5 Data Collection
This evaluation model considers both quantifying elements of performance (experience and experiment) as well
as subjective empirical. If the answer is wrong, or he/she not familiar with this question then skip to the second
question until all the question will be solved. However, the authors will record whether respondents are able to
complete tasks successfully. The criteria for successful task completion are:
• Respondent is able to give a correct answer based on his own information about the system. Any guessed or
assumed answers, whether correct or not, are not record as successfully completed tasks.
• Respondent is able to give a definite answer to the question. Where respondents indicated they are unsure
about the answer or would seek clarification, the task will record as not successfully completed.
4.6 Questionnaires
The purpose of the questionnaire is to prove:
• Handle the interpretation of the term KAPs of CMM and the company’s key objective in SM.
• Handle the aspects that come into play in KAPs, such as the existence of a strategy, the processes of quality
control of data, the content that is being managed.
• Identify the KAPs technique of willingness of cooperation for research work.
Basically, there are two types of questionnaire that we prepared as part of usability evaluating for the
respondents for the level of the questions.
4.6.1 Pre-Survey Questionnaire (background)
A series of questions designed to collect demographic information about the respondents to assess their level of
his information about the system.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 18

4.6.2 Post-Survey Questionnaire
After the test subject completed each scenario, he/she should answer a specific questions related to the tasks. To
indicate whether the tasks was clear and completed successfully. After the test subjects complete all the
scenarios, he/she will answer eighteen points.
5. Result
The result was conducted according to the methodology described in the previous section. It is starts with an
overview of data collected by analyzing trends. Correlation was measured by Pearson’s Correlation process of
SPSS 16.0, the Quantitative and Qualitative analysis for CMM KPA’s questions were described.
This study applies CMM model to evaluate and improve the quality of the SMP.
Whichever usability evaluation method was used, it is necessary to consider several kinds of usability factors in
a systematic manner. Moreover, in the case of expert evaluation using analytical methods, comprehensive
consideration of usability factors are very important to overcome its limitations.
Several types of data were collected to assess user’s performance and user’s perceptions, as follows:

Effectiveness was measured by the number of tasks successfully completed.
Efficiency was measured by the amount of time taken to complete all tasks.
Satisfaction was measured by a rating scale for several satisfaction elements.

In addition, we divided the result based on two parts; Quantitative Analysis of CMM KPAs in Software
Maintenance and Qualitative Analysis of CMM KPAs in Software Maintenance. Any task contained 9 questions
(tasks).
5.1

Effectiveness

As we can notice all respondents were able to complete all the tasks. Effectiveness was measured by the number
of tasks successfully completed. For other tasks respondents were able to complete in success percent ranging
between (50% & 100%). The Successful task completion for the individual tasks is summarized in Figure 4 and
Figure 5 (Note 2) below for the respondents. As we can see the average of successfully completion task are high,
according to this results the successfully completion task that presented the effectiveness, achieved correctly.
Moreover these results of the tasks successfully completed are high.
Figure 4 (Note 2) shows that Q2, Q6, Q8 and Q9 are completed answered successfully (100%) and the others
questions got more than 70%.
Figure 5 (Note 2) shows that Q1, Q5 and Q9 are completed answered successfully (100%) and the others
questions got more than 60%.
5.2

Efficiency

Completion time is the one factor used for measuring efficiency in this paper. Efficiency was measured by the
amount of time taken to complete all tasks. An average of 44 minutes and 3.5 seconds per respondent was taken
to complete the all tasks. However, there was much variation among the respondents, for example, the fastest
respondent took only 18 minutes and the slowest took 37 minutes and 9 seconds which are about three times
longer. Pearson’s product-moment correlation analysis was conducted to see if the respondents’ completion time
is related. The results showed that total completion time is independent, (see Figure 6 and Figure 7) (Note 2).
As a result, efficiency was measured by evaluating completion time used in this survey by each respondent.
Respondents who they were familiar with the systems in general tended to use less time to complete their tasks.
When the respondent knows how to get the answer, it takes them fewer time while when they don’t know how to
use the system, they take more time.
5.3

Satisfaction

Respondents Satisfaction measured by using the two scales (YES= respondent agreed, NO= respondent not
agreed). Satisfaction was measured by a rating scale for several satisfaction elements.
According to the result below, the satisfaction for the respondents were in moderate level.
Figure 8 (Note 2) shows the respondent satisfaction for Quantitative analysis for MAS. The Q2 and Q8 give high
satisfaction (

Mean

=5.10) out of (Maximum =6) and Q6 give high satisfaction (Mean=4.70), and Q7 also give
high satisfaction (Mean=4.40) out of (Maximum=5). All respondents feel satisfied with the system when they
fail to perform the task correctly.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 187

Figure 9 (Note 2) shows the respondent satisfaction for Qualitative analysis for MAS. Q8 give high satisfaction
(Mean=5.40) and Q9 give high satisfaction (Mean=5.50) out of (Maximum =6) and Q7 give high satisfaction
(Mean=4.20) out of (Maximum=5). 30% of respondents feel less satisfied with the system when they fail to
perform the task correctly.
6. Comparison with the CMM Model
It can be seen at first glance the content of the CMM Model which was used as an input to the adaption process
was extensively modified. When version 1.0 of the maintenance model is compared to the original CMM Model,
it can be seen in table 1 (Note 3) that four of KPAs is unchanged, six have been modified and eleven have been
added.
7. Conclusion
As there are various methods of developing software, different approaches of maintenance activities are adopted
for each different development method. Because CMM focuses on changing the system to make it better able to
adapt to user requirements throughout the software’s lifetime, maintenance becomes easier. It is also good to use
an iterative method for the maintenance phase because iteration is done for each change made to the system and
each bug that is fixed. This allows for smaller and more frequent releases. This ensures that each added part of
the product is fully working before moving onto the next change. Using smaller releases also makes it easier to
code and debug each part. In addition, it allows the developers to have a working product at the end of each
iteration. On the other hand, component based software design is built on the idea of developing software
systems by selecting appropriate off-the-shelf components and then assembling them with a well-defined
software architecture. The maintenance of such systems is done at the component level rather than at the source
code level. Therefore, if a defect is found in a specific part, the maintainers can seamlessly replace that module
with a working one. This is done easily by modifying the glue code. Finally, open source development allows
each user to serve as a co-developer. This leads to the creation of better software and allows users to customize a
program on their own. Any of these development methodologies can be used to make the maintenance phase
more productive and efficient.
References
Arthur. L. J. (1988). “Software Evolution-The Software Maintenance Challenge” John Wiley & Sons
Bevan, N. (1995). “Measuring usability as quality of use Software Quality”, Journal of Usability Measuring, 45:
567-575.
Carol, D and Barbara, E. (2002). “CMMI Capability Maturity Model Integration: What has changed?” The
Journal of Defense Software Engineering, Vol, 15 No 2

Carstens D. S., Patterson P. (2005). “Usability Study of Travel Websites”, Journal of Usability Studies, Issue 1,
volume 1, 2005, pp. 47-61.

Davis, F.D. (1989). “Perceived usefulness, perceived ease of use, and user acceptance of information”. MIS
Quarterly, pp 318.

Diane, L. Gibson, D, Goldenson. D and Keith, K. (2006). “Performance Result of CMMI-Based Process
Improvement” TECHNICAL REPORT, CMU/SEI-2006-TR-004, ESC-TR-2006-004.
Faulkner, L. (2005) “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing”.
Behavior Research Methods, Instruments and Computers, 35(3), pp. 379-383.
Gurdin, J. (1992). “Groupware and Social Dynamic” Eight Challenges for developers. Communications of the
ACM 37, 1, pp. 92-105.

Herbsleb, A. Carleton, J. Rozum, et al. (1994). Benefits of CMM-Based Software Process Improvement: Initial
Results, retrieved November 26, 2007 from http://www.sei.cmu.edu/pub/documents/94.reports/94.tr.013.html

Herbsleb, J., Zubrow, David, D. and Goldenson, et al. (1997). “Software Quality and the Capability Maturity
model”, Communications of the ACM, 40, 6
International Standards Organization. (1998). ISO document 9241- Ergonomic requirements for office work with
visual display terminals.
Johnson, D.L. and Brodman, J.G. (2000). “Applying CMM Project Planning Practices to Diverse Environments,
Software”, IEEE, Volume: 17 Issue: 4 , July-Aug, pp 40-47.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 18

Kajko-Mattsson, M. (2001). “A Conceptual Model of Software Maintenance” Journal of Software

Maintenance

14(3): 197-227
Kostaras N., Xenos M. (2004). “Assessing Educational Web-site Usability using Heuristic Evaluation Rules”, in
11th, Panhellenic Conference in Informatics, pp. 543-550.
Mathiesom. K. Keil. M. (1998). “Beyond the interface: Ease to use and Task/Technology Fit”, pp.579-592
Nicole McFarland. (2006). “Introduction to CMMI course participant” Version 1.1, Vol 3, No 1, pp 37-45

Paulk, Mark C. (1993). Report: “Capability Maturity Model for Software, Version 1.1”,

Pfleeger, S. L. (2001). “Software engineering – theory and practice” Upper Saddle River: Prentice Hall. NJ,
USA. JCS&T Vol. 5 No. 1

Pigoski., T. (2002). “Software Engineering – A practitioner’s approach” IEEE Software archive, Vol 19 No 4, pp
100 – 106

Pigoski., T. (2007). “Practical Software Maintenance”, John Wiley & Sons. Vol 49 No 5, pp 515-529

Preece. J, Rogers. E, and Sharp. B.A. (1992). “Evaluating a teen preventing of Usability Web”, Journal of
Victorian Behaviour. pp. 227-247.
Raffo, Vandeville, and Martin. (1999). “Software Process Simulation to Achieve Higher CMM Levels,” Journal
of Systems and Software, Vol. 46, No. 2/3
Sharp, S, Haworth, D.A, et Hale, D. (1991). “Characteristics of Empirical Software Studies: 1980-1989”. Journal
of Software Maintenance and Practice. Vol 3, PP. 1-5.
Swanson. (2006). “A review of past and current overall models for maintenance management”. Quality in
maintenance engineering journal, Vol.6 No.2.
Swason. E.B, and Beath, CM. (1989). “Maintaining Information System in Organization” John Wiley & Sons.
Vol 1 No 1, pp 47 – 58
Vaidyanathan. (2002). “Analysis of inspection-based preventive maintenance in operational software systems”,
Article in Reliable Distributed Systems, Proceedings. 21st IEEE Symposium on, pp.286-295
Viktor, C and Frank, N. (2004). “IT Service CMM, a pocket guide” DTPresto Design & Layout, Zeewolde-NL,
ISBN: 90-77212-35-3
Virzi, R. A. (1992). “Refining the test phase of usability evaluation”: How many subjects is enough? Human
Factors, 34, pp. 457-468.

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 189

Notes:
Note 1.

Figure 1. The KPAs by Maturity Level

Figure 2. The KPAs Assigned to Process Categories

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 190

Figure 3. Proposed Usability Model

Note 2.

Figure 4. Completed Tasks Successfully for Quantitative analysis

Figure 5. Completed Tasks Successfully for Qualitative analysis

20%

40%

60%

80%

100%

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9

Pe
rc

en
ta

ge
%

Question No

% task’s successful
completion

0%
20%
40%
60%
80%
100%
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Pe
rc
en
ta
ge
%
Question No
% task’s successful
completion
Usability
Effectiveness
Efficiency
Satisfaction

Quantitative

Analysis of

CMM KPAS

Qualitative

Analysis of
CMM KPAs

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 191

Figure 6. Time used to completed for Quantitative analysis

Figure 7. Time used to completed for Qualitative analysis

Figure 8. Respondent satisfaction for Quantitative analysis

0
2
4
6
8

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9

Ti
m
e

Question No

Mean (task’s completion
time)

0
2
4
6
8
10
12
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Ti
m
e
Question No
Mean (task’s completion
time)
0
1
2
3
4
5
6
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Question No
Mean

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 192

Figure 9. Respondent satisfaction for Qualitative analysis

Note 3:
Table 1. Comparison of KPAs and CMM Model

Level Maintenance Kea Process Area (KPAs) Key Practices for Our New CMM
Model

Unchanged Modified New

Initial

Repeatable 1. Requirement Management
2. Software Maintenance Planning
3. Software Maintenance Tracking
4. Software Subcontract Management
5. Software Quality Assurance
6. Software Configuration Management
7. Transfer to Maintenance Management
8. Problem Management
9. User Support Management
10. Organization of Maintenance Function
11. Role of the Interviewing around the

Maintenance

x
x

Defined 12. Organization Process Focus
13. Organization Process Function
14. Training Program
15. Integrated Maintenance Management
16. Software Maintenance Portfolio

engineering

x
x
x

x
x

Managed 17. Quantitative Management of the Process
18. Software Quality Management

x
x

Optimizing 19. Defect Preventive
20. Process Change Management
21. Process Improvement Management

Total Twenty one KPAs 4 6 11

0
1
2
3
4
5
6
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Question No
Mean

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 193

We observed from table 1 above, some KPAs were derived from the CMM model and modified to map more
closely to maintenance characteristics. Our own results support the validity of separate dimensions
corresponding to maturity levels 2 and 3 respectively. The existence of an additional dimension may reflect the
difference between organizational maturity as characterized by the CMM and process capability as characterized
by ISO/IEC 15504. Note too that both the 15504 studies and our own suggest the existence a similar
“Quantitative Process Implementation” construct. Such similarities based on disparate methods and data provide
additional confidence about both sets of results. Although the KPAs at levels 4 and 5 map to the same dimension
in our current analysis, we can safely conclude that each stage of the CMM corresponds to a dimension of
organizational maturity and that the KPAs in each maturity level are good items for measuring the maturity of a
common underlying construct.
QUESTIONNAIRE SHEET
Appendix A.1 Pre-Survey Questionnaire
Thank you very much for agreeing to participate in this experiment. All of your personal data that we collect will
be entirely confidential. I would like to gather a bit of background information about u.
Respondent Name__________________________________________
Gender: ______Male ______Female
Date________________________
How old are you? 20-29 30-39 40-49 50-59 60 or above
Level of education:
_____Certification Bachelor ________ Certification Diploma
__________Degree Postgraduate
Appendix A.2 Usability Evaluating Questions
The goal of this Survey to ensure that the CMM KPAs (Key process areas) will improve the quality of Software
maintenance process
I will ask you a series of questions and would like you to think out loud while you look for the answer. Please
remember that we are testing the effectiveness of the CMM and its KPAs and this is not a test of you. The whole
test should take less than one hour. Thank you
Kindly be informed that this survey generalized for all the four part of the Software maintenance:
Corrective: a detected error does not block the normal operation of the system and the solution time is not
critical.
Predictive: when new functionalities are added to the system.
Preventive: consists of the software modification to improve its maintainability and quality properties.
Adaptive: when the system will change its execution environment.
Description for How to Answer the Question:
Evaluation of the matrix: Assign yourself the following points for each
NA = 0, where 0 is doing nothing at all = NONE and
1 = Don’t Know, Not Sure or Can’t Say = NO
2 = Not Important or as Not been Addressed = MINIMALLY
3 = Partially Beneficial or somewhat Effective or Less Scope for Overall Improvement =
PARTIALLY
4 = Important or May not be effective but other associated necessary actions being taken =SUBSTANTIALLY
5 = Critical or already in place and effective = FULLY
Also, the scale can generally be summarized as follows for majority situations
‘NA 1 2 3 4 5’ is calibrated as in
‘5 (Always) 4 (Often) 3 (Sometimes) 2 (Occasionally) 1 (Never)’
NA (Not Applicable), (Note: “NA” and “1” scale values are equivalent.)

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 194

QUESTIONNAIRE – Part One (Quantitative Analysis)
1. Does your organization follow a documented plan for developing and improving its software process in the
whole process of Software maintenance?
NA 1 2 3 4 5
2. Does the standard software development process documentation describe the use of tools and techniques at
any Software maintenance process project?
NA 1 2 3 4 5
3. Is Software maintenance process mechanism used for assessing existing designed and code for reuse in the
new application?
NA 1 2 3 4 5
4. For each Software maintenance process project involving software development, is there a designed software
manager?
NA 1 2 3 4 5
5. Are Software maintenance process prototype methods used in designing the critical performance elements of
the software?
NA 1 2 3 4 5
6. Are internal design review standard applied in Software maintenance process project?
NA 1 2 3 4 5
7. Are statistics on software design errors gathered at any level of Software maintenance process project?
NA 1 2 3 4 5
8. Do people throughout your organization participate in software process improvement activities in Software
maintenance process (e.g., on teams to develop software process improvements)?
NA 1 2 3 4 5
9. Does the Software Quality Assurance (SQA) function have a management reporting channel separate from the
Software maintenance project management?
NA 1 2 3 4 5
QUESTIONNAIRE- Part Two (Qualitative Analysis)
1. Have your organization conducted a new technologies evaluated to determine their effect on quality and
productivity at any level of the Software maintenance process?
Yes No
2. Are the Software maintenance process project’s actual results (e.g., schedule, size, and cost) compared with
estimates in the software plans?
Yes No
3. Does your organization follow a written organizational policy to meet its training needs in Software
maintenance process project?
Yes No
4. Is a mechanism used for initiating error prevention actions at any level of Software maintenance process?
Yes No
5. Is review efficiency analyzed for any Software maintenance process project?
Yes No
6. Does the Software maintenance process project follow a written organizational policy for performing peer
reviews?
Yes No
7. Are estimates (e.g., size, cost, and schedule) documented for use in planning and tracking the Software
maintenance process project?

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

Published by Canadian Center of Science and Education 195

Yes No
8. Are the Software maintenance process project’s actual results (e.g., schedule, size, and cost) compared with
estimates in the software plans?
Yes No
9. Does your organization follow a written organizational policy to meet its training needs in Software
maintenance process project?
Yes No
Appendix A.3 Post-Survey Questionnaire
Thanks again for participating in this experiment. This questionnaire gives you an opportunity to tell us your
reactions to the system you used. Please circle a number on the scale to indicate your reactions. Thank you
QUESTIONNAIRE – Part One (Quantitative Analysis)
1. Is a documented procedure used for selecting subcontractors based on their ability to perform the work in the
Software maintenance process?
NA 1 2 3 4 5
2. Is a mechanism used for error cause analysis at any level of Software maintenance process?
NA 1 2 3 4 5
3. Is a formal procedure used to make estimates of Software maintenance process size?
NA 1 2 3 4 5
4. Are defect prevention activities planned in the Software maintenance process project?
NA 1 2 3 4 5
5. Are the activities for managing software quality planned for the Software maintenance process project?
NA 1 2 3 4 5
6. Does the system requirement in the Software maintenance process activities allocated to software change; are
the necessary adjustments to software plans, work products, and activities made?
NA 1 2 3 4 5
7. Is the project planned and managed in accordance with the project’s defined Software maintenance process?
NA 1 2 3 4 5
8. Are software configuration management activities planned for the Software maintenance process project?
NA 1 2 3 4 5
9. Are automated tools used to analyze the complexity of Software maintenance process?
NA 1 2 3 4 5
QUESTIONNAIRE- Part Two (Qualitative Analysis)
1. On the Software maintenance process project, do the software engineering group and other engineering groups
collaborate with the customer to establish the system requirements?
Yes No
2. Are target computer memory utilization estimates and actual tracked at any Software maintenance process
project?
Yes No
3. Has your organization developed, and does it maintain, a standard Software maintenance process?
Yes No
4. Are Software maintenance process prototype methods used in designing the critical elements of the machine
interface?
Yes No
5. Are automated tools used to analyze cross references between modules in Software maintenance process?

www.ccsenet.org/cis Computer and Information Science Vol. 3, No. 3; August 2010

ISSN 1913-8989 E-ISSN 1913-8997 196

Yes No
6. Does the system requirement in the Software maintenance process activities allocated to software used to
establish a baseline for software engineering and management use?
Yes No
7. Does SQA (Software Quality Assurance) provide objective verification that Software maintenance process
and activities adhere to applicable standards, procedures, and requirements?
Yes No
8. Are the Software maintenance process activities and work products for engineering software subjected to SQA
(Software Quality Assurance ) reviews and audits (e.g., is required testing performed, are allocated requirements
traced through the software requirements, design, code and test cases)?
Yes No
9. Does the organization follow a plan for managing technology changes in the Software maintenance process
project?
Yes No
Comments: ____________________________________________________________________________
______________________________________________________________.

V i e w p u b l i c a t i o n s t a t sV i e w p u b l i c a t i o n s t a t s

https://www.researchgate.net/publication/45363382

Impact of Classes of Developmen

Coordination Tools on Software Development
Performance: A Multinational Empirical Study

AMRITTIWANA

Iowa State University

Although a diverse variety of software development coordination tools are widely used in practice,
considerable debate surrounds their impact on software development performance. No large-scal

field research has systematically examined tbeir impact on software development performance.
This paper reports tbe results of a multinational field study of software projects in 209 software
development organizations to empirically examine the influence of six key classes of development
coordination tools on the efficiency {reduction of development rework, budget compliance) an

effectiveness (defect reduction) of software development performance.

Based on an in-depth field study, the article conceptualizes six holistic classes of development
coordination tools. Tbe results provide nuanced insights—some counter to prevailing beliefs—int

the relationships between the use of various classes of development coordination tools and software
development performance. The overarching finding is tbat the performance benefits of development
coordination tools are contingent on tbe salient types of novelty in a project. The dimension of devel-
opment performance—efficiency or effectiveness—tbat eacb class of tools is associated witb varie

systematically witb wbetber a project involves conceptual novelty, process novelty, multidimen-
sional novelty (botb process and conceptual novelty), or neitber. Anotber noteworthy insight is
that tbe use of some classes of tools introduces an efficiency-effectiveness tradeoff. Collectively,
tbe findings are among the first to offer empirical support for tbe varied performance impacts of
various classes of development coordination tools and have important implications for software
development practice. Tbe paper also identifies several promising areas for future researcb.

Categories and Subject Descriptors: D.2.9 [Software EngineeringI: Management—Productivity

General Terms: Design

Additional Key Words and Pbrases: Software development, outsourcing, field study, empirical study,
regression analysis, software process improvement, knowledge management, development tools,
software process improvement, project management, efficiency effectiveness tradeoff, knowledge
integration, coordination, collaborative software engineering. Software outsourcing, development
coordination tools.

Tbe support of the Russian Development Alliance, the Embassy of Ireland in Washington DC, tbe
Ireland Development Agency, India’s National Association of Software and Service Companies, and
Emory University is gratefully acknowledged.
Autbor’s address: Iowa State University, 2340 Gerdin, Ames. IA 50011-1350.
Permission to make digital or bard copies of part or all of this work for personal or classroom use is
granted without fee provided that copies are not made or distributed for profit or direct commercia

advantage and that copies show tbis notice on tbe first page or initial screen of a display along
with the full citation. Copyrights for components of this work owned by otbers than ACM must be
honored. Abstracting witb credit is permitted. To copy otherwise, to republisb, to post on servers,
to redistribute to lists, or to use any component of tbis work in otber works requires prior specific
permission and/or a fee. Permissions may be requested from Pubhcations Dept.. ACM, Inc., 2 Penn
Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org.
© 2008 ACM 1049-331X/2008/04-ART11 $5.00 DOI 10.1145/1348250,1348257 http^/doi.acm.org

10.1145/1348250.134825

ACMTransactionson Software Ei^neering and Methodology, Vol. 17, No. 2,ArticIe 11, Publication date; April 2008

11:2 • A. Tiwana

ACM Reference Format: • • ‘

•

Tiwana, A. 2008. Impact of classes of development coordination tools on software development per-
formance: A multinational empirical study. ACM Trans. Soflw. Engin. Method. 17, 2, Article 1

(April 2008) 47 pages. DOI = 10.1145/1348250.1348257 http://doi.acm.org/10.1145/1348250.
1348257

1. INTRODUCTION

As the scope, complexity, and geographical dispersion of software development
increases, various development coordination tools are being widely adopted

the industry. Development coordination tools are defined as software process
support tools that help maintain accuracy, consistency, and integrity ofthe var-
ious mutually interdependent artifacts that are created during the software de-
velopment process. These artifacts include requirements, architectural views,
source code files, test plans, documentation, and reports.

The implicit assumption underlying the adoption of such tools is that they
improve software development performance by helping manage the growing
complexity of software projects [Messerschmitt and Szyperski 2003]. There is,
however, an ongoing debate about whether such tools actually improve software
development performance [Brown 2003; Spinellis 2005]. The scant anecdotal
evidence that exists suggests mixed results. Others have argued that the use-
fulness of a particular tool depends on the particular project and that using
an inappropriate tool can actually decrease software development performance
[Seeley 2003]. Some cautious companies therefore have simply avoided exten-
sively adopting such tools, subscribing to the perspective that it is wiser

not use a tool than to use a “wrong” tool [Brown 2003; Spinellis 2005]. Curi-
ously, the reluctance of software practitioners to extensively adopt such tools
has partly been blamed for the declining productivity of the software industry
[Groth2004].

Understanding these issues is important because software developers are
often faced with the decision about whether and what combination of develo

ment coordination tools to use for any given project [Groth 2004; Seeley 2003].
The ambiguity about the impact of development coordination tools on software
development performance can be attributed to the surprising absence of any
large-scale industrial studies that have rigorously assessed their individual or
simultaneous impact on software development performance in organizations
that actually use them. The few studies that have examined the impact of de-
velopment coordination tools have either focused on specific, proprietary tools
within narrowly defined categories such as requirements traceability [Ramesh
1998; Ramesh and Dhar 1992] or have been too broad, focusing on generic con-
ceptualizations of computer aided software engineering (CASE) tools [Guiñan
et al. 1997; King and Galliers 1994; Orlikowski 1993]. Since the classes of fun

tionality provided by CASE tools varies from one implementation to another,
fully understanding their utility also requires delineation of whether they are
front-end (focused on the initial design stages of development), back-end {fo-
cused on the coding and testing stages of development), or integrated CASE
tools (spanning the entire software development life cycle). Prior research on

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois • 11:

development coordination tools can therefore be characterized as either being
too narrowly focused on proprietary tools or too broadly focused on CASE to

collections. Therefore, there exists httle generalizable knowledge of how the
individual classes of development coordination tools used in software devel-
opment practice impacts software development performance in actual projec

relative to projects that do not use them.

In this article, we attempt to address this gap through a multinational field
study, focusing on two research questions.

(1) How does the use of various classes of development coordination tools im-
pact software development performance (efficiency and effectiveness)?

(2) How does their influence vary across projects with different combinations
of project novelty (routine, conceptual novelty, process novelty, and multi-
dimensional novelty)?

To address these questions, we conducted a multiphase international field
study involving 209 outsourced apphcation development projects. We chose to
focus on outsourced projects because: (1) they are more likely to require greater
coordination across the client and vendor organizations during the development
process, (2) specialized, software vendors have historically been early adopters
of software development tools, and (3) an increasing proportion of applica-
tions development work is outsourced. The software projects in this study were
completed for American client companies by software development companies
belonging to the three major global software consortia in India, Ireland, and
Russia. In the initial phases of the study, we extensively interviewed 26 software
practitioners and experts in four countries to identify a field-based classifica-
tion of six distinct classes of development coordination tools: (1) requirements
management tools, (2) architectural modeling tools, (3) test automation tools,
(4) test case development tools, (5) configuration management tools, and (6)
defect and change request tracking tools. We used these conceptualizations to
conduct an extensive industrial field study in which we collected primary da

from 209 software development organizations and their clients. We then used
this data to examine empirically the relationships between the extent of us-
age of these classes of development coordination tools and both efficiency (cost
and rework) and effectiveness (defects remaining at the time of system deliv-
ery) dimensions of software development performance across all projects. We
conducted additional, finer-grained analysis to assess how the performance ef-
fects of the six classes of development coordination tools vary based on whether
a project involves conceptual novelty, process novelty, neither type of novelty
(routine), or multidimensional novelty.

Two overarching themes in our empirical findings are noteworthy. First, the
usefulness of each class of development coordination tool is contingent on the
combinations of novelty associated with the project to which it is applied. Sec-
ond, there is often an efficiency-effectiveness tradeoff in using such tools. Many
classes of tools that are associated with improvements in software develop-
ment efficiency in a given type of project (e.g., reducing rework, and prevent-
ing cost overruns) are simultaneously associated with decreases in software

ACM Transactionson Software Engineering and Methodology, VoL 17, No. 2, Article 11. Publication date: April 2008

11:4 • A. Tiwana

development effectiveness. This pattern varies systematically with project nov-
elty type. Some ofthe detailed findings counter popular, taken-for-granted be-
liefs: Some classes of tools that benefit one type of project can impede another
type, challenging the conventional assumption that adopting more development
tools can only enhance the software process. Our results show that sometimes a
class of tools can be good at what it does but ineffective at solving the knowledge
coordination problems specific to a given type of project. Overall, these results
have significant implications for choosing an optimal mix of development coor-
dination tools for different types of projects.

The rest ofthe article is organized as follows. In the next section, we discuss
the dominant knowledge integration challenges under each combination of nov-
elty and the role of development coordination tools in overcoming them. We also
describe six major classes of tools based on the in-depth qualitative front-end
phase ofthe field study. In Section 3, we formulate a testable formal model. Sec-
tion 4 describes the field study methodology, including the initial conceptual
development interviews, the large-scale, multinational survey, and follow-up
interviews. Section 5 presents the statistical analysis and results. Their impli-
cations for research and software development practice are discussed in Section
6. The paper concludes with a summary ofthe key findings.

2. CONCEPTUAL DEVELOPMENT

The central ingredients of software are: (1) technical knowledge through which
the software is designed and developed and (2) knowledge of the problem ap-
plication domain [Adelson and Soloway 1985; Robillard 1999; Rus and Lindvall
2002]. Effective software development requires integration of these two types
of knowledge during the development process [Faraj and SprouU 2000; Tiwana
and McLean 2003; Walz et al. 1993]. Such knowledge integration is critical for
successful software development because it ensures that the design and imple-
mentation decisions made on either side of the client-developer organization
interface are mutually consistent [Mookerjee and Chiang 2002; Seaman and
Basili 1998; Tiwana and Keil 2004]. Effective knowledge integration therefore
ensures that the application domain—design dependencies are effectively man-
aged and that what is built closely matches what is needed [Messerschmitt and
Szyperski 2003]. The challenge in accomplishing such knowledge integration in
outsourced projects arises from the dispersion of these two tj^es of knowledge
across the client and software development organizations. The knowledge fiow
from the client to the vendor and from the vendor to the client that facilitates
knowledge integration during the software development process can be viewed
as a two-pronged knowledge transfer loop. This is illustrated in Figure 1 and
discussed next.

Knowledge about the application domain and project requirements that is
needed for design decisions in the software development organization is lo-
cated in the client organization, but must be accurately transferred to the soft-
ware development organization [Gottesdiener 2003; Tiwana and McLean 2003;
Tiwana and McLean 2005]. Client to vendor knowledge transfer—for example,
through requirements and project specifications—represents the feed-forward

ACM TranBactionB on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools 11:

Client-Vendor Organizational Boundary

Project Requirements
(Feed-foiward loop)

Classes of Development
Coordination Tools

Client ^

O r g a n i z a t i o n ^ ^

Requiremenls managers

Architectural modelers

Test automation tools

Tesl case developmeni tools

Contiguralion management tools

Defect & change request trackers

. ^ Vendor
^ k Organization

•

(Feed-back loop)

•Design Validation-‘

Cliant-Vandor Organizational Boundary

Fig. 1. Various classes of development coordination tools can facilitate coordination of require-

ments and design validation activities across the client-vendor organizational boundary/ interface.

portion of this loop. Solely for simplicity of discussion, we label this facet
of knowledge transfer between the client and vendor as the project require-
ments loop. However, it is well recognized that initial project requirements and
specifications are imperfect and insufficient knowledge transfer mechanisms
[Messerschmitt and Szyperski 2003]. Therefore fully incorporating the client’s
intended objectives, constraints, and requirements missed in the preliminary
formal requirements requires iterative refinement. Thus information about the
in-progress design that is generated in the software development organization
must be validated by the client, requiring transferring it from the vendor to
the client [Abdel-Hamid et al. 1993; Ramesh and Dhar 1992], Such iterative
refinement and validation of the preliminary project requirements requires
the complementary feedback facet ofthe knowledge transfer loop. We label this
feedback loop the project design validation loop. The requirements loop and the
design validation loop therefore represent the two complementary/eec¿-/br¿i’art¿
and feed-back directions of knowledge transfer between the client and vendor
organizations during the software development process.

The role of various classes of development coordination tools can be ana-
lyzed by considering the loop—requirements or design validation—in which
problems are more likely to arise under different combinations of novelty. The
classes of development coordination tools that are most likely to improve soft-
ware development performance are those that facilitate overcoming the knowl-
edge integration problems specific to that type of project. The prior literature on
development coordination tools is, however, very limited and does not provide
theoretical guidance on this issue. As noted earlier, much of the attention in

ACM Transactions on Software Engineering and Methodology, Vol, 17, No, 2, Article 11, Publication date: April 2008

11:6 • A. Tiwana

that body of work has been focused on specific tools rather than classes of tools
and almost none of the prior work has examined the relationships between tool
class usage and software development performance. For example, prior work
has developed the general implementation of a requirements traceability tool
such as REMAP [Ramesh and Dhar 1992; Ramesh and Jarke 2001]. It however
remains unclear how this tool generalizes to other requirements management
tools. Such emphasis on specific implementations of tools makes it difficult to
generalize to the broader performance implications of development coordina-
tion tools unless we move away from the numerous, specific tools and abstract
towards the core characteristics and the underlying functionality of generic,
holistic classes of tools. On the other extreme, other prior work has examined
the adoption (but not the performance impacts) of computer-aided software en-
gineering (CASE) tools [Orlikowski 1989,19931. CASE tools provide a plethora
of coordination functionality and such functionality often varies from one CASE
tool package to another. For example, some CASE tools support early stages
of the software design process (front-end CASE tools), others focus primar-
ily on the later development and testing stages (back-end CASE tools), while
others support the entire software development lifecycle (integrated CASE
tools). This coarse level of granularity makes it impossible to draw general-
izable inferences about their influence on software development performance
because different CASE tools represent different bundles of development coor-
dination functionality (which makes it difficult to compare across projects or
organizations).

To move past this dual problem of high specificity and lack of granularity in
the treatment of development coordination tools in prior work, we attempted to
abstract the distinct types of functionality that different classes of development
coordination tools can provide through an extensive set of in-depth interviews
in 19 software development companies. This step was a necessary antecedent to
the subsequent empirical field study. Because without a common set of develop-
ment coordination tool definitions, it would be impossible to draw generalizable
insights about development coordination tool functionality and software devel-
opment performance given the diverse variety of proprietary and commercial
tools that software development organizations used.

We developed an abstracted list of development coordination tool classes
using a grounded, semi-structured interview methodology (Phase 1 in
Figure 2) [Creswell 1994; Morgan and Smircich 1980; Strauss and Corbin 1998].
Grounded field research refers an inductive (observations -> to -> theory) ap-
proach where observations from the field are used to generate conceptual cate-
gories. (An in-depth discussion of the methodology appears in the next section.)
This involved three rounds of interviews with software project managers in 19
software development organizations and seven academic software engineering
and project management experts. These interviews were used to identify the
most commonly used classes of development coordination tools through quali-
tative pattern mapping [Todd 1979] using a grounded conceptual development
approach [Baskerville and Pries-Heje 1999; Glaser and Strauss 1967].

The key classes of development coordination tools commonly used in prac-
tice were identified and preliminary descriptions of these tools were created

ACM Transactions an Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:7

through this process. These descriptions were then refined following extensive
feedback from these software practitioners to ensure that the descriptions of
the six classes tools provide to the field study participants in the subsequent
questionnaire were interpreted consistently and without ambiguity [Mumford
et al. 1996; Sträub 1989]. The study participants commented on the clarity and
meanings of the tool class descriptions, definitions, and conceptualizations. In
our interviews, we found that there was variance in the tools that different orga-
nizations used to accomplish similar functions. Abstracting generalized classes
and descriptions of the key classes of development coordination tools used in
practice was the most critical yet challenging aspect of the questionnaire de-
velopment process. We resolved this challenge by moving away from specific
implementations of tools to instead providing descriptions of the functionality
provided by each class of development coordination tool. The final version of the
descriptions of the functionality of each class of development coordination tools
based on these iterative interviews is summarized in Tahle I. This grounded
conceptual development process helped establish the face vahdity and content
validity of the description of various classes development coordination tools
that were used in the subsequent questionnaire-based data collection phase
of the study. This questionnaire draft was pretested once again to ensure that
the respondents correctly and unambiguously interpreted the meaning of each
class of tools. This iterative refinement process was repeated until further im-
provements in the questionnaire appeared to be marginal [Sträub 1989]. Six
salient classes of development coordination tools emerged from this phase of
the study.

We focus our attention on these six classes of development coordination tools:
(1) requirements management tools, (2) architectural modeling tools, (3) test
automation tools, (4) test case development tools, (5) configuration management
tools, and (6) defect & change request tracking tools. To the extent that a class
of tools facilitates integration of fragmented technical and application domain
knowledge in a given type of project, it is likely to enhance software development
performance.

The salient knowledge integration challenges in routine (see Section
2.1), conceptually novel, process novelty, and multidimensionally novel (see
Section 2.2.) projects are discussed next (see Table II for a summary). This
discussion is then used to theorize how different classes of development coor-
dination tools might help overcome these challenges.

2.1 Routine Projects

Routine projects represent a well-definable problem space for which the initial
state, the goal, and a set of possible operations to reach the goal from the ini-
tial state are available [Robillard 1999]. Routine projects are therefore low in
conceptual and process novelty. Here, formal requirements facilitate convey-
ing client needs to the software development organization. In routine projects,
client requirements are more likely to be known with some confidence at the
outset of the project and interactions between the system and other subsys-
tems in the client organization can be identified early on. Further, the use of

ACM Transactions on Software Engineerir^ and Methodology, Vol. 17. No. 2, Article 11, Publication date: April 2008

11:8 A, Tiwana

Table I. The Six Classes of Development Coordination Tools and Their Roles in the Software
Process

Class of Development Coor-
dination Tools

Requirements managers

Architectural modelers
Test automation tools

Test case development tools

Configuration managers

Defect & change request
trackers

Definition and
functionality provided

Maintaining system and
subsystem requirements
and their
interdependencies.

Maintaining various
modeling views and
high-level design
information.

Executing the test cases
and generating defect
reports.

Helping create
system-level test-case
descriptions that confinn
whether the known
requirements are met.

Keeping track of various
versions of all ofthe
project’s artifacts and
providing a mechanis

for associating project
artifact versions and the
delivered version ofthe
system.

Allowing any stakeholder
to report problems or
suggest changes to the
system through a central
repository. Defining and
enforcing business rules
and decision authority
allocation for
incorporating various
tjipes of changes.

Role in knowledge
integration during
software development

Maintaining integrity
between the systems
design and knowledge of
evolving client needs.

Translating client
requirements into a
high-level software
design that accurately
reflects those
requirements.

Increasing the efficiency
with which the mapping
between client
requirements and the
in-progress system
artifacts can be
frequently assessed;
lowering the overhead of
testing the integrity of
system artifacts.

Ensuring that knowledge of
client needs as specified
in project requirements
is integrated in the
system being developed.

Tracing the lifecycle of
systems functionality to
prior versions of the
underlying design
artifacts and to the
underlying client
requirements.

Providing a centralized
mechanism for keeping
track of changes
requested by project
stakeholders and
enforcing authority for
making different types of
changes across the
development
organization and client
organization.

established development processes facilitates validation of the design by the
client organization while development is in progress [Kokol 1989].

Many ofthe classic project management concepts for software development
embody models that portray development as a sequential process of design fol-
lowed by development and testing [MacCormack et al. 20011. Effective software

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Fublication date: April 2008

Classes of Software Development Coordination Tools • 11:9

Table IL The Dominant Knowledge Integration Challenges in Different Types of Projects

Project Type

Routine

Conceptual novelty

Process novelty

Multidimensional novelty

Loop in which dominant knowledge integration
challenge exists (see Figure 1)

No dominant knowledge integration challenge
exists and the utility of development coordination
tools is primarily improvement-oriented.

REQUIREMENTS LOOP i.e., accurately conveying client
organization needs to the software development
organization.

DESKiN VALIDATION LOOP i.e., e n s u r i n g t h a t t h e
delivered system matches the client’s needs.

Challenges in knowledge integration across
REQUIREMENTS LOOP a n d DESIGN VALIDATION LOO

simultaneously exist.

development in these models is characterized by an approach that minimizes
changes to the design once coding has begun. A strong assumption in such
models is that all information about potential design choices is known or can
be readily discovered during concept development. Projects that either attempt
to develop a solution that is novel in concept or attempt to follow a new de-
velopment process present a fundamental challenge to such asserted models of
software development. A second challenge that this oversimplified view faces
is that feedback on the performance and functionality of the system is not ob-
tained until much later in the development process, when a large proportion
of the functionality has already been implemented. These are relatively less
challenging assumptions in routine projects.

Even though no dominant knowledge integration challenge exists in routine
projects, development coordination tools can enhance knowledge integration
across the feed-forward knowledge transfer loop by helping better translate
project requirements into a high level software design (e.g., using architec-
tural modeling and requirements modeling tools) and the feed-back knowledge
transfer loop by helping verify through appropriate test cases that this has
been acceptably accomplished (e.g., using defect/change request tracking, test
case development, and test automation tools). We therefore expect these all
six classes of development coordination tools will be associated with enhanced
development performance in routine projects.

2.2 Novel Projects

Knowledge Integration Challenges Common to All Novel Projects. In the pres-
ence of any type of novelty, the most challenging aspect of knowledge integration
is the client organization’s inherent difficulty in precisely spelling out their
novel project concepts through formal requirements at the outset of the project
(i.e., what we labeled the requirements loop ) [Rowen 1990]. The most likely error
in such projects is that initial, formal project requirements do not fully capture
the client’s needs and constraints [McAfee 2003]. This can happen for two possi-
ble reasons. First, the initial requirements might have been incomplete because
it is much more difficult to comprehensively articulate and formally transfer
such knowledge (e.g., in the form of project requirements) to the software

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 20

11:10 • A. Tiwana <:> • n r ‘

development organization [Tiwana and McLean 2005]. Second, as the client
organization begins to see preliminary development artifacts such as archi-
tecture representations, modeling diagrams, prototypes, and mock-ups, it may
recognize that the software development organization’s understanding of the
client problem domain is incomplete or imprecise. This can introduce new
requirements during development. The software development organization
must integrate such emerging information about client needs during the
development process to avoid the risk of proceeding to develop a system that
fails to meet the client’s needs.

We therefore expect that requirements management tools, which facilitate
managing requirements and interdependencies among requirements as they
evolve (or become clearer) over the project’s lifecycle will be associated with
higher software development performance in all types of novel projects. In ad-
dition, once test cases are developed, automated testing ofthe code throughout
the development lifecycle lowers coding errors, thus can be expected to be asso-
ciated with higher software development performance across all types of novel
projects.

Knowledge Integration Challenges Unique to Conceptually Novel Projects.
The dominant challenge in conceptually novel projects lies in accurately con-
veying the novel client requirements from the client organization to the soft-
ware development organization (i.e., across the REQUIREMENTS LOOP). Therefore,
coordination tools that facilitate associating evolving client requirements with
project artifacts (requirements management and defect/change request track-
ing tools) and maintain traceability across artifact versions and system func-
tionality (e.g., configuration management tools) can enhance software devel-
opment performance. Because requirements are more likely to evolve over the
course of conceptually novel projects, there is inherent value in being able to
forward-trace and backward-trace the evolution of different versions of various
project artifacts such as requirements, specifications, and the software modules
associated with them (configuration management tools). Furthermore, classes
of tools that facilitate testing (test case development and automation tools) can
also enhance software development performance by freeing up development re-
sources ordinarily dedicated to rote testing activities to instead focus on the
conceptually new problem space. Automating the execution of test cases and
frequent generation of test reports throughout the development process can
therefore lower the likelihood of cost overruns. Paradoxically, test automation
development coordination tools can increase the efficiency of intensive and fre-
quent testing but might also lower the likelihood oî finding and subsequently
removing more defects in conceptually novel projects (especially if the test cases
are inadequate), which might be refiected in lower development rework and
costs. Architectural modeling tools however can only limitedly enhance devel-
opment performance because the preliminary project requirements that serve
as inputs for constructing architectural models might themselves be incomplete
at the outset. Except for these, we therefore expect all other classes of devel-
opment coordination tools to be associated with improvements in at least one
dimension of performance in conceptually novel projects.

ACM TranBactiona on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:11

Knowledge Integration Challenges Unique to Projects Involving Process Nov-
elty. Use of novel development processes fsuch as a proprietary methodology
mandated by the client organization or a development process with which
the software development organization is not familiar) means that there is no
longer an established, mutually understood “syntax” through which the client
can monitor and provide feedback to the software development organization
during development. Although the uncertainties might be manageable from a
technical standpoint, the use of novel development processes makes it more dif-
ficult to verify that the in-progress design is compatible with the needs of the
client organization and reflects an architecture that best captures the desired
functionality and feature tradeoffs. It is useful to think in terms of degree of pro-
cess novelty instead of thinking of it in dichotomous terms. Therefore, projects
that require the use of many new processes at once or processes that span sev-
eral different parts of an organization have higher process novelty relative to
one that uses a minor modification of an existing development process. Lack-
ing a preestabhshed process for validation, the dominant challenge in projects
involving process novelty lies in validating that the artifacts generated by the
vendor correspond to the chent organization’s actual needs (i.e., the DESIGN VAL-
IDATION LOOP). The most effective classes of development coordination tools in
such projects are the ones that help: (a) integrate information about the in-
progress project artifacts generated in the software development organization
for client design validation (requirements management tools), and (b) man-
age interdependencies among client requirements, versions of project artifacts,
and system modules, which might be more difficult to establish without the use
of preexisting development processes (e.g., configuration management tools).
This allows the development team to trace the evolution of, say a requirement
or a user interface feature, without being overly dependent on the procedural
routines of the novel process. However, since a well-established, familiar soft-
ware process that both the client and vendor are familiar with does not exist
in projects with process novelty, successfully accomplishing the automation of
specialized development activities using such test case development tools and
change request trackers is likely more difficult. Therefore, we expect limited
benefits to be realized from the use of such classes of tools.

Knowledge Integration Challenges in Projects Simultaneously Involving Con-
ceptual and Process Novelty (Multidimensional Novelty). Projects with multidi-
mensional novelty face challenges common to both conceptually and procedu-
rally novel projects because both types of novelties simultaneously exist in the
project. Therefore, the classes of development coordination tools that facihtate
knowledge integration across both the requirements loop and design validation
loop are likely to be the most beneficial. Classes of tools that facilitate knowl-
edge integration in the REQUIREMENTS LOOP (e.g., requirements and defect/change
request tracking tools) as well as the DESIGN VALIDATION LOOP (test case develop-
ment and automation tools that facilitate efficient and frequent execution of test
cases and defect report generation) can thus be expected to enhance software
development performance. Architectural modeling tools however, encounter the
same bottlenecks as in conceptually novel projects, that is, lack of a stable and

ACM Transactions tjn Software Engineering and Methodology, Vol. 17. No. 2, Article 11. Publication date: April 2008

11:12 A, Tiwana

Table III. Summary of Development Coordination Tools Classes that Were Expected to
Influence Software Development Performance

Class ofTooJs
Requirements managers
Architectural nit)delers
Test automation tools
Test case development tools
Configuration management tools
Defect & change request trackers

Project Novelty Type
Routine

Conceptual

•

•
•

Process
•

•
•

Multidimensional

•

•
•
•

complete set of preliminary requirements on which to base the high-level de-
sign. This coupled with the lack of an established process makes it more difficult
to manage interdependencies among client requirements and versions of project
artifacts, limiting the benefits realized from using configuration management
tools.

In summary, although using various classes of development coordination
tools can enhance software development performance in routine projects, their
value in other types of projects is contingent on the type(s) of novelty present
in the project. The influence of a specific class of development coordination
tools will be more pronounced when its use facilitates integrating the type of
distributed knowledge that is problematic in that type of project. A class of tools
that might enhance the development process for one type of project might yield
no observable benefits for other types. Clearly, the dimension of development
performance that each class of development coordination tool enhances in each
type of project is an empirical question. The foregoing discussion is summarized
in Table III. The bullets indicate whether the each class of tool is expected to
be associated with an improvement in software development performance for
each type of project.

In the following section, we use the foregoing discussion to develop an em-
pirically testable model for assessing the impact of each class of development
coordination tools on software development performance.

3. MODEL SPECIFICATION

A system of regression equations in which software development performance
is modeled as a function ofthe level of use of each ofthe six classes of tools (de-
fined later in this section) was first developed. These were assessed empirically
using data from the projects in the study to test the relationships between tool
class usage and software development performance. The objective ofthe formal
models is twofold: (1) to assess whether each ofthe six classes of development
coordination tools significantly influences rework, cost overrun, and residual
defects and (2) to assess the direction and relative magnitude of this relation-
ship in routine, conceptually novel, procedurally novel, and multidimensionally
novel projects.

The model is specifled as a four-step hierarchical regression model where per-
formance is a function ofthe usage ofthe six classes of tools. Three such models

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:13

are estimated, one each for the three dependent variables—development re-
work, cost overrun, and residual defects. The first two represent proxies for the
efficiency dimension and the third the effectiveness dimension of software de-
velopment performance. Development rework refers to the percentage of project
function points that were changed during the testing stages of a project. This
represents development effort that was expended after the main development
stages of the project that ideally should not have been expended at all if the
coded system were perfect at the time of its delivery to the client organization.
Cost overrun refers to the percentage by which a project exceeded its planned
budget. Residual defects refer to the defects recorded in the post-delivery in-
stallation and warranty stages of a project. We chose to focus on these three
dependent variables because budget compliance, development effort and re-
work, and quality (measured as the absence of defects at the time of delivery)
are widely used metrics for software development performance.

Each equation specifies the dependent variable as a function of the extent
of use of the six classes of development coordination tools in a project. Each
equation represents a four-stage, incremental stepwise regression model. Each
of the four steps tests the effect of the six classes of development coordina-
tion tools incrementally in routine projects followed by conceptual, process,
and multidimensional novelty. The corresponding equation for the first depen-
dent variable rework is summarized in Equations (1) through (4), as follows.
In the first regression-modeling step, the usage data on the six coordination
tool classes were entered into the model, ie represents the error term in the
regression equation.)

In the equations that follow, the variables represent the following:

RMgr = requirements management tool usage level
Modeling = architectural modeling tool usage level
Test-auto = test automation tool usage level
Test-case = test case development tool usage level
Config — configuration management tool usage level
Tracking = defect and change request tracking tool usage level
Cnew = conceptual novelty
SPNew = software process novelty
MDNew = multidimensional novelty

Rework = ßo-i- ßrmgre ßm
+ ßtestjiutot^St JlUtO + ßtaase

+ ßconfigConßg -\- ßtrackingTracking -+- E (1)

Next, the effects of each class of development coordination tools in the pres-
ence of conceptual novelty were assessed by adding six interaction terms be-
tween conceptual novelty and tool class usage.

-\- ßteßtjiuto-ct^stJzuto*Cnew -\- ßtest^ase-c’^est-case*Cnew

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:14 • A, Tiwana

+ ßconfig-cConfig*Cnew + ßtrackingcTracking’Cnew. (2)

Next, the effects of each class of development coordination tools in the pres-
ence of software process novelty were assessed by adding six interaction terms
hetween process novelty and tool class usage. This interaction terms approach
allowed a finer grained assessment using levels of each type of novelty as op-
posed to a simple hinary classification approach that would inherently be less
granular.

+ ßconfig-spConfig*SPNew + ßtracking-spTracking* SPNew Í3)

Finally, the effects of each development coordination tool class in the pres-
ence of multidimensional novelty were assessed by adding six interaction terms
between tool class usage and multidimensional novelty (a dummy-coded vari-
able) involved in the projects.

+ ßtestMuto-mdtestjxuto”MDNew + ßtestxase-mdTest-case*MDNew

+ ßconfig-mdConfig^MDNew + ßtracking-mdT racking* MDNew (4)

The final model for rework t h e n becomes t h e following.

Rework = ßü-\- ßrmgrRiAgr + ßmodelingModeling

+ ßtest^utotest jauto + ßtestxaseTestjcase + ßconfigConfig

+ ßtrackingT racking + ßrmgr-c^^s^”Cnew

•^ ßmodeling-cModeling*Cnew . ^

+ ßtestjiutoctest Jiuto*Cnew + ßtggtxase-cTest xase*Cnew

+ ßconfig-cConfig*Cnew + ßtracking-cTracking*Cnew

+ ßrmgr-sp^^sr’SPNew + ßmodeiing-spg^odeling* S P New

+ ßtestjiutosptestJxuto*SPNew + ßtestxase-spTestxase^SPNew

+ ßconfig-spConfig*SPNew + ßtracking-spTracking*SPNew

+ ßrmgr-md^^Sr*MDNeW + ßmodeling-mdgModeling* MDNeW

+ ßtestMuto-mdtest jauto* MDNew + ßtestxase-mdTest xase*MDNeu}

+ ßconfig-mdCon fig* MDNew + ßtracking-mdTracking* MDNew + e (5)

Cost overruns and residual defects were predicted by similar regression
equations (6), (7) in which the dependent variable was respectively develop-
ment rework and cost overrun. The underlying logic for these equations mirrors
that of the equations for rework.

Costouerrun \\ Residual defects = ßo + ßrmgr^^sr + ßmodelingModeling

tjiutotest Jiuto + ßtest-caseTestxase ,

ßconfigConfig+ ßtrackingTracking

gr-cR^sr*Cnew + ßtnodeiing-Modeling* C new

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois 11:

Phase

Software teams survey

Phase 3
Client project evaluations

Interviews in 19 software
companies and 7 experts

Content analysis of
inteivievtfs to define

development tool classes

Refinement of preliminary
development tool classes

and their

definitions

Final development
coordination tool class

definitions

Survey sent to 808 software
project teams

Follow-ups

Empirical project-level data
from 232 soflware

development organizations

Client project evaluation
requests sent to 232
clients from Phase 2

209 project evaluations

Follaw-up intervievra with
five soflware developers

-Qualitative phase -Quantitative phase-

Fig. 2. Overview of study methodology.

ßconfig~cConfig*Cnew -|- ßtracking-cTracking*Cnew

• ßmodeiing-spgModeling* S P New
,auto*SPNew + ßte3t_case-sp’^6stjcase*SPNew
ig*SPNew + ßtracking-BpTracking* S P New

_auto-mdtest jauto’MDNew + ßtest -case-mdTest xase*MDNew
‘config-mdConfig*MDNew + ßtracking-mdT racking* MDNew + e (6)

The regression models were estimated using empirical data collected in the
field study and the coefficient estimates and tests for their statistical signif-
icance provide insights into which classes of tools provide what performance
benefits under each ofthe four possible tj^jes of project novelty.

The regression estimation approach facilitates statistically testing the rela-
tionships between tool class usage and development performance in the pres-
ence of various comhinations of novelty in the following manner. The first mod-
eling step was to assess the impact of development coordination tool class us-
age in routine projects, that is, without conceptual or process novelty. In the
first step, the “main effects” terms for the extent of use of the six classes of

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:16 ‘ A. Tiwana • ! ‘ . . . , •

development coordination tools are entered in the regression equation. This
corresponds to routine projects. Next, six interaction terms between each de-
velopment coordination tool class and a dummy variable for conceptual novelty
is entered in the model. This corresponds to conceptually novel projects. In
the third step, six interaction terms between each of the six development co-
ordination tool classes and a dummy variable for process novelty is entered in
the model to assess how they relate to development performance in projects
with process novelty. In the fourth step, six interaction terms between each
development coordination tool class and a dummy variable for multidimen-
sional novelty (simultaneous conceptual and process novelty in the project) is
entered in the regression model to assess how they relate to development per-
formance in projects with multidimensional novelty. (To create this dummy
variable, we tagged projects that had both conceptual novelty and process nov-
elty scores simultaneously rated at the highest level ofthe scales as 1 and others
asO.)

4. RESEARCH METHODOLOGY AND DATA COLLECTION

Figure 2 provides an overview ofthe three phases ofthe field study methodol-
ogy used in the study. The field study used a sequential multimethod research
design [Mingers 2001]. The “field study” label refers to the context in which the
study was conducted, that is, real-world software development organizations.
The multimethod approach refers to the use of two methods in a sequential
manner. In this study, the upfront phase of the study used a semi-structured,
interview-based qualitative methodology (phase 1). The results ofthe qualita-
tive phases were used to inductively derive “grounded” descriptions of various
classes of development coordination tools that were then used to conduct the
large-scale industrial survey using quantitative empirical methods [Baskerville
and Pries-Heje 1999; Strauss and Corbin 1998].

In phase 1, multiple rounds of in-depth interviews with 26 domain experts
(software project managers in 19 software organizations and 7 academic soft-
ware development experts) were conducted to identify and define the devel-
opment coordination tools used in practice. Having been drawn from four
different countries, the interviewees were representative of the diversity of
the eventual sampling frame of companies that were targeted in Phases 2
and 3. This was particularly important since similar concepts might be de-
scribed differently across software development companies in the four na-
tions that the study spanned. We interviewed software project managers in
six U.S. companies, four Indian, two Irish, and seven Russian software de-
velopment companies in this phase. All interviews were conducted in En-
glish. The interviews were conducted in person and over the phone. They fol-
lowed a semi-structured interviewing protocol outlined by Yin [1994] to elicit
identification and description of various classes of development coordination
tools used in the interviewees’ companies. Content analysis procedures were
used to identify questionnaire items to measure development coordination tool
usage [Lee 1989], which were then further refined through two rounds of addi-
tiona] interviews [Mingers 2001]. (The details of this procedure were described

ACM Tranfiactionaon Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:17

in detail in the eoneeptual development section of the paper.) The use of a qual-
itative front-end to develop the content for field survey is the convention for
field-hased surveys in organizations in the social sciences including informa-
tion systems [Mumford et al. 1996; Schwab 19801. The outcome of the first phase
of the study was therefore a set of grounded deseriptions of six classes of de-
velopment coordination tools that could be used unamhiguously and reliahly to
describe each elass of tools to a variety of software development organizations.
This formed the input for Phase 2 of the study.

In phase 2, project-level empirical data on tool class usage and archival de-
feet data for each projeet stage were eoUeeted from the lead project managers
in 232 software development companies in India, Ireland, and Russia. The
top exeeutives of 818 software development organizations in the three largest
global software consortia in Russia (Russian National Software Development
AlUanee), Ireland (via Irish Investment and Development Ageney), and India
(National Association of Software and Service Companies) were requested to
identify a major project that their company had completed within the last two
years for a U.S. organization. The lead project managers of each of these projects
completed the software development organization-side questionnaire and pro-
vided arehival defeet data for eaeh stage including unit-testing, system testing,
aeeeptanee testing, installation, and warranty stages of individual projects. In
this phase of the study, objective archived data were used where possible and
all other constructs in the study were measured on a 7-point Guttmann scales
using four to six questionnaire items. A Guttman scale (also known as scalo-
gram analysis) is a perceptual measurement instrument developed using the
Guttman scaling technique. A Guttman scale is a cumulative scaling approach
for measuring concepts that have varying levels. The primary objective of the
Guttman scaling is to ensure that the instrument measures only a single trait (a
property deserihed as unidimensionality wherein a single dimension underlies
the responses to the scale). The purpose of such a scale is to establish a one-
dimensional continuum for a concept being measured. For example, the seale
for eoneeptual novelty has four measurement items. As a respondent moves
from the first to the last item of the scale, it indicates that the level of eoneep-
tual novelty has cumulatively increased. In a Guttman scale, respondents who
agree with a more extreme scale item also agi’ee with all less extreme seale
items that precede it.

Data on the extent of use of eaeh of the six classes of development coordina-
tion tools were obtained from the lead projeet manager in eaeh software devel-
opment organization. Data on the outeomes of each project were obtained from
project-manager-reported and client-reported archival development records for
eaeh projeet. This multi-souree approach was used to mitigate the threat of
common methods bias that arises from obtaining perceptual data for both the
dependent and predictor variables from the same respondent in a single survey.
Appropriate steps were taken to ensure that the measurement scales measured
distinct variables and tapped into the conceptual domain of the variable (i.e.,
exhibited unidimensionality, face validity, and content validity) IBagozzi et al.
1991; Mumford et al. 1996; Sträub 1989]. Sinee all multiitem scales (tool elass
usage) were cumulative Guttman scales and performance variables were eaeh

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Pubtication date: April 2008

11:18 • A. Tiwana

measured as a single item, no alphas ICronbach and Meehl 1955] are associ-
ated with them [Schmidt and Hunter 1989]. The survey questionnaire items
for each construct are described in the next section. The data from Phase 2 of
the study were used as the hasis for collecting additional software development
performance data from the client organization in Phase 3.

In phase 3, soflware development performance assessments of cost over-
runs were obtained from client liaisons for each of these projects. Finally, after
the statistical analysis, we conducted additional interviews with five software
developers to validate and interpret some of the empirical findings. This key
informant technique of collecting data from a highly knowledgeable individual
in each organization is a commonly used approach in field studies [Kumar et
al. 1993].

The choice of the key informants for the survey was an important consid-
eration. The ohjective was to collect project-level data from someone who was
knowledgeable about all the development coordination tools that were used in
a particular project and less likely to be biased towards providing responses
that might appear to be socially desirable. Meeting these criteria required col-
lecting data from two different individuals from the client and vendor organi-
zations, further complicating the challenges of collecting field-based data. The
lead project manager for each project was an appropriate respondent for each
project because he/she is most likely to have good overall knowledge of the ex-
tent to whieh different development coordination tools were used in the project.
In contrast, project team members might have more knowledge ofthe tools that
they personally used in the project, but might not have accurate knowledge of
the extent to which ones that they themselves did not use but other project
team members did. The client-side liaison managers from the client organi-
zation were the most appropriate source of information about deviation from
budget because they were most likely to possess budget information as well
as less likely to underreport it. In contrast, collecting this information directly
from the vendor could have introduced reporting bias. Our preliminary inter-
views in Phase 1 ofthe study also confirmed this choice of informants.

4.1 Measures

The following section describes how the key constructs in the study were mea-
sured in the survey questionnaire. Development coordination tool usage was
measured for each of the six classes of tools using a six-item, seven-point
Guttmann scale. The descriptions shown in Table I were provided to the re-
spondents at the beginning of this questionnaire section to ensure that all of
them interpreted these tools in a consistent and unambiguous manner. The
scale of 1 to 7 measured the extent to which the following classes of tools were
used in the development process for each project, based on the lead project
manager’s assessment in each software development organization. The end
points of the scale were “not at a i r to “to a great extent” with “somewhat” as
an anchor for the midpoint of the scale.

(a) Requirements managers
(b) Architectural modelers . .

ACM Transactions on Software Engiaeeringand Methodology, Vol. 17, No, 2, Article 11, Publication date: April 2008

Classes of Software Developmeni Coordination Tools • 11:19

(c) Test automation tools
(d) Test case development tools
(e) Configuration managers
(f) Defect & change request tracking tools

Conceptual novelty (CNew) was measured using a four-item Guttmann scale
adapted from [Takeishi 20021. This scaled measurement approach provides a
granular measure for different the levels of novelty, where (1) below is the least
conceptually novel and (4) the most conceptually novel. Each level is coded as
one through four for statistical analyses. The advantage of this scaled mea-
surement approach over a discrete, binary (novel/not novel) measure is that
finer grained analysis is possible through multiple regression modeling and the
ability to construct discrete classifications is retained. Both continuum-based
values and discrete classification are later used for different facets of the sta-
tistical analyses. Information on this variable was collected from the software
development organization. The four questionnaire items for this scale asked
the software development organization to assess the novelty in the project ac-
cording to the description that best characterized the project from the following
four questionnaire items.

(1) Minor modification of a system design already developed by your company
(2) Major modification of a system design already developed by your company
(3) Completely new design, but based on a concept already demonstrated in

another project
(4) Technically new to your company and a completely new design

Software process novelty (SPNew) was measured using a four item
Guttmann scale based on Takeishi [2002] and Adler [1995]. Information on
this was collected from the software development organization. The four
questionnaire items for this scale asked the software development orga-
nization to assess the novelty in the project according to the description
that best characterized the project from the foilowing four questionnaire
items.

(1) Existing methodology and development tools used with MINOR modifica-
tions

(2) Existing methodology and development tools used with MAJOR modifica-
tions

(3) Either new methodology OR development tools, but based on existing ones
(4) Entirely new methodology AND new development tools

The efficiency dimension of software development performance was mea-
sured using rework and cost over m n as proxies and the effectiveness
dimension of performance was measured using the residual, post-delivery de-
fects that were recorded in the installation and warranty stages of a project as
a proxy. Rework was measured as the percentage of the total project function
points that were changed during the testing stages of each project. Information
on this was collected from the software development organization. Cost overrun

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:20 • A. Tiwana ^ • :

was measured as the percentage by which the project exceeded its planned bud-
get allocated by the client organization. Information on this was collected from
the primary project-liaison in the client organization. This approach mitigates
the methodological threat usually referred to a “common-methods bias” in sur-
vey research [Babbie 1973). This bias arises when the dependent variable data
(cost overrun) are collected from the same organization that might be biased
to underreport cost ovemm, that is, the vendor This problem does not exist in
ohjective (nonperceptual) variables such as recorded defects counts, total team
size counts, and allocated project schedule (in months). Residual defects rep-
resent the number of defects recorded in the installation and warranty stages
that followed unit-, system-, and acceptance-testing, that is, those that existed
in the system at the time of project delivery to the client organization. However,
since the number of defects can vary with project size, we accounted for project
person-months, which is a reasonable proxy for project size. Although project
size could have been measured in thousand line of code (KLOCs), our initial
interviews revealed that different companies differed on which of these metrics
they internally used. Project person months was therefore a common metric
applicable to all of the companies in the sample.

4.2 Response Rates

We received an overall response rate of about 28% and only the 209 projects
with matched pair data were used in the statistical analyses. We received com-
pleted sets of questionnaires back from 33.5% (59 received/176 sent) ofthe Rus-
sian, 29.5% ofthe Irish (54 received/183 sent), and 25.9% ofthe Indian vendors
(119 received/459 sent) respectively. 23 of these 232 projects had to be dropped
from the sample because of missing matched-pair data, leaving 209 projects
for the purpose of our analyses. We attribute the high response rate to target-
ing software development organizations belonging to three competing interna-
tional consortia, a comparison report custom-developed for each respondent’s
company in return for participation, multiple follow-ups, as well as extensive
confidentiality agreements. Additional checks suggest no persuasive threat of
nonresponse bias (this is discussed in detail later under Threats to Validity). At
the end of this phase, we interviewed five software developers to gain additional
insights into the results from our preliminary statistical analyses.

5. ANALYSIS AND RESULTS ” ‘:’ ” ‘•

5.1 Project Characteristics and Descriptive Statistics ‘

The projects in the study were custom application development projects. The
key characteristics ofthe projects in the sample are summarized in Tahle IV.

On average, the software development organizations in the study had heen in
the software development business for about 7 years, although this varied from
startup organizations to the oldest one at 47 years. In terms of their concep-
tual and process novelty, the projects were distributed according to the pattern

ACMTranBactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois 11:21

Table IV, A Summary of the Characteristics of the Projects in the Field Study

Project characteristic

Project duration

Project team size
Percentage of total project hours spent on the de-
sign phase of the project
Vendor CMM level

Rework

Residual defects
Project cost overrun
Project conceptual novelty (4 level scale)
Project process novelty (4 level scale)

Mean

11 months

16 people
20.7

15.7%
25.

16.4

2.9
1.9

Standard
deviation

9,6 months
22.9 people

8.47%
1.07
16.1%
61.09
16,43
0.9

1.05

Minor modlticarion Major modllicsiion Noval but darfvativs

Type ol Noveliy

Comptelely new

• Conceptual Novelty O Process Novelty

Fig. 3. Distribution of projects in terms of conceptual novelty and process novelty.

illustrated in Figure 3. The average level of conceptual novelty for these projects
on a scale of one to four was approximately 2.9 (standard deviation 0.94) and
process novelty 1.9 (standard deviation 1.05). Conceptual novelty was statisti-
cally significantly correlated with process novelty iß — 0.353, T-value = 4.46,
statistically significant at the 99% confidence level). This suggests that concep-
tually novel projects are also more likely to use novel software development
processes.

As the data show, approximately 40% of the projects in the sample were con-
ceptually novel but based on a preexisting design while only 10% were truly

ACM Transactions on Software Engineering and Methodology, Vol, 17, No, 2, Article 11, Publication date: April 2008

11:22 • A. Tiwana

Oelecfchange tracking

Conliguralton management

Test development

Te5t automation

Architectural modeling

Requirements maragement

Used somswtiai Used ex tensive iy

Extent Of Usage of the Six Classes of Development Cooröinalion Tools

Fig. 4. Average usage of each class of development coordination tool i n the sample. .

routine. 28% were completely new. However, 51% of these projects involved no
process novelty. The distribution pattern also shows that there was sufficient
variance in the sample to rigorously assess the statistical interaction effects
between novelty and development coordination tools [Baron and Kenny 1986].
The distribution pattern raises a cautionary note that relatively few projects
(about 9%) were at the upper end of process novelty. However, the majority of
projects (68%) in the study were relatively high on conceptual novelty. The ways
in which this distribution pattern might infiuence our findings is discussed later
in the article. Development coordination tool usage for various tool classes is
summarized in Figure 4. All six classes of tools appear to he used hut to varying
degrees. Surprisingly, requirements management tools that have the most con-
sistent performance impacts in the later results are relatively the least used.
This gives some credence to recent alarms that the software industry under-
utilizes even tools that are already available [Spinellis 2005J. An alternative
interpretation is that requirements management tools might have been used
extensively primarily in the early project stages, leading the respondents to
view its overall usage level as being lower relative to many other classes of
tools.

Table V presents a correlation matrix for the data. The correlation matrix
shows three noteworthy patterns in the correlations among various classes
of development coordination tools. These patterns indicate which classes of

ACM Tranaactáons on Softweire Engineering ¿tnd Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois 11:23

X
I

m
P

io
n

or
re

]

1. >

S
D

ta
>

i-H

1,
00

m
en

m
an

ag
er

s
PS

i-i

on
r-
co

îi
te

ct
u

ra
l

m
od

el
er

‘^

0
0

•

-H
t-H

.H
co

t a
u

to
m

at
io

n
to

ol
s

67
**

o
•

0.
30

in
o

CO
ou
CO

el
op

m
en

t
to

*
O

32
”

0.
20

>—1

(N
in
o

fi
g
u
ra

o
n

an
ag

em
en

o
o
r-

in
d

to
0^
o

0.
24

(N
fn

CO
00
TP

3c
t

&
c

ha
ng

e
re

u
es

t
tr

U
CD
.0
0

1—1

I—1

0.
0

0,
04

0.
17

o
to
T—1

•-H

ei
op

m
en

t
re

w
or

k
1.

00
37

**
o

m
o
o

in
d

co
0.

09
0.

07
.4

3
to
o

.—1

: o
v
er

ru
n

m
U

1,
00

-0
.0

în
(N
O

rg
>—1

o
o
0.
0
to
o

.2
3*

0.
00

08
CD
to

“”1
IM

îd
u

al
d

ef
ec

ce!
ai

ACM Transactions on Software En^neering and Methodology, Vol, 17, No, 2, Article 11, Publication date: April 2008

11:24 • A, Tiwana

tools were consistently used in conjunction with others and the correlations
in the extent of their use in the projects in our sample. First, there is a
statistically significant correlation between requirements management tools
and all other classes of tools with the exception of configuration manage-
ment tools. This suggests that requirements management tools were often
used in conjunction with other tools. A second noteworthy pattern is the high,
statistically significant correlation (0.67; p < 0.01) hetween test case de- velopment tools and test automation tools, which suggests a high likelihood that they the extent of use of one class of testing tools is directly correlated with the extent of use of the other. Third, these two classes of software test- ing tools have a moderate but significant correlation with defect/change re- quest tracking tools and configuration management tools. This suggests that they are likely to be used together with testing tools, although to a lesser level.

5.2 Choice of Multiple Regression for the Statistical Analyses

The independent variables in the dataset are ordinal data measured on mul-
tilevel scales that capture the extent to which each class of tools was used in
a project [Babbie 1973]. The dependent variables measuring software develop-
ment performance are continuous. Furthermore, the dataset is relatively large
by social science conventions and does not violate normal distribution assump-
tions. Assessing the relationships between various dimensions of performance
and usage ofthe six classes of tools requires that the effect and statistical sig-
nificance (i.e., assessing whether the relationships are real and not merely an
artifact of chance) ofthe six predictors be simultaneously accounted for in the
model [Hair et al. 1995]. In other words, analyzing the effect of one tool class at
a time in the absence of others would lead to neglecting the possibility that the
usage of one class of tools might defiate or innate the relationship with perfor-
mance for others that are not simultaneously included in a model [Aiken and
West 1991]. Furthermore, any analytic technique that assumes Poisson distri-
butions (e.g., log-linear regression) or dichotomous (e.g., logistic regression) or
categorical variables (e.g., multinomial logistic regression) would be inappro-
priate. Therefore, given the type of data and the objective ofthe analyses, the
most appropriate technique for statistically testing the relationships between
tool class usage levels and the performance variables is multiple regression
[Belsley et al. 1980; Cohen and Cohen 1983]. The choice of this regression-
based analytical approach is consistent with a large body of empirical field
studies using such data that have been conducted in the past three decades
in social science fields such as MIS that have also examined software develop-
ment teams and projects [Andres and Zmud 2002; Banker et al. 1998; e.g. Barki
et al. 2001; Faraj and Sproull 2000; Ho et al. 2003; Kirsch et al. 2002]. In this
approach, the usage ]evels of all six classes of tools are considered simultane-
ously in the regression runs. Researchers should be careful not to over interpret
the results as implying that a single tool by itself will infiuence performance
because the analyses rely on entering all six classes of tools in a single step.
Therefore, the effects of one class of tools might be contingent on the presence

ACMTransactioas on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:25

of others in the model. This issue is discussed in flirther detail in discussing
the interpretations of the statistical results.

5.3 Model Assessment Results

The three simultaneous regression equations mirroring (5), one each for rework,
cost overrun, and residual defects, were statistically estimated using stepwise
regression models using the data from all projects in the sample. The results of
the analyses are summarized in Tahle VI. Each set of columns in Table VI corre-
sponds to the three proxy variables measuring the efficiency and effectiveness
dimensions of software development performance. The dataset did not exhibit
any significant violations of the assumptions for multiple regression analyses
and did not require any variable transformations. The series mean substitution
algorithm was used for missing data points, as is conventional practice in such
analyses. Both listwise and pairwise regression analysis approaches were used
in the analyses. Each model was first estimated as a baseline model with all
variables included, followed stepwise by interaction terms with conceptual nov-
elty, process novelty, and finally multidimensional novelty. The results shown in
Table VI are for the full model, which must be used for drawing interpretations
[Aiken and West 19911.

Interpretation of the Statistical Results. The dependent variables in the three
results columns of Table VI represent the three different dimensions of software
development performance. The direction of the relationship between each class
of tool and software development performance is indicated by the unstandard-
ized regression coefficient, beta. The use of unstandardized beta coefficients is
appropriate hecause the model is an explanatory model rather than a predic-
tive model, where standardized betas would instead be appropriate IPedhazur
19821. Consider for example rework (Model A in Table VI). A positive (nega-
tive) beta sign for the rework column means that the use ofthat class of tools
is associated with increased (decreased) rework. The other statistic shown in
conjunction with each beta coefficient is a T-statistic. The T-statistic is a test for
whether we have sufficient confidence in a statistical sense that the observed
relationship is real and not by probabilistic chance. The minimal acceptable
norm for T-values is to have a 95% (i.e., p < 5%) or higher level of statistical confidence that the observed relationship is not by chance [Hair et al. 1995]. It is valid to draw conclusions only for beta coefficients with T-statistics with at least 95% statistical confidence. The T-statistic corresponding to a 95% confi- dence level is 1.69, 99% confidence is 2.36, and 99.9% confidence level is 3.17. The statistically significant relationships in Table VI are highlighted in bold type.

Only the results for development coordination tool classes in bold are
statistically significant and should be used for drawing interpretations. The
statistically-significant tool classes that are associated with lower rework, cost
overrun, and residual defects have a negative sign and are highlighted in
shaded box borders. These indicate improvements in software development
performance. We used one-tailed T-tests and a statistical significance level of
5% (i.e., 95% statistical confidence level) to denote statistical significance. Our

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publieation dat«: April 2008

11:26 A. Tiwana

Table VI. Impact of the Six Clsisses of Development Coordination Tools on Rework, Cost Overrun,
and Residual Defects

Development
Coordination Tool

STEP 1: Routine
Projects (Baseline
model)

Constant {ßo)
Requirements

management (ßrmgr)
Architectural modeling

Test automation (.ßtest^uio)
Test case development

Configuration management

Defect & change request
tracKing (ßtracktng’

Conceptually Novel Projects
STEP 2: Interaction terms

for conceptual novelty
CNew* Requi rements

management ißrmgr-c)
Cnew’ Architectural

modeling ißmodehng-c^

Model A

Rework

/3(T-statistic)

(4.114>

— 2 , 4 4 4 ” ‘ i —2.SOG i

^^^^Eßi 9 69SÍIt

4.887″‘f5.680)

2.600″n4.109)

.223<.375)

1.777″ (3.012)

.999* (1.998)

Model B Cost
Overrun

/3(T-statistic)

, ” r “M

(3.895)

fl^Hl^i-5ruo)i
.827(1.374)

5.894″‘(6.648)

1.808″‘(2.773)

.764(1.246)

2.983′”(4.908)

-.147(-.286)

Model C Residual
Defects

/3(T-statistic)

(-.823)
2.308″‘(2.451)

-,406(-.643)

la;621-“(-2.7ld
1 1.343(1.753)

.853(1.247)

-.092(-.170)

Cnew’Test automation

Cnew’Test case
development {ßtestj:ase-c)

Cnew’Configuration
management ißcnnfig-c)

Cnew’Defect & change
request tracking
\ßtracking~c >

Projects with Process
Novelty

STEP 3: Interaction terms
for process novelty

SPNew’Requirements
management {ßrmgr-md)

SPNew’ Architectural
moáe\mg{ ß„n)deUn-md)

SPNew’Test automation
‘Ptest .auto’md ‘

SPNew’Test case
development

SPNew* Configuration
management
\ßconfig-mdi

8 ^ ^ 1.290′(1.792)

3.314′” (6.075)

.X52(.188)

2.738″* (4.872)

.178(.33O)

2.198″ (2.396)

1.340(1.329)

4.547*” (4.811)

2.151*(2.218) 5.777″* (5.779)

– , 7 9 3 ( – 1 . 4 0 3 )

1.938′(2.219)

1.420U.303)

3.328-(2.607)

-.618(-.589)

-.647(-.617)

ACM Transactions on Software Engineering and Methodology, Vol. 17. No. 2, Article H.Ihjblication date: April 2008

Classes of Software Development Coordination Tools

Table VI. Continued.

11:27

Development
Coordination Tool

SPNew’Defect & change
request tracking
y “tracking—md ‘

Projects with
Multidimensional
Novelty

STEP 4: Interaction terms
for multidimensional
novelty

MultiNew* Requirements
m a n a g e m e n t [ßrmgr-md )

MultiNew’Architectural
modeling ißmiidetin-md’l

MultiNew’Test automation
ypiestMato-nui ‘

MultiNew’Test case
development
yptvut ̂ ase-md ‘

MultiNew’Configuration
management
^Pconfig-md ‘

MultiNew’Defect & change
request tracking
yytrackirig-md’

Model R^
Model F-value

Model A
Rework

.694( .620)

^1.962′(-2.286)

-.747(-.787)

1.990*(1.864)

^X^ââlài^hiSSÉÊk

1.599′(2.323)

.558(.646)

54.6%***
3.65″*

Model B Cost
Overrun

2.814’* (2.441)

^3.814”” ( ^ , 3 S H

3.095*'(3.ie7)

5.627*’*(5.113)

2.374*'(3.348)

-2.457* ( 2 . 7 5 7 )

60.4%*”
3.81’**

Model C Residual
Defects

-.313(-.259)

1 1.181(1.272)

– f>45(-,6291

\ 1.874*(2.020)

.O24(.O33)

,2I7(.232)

64.4%*’*
3.69***

1. • 95%. ” 99%, — 99.9% statistical confidence level; N = 209 projects.
2. Statistically significant relationships between tool class usage and performance are in bold.
3. Results in ^ ^ ^ ^ ^ ^ ^ H highlight tools that signiñcantly are asoociated with higher development perfor-
mance.
4. Negative signs indicate improvements and positive signs indicate decrease in software development perfor-
mance.
5. The raw data used for the regression procedure is in the Appendix.
6. SPSS”™’ version 11.5 for Windows was used to compute the unstandardized beta, traditional R ,̂ and T-
statistics.
7. SPSS 11.5’s Base”̂ ” version’s implementation of the aeries mean substitution algorithm with pair wise and
list wise regression procedures was used for estimation.

subsequent interpretations for some tool classes that—counterintuitively—
were associated with decreased performance are also discussed, guided in part
by our follow-up interviews with developers in the concluding phase of the
study.

For drawing interpretations from the analyses, first consider just Model A
(the first results column) in Table VI, which is the model for rework. In step 1 of
the evaluation procedure, the six classes of development coordination tools are
introduced in the model. This baseline model represents routine projects. Step 2
models the infiuence of each ofthe six classes of development coordination tools
when projects are conceptually novel. Step 3 models the infiuence of each of

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Pubhcation date: April 2008

1 1 : 2 8 • A . T i w a n a . -• – . . .

the six classes of development coordination tools when projects involve process
novelty. Step 4 models the influence of each of the six classes of development
coordination tools when projects involve multidimensional novelty. We focus the
later discussion primarily on the classes of tools that are associated positively
with improvements in performance hut also discuss possihle reasons for the
ones that are associated with decreases in performance (i.e., have a positive
and statistically significant coefficient). The results in the columns for Models
B and C in Table VI respectively correspond to cost overrun and residual defects,
and must he interpreted as described for rework.

The routine project regression results in Table VI correspond to projects
that do not involve novelty either in the underlying project concepts or in the
development process. The corresponding results inform us about the relation-
ships between performance and the usage level of the six classes of develop-
ment coordination tools in such projects. The subsequent results for concep-
tual, process, and multidimensional novelty should he interpreted in a similar
manner.

The amount of variance explained by each model, R̂ provides an indication
of each model’s goodness of fit. The model explained 54.6% of the variance in
rework, 60.4% of the variance in cost overruns, and 64.4% of the variance in
residual defects, suggesting that development coordination tool usage is a rea-
sonable but imperfect predictor of software development performance. As can be
expected, there was significant correlation between the three dependent vari-
ables. Specifically, cost overrun was significantly and positively correlated with
development rework {correlation coefficient, y — 0.373, p < 0.001) suggesting that higher levels of rework was associated with greater project cost overruns. It is important to note that cost overrun represents the additional cost that is passed on to the client and might not equal the actual cost incurred by the ven- dor. Likewise, rework was positively and significantly correlated with residual defects (y = 0.248, p < 0.027), suggesting that greater residual defects were as- sociated with more rework. Finally, residual defects did not have a statistically significant relationship with cost overruns, although the direction was nega- tive iy = —0.017, p < 0.452, nonsignificant). It is important to note that these relationships are correlations and do not establish a causal set of relationships. Establishing causality requires longitudinal data at different points in time for each project. However, the correlations between the three dependent variables does not confound the results because each dependent variable for each project type was analyzed as a separate regression model. We also tested whether the rework, cost overruns, and residual defects systematically varied depending on the presence of conceptual or process novelty. We found projects with process novelty are more likely to have defects remaining at the installation and war- ranty stages iy = 0.292, T-value = 2.072, p < 0.05). These results are discussed in the next section.

5.4 Assessment of Threats to Validity

Before proceeding to a discussion of these results, it is important to consider
our handling of five potential threats to the validity of the study. The first

ACM Transactions on Software Engineeringand Methodology, VoM7. No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:29

potential threat is nonresponse bias. Nonresponse bias refers to the poten-
tial bias that can result from the companies that chose to participate in the
empirical study vis-à-vis those that did not participate [Babbie 1973], To as-
sess whether the non-responding organizations were biasing our sample, we
made follow-up telephone calls to nine randomly selected nonresponding ven-
dors (three in each country). Most nonparticipating organizations declined to
participate for lack of time, concerns about sensitivity of the data, or because
they were no longer in the custom software development business. Further,
we also assessed differences between the early {first 50) and late (last 50) re-
spondents and found no statistically significant differences in organizational
characteristics such as company size and age. These checks suggest that non-
response bias is not a persuasive threat to our findings.

The second potential threat is that is key informant knowledge bias, which
refers to the ability of the responding managers to knowledgably provide an
assessment of the various classes of tools used in the projects that were stud-
ied. We assessed this threat in the interviewing phase (phase 1) of the study
through our interviews with project managers in 19 software development or-
ganizations. The lead project manager for each project was an appropriate re-
spondent for each project because he/she is most likely to have good overall
knowledge of the extent to which different classes of development coordination
tools were used in a project [Weekley and Gier 1989]. In contrast, individual
team members might have greater knowledge of the tools that they person-
ally used in the project, but might not have accurate knowledge of the extent
to which ones that they themselves did not use but other team members did.
Ideally, we would want to collect information on the use of each class of tool
from each team member for each project (and calculate an averaged team-level
score). However, this approach would require collecting data directly from ap-
proximately 3,712 individuals (232*16 team members on average in each team)
in 232 project teams, which was unfeasible given the limited access that we
had to the participating organizations and our resource constraints. The client-
side liaison managers from the client organization were the most appropriate
source of information about deviation from budget because they were most
likely to possess budget information as well as less likely to underreport it.
In contrast, collecting this information directly from the software development
organizations would have introduced a reporting bias [Shore et al. 1992]. There-
fore, respondent knowledge bias is not a serious threat to the validity of this
study.

The third potential threat is the nature of the contractual agreement between
the client and vendor companies, that is, whether the contracts were fixed price
or time-and-materials based [Gopal et al. 20031. In our initial interviews, we
found that software development organizations rarely use one or the other t3T)e
of contracts; instead, they are usually a combination of fixed price elements
coupled with time-and-materials components. Therefore, we did not specifically
control for this distinction from a statistical standpoint.

The fourth potential threat is that of construct validity, that is, whether the
measure for each construct actually measures what it was intended to mea-
sure. Two aspects of construct validity should be evaluated: (1) content and face

ACMTransactionsonSoftware Engineering and Methodology, Vol. 17, No. 2. Article 11, Publication date: April 2008

11:30 • A. Tiwana

validity and (2) level of granularity [Bagozzi et al. 1991; Lawther 1986; Paese
and Switzer 19881. We addressed the content validity threat hy grounding the
definitions of each class of tool in our interviews with actual software practi-
tioners and extensively refining them in the field to ensure that they were inter-
preted unambiguously and clearly hy the respondents in the larger-scale em-
pirical survey phase that followed. The second aspect of construct validity is the
granularity (i.e., aggregation across the entire project life cycle) at which tool
class usage was measured. Recall that we measured the degree to which each
class of tool was used over the life cycle of each project. An alternative would
have heen to provide a list of tools and collect data on each tool class’s us-
age. However, this would have generated data that could not have heen com-
pared across projects because: (a) the terminology used to describe various
tools varied widely among the two hundred plus software development orga-
nizations that we studied and many of the tools were proprietary (potentially
causing unverifiahle measurement errors across companies and projects) and
(b) many integrated tool environments such as proprietary and commercial
CASE tools provide different subsets of coordination functionality. Therefore,
no cross-organizational analyses would have heen possihie across the projects
in over two hundred different organizations without relying on a categorization
of tools into broader generic functionality classes, grounded in our conceptual
development interviews in Phases 1 and 2.

The final potential threat is that some tool classes such as test case devel-
opment tools would he expected to be used largely or exclusively only in a few
project phases. Tools that are tj^jically used in fewer phases of tjqDical projects
might have evoked responses on the lower end ofthe measurement scale. We
believe that this is not a serious threat to the validity of the study for two rea-
sons. First, classes of tools that are typically used in fewer phases of software
development projects would have generated a similar pattern of data, that is,
the respondents evaluated tool class usage relative to what is typical in their
view ofthe industry. Since regression analysis is hased on variance in the data
rather than absolute values, this would not systematically hias the results. The
second piece of evidence to support this assessment appears in Figure 4, which
shows that test case development, test automation, and architectural modeling
tools, which are typically used in fewer phases of the software development
process, are all around the middle point ofthe scale (i.e., neither too high nor
too low, on average). -r.- . -i

5.5 Limitations

This study has six limitations that warrant caution before generalizing the re-
sults. Unlike the threats to validity discussed in the preceding section, these
are limitations of the dataset that cannot completely he ruled out. First, us-
age of various classes of development coordination tools in each ofthe projects
analyzed in the study was hased on the lead project manager’s assessment.
This was a necessary trade-off in order to collect a large dataset in a multina-
tional setting from over 200 different companies in this study. It would he valu-
able to collect data that are more granular from each project team members in

ACM Transactions onSoitware Engineering and Methodology. Vol. 17, No. 2, Article 11, Publicationdate: April 2008

Classes of Software Development Coordination Tools • 11:31

future studies. The model assumed that the impact of various classes of devel-
opment coordination tools is linear. This assumption, while appropriate in this
study, should be refined in future research. For example, future research can
collect longitudinal data at multiple points in each project and examine
whether some classes of tools (e.g., requirements managers) have curvilin-
ear relationships with performance as a project progresses through differ-
ent stages. Future work can also provide deeper insights by directly mea-
suring the frequency and person-hours of use for each tool, which would be
much more insightful than a simplistic perceptual assessment used in this
study.

Second, the projects in the study were small to midsized projects. It is pos-
sihle that the influence of development coordination tools will be different in
large-scale projects. Since the complexity ofthe projects was not directly taken
into consideration and there appears to be little variance in project size (a
surrogate for complexity), caution is warranted in generalizing the results to
complex projects. It is plausible that smaller projects could also be highly com-
plex, requiring directly measuring and modeling project complexity in future
studies of development coordination tools. In interpreting the results, it is also
important to remember that the beta coefficients represent the effects of tool
classes when all six classes are copresent in the model. In other words, drawing
interpretations for one class of tools in isolation would be too simplistic and
potentially misleading because each effect represents the performance effect
from the usage of a given class of tools when the other five classes are also
simultaneously accounted for in the model. Merely regressing one tool at a
time would not have isolated the unique effects of a given class of tool either
because some tools are used more in specific phases of the software develop-
ment process and it would be almost impossible to ascribe causality to that
class of tools unless the level of usage of all other classes of tools was identical
across all projects heing compared. It might, however, be possible to exam-
ine this issue in future research through a laboratory experiment in which
only one tool class is varied across an experimental group and control group
of projects. This approach will, however, introduce the trade-off that the study
context would have to be a classroom setting rather than a real world industrial
setting.

Third, residual defects were measured using recorded installation and war-
ranty defects for completed projects. Not all existing warranty defects might
have been detected at the time of the study. This is not a persuasive threat
to our findings because if this is the case, the study underestimates rather
than overestimates the effectiveness of various classes of development coor-
dination tools. However, it is also possible that the undetected defects should
not have been present in the first place, which is impossible to examine in the
data. A related limitation is that we did not measure the severity of recorded
defects, in essence treating all recorded defects as equal irrespective of their
magnitude. This remains a promising refinement opportunity in future studies.
Prototyping tools did not explicitly emerge in the preliminary interviews and
prototyping functionality appears to have been subsumed imder other classes
of development coordination tools.

ACM Transactions on Software Engineering and Methodology, Vol, 17, No, 2, Article 11, Publication date; April 2008

1 1 : 3 2 • A . T i w a n a • ‘ • •’•*^’-^ •

Fourth, the study was cross-sectional (i.e.. collected at a single point in time)
rather than longitudinal, which makes it impossible to statistically establish
causality, that is, whether the use of a given tool class causes or is merely as-
sociated with improved performance. For the same reason, the analysis fails
to account for learning curves in using development coordination tools; it is
possible that some classes of tools did not generate observable benefits in
projects with process novelty and multidimensional novelty because the de-
velopment team had to go through a learning curve to benefit fully from
their use. If the programmers do not become sufficiently familiar with a new
tool, that tool is less likely to be effective in facilitating development per-
formance improvements ISeeley 2003). Therefore the long-term versus short-
term impact of various classes of tools remains an important issue for future
field studies; these must be conducted at the software development organi-
zation rather than project as the empirical unit of analysis. Another fruitful
avenue for future research is to examine how the level of usage of various
classes of tools varies with other characteristics of projects such as budget and
duration.

Fifth, there is a wide variance in the characteristics ofthe projects and the
organizations—such as project schedules, team size, design effort, vendor CMM
level, national contexts, and age ofthe vendor—surveyed in this study. The risk
that variations in those contextual variables might confound some effects of de-
velopment coordination tools cannot be completely ruled out. However, there
is also one commonality that lessens the severity of potential confounding ef-
fects: All of the projects were completed for US clients. One can reasonably
presume that foreign software vendors who have completed projects for client
companies in the same country might have sufficient commonalities in their
characteristics that at least some of this threat would be mitigated. A second
aspect that should also he considered is that the unit of analysis in our mod-
els is the project rather than the vendor organization; therefore, inclusion of
other variables at different levels of analyses would not be methodologically
permissible.

6. DISCUSSION

The overarching implication of the results is that the performance benefit of
any class of development coordination tools is contingent on the salient types
of novelty in a project. The majority of development coordination tools are
associated with some dimension of higher performance but the dimension of
performance that they enhance—efficiency (e.g.. reducing rework, and pre-
venting cost overruns) or effectiveness (e.g., reducing software defects)—varies
systematically based on whether a project involves conceptual novelty, pro-
cess novelty, neither, or multidimensional novelty (both conceptual and pro-
cess novelty). The results provide nuanced insights—some counter to pre-
vailing beliefs—into the portfolio of development coordination tool classes as-
sociated with higher software development performance in various types of
projects.

ACMTransactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:33

The overall pattern of results for the influence of each class of development
coordination tool on software development performance is summarized in Table
VIL (These patterns are hased on the statistical results in Table VI.) Here,
rework reduction and prevention of cost overruns represent proxies for the
efficiency dimension of software development performance and reduction in
post-delivery defects as the effectiveness dimension.

Three overarching patterns in Table VII are noteworthy. First, the use of
many classes of tools in any type of project is associated with increases in some
dimensions of performance but decreases in others. It appears that software
development efficiency is achieved through the use of some classes of tools at
the expense of software development effectiveness and vice versa. This can he
viewed as an efficiency-effectiveness trade-off. Consider requirements manage-
ment tools as one example. In routine projects, such tools are positively associ-
ated with efficiency but negatively with effectiveness. In projects with only one
type of novelty, such tools are associated with improvements in effectiveness but
decreases in efficiency. Test case development tools and test automation tools
also reveal similar patterns. Second, architectural modeling tools and configu-
ration management tools are exceptions to this in that they do not introduce
an efficiency-effectiveness tension in any type of project. Third, classes of de-
velopment coordination tools that are associated with higher performance in
multidimensionally novel projects—counter to the intuitive assumption—are
not a perfect superset of those that are associated with higher performance in
projects with conceptual and process novelty.

We discuss next the findings related to each class of development coordi-
nation tools for projects with no novelty, process novelty conceptual novelty,
and multidimensional novelty. Our interpretations are based on the sign of the
path coefficients (i.e., beta values, which indicate the direction of influence of
each class of development coordination tool on each of the three performance
variables) as well as its statistical confidence (significance) level in Table VI.
Caution should be exercised in interpreting these results as being causal rather
than associative; the former requires longitudinal data over multiple periods for
each project and ruling out of alternative hypotheses [Duncan 1985]. Causal-
ity is implicitly assumed but it cannot be statistically tested with our cross-
sectional data.

The following discussion references beta values, that is, the sign of the path
coefficients in the regression model that descrihe the direction of the relation-
ships hetween tool class usage and software development performance. The
letters in the heta subscripts in Tahle VI and the following discussion denote
the class of tools and the type of project that the heta value represents sepa-
rated by a dash. For example, ßrmgr-r represents the beta coefficient for require-
ments management tools for routine projects, matching the beta labels used in
Table VI.

6.1 Requirements Management Tools

Requirements management tools were significantly associated with higher soft-
ware development performance along at least one of the three dimensions of

ACM TranBactions on Software Engineeringand Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:34 A. Tiwana

p-
%

‘Z
J J

tí

H P H
C
01

id
im

3
S

CL.

“ffl

IL
U

Q- Q
O ^

-F
Oi.O

“o c

s 3 CJ

unj.i3AQ leoQ

J[J0M9y

unjJSAn î s o n

813313(1 renpisaji

irruJ3.\0 isoo

îjjoMsy

.,»,ea,Bnp.ea

-4

-4
-4

j ^

•4

>
•4
•4

tí
d

eq
u

ir
em

‘4

t .

rc
h

it
ec

‘4
>
‘4
>
‘4
‘4
>
>

O
“‘^

O
J-1

es
t

au
to

m
-4
-4
L>
j ^
•4
^

• 4

t/j

to
o

tí

Fa
_u

es
t
ca
se
d
-4
‘4
‘4
>
>

to
o
ls

m
en
CO
ti

on
on

fi
gu

ra
t

U
<

>
• 4
-4

ra
ck

er
s

u
es
t

bn
u
ef
ec

t
an

t á

s be ^
ÍS S ^
3 -S J :

te
d

O
C

l
m

S
ra

“§
to
0)

ra
d ^

aï
111

.S
•4

ra
•5

.S
>

ACM Transactions on Software Engineering and Methodolt^y. Vol. 17, No. 2. Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11 ;35

performance across all four types of projects. However, the nature of their im-
pacts varied across project types: Their use was associated with lower rework
and cost overrun in routine and multidimensionally novel projects (as indi-
cated hy the negative and statistically significant ßmgr-r and ßrmgr-md in Ta-
hle VI), hut with fewer residual defects in projects with only a one type of
novelty (ßrmgr-sp and ßrmgr-c in Tahle VI). Conceptually, this implies that the
capahility to maintain requirements and interdependencies between require-
ments provided hy requirements management tools enhances development
efficiency in both routine and multidimensionally novel projects. In projects
with only one form of novelty (i.e., either conceptual or process novelty),
requirements management tools enhance the effectiveness dimension of
performance. However, their use appears to he associated with increased cost
overrun and rework in such projects, suggesting that cost overruns might he as-
sociated with greater defect detection and their suhsequent elimination during
the development process. Another interpretation for this pattern of results is
that requirements management tools help maintain interdependencies among
interacting requirements in singularly novel projects, which lowers residual de-
fects but that a direct cost overhead might he associated with the comprehensive
and formalized requirements management process imposed hy their use.

6.2 Architectural Modeling Tools

Architectural modeling tools were associated with significantly lower devel-
opment rework in routine projects (suggested by a negative and statistically
significant ßmodeiing-r term in Table VI). This finding is consistent with the
logic in Section 2 that a good high-level design and a sound architecture that
such development coordination tools facilitate enhances software development
efficiency when requirements are relatively well definahle and the envisioned
outcome can he more completely specified at the outset ofthe project. Architec-
tural modeling tools are associated with a decrease in cost overruns in projects
with process novelty. An interpretation for this is that hy using these devel-
opment coordination tools to coordinate the high-level design/architecture of
the software frees up development resources to execute the relatively novel
process that implementing it entails, in turn enhancing software development
efficiency. We found no statistical evidence that such tools enhanced either
software development efficiency or effectiveness in conceptually novel or multi-
dimensionally novel projects. One interpretation for this is that in conceptually
novel projects, architectural modeling functionahties do not have a significant
infiuence hecause the needs that the project must fulfill are incompletely known
at the outset. Instead, a more evolutionary approach is required to specify iter-
atively the high-level design ofthe system. In projects with multidimensional
novelty, tools that facilitate high-level architecture might impose preexisting
design patterns that they natively support and might lead the development
team to make design assumptions that follow from preexisting development
processes but do not necessarily correspond to the new concepts that the project
is intended to implement. It is impossihle to deduce from the availahle data
whether the higher number of residual defects found would have heen missed

ACMTransactionson Software Engineering and Methodology, Vol, 17, No. 2, Article 11, Publication date: April 2008

11:36 • A. Tiwana

without the use of such development coordination tools. This issue is critical for
understanding whether architectural modeling tools ought to he used in novel
projects deserves further examination in future research. Future research is
also needed to examine whether there are other benefits (such as client sat-
isfaction with the delivered system) that arise from their use, especially in
complex projects.

6.3 Test Automation Tools

Test automation tools were associated with a higher level of at least some di-
mension of software development performance across all four types of projects,
although they also introduced efficiency-effectiveness tradeoffs in three of the
four types of projects. In hoth routine and muitidimensionally novel projects,
their use was associated with fewer residual defects (indicated hy the nega-
tive and statistically significant ßtestjiuto-r and ßtest^uto-md in Table VI). This
suggests that in routine projects, automated execution of test cases lowers the
incidence of defects. Surprisingly, their use also is associated with increased
rework and cost overruns in such projects. An interpretation for this is that
the additional cost and rework might result from the increase in defects that
they help detect during the development process and that need to be fixed.
It is also possible that the additional cost of test automation tools will arise
from the learning curve that the developers might have to go through before
fully benefiting from their use. The cross-sectional nature of our data, how-
ever, constrained our ability to assess empirically this possibility. In projects
with either conceptual or process novelty, the use of test automation tools was
associated with both lower rework and lower cost overruns (indicated by neg-
ative and statistically significant ßtest^uto-c and ßtestMuto-sp in Table VI). One
interpretation for this finding is that automation of test case execution in the
presence of a singular novelty frees up developers’ attention from rote testing
activities to the more creative aspects such as the novel conceptual or process
framework, thereby increasing the efficiency dimension of software develop-
ment performance. Surprisingly, their use was associated with greater resid-
ual defects in projects with process noveity. An interpretation for this is that
automated testing might provide more frequent opportunities for detecting de-
fects in the simultaneous presence of novel software development processes.
A similar pattern for conceptually novel projects might further suggest that
automation of testing activities can plausibly increase the incidence of defects
when the novelty of project concepts hinders the ex ante formal specification of
project details.

6.4 Test Case Development Tools

Test case development tools help create system-level test-case descriptions that
confirm whether project requirements are met. Test case development tools
therefore serve a validation function that ensures that the system level tests
can be mapped back to the known requirements, thus reducing later rework
and the costs associated with it. Their use was associated with higher develop-
ment performance along at least one dimension in all projects except projects
with process novelty. They enhanced efficiency in routine (see ßtest^ase-r in

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools • 11:37

Table VI) and multidimensionally novel projects, and effectiveness in projects
with conceptual novelty ißiest^ase-c in Table VT). Overall, the performance bene-
fits of test case development coordination tools appear to be most pronounced in
routine and multidimensionally novel projects, with somewhat less pronounced
benefits in conceptually novel projects. An interpretation for this difference
is that effective use of such development coordination tools requires that the
requirements be relatively well known and relatively stable from the outset,
which is less likely in conceptually novel projects. In the case of multidimen-
sionally novel projects, they perhaps help cope with the sheer complexity of
developing test cases in the simultaneous presence of conceptual and
process novelty. However, test case development tools introduce the
aforementioned efficiency-efîectiveness trade-off in all types of projects except
routine projects; that is, they lower some dimension of performance while in-
creasing others. They lower efficiency in projects with singular novelty (see
ßtest^ase-c ^nd ßte.it^’.ase-sp) ^nd lower effectiveness in projects with multidimen-
sional novelty (see ßtesi-case-md)- An interpretation for the former is that the
absence of precise upfront project requirements makes their use less efficient
in conceptually novel projects and mapping the known requirements to test
cases in projects with process novelty is more time consuming because of the
absence of a preestablished development process. Another plausible interpre-
tation is that such tools assume an established software process (which is not
the case if the software process itself is novel), leading to critical errors in the
system going undetected while executing the resulting test cases. Correcting
these defects later in the process then imposes additional costs and rework. A
competing interpretation is that additional rework and costs arise from the cor-
rection of defects that such development coordination tools help identify. How-
ever, this interpretation is less plausible because the use of such tools were not
significantly associated with fewer residual defects. An interpretation for the
tradeoffs observed for multidimensionally novel projects is that although such
tools facilitate efficient development of test cases, this efficiency comes at the
cost of comprehensiveness and precision in developing them.

6.5 Configuration Management Tools

Configuration management tools facilitate keeping track of associations among
various versions of all of the project’s artifacts throughout the development
life cycle. The use of configuration management tools was associated with
higher development performance only in projects with a singular type of nov-
elty (ßconßg-c and ßconfig-sp)- No efficiency-effectiveness trade-offs were observed
in such projects. The capability to associate versions of artifacts with builds
ofthe system throughout the development process helps maintain consistency
with evolving requirements that emerge as development progresses in concep-
tually novel projects. This in turn proactively mitigates potential mismatches
between the most recent versions of requirements, specifications, and system
artifacts associated with them, thereby lowering the need for additional re-
work to correct the ensuing defects. Projects involving novel development pro-
cesses also benefit from the use of configuration management tools because

ACM Transactions on Software Engineering and Methodology, Vol, 17, No, 2, Article 11. Publication date: April 2008

11:38 • A. Tiwana

they help coordinate the mappings among requirements, specifications, and
system artifacts relatively independently of the development process. This in-
creases the efficiency with which traceahility information can he maintained,
reducing the extent of cost overruns and additional rework. Although novel
projects appear to benefit from the use of configuration management tools,
their use is associated with lower software development efficiency in routine
Eind muitidimensionally novel projects. An interpretation for the former finding
is that keeping track of various versions of artifacts imposes an unnecessary
overhead when requirements are stable and might have fewer versions rel-
ative to singularly novel projects. An interpretation for the latter is that it
is too complex and time consuming to keep track of requirements using such
tools when requirements have too many versions because of evolving require-
ments and the simultaneous absence of an established software development
process.

6.6 Defect and Change Request Tracking Tools

Defect and change request tracking tools were associated with higher software
development effectiveness in routine projects and with higher software devel-
opment efficiency in multidimensionally novel projects, without any efficiency-
effectiveness trade-offs. In conceptually novel projects, they were associated
with higher software development efficiency but also with a greater incidence
of residual defects. An interpretation for this finding is that in conceptually
novel projects, where project requirements are incompletely known at the out-
set, defect & change request tracking tools facilitate rigorous tracking of project
requirements as they evolve or change, plausibly increasing the likelihood of
detecting more defects in the design of the software. Our interviews also sug-
gested an additional interpretation for why more residual defects might exist:
The functionality to enforce assignment of authority for incorporating various
types of changes to the system puts certain changes out of the scope of some
team members’ work authority and access rights/ privileges. Our interviews
also suggested that at times a defect is observed hy a team member in one of
the stakeholder organizations but the inability to report it in a timely manner
due to the authority allocation rigor enforced by such tools can lead to its go-
ing unrecorded until much later in the development process. This constraint
in turn leads to defects being detected yet remaining in the system, which in
turn can increase the incidence of defects at the time of delivery. It is also note-
worthy that the use of defect and change request tools was skewed towards
higher end of the scale. An interpretation for this is that in the outsourcing
context in which the study was conducted, software development organizations
are likely to use these development coordination tools more for two additional
reasons: (1) they serve a bookkeeping function for recording defects for CMM-
type certifications and (2) they might create an improved perception of quality
management practices among clients. In projects with process novelty, the use
of such tools is associated with increased cost overrun. An interpretation for
this result is that they provide an elaborate mechanism for recording defects
and change requests but the absence of a well-established process hinders the

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools 11:39

ToolOaBS

Recpjirements managers

A tchiteciural modelers

Test auion^tbn tools

Test case developnwnt tools

Conligutation managementtools

Oetecl& change request trackers

Project Nove ly Type

Rxitirte

O
•
0
•

•
Conceptual
0

0
•
0

R’ooess

0
•
0
•
Multidimensional
•
0
0

•
Notes: 0 denotesperformancebenetits with efticencv-edecttvenesstradeofts;^ benefits withouttiadeoMs.

Fig. 5. Overall pattems of the performance implications of the various classes of development
coordination toolB.

efficient incorporation of such requests into the development process, in turn
lowering software development efficiency.

6.7 Summary and Implications for Software Development Practice

The most important insight for practice from these results is that all classes of
development coordination tools except architectural modeling tools and config-
uration management tools introduce efficiency-effectiveness trade-offs in some
types of projects. Consistent with what has been suspected [Seeley 2003], a
mismatch between the choice of tools and type of project can therefore worsen
some dimension of software development performance even when it is asso-
ciated with improvements other dimensions. This is what we labeled as the
efficiency-effectiveness trade-off. Figure 5 sjnithesizes the pattem of results from
the statistical analyses to illustrate the trade-offs across projects involving dif-
ferent combinations of novelty.

To summarize, classes of development coordination tools that are associated
with improved software development performance without trade-offs in specific

of projects are the following.

—In routine projects, architectural modeling, test case development, and de-
fect/change request tools are associated with higher software development
performance.

—In conceptually novel projects, configuration management tools are associ-
ated with higher software development performance.

—In projects using novel software processes, architectural modeling and config-
uration management tools are associated with higher software development
performance.

—In multidimensionally novel projects, requirements management and de-
fect/change request tools are associated with higher software development
performance. The classes of development coordination tools that are associ-
ated with higher performance under multidimensional novelty are therefore
not a true superset of conceptual and process novelty, as commonly asserted
in practice.

ACM Transactions on Software Engineerii^and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:40 • A. Tiwana ‘ ^̂

Software development organizations therefore must first carefully consider
the characteristics of each individual project and then select a set of devel-
opment coordination tools are more likely to have the greatest impact on its
development process. .

7. CONCLUSIONS

As hoth the complexity and novelty of software projects continue to increase,
a growing number of software development organizations are adopting de-
velopment coordination tools in an attempt to improve development perfor-
mance and productivity. In the ahsence of any prior field research on when and
how such tools improve software development performance, many organiza-
tions have leaned towards adopting a broad array of different tools under the
presumption that they can only enhance software development performance.
Others have avoided extensively adopting such tools because some industry
observers have cautioned that it might he wiser to not use a tool than to use the
wrong tool [Brown 2003; Spinellis 2005]. The objective of this early field study
was to assess empirically the relationship between the use of various classes of
development coordination tools and software development performance in the
actual industrial context in which they are used.

We first identified and conceptually delineated six distinct classes of software
development coordination tools used in the industry based on in-depth inter-
views in 19 software development organizations. We then used the results of
this phase to assess empirically the relationships between the use of these tool
classes and software development performance using primary field-based data
collected from 209 different software development companies. We statistically
analyzed this data through regression analysis techniques.

Overall, the results provide new insights into the relationship between the
use of various classes of development coordination tools and software devel-
opment performance. The results reveal two broad patterns. First, the di-
mension of performance that each class of tool impacts—that is, whether it
mitigates development rework, controls budget slippage, or incidence of post-
delivery software defects—depends on whether the project involves concep-
tual, process, or multidimensional novelty. A given class of development co-
ordination tools that is beneficial in one type of project can have no impact—
or worse, a negative impact—in another. Second, the use of many classes of
tools introduces efficiency-effectiveness tradeoffs; that is, it can enhance one
dimension of performance while lowering another. We found two exceptions—
architectural modeling tools and defect/change request-tracking tools—that do
not impose such tradeoffs. Although this study provides an initial step, consid-
erable potential exists for further fieldwork that can expand our understand-
ing about how development coordination tools enhance software development
performance.

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools 11:41

APPENDIX: RAW DATA

Usage of six classes of development
coordination tools

1 = “^ot at all,” 4 = “somewhat,” and 7 = “to a
great extent.”

T l
1
6
3
1
3
3
5

3
6

3
7

3
3
4
1
1
1
3

1
4

3
3
1
3
3
1
1
3
3

5
1

I-H

1
3
3
1
2
6

2
4

3
3

5
2

2
1

4
4

1
4
4

5
4
2

4
5

4
4
4
4

1
1
4

1
4
3
5

1
3
4
7

4
4
4
5

1
5

1—
1

1
4
4
1
3

5
5

4
1
4

1
4
1

4
4

5
4
1
4
7
4
4
4
1
1

1
4
7
4
4
2

5
3

4
6

1
4
4
2
3
5
1
1
4
4
1
4
6
6
6
4
4

1-H

T4
1
6
4

2
4

4
5
4
1
4
7
4
4
4

1
1
1
4

1
4

4
2
5
3
4
6

1
4
4
4
3
5
1
1
4
4
3

5
6

6
4
4
4
6
3

1
7

5
1

5
5
4
5
6
5

5
5
4

6
1

5
1

4
4
4
7

3
5
2
1
5
5
3
1
5

1-H
1
5

4
5
2

7
2

5
5
6
5

T6
7
1
5
1

5
5
4
5
6
5

5
5
5
4

1
7

5
5
4
4
4

5
7

3
5
5

5
5
5
4
2
5

I-H
1
5

5
3
5

7
7

5
5
6
4

Dimensions of Software
Development Performance
P I
16
30
16
16
16
16
0
16
50
16
16
16
16
16
16
30
16
16
50
16
16
16
16
16
16
5

5
16
16
40
16
16

5
16
16
16
0
0
16
16
16
16
16
16

P2
10
15
16
16
16
16
16
16
5
16
16
16
16
16
16
10
30
16
80
0

20
16
16
16
16
40
0
16
16
50
16
1

16
20
16
16
20
16

5
4
16
16
16
10
10

P 3
40
8

25
25

25
25
25

25
2

25
25
25

25
25
10

10
25
25

I-H

25
20
2

25
6

25
350
25
25

25
2
25
1-H

5
12
25
25
4

25
1

25
25
25
25
25

ACM TranBactions on Soflware Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:42 A. Tiwana

Usage of six classes of development
coordination tools

1 = “not at all,” 4 = “somewhat,” and 7 = “to a
great extent.”

3
3
3
3
1

1-
1

3
6

5
5
1

2
2
1
1
3
6
1
1
5
1
6
2
7
5
1
3
1
5
3
5
4
4
3
7
3

7
6

2
4
3
2
6
4
4
2
6

1
4
4
4
1
5
4
6
6
5
7
2
1

1—
1

3
3
6
3
6
6
1
6
4
7
7
1
4

1
2
4

5
5
5
6
7
4
5
6
3
4
5
2
6
5
4
6
3
4
5

2
4
4
4
7
7
4
3
7
3
1
3
1
1
1
3
3
1
1
6
3
4
4
6
7
1
6
7
6
4
4
4
5
2
5
4
5
6
3
3
3
1
6
5
4
6
3
3
7

2
4
4
4
7
7
4
5
7
3
1
3
1
2
1
4
3
1
1
7
3
5
4
6
7

1—
1

6
7
3
4
3
5
4
2
5
4
5
6
5
3
2
1
6
4
4
7
3
1
4

5
5
5

5
7
7

5
7
6
3
1
4
4
3
5
5
5
3
1
6
1
5
3
6
7
1
6
4
4
5
7
5
4
6
3
5
6
6
7
3
5
1
6
4
4
4
4
1
5

5
5
7

7
5

7
7
5
5
5
1
1
5

4
6
1

6
6
1
6
3
6
7
6
5
4
3
5
7
1
5
6
6
5
7
6
5
2
5
1
6
6
4
7
4
6
7

Dimensions of Software
Development Performance
20
16
16
16
10
16
16
15
16
10
16
110
20
16
16
30
50
25
16
20
2
16
16
16
16
10
50
16
16
16
50
16
16
15
16
16
20
16
10
10
16
12
10
16
16
16
16
0
13

25
16
16
16
16
2
16
30
20
35
3
5
20
16
16
2
60
5
16
40
0
30
16
16
16
1
50
10
30
16
50
16
16
10
16
16
35
1
10
2
10
15
10
30
20
5
16
5
5

4
25
25
25
25
25
25
25
25
11
8
25
7
5
2
40
20
11
25
400
4

25
25
7

25
30
100
25
35
25
25
25
25
30
2
25
12
5
7
25
10
25
25
8
25
25
25
25
30

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois 11:43

Usage of six classes of development
coordination tools
1 = “not at all,” 4 = “somewhat,” and 7 = “to a
great extent.”

3
3
4
2
4
4
1
1
6
2
7
5
1
3
5
3
3
3

5
3
4
5
6
3
5
3
3
3
1
2
3
3
3
1
1
3
3
5
4
1
1
6

4
3
5
3

3
5
5

5
6
4
5
3
4
4
4
7
6
6
5
1
4
4
4

4
4
1
4
4
2

5
3
4
4
4
4
3
5
4
5
4
2
3
4
4
4
5
4

1
6
4
4
5
4

4
3
5

7
2

5
1
3
4

1
1
4
6
1

5
2
4
5
4
4
4
1
4
6
5
2
4

ef
t

4
4

4
1
4
4
7
2

I-l

3
4
4

1
5
7

1
6
4
4
5
4
4
5

l-l

2
4
1

4
6
7
4

5
3
5

1
4
5
4
4
4
4
4
6
4

ef
t

7
2
4
4
4
1

2
4
7

2
2
4
4
4
2
5
7
1
6
1

4
5
4
4
5
7

7
6
5
7
1
4
1

1
5
6
5
4
7
5
4
5
5
5
4

5
7
7
ef
t
6

5
5
5
5
7

6
5
7

l-l

1
6
5
5
3
6
7
7
6
4
5
6
5
5
5
5

7
6
4
4
4
4
3
1
7
6
6
4
6
5
5
5
5
5
6
5
4
7
7
6
5

5
5
5
7
7
5

5
1
2
6
5
5
4

5
7
1
6
7

5
6
5
5
5
7

Dimensions of Software
Development Performance
5

15
20
30
0
0

20
0
16
15
25
16
20
16

10
16
16
16
25
16
20
16
30
10
16
16
16
16
20
15
16
16
16
10
10
16
16
15
10
16
20
15
16
16
16
16
16
15
16

10
10
10
10
3

25
10
10
16
20
15
16
16
16
15
16
16
16
50
16
10
16
20

25
16
16
16
16
20
20
16
16
16
16
\ry
16
16
20
2
16
16
5
16
16
10
16
16
16
80

4
30
7
5

25
25
25
3
25
15
8

25
0

25
25
25
25
15
25
25
25
35
25
25

25
25

25
2

25
25

25
25
7
25
25
10

5
25
22

25
25

25
25
25
25
25
7

ACMTranBactionson Software Engineeringand Methodology, Vol. 17, No. 2,Art)cle 11, Publication date: April 2008

11:44 A. Tiwana

Usage of six classes of development
coordination tools
1 = “not at all,” 4 = “somewhat,” and 7 = “to a
great extent.”

3
6
4
1
3
4
3
5
4
1

I-H

4
3
3
1

C
JI

6
5
3
3
4
3
3

C
Jl

4
3
3
1
3

4
5
2
5
3

3
5
3
6
5
3
5
5
3
3

C
JI

3
1
3
4

3
6

4
1
4
4
6
5
4
1
1

5
3

4
2
5
1

C
JI
4
4
4
4
4
7

7
4
4
1

C
Jl
C
JI
C
JI

5
1
4
4
5

4
6
1

3
5
5
4
4
7
4
5
2
1

7
6
3
1
4
4
6
7
4
7

1
5
4
4
3
6
2
6
4
4

7
4
4
7
7
4
4
5

5
5
4

ira

1
4
4

2
4
1
1
4
5
1
4
4
6
4

C
Jl
1
4

7
6
4
1
4
4
6
6
4
7
1
4

2
4
1

5
2
6
4

4
7
4
4
7
4

4
4
5
6

5
6
5
1

4
4

6
4
1
1
4
4
5
4
4
5
4

5
2
1

7
6
3
5

C
Jl

4
2
5
4
7
7
5
7
5
6
5
7
6

5
C
Jl
7
ira

5
1
7

5
5
5
6
5
5
5
5
5
5

ira
5
C
JI

1
6
4
5
5
5
2
5
1
6
6

7
6
6
5
5
4
2
7
4
7
6

C
Jl
7
C
Jl
C
Jl

5
7
5
5
5
7
5

5
6
7
5

•
5
5
5
6
5
1
5
5
5
5
7

1
6
4
5
5
5
3
5

I-H
7
7

Dimensions of Software
Development Performance
0
16
16
16

16
20
0
10
16

5
16
0
16
16
10
16
6
15
16
16
16
16
16
6
10
16
16
20
30
16
16
20
10
16
16
10
16
10
16
5
16
16
16
16
16
16
20
16
16

0
16
16
16
16
0
10
15
16
15
16
15
16
16
5
16
40
10
16
16
16
16
16
10
12

16
16
16
8
16
16
10
16
16
16
10
16
12
16
16
16
16
16
16
20
16
20
60
0

25
12
25
25
25
25
25
25
25
25
25
29
25
25
25
25
3
75
25
25
15
25
25
8
25
25
25
25
2 1
25
25
5
13
25
25
25
25
25
25
5

25
25
25
25
32
25
25
100
25

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Tools 11:45

Usage of six classes of development
coordination tools
1 = “not at all,” 4 = “somewhat,” and 7 = “to a
great extent.”
4
3
3
1
3
1
5
7
2
3
3
6
1
1
3
4
5
3
3
4
4
1

4
5
5
4

2
5
4
3

5
1
4
1
2
4
5
4
4
1
4
5
2

2
2
5
4

7
5
3
4
2
1
4
3
4
4
7
4
3
1

2
2
4
4
7

4
1
4
1
2
4
4
5
5
7
5
6
5
2
2
6
5
7
6
7
5
1
4
5
4
5
5
1
5
5
5
2
2
6
5
7
6
7
5
3
4
5

Dimensions of Software
Development Performance
12

16
16
0
16
5
5
5
3
10
16
16
5
16
16
10
3
16
15
16
16
10
16
2
5
16
10
16
16
15
0
16
16
5
1
16
2
25
25
25
25
25
40
25
0
25
25
25
18
25
25
25
4
25

1. Tool usage variables legend: Tl = Requirements management; T2 = Architectural modeling; T3 = Test au-
tomation; T4 = Test case development; T5 = Configuration management; T6 = Defect & change request tracking.
2. Software development performance variables legend: PI = Cost overrun as % of planned budget;
P2 = Rework as % of total project function points; P3 = Residual defects in installation and warranty stages.
3. Usage of each class of development coordination was measured using the project manager’s perceptual evalu-
ation.
4. Mean substitution was used for missing data in the analyses. See Table 5 for series means.
5. Defect data represents only the recorded defects and waa self-reported by the project manager for each project.
6. Scales/measurement items in the questionnaire are described in Section 4.1.

ACKNOWLEDGMENTS

Comments and feedback from Ashley Bush and four anonymous reviewers are
gratefully acknowledged.

REFERENCES

ABDEL-HAMID, T., SENGUPTA, K., AND RONAN, D. 1993. Software project control: An experimental
investigation of judgment with fallible information. IEEE Trans. Soft. Engin. 19, 6, 603-612.

ADELHON, B. AND SOLOWAV, E . 1985. The role of domain experience in software design. IEEE Trans.
Soft. Engin. 11, 11.1351-1360.

ADLER, P. S. 1995. Interdepartmental interdependence and coordination—the case of the de-
sign/manufacturing interface. Orgatiiz. Sei. G, 2, 147-167.

AIKEN, L . AND WEST, S. 1991. Multiple Regression: Testing and Interpreting Interactions. Sage,
Newbury, CA.

ANDRES, H . P. AND ZMUD, R. W. 2002. A contingency approach to software project coordination. J.
Manag. Info. Sys. 18, 3, 41-70.

BABBÍE, E . R. 1973. Survey Research Methods. Wadsworth, Belmont, CA.
BACWJZZI, R. P., Yi, Y., AND PHILLIPS, L.W. 1991. Assessing construct validity in organizational

research. Admin. Sei. Quarterly. 36, 3, 421—458.
BANKER, R., DAVLS, G., AND SLAUGHTER, S. 1998. Software development practices, software com-

plexity, and software maintenance performance. Manag. Sei. 44, 4, 433-450.

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

11:46 • A. Tiwana • – .

BARKI. H . , RIVARD, S.. AND TALBOT, J. 2001. An integrative contingency model of software project
risk management. J. Manag. Info. Syst. 17, 4, 37-69.

BARON, R. AND KENNY, D. 1986. The moderator-mediator variable distinction in social psycholog-
ical research; Conceptual, strategic, and statistical considerations. J. Person. Soc. Psychol. 51,
1173-1182.

BASKERVILLE, R. AND PRIES-HEJE, J. 1999. Grounded action research; A method for understanding
IT in practice. Account. Manag. Info. Techno. 9, 1, 1-23.

BELSLEY, D.A., KUH, E., AND WELSCH. R.E. 1980. Regression Diagnostics: Identifying Influential
Data and Forces ofColUnearity. Wiley, New York.

BROWN, D. 2003. The developer’s art today: Aikido or sumo? ACM Queue 1, 6.
COHEN, J. AND COHEN, P. 1983. Applied Multiple Regression Analysis for the Behavioral Sciences.

Lawrence Erlbaum, Hillsdale.
CRESWELL, J. 1994. Research Design: Qualitative and Quantitative Approaches. Sage Publica-

tions, Newbury. CA.
CRONBACH, L. J. AND MEEHL, P.E. 1955. Construct validity in psychological tests. Psychol. Bull. 52,

4, 281-302.
DUNCAN, 0. 1985. Path analysis: Sociological examples. In Causal Models in Social Sciences, H.

Blalock (Ed.). Aldine Publishing, Hawthorne, NY,
FARAJ, S. AND SPROULL. L. 2000. Coordinating expertise in software development teams. Manag.

Sei. 46,12,1554-1568.
GLASER, B. AND STRAUSS, A. 1967. The discovery of Grounded Theory: Strategies for Qualitative

Research. Aldine Publishing Company, Chicago.
GOPAL, A., SIVARAMAKKIRHNAN, K. KRISHNAN. M., AND MUKHOPADHYAY, T. 2003. Contracts in ofFshore

software development: An empirical analysis. Manag. Sei. 49, 12, 1671-1683.
GOTTESDIENER. E. 2003. Requirements by collaboration. IEEE Sofî. March-April, 5 2 –

55.
GROTH, R. 2004. Is the software industry’s productivity declining? IEEE Soft. 21, 6, 9 2 –

94.
GUIÑAN, P.. COOPRIDER, J., AND SAWYER, S. 1997. The effective use of automated application devel-

opment tools. IBM Syst. J. 36, 1, 124-139.
HAIR, J.F., JR.. ANDERSON. R . E . TATHAM, R.L., AND BLACK, W.C. 1995. Multivariate Data Analysis.

Prentice Hall, Englewood Cliffs, NJ.
Ho, V, ANG. S., AND STRÄUB. D. 2003. When subordinates become IT contractors: Persistent man-

agerial expectations in IT outsourcing. Inform. Syst. Resear. 14, 1, 66-86.
KING, S. AND GALLIERS, R. 1994. Modelling the case process: Empirical issues and future directions.

Info. Soflw. nch. 36,10, 587-596.
KIRSCH, L., SAMBAMURTHY, V. Ko, D.. AND PURVIS, R. 2002. Controlling information systems devel-

opment projects: The view from the client. Manag. Sei. 48, 4, 484—498.
KoKOL, P. 1989. Formalization of the information system development process using metamodels.

ACM SIGSOFT Softw. Engin. Notes 14, 5, 118-122.
KUMAR, N.. STERN. L . W . AND ANDERSON. J . C . 1993. Conducting interorganizational research using

key informants. Academy Manag. J. 36, 6, 1633-1651.
LAWTHER, W.C. 1986. Content validation: Conceptual and methodological issues. Rev. Public Per-

son. Admini 6, 3, 37-49,
LEE. A . S . A scientific methodology for MIS case studies, M/S Quart., 1989, 33-50.
MACCORMACK, A., VERGANTI, R., AND IANSITI, M . 2001. Developing products on internet time: The

anatomy of a flexible development process. Manag. Sel. 47,1, 133-150.
McAFEE, A. 2003. When too much IT knowledge is a dangerous thing. MIT Sloan Manag. Rev.

44, 2. 83-89.
MESSERSCHMITT. D. AND SZYPERSKI, C. 2003. Software Ecosystem. MIT Press, Cambridge, MA.
MiNGERS, J., 2001. Combining IS research methods: Towards a pluralist methodology. Info. Sys.

Resea. 12, 3, 240-259.
MoOKERJEE, VS. AND CHIANG, R. 2002. A dynamic coordination policy for software system con-

struction. IEEE Trans. Soflw. Engin. 28, 6, 684-694.
MORGAN, G. AND SMIRCICH. L.. SAWYER, S. 1980. The case for qualitative research. Acad. Manag.

neu 5. 4, 491-500. . . .

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Classes of Software Development Coordination Toois • 11:47

MiTMFORD, M., CosTANZA, D., AND CoNNELLY, M. 1996. Item generation procedures and background
data scales: Implications for construct and criterion-related validity. Person. Psychol. 49, 2, 3 6 1 –
398.

ORLIKOWSKI, W. 1989. Division among the ranks: The social implications of case tools for system
developers,. In Proceedings ofthe Tenth International Conference on Information Systems, J.
Degross, J. Henderson and B. Konsynski Eds. ACM Press.

ORLEKOWSKI, W. J . 1993. Case tools as organizational change: Investigating incremental and rad-
ical changes in systems development. MIS Quart. 17, 309-340.

PAESE, P.W. AMD SWITZER, F.S., III. 1988. Validity generalization and hypothetical reliability dis-
tributions: A test ofthe schmidt-hunter procedure. J. Appl. Psychol. 73, 2, 267-274.

F’EDHAZUR, E . J . 1982. Multiple Regression in Behavioral Research: Explanation and Prediction.
Dryden Press, New York, NY.

RAMESH, B. 1998. Factors influencing requirements traceability practice. Comm. ACM 41, 12,
37-44.

RAMESH, B . AND DHAR, V. 1992. Supporting systems development by capturing deliberations dur-
ing requirements engineering. IEEE Trans. Softw. Engin. 18, 498-510.

RAMESI4, B. AND JARKE, M. 2001. Towards reference models for requirements traceability. IEEE
Trans. Soßw. Engin. 27, 1, 58-93.

ROBILLARD, P. 1999. The role of knowledge in software development. Comm. ACM 42, 1, 87-92.
RowEN, R. 1990. Software project management under incomplete and ambiguous specifications.

IEEE Trans. Engin. Manag. 37, 1,10-21,
Rus, I. AND LINDVALL, M . 2002. Knowledge management in software engineering. IEEE Soßw. 19,

3, 26-38.
SCHMIDT, F. AND HLT^TER, J. 1989. Interrater reliability coefficients cannot be computed when only

one stimulus is rated. J. Appl. Psychol. 74, 368-370.
SCHWAB, D.P. 1980. Construct validity in organizational behavior. Research in Organizational

Behavior. 2, 3-^3.
SKAMAN, C. AND BASILI, V. 1998. Communication and organization: An empirical study of discus-

sion in inspection meetings, IEEE Trans. Soßw. Engin. 24, 7, 559-572.
SEELEY, D. 2003. Coding smart: People vs. Tools. ACM Queue 1, 6.
SHORE, T.H., SHORE, L.M., AND THORONTON, G.C., III. 1992. Construct validity of self- and peer

evaluations of performance dimensions in an assessment center. J. Appl. Psychol. 77, 1, 42-54.
SPINELLIS, D. 2005. The tools at hand. IEEE Softw. 26, 1, 10-12.
STRÄUB, D.W. 1989. Validating instruments in MIS research. MIS Quart. 13, 2, 147-166.
STRAUSS, A. AND CORBIN, J. 1998. Basics of Qualitative Research: Techniques and Procedures for

Developing Grounded Theory. Sage Publications, Inc., Newbury, CA.
TAKEISHI, A. 2002. Knowledge partitioning in the interfirm division of labor: The case of automo-

tive product development. Organiz. Sei 13, 3, 321-338.
TIWANA, A. AND KEIL, M. 2004. The one minute risk assessment tool. Comm. ACM 47,11, 73-77.
Tiw.\NA, A. AND McLEAN, E. 2003. Managing the unexpected: The tightrope to e-business project

success. Comm. ACM 46, 12, 345-350.
TIWANA, A. AND MCLEAN, E . R. 2005. Expertise integration and creativity in information systems

development. J. Manag. Info. Sys. 22, 1, 13-43.
ToDD, J.D. 1979. Mixing qualitative and quantitative methods: Triangulation in action. Admin-

istrative Sciences Quart. 24, 4, 602-611.
WAIJ., D., ELAM, J., AND CURTIS, B . 1993. Inside a software design team: Knowledge, sharing, and

integration. Comm. ACM 36, 10, 63-77.
WP;EKLEY, J.A. AND GIER, J.A. 1989. Ceilings in the reliability and validity of performance ratings:

The case of expert raters. Acad. Manag. J. 32, 1, 213-222.
YIN, R.K. 1994. Case Study Research: Design and Methods. Sage Publications, Newbury, CA.

Received August 2005; revised March 2006; November 2006; February 2007; accepted March 2007

ACM Transactions on Software Engineering and Methodology, Vol. 17, No. 2, Article 11, Publication date: April 2008

Turn in your highest-quality paper
Get a qualified writer to help you with

“ Application 2 – Annotated Bibliography ”

Get high-quality paper

Guarantee! All work is written by expert writers!

Still stressed from student homework?

Get quality assistance from academic writers!

Order now