INVESTIGATING THE IMPROVEMENT SPACE OF SOFTWARE
DEVELOPMENT ORGANISATIONS
Joseph Trienekens, Rob Kusters
Eindhoven University of Technology, Den Dolech 2, 5600 MB, Eindhoven, The Netherlands,
Frans van Veen, Dirk Kriek, Daniel Maton
Quality House B.V., 3404 LL IJsselstein, The Netherlands,
Paul Siemons
Metrific, 3506 GT, Utrecht, The Netherlands,
Keywords: Software process improvement, entropy, empirical research results.
Abstract: Actual results of software process improvement projects happen to be quite disappointing in practice.
Although many software development organisations have adopted improvement models such as CMMI, it
appears to be difficult to improve software development processes in the right way, e.g. tuned to the actual
needs of the organisation and taking into account the environment (e.g. the market) of an organisation. This
paper presents a new approach to determine the direction of improvement for an organisation. This
approach is based on literature research as well as an empirical investigation among eleven software
development organisations in The Netherlands. The results of the research show that software development
organisations can be classified and can be positioned on the basis of their internal and external entropy, c.q.
the level of (dis)order in the business system and its environment. Based on a possible imbalance between
the internal and external entropy, directions for software process improvement can be determined. As such
the new approach can complement and improve the application of current software process improvement
methodologies, e.g. CMMI.
1 INTRODUCTION
To be competitive in the current economy, more and
more software development organisations strive at
an improvement of their processes. This is based on
convictions that an improvement of their software
engineering processes ultimately will lead to an
increased quality of the products. Over the last ten
years several improvement models have been
developed that act as frameworks and tools to
improve software development processes (Balla et
al, 2001). Models such as the Capability Maturity
Model (CMM, CMMI) (SEI, 2006), (Paulk et al,
1993) are being applied increasingly in the software
engineering domain. Starting point for process
improvements is a so-called process assessment, i.e.
an analysis and subsequent determination of the
current performance level of the software
development process. On the
basis of the outcome of an assessment an
improvement project can be initiated. However, the
actual results of improvement projects are often
quite disappointing in practice (Kusters and
Trienekens, 2005). Case study reports and success
factor investigations offer until now only a limited
insight in the way software development
organisations can and/or should try to improve their
software development processes.
A closer look at assessment and improvement
methodologies shows that they are strongly oriented
on the internal processes of an organisation. They
focus for example on internal organisational factors
such as management commitment, on internal
human factors such as training, on internal process
factors such as project planning, on internal
technical factors such as tool support, and on
financial factors such as project budgets (SEI, 2006),
(Paulk et al, 1993). As a consequence of this internal
orientation it can be concluded that current software
3
Trienekens J., Kusters R., van Veen F., Kriek D., Maton D. and Siemons P. (2006).
INVESTIGATING THE IMPROVEMENT SPACE OF SOFTWARE DEVELOPMENT ORGANISATIONS.
In Proceedings of the Eighth International Conference on Enterprise Information Systems - DISI, pages 3-10
DOI: 10.5220/0002443300030010
Copyright
c
SciTePress
process improvement approaches do not take into
account the influence of the environment of a
software development organization on the way the
software development processes could be improved
(Kusters and Trienekens, 2005).
This paper introduces a new approach for
software process improvement. This approach uses
as kernel concept the distinction between the internal
and external entropy of a software development
organisation. Entropy is a cybernetic concept that
stands for the level of disorder of a system
(Boltzman, 2000). The main assumption in this
paper is that an organisation should be aware of its
own internal and external disorder. Differences
between the internal and external disorder should
play a role in the process of chosing a direction for
process improvement. In this paper the theoretical
concepts of internal and external entropy have been
made operational. This has been done by an
assessment of a selection of well-defined internal
and external factors of a software development
organisation. Based on an analysis of these internal
and external factors the differences between the
internal and external entropy, the so-called
'improvement space' of an organisation, is
determined. Within its own 'improvement space' an
organisation then has to strive at a balance between
their internal and external entropy. This should be
the main theme for their process improvement
activities, and the usage of improvement models
such as CMMI should be based on that theme.
This paper deals in chapter 2 with software
process improvement, its limitations and the concept
of entropy. Chapter 3 focuses on the metrification
and quantification of entropy. In chapter 4 the results
are presented of an empirical research project in that
the entropy of eleven software development
organisations has been measured. Finally, in chapter
5 conclusions and recommendations for future
research are given.
2 SPI AND THE NECESSITY OF
TAKING BOTH INTERNAL
AND EXTERNAL
ORGANISATIONAL FACTORS
AS STARTING POINT
In the software engineering domain, improvement
activities are known as Software Process
Improvement (SPI). SPI is being defined as:
The measurement-based improvement of the
performance of the software development process
aimed at delivery in due time, against agreed
budgets, and with the required quality.
A well-known and broadly accepted model for
the improvement of software development processes
is CMM (Capability Maturity Model). CMM
prescribes a specific sequence of improvement
activities that has to be followed to reach higher
levels of software process maturity. CMM and
comparable maturity models are used on the one
hand to strive at a standardisation of processes and
on the other hand to realise a decrease of product
failures by eliminating their causes. However, it is
questionable whether these improvement models are
applicable for each and every organisation. For
instance, an organisation that has to deal with an
increasing turbulent market, and that has improved
its development processes by defining and
standardising them in detail, will face difficulties in
responding to the dynamic and changing
requirements of that market. As a result there will be
a ‘mismatch’ between the strengths of the software
developing organisation and the needs of the market.
Examples of other factors in the environment of
an organisation that can influence its behaviour are
changing governmental and legal factors. However,
these types of external factors are until now not
addressed by software process improvement
methodologies. This paper proposes that both
internal and external factors should be taken into
account during a process assessment and they have
to play a role in the determination of process
improvement activities.
In order to make external and internal factors
applicable to process assessment, the concept of
external and internal entropy is introduced.. Entropy
is a concept with that the internal and external
situation of a system can be expressed (Boltzman,
2000). A low entropy means a high level of order, or
structure and stability, in a system. A high entropy
means a low, or even chaotic, level. Table 1 gives
some examples of characteristics of organisations
with high and low internal and external entropies.
These rather abstract characteristics are used as a
basis for making the concept of entropy more
operational.
Based on this type of characteristics,
organisations can be considered as dynamic entities
that are continuously moving and changing, and that
have to strive at a balance between their internal and
external entropy. For instance, in case a market is
very dynamic, and a company in that market is, as a
consequence, faced with a high external entropy, this
company then has to be able to respond quick and
adequately to the market, and in other words should
be flexible.
ICEIS 2006 - DATABASES AND INFORMATION SYSTEMS INTEGRATION
4
Table 1: Some examples of characteristics of organisations
with high and low entropies.
High external entropy Low external entropy
Environment is
unstable and
unpredictable
Environment is
stable and
predictable
Standardisation is
less important than
flexibility
Flexibility is less
important than
standardisation
High internal entropy Low Internal entropy
Flexibility is more
important than
standardisation
Flexibility is less
important than
standardisation
Organisational
processes are often
executed in a chaotic
way
Organisational
processes are
executed in a formal
and predictable way.
To increase its own flexibility, an organisation
could be forced then to increase its internal entropy
so that it becomes able to respond to the (dynamic)
market. Such an organisation has to drop for
example certain standardisation rules and has to
increase the level of decisional freedom of
employees in the development processes. The
dropping of standardisation rules and the increasing
of decisional freedom of employees should become
main themes for software process improvement in
this type of organisations.
3 MEASURING ENTROPY
To be able to use the concept of internal and external
entropy in a practical way it is necessary to make
entropy measurable. In accordance with (Boltzmann,
2000) entropy is based on the number of states that a
system can have. In order to make entropy
operational we link the state of a system to basic size
variables. Of course this link has to be validated in
practice and elaborated further, but in this paper
organisations are considered to be business systems,
consisting of interrelated components. The entropy
of such a business system increases with:
• an increasing number of components;
• an increasing number of interactions between
the components;
• an increasing number of changes in the
environment.
From the above we derive that two concepts are
playing a central role in the determination of
entropy, respectively complexity and dynamics.
These two concepts can be described as follows:
• complexity is a measure of the number of
interacting components;
• dynamics is a measure of the amount of changes
over time.
3.1 Entropy and Its Relations to
Complexity and Dynamics
Entropy can be made measurable by using the
concepts of complexity and dynamics. In principle,
measuring is the linking of a value to a parameter,
by positioning the parameter on a measurement
scale. From the four types of measurement scales:
ratio, interval, ordinal and nominal, only the ordinal
scale applies for the measurement of the concepts of
complexity and dynamics (and subsequently for the
measurement of entropy). The ordinal scale offers
the possibility to position a parameter on a
measurement scale with values low, average, and
high. In our research we have chosen to measure
complexity and dynamics by using a four value
measurement scale with the levels: low, average-
low, average-high and high.
Table 2: Levels of complexity.
Complexity Definition
Low Almost all parameters have a low score
Average-Low Most parameters have a low score
Average-High Most parameters have a high score
High Almost all parameters have a high score
Measuring complexity is executed via a number
of business system characteristics, such as the
number of employees per business function, the
number of functions per department, the number of
departments in the organisation, etc. Based on
numbers for these parameters, organisations can be
classified. For example the number of employees
can be used to classify an organisation as small (1-
9), average-small (10-99), average large (100-999)
or large (1000+).
Similarly the dynamics of a business system can
be measured. For example a classification can be
derived on the basis of the number of process
changes per time-frame. To determine the number of
changes of a specific parameter, such as employees,
information is needed on the input, the output, and
the current number of the employees. Other
parameters are for instance departments, products,
customers, suppliers, competitors etc.
INVESTIGATING THE IMPROVEMENT SPACE OF SOFTWARE DEVELOPMENT ORGANISATIONS
5
Consequently a total score for entropy can be
calculated by combining the scores for dynamics and
complexity, see table 3.
3.2 Entropy and Its Relations with
Business System Aspects
To determine the internal and external entropy of a
software development organisation a questionnaire
has been developed. This questionnaire is based on
the following structure. First the concept of business
system has been subdivided in the three aspects of
Market, Organisation and Factory. Subsequently
these aspects have been elaborated. Respectively:
Market has as sub aspects: products, design and
architecture; Organisation has as sub-aspects:
processes and structure; and Factory has as sub
aspects: people, resources and technology. The
objective is that for each of the three aspects (and
sub aspects) the complexity and dynamics can be
determined. Table 4 summarises the basic structure
of the questionnaire. By using the questionnaire,
information is gathered about the (sub)aspects and
subsequently about the complexity and the
dynamics.
On that basis the internal entropy (business
system aspect Organisation and Factory) and the
external entropy (business system aspect Market)
can be determined. To illustrate the questionnaire
and its questions, in table 5 an example is given of a
question that addresses the complexity of the
Market.
The questions about the business system aspect
Market, lead to scores with that the external entropy
can be determined. The questions about the business
system aspects Organisation and Factory lead to
scores on which basis the internal entropy can be
determined.
Table 3: Determination of entropy on the basis of measurements of complexity and dynamics.
Complexity
Entropy
Low Average-Low Average-High High
Low Low Low Average-Low Average-Low
Average-Low Low Average-Low Average-Low Average-High
Average-High Average-Low Average-High Average-High High
Dynamics
High Average-High Average-High High High
Table 4: (Sub)aspect of Market, Organisation and Factory.
Aspect Sub aspect Complexity Dynamics
External entropy Market Product, Design, Architecture
Organisation Process and Structure Internal entropy
Factory People, Resources and Technology
Table 5: Example of a question about the complexity of the Market.
COMPLEXITY (of market on the basis of the aspects product, design and architecture)
How many parties are involved in the development of the software products (e.g. as supplier of sub-
architectures, -designs and/or components)
Quantification:
How large is the involvement of third parties?
- percentage components delivered?
- number of external parties that contributes to an architecture and/or design?
- amount of time that external parties contribute?
What is the average effort that external parties contribute?
- average number of employees per external party /number of employees intern
- total number of employees of external parties /total number of employees in the project
ICEIS 2006 - DATABASES AND INFORMATION SYSTEMS INTEGRATION
6
Low
Low
High
High
EXTERNAL entropy Æ
INTERNAL entropy Æ
B
a
l
a
n
c
e
b
e
t
w
e
e
n
i
n
t
e
r
n
a
l
a
n
d
e
x
t
e
r
n
a
l
e
n
t
r
o
p
y
Room for
process improvement
CMM 1
CMM 2
CMM 3
CMM 4
CMM 5
Figure 1: Space for improvement for software development organisations.
By comparing the scores of the external entropy and
the internal entropy it becomes possible to position
organisations in a two-dimensional matrix, see
Figure 1.
A difference between the internal and external
entropy shows, what we would like to call, the
'improvement space'. The concept of 'improvement
space' indicates that it is possible to strive at a
balance between internal and external entropy. The
larger the distance an organisation has to the
balance-line, the larger the space for improvement
is, see Figure 1.
In Figure 1 the balance-line is represented by the
dotted black line. The position of this balance-line is
currently based on the usage of the data collected
(i.e. the eleven companies). This means that we
assume that on the average these companies have a
'certain' balance between internal and external
entropy. It has to be emphasised that this is just a
first step to determine the balance-line. Further
research, and preferably more data, is needed to
determine the balance-line more precisely.
The yellow bullet represents an organisation with
a particular distance to the balance-line. This
organisation has a relatively high level of internal
entropy and a relatively low level of external
entropy. Such an organisation should emphasise in
its software process activities a decrease of its
internal entropy, for example by striving at a certain
level of standardisation of its processes. By doing
so, the internal entropy will decrease until a balance
is reached. Rather intuitively, on the vertical
dimension to the right of Figure 1, the CMM-levels
are given. Notice that on CMM5, which is the
highest maturity level, the internal entropy is
extremely low. This means that a software
development organisation has an extremely high
level of internal organisational order.
4 RESULTS OF THE RESEARCH:
SPACE FOR IMPROVEMENT
FOR SOFTWARE
DEVELOPMENT
ORGANISATIONS
The questionnaire has been applied in an empirical
research project in that eleven software development
organisations in The Netherlands have been visited
for the execution of interviews. In each organisation
both a responsible person for software process
improvement and a representative product manager
have been interviewed. The software process
improvement experts, with a strong internal
orientation, had more than five years experience in
the field. The product managers, with a strong
orientation on the environment (or: external Market
factors) of the organisation had about 10-15 years of
relevant experience. The structured interviews took
about 2 hours per interview. The results, answers to
the questions, have been elaborated and have
subsequently been reviewed by the interviewed
persons.
INVESTIGATING THE IMPROVEMENT SPACE OF SOFTWARE DEVELOPMENT ORGANISATIONS
7
0
1
2
3
4
5
6
No 0-2 year 2-5 year 5-10 year 10-15 year >15 year
0
1
2
3
4
5
6
No 0-2 year 2-5 year 5-10 year 10-15 year >15 year
Figure 2: Experiences with Software Process Improvement.
Low
Low
High
High
EXTERNAL entropy Æ
INTERNAL entropy Æ
B
a
l
a
n
c
e
b
e
t
w
e
e
n
i
n
t
e
r
n
a
l
a
n
d
e
x
t
e
r
n
a
l
e
n
t
r
o
p
y
CMM 1
CMM 2
CMM 3
CMM 4
CMM 5
Low
Low
High
High
EXTERNAL entropy Æ
INTERNAL entropy Æ
B
a
l
a
n
c
e
b
e
t
w
e
e
n
i
n
t
e
r
n
a
l
a
n
d
e
x
t
e
r
n
a
l
e
n
t
r
o
p
y
CMM 1
CMM 2
CMM 3
CMM 4
CMM 5
D
C
B
A
Figure 3: Positioning of software development organisations on the basis of their internal and external entropy scores.
The research has lead to a number of results. In
the introduction of the interview sessions the
interviewed persons have been asked to answer
some more general questions. One of these questions
was to describe the level of experience with software
process improvement of the organisation. In section
4.1 we present first these experience levels. In
section 4.2 we will present the entropy scores of the
interviewed organisations.
4.1 Experience with Process
Improvement
Answers to questions regarding the experience with
software process improvement have lead to Figure 2.
In total 17 persons could answer this question, see
Figure 2. From this it appeared to be that about 80%
of the visited organisations had experience with SPI
4.2 Positioning Organisations on the
Basis of their Entropy Scores
Based on answers from the questions regarding the
complexity and the dynamics of the three aspects
Market, Organisation and Factory, the internal and
external entropy could be determined. The following
Figure 3 shows selected results.
Based on the above mentioned scores for the
business system characteristics of the software
development organisations, their scores for the
internal and external entropy could be determined.
These scores have then been used to position
each software development organisation in the
internal/external entropy balance Figure, see Figure
3.
ICEIS 2006 - DATABASES AND INFORMATION SYSTEMS INTEGRATION
8
Table 6: Characteristics of Organisation A.
Table 7: Characteristics of Organisation B.
Organisation
B
Complexity Motivation Dynamics Motivation
Market H Complex product
portfolio
HH High pressure of market which asks for lead time
reduction in combination with innovative products
Organisation L Rather rigid formal
matrix organisation
L Stable structure and standardised processes
Factory L High level of
standardisation
HH Many job changes, fast emerging technologies.
Table 8: Characteristics of Organisation C.
Organisation
C
Complexity
Motivation
Dynamics
Motivation
Market L
Stable and mature market with
restricted product portfolio
L
Long lead time per product
Organisation M
High level of maturity of business
processes
LL
Rather stable processes, only few
changes per time interval
Factory H
High level of standardisation of all
resources is a necessity
L
Few changes in resources to be
applied
Table 9: Characteristics of Organisation D.
Organisation D Complexity
Motivation
Dynamics
Motivation
Market L
Limited product portfolio, relatively
low complexity of design and
architecture
M
Influence of customers on product
portfolio increases.
Organisation L Flat, formal organisation HH
Many reorganisations necessary
due to market pressure
Factory H Low level of standardisation H
Allocation of (new types of)
resources in the business
processes is emerging
From the four tables 6, 7, 8 and 9 it can be
concluded that organisations can have totally
different characteristics regarding their
internal/external entropy balance. As a consequence
different organisations should follow rather different
directions regarding the improvement of their
software development processes. For example, it is
possible that an organisation wants to strive at a
decrease of its internal entropy by means of a
standardisation of its development processes. By
doing so, this organisation can reach a more
balanced internal/external entropy situation. Of
course it is also possible that an organisation wants
to influence its external entropy by focusing on new
emerging markets (i.e. increasing its external
entropy) or by focusing on existing stable markets
(i.e. decreasing its external entropy). The final
objective of these 'improvement directions' or
'improvement themes' is a more balanced situation
regarding the internal and external entropy of the
software development organisation.
5 CONCLUSIONS
This paper presents the hypothesis that software
development organisations can be classified and can
Organisation
A
Complexity
Motivation
Dynamics
Motivation
Market H Various products for various markets H
Product portfolio changes rapidly
Organisation H Many interrelations with suppliers H
Many changes in collaborative
processes
Factory H
Large diversity in resources and people
(skills)
H
Emerging new technologies
INVESTIGATING THE IMPROVEMENT SPACE OF SOFTWARE DEVELOPMENT ORGANISATIONS
9
be positioned on the basis of their internal and
external entropy. Based on empirical research in
eleven organisations the concept of entropy has
proven to be fruitful for the characterisation of both
the internal business system and its (external)
environment. By using these internal and external
characteristics, a possible imbalance can be
determined for an organisation and a direction for
improvement can be pointed out.
In this paper the characteristics of selected
software development organisations have been
determined on the basis of questionnaires that have
been used in formal structured interviews. The
concepts and terminology used has lead to positive
reactions of the interviewed practitioners. The
research results show that software development
organisations can have quite different
internal/external entropy characteristics, and this can
lead to quite different software process improvement
directions. Although the results gained are
interesting we realise that further research is needed
to validate the assumptions and constructs made. In
particular we will investigate further the
measurability of the internal and external entropy on
the basis of the concepts of complexity and
dynamics. Other main issues in our near-future
research are the more precise determination of the
entropy balance-line, and also the determination of
the relation between CMM-levels and the
internal/external entropy levels, which is currently
still intuitively assumed. The final result at that we
aim at, is a method (a.o. systematic analysis) and
tool (i.e. automated questionnaire), to determine the
improvement space of software development
organisations that strive at efficient and effective
software process improvement.
REFERENCES
Balla, K., T. Bemelmans, R. Kusters, J.Trienekens, 2001.
Quality through Managed Improvement and
Measurement (QMIM): towards a phased development
and implementation of a Quality Management System
for a Software Company, Software Quality Journal, 9,
177-193.
Boltzmann L., 2000. Entropie und Warscheinlichkeit,
Ostwalds Klassiker, Bd 286.
Carnegie Mellon Software Engineering Institute, 2006,
CMMI website, http://www.sei.cmu.edu/cmmi/.
Kusters R.J., J.J.M. Trienekens, 2005. Software Process
Improvement from a Business Perspective, submitted
to: The International Journal of Information
Management.
Paulk, M.C., Curtis, B., Chrissis, M.B., and Weber, C.V.,
1993. Capability Maturity Model for Software,
Version 1.1., SEI-CMU-93-TR-25
.
ICEIS 2006 - DATABASES AND INFORMATION SYSTEMS INTEGRATION
10
