MEASURING EFFECTIVENESS OF COMPUTING FACILITIES
IN ACADEMIC INSTITUTES A NEW SOLUTION FOR A
DIFFICULT PROBLEM
Smriti Sharma
Department of Industrial & Management Engineering, Indian Institute of Technology, Kanpur, U.P., India
Veena Bansal
Department of Industrial & Management Engineering, Indian Institute of Technology, Kanpur, U.P., India
Keywords: Computing Facilities, Usability, Functional Utility, User Satisfaction, Individual and Organizational Impact.
Abstract: There h
as been a constant effort to evaluate the success of Information Technology in organizations. This
kind of investment is extremely hard to evaluate because of difficulty in identifying tangible benefits, as
well as high uncertainty about achieving the expected value. Though a lot of research has taken place in this
direction, but not much is written about evaluating IT in non-profit organizations like educational
institutions. Measures for evaluating success of IT in such kind of institutes are markedly different from that
of business organizations. The purpose of this paper is to build further upon the existing body of research by
proposing a new model for measuring effectiveness of computing facilities in academic institutes. As a
baseline, Delone & McLean’s model for measuring the success of Information System (DeLone & McLean
1992,DeLone & McLean 2003) is used, as it is the most pioneering model in this regard.
1 INTRODUCTION
Given the crucial role of education in development
and the expansion of Information and
Communication technology in the global economy,
the role of IT in education cannot be ignored. Of late
there has been a major surge in the use of IT in the
territory of education. This, at the same time, has
raised the questions- How effective is IT in
academic institutions? How to measure the
effectiveness/ success of IT in educational
institutions? Effectiveness is concerned about the
impact of the information provided in helping users
do their job. It is important to evaluate the impact of
the IT on the organization as a whole rather than
looking at the quality of the system, user satisfaction
or by looking at a narrow financial perspective of the
evaluation.
The difficulties in effectively evaluating the
im
pact of information systems are widely
acknowledged in the IS literature (DeLone et al
1992, Willcocks & Lester 1996, Willcocks 1996).
Evidence suggests that poo
r performance of the
IS function is a serious inhibitor to good business
performance (Carlson & McNurlin 1992b). Better
use of information, both internal and external, relates
positively to profitability (Strassman 1990).
A lot of research has been undertaken in this
rega
rd to develop frameworks for measurement of
Information Systems’ success. Economic and
quantitative measures for the success of IS, however,
are difficult to obtain. Researchers and practitioners
alike often rely on subjective assessment and
surrogate measures, such as end-user computing
satisfaction (EUCS) instrument.
Saunders and Jones (1992) developed the "IS
Fu
nction Performance Evaluation Model" which
was used to describe how measures should be
selected from the multiple dimensions of the IS
function relative to specific organizational factors
and based on the perspective of the evaluator.
The model proposed by Delone et al (1992,2003)
to m
easure the effectiveness of Information System
is the most pioneering work in this regard. DeLone
and McLean Information Systems (IS) Success
46
Sharma S. and Bansal V. (2006).
MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION FOR A DIFFICULT PROBLEM.
In Proceedings of the First International Conference on Software and Data Technologies, pages 46-51
DOI: 10.5220/0001313900460051
Copyright
c
SciTePress
Model is a framework and model for measuring the
complex-dependent variable in IS research. It
concludes with a model of "temporal and causal"
interdependencies between their six categories of IS
success- Information Quality, System Quality, Use,
User Satisfaction, Individual Impact, and
Organizational Impact.
Their model depicts the relationships of the 6 IS
success dimensions. They contend that
System Quality and Information Quality singularly
and jointly affect both Use and User Satisfaction.
Additionally, the amount of Use can affect the
degree of User satisfaction. Use and User
Satisfaction are direct antecedents of Individual
Impact; and lastly, this impact on individual
performance should eventually have
some Organizational Impact. This model was later
on validated by many researchers including Seddon
and Kiew (1994), who tested the causal structure of
the model.
Inspite of being the most complete and a better
known model some shortcomings have been sighted
in this model by researchers. It does not take into
consideration the effect of extraneous variables both
internal and external to the organization. They
themselves accept that it is necessary to include the
organization type and its environment into context
before applying this model.
In the light of the above argument, we have made
an attempt to modify Delone and McLean’s model
to make it relevant for measuring the effectiveness
of computing facilities in academic institutes.
Information Quality and System Quality have been
replaced by Usability and Functional Utility. Use
construct is omitted from the proposed model.
Measures for evaluating success of IT in such kind
of institutes are markedly different from that of
business organizations. Therefore, for capturing
Individual Impact and Organizational Impact
measures suitable in the context of academic
institutes have been introduced.
2 PROPOSED MODEL
Following modifications have been proposed in the
Delone & McLean’s model
Replacing System Quality and Information Quality:
We are concerned with measuring effectiveness of
all the computing facilities of an academic institute
unlike (DeLone et al 1992) where focus is on an
individual Information System. Therefore, System
Quality and Information Quality have been replaced
by Usability and Functional Utility.
Omission of Use construct:
A main criticism of Delone and McLean has
centered on the Use construct. It is considered to be
an inappropriate measure of IS success. Its
implication is that if a system is used, it must be
useful, and therefore successful. Take the example
of an expensive design software, which is used only
by handful of students. If this software helps these
students to produce some excellent research work, it
will be considered as an asset for the institute,
irrespective of the number of students using it.
Hence, Use construct was considered as
inappropriate in this context.
Taking the points mentioned above into
consideration, the proposed model includes the
following five constructs- Usability, Functional
Utility, User Satisfaction, Individual Impact and
Organizational Impact. The relationship between the
constructs is as shown in Fig. 1.
Usability
Functional
Utility
User
Satisfaction
Individual
Impact
Organizational
Impact
Figure 1: Proposed model.
MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION
FOR A DIFFICULT PROBLEM
47
This model shows the interdependent nature of
success categories used.
Usability measures the extent to which the
computing facilities match user characteristics and
the skills for the tasks concerned. Functional Utility
focuses on how well the computing facilities meet
the requirements of the users. It also measures the
availability, accuracy and up-to–datedness of the
information obtained from the use of computing
facilities. User satisfaction is the most extensively
used single measure for IS evaluation (Delone et al
1992). End-user’s feelings of satisfaction arise when
he or she combines his or her perception of and
valuation of discrepancy regarding desires and
expectations from the use of computing facilities.
Individual Impact and Organizational Impact
indicate the impact of computing facilities on
individual performance and organizational
performance, respectively. Measures used for
Individual Impact are concerned with evaluating the
impact of computing facilities on an individual in
learning, course work, research work, planning and
decision making, communication and overall
productivity. Likewise, Measures of Organizational
Impact evaluate the impact of comporting facilities
on the organizational as a whole in the following
respects- innovation, research quality, pass
rate/grades, decision making, image of the institute,
capacity in terms of students, and overall
productivity of the institute.
3 MODEL VALIDATION
Aim of testing this model was to provide an
empirical evidence for the relationships between the
five constructs used in the proposed model. We
conducted a self-administered survey to collect the
primary data from the target population, which
consisted of students and faculty of five academic
institutes.
For the survey, a questionnaire was designed
based on discussions with students and faculty and
literature. Respondents were asked to fill the
questionnaire in the context of computing facilities
used in their institutes.
Questionnaire contained five sets of questions to
measure the five constructs of the model.
Questions were framed by discussions with
students and faculty of various academic institutes
and available literature. To evaluate the first
construct Usability, a set of four questions was used.
For measuring Functional Utility six questions were
framed. Four questions on Overall Satisfaction were
from Seddon and Yip (1992). To measure Individual
Impact and Organizational Usability measures the
extent to which the computing facilities match user
characteristics and Impact group of five and six
questions were used, respectively.
Likert scale was used for measurement in which
respondents indicate a degree of agreement or
disagreement with each of a series of statements
about the stimulus objects. Each statement has been
assigned seven response categories, ranging from 1
to 7. One signifies strong agreement, and seven
means strong disagreement.
3.1 Data Collection
Questionnaires were administered personally to the
students and faculty of the aforementioned institutes.
Total of 500 Questionnaires were distributed, out of
which, 411 completed questionnaires were returned
by the respondents. After screening of
questionnaires to identify illegible, incomplete, or
ambiguous responses, 31 questionnaires were
rejected. Total, 380 questionnaires were found
suitable for data analysis. Treatment of missing
values was done by substituting a neutral value.
3.2 Data Analysis and Results
To establish the model, three regression models have
been used
• Multiple regression model with Usability
and Functional Utility as independent
variables and User Satisfaction as
dependent variable.
• Simple regression model with User
Satisfaction as independent variable and
Individual Impact as dependent variable.
• Simple regression model with Individual
Impact as independent variable and
Organizational Impact as dependent
variable.
Using the abbreviations
X
1
= Usability
X
2
= Functional Utility
X
3
= User Satisfaction
X
4
= Individual Impact
X
5
= Organizational Impact
the following linear regressions are considered
X
3
= b
3.12
+ b
31.2
X
1
+ b
32.1
X
2
(1)
ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES
48
X
4
= b
4.3
+ b
43
X
3
(2)
X
5
= b
5.4
+ b
54
X
4
(3)
The eq. (1) represents a multiple linear
regression and (2) and (3) are simple linear
regressions, hereafter called Simple Regression 1
and Simple Regression 2 respectively.
Here b
3.12
, b
4.3
and b
5.4
are constants; b
43
and
b
54
are regression coefficients ,b
31..2
and b
32.1
are
partial regression coefficients. The suffix after the
dot refers to the variable held constant.
3.3 Hypotheses
The hypotheses to be tested are as follows:
H1: The partial regression coefficient b
31..2
> 0
It is assumed that if the user finds the computing
facilities easy to use, perceived usefulness of the
system will increase for him. This subsequently, will
result into increased User Satisfaction.
H2: The partial regression coefficient b
32..1
> 0
Increase in Functional Utility will result into
increased usefulness for the user and hence
increased satisfaction. The more the facilities meet
the requirements of the user the more will be the
User Satisfaction
H3: The regression coefficient b
43
> 0
This hypothesis states that if a student is more
satisfied with the computing facilities then it will
have a more positive Individual Impact e.g. better
learning or communication with students/faculty.
H4: The regression coefficient b
54
> 0
Higher Individual Impact will result into higher
Organizational Impact e.g. a positive effect of
computing facilities on learning of individual
students will result into overall improvement in pass
rate/ grades of the institute.
Data analysis was done using SPSS.
Table 1: Cronbach’s alpha.
No. of Items Cronbach alpha
Usability(X
1
)
4 .6790
Functional
Utility(X
2
)
6 .8479
User
Satisfaction(X
3
)
4 .8497
Individual
Impact(X
4
)
6 .8772
Organizational
Impact(X
5
)
7 .8796
High Cronbach’s alpha for all the variables in
Table 1, except for Usability, which is marginally
less, is an indication of high internal consistency.
Low value for Usability can be attributed to lower
number of items used to measure it.
Table 2: Pearson Correlation matrix.
X
1
X
2
X
3
X
4
X
5
Usability(X
1
)
1
Functional
Utility(X
2
)
.562 1
User
Satisfaction(X
3
)
.602 .815 1
Individual
Impact(X
4
)
.551 .774 .817 1
Organizational
Impact(X
5
)
.537 .722 .769 .812 1
Table 2 shows the Pearson Coefficient of
Correlation between all the variables. Pearson's
correlation coefficient (r) is a measure of the
strength of the association between the two
variables.
The coefficient of correlation between the
constructs Usability and Functional Utility is low,
which indicates their independence. The coefficients
of correlation are high for the constructs Functional
Utility and User Satisfaction; User Satisfaction and
Individual Impact; Individual Impact and
Organizational Impact as suggested by the model.
However, it is on the lower side for the constructs
Usability and User Satisfaction, which suggests that
dependence of User Satisfaction is higher on
Functional Utility as compared to Usability.
Table 3.
R
2
Adjusted R
2
F
(p-value)
Multiple
Regression
.695 .693 428.747
(0)
Simple
Regression 1
.667 .666 757.880
(0)
Simple
Regression 2
.659 .658 729.581
(0)
MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION
FOR A DIFFICULT PROBLEM
49
The high values of t and F- statistic in all the
cases strongly support the rejection of the Null
hypotheses, that the regression coefficients are zero.
The regression coefficients except for b
31.2
have high
positive values. Also the 95% confidence intervals
are small. The coefficients of determination show
reasonably good fit. All the above results tend to
validate the model and support all the four
hypotheses.
4 CONCLUSIONS
Results obtained from path analysis of the survey
data provide considerable empirical evidence for the
model. Results show strong dependence of User
Satisfaction on Usability and Functional Utility;
Individual Impact on User Satisfaction and
Organizational Impact on Individual Impact. All the
four Hypotheses assumed in the beginning of the
research are found be true.
An implication of the model is that because of
the causal nature of these dimensions, Usability,
Functional Utility and User Satisfaction are
sufficient to measure the effectiveness of computing
facilities.
On the basis of the small piece of work done in
this thesis, it is strongly recommended that every
academic institution should undergo through this
self screening or self assessment process. This
model can be used by academic institutes to get
regular feedbacks about their computing facilities,
which will help them in continuous improvements.
An attempt has been made to include all the
suitable measures of each construct. However, there
is a scope of including new measures for each of the
constructs. More questions can be added to the
questionnaire to measure each of these constructs,
including both positive and negative statements to
check the consistency of the respondents. Finally,
inclusion of other constructs in the model can be
investigated.
REFERENCES
Carlson, W.M., & McNurlin, B.C. (1992b). Do you
measure UP?, Computerworld, 26 (49), pp. 95-98.
Delone, W.H. & McLean, E.R. (1992). Information
systems success: the quest for dependent variable,
Information Systems Research 3, March, pp. 60-95.
DeLone, H.W. & McLean, R.E. (2003). The DeLone and
McLean model of information systems success : A
ten-year update, Journal of Management Information
Systems, 19 (4), pp. 9-30.
Saunders, C.S., & Jones, J.W. (1992). Measuring
performance of the information systems function,
Journal of Management Information Systems, 8 (4),
pp. 63-82.
Seddon, P.B. & Yip, S.K. (1992). An empirical evaluation
of user information satisfaction UIS, measures for use
with general ledger accounting software, Journal of
Information Systems, pp.75-92.
Seddon, Peter B. & Kiew, Min-Yen (1994). A partial test
and development of Delone and Mclean’s model of IS
success. Paper presented at the Fifteenth Annual
International Conference of Information Systems
(ICIS).
Strassman, P.A. (1990). The Business value of Computers:
An executive Guide, New Canaan, CI: Information
Economic Press.
Willcocks, Leslie & Lester, Stephanie (1996). Beyond the
IT Productivity Paradox, European Management
Journal 14, No. 3, pp. 279-290.
Willcocks, L. (1996). Investing in Information Systems:
Evaluation and Management, published by Chapman
& Hall, first edition.
Table 4.
Path Unstandardized
Coeff.
Std. Coeff. t (p-
value)
95%
Conf. Bounds
from to
Coeff Std. Er. Lower Upper
H1 Usability User Satisfaction .228 .037 .211 6.126 (0) .155 .302
H2 Functional Utility User Satisfaction .737 .036 .696 20.234
(0)
.666 .809
H3 User Satisfaction Individual Impact .779 .028 .817 27.530
(0)
.723 .835
H4 Individual Impact Organizational Impact .798 .030 .812 27.011
(0)
.740 .856
ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES
50
APPENDIX: SURVEY ON
COMPUTING FACILITIES IN
ACADEMIC INSTITUTES
This questionnaire uses a seven-point scale. The
scale represents a spectrum. 1 signifies that you
strongly agree with the given statement, and 7 means
you strongly disagree. For each question tick the
number that reflects what you think about each
statement. Computing facilities refer to computer
hardware, software and network of your institute.
PART A: Usability
1 Computing facilities are easy to use. 1 2 3 4 5 6 7
2 Computing facilities are user friendly. 1 2 3 4 5 6 7
3 It is easy to acquire skills for using the Computing facilities. 1 2 3 4 5 6 7
4 It requires lot of effort to use the Computing facilities 1 2 3 4 5 6 7
PART B: Functional Utility
1 Computing facilities meet most of your requirements. 1 2 3 4 5 6 7
2 The content of information obtained with the help of computing facilities meets your
requirements.
1 2 3 4 5 6 7
3 Computing facilities are available whenever required. 1 2 3 4 5 6 7
4 You can get in touch with sufficient sources of information by using computing facilities.
1 2 3 4 5 6 7
5 Computing facilities enable you to obtain accurate information. 1 2 3 4 5 6 7
6 Computing facilities enable you to obtain up-to-date information. 1 2 3 4 5 6 7
PART C: User Satisfaction
1 Computing facilities meet your information processing and computational needs. 1 2 3 4 5 6 7
2 Computing facilities are fast enough. 1 2 3 4 5 6 7
3 Computational facilities are effective. 1 2 3 4 5 6 7
4 Overall,you are satisfied with the computing facilities. 1 2 3 4 5 6 7
PART D: Individual Impact
1 Computing facilities help you in learning. 1 2 3 4 5 6 7
2 Computing facilities help you in course work. 1 2 3 4 5 6 7
3 Computing facilities help you in research work. 1 2 3 4 5 6 7
4 Computing facilities help you in planning and decision making. 1 2 3 4 5 6 7
5 Computing facilities help you in communication with teachers and students. 1 2 3 4 5 6 7
6 Computing facilities help you in improving your overall productivity. 1 2 3 4 5 6 7
PART E: Organizational Impact
1 Computing facilities help in encouraging innovation. 1 2 3 4 5 6 7
2 Computing facilities help in improving research quality. 1 2 3 4 5 6 7
3 Computing facilities help in improving overall pass rate/grades. 1 2 3 4 5 6 7
4 Computing facilities help in better decision making. 1 2 3 4 5 6 7
5 Computing facilities help in improving the image of the institute. 1 2 3 4 5 6 7
6 Computing facilities help you in increasing capacity in terms of students. 1 2 3 4 5 6 7
7 Computing facilities help in improving overall productivity of the institute. 1 2 3 4 5 6 7
MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION
FOR A DIFFICULT PROBLEM
51
