An empirical test of the DeLone-McLean model of information system success

An Empirical Test of the DeLone-McLean Model of Information System Success Juhani Iivari University of Oulu

Acknowledgment I wish to express my gratitude to Minna Perälä, M.Sc., for the data collection, and especially to Prof. Wynne Chin for his comments and for helping me to use PLS. This paper was submitted in February of 2002. Wynne Chin served as the Senior Editor.

Abstract This paper tests the model of information system success proposed by DeLone and McLean using a field study of a mandatory information system. The re-sults show that perceived system quality and perceived information quality are significant predictors of user satisfaction with the system, but not of system use. Perceived system quality was also a significant predictor of system use. User satisfaction was found to be a strong predictor of individual impact, whereas the influence of system use on individual impact was insignificant.

ACM Categories: J.1, K.6.2

Keywords: Information System Success, Infor-mation System Quality, System Quality, Information Quality, User Satisfaction, Use, Individual Impact Introduction Seddon et al. (1999) estimate that the total annual worldwide expenditure on information technology (IT) probably exceeds one trillion US dollars per year and is growing at about 10% annually. At the same time, information systems are pervading almost all aspects of human life. In view of the high investments in IT and its ubiquity, the success of such investments and the quality of the systems developed is of the utmost importance both for research and in practice.

This paper focuses on the success of individual information system applications. Following Gustafsson et al. (1982), we interpret an information system (IS) as a computer-based system that provides its users with information on specified topics in a certain organizational context. DeLone and McLean (1992) proposed in their influential paper a framework for IS success measures that distinguishes system quality, information quality, user satisfaction, use, individual impact and organizational impact. They also suggested a causal model for the success measures.

Despite the considerable interest in the DeLone-McLean model1, there is a dearth of studies that test it empirically. DeLone and McLean (2002) identify only sixteen empirical studies that have explicitly tested some of the associations of the original DeLone-McLean model. Among them Seddon and Kiew (1994) revised it considerably, by deleting system use and substituting perceived usefulness. In our view perceived usefulness reflects more the individual impact (Rai et al., 2002), i.e. the impact of the system

1 The Science Citation Index, Social Science Citation Index and Arts & Humanities Index identify 235 references to the article (as of January 10, 2002).

8 The DATA BASE for Advances in Information Systems - Spring 2005 (Vol. 36, No. 2)

on a user’s performance of his/her job.2 The idea of this paper is to test the DeLone-McLean model while sticking more faithfully to its original form. Leidner (1998) reports a partial test of the model in the case of Executive Information Systems, and more recently, Rai et al. (2002) tested both the DeLone-McLean (1992) model and the Seddon (1997) model, reporting reasonable support for both.

The composition this paper is as follows: Section 2 discusses the theoretical background; Section 3 introduces the research method; Section 4 describes the results; Section 5 discusses the results; and Section 6 concludes the paper. Theoretical Background The DeLone-McLean Model for IS Success

The DeLone-McLean model for IS success, described in Figure 1, assumes that system quality and information quality, individually and jointly, affect user satisfaction and use. It also posits use and user satis-faction to be reciprocally interdependent, and presumes them to be direct antecedents of individual impact, which should also have some organizational impact.

DeLone and McLean (1992) characterize system quality as desired characteristics of the information system itself, and information quality as desired characteristics of the information product. More concretely, they incorporate four scales from the Bailey-Pearson (1983) instrument into system quality (convenience of access, flexibility of the system, inte-gration of the system and response time) and nine scales into information quality (accuracy, precision, currency, timeliness, reliability, completeness, conciseness, format and relevance).

Much of the research on User Information Satisfaction has concerned users’ satisfaction with specific features of a system (Doll & Torkzadeh, 1988; Iivari & Koskela, 1987) or IS function (Bailey & Pearson, 1983; Baroudi & Orlikowski, 1988), covering features of both system quality and information quality. Even though the inclusion of service quality in the updated DeLone and McLean (2002) model reflects IS functions or IS organizations rather than IS application, the following will focus on the success of IS applications only. User satisfaction in DeLone and McLean (1992) refers to the overall user satisfaction (Seddon & Kiew, 1994) measured independently of

2 Davis’ original measure for perceived usefulness was developed to assess ex ante expectations of individual impact. We focus here, however, on ex post individual impact measured after six months’ experience of use.

system quality and information quality. Otherwise the relationship between system/information quality and user satisfaction would be an artifact of measure-ment.

Seddon (1997) claims that the DeLone-McLean model is ambiguous in the sense that one component of it, use, has three potential meanings (Table 1). His conclusion is that only Meaning 1 is justified in the light of the objections listed in the second column of Table 1. To me all these points of criticism seem questionable. His criticisms of Meaning 2 and Meaning 3 refer to the distinction between a variance model and a process model (Mohr, 1982).3 Without going into the details of this distinction, it is obvious that even though IS use as a process is assumed to lead to individual impact and organizational impact, it is not necessary to regard it as a discrete event to be stated (use vs. non-use), as implied by process theories (Mohr, 1982).4 This paper interprets use as the amount of use, which may be considered one measure of IS success.

DeLone and McLean (1992) characterize individual impact as “an indication that an information system has given a user a better understanding of the decision context, has improved his or her decision-making productivity, has produced a change in user activity, or has changed the decision maker’s perception of the importance or usefulness of the in-formation system” (p. 69). Seddon (1997) reinterprets individual impact to mean benefits accruing to individuals from use. Even though both DeLone and McLean (1992) and Seddon (1997) implicitly presuppose that individual impact is of benefit to the user, this paper interprets individual impact as referring to a unit of analysis rather than the benefi-ciary of the impact.5

3 As Mohr (1982, p.44) points out, he uses the term “process theory” in a highly specific meaning. “Process theory” does not imply that variance theories cannot address processes (e.g. IS acceptance). 4 We interpret the DeLone-McLean model as based on the reasoning that a system that is not used at all does not have any individual or organizational impact. On the other hand, the DeLone-McLean model also allows the hypothesis that more use is associated with more individual impact, which follows the “logic” of variance theories. 5 The likely explanation for this assumption is that the above authors implicitly assume that use of the system is voluntary. In that case a user will hardly continue to use a system if he or she does not perceive its use as beneficial. However, according to my reading, DeLone and McLean (1992) do not explicitly restrict their model to voluntary systems, although they do note that actual use makes sense only when system use is voluntary.

The DATA BASE for Advances in Information Systems - Spring 2005 (Vol. 36, No. 2) 9

System Quality

Information Quality

Use

User Satisfaction

Individual Impact

Organizational Impact

Figure 1. The DeLone-McLean Model for IS Success

Meaning Seddon’s objections Counter-objections Meaning 1: Benefits from use

(The only justified mean-ing)

What is the meaning of “use” in “benefits from use”?

Meaning 2: Use as the dependent vari-able in a variance model of future use

IS success must bring benefits to somebody.

Can’t a system (e.g. a piece of free software such as Linux) be genuinely considered a suc-cess when it is widely used without any consi-deration of its benefits or disadvantages to different stakeholders?

Meaning 3: Use as an event in a pro-cess leading to indi-vidual or organiza-tional impact

IS use is a process con-struct that should not have any place in a variance model predicting IS success.

Even though IS use as a process is assumed to lead to individual impact and organizational impact, it is not necessary to regard it as a discrete event to be stated (use vs. non-use), as implied by process theories.

Table 1. The Three Meanings of IS Use in the DeLone-McLean Model, According to Seddon (1997)

Following DeLone and McLean (1992) and Rai et al. (2002), we will specifically focus in this paper on the effect of an information system on the work perform-ance of individual users as measured by perceived usefulness.6 Hypotheses DeLone and McLean (1992) introduce the model shown in Figure 1 primarily as a causal-explanatory model of how system quality and information quality affect use and user satisfaction, how use and user satisfaction, affecting each other reciprocally, are direct antecedents of individual impact, and how individual impact leads to organizational impact. As an alternative, one could emphasize more the predictive nature of the model, how the preceding variables help to predict the dependent variables, 6 We do claim that perceived usefulness covers all aspects of individual impact. DeLone and McLean (1992) specifically focus on decision-makers as users of an information system. Assuming that the work of decision-makers is to make decisions, perceived usefulness essentially covers the impact on decision-making productivity. Perceived usefulness nevertheless misses those aspects of individual impact which do not directly concern work performance, e.g. the impact on the quality of work (Iivari, 1997).

even though the causal explanation of the relationship is not totally clear. The criticism of Seddon (1997), even though we do not accept it in its entirety, shows that some of the assumed causal relationships in the DeLone-McLean model are arguable and the model is incomplete. In particular, the model misses the feedback loops from individual impact and organiza-tional impact to user satisfaction and use. We interpret the DeLone-McLean model primarily as a predictive one that is worth testing empirically. Based on the DeLone-McLean model, we propose to test the hypotheses depicted in Figure 2 in the present paper. It is hypothesized in Figure 2 that system quality and information quality are positively associated with user satisfaction. Hypothesis H1 assumes that ceteris paribus the higher the system quality is perceived to be by users, the more satisfied they are with the system. Similarly, Hypothesis H2 posits that ceteris paribus the higher the information quality is perceived to be by users, the more satisfied they are with the system. If user satisfaction is interpreted as an attitude (Baroudi et al., 1986), hypotheses 1-2 essentially argue that the attitude is dependent on perceptions of the attitude object (Fishbein & Ajzen,


1975; McGuire, 1969). There is considerable empirical evidence for these hypotheses.

S ystem quality

I information quality

Actual use

User satisfaction

Individual impact

H7:+

H6:+

H3:+

H4:+

H1:+

H2:+

H5a:+ H5b:+

Figure 2. The Model to Be Tested

Many of the instruments developed to measure User Information Satisfaction (UIS) in terms of attributes such as system quality and information quality (e.g. Bailey & Pearson, 1983; Ives et al., 1983; Doll & Torkzadeh, 1988) have used an independent measure of overall user satisfaction to test the predictive validity of the measure. They have consistently reported significant correlations between UIS or its factors and the independent measure of overall satisfaction. Doll and Torkzadeh (1988), for example, found correlations varying between 0.51 and 0.65 between the twelve items of their end-user satisfaction measure and the criterion variable that can be interpreted as overall satisfaction. The correlations between the five factors (content, accuracy, format, ease of use and timeliness) and the criterion varied between 0.55 and 0.69, and that between the 12-item instrument and the criterion variable was 0.76. Further, Seddon and Kiew (1994) found in their path analysis that information quality and system quality are significant determinants of overall user satisfaction (both path coefficients significant at the level 0.001).7 Similarly Rai et al. (2002) report significant path coefficients between ease of use (used to measure system quality) and user satisfaction (ß = 0.30, p ≤ 0.01) and between information quality and user satisfaction (ß = 0.52, p ≤ 0.01) in their LISREL analysis of the DeLone-McLean model.

DeLone and McLean (1992) hypothesize that the higher the system quality, the more the system is used (Hypotheses H3) and the higher the information quality, the more the system is used (Hypothesis H4).

7 System quality was measured using two items from Doll and Torkzadeh (1988), four items of ‘ease of use’ from Davis (1989) and three additional items. The measure of information quality consisted of ten items from the Doll and Torkzadeh (1988) instrument. (Overall) satisfaction with the system was measured using four items: how adequately the application meets the information processing needs, how efficient it is, how effective it is and overall satisfaction.

At a general level, there is a considerable body of em-pirical research on the relationship between UIS (measured in terms of attributes such as system quality and information quality) and IS use, which suggests that the relationship is positive but relatively weak (Amoroso & Cheney, 1991; Barki & Huff, 1985; Baroudi et al., 1986; Ginzberg, 1981; Igbaria, 1990; Igbaria & Zviran, 1991; Nelson & Cheney, 1987; Srinivasan, 1985). Baroudi et al. (1986), for example, found a correlation of 0.28 between UIS and IS use, and Barki and Huff (1985) 0.39. More specifically re-lated to Hypothesis H3, the Technology Acceptance Model (Davis et al., 1989) predicts that perceived ease of use, as an aspect of system quality (DeLone & McLean, 1992), is a significant direct and indirect determinant of use, the indirect effect being channelled through perceived usefulness. After a de-cade of intensive research into TAM, there is significant empirical evidence of the indirect effect of perceived ease of use (Davis, 1989; Davis et al., 1989; Mathieson, 1991; Adams et al., 1992; Davis et al., 1992; Igbaria et al., 1995; Igbaria & Iivari, 1995; Chau, 1996; Igbaria et al., 1996; Szajna, 1996; Taylor & Todd, 1995, Gefen & Straub, 1997; Igbaria et al., 1997; Straub et al., 1997; Gefen & Keil, 1998; Karahanna & Straub, 1999; Karahanna et al., 1999; Venkatesh & Davis, 2000), whereas the direct effect is much more controversial (Gefen & Straub, 2000). Consistent with the above, Rai et al. (2002) report a relatively low path coefficient between ease of use and system dependence (used to measure system use) (ß = 0.09, p ≤ 0.10) and a somewhat higher coefficient between information quality and system dependence (ß = 0.18, p ≤ 0.01). Leidner (1998) reports positive associations between perceived EIS information quality and frequency of EIS use (ß = 0.38, p ≤ 0.01) and between perceived EIS information quality and EIS use for internal monitoring (ß = 0.34, p ≤ 0.01), but not between EIS information quality and EIS use for external monitoring nor between EIS information quality and EIS use for communication. Despite these somewhat inconsistent findings, we will follow the original model of DeLone and McLean (1992) as expressed in Hypotheses H3 and H4.

As mentioned above, DeLone and McLean (1992) posit that use and user satisfaction are reciprocally interdependent. To test this reciprocal dependence fully, one should have a piece of research in which use and user satisfaction are followed over time. This paper is confined to a single point in time, however, although hypotheses 5a and 5b, which are to be tested separately, do attempt to capture this reciprocal dependence.

Hypothesis H5a predicts that the more satisfied users are with the system, the more they will use it. Baroudi


et al. (1986) suggest that if user satisfaction is interpreted as an attitude, the Theory of Reasoned Action (Fishbein & Ajzen, 1975) supports the model that user satisfaction will influence intentions to use the system and actual use (Hypothesis 5a). As an alternative, they identify Dissonance theory (Fishbein & Ajzen, 1975), which suggests that IS use leads to user satisfaction (Hypothesis H5b). The results of path analysis supported the model that user satisfaction leads to system use rather than vice versa (Baroudi et al., 1986). Interestingly, they explain the model as follows: “The model assumes that as use demonstrates that a system meets a user’s needs, satisfaction with the system should increase, which should further lead to greater use of that system. Conversely, if system use does not meet the user’s needs, satisfaction will not increase and further use will be avoided.” This explanation suggests that, causally, IS use precedes user satisfaction, or that the relationship is reciprocal, as assumed by DeLone and McLean (1992). In fact, Torkzadeh and Dwyer (1994) found a path coefficient of 0.21 (p < .05) from user satisfaction to usage and a coefficient of 0.37 (p < .05) from usage to user satisfaction in their LISREL analysis. Rai et al. (2002) report a significant path coefficient (ß = 0.35, p ≤ 0.01) from user satisfaction to system dependence, used to measure system use.

The relatively low association between actual use and user satisfaction may be explained by the complexity of the relationship. Chin and Lee (2000) propose that overall user satisfaction is composed of expectation-based satisfaction and desire-based satisfaction. Ex-pectation-based satisfaction is a direct and multiplicative combination of the overall expectation discrepancy between prior expectation and post hoc perceptions of the system and the overall evaluation of this expectation discrepancy. Similarly, desire-based satisfaction is a direct and multiplicative combination of the overall desire discrepancy be-tween prior desires and post hoc perceptions of the system and the overall evaluation of this desire discrepancy.

Seddon (1997) reasons that although user satisfaction may have an association with IS use in a steady state, this will break down in a situation of system replacement. Favourable satisfaction with the old system is not sufficient “to cause use of the old system”. Although this may be true, it is obvious that user satisfaction is never the only cause of system use. At a minimum, the system must be accessible to a user. Seddon’s (1997) reasoning assumes that user satisfaction with the old system is not affected by the new system. To the author’s knowledge, this issue has not been studied empirically, but one could speculate that a user would compare the old and new systems. After this comparison it seems quite unlikely

that a user’s satisfaction will not be affected by the new system. One could conjecture that dissatisfaction with the new system might lead to higher satisfaction with the old system, whereas high satisfaction with the new system may reduce satisfaction with the old system.8 Further, if the user is dissatisfied with the new system after the replacement, high satisfaction with the old system may explain the likelihood of its future use.

Contrary to Chin and Lee (2000), Seddon (1997) also assumes that user satisfaction reflects past experience with the system and does not include expectations. Iivari (1987) points out, however, that if user satisfaction is interpreted as the user’s belief in the degree to which the information system (at the IS schema level) is capable of satisfying his or her information requirements (at the IS schema level) (Ives, et al., 1983), the distinction between ex post (experience-based) and ex ante (expectation-based) interpretation disappears. At the same time, he argues that in this case user satisfaction is consequent upon acts of using a system rather than being an antecedent, especially when one takes into account alternative and complementary information systems. The explanation is that information needs in a specific use situation may differ from information requirements imposed on the system at the schema level. A manager, when he or she has time, may prefer to visit the shop floor to inquire directly from the employees about the state of production, because he or she wishes to receive “soft” knowledge of a kind that a formal information system is assumed to be incapable of supplying. He or she may nevertheless be totally satisfied with the reporting system and use it when there is no opportunity to visit the shop floor. At the same time, Iivari (1987) acknowledges that user satisfaction may correlate highly with IS use. We also claim that it may predict IS use.

Hypothesis 6 predicts that user satisfaction will be positively associated with individual impact. The meaning of this hypothesis depends on the interpretation of user satisfaction which users employ. Goodhue (1986) defines “IS satisfactoriness” as the individual’s belief in the correspondence or fit between job requirements and IS functionality, and “IS satisfaction” as the correspondence between the information system’s intrinsic benefits of use, such as providing a sense of accomplishment due to a crisp, attractive output, and the needs of the individual. Iivari

8 Relative advantage, i.e. “the degree to which an innovation is perceived as being better than the idea it supersedes” (Rogers, 1995, p. 212) captures this idea of comparing the old and new systems. Our conjecture above would suggest that relative advantage is a dynamic concept that emerges in the form of an interaction between the levels of satisfaction with the two systems.


and Ervasti (1994) propose that when user satisfaction is interpreted in the sense of “IS satisfactoriness” rather than “IS satisfaction,” the link between user satisfaction and IS effectiveness can be established directly without the intervening variable of IS use. If user satisfaction is the user’s best estimate of the match between the requirements imposed on the system by his or her work and the system capa-bilities, the positive association between user satisfaction and individual impact is quite understandable. If the user’s interpretation of these requirements and estimate of the match between them and the system capabilities are correct, increased user satisfaction should be positively associated with task performance.

There are relatively few studies that have investigated empirically the relationship between user satisfaction and individual impact, especially when we focus on the effect on individual job performance. DeLone and McLean (1992) identify four papers that comprise both user satisfaction and individual impact criteria, among which only Cats-Baril and Huber (1987) actually address the relationship, reporting correlations between two measures of satisfaction on the one hand and the quality and productivity of decision-making on the other hand. Productivity was measured in terms of the number of objectives generated, the number of alternatives generated and the number strategies prioritized. The authors do not report the significances, but all the correlations were negative, albeit relatively low in absolute terms. One of the eight correlations was just over 0.30 in absolute terms. More recently, Gatian (1994) conducted a LISREL analysis of the relationship between user satisfaction, decision performance and user efficiency in the case of direct and indirect users of the same system in 39 organizations. In the case of direct users, she found a close association between user satisfaction and decision performance (r = 0.64, ß = 0.64) and similarly between user satisfaction and efficiency (r = 0.68, ß = 0.97), but in the case of indirect users only the relationship between user satisfaction and user efficiency was of interest. This was also found to be significant (r = 0.81, ß = 0.81). Her findings may partly be inflated by the fact that her user satisfaction measure was adapted from the Jenkins-Ricketts (1979) instrument and decision performance from Sanders (1985). User efficiency was also evaluated in terms such as user’s data processing correctness, report preparation and distribution timeliness. Etezadi-Amoli and Far-hoomand (1996) report that six factors of end user computing satisfaction (documentation, ease of use, functionality, quality of output, support and security) explained 50% of the variance in end user performance. They also found that satisfaction with the quality of the output and satisfaction with the

functionality of the system were the most significant predictors, whereas documentation was the least significant. The paper of Seddon and Kiew (1994) also analyses the relationship between user satisfaction and individual impact when the latter is interpreted as perceived usefulness. They report a correlation 0.70 (p < 0.001) between them.9

Finally, Hypothesis 6 predicts that IS use is positively associated with individual impact. Theoretically, this relationship can be argued based on the reasoning that a system that is not used at all will not have any impact on individual performance. Furthermore, one could also expect that a system that is used more will have higher impact on users’ performance. Chin and Marcolin (2001) point out, however, that the impact of initial usage on individual productivity may differ from that of continued usage. The interest of the present paper lies in the continued use of a system rather than its initial use. There seems to be a paucity of research into the relationship between IS use and individual impact, but the existing evidence seems to support this hypothesis. DeLone and McLean (1992) identify seven studies that address both system use and individual impact. Among these, Srinivasan (1985) reports time per session (connect time) and user type (light, average, heavy), among four indicators of system use, to be significantly correlated with the problem solving capabilities of the user (p ≤ 0.10 and p ≤ 0.01, respectively). Snitkin and King (1986) found a significant association between sys-tem usage and perceived effectiveness (p ≤ 0.05). More recently, Iivari (1996) found CASE usage to have a significant effect on the productivity of individual users (systems developers) and on the quality of their products. Leidner (1998) reports that EIS use to monitor internal information and EIS use to monitor external information were both positive and significant predictors of individual decision making speed (ß = 0.34, p ≤ 0.05; ß = 0.24, p ≤ 0.05, respectively), mental mode enhancement (ß = 0.38, p ≤ 0.01; ß = 0.28, p ≤ 0.05, respectively) and extent analysis in decision making (ß = 0.36, p ≤ 0.01; ß = 0.25, p ≤ 0.05, respectively). On the other hand, frequency of EIS use was not a significant predictor of any of the three aspects of individual impact.

On the other hand, there is an abundant literature on the effect of perceived usefulness on system use (Lee, 2003; Ma and Liu, 2004), where perceived usefulness is interpreted as an ex ante belief (expectation) regarding “the degree to which a person believes that using a particular system would enhance

9The high correlation between overall satisfaction and perceived usefulness may be explained by the fact that two of the four items of user satisfaction, concerning the efficiency and effectiveness of the application, may be interpreted in terms of perceived usefulness.


his or her job performance” (Davis, 1989). As noted above, there is significant empirical evidence that per-ceived usefulness has a positive effect on attitudes towards use, behavioural intention to use and actual use. Perceived usefulness as an antecedent of system use serves to point out that the relationship between system use and individual impact is probably not uni-directional but that a user’s experience of individual impact (in terms of its effect on his/her job performance) will affect his/her belief in the perceived usefulness of a system and in that way its use. Research Methodology The Field Study

The model of Figure 2 was tested as a part of a larger, longitudinal field study in Oulu City Council, which is a municipal organization of about 7500 employees. The organization in question formed a concrete setting where some of its employees were working on the adoption of a new information system and shaping its organizational acceptance. We saw it as an opportunity to study a specific case of “real” users’ acceptance of a “real” system in “real” time. Even though the study focuses only on one system in one organization, the research setting can be expected to increase the internal validity of the study in particular and to some extent its external validity as well (Jenkins, 1985). The choice of one organization controls for possible confounding effects of organiza-tional level variables such as institutional constraints and infrastructure arrangements, which may have an influence on individual adoption and acceptance, making it more likely that micro-level effects will be detected (Karahanna et al., 1999).

Oulu City Council renewed its financial and account-ing systems at the beginning of 1997 as an outcome of a nation-wide reform of municipal financial and accounting systems. As a part of this reform, it acquired an application package from a major vendor in Finland, including accounting, sales receivable, payments receivable and invoicing. The field study was targeted at about 100 primary users of the sys-tem who participated in the training provided by the vendor in October and November 1996, of whom 78 agreed to participate. Data collection, based on questionnaires, was conducted during summer 1997 after half a year of experience with the system. Measurement of the Variables

The questionnaire was based as much as possible on standard measures (see Appendix A), the questions being translated into Finnish. System quality was measured using six scales adopted from Bailey and Pearson (1983): flexibility of the system, integration of the system, response/turnaround time, error recovery,

convenience of access, and language. Similarly, information quality adopted six scales from Bailey and Pearson (1983): completeness, precision, accuracy, reliability, currency, and format of output. Each scale was measured using four items, as proposed by the source authors.

User satisfaction was measured using the six items of general reactions suggested by Chin et al. (1988) and actual use in terms of daily use time and frequency of use. Individual impact was confined to impact on the user’s work performance and was measured with an adaptation of the 6-item instrument for perceived usefulness suggested by Davis (1989).10 Data Analysis

Gefen et al. (2000) provide a recent comparison of traditional regression analysis and two classes of structural equation modelling, covariance-based and partial-least-square-based, which are potentially suitable for testing the model of Figure 2. Their guidelines suggest that covariance-based structural equation modelling approaches such as LISREL, EQS and AMOS do not suit the case at hand, for two major reasons. Firstly, they are suitable for confirmatory rather than exploratory analyses and require a strong underlying theory. Actually, the covariance-based structural equation modelling is oriented towards causal modelling and theory testing rather than prediction (Chin & Newsted, 1999), which is the major, purpose of the present paper. As pointed out by Seddon (1997), the underlying theory of the DeLone-McLean model is not very strong. Secondly, the number of cases in the present material, 68, is quite small for covariance-based structural equation modelling methods, which also impose tighter statistical assumptions than regression analysis and partial-least-square-based methods.

The hypothesized relationships among the study variables depicted in Figure 2 were tested by the Partial Least Squares (PLS) method, which is particularly well suited for predictive applications and theory building (Chin & Newsted, 1999). It does not imply parametric assumptions of multivariate normal distribution, and the sample size can be small, the minimum being ten times of the number of items in the most complex construct in the model (Chin, 1998; Gefen et al., 2000).

PLS recognizes two components of a causal model: the measurement model and the structural model. A structural model consists of the unobservable, latent constructs and the theoretical relationships among 10Observe, however, that perceived usefulness is mainly used to measure ex ante expectations of the system’s impact in terms of speed of accomplishing tasks, job performance, productivity, effectiveness, ease of job and usefulness in work (Davis, 1989).


them. Testing this includes estimating the path coeffi-cients, which indicate the strengths of the relationships between the independent and de-pendent variables. Furthermore, for each construct in a structural model, there is a related measurement model which links the latent construct in the diagram with a set of observed items. The measurement model consists of the relationships between the observed variables (items) and the latent constructs which they measure. The characteristics of this model demonstrate the construct validity of the research instruments, i.e. the extent to which the operationali-zation of a construct actually measures what it pur-ports to measure. Two important dimensions of construct validity are (a) convergent validity, including reliability, and (b) discriminant validity. Together, the structural and measurement models form a network of constructs and measures. The item weights and loadings indicate the strengths of the measures, while the estimated path coefficients indicate the strengths and signs of the theoretical relationships.

More specifically, a molar approach (Bagozzi, 1985; Chin & Gopal, 1995) was adopted to the testing of the model in Figure 2. Both system quality and information quality were considered to be second order concepts influenced by six scales each (see Figure 3). To measure system quality, the 24 items covering flexibility, system integration, response time, recoverability, convenience, and common language were used as reflective measures (Chin, 1998). Similarly, the 24 items covering completeness, preci-sion, accuracy, consistency, currency, and format were used to measure information quality. Results The model of Figure 2 was tested using PLS-Graph, version 03.00 software. As Hypothesis H5 in Figure 2 includes a mutual influence between use and user satisfaction that could not be tested at the same time, we tested two models: Model 1, which assumed the influence to be from user satisfaction to actual use (H5a), and Model 2 which worked from actual use to user satisfaction (H5b). Measurement Models

To test the measurement models Model 1 and Model 2, we examined (1) individual item loadings, (2) internal consistency (reliability of measures), (3) convergent validity, and (4) discriminant validity. All the item loadings except for five of the 24 items concerning perceived system quality and four of the 24 items concerning perceived information quality ex-ceeded the threshold value of 0.70. The internal con-sistencies of the latent variables, the average variances explained (on the diagonal) and the

correlations between latent variables are listed in Table 2. The two models gave very similar results. Only a few correlations between latent variables differed by the absolute value of 0.01.11

The weights and loadings of the indicators in both models are reported in Appendix B.

The internal consistencies of all the latent constructs, examined using the formula developed by Fornell and Larcker (1981), clearly exceeded the cut-off value 0.70 proposed by these authors. Convergent validity is considered adequate when the average variance extracted is 0.50 or more (Fornell & Larcker, 1981). As Table 2 shows, the minimum average variance extracted was 0.54. For satisfactory discriminant validity, the average variance shared between a construct and its measures should be greater than the variance shared by the construct and any other constructs in the model (Chin 1998). As Table 2 shows, the square roots of the average variance extracted (the lower values on the diagonal) exceed the corresponding off-diagonal correlation values in the corresponding rows and columns in all cases except for perceived system quality and perceived information quality, where the violations concern their antecedents in the molar model of Figure 3. This indicates adequate discriminant validity for the constructs.

As shown in Table 2, the correlations between flexibility, system integration, response time, recoverability, convenience and command language vary between 0.35 and 0.76. Analysis of collinearity between them showed that the lowest tolerance value was 0.22, which clearly exceeds the cut-off value 0.10 recommended by Hair et al. (1992). Similarly, the correlations were between 0.51 and 0.93. The tolerance value was 0.12, which is very close to the above cut-off value. Structural Models

The tests performed on the structural models gave the results depicted in Figure 3. The upper path coef-ficients and R2 values give the results for Model 1 and the lower ones for Model 2. The path coefficients from the antecedents of perceived system quality and perceived information quality were the same in both models and are therefore not repeated. The bootstrap resampling technique (500 resamples) was used to determine the significances of the paths within the structural model. As shown in Figure 3, perceived system quality is a very significant predictor of user satisfaction in both models (ß = 0.55 and 0.48,

11 In these two cases, Table 1 lists the higher correlations in absolute values.


Per- ceived system quality

Actual use

User satis-

faction

Per- ceived

information quality

Indivi- dual

Impact

Flexi- bility

0.19***0.21***

0.19***

0.23***

0.22***0.16***

Inte- gration

Res- ponse time

Re- cover- ability

Con. veni. ence

Langu- age

Comple- teness

0.19***0.20***

0.18***

0.23***

0.22***0.16***

Preci- sion

Accu- racy

Con. sis-

tency

Cur- rency

Format

0.30* 0.45**

0.26* 0.28*

0.55*** 0.48***

-0.19 -0.12

0.28Õ

0.15 0.15

0.52*** 0.52***

0.57 0.58

0.18 0.14

0.35 0.35

0.14Õ

Figure 3. Results of the Structural Analyses

respectively; p ≤ 0.001 in both cases), while perceived information quality is also a significant predictor (p ≤ 0.05) of user satisfaction in both models. Perceived system quality is also a significant direct predictor of system use both in Model 1 (ß = 0.30, p ≤ 0.05) and in Model 2 (ß = 0.45, p ≤ 0. User satisfaction is an almost significant predictor of actual use in Model 1 (ß = 0.28, p ≤ 0.10), and conversely actual use is an al-most significant predictor of user satisfaction in Model 2 (ß = 0.14, p ≤ 0.10). Most notably, user satisfaction is a strong predictor of individual impact in both models (ß = 0.52, p ≤ 0.001 in both), whereas actual use is insignificant as a predictor of individual impact. Overall, both models explain a considerable portion of the variance in both user satisfaction (R2 = 0.57 in Model M1 and R2 = 0.58 in Model 2) and individual impact (R2 = 0.35 in both models). Discussion The findings regarding the seven hypotheses derived from the DeLone-McLean model (1992) proposed in section 3.2 are summarized in Table 3. Overall, the results support the reasonableness of the DeLone-McLean model as a predictive model.

The paths from system quality and information quality to user satisfaction and from user satisfaction to individual impact in particular emerged in the manner hypothesized by the DeLone-McLean model. On the other hand, the paths from system quality and in-formation quality to actual use and from actual use to individual impact were not significant. This negative finding may be explained by the mandatory nature of the system, which may inflate the significance of actual use in the model. Our test of the DeLone-McLean model was therefore limited, and there is need to test it in the case of more voluntary systems.

As explained in section 3.2, we do not claim that the DeLone and McLean model provides a complete picture if interpreted as a causal-explanatory model. It is unclear, for example, whether our finding that user satisfaction predicts individual impact implies that user satisfaction in some sense explains individual impact or vice versa. Even though Seddon (1997) claims that “in the long run it is people’s observations of the outcomes of use, the impacts, that determine their satisfaction with the system, not vice versa” (p. 243), we see this more as a research issue that is quite complicated by nature. Our results may be interpreted as indicating that users perceive user satisfaction in terms of “IS satisfactoriness” or as a weighted combination of “IS satisfactoriness” and “IS satisfaction” rather than “IS satisfaction” alone (Goodhue, 1986). Assuming that user satisfaction is the user’s best estimate of the match between the requirements imposed on the system by his or her work and the system’s capabilities, a positive relation-ship between user satisfaction and individual impact is quite understandable. If the user’s interpretation of those requirements and estimate of the match between requirements and the system’s capabilities are correct, increased user satisfaction should be positively associated with task performance.

Our test of the DeLone-McLean model was incomplete in the sense that we did not include organizational impact. In addition to the data analysed above, we had access to managers’ evaluations of the organizational impact of the system (n = 38) in 15 user departments.12 Organizational impact was evaluated in terms of the impact of the system on the unit’s output, the quality of this output and the unit’s innovativeness, reputation for excellence and morale (Van de Ven & Ferry, 1980). Factor analysis of the five items based on the 38 responses gave only one factor.

12 The 38 managers’ evaluations were largely independent of the 78 user responses. There were only four common respondents in the two samples.


Inter-correlations of latent constructs and average variances extracteda

Con

stru

ct

No

of It

ems

Int.

Con

sist

ency

PSQ

PIQ

UIS

USE

Ind

Im

pact

Flex

ibilit

y

Inte

grat

ion

Res

pons

e

Rec

over

abilit

y

Con

veni

ence

Com

lang

uage

Com

plet

enes

s

Prec

isio

n

Accu

racy

Con

sist

ency

Cur

renc

y

Form

at

Perceived Syst. Quality (PSQ)

24 0.97 0.54 0.74

Perceived Inf. Quality (PIQ)

24 0.98 0.71

0.62 0.79

User Inf. Satisfaction (UIS)

6 0.91 0.73

0.65

0.62 0.80

Use

2 0.92 0.37

0.21

0.38

0.85 0.92

Individual impact

6 0.95 0.56

0.45

0.58

0.34

0.78 0.88

Flexibility 4 0.96 0.78

0.57

0.57

0.29

0.55

0.85 0.92

Integration 4 0.95 0.79

0.52

0.53

0.38

0.38

0.59

0.83 0.92

Response time

4 0.93 0.88

0.65

0.61

0.25

0.50

0.66

0.58

0.77 0.88

Recover-ability

4 0.95 0.90

0.59

0.65

0.30

0.52

0.61

0.70

0.76

0.82 0.90

Conven-ience

4 0.94 0.86

0.60

0.76

0.42

0.56

0.64

0.52

0.71

0.72

0.79 0.89

Command language

4 0.91 0.76

0.62

0.51

0.19

0.28

0.35

0.52

0.71

0.65

0.65

0.73 0.85

Complete-ness

4 0.97 0.48

0.83

0.47

0.15

0.29

0.47

0.33

0.47

0.37

0.37

0.40

0.89 0.94

Precision 4 0.98 0.67

0.86

0.59

0.11

0.38

0.63

0.51

0.58

0.56

0.51

0.56

0.72

0.91 0.95

Accuracy 4 0.97 0.64

0.89

0.56

0.11

0.38

0.48

0.45

0.59

0.56

0.55

0.59

0.62

0.71

0.89 0.94

Consistency 4 0.97 0.67

0.90

0.59

0.18

0.44

0.48

0.52

0.63

0.58

0.57

0.58

0.63

0.69

0.93

0.90 0.95

Currency 4 0.98 0.51

0.85

0.58

0.31

0.41

0.36

0.35

0.47

0.42

0.44

0.44

0.72

0.59

0.71

0.73

0.93 0.96

Format 4 0.95 0.68

0.71

0.54

0.21

0.41

0.55

0.54

0.58

0.57

0.56

0.56

0.51

0.65

0.51

0.51

0.51

0.83 0.91

a The upper bold number on the diagonal is the average variance extracted and the lower one its square root.

Table 2. Internal Consistencies, Average Variances Extracted and Inter-Correlations between Constructs, Part 2 of 2.


Hypothesis Model 1 Model 2

H1: Perceived system quality predicts user sa-tisfaction

Supported (p ≤ 0.001) Supported (p ≤ 0.001)

H2: Perceived information quality predicts user satisfaction

Supported (p ≤ 0.05) Supported (p ≤ 0.05)

H3: Perceived system quality predicts actual use Supported (p ≤ 0.05) Supported (p ≤ 0.01)

H4: Perceived information quality predicts actual use

Not supported Not supported

H5a: User satisfaction predicts actual use

H5b: Actual use predicts user satisfaction

Supported (p ≤ 0.10)

N/A

N/A

Supported (p ≤ 0.10)

H6: User satisfaction predicts individual impact Supported (p ≤ 0.001) Supported (p ≤ 0.001)

H7: Actual use predicts individual impact Not supported Not supported

Table 3. Summary of the Support Obtained for the Hypotheses

At the aggregated level of the 15 departments, user satisfaction, use and individual impact had corre-lations of 0.40, -0.36 and 0.41 with the overall measure of organizational impact (none of them significant at the level p ≤ 0.10). Even though not significant in this small population, the correlations are still notable, as user satisfaction, use and individual impact together explained 43.5% of the variance in organizational impact (p ≤ 0.10). Of the individual regression coefficients, the negative coefficient of actual use with organizational impact (ß = -0.38) was almost significant (p ≤ 0.10), suggesting that use may be a problematic predictor of organizational impact. One reason for this unexpected finding may be that one should not analyse the relationship between system use and organizational impact cross-sectionally across departments. It may be that if one could analyse variations in system use within depart-ments, use would have a positive association with organizational impact. This would require more longitudinal research.

From a more practical viewpoint, the power of perceived system quality and perceived information quality as predictors of user satisfaction suggests that they provide an effective diagnostic framework in which to analyse system features that may “cause” user satisfaction and dissatisfaction. The close association between user satisfaction and individual impact also suggests that user satisfaction may serve as a valid surrogate for individual impact.

Conclusions

The above paper reports on an empirical test of the IS success model of DeLone and McLean (1992) as a predictive model. As pointed out in the Introduction, there is a dearth of empirical tests of the model. Overall, the present findings supported the model, but, as implied in the discussion in section 3.2, there is much ambiguity related to the DeLone –McLean model as a causal-explanatory model. Much of this culminates in the ambiguity of the concept of user (information) satisfaction (e.g. Goodhue, 1986; Iivari, 1987; Melone, 1990; Iivari, 1997). It seems that little progress has been made on this front since the 1980’s. Strengthening of the underlying theory of the DeLone-McLean model would require some attention to be paid to this component. On the positive side, the findings suggest that user satisfaction may be a reasonably good surrogate for individual impact as long as it is confined to impact on work performance. It is an open question, however, whether it is a valid surrogate for organizational effectiveness, as suggested by Ives et al. (1983).

The present paper has its limitations. Since the results are based on a field study of one mandatory information system in one specific organizational context, the first question is whether the findings are specific to that system and its organizational context. A second question is whether they may be explained by the nature of the system (a mandatory operational level system based on an application package). As pointed out above, we suspect that the relatively insignificant role of actual use in the whole framework may be explained by this mandatory nature of the


system. This may also explain the fact that perceived system quality emerged as more significant than perceived information quality. Empirical testing of the DeLone-McLean model should therefore be extended to cover a wider variety of systems.

References Adams, D.A., Nelson, R.R. and Todd, P.A. (1992).

“Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication,” MIS Quarterly, Vol. 16, No. 2, pp. 227-247.

Amoroso, D.L. and Cheney, P.H. (1991). “Testing a Causal Model of End-User Application Effecti-veness,” Journal of Management Information Systems, Vol. 8, No. 1, pp. 63-89.

Bagozzi, R.P. (1985). “Expectancy-Value Attitude Models: An Analysis of Critical Theoretical Issues,” International Journal of Research Marketing, Vol. 2, pp. 43-60.

Bailey, J.E. and Pearson, S.W. (1983). “Developing a Tool for Measuring and Analyzing Computer User Satisfaction,” Management Science, Vol. 29, No. 5, pp. 530-545.

Barki, H. and Hartwick, J. (1989). “Rethinking the Concept of User Involvement,” MIS Quarterly, Vol. 13, No. 1, pp. 53-63.

Barki, H. and Huff, S.L. (1985). “Change, Attitude to Change, and Decision Support System Success,” Information & Management, Vol. 9, No. 5, pp. 261-268.

Baroudi, J.J. and Orlikowski, W.J. (1988). “A Short-Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes on Use,” Journal of Management Information Systems, Vol. 4, No. 4, pp. 44-59.

Baroudi, J.J., Olson, M.H. and Ives, B. (1986). “An Empirical Study of the Impact of User Involvement on System Usage and Information Satisfaction,” Communications of the ACM, Vol. 29, No. 3, pp. 232-238.

Cats-Baril, W.L. and Huber, G.P. (1987). “Decision Support Systems for Ill-Structured Problems: An Empirical Study,” Decision Sciences, Vol. 18, No. 3, pp. 350-372.

Chau, P.Y.K. (1996). “An Empirical Assessment of a Modified Technology Acceptance Model,” Journal of Management Information Systems, Vol. 13, No. 2, pp. 185-204.

Chin, J.P., Diehl, V.A. and Norman, K.L. (1988). “Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface,” in Soloway, E., Frye, D. and Sheppard, S.B. (eds.), CHI’88 Conference Proceedings: Human Factors in Computing Systems, NY: Association for Computing Machinery, pp. 213-218.

Chin, W.W. (1998). “The Partial Least Squares Approach to Structural Equation Modelling,” in Marcoulides, G.A. (ed.), Modern Methods for Business Research, Mahwah, NJ: Lawrence Erlbaum Associated, pp. 295-336.

Chin, W.W. and Gopal, A. (1995). “Adoption Intention in GSS: Relative Importance of Beliefs,” The Data Base for Advances in Information Systems, Vol. 26, No. 2&3, pp. 42-63.

Chin, W.W. and Lee. M.K.O. (2000). “On the Formation of End-User Computing Satisfaction: A Proposed Model and Measurement Instrument,” in Orlikowski, W., Ang, S., Weill, P., Krcmar, H.C. and DeGross, J.I. (eds.), Proceedings of the Twenty-First International Conference on Information Systems, Brisbane, Australia, pp. 553-563.

Chin, W.W. and Marcolin, B.L., (2001). “The Future of Diffusion Research,” The Data Base for Advances in Information Systems, Vol. 32, No. 3, pp. 8-12.

Chin, W.W. and Newsted, P.R. (1999). “Structural Equation Modeling Analysis with Small Samples Using Partial Least Squares,” in Rick Hoyle (Ed.), Statistical Strategies for Small Sample Re-search, Thousand Oaks, CA: Sage Publications, pp. 307-341.

Davis, F.D. (1989). “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly, Vol. 13, No. 3, pp. 319-339.

Davis, F.D., Bagozzi, R.P. and Warshaw, P.R. (1989). “User Acceptance of Computer Technology: A Comparison of Two Theoretical Models,” Management Science, Vol. 35, No. 8, pp. 982-1003.

Davis, F.D., Bagozzi, R.P. and Warshaw, P.R. (1992). “Extrinsic and Intrinsic Motivation to Use Computers in the Workplace,” Journal of Applied Social Psychology, Vol. 22, pp. 1111-1132.

DeLone, W.H. and McLean, E.R. (1992). “Information Systems Success: The Quest for the Dependent Variable,” Information Systems Research, Vol. 3, No. 1, pp. 60-95.

DeLone, W.H. and McLean, E.R. (2003). “The DeLone and McLean Model of Information Systems Success: A ten-Year Update,” Journal of Management Information Systems, Vol. 19, No. 4, pp. 9-30.

Doll, W.J. and Torkzadeh, G. (1988). “The Measurement of End-User Computing Satisfaction,” MIS Quarterly, Vol 12, No. 2, pp. 259-274.

Etezadi-Amoli, J. and Farhoomand, A.F. (1996). “A Structural Model of End-User Computing


Satisfaction and User Performance,” Information & Management, Vol. 30, No. 2, pp. 65-73.

Fishbein, M. and Ajzen, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research, Reading, MA: Addison-Wesley.

Fornell, C. and Larcker, D.F. (1981). “Evaluating Structural Equation Models with Unobservable Variables and Measurement Error,” Journal of marketing Research, Vol. 18, pp. 39-50. (reprinted in Fornell, C. (ed.), A Second Generation of Multivariate Analysis, Volume 2, Measurement and Evaluation, New York, NY: Praeger Publishers, pp. 289-316.)

Gatian, A.W. (1994). “Is User Satisfaction a Valid Measure of System Effectiveness,” Information & Management, Vol. 26, No. 3, pp. 119-131.

Gefen, D. and Keil, M. (1998). “The Impact of Developer Responsiveness on Perceptions of Usefulness and Ease of Use: An Extension of the Technology Acceptance Model,” The DATA BASE for Advances in Information Systems, Vol. 29, No. 2, pp. 35-49.

Gefen, D. and Straub, D.W. (1997). “Gender Differences in the Perception and Use of E-Mail: An Extension to the Technology Acceptance Model,” MIS Quarterly, Vol. 21, No. 4, pp. 389-400.

Gefen, D. and Straub, D.W. (2000). “The Relative Importance of Perceived Ease of Use in IS Adoption: A Study of E-Commerce Adoption,” Journal of the Association for Information Systems, Vol. 1, Article 8, pp. 1-28.

Gefen, D., Straub, D.W. and Boudreau, M.-C. (2000). “Structural Equation Modelling and Regression: Guidelines for Research Practice,” Communications of the Association for Information Systems, Vol. 1, Article 7, pp. 1-76.

Ginzberg, M.J. (1981). “Early Diagnosis of Implementation Failure: Promising Results and Unanswered Questions,” Management Science, Vol. 27, No. 4, pp. 459-478.

Goodhue, D. (1986). “IS Attitudes: Toward Theoretical and Definition Clarity,” in Maggi, L., Zmud, R. and Wetherbe, J. (eds.), Procedings of the Seventh International Conference on Information Systems, San Diego, CA, pp. 181-194.

Gustafsson, M.R., Karlsson, T. and Bubenko, J.A. Jr. (1982). “A Declarative Approach to Conceptual Information Modeling,” in Olle, T.W., Sol, H.G. and Verrijn-Stuart, A.A. (eds.), Information Systems Design Methodologies: A Comparative Review, North-Holland, Amstredanm, pp. 93-142.

Hair, J.F. Jr., Anderson, R.E., Tatham, R.L. and Black, W.C. (1992). Multivariate Data Analysis with Readings, New York, NY: Macmillan.

Igabria, M. (1990). “End-User Computing Effectiveness: A Structural Equation Model,” Omega, Vol. 18, No. 6, pp. 637-652.

Igbaria, M., Guimaraes, T. and Davis, G.B. (1995). “Testing the Determinants of Microcomputer Usage via a Structural Equation Model,” Journal of Management Information Systems, Vol. 11, No. 4, pp. 87-114.

Igbaria, M. and Iivari, J. (1995). “The Effects of Self-Efficacy on Computer Usage,” Omega, Vol. 23, No. 6, pp. 587-605.

Igbaria, M., Parasuraman, S. and Baroudi, J.J. (1996). “A Motivational Model of Microcomputer Usage,” Journal of Management Information Systems, Vol. 13, No. 1, pp. 127-143.

Igbaria, M., Zinatelli, N., Cragg, P. and Cavaye, A.L.M. (1997). “Personal Computing Acceptance Factors in Small Firms: A Structural Equation Model,” MIS Quarterly, Vol. 21, No. 3, pp. 279-305.

Igbaria, M. and Zviran, M. (1991). “End-User Effectiveness: A Cross-Cultural Examination,” Omega, Vol. 19, No. 5, pp. 369-379.

Iivari, J. (1987). “User Information Satisfaction (UIS) Reconsidered: An Information System as the Antecedent of UIS.” in DeGross, J.I. and Kriebel, C.H. (eds.), Proceedings of the Eighth Interna-tional Conference on Information Systems, Pittsburgh PA, pp. 57-73.

Iivari, J. (1996). “Why Are CASE Tools Not Used?,” Communications of the ACM, Vol. 39, No. 19, pp. 94-103.

Iivari, J. (1997). “User Information Satisfaction: A Critical Review,” Encyclopedia of Library and Information Science, Vol. 60, Supplement 23, pp. 341-364.

Iivari, J. and Ervasti, I. (1994). “User Information Satisfaction: IS Implementability and Effectiveness,” Information & Management, Vol. 27, No. 4, pp. 205-220.

Iivari, J. and Koskela, E. (1987). “The PIOCO Model for Information System Design,” MIS Quarterly, Vol. 11, No. 3, pp. 401-419.

Ives, B., Olson, M.H. and Baroudi, I.J. (1983). “The Measurement of User Information Satisfaction,” Communications of the ACM, Vol. 26, No. 10, pp. 785-793.

Jenkins, A.M. (1985). “Research Methodologies and MIS Research,” in Mumford E. et al. (eds.), Research Methods in Information Systems, North-Holland, Amsterdam, pp. 103-117.

Jenkins, A.M. and Ricketts, J.A. (1979). Development of an Instrument to Measure User Information Satisfaction with Management


Information Systems, Working paper, Bloom-ington, Indiana: Indiana University.

Karahanna, E. and Straub, D.W. (1999). The Psychological Origins of Perceived Usefulness and Ease-of-Use,” Information & Management, Vol. 35, No. 4, pp. 237-250.

Karahanna, E., Straub, D.W. and Chervany, N.L. (1999). “Information Technology Adoption Across Time: A Cross-Sectional Comparison of Pre-Adoption and Post-Adoption Beliefs,” MIS Quarterly, Vol. 23, No. 2, pp. 183-213.

Lee, Y., Kozar, K.A. and Larsen K.R.T. (2003) “The technology acceptance model: past, present, and future,” Communications of the Association for Information Systems, Vol. 12, No. 50, pp. 752-780

Leidner, D.E. (1998). “Mexican Executives’ Use of Information Systems: An Empirical Investigation of EIS Use and Impact,” Journal of Global Information Technology Management, Vol. 1, No. 2, pp. 19-36.

Ma, Q. and Liu, L. (2004). “The Technology Acceptance Model: A Meta-Analysis of Empirical Findings,” Journal of Organizational and End-User Computing, Vol. 16, No. 1, pp. 59-72

Mathieson, K. (1991). “Predicting User Intentions: Comparing the Technology Acceptance Model with the Theory of Planned Behavior,” Information Systems Research, Vol. 2, No. 3, pp. 173-191.

McGuire, W.J. (1969). “The Nature of Attitudes and Attitude Change,” in Lindzey, G. and Aronson, E. (eds.), The Handbook of Social Psychology, Volume 3: The individual in social context, 2nd ed., Reading, MA: Addison-Wesley, pp. 136-314.

Melone, N.P. (1990). “A Theoretical Assessment of the User-Satisfaction Construct in Information Systems Research,” Management Science, Vol. 36, No. 1, pp. 76-91.

Mohr, L.B. (1982). Explaining Organizational Behavior, the Limits and Possibilities of Theory and Research, San Fransisco, CA: Jossey-Bass Publishers.

Nelson, R.R. and Cheney, P.H. (1987). “Training End Users: An Exploratory Study,” MIS Quarterly, Vol. 11, No. 4, pp. 547-559.

Rai, A., Lang, S.S. and Welker, R.B. (2002). “Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis,” Information Systems Research, Vol. 13, No. 1, pp. 50-69.

Sanders, G.L. (1984). “MIS/DSS Success Measure,” Systems, Objectives, Solutions, Vol. 4, pp. 29-34.

Seddon, P.B. (1997). “A Respecification and Extension of the DeLone and McLean Model of IS Success,” Information Systems Research, Vol. 8, No. 3, pp. 240-253.

Seddon, P.B. and Kiew, M.-Y. (1994). “A Partial Test and Development of the DeLone and McLean Model of IS Success,” in DeGross, J.I., Huff, S.L. and Munro, M.C. (eds.), Proceedings of the Fifteenth International Conference on Information Systems, Vancouver, Canada, pp. 99-110.

Seddon, P.B., Staples, S., Patnayakuni, R. and Bowtell, M. (1999). “Dimensions of Information Systems Success,” Communications of the Association of Information Systems, Vol. 2, Article. 20, pp. 1-39.

Snitkin, S.R. and King, W.R. (1986). “Determinants of the Effectiveness of Personal Decision Support Systems,” Information & Management, Vol. 10, No. 2, pp. 83-89.

Srinivasan, A. (1985). “Alternative Measures of System Effectiveness: Associations and Implications,” MIS Quarterly, Vol. 9, No. 3, pp. 243-253.

Straub, D.W. (1989). “Validating Instruments in MIS Research,” MIS Quarterly, Vol. 13, No. 2, pp. 147-169.

Straub, D.W., Keil, M. and Brenner, W. (1997). “Testing the Technology Acceptance Model Across Cultures: A Three Country Study,” Information & Management, Vol. 33, No. 1, pp. 1-11.

Straub, D., Limayem, M. and Karahanna-Evaristo, E. (1995). “Measuring System Usage: Implications for Theory Testing,” Management Science, Vol. 41, No. 8, pp. 1328-1342.

Szajna, B. (1996). Empirical Evaluation of the Revised Technology Acceptance Model,” Management Science, Vol. 42, No. 1, pp. 85-92.

Torkzadeh, G. and Dwyer, D.J. (1994). “A Path Analytical Study of Determinants of Information System Usage,” Omega, Vol. 22, No. 4, pp. 339-348.

Taylor, S. and Todd, P.A. (1995). “Understanding Information Technology Usage: A Test of Competing Models,” Information Systems Research, Vol. 6, No. 2, pp. 144-176.

Van de Ven, A.H and Ferry, D.L. (1980). Measuring and Assessing Organizations, Chichester: John Wiley & Sons.

Venkatesh, V. and Davis, F.D. (2000). “A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies,” Management Science, Vol. 46, No. 2, pp. 186-204.


About the Author Juhani Iivari is a Professor in Information Systems at the University of Oulu, Finland, and the Scientific Head of the INFWEST.IT Postgraduate Education Program of five Finnish Universities in the area of information systems. He received his M.Sc. and Ph.D. degrees from the University of Oulu. Iivari serves in editorial boards of seven journals. His research has broadly focused on theoretical

foundations, development methodologies and approaches, organizational analysis, implementation and acceptance, and the quality of information systems. Iivari has published in journals such as AJIS, BIT, Communications of the ACM, Data Base, European JIS, Information & Management, Informa-tion and Software Technology, Information Systems, ISJ, ISR, JMIS, JOCEC,MISQ, Omega, SJIS and others. Email: [email protected]


Appendix A: Measures

System quality

Please assess the flexibility of the system to change in response to new demands

Rigid __ __ __ __ __ __ __ Flexible

Limited __ __ __ __ __ __ __ Versatile

Insufficient__ __ __ __ __ __ __ Sufficient

Low __ __ __ __ __ __ __ High

Please assess the ability of the system to communicate with other information systems

Incomplete __ __ __ __ __ __ __ Complete


Unsuccessful__ __ __ __ __ __ __ Successful

Bad __ __ __ __ __ __ __ Good

Please assess the response and turnaround time of the system

Slow __ __ __ __ __ __ __ Fast

Bad __ __ __ __ __ __ __ Good

Inconsistent__ __ __ __ __ __ __ Consistent

Unreasonable__ __ __ __ __ __ __ Reasonable

Please assess the ability of the system to recover from errors

Slow __ __ __ __ __ __ __ Fast

Inferior __ __ __ __ __ __ __ Superior

Incomplete__ __ __ __ __ __ __ Complete

Complex __ __ __ __ __ __ __ Simple

Please assess the convenience of use of the system

Inconvenient __ __ __ __ __ __ __ Convenient

Bad __ __ __ __ __ __ __ Good

Difficult __ __ __ __ __ __ __ Easy

Inefficient__ __ __ __ __ __ __ Efficient

Please assess the commands used to interact with the system

Complex __ __ __ __ __ __ __ Simple

Weak __ __ __ __ __ __ __ Powerful

Difficult __ __ __ __ __ __ __ Easy

Hard-to-use__ __ __ __ __ __ __ Easy-to-use

Information quality

Please assess the volume of output information (re-ports and queries)

Concise __ __ __ __ __ __ __ Excessive


Unnecessary__ __ __ __ __ __ __ Necessary

Unreasonable__ __ __ __ __ __ __ Reasonable

Please assess the completeness of the output information

Incomplete__ __ __ __ __ __ __ Complete



Inadequate__ __ __ __ __ __ __ Adequate

Please assess the precision of the output information

Insufficient __ __ __ __ __ __ __ Sufficient


Low __ __ __ __ __ __ __ High

Uncertain__ __ __ __ __ __ __ Certain

Please assess the accuracy of the output information

Inaccurate __ __ __ __ __ __ __ Accurate

Low __ __ __ __ __ __ __ High



Please assess the consistency of the output information


Low __ __ __ __ __ __ __ High

Inferior __ __ __ __ __ __ __ Superior



Please assess the currency of the output information

Bad __ __ __ __ __ __ __ Good

Untimely__ __ __ __ __ __ __ Timely

Inadequate__ __ __ __ __ __ __ Adequate

Unreasonable __ __ __ __ __ __ __ Reasonable

Please assess the format of the output

Bad __ __ __ __ __ __ __ Good

Complex__ __ __ __ __ __ __ Simple

Readable__ __ __ __ __ __ __ Readable

Useless __ __ __ __ __ __ __ Useful

User satisfaction

Please assess the system

Terrible __ __ __ __ __ __ __ Wonderful

Difficult __ __ __ __ __ __ __ Easy

Frustrating __ __ __ __ __ __ __ Satisfying

Inadequate __ __ __ __ __ __ __ Adequate

Dull __ __ __ __ __ __ __ Stimulating

Rigid __ __ __ __ __ __ __ Flexible

Actual use

Daily use: How much time do you spend with the system during an ordinary day when you use comput-ers?

Scarcely at all 1

Less than 1/2 hours 2

1/2- 1 hours 3

1-2 hours 4

2-3 hours 5

More than 3 hours 6

Frequency of use: How often on average do you use the system?

Less than once a month 1

Once a month 2

A few times a month 3

A few times a week 4

Once a day 5

Several times a day 6

Individual impact

Using the system in my job enables me to accomplish tasks more quickly.

Fully disagree__ __ __ __ __ __ __ Fully agree

Using the system improves my job performance.


Using the system in my job increases my productivity.


Using the system enhances my effectiveness in my job


Using the system makes it easier to do my job


I find the system useful in my job



Appendix B: Weights and loadings on the indicators

Indicator Model 1 Model 2 Weight Loading Weight Loading Flexibility - item 1 0.2651 0.8913 0.2651 0.8913- item 2 0.2468 0.9108 0.2468 0.9108- item 3 0.2839 0.9473 0.2839 0.9473- item 4 0.3042 0.9276 0.3042 0.9276System integration - item 1 0.2666 0.9208 0.2666 0.9208- item 2 0.2730 0.8956 0.2730 0.8956- item 3 0.2778 0.8851 0.2778 0.8851- item 4 0.2924 0.9356 0.2924 0.9356Response time - item 1 0.2605 0.8308 0.2605 0.8308- item 2 0.2716 0.8588 0.2716 0.8588- item 3 0.3134 0.8887 0.3134 0.8887- item 4 0.3118 0.9194 0.3118 0.9194Recoverability - item 1 0.2761 0.8757 0.2761 0.8757- item 2 0.2800 0.9366 0.2800 0.9366- item 3 0.2809 0.9236 0.2809 0.9236- item 4 0.3858 0.8815 0.3858 0.8815Convenience - item 1 0.2992 0.9352 0.2992 0.9352- item 2 0.3173 0.9489 0.3173 0.9489- item 3 0.2782 0.9012 0.2782 0.9012- item 4 0.2306 0.7590 0.2306 0.7590Command language - item 1 0.1303 0.6100 0.1303 0.6100- item 2 0.3190 0.9027 0.3190 0.9027- item 3 0.3612 0.9509 0.3612 0.9509- item 4 0.3478 0.9123 0.3478 0.9123Completeness - item 1 0.2556 0.9408 0.2556 0.9408- item 2 0.2652 0.9310 0.2652 0.9310- item 3 0.2749 0.9589 0.2749 0.9589- item 4 0.2671 0.9436 0.2671 0.9436Precision - item 1 0.2465 0.9385 0.2465 0.9385- item 2 0.2687 0.9623 0.2687 0.9623- item 3 0.2689 0.9596 0.2689 0.9596- item 4 0.2716 0.9522 0.2716 0.9522Accuracy - item 1 0.2662 0.9596 0.2662 0.9596- item 2 0.2595 0.9533 0.2595 0.9533


- item 3 0.2729 0.9300 0.2729 0.9300- item 4 0.2710 0.9231 0.2710 0.9231Consistency - item 1 0.2601 0.9277 0.2601 0.9277- item 2 0.2621 0.9630 0.2621 0.9630- item 3 0.2620 0.9505 0.2620 0.9505- item 4 0.2710 0.9585 0.2710 0.9585Currency - item 1 0.2606 0.9590 0.2606 0.9590- item 2 0.2567 0.9675 0.2567 0.9675- item 3 0.2549 0.9567 0.2549 0.9567- item 4 0.2692 0.9644 0.2692 0.9644Format - item 1 0.2726 0.9236 0.2726 0.9236- item 2 0.2838 0.9361 0.2838 0.9361- item 3 0.3020 0.9648 0.3020 0.9648- item 4 0.2510 0.8004 0.2510 0.8004Perceived system quality

- item 1 0.0586 0.6701 0.0587 0.6702- item 2 0.0487 0.6238 0.0487 0.6239- item 3 0.0551 0.7177 0.0552 0.7177- item 4 0.0608 0.7688 0.0608 0.7689- item 5 0.0530 0.6915 0.0530 0.6916- item 6 0.0564 0.7082 0.0565 0.7083- item 7 0.0586 0.7206 0.0586 0.7207- item 8 0.0619 0.7586 0.0619 0.7586- item 9 0.0485 0.6846 0.0485 0.6846- item 10 0.0536 0.7138 0.0537 0.7138- item 11 0.0660 0.8238 0.0660 0.8237- item 12 0.0668 0.8196 0.0668 0.8195- item 13 0.0590 0.7930 0.0589 0.7930- item 14 0.0635 0.8042 0.0635 0.8042- item 15 0.0642 0.8066 0.0642 0.8067- item 16 0.0698 0.8208 0.0697 0.8207- item 17 0.0699 0.8041 0.0698 0.8040- item 18 0.0720 0.8525 0.0719 0.8524- item 19 0.0672 0.7475 0.0671 0.7475- item 20 0.0559 0.6197 0.0560 0.6197- item 21 0.0220 0.2841 0.0220 0.2841- item 22 0.0481 0.6957 0.0480 0.6956- item 23 0.0604 0.7877 0.0603 0.7876- item 24 0.0595 0.7584 0.0595 0.7583Perceived information quality - item 1 0.0502 0.7418 0.0502 0.7419- item 2 0.0489 0.7697 0.0490 0.7698- item 3 0.0502 0.7977 0.0502 0.7978- item 4 0.0513 0.7752 0.0514 0.7753


- item 5 0.0502 0.7548 0.0502 0.7548- item 6 0.0563 0.8226 0.0563 0.8226- item 7 0.0561 0.8234 0.0561 0.8233- item 8 0.0558 0.8317 0.0558 0.8316- item 9 0.0543 0.8303 0.0542 0.8302- item 10 0.0527 0.8096 0.0527 0.8095- item 11 0.0584 0.8514 0.0584 0.8513- item 12 0.0540 0.8453 0.0540 0.8453- item 13 0.0559 0.8346 0.0559 0.8345- item 14 0.0577 0.8408 0.0577 0.8407- item 15 0.0578 0.8406 0.0578 0.8405- item 16 0.0581 0.8694 0.0581 0.8693- item 17 0.0565 0.8142 0.0566 0.8143- item 18 0.0560 0.8021 0.0560 0.8022- item 19 0.0560 0.7964 0.0561 0.7965- item 20 0.0584 0.8410 0.0584 0.8410- item 21 0.0450 0.6233 0.0450 0.6232- item 22 0.0467 0.6488 0.0467 0.6488- item 23 0.0491 0.6905 0.0491 0.6905- item 24 0.0408 0.5739 0.0408 0.5739User information satisfaction - item 1 0.2162 0.7257 0.2099 0.7219- item 2 0.1982 0.7409 0.1937 0.7384- item 3 0.2346 0.8521 0.2386 0.8542- item 4 0.2226 0.8100 0.2294 0.8131- item 5 0.2075 0.8287 0.2084 0.8296- item 6 0.1943 0.7720 0.1921 0.7720Use - item 1 0.5897 0.9349 0.6080 0.9400- item 2 0.4951 0.9063 0.4761 0.9001Individual impact - item 1 0.1362 0.8517 0.1358 0.8515- item 2 0.1814 0.9088 0.1814 0.9088- item 3 0.1968 0.8976 0.1967 0.8975- item 4 0.2161 0.9213 0.2162 0.9213- item 5 0.2098 0.9169 0.2100 0.9170- item 6 0.1901 0.7948 0.1903 0.7949


Documents

An empirical test of the DeLone-McLean model of information system success