View
212
Download
0
Embed Size (px)
Citation preview
Measuring IS Success
Measuring IS Success: Quest for the Dependent Variable in IS
ResearchThe Journey
Special IT Research Seminar
William H. DeLoneKogod School of Business
February 8, 2008
IT Research SeminarFebruary 10, 2003
Measuring IS Success
IS Success Research Stream
• UCLA Dissertation – Successful use of IS by SMEs; MISQ 1988
• IS Success: Quest for the Dependent Variable; ISR 1992 = “DeLone & McLean Success Model”
• Ten Year Update; JMIS, Spring 2003• Measuring E-Commerce Success, International
Journal of Electronic Commerce, Fall 2004• IS Success Models, Dimensions, Measures, and
Interrelationships, under review European Journal of Information Systems
Measuring IS Success
Research Motivation
• UCLA measurement course (Mason & Swanson) • Peter Keen’s 5 research challenges for MIS (ICIS
1) – What is the dependent variable? How does MIS establish a cumulative tradition?
• Dependent variable for PhD study in SMEs; Use & Impact
• If you can’t measure it, you can’t research it
Measuring IS Success
Quest for MIS Dependent Variable (ISR, 1992) – “IS Success”Purposes• Organize & summarize MIS research related to
defining the dependent variable• Measure progress on defining the dependent
variable• Improve IS research practice • Contribute to “Cumulative Tradition” –
compare apples to apples
Measuring IS Success
Theoretical Underpinnings
Mason’s 1978 article on measuring information outputProduction->Product->Receipt->Influence on Recipient->Influence on System Mason’s work was based on Shannon & Weaver’s 1949 Theory of
Communications book – Levels of communications measurement = technical, semantic, effectiveness
DeLone & McLean Success Categories System Quality->Information Quality-> Use->Satisfaction->Individual Impact-> Organization Impact
Measuring IS Success
Methodolgy
• Literature review
• IS Articles from 1981 to 1988
• Framework/model for organizing success measures
• Empirical measures grouped into six success categories
Measuring IS Success
D & M IS Success Model
UserSatisfaction
Use
IndividualImpact
Organizational
Impact
SystemQuality
Information
Quality
Figure 1 D&M IS Success Model
Measuring IS Success
Results/Conclusions
• A simple and parsimonious framework for organizing IS success measures
• IS Success - multi-dimensional and interdependent construct
• Selection of measures is contingent on objectives and context of study
• Reduce IS Success measures; build on existing measures=> “cumulative tradition” – comparison of results
• Need for organizational impact measures
Measuring IS Success
The Challenge
“ This success model clearly needs further development and validation before it could serve as a basis for the selection of appropriate IS measures.” (p.88)
Measuring IS Success
Ten-Year Update (JMIS 2003)
Purposes• Model Utility• Validate the Model – Causal relationships• Update the Model to recognize the changes in
IS• Assess progress in IS success measurement
Measuring IS Success
Utility of D&M Success Model
• Cited by more than 285 refereed journal and proceedings articles between 1993 and 2002 (according to a recent study in CAIS vol. 20, ISR 1992 article is the most cited article in MIS over the last 15 years; > 400 citations)
• Most articles used the Model “as a drunkard uses a lamppost for support rather than illumination.” Statement was rejected by editor.
Measuring IS Success
Model Validation
• Seddon & Kiew (1994) validated 4 of the proposed associations
• Rai et. al. validated overall model using goodness of fit tests (ISR 2002)
• Fifteen additional empirical studies validated one or more proposed associations
Measuring IS Success
Updated Model
• IS move from production function to production & service function => importance of service quality
• Impacts on whom? Individuals, groups, org., industry, economy => Net Benefits dimension with contextual definition
Measuring IS Success
INFORMATION QUALITY
SYSTEM QUALITY
SERVICE QUALITY
INTENTION TO USE
USE
NET BENEFITS
USER SATISFACTION
Updated Model
Measuring IS Success
Assessment & Conclusions
• D&M IS Success Model supported and validated• More careful attention to multidimensionality of IS
Success• Confusion between Independent & Dependent variables• Operationalization of the model is contextual (Seddon
et. al.)• Progress in parsimonious measure development is slow• System Use is misunderstood and undervalued• Use and satisfaction are not a substitute for Net Benefits
measures (Yuthas & Young, 1998)
Measuring IS Success
Recent Advances in Success Measurement
Measuring IS Success
Application of Model: E-Commerce Success (IJEC 2004)
• Premise: E-commerce does not need a new measurement paradigm but some new measures
• Apply the D&M IS Success Model for measuring E-Commerce Success
• Literature Review and classification of emerging e-commerce effectiveness measures
• Case examples of application of IS Success Model to E-Commerce success measurement
Measuring IS Success
Application of Model: ERP Success
Article by Sedera & Gable (ICIS 2004) – Government & University ERPs
Most comprehensive empirical test of modelFour dimensions of IS Success –
System quality Information quality Individual impact Organizational impact
** 27 Item measures
Measuring IS Success
Validated Measures for IS Success Source: Sedera and Gable (2004)
System Quality - ease of use, ease of learning, user requirements, system features, system accuracy, flexibility, sophistication, integration, and customization
Information Quality – availability, usability, understandability, relevance, format, and conciseness
Individual Impact – learning, awareness/recall, decision effectiveness, and individual productivity
Organizational Impact - organizational costs, staff requirements, cost reduction, overall productivity, improved outcomes/outputs, increased capacity, e-Government, and business process change
What happened to Use and User Satisfaction?
Measuring IS Success
Measuring Systems/Information Usage (Don Marchand)
• Don Marchand, Professor of Strategy & Information Management at IMD – Switzerland
• 20% of value realization is in deployment: 80% of value realization is in information and IT USAGE
• Challenge – Information Usage is difficult to see, measure and manage
Measuring IS Success
Measuring System Usage (Burton-Jones & Straub, ISR 2006)
• Problem – over-simplified measures of use; e.g. duration of use and breadth of use
• Importance of context & purpose • Elements of usage include: systems, user and task• Proposed Dimensions of Systems Usage –
Cognitive Absorption (engagement) + Deep Structure (use of system features that support a specific task)
• Systems Usage empirically related to task performance
Measuring IS Success
Current Research
Measuring IS Success: Models, Dimensions, Measures and Interrelationships under review at European Journal of Information Systems
• Assessing the state of IS Success Measurement (via D & M Model)
• Summarizing empirical literature, 1992 to 2006• Contributions
Summarizes measures used for each dimension of success
Validates significant relationships for 10 of the 15 causal relationships in the Updated Success Model based on empirical studies
Measuring IS Success
Conclusions
• DeLone & McLean IS Success Model remains the most popular, comprehensive framework for guiding the development of the dependent variable in IS research and for comparing results
• More IS researchers are using the D&M Model to inform and guide their measurement of the dependent variable rather than to merely justify their choice of measures
• Information quality and information use are under studied
• Satisfaction is over used as a surrogate for success; therefore much information is lost
• Bias toward ease of data collection threatens rigor and understanding