Embed Size (px)
Citation preview
Developing new model for website assessment
Applying evaluation theory approach for
assessment of Slovenian government websites
Boris KrageljUniversity of Ljubljana
Faculty of social [email protected]
The “idea” of evaluation study approach
Evaluation is...... systematic use of procedures, methods and research techniques for assessment of ideas, action plans and results of “public” programs (Rossi and Freeman 1993, 5)
...that provides answers to the following
questions:- How well is program (of government websites) conducted?- Does program (of government websites) functions in the way it
supposed to?- How well program (of government websites) achieves desired goals?- Why desired goals (of gov. websites) are not achieved and how it is
possible to achieve them? - What kind of changes did program (of gov. websites) reach and what
are its effects?
We applied this idea of the evaluation to the program of goals of Slovenian government websites.
Construction of evaluation study
A three step model:
Phase one:PROGRAM
- Clear identification of websites’ goals, mission and strategies for achieving them.- Determination of program of Slovenian government websites
Phase two:ANALYSIS
- Complete analysis of websites and measurement of its performance in all aspects related to its goals and strategy: accessibility, usability, utility…- Holistic analysis of Slovenian government web sites of all relevant aspects of Slovenia government activities on the WWW
Phase three:ASSESSMENT
- Comparson of the results of analysis with the websites’ program through the seven criteria of evaluation theory: effectiveness, efficiency, adequacy, equity, responsiveness, appropriateness and economy.- Comparison of the condition of Slovenian government websites with their program and strategic goals. Making judgments about how well the program achieved desired goals.
Phase one: PROGRAM
On the highest level we determined the program of Slovenian government websites in the following way:
- Presentation of the government bodies, offices and providing transparency of their activities (information service, contacts, government documents,... Online)
- Providing government services online (tax declaration, car registration, change of address...)
- Providing possibilities for citizen’ participation in the process of government activities (through the e-mail, forums and web polls...)
- Making communication and interaction with government bodies and offices simple and fficient
- Generally accessible government for all citizens with no social and technological discrimination
- High care for security and privacy when using information and services on government websites
Phase two: ANALYSIS
• Matrix of different aspects of websites, and methods, research techniques that were used for analysis of Slovenian government
websites
• GENERAL RULE: The more aspects included in analysis and the more different methods applied for the analysis – the better assessment in the final stage of evaluation study is possible!
Web
survey Usability
testing Expert review
HTML validation
Log analysis
Guideline review, Feature
inspection
Mystery shopping
Performance test
Content x
Usability x x
Traffic x x
Accessibility x
Performance x
Responsibility x
Customer satisfaction
x
Security& privacy
x
Phase two: ANALYSIS
Combining methods is an important advantage
COMBINIG METHODS
INDIVIDUAL METHOD
DIFFERENT METHODS APPLIED, DIFFERENT ASPECTS ANALYSED
QUALITY OF ASSESSMENT
Phase three: ASSESSMENT
• General evaluation criteria for the assessment of programs and their application on the program of Slovenian government
websitesCriteria Description Application to SLO government websites
Effectiveness Was desired effect achieved? Was desired effect of SLO government web sites achieved?
Efficiency How much effort was needed for effect achieved?
How much effort was needed for the effect of SLO government websites achieved?
Adequacy To which extent was (with achieved effect) problem solved?
To which extent was (with achieved effect) problem of SLO government websites solved
Equity Are expenses and benefits equally distributed among different groups?
Are expenses and benefits for SLO government websites equally distributed among different groups?
Responsiveness Are political effects of program in accordance with needs, preferences and values of parties involved?
Are political effects of SLO government websites in accordance with needs, preferences and values of parties involved?
Appropriateness Are desired effects really important? Are desired effects of SLO government websites really important?
Economy Were resources spent in the right and appropriate way?
Were resources for SLO government websites spent in the right and appropriate way?
RESULTS of evaluation study
Major results:
• Complete and relevant assessment of websites on the basis of their mission, achievement of certain strategic goals
• Identification of major problems (with the websites) for realization of the program of websites
• Directions and orientation for future developments of websites in the way of reaching the strategic goals
Additional value:
• Point of departure for revision of the program and strategy of websites
• Ground for measuring the progress toward the realization of the website program and its success in reaching the strategic goals
• Practical guidelines for improvement of websites• Establishing best practices • Providing ideas for reasonable changes in further developments
RESULTS of evaluation study in the case of Slovenian government websites
• Slovenian government websites are not meeting any of the evaluation criteria to the greater extent and are not Effective, Efficient, Adequate, Equal, Responsive, Appropriate, neither Economic
• The web sites (with regard to their program) therefore are not achieving their mission as they are:
– not generally accessible to all citizens, – Unfriendly and very difficult to use– lacking of the right content, information and services,– they take very little care for privacy and security matters– and they are without appropriate, common appearance
• The major problem presents very poorly defined program of Slovenian government websites, what finally results in desynchronized functioning of government websites
• As an additional value of evaluation approach some important solutions for renovation of Slovenian government websites were provided:
– A document with guidelines for essential reformation and improvement of Slovenian government websites was prepared
– Draft of the strategy for development of Slovenian government on the WWW was launched
• Groundwork for measuring the progress toward the mission
Conclusion
One final question:
Is evaluation study approach for website assessment applicable to all different kinds of
websites, or can it be used only for assessment of government websites?
This dilemma will remain unsolved until the next case of assessing websites with evaluation study approach from
another area will be tested!
Developing new model for website assessment
Applying evaluation theory approach for
assessment of Slovenian government websites
Boris KrageljUniversity of Ljubljana,
Faculty of social sciences
