Download pdf - Assessment-based learning systems—learning from best projects

SOFTWARE PROCESS IMPROVEMENT AND PRACTICESoftw. Process Improve. Pract. 2007; 12: 569–577

Published online 3 September 2007 in Wiley InterScience(www.interscience.wiley.com) DOI: 10.1002/spip.347

Assessment-based LearningSystems – Learning fromBest Projects

Practice SectionRichard Messnarz* and Damjan EkertRichard Messnarz, ISCN GesmbH, Schieszstattgasse 4, A-8010 Graz,Austria

1. INTRODUCTION

In most cases, assessments are purely used todetermine a capability level profile of a set ofprojects and to derive improvement actions for anorganizational unit. Since 2003, a group of majorfirms (sizes from 200 up to 90,000 staff) startedusing a knowledge based assessment portal systemin which the assessment results of all projectsand divisions are stored in a central multiuserportal system. Here they also store all improvementfindings of all assessors in all assessments and it ispossible to extract improvement recommendationsfrom a large set of data across the enterprise andto compare projects (which projects carry synergypotentials of a reusable knowledge).

Since 2005, expert linking and team learningfacilities have been added, so that projects thathave weak areas can find expert projects (who arestrong in this area) inside the corporate firm. Also, itis possible to use team learning and training portalsso that if a project is weak in a certain area it canenter a virtual training room for upgrading the skillsin this area.

This approach is called ‘Assessment-based Learn-ing Systems’.

In this article, this assessment-based learningapproach is introduced, where two systems, theCapability Adviser Web Assessment Tool and the

∗ Correspondence to: Richard Messnarz, Richard Messnarz,ISCN GesmbH, Schieszstattgasse 4, A-8010 Graz, Austria†E-mail: [email protected]

Copyright 2007 John Wiley & Sons, Ltd.

Learning Management System, Modular Object-Oriented Dynamic Learning Environment (Moo-dle), are combined to support the improvementactions in a corporate environment supported byreference projects (best in the area and reusableknow-how) and an online training and tutoringlearning cycle.

2. ASSESSMENT-BASED LEARNINGCENTERS APPROACH

A formal project assessment usually results in acapability level profile (Messnarz and Tully 1999).It is typical that strengths (high levels) and weak-nesses (low levels) are represented in such a profile.Some processes may be already on a higher level,while some processes might show significant gaps.

The assessment-based learning centers approach(Figure 1) (Messnarz et al. 1999a,b, 2001) is basedon the improvement of these gaps by exploit-ing synergies (O’Keeffe and Harrington 2001) inthe corporate firm through learning from referenceprojects (expert links), in-house trainings and tutor-ing from improvement projects. A reference projectis a company project (can be from the same divi-sion or another worldwide location) in which theprocesses are on a high/higher level than in thecurrently assessed project. The collected evidences,assessment reports, assessor comments, and ratingsfrom these reference projects (best practice exam-ples) help to exploit corporate synergies for theimprovement of the currently assessed project.

In addition to that, an improvement project (aservice project to develop a new best practice to beused in the projects) might provide virtual learningspaces. An improvement project is usually started

Practice Section R. Messnarz and D. Ekert

Project

AssessmentCapability Adviser

Learn from ReferenceProjects

Learning ManagementSystem Moodle

Improvement projectwith available tutoring

Capability Level with gaps

Improvement

Receive onlinetutoring

Perform Exercises andHomework

Figure 1. The approach

if all the projects in the company have significantgaps in a certain area or process and new best prac-tices must be developed for the organization. Theresults of the improvement projects are then usedto improve all the other processes in the company’sprojects. From both, the reference projects andimprovement projects, experts act as tutors to helpimprove the processes by offering online tutoring,trainings, presentations, learning materials, leadingdiscussion forums, preparing exercises and assign-ments, etc. All this activities are performed onlinein a Learning Management System.

3. THE IMPLEMENTATION

3.1. The Underlying Systems

For the implementation, the following two systemswere used: the process assessments and capabilitylevel calculations were done with the CapabilityAdviser Web Assessment1 System; for the tutoring,training, and online courses the learning manage-ment system Moodle2 was used.

1 www.iscn.com/projects/piconew/2 www.moodle.org

Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569–577

570 DOI: 10.1002/spip

Practice Section Assessment-based Learning Systems

Figure 2. Capability level profile

The Capability Adviser System is an online webassessment system. It supports different processsets based on different process models (for exampleISO 15504, CMMI and Automotive SPICE), andoffers a complete management of assessments,assessors, and projects. The system can export theresults to various formats such as XML (accordingto the INTACS Scheme), Microsoft Excel, andWord3. Some of the features included are evaluationand reporting of results, benchmarking with otherprojects, online browsing, online self and formalassessment, capability profiles, attribute ratings,ratings on the work product level, virtual accessto assessor comments in a team, etc.

Assessors can share their views and commentsduring an assessment. The assessors’ results andtheir valuable comments are kept in the knowledgeSQL database so that upcoming assessment can alsolearn from the mistakes of the previous ones.

In a previous Minerva project called EPI4 (Edu-cational Partnerships through ICT) the CapabilityAdviser was integrated with the free and open

3 Microsoft, Windows, Excel and Word are registered trademarksof Microsoft Corporation in the United States, other countries, orboth.4 http://www-deis.cit.ie/epi/default.html

source5 course management system Moodle. A veryimportant reason why Moodle was chosen is thatit is easy to use and is popular in the univer-sity and industry sectors. Moodle offers a wideset of activities (assignments, chat area, discussionforum, presentation, workshops etc.) and courses ona daily, weekly or topical basis; user management;translations to over 70 languages; and a growingdevelopment community.

3.2. The Assessment and Knowledge Links

One of the goals of the formal assessment is to deter-mine the capability level of the assessed processes.In the example below (Figure 2) the assessment wasperformed for a group of Engineering processes.The capability level chart for the project shows thatsome of the Engineering processes are achieveda level 2, while others like Requirement Elicita-tion, System Requirements Analysis, and SoftwareRequirement Analysis are still on level 1. With themore detailed Attribute Ratings Profile (Figure 3),some minor gaps can be identified on level 1 (LargelyAdequate instead of Fully Adequate), but major gapscan be found on level 2 where the process attributes

5 http://www.opensource.org/docs/definition.php


DOI: 10.1002/spip 571


Figure 3. Capability level attributes

are rated as Partly Adequate and a lot of improve-ment is needed. The next example will focus on theimprovement of the process ‘Requirements Elicita-tion’.

In Figure 4 the process Requirements Elicitationhas been selected. The assessment-based learningsystem offers a list of company references andimprovement projects with tutors and in-housetrainings and, in addition, a list of available publiccourses containing similar topics. The list of projects,trainings, and courses depends mainly on theselected target level. In general, the lower the levelthe more the number of available reference projectsand improvement trainings. To provide a wideselection of reference projects with tutoring andin-house training, strong management support isneeded. It is expected that the tutors use 20–30% oftheir time for training and tutoring. The selection ofthe tutors also plays a very important role. Not allexperts can pass their knowledge. Training for thelearning management system, time to prepare thematerials, etc. must be also taken in account beforeselecting a project and for their tutor to become areference project.

The assessment data containing the assessorcomments and rated evidences from each assessor

from a reference project can be accessed throughthe Assessment Log. This way the project canalso improve some of their evidences (in mostcases they are documents) by just reviewing thereference materials consisting of the assessmentreport, assessor comment, evidences, improvementplan, etc. In most of the cases (especially with thelow level processes), studying the reference projectmaterials is not sufficient. Guidance from experts inthe form of tutoring and in-house training is needed.In the system, the Register button automaticallyregisters the user (project member) for the selectedtutorial or training in the learning managementsystem Moodle. The same user name and passwordare used by both systems.

Similar to the reference project assessment data,the improvement project data can be accessed. Animprovement project is started if all the projectsfrom the company have gaps in a particular processor a process area. The improvement project resultsare afterwards used by the projects. Tutoring of animprovement projects is very important.

The second list offers additional online Moodle(public) courses connected to the selected topic. Thecourses are not always public, but rather are heldin a small community of companies with the same


572 DOI: 10.1002/spip


Figure 4. Available tutoring and courses

focus (for example as in the Soqrates6 group). Thechosen Customer Relationship Management coursefrom the Innovation Manager training7 has the samefocus as the Requirement Elicitation process but ina more holistic approach. The course deals withtopics like learning from customers, how to capturecustomer innovation plans in advance, establishingknowledge management strategies for collectingand sharing product ideas/requirements etc.

3.3. The Learning Cycle and the Tutoring

After the gaps have been identified and the traininghas been selected, the improvement and learningcycle starts. The figure shows the six steps of thelearning cycle and tutoring:

• In addition to the presentation, discussions,etc. the tutor sets homework/exercises for thetraining;

• The project participants receive their homework,and can discuss and clarify with the tutor; and

• Upload their homework back to Moodle;

6 www.soqrates.de7 Certified Innovation Manager: www.innovationmanager.org

• The tutor reviews the homework and providesfeedback.

The first four steps can be repeated until ‘enough’homework/exercises are performed. This home-work/exercise can be uploaded to the CapabilityAdviser system as new evidence (step 5), whichresults in a higher capability level (step 6).

The online training in Moodle is divided intotwo parts. The first part offers some generaltraining information as well as informal means ofcommunication: for example, a chat room and adiscussion room. In the second part the course isplanned for a number of weeks. The first week,for example, offers some training materials inPowerPoint, list of references materials, exampleswith case studies, and also an assignment. Thetraining program can also be divided to topicsinstead of the weekly structure. Moodle offershere a wide set of activities and possibilities toconfigure the training individually. The tutoringdiffers from the training in a way that no additionalmaterials and presentations are provided; the tutorprovides and reviews homework and participate indiscussion forums.


DOI: 10.1002/spip 573


Figure 5. Learning cycle and tutoring

4. SYSTEM – ROLES ANDIMPLEMENTATION

The Capability Adviser system supports differentroles:

• Administrator: The Administrator creates orga-nizational accounts for the different divisionsand creates a corporate-wide assessor pool.

• Organization: Each organization can administerprojects, assessments, pool of assessors, andtailoring of process selections specific to theorganizational unit, and it does the maintenanceof project and assessor accounts.

• Content Provider: This is a specific user whocan enter new assessment models, learning ref-erences, and assign courses to specific processes.The system allows entering knowledge links.

• Assessor: The assessor has a work bench toassess projects, review evidences (uploaded),rate processes (base practices and generic prac-tices), exchange the data between the wholeassessment team online, generate profiles, gen-erate reports, generate assessment logs, etc.

• Participant/Learner: The participant is usually aproject representative who can log in, perform aself assessment, see the assessor comments andresults, generate reports, and upload evidences.They can also request best practice experiences


574 DOI: 10.1002/spip


Figure 6. Requirements Elicitation Tutoring

or access online organized courses to learn thebest practices.

The Moodle system supports different roles.

• Administrator: The Administrator createscourses, teaching modules, homework exercises,tutoring facilities in Moodle (www.moodle.com)and configures all courses.

• Trainer: The trainer is assigned to specificonline courses, is available at certain onlinetimes, performs online discussions, provideshomework, corrects homework materials onlineand summarizes the results in the learning team.

• Student:The student participates in the courseonline, accesses the learning material, discusseswith the tutor, performs a homework, receivesfeedback, refines the applied homework (appli-cation of concept in the work place), and usesthe refined material as an evidence later toprove in assessment that this competence iscovered now.

The systems are mostly used for ISO 15 504 andcombined courses are agreed for a corporate-wideimprovement in the firm.

Project personnel act as participants in theCapability Adviser System as well as students inthe Moodle system.

5. RESULTS

The public online courses have a more generalapproach and a wider target audience then thein-house training courses, which are tailored to thecompany processes. The Moodle-configured Cus-tomer Relationship Management course can be seenin Figure 5. The course offers PowerPoint Slidesextended with English and German voice presen-tation, as well as exercises, reference materials,discussion area, etc. In the year 2005, the Soqrates8

group (formed as a Bavarian Initiative to shareknowledge in important SPI and innovation fields)

8 www.soqrates.de


DOI: 10.1002/spip 575


Figure 7. Online Customer Relationship Management Course

used the course already to improve their processRequirement Elicitation process.

The following major organizations are alreadyusing the system.

• Continental• ZF Friedrichshafen AG• Giesecke & Devrient• T-Systems (German Telecom) and others.

Meanwhile experiences show that using the synergypotentials inside a corporate firm in a tactical wayleads to about 50% time reduction in achieving ahigher level in a project. (Presuming that expertlinks can be found and exploited).

6. CONCLUSIONS AND OUTLOOK

One of the most pertinent questions a businessmanager can ask is the following: ‘How can I makemy firm succeed where another fails?’

A key concept of the approach is the notionof lever (Messnarz and Tully 1999). Levers aremeans used by a firm to increase its resourcegenerating ability, just as a mechanical lever isused for increasing the force applied to an object.The analogy goes even further. Just as a force can

be applied in many different ways to the objectresulting in a similar displacement, the use of thedifferent levers can increase the resource generatingability of the firm resulting in similar businessbenefits. Finally, the resources are used to increasethe assets of the firm and to reward employees andstakeholders.

In the business management world the followinglevers are considered (Biro and Tully 1999):

• Operating leverage• Production leverage• Human leverage• Marketing leverage• Financial leverage

The approach described in this article directlyrelates to the human leverage. The exploitation ofhuman leverage is particularly important in soft-ware process improvement since software develop-ment is fundamentally a human mental process.

An exchange of best practices and a guided(using a learning system underlying the assessmentportals) and tutored learning of best practicesenable a learning culture that increases the speed ofimprovement and distribution of knowledge.


576 DOI: 10.1002/spip


SPI is more than a technical issue, and we needmore systematic approaches to include the humanlearning process in such systems in the future.

1 www.iscn.com/projects/piconew/2 www.moodle.org3 Microsoft, Windows, Excel and Word are reg-

istered trademarks of Microsoft Corporation inthe United States, other countries, or both.

4 http://www-deis.cit.ie/epi/default.html5 http://www.opensource.org/docs/

definition.php6 www.soqrates.de7 Certified Innovation Manager: www.innovation

manager.org8 www.soqrates.de

REFERENCES

Biro M, Tully C. 1999. The software process in the con-text of business goals and performance. In Chapter in theBook Entitled Better Software Practice for Business Benefit,Messnarz R, Tully C (eds). IEEE Computer Society Press:Washington, DC, Brussels, Tokyo, (ISBN 0-7695-0049-8).<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0769500498.html>.

Messnarz R, Tully C (eds). 1999. The PICO – Book: BetterSoftware Practice for Business Benefit – Principles andExperience. Wiley-IEEE Computer Society Press, 1stedition (September 27, 1999), ISBN-10: 0769500498,ISBN-13: 978-0769500492, Berlin, Tokyo, Washington,September 1999.

Messnarz R, Stubenrauch R, Melcher M, Bernhard R.1999a. Network based quality assurance, Proceedings ofthe 6th European Conference on Quality Assurance, Vienna,Austria, 10–12 April 1999.

Messnarz R, Stockler C, Velasco G, O’Suilleabhain G.1999b. A learning organization approach for processimprovement in the service sector, Proceedings of theEuroSPI 1999 Conference, Pori, Finland, 25–27.

Messnarz R, Nadasi G, O’Leary E, Foley B. 2001.Experience with teamwork in distributed workenvironments. Proceedings of the E2001 Conference, E-Workand E-commerce, Novel Solutions for a Global NetworkedEconomy, eds. Brian Stanford Smith, Enrica Chiozza, IOSPress: Amsterdam, Berlin, Oxford, Tokyo, Wash-ington.

O’Keeffe T, Harrington D. 2001. Learning to Learn: AnExamination of Organisational Learning in Selected IrishMultinationals, Vol. 25, Number 2/3/4. Journal ofEuropean Industrial Training, MCB University Press.


DOI: 10.1002/spip 577