Assessment-based learning systems—learning from best projects

  • Published on

  • View

  • Download

Embed Size (px)


  • SOFTWARE PROCESS IMPROVEMENT AND PRACTICESoftw. Process Improve. Pract. 2007; 12: 569577

    Published online 3 September 2007 in Wiley InterScience( DOI: 10.1002/spip.347

    Assessment-based LearningSystems Learning fromBest Projects

    Practice SectionRichard Messnarz* and Damjan EkertRichard Messnarz, ISCN GesmbH, Schieszstattgasse 4, A-8010 Graz,Austria


    In most cases, assessments are purely used todetermine a capability level profile of a set ofprojects and to derive improvement actions for anorganizational unit. Since 2003, a group of majorfirms (sizes from 200 up to 90,000 staff) startedusing a knowledge based assessment portal systemin which the assessment results of all projectsand divisions are stored in a central multiuserportal system. Here they also store all improvementfindings of all assessors in all assessments and it ispossible to extract improvement recommendationsfrom a large set of data across the enterprise andto compare projects (which projects carry synergypotentials of a reusable knowledge).

    Since 2005, expert linking and team learningfacilities have been added, so that projects thathave weak areas can find expert projects (who arestrong in this area) inside the corporate firm. Also, itis possible to use team learning and training portalsso that if a project is weak in a certain area it canenter a virtual training room for upgrading the skillsin this area.

    This approach is called Assessment-based Learn-ing Systems.

    In this article, this assessment-based learningapproach is introduced, where two systems, theCapability Adviser Web Assessment Tool and the

    Correspondence to: Richard Messnarz, Richard Messnarz,ISCN GesmbH, Schieszstattgasse 4, A-8010 Graz, AustriaE-mail:

    Copyright 2007 John Wiley & Sons, Ltd.

    Learning Management System, Modular Object-Oriented Dynamic Learning Environment (Moo-dle), are combined to support the improvementactions in a corporate environment supported byreference projects (best in the area and reusableknow-how) and an online training and tutoringlearning cycle.


    A formal project assessment usually results in acapability level profile (Messnarz and Tully 1999).It is typical that strengths (high levels) and weak-nesses (low levels) are represented in such a profile.Some processes may be already on a higher level,while some processes might show significant gaps.

    The assessment-based learning centers approach(Figure 1) (Messnarz et al. 1999a,b, 2001) is basedon the improvement of these gaps by exploit-ing synergies (OKeeffe and Harrington 2001) inthe corporate firm through learning from referenceprojects (expert links), in-house trainings and tutor-ing from improvement projects. A reference projectis a company project (can be from the same divi-sion or another worldwide location) in which theprocesses are on a high/higher level than in thecurrently assessed project. The collected evidences,assessment reports, assessor comments, and ratingsfrom these reference projects (best practice exam-ples) help to exploit corporate synergies for theimprovement of the currently assessed project.

    In addition to that, an improvement project (aservice project to develop a new best practice to beused in the projects) might provide virtual learningspaces. An improvement project is usually started

  • Practice Section R. Messnarz and D. Ekert


    AssessmentCapability Adviser

    Learn from ReferenceProjects

    Learning ManagementSystem Moodle

    Improvement projectwith available tutoring

    Capability Level with gaps


    Receive onlinetutoring

    Perform Exercises andHomework

    Figure 1. The approach

    if all the projects in the company have significantgaps in a certain area or process and new best prac-tices must be developed for the organization. Theresults of the improvement projects are then usedto improve all the other processes in the companysprojects. From both, the reference projects andimprovement projects, experts act as tutors to helpimprove the processes by offering online tutoring,trainings, presentations, learning materials, leadingdiscussion forums, preparing exercises and assign-ments, etc. All this activities are performed onlinein a Learning Management System.


    3.1. The Underlying Systems

    For the implementation, the following two systemswere used: the process assessments and capabilitylevel calculations were done with the CapabilityAdviser Web Assessment1 System; for the tutoring,training, and online courses the learning manage-ment system Moodle2 was used.


    Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569577

    570 DOI: 10.1002/spip

  • Practice Section Assessment-based Learning Systems

    Figure 2. Capability level profile

    The Capability Adviser System is an online webassessment system. It supports different processsets based on different process models (for exampleISO 15504, CMMI and Automotive SPICE), andoffers a complete management of assessments,assessors, and projects. The system can export theresults to various formats such as XML (accordingto the INTACS Scheme), Microsoft Excel, andWord3. Some of the features included are evaluationand reporting of results, benchmarking with otherprojects, online browsing, online self and formalassessment, capability profiles, attribute ratings,ratings on the work product level, virtual accessto assessor comments in a team, etc.

    Assessors can share their views and commentsduring an assessment. The assessors results andtheir valuable comments are kept in the knowledgeSQL database so that upcoming assessment can alsolearn from the mistakes of the previous ones.

    In a previous Minerva project called EPI4 (Edu-cational Partnerships through ICT) the CapabilityAdviser was integrated with the free and open

    3 Microsoft, Windows, Excel and Word are registered trademarksof Microsoft Corporation in the United States, other countries, orboth.4

    source5 course management system Moodle. A veryimportant reason why Moodle was chosen is thatit is easy to use and is popular in the univer-sity and industry sectors. Moodle offers a wideset of activities (assignments, chat area, discussionforum, presentation, workshops etc.) and courses ona daily, weekly or topical basis; user management;translations to over 70 languages; and a growingdevelopment community.

    3.2. The Assessment and Knowledge Links

    One of the goals of the formal assessment is to deter-mine the capability level of the assessed processes.In the example below (Figure 2) the assessment wasperformed for a group of Engineering processes.The capability level chart for the project shows thatsome of the Engineering processes are achieveda level 2, while others like Requirement Elicita-tion, System Requirements Analysis, and SoftwareRequirement Analysis are still on level 1. With themore detailed Attribute Ratings Profile (Figure 3),some minor gaps can be identified on level 1 (LargelyAdequate instead of Fully Adequate), but major gapscan be found on level 2 where the process attributes


    Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569577

    DOI: 10.1002/spip 571

  • Practice Section R. Messnarz and D. Ekert

    Figure 3. Capability level attributes

    are rated as Partly Adequate and a lot of improve-ment is needed. The next example will focus on theimprovement of the process Requirements Elicita-tion.

    In Figure 4 the process Requirements Elicitationhas been selected. The assessment-based learningsystem offers a list of company references andimprovement projects with tutors and in-housetrainings and, in addition, a list of available publiccourses containing similar topics. The list of projects,trainings, and courses depends mainly on theselected target level. In general, the lower the levelthe more the number of available reference projectsand improvement trainings. To provide a wideselection of reference projects with tutoring andin-house training, strong management support isneeded. It is expected that the tutors use 2030% oftheir time for training and tutoring. The selection ofthe tutors also plays a very important role. Not allexperts can pass their knowledge. Training for thelearning management system, time to prepare thematerials, etc. must be also taken in account beforeselecting a project and for their tutor to become areference project.

    The assessment data containing the assessorcomments and rated evidences from each assessor

    from a reference project can be accessed throughthe Assessment Log. This way the project canalso improve some of their evidences (in mostcases they are documents) by just reviewing thereference materials consisting of the assessmentreport, assessor comment, evidences, improvementplan, etc. In most of the cases (especially with thelow level processes), studying the reference projectmaterials is not sufficient. Guidance from experts inthe form of tutoring and in-house training is needed.In the system, the Register button automaticallyregisters the user (project member) for the selectedtutorial or training in the learning managementsystem Moodle. The same user name and passwordare used by both systems.

    Similar to the reference project assessment data,the improvement project data can be accessed. Animprovement project is started if all the projectsfrom the company have gaps in a particular processor a process area. The improvement project resultsare afterwards used by the projects. Tutoring of animprovement projects is very important.

    The second list offers additional online Moodle(public) courses connected to the selected topic. Thecourses are not always public, but rather are heldin a small community of companies with the same

    Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569577

    572 DOI: 10.1002/spip

  • Practice Section Assessment-based Learning Systems

    Figure 4. Available tutoring and courses

    focus (for example as in the Soqrates6 group). Thechosen Customer Relationship Management coursefrom the Innovation Manager training7 has the samefocus as the Requirement Elicitation process but ina more holistic approach. The course deals withtopics like learning from customers, how to capturecustomer innovation plans in advance, establishingknowledge management strategies for collectingand sharing product ideas/requirements etc.

    3.3. The Learning Cycle and the Tutoring

    After the gaps have been identified and the traininghas been selected, the improvement and learningcycle starts. The figure shows the six steps of thelearning cycle and tutoring:

    In addition to the presentation, discussions,etc. the tutor sets homework/exercises for thetraining;

    The project participants receive their homework,and can discuss and clarify with the tutor; and

    Upload their homework back to Moodle;

    6 www.soqrates.de7 Certified Innovation Manager:

    The tutor reviews the homework and providesfeedback.

    The first four steps can be repeated until enoughhomework/exercises are performed. This home-work/exercise can be uploaded to the CapabilityAdviser system as new evidence (step 5), whichresults in a higher capability level (step 6).

    The online training in Moodle is divided intotwo parts. The first part offers some generaltraining information as well as informal means ofcommunication: for example, a chat room and adiscussion room. In the second part the course isplanned for a number of weeks. The first week,for example, offers some training materials inPowerPoint, list of references materials, exampleswith case studies, and also an assignment. Thetraining program can also be divided to topicsinstead of the weekly structure. Moodle offershere a wide set of activities and possibilities toconfigure the training individually. The tutoringdiffers from the training in a way that no additionalmaterials and presentations are provided; the tutorprovides and reviews homework and participate indiscussion forums.

    Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569577

    DOI: 10.1002/spip 573

  • Practice Section R. Messnarz and D. Ekert

    Figure 5. Learning cycle and tutoring


    The Capability Adviser system supports differentroles:

    Administrator: The Administrator creates orga-nizational accounts for the different divisionsand creates a corporate-wide assessor pool.

    Organization: Each organization can administerprojects, assessments, pool of assessors, andtailoring of process selections specific to theorganizational unit, and it does the maintenanceof project and assessor accounts.

    Content Provider: This is a specific user whocan enter new assessment models, learning ref-erences, and assign courses to specific processes.The system allows entering knowledge links.

    Assessor: The assessor has a work bench toassess projects, review evidences (uploaded),rate processes (base practices and generic prac-tices), exchange the data between the wholeassessment team online, generate profiles, gen-erate reports, generate assessment logs, etc.

    Participant/Learner: The participant is usually aproject representative who can log in, perform aself assessment, see the assessor comments andresults, generate reports, and upload evidences.They can also request best practice experiences

    Copyright 2007 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2007; 12: 569577

    574 DOI: 10.1002/spip

  • Practice Section Assessment-based Learning Systems

    Figure 6. Requirements Elicitation Tutoring

    or access online organized courses to learn thebest practices.

    The Moodle system supports different roles.

    Administrator: The Administrator createscourses, teaching modules, homework exercises,tutoring facilities in Moodle ( configures all courses.

    Trainer: The trainer is assigned to specificonline courses, is available at certain onlinetimes, performs online discussions, provideshomework, corrects homework materials onlineand summarizes the results in the learning team.

    Student:The student participates in the courseonline, accesses the learning material, discusseswith the tutor, performs a homework, receivesfeedback, refines the applied homework (appli-cation of concept in the work place), and usesthe refined material as an evidence later toprove in assessment that this competence iscovered now.

    The systems are mostly used for ISO 15 504 andcombined courses are agreed for a corporate-wideimprovement in the firm.

    Project personnel act as participants in theCapability Adviser System as well as students inthe Moodle system.

    5. RESULTS

    The public online courses have a more generalapproach and a wider target audience then thein-house training courses, which are tailored to thecompany processes. The Moodle-configured Cus-tomer Relationship Management course can be seenin Figure 5. The course offers PowerPoint Slidesextended with Eng...


View more >