FP7ICTSTREPProject

http://www.learnpad.eu

DeliverableD8.4

Demonstratorsassessment

Templatev.0.3

ProjectNumber : FP7-619583

ProjectTitle : LearnPAdModelBasedSocialLearningforPublicAdministrations

DeliverableNumber : D8.4

TitleofDeliverable : Demonstratorsassessment

NatureofDeliverable : Report

Disseminationlevel : Public

Licence : CreativeCommonsAttribution3.0License

Version : 4.0

ContractualDeliveryDate : 31October2016

ActualDeliveryDate : 31October2016

ContributingWP : WP8

Editor(s) : AndreaSergiacomi(MAR)

Author(s) : CinziaAmici(MAR),AlessiaBalducci(MAR),SerenaCarota(MAR),LauraGiorgio(MAR),BarbaraRe(UniCam),AndreaSergiacomi(MAR)

Reviewer(s) : AntoniaBertolino(CNR),GuglielmoDeAngelis(CNR),AndreaPolini(Unicam),BarbaraRe(Unicam)

AbstractWork Package 8, titled Demonstrators, aimed at providing support to the technical and research work-packages for better focusing on their activities, and at assessing the applicability, acceptance andeffectivenessof theproposed solutions.Anearly validationwas carriedout considering the first runninginstanceof theLearnPAdplatformavailable (since06November2015).Agroupofusers, sampled fromboth project demonstrators (EPBR and SUAP), tested the Learn PAd platform and provided us withfeedbackandcommentstoimproveit.Fromamethodologicalperspective,intheearlyvalidationwerunaquestionnaire and collected technical and detailed opinions from a focus group with influential people,skilledincomputerscienceorineducation.

Inthisdeliverablewepresenttheaims,settingandresultsofthefinalvalidationrunonthelatestrunninginstanceoftheLearnPAdplatformavailable,whichincludesalltheLearnPAddefinitivecomponents.Froma methodological perspective, in the final validation we submitted a questionnaire ex-ante to assesslearners’competencesandexperiences,andaquestionnaireex-posttoassesstheirpossibleimprovementsinknowledge,skillsandcompetence,acquiredattheconclusionofatrainingsession lastingtwoworkingweeks (since14September2016).Theex-postquestionnairealsoaimedat surveying thedegreeofusersatisfactionandengagementandataskingtheiropinionregardingusability,efficacyandcompletenessoftheLearnPAdPlatform.Thefinalvalidationinvolved61users(31learnerswereinvolvedinusingtheLearnPadplatformand30learners,referredtoasthematchingcontrolgroup,workedwithinanotherregionale-learningplatformcalledMARLENE).

ii

Keywordlist

BusinessProcess,EuropeanProjectBudgetReport,ModellingEnvironment,LearningPlatform,PublicAdministration,SportelloUnicoAttivitàProduttive,OneStopShop,Validation,Accessibility,UserExperience,Gamification,TitoloUnico,SegnalazioneCertificatadiInizioAttività,UrbanVariant,ConferenceofServices,EuropeanQualificationsFramework,LearningMaterial,Simulation,Recommender,KeyPerformanceIndicators,Learningapproach

iii

iv

Document History

Version Changes Author(s) 0.1 TableofContent(TOC) AndreaSergiacomi(MAR)

1.0 FirstDraft:Contenttobeimproved

AndreaSergiacomi(MAR)

1.1 SecondDraft:Contenttobemergedandfixed

AndreaSergiacomi(MAR),AlessiaBalducci(MAR),LauraGiorgio(MAR),AntoniaBertolino(CNR)

2.0 Internalrelease:Contenttobefinalized

AndreaSergiacomi(MAR)

2.1 Secondinternalrelease:Conclusionstobefixed

AndreaSergiacomi(MAR),AlessiaBalducci(MAR)

2.2 Thirdinternalrelease:Allcontentaddedandcheckedfortheinternalreview

AndreaSergiacomi(MAR),AlessiaBalducci(MAR)

3.0 Submission AndreaSergiacomi(MAR)

Document Review

Release Date Ver. Reviewer(s) Comments ToC October3,2016 0.1 AntoniaBertolino,

BarbaraReSuggestiontoaddthelearningapproachdescription

Draft October14,2016 1.x AntoniaBertolino,GuglielmoDeAngelis,AndreaPolini

addedpartnerintegrationsandcontributesfromothertechnicaldeliverables

Internalrelease October24,2016 2.x AntoniaBertolino

Candidatefinal October27,2016 3.0

v

vi

Glossary, acronyms & abbreviations

Item Description Accessibility Incomputerscienceisthecharacteristicofacomputingsystemtobeeasilyusable

byanyuser,regardlessofhissensory,motororpsychicindividualcapacities,ofanytemporaryorpermanentdisability,ofevenculturalaspectsoraspectsrelatingtofamiliaritywiththeuseofICTinterfaces.

Bloom'staxonomy

isasetofthreehierarchicalmodels(coveringthelearningobjectivesincognitive,affectiveandsensorydomains)usedtoclassifyeducationallearningobjectivesintolevelsofcomplexityandspecificity.

BP BusinessProcess

CoS ConferenceofServices

EPBR EuropeanProjectBudgetReport

EQF EuropeanQualificationsFramework

Gamification Theuseofelementsborrowedfromthegamesandotherfuntechniques,adaptedtoexternalcontexts(eg.scores,levels,rewards,badges,giftsandotherloyaltymethodsorpracticestoarouseinterestorcontributetotheusersatisfaction)

Item Description

KPI KeyPerformanceIndicatorssettomeasurestrategic,organizationalandlearninggoals

LearningApproach

TheoverallarchitectureadoptedbytheLearnPAdproject,mappingtheMethodology(inspiredbytargets,contextandexistinglearningparadigmsforPAs),thechoicesmade(referredtothecentralityoftheworkprocesses,thetransmissionofknowledgeandusefulcompetences,theinteractivedidacticmethodologiesandthemixofe-learningmodels:selflearning,assistedlearning,collaborativelearning,organizationallearning),thesubjectstobeinvolved(mandants,modelers,learners,experts)andtheevaluationsystem(basedonaKPIDashboard)

LearningMaterial

Allentitiesnotrepresentingmodelsbutrelevantforlearningasbooks,tutorials,learningaudioandvideofilebutalsobrowsingandsimulation

MARLENE MarcheLearningNetworke-learningplatform

OSS OneStopShopforProductiveActivities(synonymofSUAP)

PA PublicAdministration

vii

Item Description Recommender Learningmodethatprovidesemanticallycontextualizedlearningcontent

(accordingtotheuserprofileorthefunctionalitieshe/sheisplaying)throughasidebaroftheLearnPAdplatform

SCIA ‘SegnalazioneCertificatadiInizioAttività’meansinEnglishcertifiedreportforstartingabusinessactivity;hereusedtonameanotherkindofSUAPbusinessprocess

Simulation Learningmodethatsimulatestheexecutingof(partof)aprocessinatrainingenvironmentwithtrainingdata

SUAP SportelloUnicoAttivitàProduttive

TitoloUnico ‘Titolounico’meansinEnglishstandardrequestforstartingabusinessactivity;hereusedtonamethebusinessprocessconsidered

UrbanVariant IsanotherkindofSUAPbusinessprocess,accordingtowhichtheentrepreneur'srequestmakesnecessarytoinvolvethecitycouncilinformallychangingtheurbanisticdestinationofazonefromitsoriginalplan

UX UserExperience.ThesubjectivefeelingsthatanuserexperiencedusinganICTsystem,withreferencetotheempiricandaffectiveaspects,tothevalueandmeaningattributionhe/sherelatedtotheavailabilityandtheinteractionwiththesolution,aswellastotheoverallperceptionsandpersonalthoughtshe/shehadaboutitseffectiveness,theeaseofuseoftheinterface,thesystem'sefficiency,etc.

viii

TableofContent(ToC)ListofTables......................................................................................................................................................x

ListofFigures...................................................................................................................................................xi

1Introduction...................................................................................................................................................1

1.1Purpose...................................................................................................................................................1

1.2Relateddeliverables................................................................................................................................1

1.3Structureofthedeliverable....................................................................................................................2

2Learningapproach..........................................................................................................................................4

2.1Explanationofthemethodology.............................................................................................................4

2.2Thechoices..............................................................................................................................................6

2.3Subjectsinvolved.....................................................................................................................................7

2.4Evaluationsystem...................................................................................................................................8

3LearnPadPlatform:finalvalidation...............................................................................................................9

3.1Resultsfromearlyvalidation...................................................................................................................9

3.2Validationintopractice.........................................................................................................................10

3.2.1Referencedplatforms.....................................................................................................................11

3.2.2Validationscenario.........................................................................................................................18

3.2.3Usedmethodology.........................................................................................................................21

3.2.4Questionnaires...............................................................................................................................22

3.3Resultsfromquestionnaires..................................................................................................................27

3.3.1Profilingandcontextdata..............................................................................................................27

3.3.2Competenceself-assessmentdata.................................................................................................31

3.3.3Learningassessmentdata..............................................................................................................34

3.3.4Usabilitydata..................................................................................................................................37

3.4Synthesisofthecollectedfeedback......................................................................................................39

3.4.1Positiveoverallopinionsandplatformsstrengths.........................................................................39

3.4.2Negativeoverallopinionsandmissingfunctionalities...................................................................40

4Resultsanalysisandconclusions..................................................................................................................42

Appendix.........................................................................................................................................................49

ExanteQuestionnaire(inItalian)................................................................................................................49

ExpostQuestionnaire(inItalian)................................................................................................................53

Collectedfeedback(inItalian).....................................................................................................................61

Collectedfeedback(inenglish)...................................................................................................................67

ix

x

ListofTablesTable1CentralchoicesmappedinthemodelsanddidacticsolutionsadoptedwithintheLearnPAdapproach...........................................................................................................................................................7Table2Usabilityresultsfromtheearlyvalidation..........................................................................................10Table3DescriptorsdefininglevelsintheEuropeanQualificationsFramework(EQF)...................................25Table4Aggregateddatarelatedtothetwosamplesinvolvedinthetesting................................................28Table5PerceivedimprovementsinEQFskills................................................................................................33Table6Profitresultsforeachclusterofquestions.........................................................................................35Table7Dataandfiguresrelatedtotheusabilityexpostanswers.................................................................38Table8Overallstatisticsforprofitanswersresults(themeanofthetotalscoreacquiredandthemedianscore)...............................................................................................................................................................43Table9Pearsoncorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(PAexperience,SUAPexperience,anddedicatedtime).......................................................................................43Table10Kendalltaucorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(age,SUAPexperience,anddedicatedtime)..................................................................................................44Table11PearsoncorrelationbetweenthenumberofindividualcorrectanswersandthesumofdeclaredlevelsinthefiveEQFcompetences.................................................................................................................46Table12Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandtheusersoverallratingoftheplatforms..............................................................................................................................................47

xi

ListofFiguresFigure1TheLearnPAdplatformhomepage..................................................................................................11Figure2Thebrowsingmode:awikipagerepresentingatask.......................................................................12Figure3Thevalidatedoutputofacontributionbythequalityassessmentsystem......................................14Figure4Afterthesimulationisactivated,therecommenderbarproposescontextualizedsimilarcases.....15Figure5TheLearnPAdKPIDashboard...........................................................................................................16Figure6TheMARLENEinterfacetoaccessaSCORMcourse..........................................................................16Figure7ApageofcontentfromtheMARLENEcourseaboutSUAPCoS........................................................17Figure8TheSUAP“TitoloUnico”BusinessProcess.......................................................................................19Figure9The“ConferenceofServicesmanagement”sub-processwithinthe“TitoloUnico”BP...................20Figure10a/bTheLearnPAdandMARLENEsamplesbygender....................................................................28Figure11a/bTheLearnPAdandMARLENEsamplesbyage..........................................................................29Figure12a/bTheLearnPAdandMARLENEsamplesbyqualificationineducation.......................................29Figure13a/bTheLearnPAdandMARLENEsamplesbypublicrole...............................................................30Figure14a/bTheLearnPAdandMARLENEsamplesbyyearsofworkat(oronbehalfof)aPA...................30Figure15a/bTheLearnPAdandMARLENEsamplesbyyearsofexperienceinOSS/SUAP...........................31Figure16Theplatformsoverallusagetime(inhours)...................................................................................31Figure17ThedeclaredimprovementinEQFskillswithinthetwosamples...................................................32Figure18ThepercentageofdeclaredimprovementineachrelevantEQFskillwithinthetwosamples......33Figure19Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithinthetwosamples..............................................................................................................................................34Figure20Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithinthesamplesandperclustersofquestions......................................................................................................36Figure21Acomparingoverviewonacquiredopinionsaboutusability,userexperienceandplatformeffectivenessbytheusersofLearnPAdandMARLENE..................................................................................39Figure22a/bThePOSITIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE.......................................39Figure23a/bTheNEGATIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE.....................................40

1

1Introduction

1.1PurposeInthecontextoftheLearnPadproject,demonstrators(i)helpthetechnicalandresearchwork-packagestobetterfocustheiractivities;and(ii)theypermittoassesstheapplicability,acceptanceandeffectivenessoftheproposedsolutionwithinrealworkingcontexts.AccordingtoWP8objectiveswehadtovalidateprojectresults considering the platform effectiveness in supporting civil servants training and collaborativeactivities, by directly involving the end users. During the third year of execution of the project, theobjectives of the demonstrators evolved according to the state of development of the project, taking indeep consideration the effort of the whole consortium and of the partners in each single WP anddeliverable that represented, for the platform, a new release, an empowerment or a content adaptingrequest.IntheY2reportthereviewerssetsomespecificrequirementsforsuchvalidationrelatedtothesizeandthequalityoftheexperiment,whichhadtoprovidestatisticallymeaningfulconclusions.TheyalsoaskedtotesttheeffectivenessoftheconceptuallearningframeworkinordertoassesssomerealtangiblelearninggoalsachievedbythecivilservantsusingtheLearnPAdplatformintheworkplace.WefocusedonthecaseofOSS(SUAP)tobeabletoinvolvethelargestnumberofavailableusersaroundacommunity of practice clearly identified. To pursue the required results, the final validation has beenplanned to last aminimum of twoweeks, while the experimenters, involved in the SUAPmanagementprocesses,hadcontinuedworkingasusualintheirdailydutiesandroutines.The validation, better described in Section 3.2, was carried out dividing the 72 recruited people in twogroups,equallypopulatedandashomogeneousaspossible: theusersofa running instanceof theLearnPAd platform (available at the address http://Learn PAd.regione.marche.it:8080/xwiki/) and the controlgroup, usingMARLENE, amore traditional e-learningMoodle platform, already available in theMarcheRegion(http://marlenescuola.regione.marche.it/moodle/).Apreviousquestionnairewasprovidedtoall theexperimenters,todefineexantetheirownmasterdataandthestatusoftheirexperience,EQFcompetencesandexpertiseinthematterofthecourse.Subsequently,afterthetwoweekstrial,amoredetailedexpostquestionnaire-alwaysthesameforbothgroups-hadbeensubmittedtoretrieveusefuldatainordertounderstandifandhowthoseplatformshadhelped them to learn, and the judgment they gave to this experience, in terms of usability andeffectiveness.Attheendoftheexperimentationwereceivedback61compiledquestionnaires.Thesedata,bothmeasurableandbasedoncommentstobe interpretedandrepresented inverbal form,were anonymized, aggregated and processed to provide a clearer picture of the evaluation results, asexplainedlater.

1.2RelateddeliverablesThefollowingdocumentshavebeentakenasreferencepoints:• D1.2.RequirementsAssessmentReport.Thisdeliverablewasconsideredasitreportsonassessmentof

realizationstatusofthoserequirementsthathavebeencollected,analysed,andreportedinD1.1.

2

• D2.4CorePlatform Implementation– SecondVersion. Thedeliverable consistsof a secondand finalversionoftheLearnPAdcoreplatform,andisofsoftwarenature.

• D5.4 - KPIOntology and LearnersAssessmentMechanism. This deliverable describes thedesignof acomprehensive solution to enable PA employees meeting their business goals on a strategic andoperational level, deriving operational goals, learning goals and competencies needed to achieve acorrectexecutionofrelevantbusinessprocesses.ItoutlinesthetypicalgoalsandKPIsrequiredtoplanandtracklearninginarealworkplaceenvironmenttakingasexamplethe“TitoloUnico”SUAPprocess.

• D8.1.DemonstratorsBPandKnowledgemodels.Thedocumentidentifiesthedemonstratorsrelatedtothe Learn PAd project within the two following scenarios: European Project Budget Reporting and“SportelloUnicoAttivitàProduttive”.

• D8.1. (Addendum)UserPerspectiveandProjectEvaluationStrategies.Thisadditionaldeliverablewastriggered by the exigency of a deeper understanding of user needs is required. It is referred as itprovidedearlyinsightsandguidelinesonhowtheevaluationoftheimplementationshouldbecarriedout.

• D8.2.PlatformPrototypeEarlyValidationResults.Itreportstheresultsofthepreviousvalidationwhichhavebeencomparedwiththeresultsshowedinthepresentdeliverable.

• D8.3. Demonstrators Populated Learning Platform. It consists of the final version of the Learn PAdcontentforthetwodemonstratorsdevelopedintheproject:EuropeanProjectBudgetReport(EPBR),andSportelloUnicoAttivitàProduttive(SUAP).Thedeliverableisofsoftwareandreportnatureanditprovidestheofficialreleasedversionofcontent,alsoincludinginformationonhowsuchcontentscanbeaccessed. Inparticular,werefer toboth theLearnPAdModelingEnvironmentandtheLearnPAdLearningPlatform.

Moreover,thevalidationusedasabackgroundthefollowingdeliverableswhosenatureisReport,Softwareorboth:• D3.3-FinalLearnPAdMetamodelsandImplementationofModelTransformationsforManaging

BusinessProcessesModelsinPublicAdministrations• D4.3-QualityAssessmentMechanismsImplementation• D5.1-ModelsforSettingtheWiki• D5.5-SemanticsfortheWiki–FinalIteration• D6.3-LearnPAdSimulationEnvironment:Finalrelease• D7.2IntegrationToolsandSystems• D9.5Technology-orientedLearnPadwhitepaper• D9.7LearnPAdExploitationPlan–FinalIteration

1.3StructureofthedeliverableBesidesthisbriefintroduction,thisdeliverableisstructuredinthreeparts,respectivelyfocusingon:

A. Learningapproach(Chapter2)In this chapter we highlight the learning framework defined by the consortium, describing it intermsofadoptedmethodology(inspiredbytargets,contextandexistinglearningmodelsforPAs),choicesmade, involved subjects andevaluation system.As required in theY2 review report, theconceptualframeworkforlearningintheworkplacehasbeenusedasareferenceforbothtotheevaluationapproachandthelearninggoalsmappedinthedeployedKPIDashboard.

3

B. Finalvalidationoftheplatform(Chapter3)ThissectiondescribesthevalidationofthefinalversionoftheLearnPAdplatform.Inparticular,atfirstweprovidetheresultsfromthepreviousearlyvalidation,asapossibleyardstickto interpretthe actual findings. Then we explain how we conducted the validation describing: thefunctionalities of the platforms that have been tested, the chosen scenario, the validationmethodology,andthedevelopedquestionnaires.Finally,theresultsobtainedfromtheanswerstothequestionnairesarerepresented,andasummaryofthecollectedfeedbackisreported.

C. Resultsandconclusions(Chapter4)Thissectionprovidesevidenceandreportssomeconclusions,summarizingandfinalizingtheworkdoneforthefinalvalidation.

4

2LearningapproachThisChapterwill illustrate the learningapproachutilised for theLearnPAdprojectand its impactonthedidacticandtechnologicalchoices.

2.1ExplanationofthemethodologyThearchitectureofthelearningmodelproposedderivesfromacross-analysisofthreesystemvariables:

a. Targetcharacteristicsb. Characteristicsofthecontextinfluencingthemodelsandthelearningobjectivesc. ExistinglearningmodelsforPA

a. Targetcharacteristics

TheLearnPAdproject isaimedat thecivil servant, thereforeanadult inaworkingenvironment,who isgenerallyusedtodoinghighlyprescriptive(typicallynormedbystrictrules)professionalactivities.

The working context is still characterised by particularly fragmented working processes among theemployees, a condition that does not facilitate the understanding of the logic at the basis of theprofessionalactivityofeachindividual.

Beinganadult,thecivilservantisparticularlyorientedat:

• Learning from experience: this highlights the need to focus on experiential techniques whenconstructing the learning setting. Such techniquesmake use of the learner’s experience. Amongthem,themostefficientandcommonare:simulations,problemsolving,andcasestudies.

• Selective learning: the adult only learns what he/she needs to know and to do in order toefficientlyfacerealsituationsinlifeandatwork.

• Beingengagedbysupportinglearningprocesseswhentheseareshowntoassisthimintasksandproblemsfacedindailylife.Infacttheadultgainsnewknowledge,understanding,abilities,valuesand attitudes more effectively when these are presented and applied in the context of realsituations.

Froma technological skillsperspective, thecivil servant isgenerallyaccustomedto the technologies thatsupporthis/herprofessionalactivities,aswellaslearningsystems.Furthermoretherearewidespreadandampletechnologicalfacilitiesintheworkplace.

b. Characteristicsofthecontextinfluencingthemodelsandthelearningobjectives

Europeanandnationaldirectivesregardingsimplification,anti-corruptionandefficiencyinthePApushforreorganisation of the PA based on lean work processes, in order to ensure procedure deadlines andtransparencyofthedecisionmakingprocesses.

Thischange inpacehasanenormous impactonthe learningsystemsthatmustsupportsaidchangeandthereforemustreorganiseknowledgeintermsofcompetenceandworkprocesses.

5

c. ExistinglearningmodelsforPA

Afterananalysisofthetargetandtheenvironmentinwhichitoperates,thechoiceofmethodologieswasbasedonthelearningexperiencesclosesttothePAcontextandthemostcommone-learningmodels.

Regarding the learning experience, the didactic model of the Training School of the Region ofMarche,whichasaninternalstructureexclusivelytrainscivilservants,wasadopted.

Regardinge-learningmodels,referencewasmadetothedirectiveoftheMINISTRYFORINNOVATIONANDTECHNOLOGYandtheMINISTRYFORPAconcerninge-learningmodelsforPA

Thisdirectiveidentifies:

• The Self learning model, in which the technology predominantly provides access to learningmaterials offering repositories of material structured and organised around self learning. Staticlearningcoursesandtoolsforself-assessmentareproposed.Inthiscasetheuserinteractsalmostexclusivelywiththecontentmanagementsystem.

• The assisted learning model, in which assistants, tutors and technological staff play a role,interactingwithlearners,andhelpingthemusethematerials.Inthismodeltheaccesstomaterialsremains important but less than in the previousmodel. In fact human communicationmediatedthroughthecomputerispresentfacilitatingandenrichingthelearningprocesswhichisnolongerpurelybasedonstudyingmaterials.

• The collaborative learningmodel, inwhich interpersonal relations are evermore important, notonly with the facilitators but also between learners. This model comprises systems based oncollaboration and community practice. Activities involving cooperation between the participantsareofgreatimportance,aswellastechnologyaimedatfavouringinteraction.

RegardingthemodelutilisedbythetrainingschooloftheRegionofMarcheforPA,thetraditionallearningmodel based on the transfer of knowledge has been superseded in favour of a model centred on thedevelopmentofcompetencesthroughtheuseofactivedidacticmethodologies.

Thebasisof thismethodologyderived from themost commonknowledge theories is the following: ‘theadult in aworking context learns (gains new competences and develops new attitudes) if the followingconditionsaremet:

• Trainingcontract:thisisdeterminedwhenthetrainingofferedisrecognisedasadvantageousandusefulforprofessionalimprovement,bothforthelearnerandfortheorganisation.Inorderforthisconditiontoexisttheproposalmustbemadeonthebasisofapreliminaryanalysisoftheworkthatallowsthedesignertoidentifytheareasofresponsibilityforeachparticipantandthereafterdesignacoursethatiscoherentwiththeirworkingprocesses.

• Activeparticipation:thelearnerneedstoplayanactiveroleinpersonallyverifyingtheefficiencyofthe information provided, even if simulated, within his/her professional role. To guarantee thiscondition the school adopts an interactive didactic methodology oriented to solving realprofessionaltasks.

• MandantInvolvement:fromtheanalysisstage,thetrainingmustbestrictlyrelatedtotheoperativegoals of theorganisational structure and consequently the responsibility for the learning/changedoes not depend solely on the school and the individual learner, but also on the entireorganisationalsystemandinparticularthemanagement.

6

• Assessmentoftheresults:afurtherconditionforeffectivenessisrelatedtotheassessmentofthelearning outcomes. To this end, the school adopts a system monitoring all the variables thatinfluence the overall result: assessment of the training quality, teacher assessment, competenceassessmentandassessmentoftheoutputquality.

2.2ThechoicesOn the basis of the aforementioned criteria, principled choices on the matter were made, whichdetermined the orientation of the Learn PAd approach. These choices focus on five main principalfeatures:

1. Thecentralityoftheworkprocessesasafactororientingtheorganisationoftheinformationandthecontent.

2. The transmission of knowledge and competences seen as useful for the learner in that they aredirectlyconnectedtoconcreteprofessionalexperience.

3. Functional development sustaining interactive didactic methodologies oriented to solving realprofessionaltasks,whichfosteractivelearnerparticipation.

4. The adoption of amix of e-learningmodels fromwhich themost functional and target orientedfeatureswereselected.

5. Theadoptionofacollaborativelearningperspectivethatcansupportchangesintheorganizationalprocesses.

The table below presents the solutions selected at a training system level based on the choices in thematter.

ModelsandstandardsatthebasisofLearnPAd

Didactic/technologicalsolutionsadopted

Centralityoftheworkprocesses

• Businessprocessmodel• Organisationalmodelrelated

torecommender• Documentmodel• KPImodel

Interactivegraphicrepresentationoftheworkprocesses

Transmissionofknowledgeandcompetencesseenasuseful

• Businessprocessmodel• Competencemodel• Documentmodel• KPImodel

InteractivegraphicrepresentationoftheworkprocessesSimulationbasedonrealcasestudiesAvailabilityofformstosupportexplanationoftheprofessionalactivity

Interactivedidacticmethodologiesorientedtosolvingrealprofessionaltasks

• Organisationalmodel• Businessprocessmodel• CaseManagementModel

SimulationGamificationRecommenderCommentingandprovidingcontributions

7

Mixofe-learningmodels UseofWIKItechnology Selflearningmodel• Browsingmode• Integrativedidacticmaterial

forselflearningandindividualin-depthstudy

Assistedlearningmodel• Expertconsultation• Recommender• SimulationCollaborativelearningmodel• Feedbackvs.modeller• Commentvs.platform• Collaborativesimulation

Adoption of acollaborative learningperspective that cansupport changes in theorganizationalprocesses.

UseofWIKItechnology

Feedbacks;Commentingandprovidingcontributions.

Table1CentralchoicesmappedinthemodelsanddidacticsolutionsadoptedwithintheLearnPAdapproach

2.3SubjectsinvolvedLearnPAdlearningmodelinvolvesinitsprocessesasetofactorsthatsupportthedifferentphaseoftheprocessmodellingandimplementation.Theyare:

Themandant

Themandantisalwaysaninternalstaffmemberwithintheorganisationpromotingthetrainingprocess,whocorrespondsto(someof)thedescriptionsbelow:

• He/shecontributestothedescriptionandmappingofthebusinessprocessesandorganisationalmodel.

• He/shedefinesthestrategic,organisationalandoperativegoalsincollaborationwiththemodeller.• He/sheisthedecisionmakerwithrespecttolearningpolicieswithintheunitororganisation.• He/shereceivestheresultsofthelearningprocessesandinthisrolehe/sheisalsoresponsiblefor

thelearningimpact.• He/sheisthefinaldecisionmakerwithrespecttoproposalsforinnovationpresentedbythe

modelleronthebasisoffeedbackfromthelearners.

Themodeller

Themodellermanagesthedesign,maintenance,andmodificationsofmodelsandcontentsoftheplatform,alsotakingintoaccountlearnerfeedback.Inthisenvironmenthe/sheisresponsiblefortheassessmentoffeedbackqualityandofitsimpactonthemodels.

Itisuptothemodellertovalorisepotentialaspectsforinnovationcomingfromthelearnerfeedback.He/sheisthereforetheprincipalguarantorfortheprocessofmodelinnovation.

8

Thelearner

Thelearnerisalwaysaninternalstaffmemberwithintheorganisationpromotingthetrainingprocess.

He/sheisresponsibleforindividuallearning.

The said learner is also co-producer of knowledge through the production of feedback, interactionwithexperts,productionoflearningobjects,andinteractionwithotherlearners.

Theexpert

Maybeinternal/externaltotheorganisation.

• He/sheisresponsibleforthethematicin-depthlearning.• He/sherespondstolearnerrequestsregardingspecificstothecourse.• He/she assesses the importance and the quality of feedback relative to potential innovation or

changeswithin thebusinessprocess.He/she relates to themodeller and themandant regardingdecisionimpactingonthemodel.

2.4EvaluationsystemThe evaluation approach adopted, in coherence with the methodological choices, has two primaryobjectives:

a. Evaluate the learning process by measuring competences before and after the training andevaluatetheimprovementinworkerperformance.ThelearningassessmentismadebyutilisingthedefinitionsofKPIsetswhichgroupdatarelativetothequalityofworkerperformance,correlatedtostrategic,organisationalandoperativegoals.Inthismannerthelogicbehindtheworkprocesses,afundamentalelementinthelearningprocess,isretained.Such validation should be done over a long period, and extended to the organization (not onlyindividual)level.

b. Evaluatetheefficiencyof theplatformusing indicators thatmonitor the levelsandqualityof thelearner/platforminteraction.Atthesametimetheefficiencyofthedidacticmethodologyadoptedcanbeevaluated.The KPI dashboard collects and provides information relative to the intensity of platform usage(greateror lesserapplicationof the toolspresenton theplatformsuchas simulation, theuseofmaterials and the recommender etc.) and therefore contributes to a better understanding, alsoindirectly,oftheeffectivenessofsomemethodologies inrespecttoothers,and inrelationtothetypeofuser.

TheevaluationsystembasedonKPImodelisdescribedindetailinDeliverable5.4.

9

3LearnPadPlatform:finalvalidationAccording to Deliverable 8.1 addendum and considering the latest suggestions recommended by thereviewers,theLearnPAdconsortiumcarefullyconsidereddifferentstrategiesfortheassessmentofprojectresultsandforgetting indicationsthatmayhelptheanalysisof impactforpotentialadopters.Duringtheearly validation we aimed to collect as much feedback as possible to properly address the prototypingactivityofthefinalversionoftheLearnPAdplatform.Inthefinalvalidationwecollectedamoredetailedamountofdataandcommentstobeevaluated.This chapter describes the final validation we carried out on the Learn PAd platform at the end of theproject.TheassessmentoftheLearnPAdplatformhasbeendoneontheversiondeployedonatest-bedsettled in the cloud datacenter of Marche Region in Ancona, and available at the address http://LearnPAd.regione.marche.it:8080/xwiki/ (release 14 September 2016). In parallel a control group tried andtestedanotherregionale-learningplatform“MARLENE(http://marlenescuola.regione.marche.it/moodle/)tobeable tocompare theachievements, in termsofmeasurable learninggoals,and theopinionsof thelearnersaboutthetwodifferentlearningenvironments.Despite the different ways of use, the content enjoyed by the two groups were essentially similar andrelatedtotheSUAP“TitoloUnico”workprocessesintheorganizationofaconferenceofServices.Wedecidedtoperformanassessmentaimingatexaminingtheoverallstatusof theprojectaccordingtothedifferent learningparadigms,asdescribedinthepreviouschapter,andthevariousaspectsrelatedtotheactivitiesthatanemployeehastocarryoutwithinaBPenactment.Allthedidacticandtechnologicalsolutions,releasedandadoptedcoherentlywiththeLearnPAdlearningapproach, have been evaluated considering both the users learning profit results (also analyzing theanswerstheygivegroupedbyclustersofenablingfunctionalities),andtheiropinionsontheusabilityoftheproposedfunctionalities.WefocusedonaspecificsetofmodelsrelatedtotheSUAPscenario,whichhasbeenimplementedintermsofcontentsandlearningmaterials,accordingtothespecificfunctionalitiesofeachplatform,aswebetterdescribe later. In particular, the platform final validation involved a quantitative assessment and thecollectionandanalysisofdataandfeedbackviaquestionnaires.Thechapterisorganizedasfollows.Insection3.1weresumetheresultsobtainedfromtheearlyvalidationat M18. In Section 3.2 we describe how and for what scenario we conducted the validation. We alsointroducethemainfeaturesofthetwoplatformsandpresentthequestionsasked inthequestionnaires,togetherwiththeratiooftheirprovision. InSection3.3wepresentinfactsandfiguresthecharacteristicdatarepresentingthesamplesandtheresultsobtainedfromthecollectedanswers.TheninSection3.4wereport a summary derived from all the verbose feedback we collected from the users (which will bereportedinappendix).

3.1ResultsfromearlyvalidationThemainresultsobtainedfromtheearlyvalidationaresummarizedinthetablebelow,whichrepresentstheaverageopinionsofthememberswhoparticipatedinthe2015tests,answeringonspecificquestionsrelated to theperceived characteristics of theplatform (usability, effectiveness, accessibility, innovation,etc.). The majority of these questions was proposed again in the final validation, in order to set acomparative meter with reference to the possible evolutions found on the current Learn PAd platform

10

functionalities.Sothesevalues,evenifcollectedbyonly7testersofanearlierversionoftheplatform(asof6 november 2015) that hadbeen assistedbyplatformexperts,will be also considered and compared insection3.3.Criterions Averageratingduringtheearlyevaluation

phase[from1=lowto5=high]Overalldegreeofsatisfactionintheuseofthise-learningsolution 3,14Degreeofeffectivenessofthevariousfunctionalitiesavailablewithintheplatform

3,32

Degreeofusability,easeofuseandaccessibility 2,42Degreeofcompletenessoftheavailablefeatures 3,42Degreeofintegrationandhomogeneityofallthemodulesavailablewithinthelearningenvironment

3,42

Degreeofinnovationoftheproposedsolution 4,33Degreeofusefulnessandappropriatenessoftheplatformwithrespecttothetrainingneedsoflearners

4,00

Detectedneedofagreaterpresenceofawizard-drivencontentfruition

3,71

Detectedneedofagreaterpresenceofgamificationfeatures 4,20Detectedneedofagreaterpresenceofanappropriateuseofmultimedia

4,71

Detectedneedofagreaterpresenceofinteractivitybetweenlearnerandcontent

4,50

Detectedneedofagreaterpresenceofmultilingualismorexplanatorytechnicalvocabularies

4,28

Detectedneedofagreaterpresenceoftutoringandexternalteachingsupport

4,00

Table2Usabilityresultsfromtheearlyvalidation

3.2ValidationintopracticeInthissectionwedescribeindetailhowwepracticallyrunthevalidationtestsoftheLearnPAdplatform,incombination with a matching control group using MARLENE e-learning platform. We report the usedmethodology andwe describe the validation scenario supported by the final instance of the Learn PAdplatform(aswellasbythemoodletrainingmoduleproposedwithinMARLENE).Finally,weintroducetheexanteandexpostquestionnaireswedevelopedfortheevaluationpurposes(intheappendixwereporttheItalianverbatimversionofthesequestionnaires).Suchvalidationinvolvedatotalnumberof72end-usersoftheSUAPdemonstrator,whohadnoknowledgeof the learning platforms before the validation, and that have been invited to join by Marche Region,havingconfirmedtheirreadinesstoparticipateinthetests.36wereaskedtousetheLearnPAdplatform;36toowereaskedtotesttheMARLENEplatform.Eventually, we collected 61 answered questionnaires (31 from the Learn PAd users and 30 from theMARLENEusers).ThereceiveddataandfeedbackarereportedinSections3.3and3.4ofthisdeliverable.AsfortheLearnPAdplatformvalidationresults,wecananticipate(seeinthefollowingofthisdocument-Chapter4 - amoredetailedanalysis andexplanation) that theactual release isonaverageperceivedasquiteeffective,evenif,probably,itneedsahigherusagetimeinordertorecordmoresignificantlearningoutcomes and even if some further development and content customization are needed.We expect to

11

fulfillthoseevolutionarymaintenancerequests,abletotransformaprototypeintosomethingmoresimilartoamarketstablerelease,duringthenextcomingexploitationphase.

3.2.1ReferencedplatformsWe now turn to describe briefly each of the platforms used in the experiment and their mainfunctionalities.

3.2.1.1LearnPAd

Figure1TheLearnPAdplatformhomepage

Briefly, in the LearnPAdweb/wiki platform, after the log-in (using the assigned credentials, as from theusers list represented inhttps://github.com/LearnPAd/LearnPAd/blob/master/lp-collaborative-workspace/lp-cw-

component/lp-cw-component-ui/src/main/resources/LPUsers/UsersListY3Demo.xml#L42) and the selectionof thespecific case/course “ModelSet – SUAP Titolo Unico”, the end user can find the list of the managedBusinessProcesses(orsignificantpartsofthem),withasetoflinks,iconsandbuttons,mostlyto:

• browse hierarchically the single tasks or choice gateways, roles in the organizational units,documentsandotherobjectsinvolvedanddescribedbythemodels

• viewanavigablemapoftheprocessorsub-processactivities• contributeandgenerallytointeractwiththecontentandthefunctionalitiesoftheplatform.• simulatearealcase

12

Asanindicative,althoughapproximate,unitofmeasuretheusershadattheirdisposalforthevalidationahugeamountofmaterial (almost90wikipages), includingBPand sub-process tasksandothermodelingobjectstobevisited,collaborativeandsimulationactivitiestoplayandrecommendedcontenttoexplore.

Figure2Thebrowsingmode:awikipagerepresentingatask

More in detail, the main implemented functionalities of the Learn PAd platform under validation havebeen:

• Thehomepage(providingintroductiveinformationandexplanations,anonlinehow-tohandbook,someguidedtoursonfeatures,etc.)–Fig.1;

• TheBrowsingMode(whichallowstonavigateamongthetextualcontentsandtheimagesrelatedto the interconnectedmodels – BP, documents, organizational structures,… - , and in particularamongtheprocesstasks,representedwithinawikipage,thataretheknowledgesegmentsofanoverallproceduralflow)–Fig.2;

• TheCollaborativefeatures(includingthepossibilityto:(i)providecommentsineachpage,(ii)givefeedback to the modelers reporting logical errors or suggesting variations in the flow [this,indirectlyandpotentially, transforman individual learningprocess intoanorganizational learningprocess], (iii) contribute in writing extended wiki documents – that can also be analyzed andvalidatedbythequalityassessmentmechanismintermsofqualityforcorrectness,simplicity,nonambiguity,contentclarity,presentationclarityandcompleteness)–Fig.3;

13

• TheSimulator(inwhichtheusercanpracticeanexercisebasedonarealusecasehe/shechooses-inthiscontextaSUAPinstance- followingsomeexecutivesteps,providingresultsandansweringquestions.As soonas theuser correctlyperformsall the required jobs, the simulator returns anoverallscore.Thesimulationcanbeperformedinindividualmode,withrobots,orinvolvingotherphysicalend-userstoplaytheroleofdifferentactorsinvolvedasthirdpartiesintheprocess)–Fig.4;

• The Recommendation bar (which provides additional semantically contextualized contents –meaning experts to be contacted, more learning materials from the document model, real usecasessimilar–thatvariesaccordingtospecificparametersof theuserprofile, typesofwikipagetheuserslandin,andthespecificcontentthereinselectedbytheusers)–Fig.4.

Thesefunctionalitiesarebetteroutlinedinthedeliverablesthatwetakeasreferenceorbackground,and,ofcourse,havebeenimplementedintheMarcheRegiontest-bedandwillbedemonstratedduringtheY3review.Anotherfundamentalfunctionality,whichwasnotplannedtobeincludedinthefinalvalidationbutworthmention, istheKPIdashboard(“pannellodicontrollo”–Fig.5). Itrepresentsalltherelevantaspectsofalearningscorecard, in termsof learningandstrategicgoals, theirconnectiontoorganizationalgoals,KPIsforassessingtheirachievement.Itcollectsalsoinputvaluesthatresultfromtheactionsthatusersmakeonthecollaborativeworkspaceandthesimulationenvironmentoftheplatform(eg.n°ofvaluablefeedback,totaltimeofuse,globalscorepersimulationanduser,globalactionsperusers,etc.).Someoftheseinputvaluesneedtoreachacertainthreshold,otherwisetherecommendersystemprovidessomeadvicesbothtothelearnersandtotheirorganizationalunits, inordertoimprovebadperformancescores.AllthesefunctionalitiesarebetterdescribedinDeliverable5.4.Forthe learningassessmentandthefinalevaluationoftheplatformeffectiveness,wepreferredtouseaplatform-independent tool, structuring specific questionnaires, also in order to compare morehomogeneousresultsgotbytheLearnPAdtestandthecontrolgroup’sone.TheuseoftheKPIDashboardfor evaluation purposes makes a greater sense in the context of a real training in the workplace pre-organized set, where the aspects of motivation of the users and PA commitment exist, and theorganizationalgoalshadbeendefinedandtunedaccuratelywiththePAmanagement.

14

Figure3Thevalidatedoutputofacontributionbythequalityassessmentsystem

15

Figure4Afterthesimulationisactivated,therecommenderbarproposescontextualizedsimilarcases

16

Figure5TheLearnPAdKPIDashboard

3.2.1.2MARLENE

Figure6TheMARLENEinterfacetoaccessaSCORMcourse

The MARLENE (MARche LEarning NEtwork) platform is the regional web e-learning system, adopted,followingasignedagreementwiththeTuscanyRegion,forthereuseoftheentiresystemcalledTRIOandits available learning objects, with the goal of maximizing the public investments dedicated to thedevelopmentofanyonlinetrainingactivityforcivilservantsofthelocalPAs.

17

Thesystemisoperationalandofferstoitsusersawiderangeoflearningresources,basedonAICC/SCORMandMoodlestandards.Among the available courses, the module “SF0015-MD-2015-W” (La conferenza di servizi: quadronormativo di riferimento in ambito nazionale e regionale) provides a lesson about the regulatoryframework,atnationaland regional level,on theconferenceof services.Thismodulehasbeen takenasreferencefortheexperimentationofthecontrolgroup,asitprovidesmanycontentsrelatedtothesamescenarioimplementedwithintheLearnPAdSUAPdemonstrator,evenifatamoretheoreticallevel.Asanindicative, although approximate, unit of measure the users had at their disposal for the validation 28pagesofcontent.Withregardstothefunctionalities,theplatformprovidesamanualandsomeonlinehelp.Whenthecoursebegins,awebpop-upwindowsappearstoguidethelearnersequentiallythroughthecontents,offeredinthe form of slides, accompanied by timed and animated texts, images and audio support (Fig. 7). Someutilitybuttonsallowtheusertohandlethesubtitle,topausetheaudiospeech,tomovebetweenthenextorpreviousslides,toaccesstheglossaryandthereferences,toconsultagraphicalhelponline,toprintthecontentsofthecurrentpage.

Figure7ApageofcontentfromtheMARLENEcourseaboutSUAPCoS

18

The slides alternate with stages of verification (by multiple choice questions) exclusively for learningreinforcement, and, at the end of the fruition, a final assessment questionnaire is proposed to test theoverall knowledgegained.Thecourse is successfullypassedansweringcorrectlyat least70%of the finalproposedquestions.AMARLENE back-end is also available to assess the state of completion of the course by each learnerengaged,returninginformation(takenintoaccounttovalidatesomeofthequestionnairesanswersandtoassess the final conclusions)on thedatesof registrationandcompletion, scorepoints, total timeofuse,numberofaccesses,statusand%ofcompletion.Ifneeded,wewillbeabletoshowpracticallytheoperationofthissystemduringtheY3review,asitcanbereached through the Internet at the address http://marlenescuola.regione.marche.it/moodle/ by loggingwiththeusercredentialsownedbytheofficersofMarcheRegion.

3.2.2ValidationscenarioTheLearnPAdprojectdemonstratorsfocusontwodifferentworkplacecontexts.The first demonstrator, referred to the European Project Budget Reporting (EPBR) scenario, engages aPublicAdministration(PA),partnerinafundedproject,whichhavetomanagesomeinternaladministrativedocumentationandfinancialproceduresinordertoreportdataandinformationtotheEUCommunity;inthiscasethemodelstrytomapaBusinessProcess(BP)whichdoesnotcrosstheborderofaPA.Theseconddemonstrator,referredtothe“SportelloUnicoAttivitàProduttive”(SUAP)scenario,representsa more complex inter-organizational scenario, involving many Pas engaged in providing a unique andsharedformalanswer(orauthorization)toentrepreneurswhowanttostartabusinessactivityortobuildproductiveorcommercialpremises.Bothof themweredevelopedon thebasisof several interviewswithcivil servantsworking in thePublicAdministrationandinvolvedinthecases.For both scenarios, themodels we produced are those resulting from theWP3 (Organizationalmodels,Business Process models, Documents and Knowledge models Competence models, Business Motivationmodels,KPIs,etc.).Inparticular,weconsideredtheorganizationalstructureofeachoftheinvolvedPAs.In some cases (mainly due both to reasons of synthesis, readability, complexity decrease and spaceavailability)we focusedonlyonpartof those structures, considering theofficesand theemployees thatmainly impactontheanalyzedscenario,exercisingtheirrole.Ahugeeffortwasdonewithrespecttothecompetencesofemployees,sincetheyarerelevantintermsoflearning.ThesewereelaboratedaccordingtothewellknowEuropeanstandardcompetenciesframework(EQF).It was originally planned to carry out a final validation with users on both SUAP and EPBR scenario.However,weinsteadconcentratedalleffortsforfinalvalidationonlyonSUAP.OnereasonwasthattheY2review report requested to collect a sufficient number of users in the final validation (30 + 30) to getstatistically significant results and the number of EPBR users would be quite far from such target.Moreover,weconsideredriskytodividetheconsortiumeffortsandattentionontwodemonstratorsintherelativelylimitedtimeavailable,andpreferredtofocusallattentioninonestudy.For these reasonswe focusedon thecaseof SUAP,and inparticularon theConferenceof Services sub-processforthe“Titolounico”process,tobeabletoinvolvethelargestnumberofavailableusersaroundacommunityofpracticeclearlyidentified.

19

Inthefollowing,weintroduceandexplainbrieflytheconsideredSUAPscenario,chosenforthevalidation.ThescenarioisalreadydetailedandexplainedinDeliberable8.1:wejustwouldliketoprovideanoverview,withina self-containeddocument, tobetterhighlight someof thechoiceswemade in thesettingof thevalidationstudy.The SUAP office can be considered as an intermediary between entrepreneurs and local governmentbodies.Thismeansthat, ifanentrepreneurwantstostartanewbusinessactivityormakeaconstructioninterventiononafactory,he/sheonlyneedstocontacttheSUAPofficers.Thelatterareinchargeof:

• collectingandformallyverifyingtheapplicationandanysupportingdocumentation(eg.featuresoftheapplicant,possessionoravailabilityofthepropertytitles,etc.);

• acquiringanyintermediateanswersfromalltheotherauthoritiesoroffices,involvedonthebasisoftheirindividualresponsibilities(eg.firesafety,buildingpermissions,protectionoflandscape,protectionofarchaeologicalsitesandculturalheritage,environmentalpermits,foodhygiene,sewagedischarges,connectiontothemunicipalutilities,etc.);

• issuinganofficialfinaldecisionthatisenforceableagainstthirdparties.A SUAP office can be organized as an office of theMunicipality, or as a district office - managed by aconsortiumof PAs, by an aggregatedadministrativeentityorby a locally superordinatepublic body (eg.mountain unions) - that provides services on behalf of several Municipalities. This distinction wasconsidered in terms ofOrganizationalModel:we assigned the user credentials distinguishingwhether auser belonged - or simulated to belong - to a "district desk" or a "municipal desk"; this parameter hasenabled,forexample,tovary,dependingontheuser,thehintoftheappropriatereferencedexpertsontherecommenderbar.So,alltheusersreallyworkingforamunicipalSUAP,orforathirdpartyadministration,weregivena“municipaldesk”account;alltheusersreallyworkingforadistrictualSUAP,orforanotherPA,indirectly involved in the SUAP processes for coordination, training and support, were given a “districtdesk”account.

Figure8TheSUAP“TitoloUnico”BusinessProcess

The “TitoloUnico”application, also called standard request - Fig. 8 -, isoneof theprocesses runby theSUAP offices (typically the most ordinary one). The legislation introduced other peculiar types ofprocedures, such as the “SCIA” or the “Urban Variant”. Here, for the validation purposes, we have not

20

includedthose,eventhoughalongtheprojecttheyhavebeenthesubjectofintenseanalysisandmodelingactivities.

Figure9The“ConferenceofServicesmanagement”sub-processwithinthe“TitoloUnico”BP

One of the sub-processes specified under the “Titolo Unico” is the management of the Conference ofServices(CoS).TheCoSisameeting,activatedwhenrequiredbylaw,inordertodrawacollectivedecisionasquicklyaspossible; typically it is required when is necessary to obtain an agreement, a clarification, an act ofconcertationorassentbyoneormorePAsinvolved.Thiscanhappenoncomplexdossiers,orwhensomePAs show a delayed response time if compared to the standard timing of completion of the overallprocedure, or in any casewhere there is no consensus among all the public actors involved (and even,sometimes,thesameprivateapplicant).Inparticular,thesimultaneousCoS,mappedinthesub-processinFig. 9, is a meeting where all the involved participants (i.e SUAP offices, other municipal offices, othergovernment units and possibly the applicant) discuss on the case, from all points of view, together andconcurrently,anddecidewhether therequestcouldbeeligibleornot.According to theprocedural flow,theSUAPoffice:(i)announcestheCoSandconvenestheadministrationsinvolved-sendingalltheencloseddocumentation, (ii) manages the agenda and the works of the meeting, (iii) collects the opinionssummarizing them in a report, and then (iv) takes the final decision (which can be positive, negative,conditionedtothefulfillmentoffurtherrequirementsbytheentrepreneur,etc.).When using the Learn PAd platform, the learnerswere asked to play the role of a SUAP representativecontact person. In some phases of the simulation, they had also the option to play the role of a thirdadministrationreferee,calledtoexpressitsopinioninaCoS.

21

Anewregulation,changingconsistentlymanycircumstancesandaspectsofprocedural flowof theSUAPCoS,hasbeenrecentlyintroducedbytheDecree“DLgsn.12730/06/2016”.UnfortunatelythishappenedatalatestageofthedeploymentofthecontentsontheLearnPAdplatform;similarlywedidnothavethetime toadapt theCoSmoodle courseon theMARLENEplatform. For validationpurposeswedecided toleave, in both platforms, the outdated content, ie responsive to the previous legislation dictates.Neverthelesswe reactedoffering theusers someadditional learningmaterial, asbetterdescribed in thenextsection.

3.2.3UsedmethodologyAt thebeginningofAugustwe invitedviaemailmore than90publicmanagers,officers, employers, civilservants or even freelance professionals, involved in the provisioning of theOSS/SUAP services towardsentrepreneurs,tojointheLearnPAdfinalvalidationactivities.TheywerepeopleworkingforSUAPofficesor for local government bodies called to give an opinion on OSS practices or even other publicadministrationsindirectlyinvolvedintheOSSprocessesforcoordination,training,technicalandregulatorysupport,etc.We explained them about the Learn PAd project - and its aim to develop a prototype platform for thetrainingofPublicEmployeesbasedonworkprocesses-andweaskedfortheiravailabilityandwillingnessto participate in this trial, carried out by theMarche Region ICT Department, in collaboration with theRegional SchoolofPublicAdministration,aswell as for theirhelp in findingotherpeople toengage.Weinformedthemaboutestimatedtiming,modalitiesandrequiredeffort for thevalidationexperiment.Wetold them that we would have assessed by questionnaires, aggregating their answers, the degree ofteachingeffectivenessofthenewWikimodel-basedLearnPAdplatform,accessibleviaawebbrowser24hourson24duringthetrialtimewindow,alsoincomparisonwithmoretraditionale-learninginstruments.We also specified that the contents found in the platform, related to the SUAP,were to be consideredpurelyasameanstotestitsvalidity.Forthisreasonwewouldnothaverequiredanypreviousorextensivepreparationonthesubject,meaningthatevenaninexperiencednovice,aslongasinsider,couldattendthetesttoseeiftheplatformhadprovedausefullearningtool.Wereceivedback72candidaturesfrompeopleavailabletojointheexperimentation.InSeptemberwecommunicatedtheofficialstartofthetrial,from15/09till26/09,sendingtheusersafirstexantequestionnairetoassess,briefly,their:

• personaldetails,relevantcharacteristicsandprofileinformation,neededtoclassifyandaggregatethesamples

• aself-assessmentofthelevelofexpertise–from1to8–theybelievedtohavewithrespectto5mainEQFqualificationsrecommendedtoworkintheSUAParea

Beforethetestsbegan,wegatheredthecompletedquestionnaires.We divided the candidates in two groups (the main one of Learn PAd users and the control one ofMARLENEusers),ashomogeneousaspossible,consideringtheirexperienceandexpertiseinOSS/SUAP,onthebasisofourknowledgeandunderstanding.Asaresult,eachofthetwogroupsresultedcomposedof:almosta50%ofnoviceusers(withlessthan1yearofworkpractice),a25%ofintermediateprofessionalsanda25%ofexpertusers(withmorethan5yearsofworkpractice).

22

Thenweprovidedtheuserswithall the technicaldetails tostart (webaddress: respectivelyhttp://LearnPAd.regione.marche.it:8080/xwiki and http://marlenescuola.regione.marche.it/moodle/, credentials,operativeinstructionsandmanuals,telephoneandemailcontacttorequestsupport).Wealsoaskedthemto keep track of the real daily time spent in the learning activities, as thiswould also be the subject ofspecificfollow-upquestions.For the Learn PAd sample,we created and assigned to users individual accounts to distinguish thembyprofiles(novice,intermediateprofessional,expert)andtypeofSUAP(districtualormunicipal)accordingtothe ex ante information provided. Those profiles were needed to customize the user experience withreferencetothecontentproposedbytherecommendationsystem.Thisdistinctionhasnotbeennecessaryfortheusersofthecontrolgroup,whoaccessedbyauthenticatingthemselveswiththelocal“FedCohesion/SPID”identificationsystem.AssaidbeforetheDecree“DLgsn.12730/06/2016”introducedsignificantchangesinmanycircumstancesand aspects of procedural flow of the SUAP conference of services. However, for integration andrectification,inthetrialstartingemailwealsoattachedthreedocuments:(i)thenewlegislationitself,(ii)abrief guide to the changes introduced, (iii) an extensive interpretative guidance of the new regulatoryframework.Thesesupplementarydocumentsweredraftedwiththree levelsofdetail, respectively:(i) forexperienced staff, (ii) for personnelwith intermediate knowledge (iii) for staffwith basic skills. Both theMARLENE and Learn PAd userswere advised to consult the informationmaterial consistentlywith theirknowledge;moreover, in the LearnPAd case, thedocumentswerealso referencedas learningmaterialsthat the recommender would have returned on the basis of the appropriate user profile (if expert,intermediateornovice).ParticularlyforLearnPAd,sinceit isaprototype,somesectionsoftheplatformcouldhaveresulted,toafirst approach, not very easy to touse andnot immediately understandable. For this reason, during thetests, two informationmeetings (on 16 and21 September at theMarcheRegion) hadbeenprovided toillustrate, with practical demonstrations, the platformmain features. They were attended by almost 10users.Attheendofthetrialperiodwesenttheusersanexposttestingquestionnairetoassesstheeffectivenessofthetrainingplatformandofthepeculiarteachingmodule.We informedthemthat the informationanddataretrievedwouldhavebeenhandledbythepartnersoftheProject LearnPAd soas toensure,under the Italian “Privacy code (Dlgs196/2003)”, its security andconfidentiality. So we therefore asked them to sign a waiver of consent to the processing of theinformationanddataprovided,whendeliveringthecompletedquestionnaire.Finallywecollectedtheexpostquestionnaires(31fromtheLearnPAdsampleand30fromtheMARLENEsample). The tests results havebeenprocessed anonymously andon an aggregative basis, and analyzedcomparingthetwosamples.Theresultsarereportedinsection3.3and3.4.TheconclusionsarereportedinChapter4.

3.2.4QuestionnairesAsalreadyexplainedwedevelopedandsubmittedtothelearnerstwokindofquestionnaires.

23

3.2.4.1TheexantequestionnaireTheexantequestionnaire(fullyreportedintheappendix)askedtheuserssomepersonaldetails,inorder(i) to be able to classify and aggregate the samples by relevant characteristics, (ii) to give each user anappropriate profile with respect to the parameters of experience (expert, intermediate, novice) andorganization of their SUAP offices (districtual or municipal), and (iii) to verify a properly homogeneousdistribution of the users between the Learn PAd group and the MARLENE one. We collected usefulinformation regardingGender, Age,Geographical distributionwithinMarche region,Qualification, Publicrole,Yearsofworkat (oronbehalfof)aPA,Yearsofexperience inOSS/SUAP,Typeofpublicbody theyserve.Moreover,theexantequestionnaire,submittedbeforethestartingoftheplatformtestingactivities,aimedalso to get from each user a self-assessment of the level of expertisewith respect to 5main EuropeanQualificationsFrameworkskillsweanalyzeasrecommendedtoworkintheSUAParea.TherecommendedEQFskillswere:

• Front-officeactivities,information,communicationandmanagementoftheexternalrelationswiththepublic[neededforthetasksrelatedtothepreparationofworkandtoprovideinformationtotheSUAPdeskexternalusers]

• Assessing the administrative and procedural regularity of a request, through checking itscompletenessandformalcorrectness[neededforthetasksrelatedtotheexecutionofaBPandtochecktheadmissibilityofaSUAPapplication]

• Managementandcoordinationofspecificadministrativeprocedures[neededforthetasksrelatedto theexecutionofaBPand i.e. to senddocumentation tootherPAs, toask for integrations, toactivatetheCoS,tosolicitthethirdpartiesincaseofdelayornon-compliance,etc.]

• Verifying the congruency and pertinence of data and documentation submitted in an instance,consideringthemeritswithrespecttotheactivitythattheuserintendstostartortheinterventiontobeundertaken[neededforthetasksrelatedtodeepcheckingandmonitoringandnotonlytoaccomplisha finalcheckof thecorrectnessof thewholeacquireddocumentation,butalso tobeabletoissueanexpertopinionontheadmissibilityofaSUAPauthorization]

• Drawingupformaldocuments(Decrees,reports,letters,etc.)duringtheexecutionorattheendofanadministrativeprocedure [needed for the tasks related to the finalauditand releaseofSUAPformalactsandtobeabletowriteandorganizeadministrativetextsinapropermanner]

Weaskedlearnerstoattributethemselvesalevel,from1(low)to8(high),asspecifiedinthetablebelow,derivedfromtheEuropeanQualificationsFramework(EQF).

24

EQFlevelKnowledge

[theoreticaland/orfactual]

Skills

[cognitive(involvingtheuseof

logical,intuitiveandcreative

thinking),andpractical

(involvingmanualdexterityand

theuseofmethods,materials,

toolsandinstruments)]

Competence

[responsibilityandautonomy]

Level1 Basicgeneralknowledge Basicskillsrequiredtocarryoutsimpletasks

Workorstudyunderdirectsupervisioninastructuredcontext

Level2 Basicfactualknowledgeofafieldofworkorstudy

Basiccognitiveandpracticalskillsrequiredtouserelevantinformationinordertocarryouttasksandtosolveroutineproblemsusingsimplerulesandtools

Workorstudyundersupervisionwithsomeautonomy

Level3 Knowledgeoffacts,principles,processesandgeneralconcepts,inafieldofworkorstudy

Arangeofcognitiveandpracticalskillsrequiredtoaccomplishtasksandsolveproblemsbyselectingandapplyingbasicmethods,tools,materialsandinformation

Takeresponsibilityforcompletionoftasksinworkorstudy;adaptownbehaviourtocircumstancesinsolvingproblems

Level4 Factualandtheoreticalknowledgeinbroadcontextswithinafieldofworkorstudy

Arangeofcognitiveandpracticalskillsrequiredtogeneratesolutionstospecificproblemsinafieldofworkorstudy

Exerciseself-managementwithintheguidelinesofworkorstudycontextsthatareusuallypredictable,butaresubjecttochange;supervisetheroutineworkofothers,takingsomeresponsibilityfortheevaluationandimprovementofworkorstudyactivities

Level5 Comprehensive,specialised,factualandtheoreticalknowledgewithinafieldofworkorstudyandanawarenessoftheboundariesofthatknowledge

Acomprehensiverangeofcognitiveandpracticalskillsrequiredtodevelopcreativesolutionstoabstractproblems

Exercisemanagementandsupervisionincontextsofworkorstudyactivitieswherethereisunpredictablechange;reviewanddevelopperformanceofselfandothers

25

Level6 Advancedknowledgeofafieldofworkorstudy,involvingacriticalunderstandingoftheoriesandprinciples

Advancedskills,demonstratingmasteryandinnovation,requiredtosolvecomplexandunpredictableproblemsinaspecialisedfieldofworkorstudy

Managecomplextechnicalorprofessionalactivitiesorprojects,takingresponsibilityfordecision-makinginunpredictableworkorstudycontexts;takeresponsibilityformanagingprofessionaldevelopmentofindividualsandgroups

Level7 Highlyspecialisedknowledge,someofwhichisattheforefrontofknowledgeinafieldofworkorstudy,asthebasisfororiginalthinkingand/orresearch

Criticalawarenessofknowledgeissuesinafieldandattheinterfacebetweendifferentfields

Specialisedproblem-solvingskillsrequiredinresearchand/orinnovationinordertodevelopnewknowledgeandproceduresandtointegrateknowledgefromdifferentfields

Manageandtransformworkorstudycontextsthatarecomplex,unpredictableandrequirenewstrategicapproaches;takeresponsibilityforcontributingtoprofessionalknowledgeandpracticeand/orforreviewingthestrategicperformanceofteams

Level8 Knowledgeatthemostadvancedfrontierofafieldofworkorstudyandattheinterfacebetweenfields

Themostadvancedandspecialisedskillsandtechniques,includingsynthesisandevaluation,requiredtosolvecriticalproblemsinresearchand/orinnovationandtoextendandredefineexistingknowledgeorprofessionalpractice

Demonstratesubstantialauthority,innovation,autonomy,scholarlyandprofessionalintegrityandsustainedcommitmenttothedevelopmentofnewideasorprocessesattheforefrontofworkorstudycontextsincludingresearch

Table3DescriptorsdefininglevelsintheEuropeanQualificationsFramework(EQF)

3.2.4.2TheexpostquestionnairesOntheotherside,theexpostquestionnaire(fullyreportedintheappendix)askedeachlearnertorepeatthesameEQFself-assessmentattheendofthetrainingactivities,inordertoverifywhethertheuseoftheplatformhadincreased-andtowhatextent-hisknowledgeandhisskills.We also asked the user howmuch time he/she devoted to the use of the platform, in terms of overallperiod(indays)andintermsofactualtime(inhours).At the same time, we asked the users to answer 15multiple choice questions related to specific SUAPcontent. Eachquestionanswered correctlywasworth1point andonlyoneof thepossibleanswerswascorrect.Multiple choiceswere generally 5 for each question, except for the questions: 1 , 2, 9, 10 (thepossiblechoiceswere3)andforthequestions:5,14(thepossiblechoiceswere6).The15questionscouldbegroupedinto4homogeneousclusters,bytypeofskills/knowledgeassessed:• group A (theoretical questions: n° 4, 5, 6 and 7) aimed at verifying a successful learning of content

deliveredthroughdidascalicfunctions.

26

• group B (questions on the new CoS regulatory reform introduced by the Decree “DLgs n. 12730/06/2016”: n° 3, 9 and 10) aimed at verifying a successful learning of content delivered bydownloadingsupplementarymaterial.Inotherwords,wewantedtotestiftheusershadbeenabletoread the documentation attached to an email, or downloaded as an additional learningmaterial, toanalyzeitandtolearninamoretraditionalmode(notfullybrokeredbytheplatform).

• group C (questions related to BPs: n° 1, 2, 8, 11 and 12) aimed at verifying a successful learning ofcontent delivered through Business ProcessModeling and the logic of the procedural flows. It wasrequiredthecapabilityoftracingamentalpathandmovingbetweenthedifferenttaskswhichmakeuptheCoSprocess,incompliancewiththeassociatedrulesandtiming.

• groupD(questionsaboutpracticalcases:n°13,14and15)aimedatverifyingasuccessfullearningofcontent delivered through simulations or hands-on practice on real and concrete use cases. It wasrequiredthecapabilityofapplyingthetheorytoaparticularconcretecasetakenfromtherealworld,synthesizingandevaluatingthemainelements,andstrivingtosolveit.

Finally, we asked the users to give their opinion, mostly expressing a numerical value to the elementssubmitted to their judgment [on a Likert rating scale from 1 = low to 5 = high], about usability, userexperience,platformeffectivenessandothertopics.Indetail,theaspectsbroughttotheirattentionwere:

1. Overalldegreeofsatisfactionintheuseofthise-learningsolution2. Degreeofeffectivenessofthevariousfunctionalitiesavailablewithintheplatform3. Degreeofusability,easeofuseandaccessibility

Weinformedtheuserthat"Accessibility",inourcontext,meansthecharacteristicofacomputersystemtobeeasilyusablebyanyuser,regardlessofhissensory,motororpsychicindividualcapacities,ofanytemporaryorpermanentdisability,ofevenculturalaspectsoraspectsrelatingtofamiliaritywiththeuseofICTinterfaces.

4. Perceiveduserexperiencewithregardsto:a. graphicsandcolorsb. theunderstandingofsymbolsandiconsc. themeaningoftextualcontentsd. searchandsuggestionofcontenttheywanttofinde. orientationandguidanceinbrowsingthepagesorthecontentstructuref. thestructuring,placementandpositioningofthecontent

Weinformedtheuserthat“UXoruserExperience”,inourcontext,meansthesubjectivefeelingsthathe/sheexperiencedusingtheplatformwithreferencetotheempiricandaffectiveaspects,tothevalueandmeaningattributionhe/sherelatedtotheavailabilityandtheinteractionwiththesolution,aswellastotheoverallperceptionsandpersonalthoughtshe/shehadaboutitslearningeffectiveness,theeaseofuseoftheinterface,thesystem'sefficiency,etc..

5. Degreeofcompletenessoftheavailablefeatures6. Degreeofintegrationandhomogeneityofallthemodulesavailablewithinthelearning

environment7. Degreeofinnovationandnoveltyoftheproposedsolution8. Degreeofusefulnessandappropriatenessoftheplatformwithrespecttothetrainingneedsof

learners9. Detectedneedofagreaterpresenceof:

a. awizard-drivencontentfruitionb. gamificationfeatures

27

Weinformedtheuserthat"Gamification",inourcontext,meanstheuseofelementsborrowedfromthegamesandotherfuntechniques,adaptedtoexternalcontexts(eg.scores,levels,rewards,badges,giftsandotherloyaltymethodsorpracticestoarouseinterestorcontributetotheusersatisfaction)

c. anappropriateuseofmultimediad. interactivitybetweenlearnerandcontente. interactivityandknowledgeexchangebetweenplatformusersf. multilingualismorexplanatorytechnicalvocabularies,glossaries,dictionaryofacronyms,

bibliographicalreferences,etc.g. tutoringandexternalteachingsupport

Theobtainedresults,bothfromcontentandusabilityquestions,oftheLearnPAdgrouphavebeencomparedwiththecontrolgroup.Moreoverwe compared the Learn PAdusability results to the feedback received in the early evaluationphase(exceptfortheelements:4-fromatof-and9e,thatwerenotpreviouslycollected).Thequestionsn°1,2and5requiredalsoafree-textcomment(insynthesis,respectively:1-motivateyouropinion;2-whichfeaturesaretobeconsideredmostuseful;5-whatfeaturesaremissingorinadequate,and why, to consider the platform a valuable learning tool). Those feedback have been analyzed andsummarizedinsection3.4,alsoseparatingpositiveandnegativenotes.We present all the data and figures received from the ex ante and ex post questionnaires in the nextsection.

3.3ResultsfromquestionnairesIn this section the results obtained from the answers to the ex ante and ex post questionnaires arerepresentedintables(andalsographically,forthekeyaspectstohighlight).

3.3.1ProfilingandcontextdataWe beginwith representing the demographical data and the context information of the Learn PAd andMARLENEsamplegroups:

Sample: LearnPAd MARLENEN°users Invitedtotesting:36

Questionnairescollected:31Invitedtotesting:36Questionnairescollected:30

Gender Males:17Females:14

Males:18Females:12

Age Mean:43,61Compositionbyagegroups:2under30years,9from30to40,14from40to50,6over50

Mean:44,40Compositionbyagegroups:2under30years,6from30to40,16from40to50,6over50

GeographicaldistributionwithinMarcheregion

18intheprovinceofAncona,4intheprovinceofAscoliPicenoandFermo,4intheprovinceofMacerata,5intheprovinceofPesaroeUrbino

15intheprovinceofAncona,2intheprovinceofAscoliPicenoandFermo,7intheprovinceofMacerata,6intheprovinceofPesaroeUrbino

Qualification Universitygraduates:23Schoolgraduates:8

Universitygraduates:23Schoolgraduates:7

28

Publicrole 0executivemanagers,12officers,12employees,7freelancersservingaPA

1executivemanager,17officers,8employees,4freelancersservingaPA

Yearsofworkat(oronbehalfof)aPA

Mean:14,81CompositionbyPAexperiencedclusters:7upto5years,4from6to10,5from11to15,8from16to20,7morethan20

Mean:13,27CompositionbyPAexperiencedclusters:6upto5years,4from6to10,10from11to15,5from16to20,5morethan20

YearsofexperienceinOSS/SUAP

Mean:2,35CompositionbySUAPexperiencedclusters:20novices(upto1years),7intermediateprofessionals(from2to5),4experts(morethan5)

Mean:3,87CompositionbySUAPexperiencedclusters:15novices(upto1years),7intermediateprofessionals(from2to5),8experts(morethan5)

Typeofpublicbodyserved

1workinadistrictualSUAP,3inamunicipalSUAP,15inathirdpartyadministration,12inanotherPA(indirectlyinvolvedintheSUAPprocessesforcoordination,training,support)

7workinadistrictualSUAP,2inamunicipalSUAP,18inathirdpartyadministration,3inanotherPA(indirectlyinvolvedintheSUAPprocessesforcoordination,training,support)

Table4Aggregateddatarelatedtothetwosamplesinvolvedinthetesting

Figure10a/bTheLearnPAdandMARLENEsamplesbygender

29

Figure11a/bTheLearnPAdandMARLENEsamplesbyage

Figure12a/bTheLearnPAdandMARLENEsamplesbyqualificationineducation

30

Figure13a/bTheLearnPAdandMARLENEsamplesbypublicrole

Figure14a/bTheLearnPAdandMARLENEsamplesbyyearsofworkat(oronbehalfof)aPA

31

Figure15a/bTheLearnPAdandMARLENEsamplesbyyearsofexperienceinOSS/SUAP

As for theoverall timespenton theplatforms, LearnPAdusage time totalized169hours in90days (onaverageperuser:5:45hoursin3days);MARLENEusagetimetotalized96hoursin64days(onaverageperuser:3:20hoursin2days).

Figure16Theplatformsoverallusagetime(inhours)

3.3.2Competenceself-assessmentdataWithin the Learn PAd sample, 14 users out of 31 (45%) said they had improvedoneormore EQF skills;regardingMARLENE,13usersoutof30(43%)saidthis.

32

Figure17ThedeclaredimprovementinEQFskillswithinthetwosamples

Thetotalperceivedimprovementsineachcompetence-calculatedasapercentageofthesumoftheexpostincreasesonthesumoftheexantevaluesdeclared-amountedtothefollowingvalues:

33

EQFSkill LearnPAd MARLENE

%ofusersperceivedimprovementonaverage

standarddeviationamongthesample

%ofusersperceivedimprovementonaverage

standarddeviationamongthesample

Front-officeactivities,information,communicationandmanagementoftheexternalrelationswiththepublic

9% 0,57 2% 0,25

Assessingtheadministrativeandproceduralregularityofarequest,throughcheckingitscompletenessandformalcorrectness

9% 0,63 5% 0,37

Managementandcoordinationofspecificadministrativeprocedures

15% 0,70 6% 0,50

Verifyingthecongruencyandpertinenceofdataanddocumentationsubmittedinaninstance,consideringthemeritswithrespecttotheactivitythattheuserintendstostartortheinterventiontobeundertaken

12% 0,79 4% 0,34

Drawingupformaldocuments(Decrees,reports,letters,etc.)duringtheexecutionorattheendofanadministrativeprocedure

6% 0,47 6% 0,48

Table5PerceivedimprovementsinEQFskills

Figure18ThepercentageofdeclaredimprovementineachrelevantEQFskillwithinthetwosamples

34

3.3.3LearningassessmentdataSummarizingthelearningoutcomesforthecorrectanswerstothemultiplechoicequestions:In the Learn PAd sample, 15 users out of 31 (48% within the sample / 25% of the whole users) havecorrectly answeredmore than 70% of questions (that is theminimum threshold set by the PA TrainingRegional School to assess the successful passing of a course). The sample obtained a 62% of the totalachievablepoints.IntheMarlenesample,16usersoutof30(53%withinthesample/26%ofthewholeusers)havecorrectlyansweredmorethan70%ofquestions.Thesampleobtaineda61%ofthetotalachievablepoints.

Figure19Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestions

withinthetwosamples

We can also go deeply in the analysis separating the 15 questions into the 4 homogeneous clusterspreviouslyexplained:

35

Clustertype

N°ofinvolvedquestions

LearnPAd MarleneN°Userswith>70%correctanswers(outof31)

Userspercentageamongthesample

%oftotalpointsachievedcomparedtothemaximumachievable

N°Userswith>70%correctanswers(outof30)

Userspercentageamongthesample

%oftotalpointsachievedcomparedtothemaximumachievable

Atheoreticalquestions

4 9 29% 52% 12 40% 48%

Bquestionsonanewregulatoryreform

3 18 58% 76% 20 67% 80%

CquestionsrelatedtoBPs

5 12 39% 61% 11 37% 62%

Dquestionsaboutpracticalcases

3 9 29% 59% 10 33% 56%

Resultsonthewholesetofquestions

15 15 48% 62% 16 53% 61%

Table6Profitresultsforeachclusterofquestions

36

Figure20Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithinthesamplesandperclustersofquestions

37

3.3.4UsabilitydataAsfortheUsabilitycriterionswecalculatedtheseresults:Criteria LearnPAd

earlierplatformv.6/11/2015

LearnPAdfinalplatformv.14/09/2016

MARLENEplatform

Averagerating[from1=lowto5=high]

Averagerating[from1=lowto5=high]

N°userswhoexpressedanopinionaboveaverage[over3]

%userswhoexpressedanopinionaboveaverage[over3]

Averagerating[from1=lowto5=high]

N°userswhoexpressedanopinionaboveaverage[over3]

%userswhoexpressedanopinionaboveaverage[over3]

Overalldegreeofsatisfactionintheuseofthise-learningsolution

3,14 2,97 8 26% 3,07 10 33%

Degreeofeffectivenessofthevariousfunctionalitiesavailablewithintheplatform

3,32 2,94 10 32% 3,13 11 37%

Degreeofusability,easeofuseandaccessibility

2,42 2,90 12 39% 3,33 13 43%

perceivedpositiveuserexperiencewithregardstographicsandcolors

N.A. 3,03 8 26% 3,50 18 60%

perceivedpositiveuserexperiencewithregardstotheunderstandingofsymbolsandicons

N.A. 2,84 7 23% 3,43 15 50%

perceivedpositiveuserexperiencewithregardstothemeaningoftextualcontents

N.A. 3,10 10 32% 3,60 17 57%

perceivedpositiveuserexperiencewithregardstosearchandsuggestionofcontenttheywanttofind

N.A. 2,68 5 16% 3,07 11 37%

perceivedpositiveuserexperiencewithregardstoorientationandguidanceinbrowsingthepagesorthecontentstructure

N.A. 2,65 8 26% 3,57 18 60%

perceivedpositiveuserexperiencewithregardstothestructuring,placementandpositioningofthecontent

N.A. 2,97 11 35% 3,40 16 53%

Degreeofcompletenessoftheavailablefeatures

3,42 3,06 10 32% 3,17 9 30%

38

Degreeofintegrationandhomogeneityofallthemodulesavailablewithinthelearningenvironment

3,42 3,16 9 29% 3,07 11 37%

Degreeofinnovationoftheproposedsolution

4,33 3,61 19 61% 3,07 12 40%

Degreeofusefulnessandappropriatenessoftheplatformwithrespecttothetrainingneedsoflearners

4,00 3,06 11 35% 2,80 5 17%

Detectedneedofagreaterpresenceofawizard-drivencontentfruition

3,71 3,19 9 29% 3,27 9 30%

Detectedneedofagreaterpresenceofgamificationfeatures

4,20 2,90 9 29% 3,33 8 27%

Detectedneedofagreaterpresenceofanappropriateuseofmultimedia

4,71 3,45 6 19% 3,73 2 7%

Detectedneedofagreaterpresenceofinteractivitybetweenlearnerandcontent

4,50 3,35 6 19% 3,67 2 7%

Detectedneedofagreaterpresenceofinteractivityandknowledgeexchangebetweenplatformusers

N.A. 3,42 4 13% 3,77 5 17%

Detectedneedofagreaterpresenceofmultilingualismorexplanatorytechnicalvocabularies

4,28 3,35 8 26% 2,73 13 43%

Detectedneedofagreaterpresenceoftutoringandexternalteachingsupport

4,00 3,23 5 16% 3,27 8 27%

Table7Dataandfiguresrelatedtotheusabilityexpostanswers

39

Figure21Acomparingoverviewonacquiredopinionsaboutusability,userexperienceandplatform

effectivenessbytheusersofLearnPAdandMARLENE

3.4SynthesisofthecollectedfeedbackInthissectionasyntheticsummaryofthecollectedverbosefeedbackisdescribed.Thefull lists(inItalianandinEnglish)arereportedintheappendix.

3.4.1Positiveoverallopinionsandplatformsstrengths

Figure22a/bThePOSITIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE

40

RegardingtheLearnPAdplatform,wereceived20among31positivefeedbackrelatedtotheBPbrowsingfunctionality, said to be a simple, logical, intuitive, fast and flexible way to navigate, by content or bygraphic schemes, through the workflow; the processes themselves and their related content weredescribedasaccurate,wellstructured,clearlyexplainedandeasytounderstand.10usersalsoappreciatedthesimulationmodeanditsexercisesbasedonrealcases.4usershavepraisedthesocial interactionfeaturesbasedoncomments,thechatandtheotherformsofcontribution;4more,inparticular,evidencedthefunctionalitiesfordrawingupandaddingcontributivelystructuredtextsthroughthewikienvironment.2learnersnoticedtheadvantagesofthedynamicrecommendationofnewcontentbasedonsemantics.Finally2usershavespokenwellofthehomogeneousandresponsivevisualidentityand1userspokeofanewerandinterestingwaytolearn.About MARLENE 13 users among 30 liked the intermediate comprehension tests via multiple choicequestions.6userssucceededinupdatingtheirknowledgeandinquicklymemorizingSUAPnotions,thankstothewelldescribedandsyntheticcontentandtosomein-depthssections.5userstalkedaboutaneasytousetool,withaclearinterfaceandasimplesequentialflowthatlinksallthepages.3 users judged valid the fact that the content were presented through timed animation, highlights andaudio.Finally2usersrecommendede-learningandonlinewebcontentasawaytosavetimeandmoney.

3.4.2Negativeoverallopinionsandmissingfunctionalities

Figure23a/bTheNEGATIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE

41

Speaking about Learn PAd, 19 users among 31 have complained about an initial feeling of disconcert,uncertainty,displacement,unclarity. They reported that theplatformwasdifficult toexplore since therewasnotapredefinedorientingpathtofolloworexplainingtheprogression.Thereweretoomanyunknownavailablelinksandfeatures,vehicolatedthroughnonexhaustiveintroductionsandthroughatrickyandnotso intuitive interface. One also remarked that just learning to use the platform itself is hardly timeconsuming.10learnerswrotethatsimulationdidnotseemtoworkwellorwasnotwellexplainedoreventhatitwasshortlyeffectiveinstimulatingrealproblemsolvingattitudes4usersreportedthatthegraphicalschemesrepresentingtheflowchartsweredifficulttouseandread3usersnoticedthepresenceofdisturbingtypos(eg.e&apos)andenglishwords3 users suggested to strengthen and make more attractive the social collaboration and interactioncapabilities2 learners believe that the Learn PAd platform is not suitable for beginners as the course and even theprofitquestionnairerequiredasaprerequisiteapreviouslyacquiredknowledge.2peopleaskedtointroducetheuseofvideo,multimediaandanimationsinordertofacilitatelearning.2 users commented that the recommending bar showed poor content and it could have been moreeffective.AnisolatedopinionreportedthatcontentaboutCoSwasoutdated.

ConcerningMarlene, 11 users among 30 suggested to introduce, as more advanced e-learning systemsprovide,interactiveandsocialfunctionalities,togaintheopportunityofhavingchat,submittingquestionson regulatory concerns and complex situation to handle, receiving in-depth opinions and legislationinterpretationfromexpertsorothercolleaguesattendingthecourseorexperts.7 learners described the course as very basic and theoretical, suitable for beginners, lacking insights oneverydaypracticalwork,realapplicationsandcomplexsituationtohandle.4userscomplainedaboutaninsufficientaccessibilitybecauseofthepop-upblockerandthebrowserorI/Odevicesconfigurationchangestobeputinplace.Other4usersreportedthelackofacontentsearchsystemtogodirectlytothecontentofinterest.ResidualsingleopinionsnoticedoutdatedcontentforCoS,thehighamountoftimeneededtocompleteallthetrainingmodule,theabsenceofgamificationaspectsandofvideosasresourcesforlearning.

42

4ResultsanalysisandconclusionsInsummary,thevalidationdatacollectedareofdifferenttype.Precisely,foreachrespondent,theyinclude:

X. Aself-assessmentevaluationonowncompetenceconcerningthe5expectedEQFcapabilities,collectedexanteandexpost;

Y. 15answerstomultiple-choiceprofitquestionsrelatedtolearningcontents(withintheSUAPcontextasavalidationscenario);

Z. Qualitativeanswers,onaLikertscale,on9questions(andfor3ofthembyanaccompanyingcommentary),assessingusabilityandlikingoftheplatform.

Assaid,alltheabovedatahavebeenidenticallycollectedfrombothusersofLearnPAdandofMarlene,asabaseline.

Thequestionsweaddressedintheexperimentforvalidationcanbestatedasfollows:

VQ1:HowdoeslearningthroughtheuseofLearnPAdapproachcomparesagainstlearningthroughtheuseofamoretraditionale-Learningplatform(suchasMarlene),intermsoflearningeffectiveness?

VQ2:HowdousersassesstheusabilityandthefunctionalityoftheLearnPAdplatform?

Theconsiderationsrelatedtodatacollectedbynon-statisticallysignificantsamplesshouldbetakenwiththenecessaryprecautions.Howeverwetriedtosummarizesomegeneralconclusions.

ToaddressVQ1weexamineboththequalitativeself-assessmentbyusers(answerstogroupXquestions)andtheanswerstoquestionstestingusersacquiredknowledgeonthesubjectoflearningexperiment(groupYquestions).Thefirstgroupofanswerssupportsasubjectivemeasureoflearningeffectivenesswhereasthesecondgroupsupportsamoreobjective-eventhoughlimited-measure.Themeasurementoflearningeffectivenesshasbeenassessedalsoinrelationtothelearnersprofile(suchastheyearsofworkingexperienceinaPA).

ToaddressVQ2weexaminetheanswerstothegroupZquestions(boththeresultingratingsandthefree-textcomments,aggregatedforrelevancyandsimilarity).

RESULTSOFGROUPY(profitanswers)

We start the analysis considering answers to group Y. Overall we can see (Table 8) that the providedanswersarebetterfortheMarlenegroupthathaveahighermedianvalueandlowerstandarddeviation,althoughthedifferencesaresmall.Wethenperformedastatisticaltesttoassessthenullhypothesisthatthedifferencebetweenthenumberofcorrectanswersforthetwogroupsfollowsasymmetricdistributionaroundzero,i.e.,thenullhypothesisisthatthemedianvaluesarestatisticallyequivalent.Becauseourdatacouldnotbeassumedtobenormallydistributed,weadoptedanon-parametricstatisticalhypothesistest,theWilcoxonsigned-ranktest.Witharesultingp-valueof0.5604,thenullhypothesiscouldnotberejected,i.e.,themedianvaluesobservedarestatisticallyequivalent.

43

Marlene LearnPAdMean 9,93 9,23Median 10 9St.Dev. 3,38 3,5

Table8Overallstatisticsforprofitanswersresults(themeanofthetotalscoreacquiredandthemedianscore)

Werepeatedthetestforthe4clustersofquestionsdescribedinSection3.2.4.2(Atheoreticalquestions,Bquestionsonanewregulatoryreform,CquestionsrelatedtoBPs,Dquestionsaboutpracticalcases)andtheresultsarethesame:themediansarestatisticallyequivalent.

Itmeansthatthelearningassessmentareequivalentbetweenthetwosamples,andwecansaythatitisagood result for theLearnPAdplatform,as it isaprototypewith respect to themore testedandutilizedMarleneplatform.

We further examined these results looking for possible correlation with some of the factors that wecollected. We hypothesized that demographic characteristics of the learners, such as age or maturedexperienceinthetopicofthelearningsession(SUAP),mightimpactthelearningresults.Wealsotestedthehypothesis that the time spenton the learningplatform (eitherMarleneor LearnPAd) is not correlatedwithsuchresults.

Weperformed,forasetofcomparisonvariablesconsideredsignificant,boththePearsoncorrelationtestandtheKendalltaucorrelationtest.KendalltauissimilartothemorecommonlyusedPearsoncoefficientbutitdoesnotrequirethevariablestobenormallydistributed.TheresultsarereportedinTable9and10.

Marlene LearnPAd

YearsofPAexperience

0,360 0,577

YearsofSUAPexperience

0,467 0,563

Hoursdedicatedtotesting

0,189 0,096

Table9Pearsoncorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(PAexperience,SUAPexperience,anddedicatedtime)

With the Pearson index we noticed a no significant correlation of the individual learning results withrespect to theplatformtimeusageandamoderatecorrelation,moreevident for theLearnPAdsample,with respect to theyearsofworkingexperiencewithinaPAand regardingSUAP.This isplausibleand itcouldindicatethat,astheLearnPAdapproachiscenteredonworkprocesses,learningiseasierandmoreimmediateforthosewhoarethelongestmorefamiliarwiththeseprocesses.

44

Marlene LearnPAd

Ageoflearners

0.2827474 0.403308

YearsofSUAPexperience

0.5207782 0.469423

Hoursdedicatedtotesting

0.1636847 0.15134

Table10Kendalltaucorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(age,SUAPexperience,anddedicatedtime)

ForinterpretingthedatadisplayedinTable10(whereallentriesarestatisticallysignificantata95%level),weuse theGuildford scale1 [1], inwhich correlationswith absolute value less than0.4 are described as“low”,0.4to0.7as“moderate”,0.7to0.9as“high”,andover0.9as“veryhigh”.

Weobservedsomeinterestingandconvincingresults,asfollows.

Theageof respondentshasa lowcorrelationwith thenumberof correctanswersboth forMarleneandLearnPAd.Thetimespentontheplatformdoesnothaveastatisticalcorrelationwiththeamountofcorrectanswers.Thisisinlinewiththecurrentstate-of-the-art.Even if time had no relevance on the progression of learning, we found that Learn PAd, to be tested,requiredagreatereffortinhoursthanMarlene(abouttwiceofwhatusersspentonMarlene).LearnPAdsample users provided a higher level of commitment during the tests,maybe also because of the hugeamount of content that the platform contains (about three times the amount that is contained in theMarlenecourse).Weacknowledgethattwoweeksoftrialhadbeentoofew,especiallyfortheLearnPAdplatform,whoseusageneedstobeextendedovertimebeforeitcanfulfillallitspotentialstrengths–suchasgeneratingarealcommunityofpractice-andbeforeitisgrantedthepossibilitytowalkthrough,studyandunderstandallthetasksofBP.However,fororganizationalreasons,thefinalvalidationcouldnothavebeenstructuredotherwise.TheexperienceonSUAPhasamoderatecorrelationwiththenumberofcorrectanswersbothforMarleneandLearnPAd,andrepresentsthefactorthatisbettercorrelatedwiththenumberofcorrectanswers.Basedonsuchoutcome,andnoticingthatwithintheLearnPAdsamplealargerpartofsubjectsdeclaredtobe novices in terms of expertise about SUAP, wemade a further analysis by computing the correlationbetweenSUAPand thenumberof correct answersafter removing thoseuserswhodeclared less than1years of SUAP experience. The correlation coefficient improved for both Marlene (from 0.5207782 to0.6958166)andLearnPAd(from0.469423to0.5957597).Suchresultsseemtosuggestthat,independently

1J.P.Guilford.FundamentalStatisticsinPsychologyandEducation.McGraw-Hill,1942

45

fromtheplatformused,learningeffectivenessisbetterwhensomepreviousknowledgeexists,whichmightbeexplainedbytheextremecomplexityofSUAPregulations.

Asforthefinalprofitresultsoflearning,only48%ofLearnPAdusershasexceededthethresholdof70%ofcorrect answers (which defines the minimum acceptable level). Similarly, the percentage of MARLENEsufficient results stoodat53%ofusers.Given that the LearnPAdgroup includedahigherproportionoflearnerswithupto1yearsofSUAPexperience(20outof31)thanMarlene(15outof30),inpracticewecanalsoaffirmthatourassessmentisslightlybiasedinfavorofMarlene.

Althoughthefinallearningoutcome,initself,isnotsatisfactory,wenoticedthatitisacross-platformeffect(independentoftheplatformused).WhatitcanbesaidinfavorofLearnPAd-unlikeMARLENE-itisthatLearnPAd is amorecomplexandmulti-facetedplatform,which surelyneedsmoreassimilation time todeploybenefits(especiallythoseofthecollaborativelearning).

According to theMarcheRegionPATrainingSchool, the learning resultsofonlinemethodsaregenerallylowerthanthoseofthetrainingbasedoncoursesinpresence(andalsothedrop-outrateismuchhigher).ThiscanbeduetoagenerallackofhumanizationandpersonalrelationshipsineLearningpractices,butasforthefinalvalidationalsoalackofmotivation-fromthelearnersside-andtheabsenceofamandantoranagreedtrainingplan-fromtheorganizationsinvolved-couldhaveinfluenced.

Finally,wealsoperformedaKendall taucorrelationanalysisbetween theexperiencewithSUAPand theindividual’s self-assessment (ex post EQF skills values declared).We observed a negative correlation forboth platforms with correlation coefficients of -0.336554 and -0.05058633 for Marlene and Learn PAd,respectively.Thisresultisconsistentwiththeexpectation:individualswhoarelessexperiencedwithSUAPwill probably notice a greater improvement after using the platform, than those who are alreadyexperienced.

RESULTSOFGROUPX(EQFself-assessment)

WenowdiscusstheresultsfromgroupXanswers.Asimilarnumberofsubjects(45%ofLearnPAdusersand43%ofMarleneusers)assessedthattheyhaveacquiredsomecompetence,althoughthisnumberisnothigh.

Table5(andFigure18)inSection3.3.2showthat,attheendofthetest,theLearnPAdusers,morethantheMARLENEones, said theyhadachieved improvements in their competences.However the figuresofthe standard deviation shows that, within the MARLENE sample, there has been a more equitabledistributionofthe improvementsbetweenusers(andso,thatthehighest improvementattributedtotheuseofLearnPAdmaybeduetothepresenceofsomeoutliers).Checkingdata,wefoundasingleLearnPAduserself-assessingimprovementinhislevelsof2,3,3,4and2pointsforthefiveEQFSkills(thatcouldbeexaggerated,orperhapsitcouldmeanthantheuserunderestimatedhisskillsintheexantequestionnaire).ThepercentageofimprovementsvariesacrossthefivedifferentEQFcompetences,andwethinkitmaybeusefultolookattheresultssingularlyastheyaddresseddifferenttypeofcompetences.An interesting thing tonotice is that thehigher incrementalpercentage for the LearnPAd sample is the15% on the EQF Skill “Management and coordination of specific administrative procedures”, which,notably, is themostused skillwhenworking, for instance,with the coordinationof the taskswithin the“TitoloUnico”BP.And,bytheway, theBPbrowsingmodehasbeenconsideredthebest functionalityof

46

theLearnPAdPlatform,asassessedbythepositiveverbal feedbackprovidedbythevastmajorityoftheusers(64%).Also the better improvements in “Assessing the administrative and procedural regularity of a request,throughcheckingitscompletenessandformalcorrectness”(9%amongtheLearnPAdsampleinsteadofthe5%withinMarlene)andin“Verifyingthecongruencyandpertinenceofdataanddocumentationsubmittedin an instance, considering themerits with respect to the activity that the user intends to start or theintervention to be undertaken” (12% among the Learn PAd sample instead of the 4% withinMarlene),couldbe related to the LearnPAdcasebasedactivity,performed in simulationmode,whereuserswereaskedtoreadasubmittedapplicationformtocheckitanddeterminehowtoproceed.

Marlene LearnPAd

exantedeclaredlevelinEQFcompetences

0,463 0,454

expostdeclaredlevelinEQFcompetences

0,565 0,413

Table11PearsoncorrelationbetweenthenumberofindividualcorrectanswersandthesumofdeclaredlevelsinthefiveEQFcompetences

With the Pearson indexwe also noticed aweak correlation, but present in both cases, of the individuallearningresultswithrespecttotheself-assessedexanteEQFlevels.ForwhatconcernsthedeclaredexpostEQF competence, it seems thatMARLENE self-assessments has a slightly higher level of coherencewithrespecttothelearningresults.RESULTSOFGROUPZ(usabilityrelatedanswers)

As a general comment, we observed that the average of ratings stands tomid-range for almost all thecriteria,andforboththeplatforms.

There isapoorcorrelationbetween theprofit total scoresobtainedbyusersand their ratingsofoverallsatisfactionontheuseoftheplatform.Andatthesametimethesejudgmentsareplacedonintermediatevaluesthatmakesuchanindicatorapoorproxy.

ForLearnPAdwerecordedvaluesbelowaverage,thoughslightly,on:overallsatisfaction;effectivenessofthevariousfunctionalities;usabilityandaccessibility;userexperience(UX) for symbols and icons; UX for search and suggestion of content; UX related to orientation andguidanceinbrowsingthepagesorthecontentstructure;UXforstructuring,placementandpositioningofthecontent.Webelievethatthenon-positivescore,mostlyabouteffectivenessoffunctionalities,usabilityand accessibility, search and suggestion of content, guidance for browsing the content structure (also

47

corroboratedby theverbalcommentsofusers)willhaveundoubtedly to inspire thenextactions for theevolutionoftheplatform.

Marlene LearnPAd

Overalldegreeofsatisfactionintheuseofthise-learningsolution

0,357 0,242

Table12Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandtheusersoverallratingoftheplatforms

About the perceived degree of usability, ease of use and accessibility, the judgment of the users isimprovedcomparedtothepreviousassessmentintheearlyevaluation.

With the exception of the need for greater presence of multilingualism or explanatory technicalvocabularies, Learn PAd presented better results than the previous assessment and than MARLENEconcerningalltheother10indicatorsmonitoringdeficientorabsentfeatures,meaningthat:

• the consortium addressedmany of the comments that emerged in the early evaluation, solvingrequestsandissues

• on average Learn PAd is considered, with a very slight margin of advantage, a completeenvironment,concerningtheneededfunctionalitiestobeeffectiveintermsoflearning.Clearlythisdoes not imply that the quality of the implemented features is good enough for users or mayrequirefurtherimprovements.

LearnPAdpresentedbetterresultsthanMARLENEonlyon:degreeof integrationandhomogeneityofallthemodules available within the learning environment; degree of innovation of the proposed solution;degreeof usefulness and appropriatenessof theplatformwith respect to the trainingneedsof learners(and this last is a good result because it considers the content of the platform and how they aretransmitted).

Unless for completeness, innovation and usefulness for training needs, MARLENE scored for the othercriterionsthehighestnumberofusersthatexpresspositivejudgments.Thisresultisnotsurprising,becauseMarleneisaplatformusedinproductionforseveralyears,whileLearnPAd,atthemoment,isjustaresearchprototypethat(althoughproducingevidencesoflearning)isheavilybasedonabusinessprocessmodel,inwhichthelocalcivilservantshavelittleornoexperienceatall.Analyzingthefree-textfeedbackprovidedbytheusers,wenoticedthatmanyofthemconsideredMarleneamoresuitabletoolforbeginnersthanLearnPAd,mainlybecauseitusesamoresimplifiedandsequentialprocess of acquiring information and knowledge. Instead Learn PAd would be more suitable forexperiencedpeople, alsobecause itmakes awideuseof features for socialization, sharing, contributionandcollaboration.

48

It is not easy to deduce clear recommendations from the verbal feedback related to the Learn PAdplatform.Forexample,simplifyingtheanalysis,10usersamong31arguedthatsimulation isaverygoodidea,but,ontheotherend,thesameamountofuserssaiditdidnotworkwellandshouldberevised;20outof31usersbelievedvaluablethebrowsingmode,but,atthesametime,other20userstoldtheydidnotfeelwell-orientedduringthenavigation.

Trying to draw some conclusions we believe that, unsurprinsingly, there is still work to do around theplatform.Theactualrelease(andthelearningapproachitrepresents)isonaverageperceivedaseffective,evenif,probably, itneedsahigherusagetimeinordertorecordmoresignificant learningoutcomesandevenifsomefurtherdevelopment(oradjustment)offeaturesandcontentcustomizationareneeded.Weexpect to fulfill those evolutionarymaintenance requests, able to transforma prototype into somethingmoresimilartoamarketstablerelease,duringthenextcomingexploitationphase.

Weprovideaquitehigheffort toupdatecontent in theplatform: fromprocessmodelingwithinasinglereferencemeta-model,tillthecollectionofeducationalmaterials,theeditingofexplanatorytextsandtheverification of the state of a constantly changing legislation, passing through the personalization of thesimulation exercises, of the recommender suggestions andof theKPIDashboard goals to be fed.At thesame time it was also challenging to maintain aligned the technologies and the functionalities of theplatform around a single and integrated application infrastructure. It is therefore logical to assume thatthosewho intendtopursue theresultofmakinguseofLearnPAdasa teaching tool,areequippedwithresourcestomaintainup-to-datetheplatform,eventechnologically,andtocustomizeitscontent(andthemanagedbusinessprocesses)accordingtosimilarneedsandrequestscomingfrommorethanonePAasaprofitable market segment, and consequently they should be subjects able to reap economic ororganizationalgainsfromtheseactivities.

49

Appendix

ExanteQuestionnaire(inItalian)Datianagraficiediprofiloutente

• NomeeCognome…………………………………………..…………………………………………..………………………………………..• Sesso(M,F)……...• Età……………………..• Sedelavorativa(Comune)……………………………………………………………………………………………………………………….• Titolodistudio………………………………………………………………………………………………………………………………………..• Ruolopubblico(Dirigente,Funzionario,Impiegato,Liberoprofessionistaoaltrafiguraalserviziodiuna

PA)……………………………………………………………………………………………………………………………………..• Annidilavoropresso(opercontodi)PA………………...............• AnnidilavorosvoltiinambitiattinentialSUAP……………………• Entediappartenenza/svolgimentoservizioetipologia(Enteterzo,SUAPcomprensoriale,SUAP

municipale,Altro)…………………………………………………………………………………….………………………………………......

Lechiediamodicompilare,primadell’avviodelleattivitàformative,un’autovalutazionerispettoallivelloeuropeo(EQF)dicompetenzeposseduteinambitoSUAP(SportelloUnicoAttivitàProduttive)ocomunqueneiprocessidilavorodiunaPubblicaAmministrazione.PerognispecificacompetenzadescrittanellaTab.1(colonnacompetenza),dovràindicareillivellodicompetenzacheritienedipossedere(1=bassa/nulla–8=alta).IdiversilivellisonomegliodescrittinellaTab.2Tab.1Competenza Livelloda1a8(rispetto

allatabellaEQFsottostante)a. Attivitàdisportello,diinformazione,dicomunicazioneedigestionedelle

relazioniconilpubblico

b. Verificadellaregolaritàamministrativaeprocedimentalediun’istanza,attraversocontrollidicompletezzaecorrettezzaformale

c. Gestioneecoordinamentodispecificheprocedureamministrative d. Verificadimeritodiun’istanzainordineallacongruenzaeallapertinenza

deidatiedelladocumentazionepresentatarispettoall’attivitàchel’utenteintendeavviareoall’interventochesiintenderealizzare

e. Stesuradidocumentiformali(atti,verbali,lettere,etc.)nell’esecuzioneoallaconclusionedeivaripassaggiprocedurali

Tab.2

Livello

EQFConoscenza Abilità Competenza

50

1 Generaledibase.

Basilarinecessariepersvolgere

compitisemplici,anchein

assenzadiesperienzapregressa.

Lavorareostudiaresottola

direttasupervisione,inun

contestostrutturato.

2Praticadibaseinunambito

lavorativoodistudio.

Cognitiveepratichedibase

necessarieperutilizzarele

informazionirilevantialfinedi

svolgerecompitierisolvere

problemidiroutineutilizzando

regoleestrumentisemplici.

Lavorareostudiaresottola

supervisioneconunacerta

autonomia.

3

Conoscenzadifatti,principi,

processieconcettigenerali,inun

ambitolavorativoodistudio.

Cognitiveepratichenecessariea

svolgerecompitierisolvere

problemiscegliendoe

applicandometodidibase,

strumenti,materialied

informazioni.

Assumersilaresponsabilitàper

ilcompletamentodelleattività

nellavoroenellostudio.

Adeguareilproprio

comportamentoalle

circostanzenelrisolvere

problemi.

4Praticaeteoricainampicontesti,

inunambitolavorativoodistudio.

Cognitiveepratichenecessariea

risolvereproblemispecificiinun

campodilavoroodistudio.

Autogestionenell'ambitodelle

lineeguidaincontestidilavoro

odistudiochesono

solitamenteprevedibili,ma

soggettiacambiamenti.

Supervisionareillavorodi

routinedialtri,assumendosi

unacertaresponsabilitàperla

valutazioneeilmiglioramento

diattivitàlavorativeodistudio.

5

Praticaeteorica,completae

specializzatainunambito

lavorativoodistudioe

consapevolezzadeiconfiniditale

conoscenza.

Unagammacompletadiabilità

cognitiveepratichenecessarie

persvilupparesoluzionicreative

aproblemiastratti.

Gestireesorvegliareattivitàin

contestidilavoroodistudio

espostiacambiamenti

imprevedibili.

Controllareesvilupparele

prestazioniproprieedialtri.

6Avanzatainunambitolavorativoo

distudio,chepresupponeuna

Avanzate,chedimostrino

padronanzaeinnovazione

Gestireattivitàtecnicheo

professionalicomplesseo

51

comprensionecriticaditeoriee

principi.

necessariearisolvereproblemi

complessiedimprevedibiliinun

ambitospecializzatodilavoroo

distudio.

progetti,assumendosila

responsabilitàperilprocesso

decisionaleincontestidilavoro

odistudioimprevedibili.

Assumersilaresponsabilitàdi

gestirelosviluppo

professionaledipersonee

gruppi.

7

Altamentespecializzata,chepuò

costituirel'avanguardiadella

conoscenzainunambito

lavorativoodistudio,comebase

delpensieroe/odiricerca

originale.Consapevolezzacritica

delleproblematichelegatealla

conoscenzainuncampoe

all'interfacciatracampidiversi.

Problemsolvingspecializzato

necessarionellaricercae/o

nell'innovazione,alfinedi

svilupparenuoveconoscenzee

procedureeperintegrare

conoscenzeprovenientida

ambitidiversi.

Gestireetrasformarecontesti

complessidilavoroodistudio

imprevedibilicherichiedono

nuoviapproccistrategici.

Assumersilaresponsabilitàdi

contribuireallaconoscenzae

allapraticaprofessionalee/odi

verificareleprestazioni

strategichedeigruppi.

8

Livelloconoscitivopiùavanzatoin

unambitolavorativoodistudioe

all'interfacciatracampi.

Tecnichepiùavanzatee

specializzate,tracuilasintesie

lavalutazione,necessarieper

risolvereproblemicomplessi

dellaricercae/odell'innovazione

eperestendereeridefinirele

conoscenzeesistentiolapratica

professionale.

Dimostrareeffettivaautorità,

innovazione,autonomiae

integritàtipicadellostudiosoe

delprofessionistaeunimpegno

continuonellosviluppodi

nuoveideeoprocessi

all'avanguardiaincontestidi

lavoroodistudio,tracuila

ricerca.

Ai sensi del Dlgs196/2003 - "Codice Privacy”, desideriamo informarla che le informazioni ed i dati personali forniti in tale contesto(sperimentazioneevalidazionedellapiattaformaLearnPAd)sarannoutilizzatiesclusivamenteperfinalitàstatisticheescientificheetrattatiinconformità al documento programmatico per la sicurezza informatica in vigore, nonché nel rispetto della normativa vigente, inmodo dagarantirnelasicurezzaelariservatezza.IrisultatideitestediquestionaricompilatisarannoanalizzatiedutilizzatidaipartnersdelProgettoLearnPAd(RegioneMarche:PFSistemiInformativieScuoladiformazioneregionale,CNRConsiglioNazionaledelleRicerche:ISTI,UniversitàdiCamerino,Universitàdell’Aquila,UniversitàFHNWSwitzerland,BOCAssetManagementGmbHAustria,NoMagicEuropeLituania,LinagoraGSOFrancia,XWIKISasFrancia),confrontaticonquelli raccoltidaglialtripartecipantial testecomunquerielaborati in formaaggregataedanonimaaifinidiunapresentazioneversoisoggettiterziabilitati(ades.funzionarievalutatoriUE).EssisarannoutilizzatipercomprendereilivellidiefficaciaraggiunticonilnuovomodelloformativosperimentatonelprogettoeuropeoLearnPAd.

Tali dati sono conservati da Regione Marche ed accessibili solo da parte di personale autorizzato fino alla conclusione del progetto; lacancellazionedeidatiavvienesurichiestaviamail,faxoletteradegliinteressati,senzaindugio.Inognicasoidatisarannocancellatialterminedell'iniziativaeaseguitodelloroutilizzoneiterminiindicati.

I soggetti cui si riferiscono idatipersonalihanno ildiritto,ai sensidell'art.7delD.Lgs.n.196/2003, inqualunquemomento,diottenere laconfermadell'esistenzaomenodeimedesimidatiediconoscerneilcontenutoel'origine,verificarnel'esattezza,chiedernel'integrazioneol'aggiornamento,oppure la rettifica, lacancellazione, la trasformazione in formaanonimao ilbloccodeidati trattati inviolazionedi legge,nonchédiopporsiinognicaso,permotivilegittimi,allorotrattamento.

52

InbasealdocumentoprogrammaticosullasicurezzadeidatipersonalidellaRegione,iltitolaredeltrattamentodeidatipersonalièlaGiuntaregionale, il responsabileè ildirigente titolaredellaPFSISTEMI INFORMATIVIETELEMATICIegli incaricati sono i responsabili regionalideiprocedimentiedeiprocessicorrelatialprogettoLearnPAd.

Percomunicazioni:

RegioneMarcheP.FSISTEMIINFORMATIVIETELEMATICIDirigenteDott.ssaSERENELLACAROTAViaTiziano4460125Ancona(AN)Fax0718063071funzione.sisinf@regione.marche.it

Ilsottoscrittoacconsentepertanto,almomentodiriconsegnareilquestionariocompilato,aconcedereunaliberatoriaperiltrattamentodelleinformazioniedeidatiraccolti,neiterminisopraindividuati.

Infede____________________________________________________

53

ExpostQuestionnaire(inItalian)

SPERIMENTAZIONEPIATTAFORMA…………………………..–QuestionariodivalidazioneexPOST

• NomeeCognome…………………………………………..…………………………………………..………………………………………..• NomeutenteLearnPAdassegnato…..…………………………………………………………………………………………………..• Quantotempohadedicatoallafruizionedellapiattaforma,interminidiarcotemporalecomplessivoin

giorni………………………………………edinterminiditempoeffettivodiutilizzoinore……………….…………………

Lechiediamodicompilare,alterminedelleattivitàformative,un’autovalutazionerispettoallivelloeuropeo(EQF)dicompetenzecheritienediaveracquisitoneiprocessidilavorodiunaPubblicaAmministrazionee/oinrelazioneaitemidelSUAP(SportelloUnicoAttivitàProduttive);ciòascopodiverifica,relativamentealfattosel’utilizzodellapiattaforma,asuoavviso,abbiaincrementato–einchemisura–lesueabilitàelasuaconoscenza.PerognispecificacompetenzadescrittanellaTab.1(colonnacompetenza),dovràindicareillivellodicompetenzacheoraritienedipossedere(1=bassa/nulla–8=alta).IdiversilivellisonomegliodescrittinellaTab.2riportatanelquestionarioexanteTab.1Competenza Livelloda1a8(rispetto

allatabellaEQFgiàpropostanelquestionarioexante)

f. Attivitàdisportello,diinformazione,dicomunicazioneedigestionedellerelazioniconilpubblico

g. Verificadellaregolaritàamministrativaeprocedimentalediun’istanza,attraversocontrollidicompletezzaecorrettezzaformale

h. Gestioneecoordinamentodispecificheprocedureamministrative i. Verificadimeritodiun’istanzainordineallacongruenzaeallapertinenza

deidatiedelladocumentazionepresentatarispettoall’attivitàchel’utenteintendeavviareoall’interventochesiintenderealizzare

j. Stesuradidocumentiformali(atti,verbali,lettere,etc.)nell’esecuzioneoallaconclusionedeivaripassaggiprocedurali

DomandearispostamultiplasuicontenutirelativiaSUAP,TitoloUnico,ConferenzadiServizi(soltantounadellepossibilirisposteècorretta–EVIDENZIARELARISPOSTASELEZIONATAOSCRIVERELALETTERAALLAFINEDELLADOMANDA)

1. nelcasoincuialSUAPpervengaunaistanzaperlarealizzazionediunanuovastrutturasanitaria,dicuiallaL.R.20/00,inqualicasipuòindirelaconferenzadiservizi?

a. Potràindirelaconferenzadiservizisolosesonodecorsi30giornidallarichiestadeiparerialleamministrazioniinteressate(AziendaSanitariaUnicaRegionaleASUR;ServizioSanitàdellaRegioneMarche,ecc.)senzachequestesisianoinalcunmodopronunciate

b. Dovràinognicasoindirelaconferenzadiservizi,inquantoènecessarioc. Nonpotràmaiindirelaconferenzadiservizi,madovràrichiedereipareriallesingole

amministrazioniinteressate(ASUR;ServizioSanitàdellaRegioneMarche,ecc.)

54

2. incasodiassenzadell’ASUR,regolarmenteconvocata,allaconferenzadiservizi,comedeveessereintesoilrelativoparere?

a. Espressofavorevolmenteb. Nondeveessereintesoinalcunmodo,mailprocedimentoamministrativovieneinterrotto

finoall’acquisizionedelpareredapartedell’ASUR,chepotràessereacquisitoanchealdifuoridellaconferenzadiservizi

c. Espressonegativamente3. Checosaèlaconferenzadeiservizisincrona?

a. MeetingtrairesponsabiliSUAPcoinvoltinelmedesimoprocedimentoamministrativob. IncontrotrailegalirappresentantidiciascunComunec. Tuttelerisposte(trannelae)d. StrumentocheriuniscenellostessoluogolePP.AA.coinvoltenelprocedimentoamm.e. Nessunadellerisposte

4. Qualèloscopodellaconferenzadeiservizi?a. Riunireiresponsabilidelleamministrazionicoinvoltenelmedesimoprocedimentob. FarincontrareilegalirappresentantidiciascunComunec. Tuttelerisposte(trannelad)d. Nessunadellerispostee. Ridurreitempieilcontemperamentodegliinteressipubblicicoinvolti

5. Qualisonolenormechesonointervenuteprevedendol’utilizzodistrumentiinformaticiperilavoridellaconferenzadiservizi?

a. Tuttelerisposteb. laLegge18giugno2009n.69c. ilD.L.5/12,convertitonellaL.35/12d. laLegge11febbraio2005n.15e. laLegge7agosto2015n.124f. ilDLgsn.127del30/06/2016

6. Qualisonoivantaggidellaconferenzadeiservizitelematica?a. Prevalentementelasemplificazionedeirapportitraisoggettipartecipantialleconferenze

servizib. Prevalentementel’azzeramentodelledifficoltàcheconsentonodipartecipareintempi

diversic. Prevalentementelatrasparenzaelatracciabilitàdellaproceduranonchélariduzionedei

tempidirispostad. Prevalentementeladematerializzazionedelladocumentazioneeladigitalizzazionedel

procedimentoe. Tuttelerisposte

7. InqualeprocedimentodicuialD.P.R.160/2010trovaapplicazionelostrumentodellaconferenzadiservizi?

a. Tuttelerisposte(trannelad)b. Nelprocedimentoordinarioc. Nelprocedimentodivalutazionedellaconformitàdeiprogettipreliminarid. Nessunadellerispostee. Nelprocedimentoinvarianteallostrumentourbanistico

8. Aisensieperglieffettidell’art.7delD.P.R.160/2010,entroquantigiornidalricevimentodell’istanzailresponsabiledelSUAPdeveverificarelacompletezzaformaledell’istanza?

55

a. 30giornib. 20giornic. 25giornid. 35giornie. 15giorni

9. IlDLgs30giugno2016n.127introduceunanuovadisciplinae5tipologiediConferenzediServiziattivabili;essesono:

a. Decisoria,Automatica,Consensuale,inVariante,Istruttoriab. Decisoria,Semplificata,Simultanea,Preliminare,Istruttoriac. Decisoria,Automatica,Parziale,Preliminare,Procedimentale

10. ChiconvocalaConferenzadiServiziDecisoriaa. L’enteterzoincasodiemissionediparerenonfavorevolealrilasciodelTitolounicob. Iltitolarerichiedente(osuointermediariodelegato)quandol’attivitàdelprivatoè

subordinataadununicoattodiassensodaadottareaconclusionediunprocedimentodicompetenzadiunaamministrazionepubblica

c. L’amministrazioneprocedente,nellapersonadelresponsabiledelprocedimento,quandoperlaconclusionedelprocedimentodevonoessereacquisitialmenoduepareri,intese,concerti,nullaostaoaltriattidiassenso,dapartedidiverseamministrazioni,inclusiigestoridibenioservizipubblici.

11. Inunaistanzadititolounico,qualora,aseguitodell’esameformaledapartedelSUAP,ladocumentazioneallegatarisultiesserenonconformeallanormativa:

a. IlSUAPrichiedeulteriorichiarimentieintegrazionialtitolaredellapraticab. Glientiterzicoinvoltiemettonoparerenegativo,ciascunoperl’endo-procedimentodi

competenza(ovveroperognisub-attività-BusinessProcesstask-incuivengonocoinvoltiechiamatiademetterepareredimerito)

c. IlSUAPdeveconvocareunaconferenzadiservizid. IlSUAPrigettal’istanzainquantoinammissibileoirricevibilee. IlTribunaleAmministrativoRegionale(TAR)deveprendereinesamed’ufficiol’istanzaela

documentazioneallegata12. Se,aseguitodiunaconferenzadiserviziodelcontrollodimeritodiun’istanza,unenteterzo

esprime,perquantodicompetenza,unparerefavorevolecondizionatoall’avviodiun’attivitàproduttiva,ciòsignificache:

a. L’enteterzohanecessitàdiulteriorichiarimentiprimadipotersiesprimeredefinitivamente,pertantoemetteunpareresospensivodeiterminidelprocedimento

b. L’attivitàproduttivadaavviaresiintendeesercitabile,previaverificadellecondizionidiidoneitàprofessionaledapartedelrichiedente

c. L’enteterzohaespressounpareresostanzialmentefavorevole,previoadempimentodapartedelrichiedentediunaopiùprescrizionivincolantiindividuate

d. L’utenterichiedenteègiudicatoincondizionediesercitarel’attivitàproduttiva,masoloinriferimentoallespecifichecompetenzeapronunciarsichel’enteterzohaincapopernorma

e. L’enteterzohaespressounparerepositivo,segnalandoall’utenteunaseriediinterventiaggiuntiviopzionalinonobbligatoridaporreinessere

13. UnutentehacorrettamentepresentatoistanzadiTitoloUnicoalSUAPdelComunediSenigallia,inordineallaristrutturazionediunostabilimentobalnearesuunaspiaggiaricadenteinarea

56

paesaggisticatutelataanormadellaparteIIIdelD.Lgs.42/2004;IlSUAPconvocaunaconferenzadiservizi:qualidiquestientiterziverrannosicuramenteinvitati?

a. MultiserviziSpAinquantoentegestoredellaretefognaria(perl’esamedelrefertoanaliticosulloscaricodiacquereflue)eARPAM(perlavalutazionediimpattoacusticoaisensidellaL.447/95edellaL.R.14novembre2001,n.28)

b. IlDipartimentodiigienealimentareenutrizionedell’AziendaSanitariaUnicaRegionale-ASUR(perilparereigienicosanitariopreventivoinordineall’esecuzionedeilavoriedili)elaPFVALUTAZIONIEDAUTORIZZAZIONIAMBIENTALIdellaRegioneMarche(perlarichiestadiautorizzazioneintegrataambientaledicuialDLgs59/05,inordineaimpianticoncapacitàproduttivapariosuperioreaivaloridisogliaindicatinell’AllegatoI)

c. L’areaEcologiadellaProvincia(perilparereinordineall’adozionedell’AUA-autorizzazioneunicaambientale–perleemissioniinatmosfera)edilDIPARTIMENTODIPREVENZIONEINCENDIdeiVigilidelFuoco(perlavalutazionedelprogettoaisensidell’art.3delDPR151/2011)

d. LaReteFerroviaItalianaSpA-exFF.SS.(perilparereinmeritoaiconfiniedistanzedallalineaferroviariaaisensiDPR753/1980)edilDIPARTIMENTODIPREVENZIONEESICUREZZANEILUOGHIDILAVOROdell’ASUR(perl’acquisizionedell’autorizzazionedicuiall’art.5delR.D.147/27inmateriadiutilizzodigastossici)

e. Gliufficicomunaliambiente(perilpareredicompatibilitàpaesaggistica)ededilizia(perl’ottenimentodelpermessodicostruire)

14. ConclusasiunaconferenzadiserviziincuisiastataconvocatalaSoprintendenzaperibeniartistici,architettonici,ambientaliestorici,affinchésiesprimaaisensidell'art.146DLgs42/2004esmieart.14-tercomma3bisL.241/1990,qualoralaSoprintendenzanonabbiaemessoparereformalenéabbiapartecipatoailavori:

a. IlSUAP,aisensidellanormativasulsilenzioassenso,intendeacquisitounparerefavorevoleb. LaSoprintendenzaha30giorniditempodall’emissionedelverbaleperintegralocon

eventualiesuccessiveosservazioni,richiestedichiarimento,prescrizionic. IlSUAPsollecitalaSoprintendenzaadesprimeredefinitivamenteilparereentroulteriori15

giornidall’emissionedelverbaled. LaSoprintendenzaha15giorniditempodall’emissionedelverbaleperconvocareuna

nuovaconferenzadiservizie. L’utenterichiedenteacquisiscedidirittol’autorizzazionearealizzarel’interventooavviare

l’attivitàd’impresaf. IlSUAPècostrettoacomunicareall’utenteildiniegoall’autorizzazione

15. Conclusasiunaconferenzadiservizirelativaadun’istanzadiTitolounicoincuihannopresoparte,oltrealSUAP,tuttiglientiterzinecessari,qualoralasoprintendenzaperibeniartistici,architettonici,ambientaliestoriciabbiaemessoparerefavorevoleaisensidell'art.146DLgs42/2004esmieart.14-tercomma3bisL.241/1990,l’ufficiopatrimoniodelcomuneabbiaaccordatoilNULLAOSTAall’interventoinqualitàdiEnteproprietariodell'area,l’ufficioambientedelcomuneabbiaevidenziatolacompatibilitàdell’interventoalivellopaesaggisticoaisensidell'art.146DLgs42/2004el’ufficiocomunaleediliziaabbiaevidenziatolanonammissibilitàdell'interventoaseguitodellacontestualerichiestadelpermessodicostruire:

a. AlterminediprocedimentoilSUAPemetteràtitolounicoaprocedereall’interventob. Ilconsigliocomunalesaràchiamatoapronunciarsisulprocedimentodivarianteurbanisticac. AlterminediprocedimentoilSUAPcomunicheràalrichiedenteildiniego(oilpreavvisoal

diniego)perl'effettuazionedell'intervento

57

d. L’ufficiocomunaleediliziasaràinvitatoariesaminareilpropriopronunciamentolegatoalpermessodicostruireentroilterminedi15giorni

e. Alterminediprocedimentol’utentepotràpresentarealSUAPlaSegnalazioneCertificatadiInizioAttività

Domandesull’usabilitàdellapiattaforma(EVIDENZIARELARISPOSTASELEZIONATAOAGGIUNGEREUNAXNELLARELATIVACASELLA)

1. Ingenerale,quantoèsoddisfattodellafruizioneinmodalitàe-learningacuihaavutoaccesso?(Scaladiriferimento1=insoddisfatto;5=totalmentesoddisfatto)

1 2 3 4 5

Puòspiegarciinsintesiperché……………………………………………………………………………………………………………………….

……………………………………………………………………………………………………………………………………………………………………….

2. Comegiudicanelcomplessol’efficaciadellediversefunzionalitàresedisponibilidallapiattaforma?(Scaladiriferimento:1=pernienteefficaci;5=moltoefficaci)

1 2 3 4 5

Inparticolarequalifunzionalitàgiudicapiùutili:ades.esercizidisimulazionebasatisucasireali,raccomandazionedinamicadicontenuti,navigazionesequenzialedelflussodilavoro,interazionesocialconcommentiealtreformedicontribuzione,elementidiapprofondimentoevidenziatiattraversograficaomultimedia,momentidiverificaintermedicondomandearispostamultipla,attribuzionedipunteggifinali,etc.......................................................................................................................................................................

……………………………………………………………………………………………………………………………………………………………………….

……………………………………………………………………………………………………………………………………………………………………….

3. Comegiudica“facilitàd’uso”ed“accessibilità”deiserviziresidisponibili?Per“accessibilità”siintendelacaratteristicadiunsistemainformaticodiesserefacilmentefruibiledaqualsiasitipologiadiutente,indipendentementedallecapacitàsensoriali,motorieopsichichedell’individuo,daeventualidisabilitàtemporaneeostabili,oanchedaaspetticulturaliorelativialladimestichezzaconl’usodelleinterfacceedelletecnologieICT.(Scaladiriferimento1=difficile;5=facile)

1 2 3 4 5

4. Interminidi“usabilità”ed“esperienzautente”comevalutaiseguentiaspetti?Per“esperienzautente”(UserExperience-UX)siintendelasensazionesoggettivacheaveteprovatoutilizzandolapiattaformaconriferimentoagliaspettiesperienziali,affettivi,diattribuzionedisensoevalorecollegatialla

58

disponibilitàdelprodottoeall’interazioneconessononchéallepercezionicomplessiveeaipensieripersonalicircal’efficacianell’apprendimento,lasemplicitàdiutilizzodell’interfaccia,l’efficienzadelsistema.(Scaladiriferimento:1=esperienzadeltuttonegativa;5=esperienzadeltuttopositiva)

Graficaecolori 1 2 3 4 5

Comprensionedeisimboli 1 2 3 4 5

Significatodeitesti 1 2 3 4 5

Ricercaesuggerimentodeicontenutidesiderati 1 2 3 4 5

Orientamentonellanavigazione 1 2 3 4 5

Strutturaeposizionamentodeicontenuti 1 2 3 4 5

Altro(specificareeaggiungererighesenecessario)

…………………………………………………………………………………………………………………………………………..

1 2 3 4 5

…………………………………………………………………………………………………………………………………………..

1 2 3 4 5

5. Comegiudicala“completezza”dellefunzionalitàresedisponibili?(Scaladiriferimento1=incomplete;5=complete)

1 2 3 4 5

Inparticolarequalifunzionalitàmancherebberooasuogiudizioreputacarenti(eperché)perpoterconsiderarelapiattaformaunvalidostrumentodiapprendimento………………………………………………………………

……………………………………………………………………………………………………………………………………………………………………….

……………………………………………………………………………………………………………………………………………………………………….

6. Quantovalutaefficacel’integrazionedegliambientididatticipresenti?Inaltreparoleleèsembratodioperareinuncontestodiapprendimentosufficientementeomogeneo,rispettoatuttiimoduli,icontenutielefunzionalitàesplorati?(Scaladiriferimento1=pernienteintegrati;5=moltointegrati)

1 2 3 4 5

7. Comevalutailgradodiinnovativitàdellasoluzioneproposta?Leèsembratodiapprendereinunamodalitànuovarispettoadesperienzedidattichegiàavute?(Scaladiriferimento1=pernienteinnovativo;5=moltoinnovativo)

59

1 2 3 4 5

8. Comevalutal’utilitàdellasoluzioneproposta?Lareputaadeguatarispettoallesueesigenzeformative?(Scaladiriferimento1=pernienteutile;5=moltoutile)

1 2 3 4 5

9. Aifinidiunamiglioreefficaciadellapiattaformainterminidiapprendimentoritieninecessariaunamaggiorepresenzadeiseguentiaspetti?(Scaladiriferimento:1=pernientenecessario;5=deltuttonecessario)

Fruizioneguidatadeicontenuti 1 2 3 4 5

“Gamification”ovverol'utilizzodielementimutuatidaigiochiedelletecnicheludicheriadattateacontestiesterni(ades.punteggi,livelli,ricompense,distintivi,doniealtrimetodidifidelizzazioneoperdestareinteresseocontribuireallasoddisfazionedegliutenti)

1 2 3 4 5

Multimedialità(usoappropriatodiaudio,video,immagini,…) 1 2 3 4 5

Interattivitàtradiscenteecontenuti 1 2 3 4 5

Interattivitàescambiodiconoscenzatrautentidellapiattaforma 1 2 3 4 5

Multilinguismoocomunqueancheusodididascalieingradodispiegareilsignificatodiacronimi,vocabolitecnicidaaddettiailavori,etc.

1 2 3 4 5

Tutoring(animazioneperidiscenti,supportoalladidattica) 1 2 3 4 5

Altriaspettinecessari(specificareeaggiungererighesedelcaso)

…………………………………………………………………………………………………………………………………………..

1 2 3 4 5

…………………………………………………………………………………………………………………………………………..

1 2 3 4 5

Ai sensi del Dlgs196/2003 - "Codice Privacy”, desideriamo informarla che le informazioni ed i dati personali forniti in tale contesto(sperimentazioneevalidazionedellapiattaformaLearnPAd)sarannoutilizzatiesclusivamenteperfinalitàstatisticheescientificheetrattatiinconformità al documento programmatico per la sicurezza informatica in vigore, nonché nel rispetto della normativa vigente, inmodo dagarantirnelasicurezzaelariservatezza.IrisultatideitestediquestionaricompilatisarannoanalizzatiedutilizzatidaipartnersdelProgettoLearnPAd(RegioneMarche:PFSistemiInformativieScuoladiformazioneregionale,CNRConsiglioNazionaledelleRicerche:ISTI,UniversitàdiCamerino,Universitàdell’Aquila,UniversitàFHNWSwitzerland,BOCAssetManagementGmbHAustria,NoMagicEuropeLituania,LinagoraGSOFrancia,XWIKISasFrancia),confrontaticonquelli raccoltidaglialtripartecipantial testecomunquerielaborati in formaaggregataedanonimaaifinidiunapresentazioneversoisoggettiterziabilitati(ades.funzionarievalutatoriUE).EssisarannoutilizzatipercomprendereilivellidiefficaciaraggiunticonilnuovomodelloformativosperimentatonelprogettoeuropeoLearnPAd.

Tali dati sono conservati da Regione Marche ed accessibili solo da parte di personale autorizzato fino alla conclusione del progetto; lacancellazionedeidatiavvienesurichiestaviamail,faxoletteradegliinteressati,senzaindugio.Inognicasoidatisarannocancellatialterminedell'iniziativaeaseguitodelloroutilizzoneiterminiindicati.

I soggetti cui si riferiscono idatipersonalihanno ildiritto,ai sensidell'art.7delD.Lgs.n.196/2003, inqualunquemomento,diottenere laconfermadell'esistenzaomenodeimedesimidatiediconoscerneilcontenutoel'origine,verificarnel'esattezza,chiedernel'integrazioneo

60

l'aggiornamento,oppure la rettifica, lacancellazione, la trasformazione in formaanonimao ilbloccodeidati trattati inviolazionedi legge,nonchédiopporsiinognicaso,permotivilegittimi,allorotrattamento.

InbasealdocumentoprogrammaticosullasicurezzadeidatipersonalidellaRegione,iltitolaredeltrattamentodeidatipersonalièlaGiuntaregionale, il responsabileè ildirigente titolaredellaPFSISTEMI INFORMATIVIETELEMATICIegli incaricati sono i responsabili regionalideiprocedimentiedeiprocessicorrelatialprogettoLearnPAd.

Percomunicazioni:

RegioneMarcheP.FSISTEMIINFORMATIVIETELEMATICIDirigenteDott.ssaSERENELLACAROTAViaTiziano4460125Ancona(AN)Fax0718063071funzione.sisinf@regione.marche.it

Ilsottoscrittoacconsentepertanto,almomentodiriconsegnareilquestionariocompilato,aconcedereunaliberatoriaperiltrattamentodelleinformazioniedeidatiraccolti,neiterminisopraindividuati.

Infede____________________________________________________

61

Collectedfeedback(inItalian)LearnPAdNegativeoverallopinionsandmissingfunctionalities• L’impattoinizialeconlapiattaformasconcertaegeneraincertezze–nonessendociunpercorso

predefinitodaseguire,maunaseriedifunzionalitàattivabiliapiaceredicuiinizialmentenonseneconosconoutilitàocomportamento,l’utentesitrovaspiazzato->siconsigliadipotenziaregliaspettiintroduttivi(es.testoesplicativoiniziale,videotutorial,etc.).

• Perquantohoavutomododivedere,alcunefunzionalitàsembranononfunzionarebene(es.barraraccomandazioni:nonvedoalcunicontenuticheimieicolleghiinvecehannosperimentato)–altrenonsonobenspiegate(es.simulationnonèchiara,néneicontenuti,nénellapresentazionedelleazionidafare)

• Difficoltàanavigareall’internodellapiattaforma.Honotatochevisonoalcuneimprecisionisuicontenutirelativiallaconferenzadiservizi(nonsonocontemplatigliaggiornamentidicuialrecenteDlgsdiriformadellamateria)

• Pocoefficacelasimulazionecherisultaesseresemplicementeunacompilazionedicampi(quasiuntest)senzaprevederefunzionichestimolanoilproblemsolving.Inoltreipunteggifinalinonsonorapportatiadunascaladiriferimentopercuiperdonoognisignificatovalutativo

• Interfacciapocointuitiva.Difficilecapiredovecisitrova• Conlavastitàdelleinformazioniedeilinkchecisitrovadavantirisultanonfacilecapiresesisiano

esploratetuttelefunzionalità• Lapiattaformaèancorainunostatoembrionaleincuiicontenutiguidasonopocochiarielafruizione

delleinformazionirisultaosticatramitel’interfacciaproposta• Ilcorsorichiedeungradopregressodiconoscenzaecompetenzaspecificasullamateriatrattata(non

adattoaneofiti)• LAPARTEDESCRITTIVAE’COMPLETAEDIFACILECOMPRENSIONE(ADECCEZIONEDIALCUNIERRORIDI

ORTOGRAFIAES.e&apos);ICONTENUTISONOBENDESCRITTI,MAHOTROVATODIDIFFICILEFRUIBILITA’ELETTURAGLISCHEMIGRAFICI

• Lafruizionedellapiattaformanonèmoltointuitiva,bisognaprendercidimestichezzaeconciliareconcettualmenteleduemodalitàdiorganizzazionedeicontenuti(quelladiflusso,cheèmoltochiaraeavvienetramitelaschematizzazione,equelladispiegazionedeiprocessi)

• Lapiattaformanonmièsembrataparticolarmenteintuitiva,la“mappa”navigabileèinteressante,manonèimmediatoilcollegamentoconilinkeitestisottostanti

• Ibusinessprocessdovrebberoregolareilfunzionamento,manondiventareprotagonistidelcorso,l’utentenondovrebbevederli,lafruizionedelcorsodeveessereaccompagnatadacalltoactionsodescrizionisemplicicheloguidinoattraversoicontenutieleesercitazioni/simulazioni.Finchénonmel’hannospiegatononsonoriuscitoacapirciniente,mancanoledescrizionidellesezioni,mainognicasounsoftwareèusabilequandosispiegadasoloenonhabisognodiguide

• Pocointuitiva• L’attivitàformativasullaconferenzadeiservizi,comeprimapagina,illustrailflussodocumentalechesi

attivainrelazioneadunaConferenzadeiServizi,tuttavianonappareconevidenzanellapaginaintroduttival’illustrazionedellefinalitàedellastrutturadellapiattaforma(oalmenoiononsonoriuscitaatrovarlaconimmediatezza–poihovistodopolasezioneHOWTO).LasezioneHOWTOdovrebbeessereresapiùevidentenellaprimapaginaocostituireproprioicontenutidellaprimapaginaperesplicitarelefinalitàdell’attivitàchesiandràasvolgere.E’essenzialechelefinalitàegliobiettivi,nonchélastrutturaelefunzionalitàdellapiattaformasianoresechiaresindasubito.Noncredoqueste

62

informazionipossanoesserelasciateallaliberasceltadichiaccedeallapiattaforma.E’essenzialechesianoenunciateesplicitamenteprimadiavviarelanavigazioneperunaesigenzadichiarezzaetrasparenzaeperconsentireaifruitoridiorientarsimaggiormentenelpercorso,avendoanchechiaraladirezioneacuitendere.

• Visualizzareilcontenutodei“tasti”checostituisconoildiagrammadiflussononèsemplice.Sonodidimensionitroppopiccolee,dunque,scarsamentefruibili(daunpuntodivistagrafico).D’altrocanto,èottimochesianocliccabili,permettendoachientrainpiattaformadiorientarsiinognifasedelpercorso,dopounprimomomentodiconfusione(dovutaalfattochenonsonochiarigliobiettivielefinalitàdell’attività,amioparere).Lapiattaformanonrispondeacriteridiaccessibilitàpertutti(glistepneldiagrammadiflussosonodescritticoncaratteritroppopiccoliehoavutodifficoltàacapirecosacisiascrittoinognitasto).

• Ingenerale,l’interfacciaapparepocointuitivaedinonimmediatacomprensioneperipotenzialifruitorichesiavvicininodaprofaniall’utilizzodellapiattaforma.Adesempio,nellaschermatainizialedell’attivitàsipotrebbeindicaregliobiettiviformativiechiarirelefunzionalitàfondamentali,ovvero:navigare,affrontaresimulazionidilavorodigruppoopersingoliutenti,contribuireattraversoproprieconsiderazioniedannotazioni.

• E’ottimal’ideachesiapossibilefaredellesimulazioni,anchesepoinonsonoriuscita(ononsiriesce)adavviarle.E’comeselasimulazionefosseraccontata,manoncifosselapossibilitàdiattivarla.Nonhobencapitocometalefunzionalitàsiaattivabile.Hoprovatoacliccaresuimenùatendinaperavviareunasimulazione,maalsubmitnonèsuccessonulla.Nonsocosailsistemaabbiagenerato,seabbiageneratoqualcosaecomequestopossaattivareilcontrollo(casierisposteinserite–cheperònonsonoriuscitaadinterpretare).Inaltreparolelasimulazione,amioparere,nonfunzionainmodoadeguato(pocochiareleistruzioniperfarlapartire-mancanodocumentinecessarixcomprendereilcasoreale).

• Imodellidilavoro(formatconvocazioni,formatpercomunicareorichiedereintegrazioni,formatperredigereverbali,etc.)nonsonoutilizzabilienonsembrasianodisponibilinelcorso.L’interfacciagraficaèassolutamentedamigliorareperchérappresentailgrandelimitedellapiattaforma,rendendolapocointuitiva,pocoaccessibileenonchiaramenteorganizzataperun’idealefruizionedellefunzionalitàcheprevede.Perpoterrisponderealledomandedicontenutoneltestènecessariofareriferimentoadocumentichenonsonodisponibiliinpiattaforma

• Hotrovatoilsistemaabbastanzaapprossimativoepocointuitivo• Ladimensionedeifontètroppopiccola• Lefunzionalitàdicollaborazionesocialandrebberoirrobustiteeresepiùaccattivanti• Lefunzionalitànecessariesonotuttepresenti,dovrebberoperòessereimplementateinmanierapiù

efficaceinterminidi:organizzazionelogica,interattività(illivellodiinterattivitàtrauserecontenutiètroppobasso),progressionenelcompletamentodelpercorsodiapprendimento(nonsiintuiscequalèilcomplessivoammontaredeicontenutidafruirepercuinonsipercepisceilquantoèstatofattoequantodeveessereancorafatto);esplicitazionedegliobiettividelpercorsoformativo

• Lanavigabilitàpotrebbeesseremigliorata,cosìcomeleinformazionirelativeadovecisitrovainognipagina(breadcrumbsinsufficienti)

• Nonèchiaralamodalitàdifunzionamentodellasimulation• Lefunzionalitàdisimulationerecommendationsonoancoraincompleteoinunostatoembrionaleche

nerendedifficilelafruizione.Avolteitestidell’interfacciasonoininglese

63

• Itutorialsonobensviluppatiearticolati.Occorrededicarecomunquemoltopiùtempo.Lascrivente,perilpocochehautilizzato,hatrovatovantaggio.Almenoduegiornatelavorativedovrebberoesserededicate(apprendereautilizzarelapiattaformaèdipersétimeconsuming)

• LAPARTEDEICASIPRATICINONRISULTAFUNZIONANTE• Risultacarentelafunzionalitàdellesimulazioni,cheandrebberoresepiùinterattiveemenocomplesse• Mancano:lapossibilitàdifareannulla/escidallasimulazione;unostrumentodidesigndeiprocessidi

businessaccessibile;unpercorsoformativoatappe,nelsensochetuttiicontenutisembranorappresentatiinmodonondifferenziatoallostessolivello(mentrealcunirisultanoosticiedaltripiùaccessibili)epersonalmentenonhocoltoqualiargomentisono“propedeuticiper…”

• Sarebbeopportunoinserire,sullasinistra,una“barra”deicontenuticheevidenzilastrutturadelcorsoelagerarchiadeicontenutistessi.L’utilizzodivideoedianimazioniagevolerebbel’apprendimento

• Cisonodellepartiinlinguainglesecheandrebberotradotte;igrafici,moltoutili,dovrebberoesserepiùleggibili

• Aggiungerevideo:nellahomeperintrodurreilsistemaeilsuofunzionamento,nellevarielezioniperspiegareicontenutichevisitroveranno.Aggiungerespiegazionitestualidicosafaognisingolapagina/sezionechesivisita(adesempiolasimulazione)

Positiveoverallopinionsandstrenghts• Nelcomplessoèunmodototalmentenuovoedinteressantediimparare• Moltovalidoilmodellorelativoalprocessodilavoroeilfattochesianavigabile• Superatoilmomentodismarrimentoinizialeecompresalalogicadicomeinteragireconlapiattaforma

lafruizioneèrisultatasempliceeintuitiva• Modalitàdie-learningveloceinquantopermettedisaltareicontenutichenoninteressanoper

dedicarsiaquellidaapprofondire• L’usodeibusinessprocessmodelinquestoambitodaungrandevaloreaggiunto,poichécredosiano

utiliperavereunavisionegraficadiunprocessocomplesso.Tuttavial’aggiuntadellainterattivitànell’immaginechedescriveilBP,perrenderlanavigabile,garantirebbeunamaggioreusabilitàdellapiattaforma

• Benstrutturatoedifacilefruizione• L’illustrazionedelflussodocumentaleècompletaechiaraalivelloconcettuale.Itastidiognistepsono

cliccabilierimandanoadognifasedilavoro.Cliccandoognistep,compareunasinteticaechiaradescrizionedellafasedilavoroincuicisitrova.E’ottimocheinognistepcisiaanchelapossibilitàdiredigeredocumentiattraversoilwiki,lavorandoinformacollaborativaIngenerale,lefunzionalitàessenzialicisono.Inparticolareildiagrammadiflusso,cliccabileadognistepconlerispettiveopzioni,èunottimostrumento,chiaroecomprensibiledalpuntodivistaconcettuale

• Lagraficasempliceelineareaiutaefavoriscel'apprendimentoattraversolanavigazionedelprocesso• Utilel’eserciziodisimulazionebasatosucasireali(mavacuratomeglio);efficace(madaimplementare

meglio)ilmeccanismodelloSCORE• Interessantelaraccomandazionedinamicadicontenuti(potenzialitàsemanticachetuttavianonho

avutomododivedereinesecuzione–mièstataraccontata)• Utilissimalanavigazionesequenzialedelflussodilavoroancheattraversol’immaginedelprocesso,ma

daottimizzare(purtroppol’immaginenellafasedinavigazioneètroppopiccolasulloschermo,mahoapprezzatocheisingolielementisonocliccabili/ancorapiùutilesel’esplorazionepotesseavvenire

64

comenellasimulazione,doveadognistepsievidenziagraficamentel’attivitàdelprocessoincuisono.Nellasimulazioneilriquadrocheevidenziaquestoscorrerenelflussoètropporistretto)

• L’interazionesocialconcommentiealtreformedicontribuzioneèabbastanzautile• Buonaidealafunzionalitàcontribuisci,conlapossibilitàdiaggiungeretestoesplicativo• Lanavigazionesequenzialedelflussodilavoroèl’aspettopiùinnovativoedefficacedellapiattaforma• Giudiziopositivoperlaraccomandazionedinamicacontenuti• Lasimulationèunafunzionalitàutile,mavaresapiùintellegibile• Presenzadeglistessielementidiapprofondimentoinognipagina,evidenziatiattraversografica

omogenea• Lasimulazionesucasirealiaiutamoltissimonelprocessoditraining.Unicoaspettonegativoèchelo

spazioristrettodellafinestracontenenteilflussodeitaskviaviaattivinonpermettevadicapireimmediatamente(senzascrollorizzontaleeverticale)inqualefasedelprocessomitrovavo

• Leimmaginidelprocessorendonochiaralavisioned'insieme.Chiaroiltestodescrittivodeisingolipassaggi

• Validigliesercizidisimulazionebasatisucasirealinonchélefunzionidiinterazionesocialconcommentiedaltreformedicontribuzione.Ledescrizionideicontenutisonocompleteedifacilecomprensione

• Traimodulipiùinteressantisegnalo:lanavigazionesequenzialedelflussodilavoro;l’interazionecollaborativadegliutenti;lesimulazioni,madarenderepiùstrutturateepiùaderentiacasirealicomplessieverosimili

• Hoapprezzatogliesercizidisimulazionebasatisucasireali,lanavigazionesequenzialedelflussodilavoroelavisionegraficadelprocesso

• Ritengoutililemappenavigabili,l’interazioneneicommentielesimulazionidicasireali• Lasimulazioneèunabuonaeutileidea,mal’usoditerminicomeroboteumanoandrebberocambiati

ospiegatiinqualchemodo• OKperinterazionesocial,simulazione,navigazionesequenziale• Lanavigazionesequenzialedelflussodilavorobasatasuclicèfacilmentecomprensibilealivello

concettuale.Ledescrizionisonochiareesintetiche.Ottimalapossibilitàdico-produrredocumentiedicontribuireinmodalitàcollaborativa(wiki)

• Puntidiforza:l’eserciziodisimulazioneconcasireali;lanavigazionesequenzialedelflussodilavoro;linearitàesemplicitàdellagrafica

• LANAVIGAZIONESEQUENZIALEDELFLUSSODILAVOROMISEMBRALAMIGLIOREFUNZIONALITA’DELSISTEMA

• Tralefunzionalitàpiùutili:navigazionesequenzialedeiflussidilavoroMARLENENegativeoverallopinionsandmissingfunctionalities• Difficoltànell’integrareledispenseconleinformazionifornitenelcorso• Duranteilcorsoinmodalitàe-learningsonotroppelesezioniincuivienerichiestaunasemplicelettura

deltesto(senzal’ausiliodellavocenarrante)• Mancanoapprofondimentisucasirealiedesercizipratici• Fruibilitàdellapiattaformascarsa,perviadelbloccopopupedinumerosiblocchidurante

l’avanzamentodelleslide• Corsomoltoteoricoconpocheevidenzepraticherispettoallavoroquotidiano

65

• IcontenutinonsonoaggiornatirispettoalnuovoDLGScheriformalanormativasulleConferenzediServizi

• Icontenutipropostisonobasicisarebbemeglioaverecontenutipiùapprofonditi• Utileperdipendentipubblicialleprimearmi,maicontenutivannoaggiornatirispettoallanormativa

vigenteperleconferenzediservizi• Icontenutidialcuniargomentinonsonoaggiornati,(ALMENOPERITESTACUIHOAVUTOACCESSO),

sarebbeutileaverelapossibilitàdiinteragirepersottoporredomandesudubbinormativi• Probleminelladisattivazionedellepolicydisicurezzadelbrowserperlariproduzionepluginjava;Non

sufficientequalitàdell’insegnamentorispettoalmio(basso)livellodiconoscenza• Occorrededicaremoltotempoalcompletamentodelmoduloformativo.Servirebbeunamodalitàpiù

snella• Occorremaggiorespaziocollaborativoefunzionidedicateperdiscutereeapprofondirenormativa,

interpretazioni,prassitracolleghidelSUAP• Esistonosistemidie-learningpiùall’avanguardiacuiispirarsi• Mancanospecifichepaginededicateallarisoluzionedicasirealioconinterpretazionedellenormeper

situazioniparticolarmentedifficilidagestire• Strumentodiapprendimentononevolutointerminidiweb2.0(interazionetradiscenti,possibilitàdi

postaresuisocialnetworkiprogressi,interoperabilitàconyoutubeoaltricontenutiwebesterni,…)ewebsemantico(ricercacontenuti,propostaautomaticadicontenutiinbaseainteressiecaratteristichedell’utente,…)

• Nonpresenti:spazidiapprofondimentocheintegranolevariazioninormativeintercorserispettoallapregressadisciplina

• Perpoterconsiderarelapiattaformaunvalidostrumentodiapprendimentomancaunsistemadiricercadeicontenutiall’internodellapiattaforma.Lostrumentohaunapprocciosolosequenziale

• Mancanochatostrumentidiscambioinformazionitrapartecipantialcorso• Carenze:possibilitàdiscambiarsiopinioniconespertioaltriutentisucasistichereali;gamification,

comepiùavantidescritta• NONMISEMBRAADATTALAPOSSIBILITA’DIINTERAGIREPERPOTERPORREDOMANDEOPOTER

ESPRIMERELANECESSITA’DIUNAPPROFONDIMENTO• Mancanoaltricanaliutiliall’apprendimentoqualiilconfrontoconglialtridiscentioconundocente

espertoperapprofondiretemispecifici.Inoltreicontenutipresentati,sebbeneutili,nonentranonelmeritodelleconoscenzecheservononell’esperienzaquotidianadellavoro–ècomesesirimanesseinunpianoancorateorico(percuirestacomunqueunostrumentoutileperriceverenozioniadesempionormative)

• Nonc’èlapossibilitàdisaltarealcuneslideperdedicarsidirettamenteaicontenutidiinteresse• Nonhotrovatospazicollaborativi,didiscussioneediinterscambiotrautentieversountutoresperto• AssentelafruizionedivideoPositiveoverallopinionsandstrenghts• VALIDOPERLAPRESENZADIFUNZIONALITA’DIFFERENTICOMECONTENUTIDINAMICI,TEST

INTERMEDI,APPROFONDIMENTI• Icontenutisonoesaustiviedisemplicefruizione• Buonstrumentodiaggiornamento• Conl’E-learningsievitadidoverpartecipareacorsiinpresenza(edunquedispostarsifisicamente)e

ciòpermetteunafruizionepiùamisuradegliimpegniprofessionaliinessere

66

• Abbastanzasemplicedausareeschematicopermemorizzareglielementiprincipali• Pratico,sinteticoeveloce• Interfacciachiara–flussosequenzialesempliceescorrevole• Utileperriceverenozioninormative,credosiaunvalidostrumento• UTILISSIMELEVERIFICHEINTERMEDIE• Apprezzabiliimomentidiverificaintermedi• Interessantelapresenzadilinkagliargomentitrattatinellevarieslideedeimomentidiverifica

intermedicondomandearispostamultipla• Puntidiforza:Verificaintermediaalterminedellesingoleunità;Guidaaudiosulleslide• Validoapproccioquellodelledomandediverificaintermedia• Ottimoilflussodiscorrimentosequenzialedelcorso• Benegliintervallidiverificaarispostamultipla• OKVERIFICHEINTERMEDIEEFINALI• Positivalafruibilitàonlinedeicontenuti• Maggiormenteutili:momentidiverificaintermedieriferimentinormativi• Piuttostoutiliimomentidiverificaintermedi• Validal’enfasisuicontenutiprincipalidatacongrassettoeriquadriinevidenza• Imomentidiverificaaiutanoaricapitolareeragionareeintervallandoilcorso,spezzanolamonotonia

dellalettura• Ledomandediverificaintermediapermettonodifarsiunrapidoriassuntomentaleeverificarei

principaliconcettiappresi• Imomentidiverificaintermedicondomandearispostamultiplamisonopiaciuti

67

Collectedfeedback(inenglish)LearnPAdNegativeoverallopinionsandmissingfunctionalities• Theinitialimpactwiththeplatformdisconcertsandgeneratesuncertainty-sincethereisnota

predefinedpathtofollow,butavariousnumberoffeaturesthatcanbeactivatedatwill,whichutilityorbehaviorwedonotknowinitially.Sotheuserisdisplaced->werecommendtopotentiatetheintroductoryaspects(eg.theinitialexplanatorytext,videotutorials,etc.).

• AsfarasIgottosee,somefeaturesdonotseemtoworkwell(eg.recommendationsbar:icouldnotseeallthecontentthatmycolleaguesontheotherhandhaveexperienced);otherfeaturesarenotsowellexplained(eg.Simulationisnotclear,neitherincontentnorconcerningthepresentationoftheactionstobedone)

• Itisdifficultnavigatingwithintheplatform• Ihavenoticedthattherearesomeinaccuraciesincontentrelatedtoconferenceservices(thechanges

referredtotherecentreformDecreeonthematter,arenotcovered)• Thesimulationisshortlyeffective:inthelongrunitissimplyacompilationoffields(almostatest)

withoutprovidingfunctionsthatstimulatearealproblemsolving.Inaddition,thefinalscoresarenotcomputedonascaleofreferenceandsotheyloseanyevaluativemeaning

• unintuitiveinterface.Hardtoknowwhereyouare• Withthevastnessoftheinformationandlinksthatyoufaceisnoteasytounderstandifyouhave

exploredallthefunctionality• Theplatformisstillinanembryonicstateinwhichtheguidecontentsareunclearandtheuseof

informationbytheproposedinterfaceistricky• Thecourserequiresapreviouslevelofknowledgeandspecificexpertiseonthesubjectmatter(not

suitableforbeginners)• THEDESCRIPTIVEPARTISCOMPLETEANDEASYTOUNDERSTAND,EXCEPTFORSOMETYPOS(ES.

e&apos);THECONTENTSHEREINAREWELLDESCRIBED,BUTIFOUNDDIFFICULTTOUSEANDREADTHEGRAPHICSCHEMES

• Theuseoftheplatformisnotveryintuitive,wemustgetfamiliarandconceptuallyreconcilethetwomodesofcontentorganization(theflow,whichisveryclearandtakesplacethroughtheschematization,andthedescriptive/explanatorysectionoftheprocesses)

• Theplatformdonotseemparticularlyintuitive;thenavigablemapisinteresting,butitisnotimmediatetocatchtheconnectionwiththelinksandtheunderlyingtexts

• Thebusinessprocessesshouldregulatetheoperation,butnotbecometheprotagonistsofthecourse:theusershouldnotseethem;thefruitionofthecoursemustbeaccompaniedbycalltoactionsorsimpledescriptionsguidinghimthroughthecontentsandexercises/simulations.Aslongastheyhavehaveexplainedittome,iwasnotabletounderstandanything:therearemissingdescriptionsofthesections,butinanycaseasoftwareisusablewhenitisself-explanatoryanddoesnotneedguides

• Unintuitive• Thefirstpageofthetrainingshowsthedocumentflowthatisactivatedinconnectionwitha

ConferenceofServices;however,itdoesnotappearclearly,intheintroductorypage,anillustrationoftheaimsandtheorganizationoftheplatform(oratleastIcouldnotfinditwithdirectness-onlylaterIsawtheHOWTOsection).TheHOWTOsectionshouldbemademoreevidentwithinthehomepage,orevenitshouldbecomeitselftheuniquecontentofthefirstpage,inordertoclarify,sincethebeginning,thepurposeofthetasksyouwillperform.Itisessentialthatthelearningobjectives,aswell

68

asthecontentstructureandthefunctionsoftheplatform,aremadeclearfromtheoutset.Idonotbelievethatthisinformationcanbelefttothefreechoiceofeachuseraccessingtheplatform.Theyneedtobeexplicitlysetoutbeforeyoustartnavigating,forclarityandtransparency,andalsotoallowtheuserstofocusmoreinwhattheywillfindduringthepath,havingasaprerequisiteanevidentdirectiontogo.

• Viewingthedescriptionoftasksthatconstitutetheflowchartitisnotsimple.Theyaretoosmalland,therefore,poorlyusable(fromagraphicalpointofview).Ontheotherhand,itisgreatthattheyareclickable,allowingthosewhoentertheplatformtoorientthemselvesineverystepoftheway,bypassingtheveryfirstmomentofconfusion(duetothefactthat,inmyopinion,theaimsandpurposesoftheactivitiestotryarenotsoclear).Theplatformdoesnotmeetthecriteriaofaccessibilityforeveryuser(thestepsintheflowchartaredescribedwithtoosmallfontsandIhadtroubleunderstandingwhatiswrittenoneachsquare).

• Ingeneral,theinterfaceappearslessintuitiveandnotreadilyunderstandablebythepotentialusers,approachingasprofanetheuseoftheplatform.Forexample,onthehomescreen,itshouldbeindicatedwhatthetraininggoalsareandalsobesummarizedthefundamentalcapabilitiesoftheplatform,namely:browsing,tacklingteamworksimulationsor,forindividualusers,contributingbypostingtheirownconsiderationsandannotations.

• It'sagoodideathatyoucandothesimulations,eventhoughIcannot(oramnotableto)startthem.Thesimulationwasintroduced,buttherewasnochancetoactivateit.Idonotunderstandhowthisfeaturecouldbeactivated.Itriedtoclickonthedropdownmenutostartasimulation,butaftersubmittingnothinghappened.Ididnotknowwhatthesystemhavegenerated,ifihadreallycreatedsomethingchosingarealcase,howthiscouldbecontrolledandwhatanswersineededtoinserted-maybealsobecauseIfailedtounderstandsomeofthequestions.Inotherwords,andinmyownopinion,thesimulationdoesnotworkproperly(mainlyduetounclearinstructionstogetitstarted-andlackofthenecessarybackgrounddocumentstounderstandtherealcase).Theadministrativemodules(convocationsformat,communicationnotes,letterstorequireintegrations,draftsforreports,etc.)doesnotseemtobeavailableandusableamongthesimulationstages.

• TheGUIneedtobeimproved,becauseitrepresentsthemajorlimitationoftheplatform,makingitnon-intuitive,difficulttobeaccessedandnotclearlysetupforanidealuseoftheprovidedfunctionalities.

• Inordertoanswertosomeofthequestionscontainedinthetest,ithasbeennecessarytorefertodocumentsthatwerenotavailablewithintheplatform

• Ifoundtheprototypequiteroughandunintuitive• Thefontsizeistoosmall• Socialcollaborationcapabilitiesshouldbestrengthenedandmademoreattractive• Thefundamentalbasicfeaturesareallpresent,butshouldbeimplementedmoreeffectivelyinterms

of:overalllogicalorganization;interactivity(thelevelofinteractivitybetweenuserandcontentistoolow);progressioninthecompletionofthelearningpath(icouldnotguesswhatthetotalamountofcontenttobeenjoyedwas,sowithoutthatperceptionicouldnotfindwhathasbeendoneandwhatwasstillneededtobedone);explicationofthetrainingobjectives

• Navigabilityshouldbeimproved,aswellasinformationaboutwhereyouareineachpage(breadcrumbsseeminsufficienttome)

• Itisnotcleartheoperationalmodeofthesimulation• Thesimulationandrecommendationfeaturesarestillincompleteorinanembryonicstate,which

makesitdifficulttouse.SometimestheinterfacetextsareinEnglish

69

• Thetutorialsarewelldevelopedandarticulated.However,theplatformrequiresalotoftimetobedevotedto.Forthelessiusedit,ihavefoundsomeadvantages.Atleasttwomoreworkingdaysshouldhavebeenspenton(thatistosaythatjustlearningtousetheplatformitselfishardlytimeconsuming)

• THEPRACTICALCASESARENOTWORKING• Thesimulationfunctionalityaredeficientandshouldbemademoreinteractiveandlesscomplex• Itlacks:thecapabilitytomakeundo/quitthesimulation;adesigntoolavailabletocorrectthe

businessprocessesonthego;atrainingcoursebystages,meaningthatallcontentappearslikeifitisrepresentedinanundifferentiatedmanner,allatthesamelevel(whileinrealitysomepartsaremoredifficultandsomeothermoreaccessible),andpersonallyIhavenotgraspedwhattopicsareprerequisites"inpreparationfor..."

• Itwouldbeappropriatetoinclude,ontheleft,a"bar"ofcontentsthatshowsthecoursestructureandcontenthierarchy.Theuseofvideoandanimationsfalsowouldfacilitatelearning

• TherearesomepartsinEnglishthatshouldbetranslated;graphics,veryuseful,shouldbemorereadable

• Videosneedtobeadded,i.e.inthehometointroducethesystemanditsoperation,inthevariouslessonstoexplainthecontentthatyouwillbeabletofind.Alsoitwouldbeadvisabletoaddtextualexplanationsofwhateachsinglepage/sectionyouvisitstandsfor(duringsimulationitisreallyamust)

Positiveoverallopinionsandstrenghts• Overall,itisatotallynewandaninterestingwaytolearn• Themodeloftheworkprocessisveryaccuratedandthefactthatitisnavigableisavalidfeature• Goingthroughaninitialmomentofconfusionandonceunderstoodthelogicofhowtointeractwith

theplatform,thefruitionwassimpleandintuitive• Thise-learningmodeprovestobefaster,becauseitallowstheusertoskipnotofinterestcontent,to

embracedirectlythosejudgedtobeexplored• Theuseofbusinessprocessmodelinthisareaisabigplus,becauseIthinktheyareusefultogetaeasy

understandablegraphicalviewofacomplexprocess.HowevertheadditionofmoreinteractivityfeatureswithintheimagedescribingtheBP,egtomakeitfullynavigable,wouldguaranteeagreaterusabilityoftheplatform

• Wellstructuredandeasytousewebpages• Theillustrationoftheflowofstepsoftheprocessiscompleteandclearataconceptuallevel.Theareas

ofeachstepareclickableandrefertoeachstageofwork.Clickingoneachsteparea,youwillseeaconcisebutcleardescriptionoftheworkphaseinwhichyouarelocated

• Itisgreatthat,ineverytaskoftheBP,thereisalsothepossibilityofdrawingupdocumentsthroughwikiandalsotovalidatethem,workingwithtextsinacollaborativeandassistedway

• Ingeneral,theessentialfeaturesareincluded.Inparticular,theflowchart,clickableateachprogressionandevidencingtherespectiveoptions,isagreattool,clearandunderstandablefromalogicpointofview

• Thesimpleandstraightforwardgraphichelpsandencourageslearningthroughthenavigationoftheprocess

• Userscangetprofitfromthesimulationbasedonrealcases(butitshouldbebetterconfiguredandcurated);themechanismoftheSCOREiseffective(butitneedstobetoimplementedmoreaccurately,alsolettingtheuserstounderstandthelogicofthepointsawarding)

70

• Aninterestingfeatureisthedynamicrecommendationofcontent(asemanticpotentialitythatneverthelessihavenotnoticedwhilerunningthesystem,becauseitworksinbackground-Iwastoldaboutandthenirealizeditwatchingacollegueofminetestingtheplatform)

• Theforwardandbackwardnavigationoftheworkflowthroughtheimageoftheprocessisveryuseful,butstillneedtobeoptimized(indeed,unfortunately,thepictureshowedinthenavigationphaseistoosmallonthescreen,evenifIappreciatedthattheindividualelementswereclickable;atthatpointigotagreaterbenefitdownloadingthepicturetomydesktop).Forinstanceitwouldbemoreusefuliftheexplorationofthesingleactivitiesinthebrowsingmodecouldtakeplaceasinthesimulation,whereeachstepisgraphicallyhighlightedshowingthecurrentactivityoftheprocessinwhichwestand.However,duringsimulation,thepanethatcontainsthedynamicflowpictureistoosmallandweneedtoscrollhorizontallyandvertically(anditbothers).

• Thesocialinteractionwithcommentsandotherformsofcontributionisquiteuseful,alsotoaskhelptootherusers

• Goodistheideaofthewindowtocontribute,withtheabilitytoaddbrandnewexplanatorytexts• Thesequentialbrowsableprocessworkflowisthemostinnovativeandeffectiveaspectoftheplatform• Apositiveratinggoestothedynamiccontentrecommendation• Thesimulationisausefulfeature,butitshouldbefixedandmademoreintelligible• Ilikedthepresenceofthesameelementsoneachpage,identifiedthroughstandardgraphicsandicons• Thesimulationofrealcaseshelpsalotduringthetrainingprocess.Theonlydownsideisthatthe

confinedspaceofthewindow,containingthetasksflowandtheactivetask,doesnotallowtounderstandimmediatelyandwithoutscrollingatwhatstageoftheprocessIam

• Theprocessimagesgiveaclearoverview.Alsocleararethetextualdescriptionsofeachindividualstep• Thesimulationexercisesbasedonrealcasesarevalid,aswellasthecollaboirativeandsocial

interactionfunctionsforpostingcomments(andcommentstocomments),givefeedbackandaddacontribution.Asforthecontributions,theheaderofeachsectionofthedocumentshouldbetranslatedinItalian.Thecontentdescriptionsarecompleteandeasytounderstand

• Amongthemostinterestingmodulesipointout:1)thenavigableworkflow;2)thecollaborativeinteractionoftheusers;3)thesimulationofarealcaseandthepossibilitiestochatwithotherlearnerswhiledoingit,buttherealcaseitselfhastobemademorestructuredand-forlearningpurpose-moreadherenttolikelytohappencomplexcases

• Ienjoyedthesimulationexercisesbasedonrealusecases,andthenavigationoftheConferenceofServicesworkflowplusthegraphicaloverviewoftheprocess

• Iconsiderusefulthenavigablemaps,theoptiontoaddcommentsandthesimulationsoftheSUAPapplications

• Thesimulationisagoodandusefulidea,buttheuseoftermssuchasrobotsandhumanhastobeavoided...otherwisetheseoptions(andhowtousethem)havetobeexplainedinsomeway

• OK:socialinteraction,simulation,sequentialnavigation• Thebrowsingmode,basedonclickstomovethroughtheTitoloUnicoprocess,iseasytouse,to

understandandeffectivefortrainingonthejob.Thedescriptionsofeveryactivityinvolvedareclearandconcise.Excellenttooistheopportunitytoco-producedocumentsandtocontributeinthecollaborativemode(wiki)

• Strengths:theexercisessimulatingtherealcases;thesequentialnavigationoftheworkflow;thelinearityandthesimplicityofthegraphic,whichworksevenwithmobiledevice

• THESEQUENTIALITYOFPROCESSBROWSINGSEEMSTOMETHEBESTFUNCTIONALITYOFTHESYSTEM

71

• Themostusefulfunctionalityisthenavigablebusinessprocess,bothbypageclickandbyinteractingwiththeimage

MARLENENegativeoverallopinionsandmissingfunctionalities• Iexperienceddifficultiesinintegratingtheattacheddocuments(aboutthereformationofthe

conferenceofservices)withtheinformationprovidedwithintheMarlene'scourse• Duringthecourseine-learning,toomanysectionsrequiredasimplereadingofthetext(withoutthe

helpofthenarratorvoice-functionthat,whenpresent,makesmelearnfaster).AtthebeginningihadsomeproblemswiththeconfigurationofmyPCaudiodevices

• Itlacksinsightsonrealapplicationsandpracticalexercises.TheSUAPorganizationisdefinitivelymorecomplexthanwhatthelawssay

• Poorusabilityoftheplatform,becauseofthepop-upblockerandvariousotherblocksduringtheforwardmovementoftheslides

• Averytheoreticalcourse,withfewpracticalevidenceswithrespecttomydailywork• ThecontentsareoutdatedcomparedtothenewLegislativeDecreereformingtheregulationof

ConferencesOfServices• Theproposedcontentsarebasic.Itwouldbebettertohavemorein-depthcontent• Usefulforcivilservantsbeginners,butthecontentsneedtobeupdatedwithrespecttothecurrent

regulationsforSUAPandforconferencesofservices• TheSUAPcontents,atleastforsomeargumentstowhichihadaccessforthetest,areoutofdate.It

wouldbeusefultohavetheopportunitytointeracttosubmitquestionsonregulatoryconcernsandtoreceiveopinionsfromothercolleaguesattendingthecourse

• Ihadtechnicalproblemsinthedeactivationofbrowsersecuritypolicyforthejavapluginplayback• Idonotfindasufficientqualityoftheteachingaccordingtomy(low)levelofknowledge.Ifindithard

learning• Ittakestoomuchtimetocompleteallthetrainingmodule.Itwouldservealeanermodetogothrough

thecontentandtogiveasfinishedeachslide/chapter• Amorecollaborativespaceisneeded,andalsosomededicatedfunctionstodiscussandexploredeeply

thelegislation,togiveinterpretationsandtoexchangebestpracticesamongSUAPofficersandmanagers

• Therearemanymoreadvancede-learningsystemstogetinspiredby• nospecificpagesaddressedtothesolutionofrealcasesorinterpretationoftherulesforparticularly

difficultsituationstohandle• Marleneisalearningtoolnotevolvedintermsofweb2.0(interactionbetweenlearners,abilitytopost

individualprogressonsocialnetworks,interoperabilitywithyoutubeorotherexternalwebcontent,...)andthesemanticweb(contentsearching,automaticallyofferedcontentonthebasisoftheuser'sinterestsandcharacteristics,...)

• Thereweren'tenoughstudiesthatintegratethechangesoccurredwithrespecttothepreviousregulationsgoverningtheconferenceofservices

• Inordertoconsidertheplatformavaluablelearningtool,itlacksacontentsearchsystemwithintheplatform.Theinstrumenthasonlyasequentialforwardandbackwardapproachtoreachcontent

• Achat-orevenasimplercommunicationtooltoexchangeinformationbetweentheparticipantsatthecourse-ismissing

72

• Weaknesses:idonotfindanypossibilitytoexchangemypointsofviewwithexpertsorotherusersonrealcasestudies;alsogamificationaspects-asdescribedinthequestionnaire-arenotpresent

• ITDOESNOTSEEMSUITABLETOME,ASALEARNINGINSTRUMENT,BECAUSEICAN'TFINDTHEOPPORTUNITYTOINTERACTWITHSUAPEXPERTSINORDERTOASKQUESTIONSORTOBEABLETOEXPRESSMYPECULIARNEEDSFORIN-DEPTH

• Otherusefullearningchannels,suchascomparisonwithotherlearnersorwithanexperiencedteachertofocusonspecifictopics,needtobeaddedMoreover,thepresentedcontent,althoughuseful,donotenterintothemeritsoftheknowledgeandskillsneededineverydaywork-itisasifthecourseremainsinamoretheoreticallevel(soitisstillausefultooltoreceivesuchregulationsnotions,butnottolearnhowtoconductaconferenceofservices)

• Istherethepossibilityofskippingsomeslidestogodirectlytothecontentofinterest?Icannotfindit...

• Ididnotfindcollaborativespacesfordiscussionandinformationexchangebetweenusersorfromanexperttutor

• Absenceofanyconnectionorusageoflearningbyvideo

Positiveoverallopinionsandstrenghts• THESYSTEMISVALIDFORTHEPRESENCEOFDIFFERENTFUNCTIONALITIESASDYNAMICCONTENTS,

INTERMEDIATECOMPREHENSIONTESTS,IN-DEPTHSSECTIONS• Thecontentiscomprehensiveandthetooliseasytouse• Agoodtooltoupdateone'sknowledge• WiththeE-learningyouavoidhavingtotakecoursesinthepresence(andthereforeyouquitphysically

transferts).Thisenablesamoretailoredusageoftime,inrelationwiththeoutstandingprofessionalcommitments

• Prettysimpletouse.ItProvidesaschematicapproachtostoreinmindquicklythemainelementstobelearned

• Practical,synthetic,quicktounderstandandfasttouse• Aclearinterface-Asimpleandsmoothsequentialflow• Usefultoreceiveregulatorynotions,Ithinkitisagoodtool• INTERMEDIATECHECKSAREUSEFUL• Theintermediateverificationmomentsareappreciable• Theplatformisinterestingforthepresenceoflinkstothetopicscoveredinthevariousslidesandfor

themultiple-choicequestions,bothforintermediateandfinalassessmentoflearning• Amongthestrengths:intermediatechecking-bytheroreticalquestions-attheendofeachunit;audio

guideoneachslide• Aviableapproach,thatofthemid-termexaminationquestions• Excellentforthesequentialslipstreamofthecourse• Wellfortheintervalswiththemultiplechoicequestions• OKFORTHEINTERMEDIATEANDFINALCHECKS• Positiveistheonlineavailabilityofthecontent• Themostusefulfeaturesare:theintermediatemomentsofverificationandthepresenceofnormative

andstandardreferences• Ratherusefultheintermediatemomentsofverification• Howeffectiveistheemphasisonthekeycontent,conveyedthroughbold,colorsandhighlightboxes

73

• Themomentsofverificationhelpthelearnerstosummarizeandreasonand,spacingoutthecourse,theybreakupthemonotonyofreading

• Theinterimreviewquestionshelpyoumakingaquickmentalrecapandcheckingmentallytheeffectiveassimilationofthemainconceptstolearn

• Ilikedthemomentsofintermediatecheckbasedonmultiplechoicequestions