Embedding Benefits Final Report PotsPan Project

Project Information

Project Title Pilot of the SWANI Project Administrative Network (PotsPan)

Start Date May 2012 End Date April 2013

Project URL/Blog URL Project Website: http://potspan.pbworks.com/

Project Blog: http://potspan.blogspot.co.uk/

Design Studio URL http://jiscdesignstudio.pbworks.com/w/page/64840266/The PotsPan Project Home Page

Lead Institution Coleg Sir Gâr

Project Lead Contact Name

Dave Howells (Project Director); Tony Toole (Project Manager)

Contact Email address dave.howells@colegsirgar.ac.uk ; tony.toole@e-college.ac

Programme Name Embedding Benefits Category

Programme Manager Rob Englebright (r.englebright@jisc.ac.uk)

1. Summary

The PotsPan (Pilot of the Swani Project Administrative Network) project was funded as part of the Jisc e-Learning Embedding Benefits Programme. The goal was to take the outcomes of the successful Swani (Secure Work-based learning through Networked Infrastructure) concept-proving project completed by Coleg Sir Gâr in 2012 and to pilot it more broadly across the regional partnership.

The Jisc Swani Project created a digitised administrative and management documentation system for Work Based Learning in Wales that included a digital signature system that satisfied the audit requirements of the EU and the Welsh Government.

The PotsPan project piloted the system with two of the partners in the Skills Academy Wales South Wales WBL consortium and also evaluated the approach in a partner University for potential use in the administration of a new distance learning WBL programme.

The purpose of the PotsPan project was to take what was a successful concept proving project and develop it into a practical digitised WBL administrative system that would not only satisfy the needs of the SWASW partnership, but would also be of value to other institutions addressing similar issues.

2. What resource(s) did you package/collate/disseminate for use by other institutions?

The main need of the target stakeholders in the PotsPan project was to have a digitised administrative and management system for work based learning in the region that delivered significant benefits in terms of cost effectiveness, operating efficiencies and data consistency and security. A further need was to have an audit trail for WBL that met the requirements of both the EU and the Welsh Government who were jointly funding the provision in the region.

1

The original Swani project successfully addressed both of these needs in a concept-proving context and the PotsPan project was intended to test the systems in the field with other partners in the network.

A key activity in the PotsPan project was to fully populate and test the central document management system based on the open source JOOMLA application. The objective was to ensure that the system was complete and fully debugged and represented a sharable resource for use by all partner institutions.

The benefit of this preliminary activity was that the document management system would be transferrable to other institutions that had related information management requirements. The ability to re-brand documents according to the institution that was using the system demonstrated how flexible the system was and how it could be adapted and repurposed by other institutions.

3. How did you go about embedding your resources / outputs / outcomes into the wider community?

The PotsPan project methodology followed a straightforward testing and evaluation process that began with a period of system refinement at Coleg Sir Gâr where the document management system was hosted. The web designer and the project manager worked together to test each of the system functions and the usability of each of the forms, including how effectively they shared common data fields.

Workshops were held at both Coleg Sir Gâr and Pembrokeshire College with the WBL teams that would be testing the system in the field. Each team selected an appropriate cohort of work based learners and completed a testing and evaluation exercise that involved completing all the digitised versions of the administrative documents in parallel with completing the existing paper based versions. Each exercise concluded with a comparison of the two approached with the aim of judging the relative advantages in terms of efficiency and effectiveness. These comparisons would be presented in the form of case studies.

An additional exercise was carried out at Swansea Metropolitan where the use of electronic signatures was explored. It was felt that this solution for document verification would be eventually accepted by the EU for audit purposes, but that was not the case currently. The exercise at Swansea Met, being unconstrained by EU rules, was designed to take that agenda forward by testing and evaluating current electronic signature technologies and their appropriateness for educational administration, particularly in the distance learning context.

Following the usability testing exercise at Coleg Sir Gâr, the two partner cohorts for the WBL implementation testing and evaluation were chosen and the process begun. The paper based administration forms were completed as normal and were then transcribed onto the digital system. The ease with which this was completed was noted, as was the potential saving in time with the sharing of common data fields.

A perception at the outset was that there were administrative processes, particularly with the registration of new learners, for example, where multiple forms were completed by hand that contained the same information in terms of learner name, address, NI number etc. An anticipated benefit was that the digitised system would only require these data fields to be entered once, would ensure data consistency and, where editing was needed at a later date, the edits would automatically update all forms that used that data.

The testing of electronic signature usage at Swansea Metropolitan was focussed on the development of a distance learning version of a WBL programme delivered to the armed forces in Germany. Of particular interest was the use of such signatures to verify the authenticity of written assessment materials submitted online.

2

The outcomes of the testing were recorded using a common case study template for ease of comparison. The outcomes were also shared on the project blog and wiki and were presented for wider dissemination at ALT-C and at the JISC Online Conference 2012.

4. What impact has your embedding benefits project had and who are the beneficiaries? Include evidence of impact wherever possible (e.g. survey results, evaluation, cost benefit analysis etc.)

The major benefit delivered by the PotsPan project has been the implementation and experience gained by institutions in the WBL regional partnership. Pembrokeshire College, lead institution for Skills Academy Wales (South West) consortium, and Coleg Sir Gâr are the two main partners and both participated in the pilot exercise. The two other partners, PRP Training and the Prince’s Trust, did not pilot the system as part of the project but were involved in the process and will benefit from the results as members of the network.

It was agreed at an early stage in the PotsPan project that a detailed piloting exercise by the two main partners in SAWSW would be the most efficient and effective way of testing and evaluating the digitised WBL administrative system developed by the Swani project. The outcomes would then be shared with the other partners and made available to all other institutions in the JISC community seeking to achieve similar goals.

The additional involvement of Swansea Metropolitan University (now merged as part of University of Wales Trinity Saint David) was planned as a way of broadening the testing and evaluation of electronic signatures in the audit trail process.

The impact of the PotsPan embedding benefits project was evidenced by:

Increased awareness: The original Swani project outcomes were piloted, developed and evaluated by two of the four institutions in the SAWSW consortium and the outcomes of that process shared across the consortium and with the wider Jisc community;

Greater take-up: The use of electronic signatures and digital pens as part of the document audit process was taken further by the piloting exercise at Swansea Metropolitan which was not a partner in the original Swani project. This exercise specifically tested the transferability and benefits of the project outcomes in a different educational information management context;

Use of resources: It will be a matter for the SAWSW consortium to agree the future use of the digitised document management system developed by Swani and piloted through PotsPan. However, the evaluation outcomes were very favourable and it is expected that the system will be adopted by the consortium as its central document management resource;

Community of interest: The outcomes of the project have been disseminated widely through ALT and Jisc conference presentations. Dissemination has also included the sharing of information and resources on the project website, project blog and through the Jisc Design Studio;

Evaluation evidence: Each of the three pilot exercises carried out through the PotsPan project resulted in the production of a case study that detailed the outcomes. These are available for viewing and download on the project website at http://potspan.pbworks.com/w/page/53357795/Project Deliverables.

5. What outputs has your project produced?

The Outputs and Results for the JISC PotsPan project included an updated and tested WBL Document Management System, a series of three Case Studies that detail the evaluation and testing of that system, and a project Blog that provides a narrative account

3

of how the project evolved. The final output was this report that provides an overall summary of activities, achievements and deliverables.

The primary deliverable was the JOOMLA based document management system that was originally developed during the Swani project. As a result of the PotsPan testing and evaluation activities the presentation of the documents was refined and the efficiency and consistency of data sharing was improved. The system can be viewed at http://sawsw.colegsirgar.ac.uk/.

Examples of the documents and the way they are presented are shown on the project blog at http://potspan.blogspot.co.uk/ and are available for download from the project Wiki at http://potspan.pbworks.com.

There were three structured testing and evaluation exercises carried out as part of the PotsPan project. Two of them tested the document management system, and the use of the digital pens for audit purposes, with selected WBL cohorts at two of the partner institutions. The third exercise explored the use of electronically signed documents, rather than digital pens, in the administration of a work based distance learning programme at a third partner institution.

The outcomes are presented in the form of three case studies which can be viewed and downloaded on the project Wiki. The outcomes are also available on the Jisc Design Studio at http://jiscdesignstudio.pbworks.com/w/page/64840266/The PotsPan Project Home Page.

6. How will the embedding benefits activity be developed further/sustained?

The expectation is that the success of the pilot projects will lead to SAWSW adopting the digitised system for its document management processes. The outcomes of the work are being disseminated and it is possible that the system will be adopted across the whole of Wales. This will be a matter, however, for the managers of the WBL programme and the partner institutions involved.

Clearly the digitisation of the document management system will change administrative practice when adopted. One of the key benefits is in the overall management of the scheme as the digitised information is available at all times for any partner or manager. The system will also have the capacity to communicate and share data with the wider student information management systems in the institutions.

The system is not yet embedded. It is hoped that the outcomes of the pilot exercises and the clear benefits of the digitised system (which have already been recognised and welcomed by all participants) will lead to that process taking place. There is still work to be done, however, to achieve this.

The electronic signature/digital pen solution for the document audit trail has yet to be resolved and may remain a barrier to adoption. The project has made a positive contribution to that debate and has been disseminated through several conference presentations. It is hoped that the project team will continue to be involved in those discussions both with the Welsh Government and in Brussels.

The impact on other institutions is not able to be judged at present. However, the two key project deliverables: the digitised document management system and the electronic signature system will both be of value to other institutions addressing the same issues. The benefit of the document management system is in the use of open source software as a sustainable solution. The benefit of the electronic signature system will be, particularly if eventually accepted for audit purposes, in the ability to fully digitise a secure and verifiable document management system.

7. Summary and Reflection

4

The original Swani project and the subsequent piloting through PotsPan represented a serious attempt to address a seemingly intractable problem: that of creating a digitised document management system for the work-based learning programme in Wales that met the audit requirements of the European Commission.

The EU audit system requires hand written signatures on all audit documents. This, and the resulting Welsh Government requirements, meant that the WBL administrative document system was paper based. The projects set out to find a solution to this problem.

Two solutions were found. The first was a ‘quick and dirty’ solution that involved the use of a digital pen. This enabled the creation of document with a hand written signature, while at the same time creating a digital version that could be uploaded to the digitised document management system. This met the requirements of the current audit system and also allowed full document digitisation. However, it was not ideal as it still required a paper version as part of the system.

The second solution was the creation of an electronic signature system for digital documents that met the EU requirements. A typical system of this kind, acceptable globally for secure online financial transactions, involves the use of verified digital certificates that authenticate signatories of digital documents. It was this solution that was piloted at Swansea Metropolitan.

The Swansea Metropolitan pilot demonstrated that a process that met the published EU guidelines was possible. It also showed that it met the other project requirement of being optimally cost-effective through the use of open source software and freely available web based services.

There will need to be decisions made in both the European Commission and the Welsh Government for an electronic signature system to be accepted for audit purposes, but it is felt that the outcomes of the Swani and PotsPan projects will make a valuable contribution to that ongoing debate.

In summary:

Lessons learned: The full digitisation of the WBL administrative document system was successful and delivered the operating efficiencies and data consistency goals. The digital signature system was also shown to work, but there was some progress remaining on EU and Welsh Government acceptance;

Project expectations: It was always felt that the digitisation of the document management system was achievable, but that the acceptance of the electronic signature process would only be a contribution to the debate. The use of the digital pen was a compromise solution;

Future practice: The online document management system is clearly advantageous in terms of data consistency and security. If the electronic signature issue is also resolved, then the system will significantly benefit the management of the WBL programme, both in the South West and in the other regions in Wales;

Value to others: The approach of looking for a digitised document management system that was based on open source software and freely available online resources would be of interest to any institution with similar goals. It is hoped that the project will also contribute positively to the eventual acceptance of electronic signatures for EU audit purposes.

5