PROWEPersonal Repositories Online:

Wiki Environments

Open University and University of Leicester

(www.prowe.ac.uk)

EVALUATION REPORT

August 2007

Chris PeglerInstitute of Educational Technology

Open University

Document HistoryVersion Date Comments

1.0 2007-08-29 1st draft CP

1.1 2007-11-30 2nd draft CP/AG (repackaging)

Table of ContentsPROWE...............................................................................................................................1Personal Repositories Online: Wiki Environments............................................................1EVALUATION REPORT...................................................................................................1August 2007.........................................................................................................................1Table of Contents.................................................................................................................3Table of Contents.................................................................................................................3Acknowledgements..............................................................................................................5Executive summary..............................................................................................................61.0 The Project.............................................................................................................102.0 The PROWE central research question.................................................................133.0 Approach to evaluation..........................................................................................15

3.1 Use of tracking information...........................................................................184.0 Brief description of institutional contexts..............................................................195.0 Project participants................................................................................................23

5.1 Open University: Project Team.....................................................................235.2 University of Leicester: Project Team...........................................................245.3 Open University tutor participants.................................................................245.4 University of Leicester teaching associates and participants.........................26

6.0 Surveys, establishing the start point......................................................................286.1 How do tutors organise their resources currently?........................................306.2 Do tutors currently reuse materials?..............................................................316.3 Prior knowledge of wikis and blogs..............................................................336.4 Community and professional development...................................................336.5 Community policy and ground rules.............................................................34

7.0 Focus groups..........................................................................................................367.1 The Open University meetings......................................................................367.2 The University of Leicester focus group meeting.........................................43

8.0 Interviews with participants...................................................................................459.0 Participant views on the technologies....................................................................46

9.1 OU PROWE blog usability............................................................................469.2 Uploading into the OU PROWE blog............................................................479.3 Help in using the OU PROWE blog..............................................................489.4 Navigating the OU PROWE blog..................................................................499.5 RSS feeds and finding content.......................................................................499.6 Offline fallbacks?...........................................................................................509.7 UoL PROWE blog/wiki (Plone v 2.1.3) and the University of Leicester users

509.8 Preparing content offline...............................................................................51

10.0 Theme one: Informal repository use......................................................................5310.1 When it’s ‘informal’ will there be high quality?...........................................5410.2 Provenance and personal profiles..................................................................5410.3 Informal tagging systems – the attraction of taxonomies..............................5610.4 Non-formal tagging: Folksonomies...............................................................58

11.0 Theme two: Personal repository use......................................................................5911.1 Different types of publishing space...............................................................5911.2 Versions and back-ups...................................................................................6311.3 Personal profiles............................................................................................64

12.0 Theme three: Wikis and blogs as repositories.......................................................6512.1 ‘Wiki-published’............................................................................................6512.2 Affordances of blogs and wikis.....................................................................66

13.0 Theme four: CPD and sharing resources...............................................................6813.1 How sharing currently happens.....................................................................7013.2 Discipline differences....................................................................................7213.3 Researchers use of personal and informal repositories..................................72

14.0 Sustainability – what happens next?......................................................................7414.1 The Open University......................................................................................7414.2 The University of Leicester...........................................................................7614.3 What level of moderation is required?...........................................................76

References..........................................................................................................................78APPENDIX: Evaluation plan............................................................................................80APPENDIX: Survey questions.........................................................................................83

Open University survey questions.............................................................................83University of Leicester survey questions...................................................................83

APPENDIX: Focus Group questions.................................................................................85APPENDIX: Interview Questions.....................................................................................87

Open University version............................................................................................87University of Leicester version..................................................................................88

APPENDIX Accessibility assessment..............................................................................89

Acknowledgements

I would like to thank all members of the PROWE team across the Open University and University of Leicester as well as the tutors who took part in interviews, focus group meetings and the surveys. I owe a particular debt to the two OU Annes (Anne Gambles and Anne Hewling) for their patience and support at key points, Susan Eales for her quiet encouragement and Gilly Salmon for her great ideas and sustained interest. Roger did all the ‘leg work’ for me at Leicester and he – with Richard Mobbs and Tony Churchill – are key to my appreciation of the very interesting work that UoL undertakes.

JISC should also be thanked for funding the project and the various other very interesting projects within the repository cluster. These provided relevant work for this project to reflect upon and learn from. Particular influences have been the CD-LOR, RepoMMan, TrustDR, and Rights and Rewards projects.

Executive summary

This report evaluates the experiences across two demonstrator communities of practice operating in the course of the PROWE project. The demonstrators were using open source systems, Elgg (version 0.4) and PmWiki (version 2.1.11) at the Open University, and Plone (version 2.1.3) at the University of Leicester.  

The evaluations concentrate on and are restricted to typical and actual use of the demonstrators within the PROWE project. The comments and observations within this report are aimed at the PROWE experience and context and not an analysis of the underlying systems, or comments on their applicability in other contexts.  

The PROWE project looked at the use and potential of personal and informal repositories used by teaching staff at The Open University (OU) and The University of Leicester (UoL) from mid-2005 to mid-2007. Although originally envisaged as a project where the same system and types of users would be studied across both sites, there have been separate systems used and considerable variety across the sites and within UoL in terms of the types of personnel using and controlling the systems and their motivation and uses.

The systems used were Elgg (version 0.4) and PmWiki (version 2.1.11) (OU) and Plone (version 2.1.3), Simple Blog (version 1.2.1) and Zwiki (version 0.58) (UoL) (a good technical description is given in Mobbs, 2007). The two institutional virtual learning environments (VLEs) Blackboard (UoL) and Moodle (OU) were also drawn on by users, as was the FirstClass conferencing system which currently offers OU tutors a route to sharing resources online within and across course communities.

The evaluation approach for the project was modified to take account of this variability (see Approach to evaluation). The most notable findings were:

The contractual arrangement, or what is understood to be the contractual arrangement, for tutors is likely to affect whether and how they share content (e.g. contributors or consumers). Some UoL part-time lecturers may be devising the curriculum for modules which they, and only they, teach. There is little incentive for them to expose this material to others. OU lecturers are very likely to be one of several tutors teaching the same subject. They will however have had little say in the design of most of the course material. They may see themselves as not having the right to share that material more broadly – for example with other institutions where they teach similar courses. These are two of the many IPR concerns that sharing raises in these contexts.

Several users (both sites) reported that they were now making use of online file storage, such as Google Docs or storage available though free email accounts hosted outside the institution (e.g. GMail). This is a recent development and may

explain the relative lack of interest in using either of the two PROWE systems for storage of ‘private’ documents.

Tutors and other academic and research staff interviewed were unlikely to, and in some cases uneasy with, the idea of storing private and public content within the same repository. They were more likely to keep the two separate. For instance keeping private content (not for wider sharing or not yet ready for public viewing) on USB pen drives, on portable hard disks, on a laptop, or online on a completely separate system. This content would be shared by email and so was kept distinct from repository content. This mirrors findings on sharing work-in-progress by email already reported in the CD-LOR Report on Personal Resource Management Strategies.

There was a tendency for tutors at either site to automatically see the PROWE systems as places to ‘publish’. The idea of putting material onto either system that only they could access, or see, was almost counterintuitive. So although this facility was there it was hardly used at all and then sometimes on an experimental, or accidental basis.

The interest in sharing with small groups of friends or project teams, either within or beyond the institution was seen as a significant advantage of personal and informal repositories (Macdonald and Hewling, in press). Having local control over how these groups were constituted and what they could access was considered very important across both sites.

Users at OU and UoL had reservations about the usability, particularly the file upload ease, for their respective systems (Elgg (v0.4) and Plone (v2.1.3)). This may have impacted on the type of use made. In one case a single person within the unit did the uploading of documents for all staff in order to overcome this problem.

While there was considerable interest in blogging and wikis, some users – particularly the UoL users – were more likely to see their usefulness in terms of teaching (i.e. communicating with or as a student), or for research, than for staff development. The spontaneous sharing of ideas and resources with other teachers is still a new and relatively untried idea on both sites although PROWE is credited with having taken this practice forward through a number of initiatives.

There were some concerns that blogging was ‘vanity publishing’ and could be too ‘Café chat’ in style and too variable a quality to be useful.

Having a functional community was felt to be important if credence were to be given to resources and advice offered in blogs and wikis. There is a high risk that if someone were to abuse trust (for example taking and reusing material without attribution or posting inaccurate or inappropriate material), that others would stop posting material and stop using the repository altogether. This is perhaps one

reason why some level of moderation was felt to be necessary – not in organising or controlling discussions and approving content, but in having someone who could intervene and advise if necessary. So informal use is still perceived as needing some level of control, even if never used.

There is a tension between wishing to be in a community, or sharing a repository, with like-minded and similarly experienced people (e.g. tutors on the same course) and interest in ‘leaky boundaries’ between repositories and communities. Having leaky boundaries would allow unexpected serendipitous connections to be made.

There was a tendency to see the usefulness of repository materials as being directed at new and relatively inexperienced tutors. The participants in this project, particularly those who were most experienced, did not usually see themselves as being the recipients, but rather the providers of shareable material.

There were challenges to the project on both sites on how to create and sustain informal communities. At UoL the repository itself was more formal and there is some indication that the blog and wiki tools integrated into Blackboard will prove more popular with part-time and other teaching staff than the Plone system which is perceived not to have this level of official ‘sanction’ as a teaching platform. The project raised questions about how informal use can be facilitated within either embedded institutional systems such as Plone at UoL (which are aligned with formal institutional activities, e.g. Marketing), or projects set up for the purpose of studying informal use over a set time period and within a small project. Informal use, by definition, cannot be managed too directly.

The group blogging and other informal community sharing activities within the OU PROWE blog were seen as its most appealing and interesting aspect. The juxtaposition of informal online communities and repositories is one which could encourage building relationships which support users in knowing more about the provenance of materials submitted.

Where colleagues were co-located, and/or frequently meet, the emphasis is more towards formal use of any online repository. Where there is no clear community with a shared purpose using the systems then there is little incentive to put time into learning to use them, especially if you are employed on a part-time basis and the use is informal rather than linked to clearer more specific or strategic objectives. This may be particularly the case for those academics who see experimentation with technology as something which they should not be expected to do.

Technically competent tutors did not seem to have a problem with switching between different sorts of repository and some mentioned that to maximise their coverage they might use more than one.

There was a preference by some users for an informal folksonomy approach to tagging content.

There is less emphasis than anticipated on sharing resources across courses and institutions than had originally been envisaged.

The personal repository is necessarily a much broader set of resources than the ‘published’ repository. It contains things that were never intended for publication.

The project also raised questions about how informal use can be facilitated within either embedded institutional systems such as Plone at UoL (which are aligned with formal institutional activities, e.g. Marketing), or projects set up for the purpose of studying informal use over a set time period and within a small project.

1.0 The Project

PROWE (Personal Repositories Online: Wiki Environments) was funded as part of the JISC Digital Repositories Programme 2005-7 and ran from June 2005 to August 2007. It was one of two projects funded within this programme to specifically look at personal and/or informal repository use. The other project was the Secure personal institutional and inter-institutional repository environment (SPIRE) project (http://spire.conted.ox.ac.uk/cgi-bin/trac.cgi) based in Oxford University’s Technology Assisted Lifelong Learning (TALL) team. It is worth noting that the original name for PROWE was ‘PROBE’, referring to ‘bliki’ (blog/wiki) rather than ‘wiki’ environments. This acronym was dropped in favour of PROWE, but the commitment to look at blogs and blog-like solutions as well as wikis continued.

The PROWE project set out to look at personal and informal repository use by part-time tutors working at a distance from the institution and within a distance learning framework. This group was identified as one which was likely to have a challenging set of requirements, which could include:

relative lack of time to give to keeping up to date and learning to use new technologies;

poor or uneven access to resources, possibly dependant on location; more than one employment role (e.g. tutoring different courses under separate

contracts) between which they might need to transfer materials and resources; more than one employer and access therefore to more than one set of institutional

resources complex – possibly individual – staff development requirements poorer opportunities to participate in informal sharing of content than full-time

and/or campus based teachers

The project’s broad aim and its ‘central research question’ (taken from the Project Plan of Sept 2005) were to ‘investigate the use of informal repositories within wiki and blogs by part-time tutors on distance education programmes in the partner institutions for sharing and storing resources in the context of their own professional development needs’. (See also 2.0 The PROWE central research question).

The original project objectives were identified as:

Identify the most sustainable technology solution to support the establishment and the development of a proactive community of practice for the partners’ distance education tutors. 1

1 In Autumn 2006 this was modified to refer to ‘proactive communities of practice’ (plural)

Undertake a user needs analysis and feasible metadata model to underpin the development of an electronic collaborative working environment for the part-time tutor community which is both cost effective and non resource intensive.2

Evaluate existing collaborative communication tools leading to the recommendation of a clearly preferred, Web Service based and fully accessible communication tool to support proactive online collaboration for other communities of practice.3

Evaluate the use of informal and personal repositories by staff across both institutions using data capture methods to record the use and explanations of the users for comparative purposes and deeper analysis of use in practice4.

Ensure the full interoperability of the collaborative communication tool with other toolkit resources, making full use of RSS feeds to provide additional ‘continuous professional development’ updates5.

Deliver a shared document repository of a range of case studies in teaching and learning.6

For reasons detailed elsewhere in this report this plan was not fully realised. The project became broader in scope and consequently less targeted than the planned approach suggests. This was mainly a consequence of the differences in context across which PROWE necessarily operated. It is also perhaps an indication that for this group of staff in particular, this type of research project – where diverse participants use technologies which are not yet stable – presents particular challenges, particularly when those uses are a blend of professional and personal/informal.

The institutions involved – The Open University and the University of Leicester – are both based in the Midlands. They were chosen not for their physical proximity, but because they together represent the two largest distance learning providers in the UK university sector. The OU has c.180,000 distance learning students and c.7000 part-time teaching staff and the University of Leicester has 6,000 distance learning students and c. 350 part-time teaching staff. Although distance learning is not such an important form of educational delivery across other UK HEIs, there is an argument that there has been a

2 In Autumn 2006 this was modified to refer to ‘user needs analysis and research; and reference was made to environments (plural) 3 Modified in Autumn 2006 into: ‘Evaluate existing collaborative communication tools leading to the development of guidelines for web based accessible communications tools to support proactive online collaboration for communities of practice.4 The ‘typical user’ was not available and may not have existed within this project. The observation of use was of an accessibility expert demonstrating accessibility concerns. 5 In Autumn 2006 this was modified into: Ensure the interoperability of collaborative communications tools used with other systems resources, making full use of RSS feeds and other options to provide additional ‘continuous professional development’ updates to PROWE sites.6 In Autumn 2006 modified to read: ‘Deliver case studies of the application of communications tools in practice in teaching and learning based on experiences in the two partner institutions’.

convergence of distance learning and campus based teaching with the onset of blended e-learning. Many part-time students who receive instruction through blended e-learning are to some extent distance learners. Also many part-time lecturers will – like their distance teaching equivalents – be only infrequent visitors to the intuition’s main campus, perhaps attending only in evenings, or teaching mainly at satellite campuses. They may work part of their time at other institutions or in non-teaching jobs. There is a clear staff development issue for such distributed staff, particularly in terms of how staff development can be personalised for them and provide an equivalent to informal networks for passing on experience and sharing resources.

Because of the way that the project was introduced and maintained at the two different institutions (see Context and Descriptions and Technology at OU and UoL) it has been easier to produce a picture of institution-wide practice at the OU than it has been at UoL. In part this was anticipated at the start when we realised that at UoL there were 11+ departments offering distance learning alternatives, each with a different approach to supporting its part-time tutors. This range of approaches varies from a part-time tutor who controls the content of a specific module and develops most of the resources with a high degree of autonomy, to one whose role is to mark student scripts and who may have little other contact. Every department in the UoL that offers distance education also offers its part-time tutors different types of teaching contract and different levels of staff support. There is also variation between the arrangements for different types of tutors within some of the larger departments (e.g. the Management School). Throughout the PROWE project, UoL has engaged in a continuing review of such arrangements. It is, for example, currently engaged in a major project to create a flexible management information system to meet the needs of all distance elearning courses.

Another marked difference between PROWE at the two institutions, was the rate of progress in introducing the technology over the two year term. The original plan allowed for one year of operating the project and refining the explorations of technologies and experiences including some evaluation. This was to be followed by a further year during which there would be a more in-depth evaluation. However, the two institutions did not follow the same pattern. Staffing and other problems at the set up meant that both projects were slightly delayed and for both some activities ‘overran’ into the second year. At the UoL there were further problems with gathering survey and focus group information which delayed some of the research. Balancing this, the nature of the technical implementation there meant that the use of the UoL PROWE blog/wiki was on-going throughout the two years, while at the OU the use of the OU PROWE blog and wiki petered out once the active phase was over. The Sustainability section offers more detail on this.

Plone (a content management system, see Mobbs, 2007) was used at the UoL site, while Elgg (v0.4) and PmWiki (v2.1.11) were used at the OU (see Technology at OU and UoL for more information). Although the original project planned to offer a single technical ‘solution’ and focus on part-time teachers working on distance learning courses in both institutions, this proved impossible. The project evaluation therefore focuses on the common themes and experiences across the two institutions. The strength of drawing on

such a variety of contexts, at both the OU and UoL is that common ground between them is likely to also be replicated in and relevant to a wide range of other UK HEIs.

2.0 The PROWE central research question

The central research question for the project was the subject of much discussion during the first part of the project. It was agreed as: ‘In what ways could wiki and wiki-type environments be useful and useable as personal and informal repositories to support professional development within part-time tutor communities of practice?’

As the Approach to evaluation, shows, there has been considerable adjustment in the ways in which this question was addressed. The project has not followed the simpler form that was originally envisaged.

Points to note:

‘wiki and wiki-type environments’. Although wikis were used or linked to on both systems used for the project, there was as much emphasis on blogs as wikis as forms of informal and personal repositories.

‘useful and useable’. Although the original evaluation assumed that there would be a usability study of the single system introduced for PROWE a variety of systems used and referred to by participants in addition to the two separate PROWE systems (i.e. the OU PROWE blog and wiki and the UoL PROWE blog/wiki). These include (at the OU) FirstClass and TutorHome and (at the UoL) Blackboard and its wiki and blog plug-ins (including the Learning Objects wiki7). It would have been difficult, given the variety of uses and users for the Plone to discover anything very specific about part-time tutor use in a technical sense, as the system there was very different to that at the OU not only in terms of the technology but also in terms of the variety of users (full time and non-teaching staff rather than part-time staff) and the range of uses (including marketing courses to students, news bulletins, research and surveys). Instead users of both sets of systems were asked about their use of the relevant system in a general sense and points made about their experiences of the separate systems. An additional accessibility evaluation of the core functions of the two systems was also undertaken during August 2007 (see separate report by Colwell and Pegler, 2007).

‘personal and informal repositories’. Few of the participants had thought in this way about their devices and systems for collecting, organising and sharing content, although many of them used personal devices such as USB pens throughout the project. In terms of PROWE use, there were more examples offered by participants at UoL of use of personal and informal repositories for research than there were for teaching – despite the teaching emphasis of the project. Examples of online teaching repositories tended for all except the most experienced ICT users to be relatively formal and institutionally supported (e.g.

7 This wiki was introduced at the end of the project (from Feb 2007) as an alternative.

Blackboard at the UoL and FirstClass at the OU). There was a clear division between use of spaces which participants knew could be published from (i.e. the PROWE systems) and their personal storage systems. This difference in assumptions about public and private spaces is a significant aspect of the project.

‘to support professional development within part-time tutor communities of practice’. This highlights the biggest difference of experience and practice across the two institutions. At the OU there was a clear understanding of what staff development tutors means across the university (independent of region or faculty). There are already several established communities of practice supporting tutors. Frequently, these are online communities.

Comparisons are therefore more readily made by OU tutors between what PROWE offers in contrast to what they already receive and expect. At the UoL over the period of the project there has been an increasing involvement of part-time tutors in developing communities of practice around their teaching as part of the university-wide professional developments in online teaching. What shape this will take, and how – and whether – it will be able to involve all part-time tutors is as yet open to question. It offers a striking contrast with the OU model and, as has been observed elsewhere, is probably an accurate reflection of what is happening in mixed mode institutions elsewhere.

3.0 Approach to evaluation

The approach to the evaluation of PROWE has necessarily shifted over time (see APPENDIX: Evaluation plan). It assumed that there would be a single system used over both sites and that the participant users of this technology would have similar requirements as part-time employees working at a distance from the institution teaching students via distance learning. The factors to be evaluated were as follows (references to sections are to the PROWE Project Plan (2005)):

1. The potential of informal and personal repository tools with part-time campus-based tutors and part-time (distance and campus-taught) students. (REF section 4)

2. Usability and accessibility implications for demonstrator and toolkit 8 (REF Section 1)

3. Technical adaptability and robustness, including any relevant interoperability concerns with regard to existing OU and UoL systems. (REF Section 2, 3)

4. Use, re-use and re-versioning of content held in personal and informal repositories. (REF Section 3)

5. Long term future of the project and its sustainability. (REF Sections 3, 19)

Where possible the project shared evaluation instruments across the OU and UoL with minimal local adaptation (e.g. with the survey questionnaire for potential tutor participants). It had been intended to share tools throughout, but by late 2005 it became clear that the project would not be developing a common system for use across the two locations. The technical development from that stage on moved in parallel with both systems incorporating some similar functionality, but quite separate in terms of the technology, support and implementation. This led to the decision in Spring 2006, by both OU and UoL, that there could not be a single cross-site evaluation with a common approach. Instead evaluation progressed as two separate evaluations (appropriate to each site and managed during the active stage by the respective Project Officers). These addressed factors 1, 3, 4 and 5.

By Spring 2006 it was also clear that access to the same sort of participants across the two sites would not be possible, although the scale of this problem only became clear in the later stages of the project. It was already proving very difficult to obtain survey responses from part-time distance teaching staff at the UoL, so a variety of options – including paying for participation – were considered. It was clear from the initial survey that responses were not strictly comparable (being a mix of largely full-time staff

8 Note that as no single system was developed/used the usability/accessibility assessment was based on Elgg (v0.4)/PmWiki (v2.1.11) and Plone (v2.1.3).

teaching on campus students at UoL and purely DL tutors teaching part-time students at the OU), although the survey and focus group meetings did provide a better insight into the different contexts in which part-time tutors were operating across the two institutions.

A revised evaluation plan was agreed in mid-2006 with modifications to the way in which the central questions would be addressed. The revised plan took account of the greater complexity of the evaluation required (diverse systems and diverse users) and the lack of data available against some of the points due for evaluation. It simplified the evaluation (removing some redundant questions) and updated the evaluation remit in the following respects:

These questions could no longer be addressed, because of the use of two separate systems/sets of users:

Analysis of interviews and data capture (audio visual and computer screen) records of different tools in use by representative sub-set of users. (As there was no representative sub-set of users)

Technical adaptability and robustness, including any relevant interoperability concerns with regard to existing OU and UoL systems. (REF Section 2, 3)

Usability and accessibility implications for demonstrator and toolkit (REF Section 1)

Because of the limited incidence of sharable resources within the OU system and the diversity of uses of the UoL system it was not possible to address the following in a direct or systematic fashion:

Has there been an improved or altered uptake of shared resources? Use, re-use and re-versioning of content held in personal and informal

repositories. (REF Section 3)

The questions from the original plan which were addressed are as follows:

What alternatives currently exist and how are these used?

How do part-time tutors use these repository tools in practice? What improvements or enhancements would increase/improve their experience so far?

Would users find benefit in having the repository tools adapted so they could fit into their local / community portal?

Which support or curriculum-based factors influence activity or enthusiasm in using these repositories?

Were expected benefits delivered? (E.g. enhanced peer to peer community for sharing ideas and experience and quicker access to material to support their teaching). (REF Section 5)

How appropriate is the demonstrator for novice through to expert users across both institutions? (REF Section 5)

What improvements or options should be considered?

What alternatives currently exist and how are these used?

Is the community self-sustaining and self-regulating?

What resourcing (e.g. moderation, facilitation and staff development support) is required?

Is the regular monitoring and removal of inappropriate material sustainable or scalable?

Many of these questions were explicitly included in the interviews held in Spring 2007. The use of interviews at this stage was another change to the original evaluation plan, necessary when it became clear that the OU tutors could not be brought onto campus for focus group meetings because of their various prior commitments. The alternative was to conduct telephone interviews with seven representative tutors who had each participated in the PROWE project, several of whom [OU T3, OU T4, OU T5 and OU T7] had completed the early survey and attended one of the focus group meetings.

A similar programme of telephone interviews was initiated with UoL. Given the initial problems with setting up a UoL focus group meeting which included part-time distance teaching staff this approach resulted in the best range of views about Plone and the project, but although seven academic staff were interviewed the only part-time distance tutor whose views were collected was Roger Dence (who was also Project Officer for PROWE at UoL).

Following the shift to twelve telephone interviews from two focus meetings, and the greater load in transcription that resulted from this9, this final data collection stage took considerably longer than planned. The limited availability of the accessibility expert also shifted the completion of that part of the report to the end of August 2007. It appears as an addendum to this report. Its coverage is shown in APPENDIX Accessibility assessment.

9 The final transcription was not received until the end of June 2007 and necessitated an extension in the deadline for the evaluation report for the project.

3.1 Use of tracking informationAlthough the assumption of the evaluation plans as they evolved was that information would be available at the end of the project in a variety of forms (e.g. tracking information about on-site activity) this was not available at the University of Leicester site because of the type of system used there. Users of Plone are able to see information relating to their own activity (what resources posted at what date, time of last login etc), but not information relating to use of their resources by others, nor aggregated summary information about patterns of usage. It is worth noting that the current system used to support tutor contact with students at the OU, FirstClass, offers a message history facility. This allows tutors there to see whether specific students or colleagues have accessed or read a particular message. The message history is visible to any user of the system, not just the person posting the message. Some analysis of the OU PROWE blog and the OU PROWE wiki user statistics was attempted to determine which users to focus on in the interviews, but deeper levels of tracking was not pursued because all of the interview respondents pointed out that the information available on the site was not sufficient to require them to make choices between content. They each read all the content that was available during the period that they used the system. It was also clear by this stage that a variety of experimental ‘play’ activities and experiments had taken place that would affect any tracking information.

All messages posted by PROWE participants in the OU PROWE blog and wiki were reviewed for this report and quotations from these have been used. The development of the the OU PROWE wiki and its subsequent use is taken up in the PROWE Case Study 1: The Open University, while the use of the UoL PROWE blog/wiki is explored in PROWE Case Study 2: University of Leicester. A short technical report of the two systems is provided by Richard Mobbs on the PROWE site (Mobbs, 2007).

4.0 Brief description of institutional contexts

The OU is a distance teaching university, so although the project is based in Milton Keynes, and many academic staff, and all the project staff, work there, this is not the ‘campus’ in any normal sense. The contact with students and the delivery of the courses is managed on a regional basis by one of 13 centres across the UK and some tutors will be based in Europe but ‘managed’ from the UK. Tutors are rarely at Milton Keynes and their teaching and day-to-day work is likely to be on premises where they have no opportunity to meet and share experiences and ideas with others teaching the same course. The fact that there may be many tutors across the UK who are all tutoring the same course at the same time and yet may never meet or communicate with each other is highly unusual. It is a very testing environment for sharing informally.

This is in contrast to The University of Leicester which is located on a fairly conventional, geographical centre with a main campus where teaching of non-distance versions – and some distance learning versions – of its courses takes place. At the UoL (unlike the OU) there are many more conventional lecturing staff and many more campus taught students (18,000) than there are distance learners (6,000). Staff who teach or support students in distance learning mode may also have other on-campus roles and therefore access to on-campus systems. They may work for UoL full time, operating across a portfolio of roles. There are – at the other extreme – some UoL part-time distance learning tutors who perform a limited and very specific range of tasks. Declining the offer to take part in the initial survey one of these commented: ‘I don't actually teach at the university, I merely mark essays and supervise dissertation students’.

Because OU and UoL part-time tutors may be working at home while tutoring, or may be tutoring while working from several different locations, they can have different and more complex technical requirements than those of the office-based lecturer. The on-campus lecturer will probably have an ‘office’ setup, where their desktop machine is for their sole use and gives them easy access to the institutional resources and services. The part time tutor might be carrying a machine with her so that she can work where she can – examples mentioned by OU tutors included logging on while away from home at dinner parties and whilst babysitting. Even having her own machine with her may not always guarantee that the tutor can get online. This will fundamentally affect how useful any online service is to that tutor:

I think if you’ve got a situation where you’ve got a desktop at the office and a desktop at home and things like that, then actually having a web repository actually makes sense. The way my life works at the moment it’s not like that. I work for the OU as an AL but when I go up to Milton Keynes if I don’t take my laptop, I don’t have a computer unless I go up to the library so a web repository isn’t much use really. When I work for exam boards it’s the same situation. I won’t have access to their desktops up there, so if I want a computer I have to take my laptop. If I’m lucky I’ll be able to log into a wireless something

somewhere. I can’t guarantee that, so I’m much better having it on my machine. [OU T2]

We also cannot assume that the laptop that the tutor uses is for their sole use. This may be another reason for wanting to store some things online, again a reflection on access problems. This can impact not only on the part-time tutor, but also on the campus-based lecturer who does not work solely from his office:

I do tend to work at home a lot and we’re now up to three computers at home. Depending on which computer I can prise members of my family away from then I don’t know where I’m going to be and so it’s much easier to have access to things online and not have to worry about what’s on the hard disk. No, having said that, the background to all of this is that I have a lot of data that I wouldn’t consider putting anywhere apart from my disk with my password so it is a bit schizophrenic really. [UoL T4]

In terms of searching or browsing, with the OU PROWE blog and wiki there was a clear sense of this as a project with some time allocated for playing around. However, the reality for part-time tutors is that they have little time for undirected exploration of this type. This sort of informal exploration is an unusual thing to ask a part-time tutor to do and some tutors were less comfortable with it than others. As one OU tutor puts it, she would sometimes would look for things and find them through ‘sheer nosiness’ but preferred a search with a purpose: ‘It’s easer if I go off searching, because I’ve got something in my mind than just browsing for the fun of it.’ (OU T2).

Some of the differences in responses between OU and UoL users might be attributed to the way in which they were each funded to work within their respective systems. At the OU all the ALs received some degree of funding to be part of the project, in particular to play a part in the evaluation (e.g. for time spent attending focus group meetings and being interviewed). At the UoL most of the users of Plone are full time staff who are using it for formal purposes not connected directly with the PROWE objectives (e.g. for marketing courses, or for research). Some of the UoL part-time teaching associates (who did not receive payment and are largely unrepresented in the end-of-project interviews) made comments at the start of the project clearly indicating that they were not paid to interact with other staff and so would be unlikely to do this (see Surveys, establishing the start point).

It was interesting to hear the views of staff who worked across the two institutions. Roger Dence was one example of this. So was UoL L1 (who also tutors for the OU) and OU T4 who has recently taught at UoL:

my bottom line is that you have to have a purpose. Because we are all busy and nobody’s going to sit down and do this unless they absolutely have to, unless they are the type of people who keep diaries and write for themselves for pleasure. So I can’t …. If you think of terms of ALs [OU Associate Lecturers] we don’t have time, but I’d do it if was paid. I use my [OU FirstClass] conference which is

absolutely adequate for what I need to communicate with other tutors on the course. I could see its use if I was developing a paper, I can see a lot of use for it in teaching, if only my students at the Open University – who are distance students – had this resource we could use this together to develop something. I can see it being marvellous. But for my own communication it has no purpose. [UoL T1]

One thing to bear in mind is that there is a difference between the practice and the requirements of tutors who teach at a distance and part-time and those who teach wholly on-campus, or wholly face-to-face. One of the differences which appears very important to the tutors themselves is who creates most of the content. There is sense in which DL tutors see themselves as teaching using other people’s resources which they are not empowered to change, or add to. ‘There’s probably less push to create with the OU than there is with the other university because we’re not creating the course and that’s what were obliged to use and it’s only if you’ve got extra energy or extra time that you might create’. [OU T3]

It is not required that OU tutors or many of the UoL tutors should create their own teaching resources, but it is generally recognised that some will do so and that there will also be variation in the ways in which the ‘received’ resources from the course team are reused and repurposed ‘in the field’. Exchanging ideas on how to teach a course, and exploring alternative resources has been the focus of many Tutor conference (forum) discussions in the past. As online teaching has developed it has become more common for tutors to exchange links with each other and with their students, often within the password protected institutional systems, but sometimes using proprietary systems. A blog developed by an OU tutor was mentioned on in the OU PROWE blog (Anne Stott’s blog on the Arts course A207 Enlightenment to Romanticism) http://enlightenmenttoromanticism.blogspot.com/ . (Anne was not a participant in this project although her use of this blog shows some of the behaviour that we were trying to explore).

At the OU there are already – in addition to the TutorHome intranet, online conferences (forums) in FirstClass where tutors can meet and exchange ideas. This was a point for comparison which was lacking on the UoL side and which the OU tutors returned to several times in the later stages of the project – contrasting what PROWE offered and what they were already used to. For example a message on the OU PROWE blog responding to the question ‘Have you changed anything about your personal resources management strategy?’ drew this response from one tutor (later interviewed as OU T5):

‘Not really . The problem I found is that there really is not anything here I found which I can’t do in FirstClass better – except perhaps for the inclusion of the photo on the messages, but that is on my FC resume. The only thing that is different is making ‘open to all’ postings and communities.’ He goes on to

comment that it is possible to create conferences (forums) in FirstClass which could address this, but that tool is only ‘for the select few’10.

Comparisons at the UoL tended to be to Blackboard, which is the institutional online teaching tool. However, notwithstanding the comments above, the control that a tutor has over the VLE is very limited and based primarily on working with students within courses. The freedom that Plone offers to create sites to support projects and communities is seen as a significant gain by many, particularly those interested in its potential to support personal and group research and development rather than teaching.

10 The decision on whether tutors can create their own conferences and sub-conferences (different levels of online fora) in FirstClass is taken at the level of each course team. While the default is for the central academics to determine which online conferences exist in the course, some courses allow all tutors to create their own sub-conferences.

5.0 Project participants

5.1 Open University: Project TeamThere was an unanticipated level of turnover in staff on the OU project team over the course of the first year. This was the period during which the project was in its most ‘active’ phase.

Project Directors Anne Ramsden Left OU in Jan 2006

Susan Eales Replaced Anne Ramsden in June 2006

Projects Managers Anne Gambles

Jane Lamberton provided maternity cover for Anne (Jan-Sept 2006)

Project Officer Anne Hewling October 2005 – October 2006 (full-time) and then as occasional consultant

Metadata consultant Lara Whitelaw

Evaluation consultant Chris Pegler

Accessibility consultant Chetz Colwell

Staff Development advisor Janet MacDonald

Technical developer Murray Altheim June 2005 to Feb 2006

Web developer Mark Stride

Project Administrator Jeanette Stanley October 2005 – April 2006

Telephone interviews with Anne Gambles, Susan Eales and Anne Hewling were held in April/May 2007 as part of the project evaluation.

5.2 University of Leicester: Project Team

Project Leader Gilly Salmon

Research Assistant Roger Dence

Technical developer Richard Mobbs

Staff Development advisor Tony Churchill

Metadata Advisor (UoL) Selena Lock June 2005 – March 2006

Joanne Dunham Replaced Selena Lock

The availability of tutor/teacher participants for the two projects, throughout the two projects was largely realised in the OU project but not at University of Leicester (see Context and Background section for a description of key differences between the institutions).

Telephone interviews with Roger Dence, Richard Mobbs and Tony Churchill were held in April/May 2007 as part of the project evaluation.

5.3 Open University tutor participantsTutor participants for the project were initially recruited through a notice on the TutorHome intranet site which all OU tutors can access. 40 tutors made contact and 16 went on to complete a survey as part of the requirements gathering stage (see PROWE: Understanding the OU user perspective (Hewling, 2006a) for a detailed analysis of responses). 18 of the tutors who had completed surveys attended one of two focus group meetings in December 2005 (see Focus groups). Some of these tutors, with others unable to attend the meetings, became users of the OU PROWE blog and wiki.

Tutors involved at the OU side of the PROWE project have been to a large extent self-selecting. Although analyses of attendance and survey completion show they do represent a fairly heterogeneous group, we do see some bias towards online tutors and subjects where online communications are a feature of the course (e.g. M150 and T17511). The ICT skills of the tutors involved in the project were therefore at the high end of those you would expect from part-time tutors, and that experience is likely to affect how they view the PROWE project and what that wanted and took away from their experience of it.

11 M150 is ‘Data, computing and information’ offered by the Maths and Computing faculty and T175 (the successor to T171 Error: Reference source not found) is ‘Networked living: exploring information and communication technologies’ offered by the Technology faculty.

I for some time have [had] my own web page on its own domain and for some time I’ve had the idea of developing it into a repository but it’s never actually happened. So the Elgg12 thing was a kick up the backside to get on and do something along those lines and it also enabled me to experiment with blogging. [OU T6]

In the final evaluation stage the seven tutors interviewed were drawn from those users who had completed surveys and used the OU PROWE blog and wiki (i.e. with some activity showing in their online log). The selection of tutors invited to participate at this stage reflected a wide range of courses, experience and regional location with a balanced group by gender. Briefly they were as follows:

OU T1 (Female)

Tutors seven Maths courses including two that are part of the Graduate Diploma in Mathematics Education, two first level and one entry level courses, a residential school, and the Openings (access) course in Maths.

OU T2(Female)

Tutors two Openings (access) courses Open to Change and Understanding Health. Is also registered as a research student and so has her own OU tutor for this course.

OU T3(Female)

Tutors two computing courses (Data Computing and Information (M150)) and also a course – same faculty – on Java programming a second level UG course.

OU T4(Male)

Tutors Data Computing and Information (M150) and also acts as co-moderator (a ‘super’ online tutor) for the Technology faculty course The client-side of Application development a second level UG course.

OU T5(Male)

Tutors two PG courses (British Cinema History 1930-1995 and Dissertation module (History)), a third level UG course (Film and television history) and the online version of the foundation level course Introduction to the Humanities (AZX103) – all for the Arts faculty.

OU T6(Male)

Teaches the foundation level course Introduction to the Humanities (AZX103) and an equivalent foundation course within the Technology Faculty (Networked Living: Exploring Information and Communication Technologies (T175). Also – with OU T7 works on Tutor toolkits – used as staff resources for professional development and induction of tutors.

12 This refers to Elgg v0.4.

OU T7(Male)

Tutors on Data Computing and Information (M150) and Networked Living: Exploring Information and Communication Technologies (T175).

Note that at the OU computing courses may be taught by staff in the Maths and Computing Faculty (e.g. M150) and also by the Technology (Telematics) Faculty (e.g. T175). It is common for staff to hold contracts from more than one faculty and six of the seven interviewed are in this position.

The OU tutors between them held contracts for tutoring in the West Midlands, East of England, North, North West, London and South East and one tutor (OU T1) held contracts in four regions. This multi-region employment pattern is more common for low population courses.

5.4 University of Leicester teaching associates and participantsAlthough initially access to equivalent participants at the UoL was assumed feasible, this subsequently proved very difficult. There is no central database of tutors from which to gather information on backgrounds and no obvious way to make contact with them. Distance learning activity at UoL is not centralised and even though the number of tutors (teaching associates) was substantial – c. 350 across 11+ departments (Hewling and Dence, 2006). Contact with tutor participants at the UoL has therefore been made on a one-to-one basis through the UoL Project Officer Roger Dence, himself a part-time tutor at both the OU and UoL.

It has been necessary, as the project has progressed, to accept substitutes for the part-time tutors that we initially imagined were largely responsible for DL tutoring at UoL. We now know that fewer UoL tutors are directly comparable with OU associate lecturers than was first thought (this point is made in the final PROWE Project Report). It was also not possible for UoL tutors to be recompensed for their time in the same way as those at the OU. This may have accounted for the level of attrition over the two years, although it is also unlikely that UoL part-time DL tutors will have the same level of motivation to use the system for staff development as their OU counterparts have, given the lack of payment for and recognition of this activity for UoL teaching associates.

While 52 tutors and other teaching/advisor staff at UoL were invited to complete initial survey, only 17 completed this and only five of these were part-time DL staff (see Surveys, establishing the start point for more on those responses). None of those who completed the survey were available for interview at the end of the project although one of the original focus group participants was interviewed (UoL T4 below).

UoLFG (Males x 2 + Female x 1)

Three teaching/research staff, who although full-time UoL staff are located in Leicester Royal Infirmary – i.e. a short distance from the main campus.

UOL T2(Female)

Works full-time at the UoL as a research associate within the Beyond Distance Alliance team (since Sept/Oct 2006) and also tutors part-time for the OU. Her research also extends to working with UoL teaching colleagues on an action research teaching development programme. User of Plone.

UoL T3(Female)

Full-time lecturer at UoL (Company Law taught across several courses). She has taught at UoL for 10 years. This is the only HEI that she has experience of. Has used a wiki to support a student tutorial. Her only DL contact is as a PhD supervisor. Most of her students are full-time and campus-based.

UoL T4(Male)

Full-time lecturer in Biochemistry, teaching a number of different modules at UG level. Has taught using the web for 10 years and runs his own web server. He has recently taught/assessed students using wikipedia. Did not use Plone.

UoL T5(Male)

Professor in Languages department with extensive experience of using Blackboard and some of using Plone. Full-time staff member and campus-based with no direct involvement with DL.

UoL T6(Male)

This response – from Roger Dence, Project Officer for the UoL side of PROWE – was submitted by email. He is the only part-time UoL DL tutor to have been ‘interviewed’ at the end of the project. His area is Management and he also is an OU AL.

6.0 Surveys, establishing the start point

As part of the process of better understanding the requirements of potential users of PROWE a short survey was circulated in the early part of the project (Autumn 2005). This contained questions about current practice in use of technology and also in approaches to and needs for sharing of resources. At the OU the questions were devised by the Project Officer Anne Hewling in conjunction with Chris Pegler. These were then passed to the UoL Project Officer, Roger Dence, who created and circulated an extended and amended version over the course of several weeks (Autumn 2005 into Spring 2006). (See APPENDIX: Survey questions for both OU and UoL versions of the questionnaire).

Comparisons of the results of each of these are interesting, and reveal something of the differences between the two institutions and their tutoring and IT arrangements. Before exploring these, the differences between the two sets of respondents needs to be noted.

At the OU only Associate Lecturers (ALs) – part-time teaching staff – were invited to complete the survey. Those targeted were the 40 ALs who had responded to information about the project posted to TutorHome (the online space which all tutors automatically access when they log into the OU system). This message gave brief information about the project, invited participation in the focus group meetings and payment plus travel expenses for attending these. It invited those that were interested to contact the Project Officer and pointed out that there would be other opportunities to be involved for those unable to make the meetings.

Of the 40 OU tutors invited to complete the survey 15 did so and their responses are detailed in PROWE: Understanding the OU user perspective (Anne Hewling, 2006a). The group who responded to the message and therefore those who expressed interest in the project were all more likely to be teaching online courses already, since tutoring online increases the number of times that a tutor will log into the OU system over any given period and thus increased the likelihood of seeing the message about the focus group meetings. These tutors may not therefore be representative of less ICT-competent OU or distance learning tutors.

At the UoL, 18 responses to a similar survey (see Appendix 1) were received and 16 of these were usable13. However only seven of the responses were from part-time teaching staff – the rest were from full-time staff, most of whom were campus-based. (Two of the full time staff were teaching at institutions other than UoL, although they had some connection with UoL – e.g. as a member of the Colleges-University of Leicester Network (CULN). One of the part-time staff although involved in distance education was an advisor rather than a teacher14). Only nine of those UoL staff who responded had a

13 One file was corrupted and one file respondent although answering the questions asked that their responses not be used in this research. Both these responses are omitted from the analysis and percentages here. 14 The ‘advisor’ response is also removed from the analysis so that each set of responses is drawn only from teaching staff – 15 from each institution.

distance teaching role, plus one who supervised DL dissertations. Only five of the UoL respondents matched the profile of users within PROWE15 (i.e. were part-time DL tutors16).

The arrangement for gathering expressions of interest was also necessarily different at the UoL as there is no central ‘TutorHome’ equivalent for all part-time DL tutor there. The survey responses were obtained by Roger Dence through a variety of different approaches. In all he contacted 52 tutors and teaching staff (34 of these did not respond despite chasing and 6 refused directly to take part). The participants here were therefore not ‘opting in’ in quite the same way that the OU respondents did. One responded by saying: ‘Here is my form. Sorry it is rather negative. I am afraid I am suspicious of the University's desire to 'harvest' material from its paid by the hour staff and I don't feel myself that I have spare capacity for taking an active part in such a 'virtual community'. In addition to obtaining a range of responses (from enthusiastic to rather cynical) this does make it unlikely that the survey at this institution was completed by anyone not previously known to the UoL team.

At both institutions staff from a range of departments and faculties completed the survey, although that range is smaller at the UoL when considering only responses from part-time and distance teaching staff so both sets of responses are recorded below.

Variety of respondents (range of discipline areas served and amount of experience within that institution/role):

OU (n=15)

UoL (All) (n=15) UoL (DL and part-time only) (n = 5)

Faculties represented

Arts, Business, Health and Social Care, Maths and Computing, Science, Technology

Computer Science, Criminology, Culture Innovation and Education, Education, Engineering, Law, Management, Mass Communications, Museum Studies, Physics (+ Astronomy), Social Sciences,

Criminology, Management, Mass Communications

Years as an AL (OU) orYears at this institution (UoL)

2-207.4 years average

1-144.4 years average

1-83.8 years average

15 One of these is an MSc external assessor at UoL. Their involvement in teaching there is not stated. 16 None of the UoL full time staff were also tutoring DL courses although this does happen and one of them was a supervisor of at least one DL PhD.

In terms of personal technology use and online connection both the OU and UoL respondents were similar. They were using broadband or network connections rather than dial-up access – only one tutor at the OU and none at UoL were reliant upon dial-up access. They also used similar technologies (i-Pod, PDA/Palm, iPaq, Blackberry and laptops) although the OU tutors mentioned a wider range of technologies including mobile phones.

6.1 How do tutors organise their resources currently?The range of ways in which respondents across both institutions organised resources was varied (USB pens, CDs, hard drives on desktops, files on laptops, personal websites and using email). The most notable difference was that at UoL there was more mention of use of Blackboard (their institutional VLE) and use of IT services, saving to the central or departmental servers (‘x’ and ‘z’ drives)17. This reflected the numbers in this group of respondents who were full-time staff and so had best access to on-campus services. None of the teaching associates at UoL (the equivalent of OU ALs) used the VLE or IT services for storage. The UoL respondents did not report any other use of online storage (compared to the OU tutors’ use of FirstClass, blogs and personal websites).

How do you organise any digital resources you may use in your teaching?

OU (n=15)

UoL (All) (n=15)

UoL (DL and part-time only) (n = 5)

Laptops 2 3 1Desktop 5 3USB pens 1 2CD 1Email 1 1 1Beamer 1Website/blog 2FirstClass (OU only) 2 N/aBlackboard (UoL only) N/a 6IT services N/a 3Powerpoint/Excel/Word 1 1None/Nothing to organise

2 3 2

Invalid answer (not specified)

1 1

17 At UoL individual z drives were made web accessible early in the life of the project and departmental x drives were made web accessible towards the end of the project period.

This table records the numbers of mentions of each approach – several of the respondents used several different types of organisational strategy.

For both sets of respondents the use of the term ‘learning repository’ was unfamiliar. It is doubtful – looking at the interview responses – that they would have understood the term ‘teaching repository’ either, although this question was not asked. At the OU seven said that they did not have experience and four said that they did not know what the term meant. At UoL, eleven said that they did not have experience of one (including some who had stated that Blackboard was a way in which they organised their own resources), one said that they did not understand the term. A few (three at the OU and two at UoL assumed that this would be a VLE such as Blackboard). Two of the UoL teaching associates (part-time distance learning tutors) familiarity with systems outside the UoL which they felt could be learning repositories – one cited LEARN at Loughborough and one mentioned FirstClass at the OU.

Tutors were asked how they currently transferred resources between sites. The answers indicate that this did not happen at all at UoL and only occurred occasionally at the OU. However, as the OU focus group meetings revealed (see 7.0 Focus groups), part-time tutors were very cautious about the legal situation of transferring resources from one institutional context to another (four of the 15 OU tutors said here that they did this but may have been referring to teaching at local colleges for the OU). At the UoL, the responses will have been influenced by the large proportion of full-time staff. It may be that many of these respondents (from both sites) do not work outside the OU or UoL, so have no need to transfer from one context to another.

6.2 Do tutors currently reuse materials?On the basis of this survey the answer would appear to be ‘Yes’, most often their own material, or material which they have used in the past. Materials which were felt to be successful were very likely to be reused in some form, sometimes with updating, sometimes with ‘top-and-tail’ modification. As Hewling in her analysis18 observed ‘reuse19 occurs much more often with successive offerings of the same course; i.e. reuse is concentrated in the same teaching/learning context, rather than being used for the development of new contexts’. Reinforcement of this point appears in the UoL responses, for example: ‘I save them on Blackboard for the following year; they can be easily amended if necessary’.

It is worth pointing out that the two UoL respondents who claim not to reuse currently were both teaching associates (part time, distance teaching staff). A third teaching associate (from Criminology) offered this comment: ‘All the materials I use are produced by the University in the form of modules, so they are reused on a constant basis’.

18 PROWE – Understanding the OU user perspective (September 2006) 19 Reuse in the sense used here encompassing minor and even major revisions to material which would quality as ‘repurposing’.

Do you like to reuse teaching materials? If so, when and how do you do this?

OU (n=15)

UoL (All) (n=15)

UoL (DL and part-time only) (n = 5)

Reuse/Repurpose material which I created or have previously used

11 12 2

Reuse/Repurpose materials made by others teaching the same course

1

The institution provides the material and I reuse it.

1 1

Reusing materials from other universities

1

New staff so no reason as yet to reuse.

1

No/Not applicable 2 2 2

Reusing materials produced by colleagues may not be easy for part-time staff who work largely on their own. As this part-time distance learning assessor says: ‘I have no contact with other markers and assessors. I would like to but there is no facility for doing this. I have direct contact with dissertation students and some contact with full-time staff’. Full time lecturers have a wider range of options and opportunities: ‘I share a core class together with two other lecturers. So we share a textbook and write exam papers together’. Full-time staff have the advantage of physical proximity to others teaching in the same area, and also a level of personal control over the curriculum that associate lecturers and teaching associates do not generally have.

The most striking statement made by a respondent was possibly this by a UoL part-time tutor who was not involved in distance education. In response to a question about what types of material they would share with others: ‘I would only do this if got paid for my time, because I am paid per piece of work. I would like to share research material, but that hasn’t really got much to do with the teaching I do’.

The difference in perceptions of reward and opportunity between different categories of teaching staff is marked. It is worth considering what the response of the OU participants would have been if they had not been paid for their involvement in PROWE. Realistically it is difficult to motivate part-time tutorial staff to give unpaid time to explore innovative approaches which may not help them in any very direct way. Teaching staff at the UoL were much more likely to see the potential of the PROWE technologies for sharing research ouputs and ideas (i.e. working with and delivering) resources to students than

they were as a form of professional development (see 13.0 Theme four: CPD and sharing resources, for more on this).

6.3 Prior knowledge of wikis and blogsAs anticipated, some of the respondents were not familiar with wikis and/or blogs at the start of the project. In the words of one respondent: ‘The prospect of a wiki/blog being part of my work is so beyond my present experience I can’t really answer this question’.

The question about how wikis/blogs would be used was interesting for throwing up the differences in how teaching staff perceive these two tools. The blog was seen by some as a less formal and more personal tool ‘generally, full of opinionated ideas’ while the wiki was seen as more clearly having a purpose for sharing and working collaboratively – compared favourably by some to standard tools using track changes to note alterations. This difference in views of the two types of tools and the behaviours that they afford was reinforced in the interviews at the end of the project.

Several of the comments made here – and throughout the project – dwell on sharing with students in the sense of teaching students. Several comments were made on how to use blogs and wikis as teaching tools – which was outside the scope of this project.

Two respondents – both from the UoL – made good points about the coverage of a wiki:

‘I am interested in a narrow subject area wiki-type resource. I think it [a wiki] is only sensible for a much larger community’.

‘Maybe but we are diverse and resources needed for Computer Science are different from others. How will this benefit me?’

In later interviews with OU tutors several felt that the size and specificity of the community played a large part in its success as an informal repository. In neither institution did the project tap into a sizeable specific community to test out the usefulness of PROWE, relying instead on the creation of a community around the project. This is in retrospect a significant weakness albeit a necessary trade-off when trying, as this project did, to look at practice over a range of different discipline areas and with users ranging from ICT novices to experts.

6.4 Community and professional developmentThere was enthusiasm at the survey stage for the idea of using blogs and wikis for community building and using online tools in a social but work-related way. Additionally, there were several comments from OU tutors which indicated enthusiasm for the idea of learning from each other – clearly seeing the project tools as being as much about exchanging ideas and experiences and views as about sharing resources. The following comments about expectations are from different OU tutors:

‘The more experienced can learn things from the new just as much as the other way around. I see such interaction as offering a valuable input to the mentoring scheme, for example, which at the moment seems a very good idea which is dealt with in a rather casual and piecemeal fashion.’

‘Helping peers and learning from peers – essential continuing professional development. In OU terms it definitely helps to cut down on the isolation felt by ALs – problems with materials and/or students are more manageable when shared with other tutors. I’ve had some good advice provided to me on how to cope with a Tourette’s Syndrome student in a tutorial – common sense really, but very helpful indeed’.

‘Sharing teaching materials and not reinventing the wheel’.

‘Finding out how others are approaching a particular task etc. and being able to compare practice’.

The UoL responses mirrored the same themes. For example:

‘Discussing methods of interaction with students. I see myself as a facilitator rather than a creator of resources. I add value from my non-academic working life experiences and from showing practical application of techniques and concepts.’

‘For a distance learning tutor, breaking down the distance barrier’.

‘Share ideas, support’.

6.5 Community policy and ground rulesThe surveys contained a question asking respondents what type of community policy or ground rules they would expect to be attached to the wiki/blog used in PROWE. This was a very interesting question to pose at this stage as none of the respondents had seen or used the PROWE systems on either site and some of them did not know what a blog or wiki was, or had no idea what they might use it for.

One common theme across both institutions at this stage was concerns about the legal position and for some level of moderation, including self-control. These are some of the ‘ground rules’ suggested:

Developing a common understanding of IPR, including using credits for the resources (e.g. ‘written by …’) and no appropriation without consent.

Respect for others views, with no libel and no offence allowed. Perhaps going as far as to exclude abusers.

Professionalism, which includes responsibility and relevance (i.e. not wasting people’s time)

Keeping in line with institutional policies on diversity and accessibility features applied elsewhere.

Observing standard netiquette (general rules of good behaviour online). Appropriate security – ensuring that private/restricted content stays

private/restricted. Respecting confidentiality and terms of employment, so that constructive

criticism can be aired without fear of reprisal. A friendly tone (mentioned by one OU tutor only). No anonymity allowed (mentioned by one OU tutor only).

More suggestions were received from the OU tutors than from those at UoL and perhaps this derives from their more extensive experiences of online forums for tutors.

7.0 Focus groups

After initial discussions in the Project Team and Steering Group about the technology options, the project staff at both institutions arranged separate focus group meetings with potential users of the blog and wiki options20.

7.1 The Open University meetingsAt the OU it was necessary to bring staff onto the campus at Milton Keynes, to meetings held on Friday 9th and Saturday 10th December 200521. Most tutors (13) attended the first meeting (Friday) and a smaller group (5) attended the Saturday meeting.

The evaluaton consultant (Chris Pegler) acted as facilitator while the Project Officer (Anne Hewling) took notes and demonstrated blogs and wikis. Jeanette Stanley (Project Administrator) also took notes. Although arrangements were made to create a digital recording, and consent for recording obtained from the participants, the technology failed and no audio was obtainable from either event.

The discussion was held around a set of trigger questions prepared by the evaluator and project officer (see) and was preceded with hands-on demonstrations of several different wikis and blogs. Tutors were specifically asked not to identify the specific courses that they taught on in their answers to questions, so that the group did not split into recognisable discipline groups from the start. It was however impossible to avoid mention of specific courses where particular approaches to sharing resources already prevailed.

A summary of responses is shown here:

1. How does sharing happen now?

o Currently, for these tutors, this is primarily through FirstClass22 when talking about sharing with other tutors. The OU ALs are familiar with using FirstClass (FC) to request and exchange resources. One tutor used FC mail to keep all shareable resources – although several others had more sophisticated solutions which included personal webpages and blogs (used

20 At this stage the precise technical configuration of the OU system had not been decided. 21 A choice of a Saturday meeting slot was offered because some tutors work full-time in non-OU posts and this would make engagement by these easier. It was however the least popular option amongst those tutors who chose to attend. It is perhaps reasonable to assume that tutors whose OU tutoring is additional to full-time work elsewhere would be unlikely to engage with a project which offered yet more work, albeit paid work. 22 FirstClass is the online conferencing system used at the OU by tutors to communicate with their students and, to a less extent with their peers. It incorporates an email facility (in the past the usual email account of tutors) and also a number of tools such as instant messaging, voice messaging, searches within messages and conferences, resumes, etc).

primarily with students and often as part of courses which themselves teach webskills).

o One course in particular (T171: You, your computer and the Net 23) had had a very large influence in encouraging use of the tutor conference to comment on and ‘improve’ course resources and develop additional tutor resources. Several tutors from T171 had now moved to M150 and they (and others) recognised that this type of sharing was not yet common at the OU.

o Another course (this time an online Arts foundation course) has a tutor conference where tutors can and do volunteer information. A moderator organises and ‘keeps’ the resources.

o In a Film History course the moderator has set up sub-groups in the online version of the course to create a ‘best practice’ collection or library of resources. There was interest in this suggestion and the potential of sharing tutor reviews of third party materials.

o There was a perception that different regions are reinventing the wheel. It was not currently possible to see (expect in rare cases) what resources are being used within different regions if the tutor conferences are organised regionally.

o Cross course re-use is not possible/not feasible. This is in part because of the structure of OU courses, but it is also impossible for tutors to see what is being used on another course within the same or similar programmes unless you are also tutoring on those other courses.

o Tutors would always try to keep their own ‘back up’ versions of original documents. All of the ALs present did keep local copies. For those that only used one machine this was – they felt – sufficient to be their personal repository.

2. Issues around sharing

o Examples of what tutors might find useful to share ranged from the highly specific: ‘This is how to approach X activity in Y block in Z course’ to ‘How to mark online using the eTMA system24’.

23 T171 was the first OU online course with large numbers of students (12,000 in its first year of presentation) and a tutoring population that numbered several hundred – many of them newly recruited as there was no previous version of this course. The staff development needs of those new tutors and the universal use within this course of online communications opened up the potential to share teaching advice and resources online using a central or regional course tutors’ conference. 24 The eTMA system is the OU’s electronic Tutor Marked Assignment system which is gradually being rolled out to all courses – a process which has already taken several years.

o Some types of courses have very specific needs – the Openings courses25 were mentioned. The problem of sharing resources which might not be suitable was considered – students in prison were specifically mentioned as presenting challenges in reusing/adapting generic materials.

o There were issues around reusing materials generated outside the OU within OU courses. This was felt to be discouraged, perhaps even forbidden, by course teams who expected tutoring to adhere to the materials they had specified/provided.

o The use of OU materials beyond the OU was felt to have some awkward rights implications although it was acknowledged that OU materials were much more in the public domain than teaching materials from other HEIs because they were often already available as separate print publications.

o FirstClass was seen as having significant weaknesses in terms of the potential to co-create documents.

o New ALs have the most to gain from sharing by others, but sharing/support is the hardest for them to get (due to lack of confidence and no knowledge of who/what to ask etc.) Some of the newer ALs attending made the point that they did not actively contribute (comment on or post messages to) tutor conferences for several years. They would not want to join a community where it was expected that everyone would post or comment within a set period (one of the suggestions made to enhance activity).

3. How would they like to share?

o Some tutors would like a culture where if you create something you share it (as happened in a secondary school where one AL worked). There were mixed feelings about this. Some resources they would share and some they would keep personal. The idea of a Sandbox/Repository split would encourage them to put things up (in the Sandbox) that they were uncertain about the quality of. Otherwise there may be nervousness. Many already share although there was a certain expressed frustration about making resources available to the Region at the end of each year but with no onward dissemination to other tutors or beyond. It was clear that regional practices/approaches vary.

o The tutors wanted to use any system like PROWE to share with students and not just ALs. Some of the ALs could see the value of using an online

25 Openings courses are part of the access programme for the OU. There is more intensive tutoring and students may need more tutor support as they are likely to have very little prior educational experience.

repository as opposed to the hard disk because they could direct students easily to an online resource. There was discussion about permissions and partitions so that different areas would be made more or less visible/usable by different groups, but not resolution on how to do this26.

o Some tutors saw social sharing as something that helps overcome physical remoteness (a particular problem on smaller population courses where there are fewer ALs, e.g. some at post-foundation or masters level). Several ALs attending the Friday meeting had not met other ALs before and some of those there on Saturday 10th December were tutoring the same courses but had not previously met.

o Sharing with peers could offer a peer review process that some ALs – after first year mentoring – felt that they lacked. This led into an interesting discussion about star ratings, whose opinion you could trust, what sort of comments would be valuable27. An example of a valuable comment would be contextual information, e.g.: ‘This is really useful for ****, I adapted it like this ****’ etc.

4. Do they want the active participation of others in production of their documents?

o In the larger group (on Friday, 9th December) there were concerns about quality and nervousness about posting something up as shareable which might not yet be finished. Fewer reservations were expressed at the Saturday meeting (see Sandbox comment above) with that slightly more ICT experienced group. There was openness to refining of resources with others on both days.

o Tutors wanted to be able to (legally) keep original version as well as adaptations, so preferred a system which allowed storage of multiple versions. They pointed out that the earlier version of something might be what you wanted, so it should be made easy to get back to this.

o Some preferred the idea of ‘comment on…’ style input rather than ‘track changes’ – i.e. so that others can’t do a complete or definitive edit. The commenting that we were talking about was akin to review comments.

26 Anticipating how to partition space could be considered as at odds with ‘informal’ use. 27 In the end of project interview one tutor suggested that he would like to see user rating systems for resources to make it easier to see what was useful. He mentioned the Digg system (http://digg.com) in this context. He would look for items that got the most votes and then ‘follow the chain’.

5. Issues around sharing and access:

o Tutors asked who would control power/level of possible edits and whether there would be limits on the numbers of possible versions. Who would decide what is kept/lost?

o If wikis are borderless, tutors asked, how does this relate to OU structure which is very course centric? There was the impression here that this more open cross cutting activity might not be allowed. For example, Course Teams may have reservations about extra resources being added by tutors which increase the workload for students.

o There were questions about copyright. These were directed at making sure that materials were shareable rather than concerns with retaining the copyright in the materials the tutors produced themselves.

o There were concerns that if something starts on FirstClass (FC) how could it be migrated over to a new environment? ALs were here concerned about the new VLE (Moodle). There is awareness that FC may not last forever and so any replacement should be easy to migrate resources into and from.28 This may lead to the ‘keep a hard disc copy’ mentality – a case of keeping a local copy just-in-case-it-disappears.

o FirstClass client software currently supports an offline reader (FirstClass Personal). One heavy user of FC Personal wanted to be able to log in and download all the changes within PROWE and then read offline. This was related to her personal (poor) access to the internet, and may not be an issue for those with broadband access.

o There was some confusion throughout about what role discussion played – i.e. some confusion with what FirstClass does now (this differs for different courses). There was gradual realisation that PROWE is a repository which sits beside FirstClass and could be accessed from it, it does not replace that primary environment for discussion and online teaching.

o There was reference to the circular nature of some discussion in FirstClass, e.g. the ‘memorial threads’ (same questions coming around again and again). This reflects on the usability of FC – it is easier to make up a new answer rather than retrieve the answer from the last time around. 29

28 Currently FC is used as the conferencing system (i.e. the online forums) and linked to from the OU Moodle VLE.29 This point was revisited by one tutor in the end of project interview when he said this ‘I can know that I have answered this question six times before on this course but its quicker to answer it again rather than find out where you answered it before’. [OU T4]

o The conference for the whole AL community in AL World (within FirstClass) was felt to be something that was negative30 in terms of the lack of control within it rather than because of its broad base.

o Who will be able to have access to any repository based on PROWE, and for how long? ALs between contracts (i.e. courses) can lose access after a time their to FirstClass accounts. This means that this is not a sustainable location for anything long-term and when the new course starts the tutor will be locked out of their ‘old’ course. One of the ALs attending the Saturday focus group meeting was not currently tutoring. She may miss out on TutorHome announcements and also will not have access to tutor conferences for courses which she has taught in the past.

o There was a question about whether access to PROWE would be on a personal or professional basis. There is a complex relationship between personal learning (and interests) and professional sharing. We need to recognise and meet both those needs.

o The ability to personalise feeds from PROWE was attractive and may encourage ‘off-the-cuff’ interaction and sharing, i.e. seeing something interesting/different and wanting to know more without searching.

o The tutors felt that they needed different areas for different things/purposes (i.e. permission levels) so that they can share with students, share with ALs, keep to self etc. This mimics the current OU FC set-up and that may be the benchmark for OU tutors.

o Access needs to be ‘instant’ from the desktop to encourage use – perhaps using MyOpenLibrary31 principles.

6. Finding stuff:

o The tutors asked for lots of different ways of finding things – browsing, searching, keywords and bookmarking. They wanted choice. Searching by level was felt by some to be most (re)useful.

o Also need comment feature along lines of ‘could also be used for…’ In addition they also talked about an Amazon-like feature along the lines of

30 At the time of this meeting a variety of highly political discussions about the treatment of ALs at the OU were occurring in AL World and some tutors felt that this ‘debate’ represented a minority opinion with a huge audience. 31 MyOpenLibrary is an online and personalisable personal library/bookshelf service available to tutors and students on some courses.

‘you may also be interested in …’

o They felt that the structure detail (of how it is organised) needed to be transparent – with top level editing, not too fine detail, broad brush.

o Metadata – auto-generate as much as possible was the request although there was recognition that some sort of tagging was okay they would like this to be as pain free as possible. Perhaps with professional librarian input to determine keywords.

o They would be happy to select keywords from choice of a group of them but the system must be dynamic enough to accept new ones without fuss.

o They felt that it might be hard to do abstracts of own stuff.

o Need for aggregators.

o They would need good technical help and support.

7. ‘Valuing’ stuff – assessing its potential:

o Star ratings and comments could be useful to indicate what was most valued by the community, but tutors were also clear that there was a balance to be struck between ego trips and the danger of bruising comments putting people off sharing.

o At the Saturday meeting it was suggested that peer review (a filtering process rather than blind peer review) was needed. However there were also concerns that moderation (such as this) would need to be resourced and paid for and could be a brake on the development of the repository. Star ratings and trusted reviewer comments were thought to be viable options to save time in locating suitable material.

o Anonymous review and comment was felt to be undesirable. There was a feeling that knowing the provenance of comments was important (an analogy was made with Amazon ‘Trusted Reviewer’ ratings). Although the amount of comment required on different resources should be flexible.

o Criticism was felt to often be counter-productive to sharing. If there is too much criticism then tutors will choose not to share because it won’t be worth the effort. The negative effects of criticism can be made worse by distance context, i.e. lack of face-to-face knowledge of the others involved. There may be no shared/common experience to build up confidence

8. What to share?

o Tutorial information including PowerPoint presentations and a pool of practice exam questions were suggested as two types of useful resource. The emphasis in suggestions made was on efficient use of time/resources.

o As one model of creating content collaboratively it was suggested that multiple choice questions could be created by all tutors (e.g. 4 each across all ALs on a course). This would make a better resource and be more productive than 100 questions generated by one AL.

o There was some debate about what tutors might want to share and what they might want to keep for their own personal use. For example, tutor-Marked Assignment (TMA) model answers (and then model feedback) might be kept in a personal repository rather than shared widely.

o A resource which might benefit from multiple contributors was a set of ‘Problem solutions’ – i.e. discussion/support materials for university wide issues e.g. how to tutor students who are in prison.

o There was a feeling that a PROWE-like system could, over time, build into something which would be significant if given continuity and if a depth of trust could be fostered. It was felt to be particularly useful for new ALs who might prefer to look and search rather than dive into the FirstClass conferences and ask questions.

7.2 The University of Leicester focus group meetingThe UoL focus group meeting also took place in December 2005. Only five UoL staff were present and one of these was not a distance tutor (later interviewed as UoL T4). The evaluator was not present at this meeting.

'Pointers' about the use of wiki type innovations were taken from this discussion to help direct later thinking (within PROWE at UoL and in wider use of blogs and wikis). For example at the Beyond Distance conference in January 2006 and in the the Beyond Distance working paper by Roger Dence and Richard Mobbs (May 2007). This identified three points highlighted through the UoL focus group meeting:

1. Participants organised their personal resources in a variety of ways. Their preferences were informed and influenced both by practice and experience and by the features/limitations of the available technology.

2. There was a problem of limited awareness of the nature and potential of new and emerging technology.

3. Social and cultural factors relating to the use/user context were potentially as significant as the technological aspects, of what is adopted.

Similar issues (to those used in the focus meetings) were also raised and explored in other workshop and briefing sessions and included a demonstration of the features of the then current version of Plone. Some attendees at these sessions were participants in the PG Certificate in Academic Practice programme. Roger Dence also attended one session and some of the part-time tutors there (in addition to Roger himself) became respondents to the interview stage of the evaluation.

8.0 Interviews with participantsDuring the second year access to part-time tutors at both institutions proved more problematic than it had been at the start. In neither the OU nor UoL was it possible to arrange the broad focus group interviews envisaged and so several individual telephone interviews and one group face-to-face interview were conducted (see 5.0 Project participants section for further information on who participated and how they were selected). While all the OU participants interviewed had used the OU PROWE blog and wiki within the context of PROWE (i.e. for informal sharing of content with colleagues that they did not work with on a regular basis), the users at UoL were more likely to have made use of Plone (the UoL PROWE blog/wiki) for communicating and sharing formal content with on-campus colleagues who they do work with on a regular basis.

The questions used in the interviews with tutors at both OU and the UoL were derived from the evaluation plan agreed at project and steering group meetings. Consent was obtained from each participant for use of their comments in work associated with the project. 32

The questions (see APPENDIX: Interview Questions) were emailed to participants in advance of the interview. Most of those interviewed had read the questions and thought of answers in advance. Some used these pre-prepared answers as an aid. Others had not read the questions and talked ‘off the cuff’, this often resulted in longer interviews.

Each interview was semi-structured, following the order of the questions circulated. They ranged in duration from 40 – 100 minutes with 50m minutes as an average. Participants were assured in advance that they could ask questions and request explanations at any time. Terms such as metadata, taxonomy and folksonomy were not well understood by many of the participants, particularly those from UoL. 33

32 UoL participants were asked to agree that their interviews should be used for a broader spectrum of research. All those interviewed consented to the requested uses of their comments. 33 Most of the OU participants understood the term ‘metadata’ and three were also interested in folksonomies, having previously discussed these with the OU Project Officer Anne Hewling.

9.0 Participant views on the technologiesAs part of the requirements gathering for the project both institutions considered very carefully what technologies should be used. Anne Hewling completed extensive research in order to offer advice on the most suitable wiki and blog environments (see for example Hewling, 2006b) and one of the key requirements was that the system should be easy to use. This point was also reinforced by Gilly Salmon speaking for UoL participants at an early project meeting where she emphasised the need for a WYSIWYG editor rather than wikis which (more authentically) used simple tags to format messages and create links. That we were at the start of this project (Summer 2005) looking at wikis without WYSIWYG input serves as a reminder of how fast the technology, and our expectations of it, have developed in the past two years.

At the OU the system chosen was Elgg (v0.4) with PmWiki (v2.1.11) and at UoL – where this system was already starting to be used for a variety of purposes – it was decided to use the Plone (v2.1.3) content management system with Simple Blog (v1.2.1) and Zwiki (v 0.58) (see Mobbs, 2007 for more detail).

As mentioned in the PROWE Final Report, having started the development of the demonstrator on v0.4 in February 2006 the decision was taken not to upgrade. Elgg v0.6 was assessed by the team in June/July 2006 as being too technically unstable, at that stage, for use by the project. We couldn’t afford to have any down-time due to the tight window within which our tutors were using it.  By running Elgg v0.4 we opted for a lower level of functionality in favour of a more stable system – not least because by then we had “workarounds” for some common issues and we had developed a comprehensive user guide to prepare ALs for issues that might arise. This dilemma is common to projects that stick with an older software version due to the disruption that up-grading may cause.  There were significant structural changes between version 0.4 and 0.6, it was not a simple upgrade.

9.1 OU PROWE blog usabilityThe OU demonstrator was based on Elgg v0.4 (implemented in February 2006 and trialled until August, 2007), using the typical default system installation “out of the box” and without modification.  Following PROWE’s initial Elgg (v0.4) trial and associated evaluation results, in July 2006 a standard implementation of PmWiki (v2.1.11) incorporating the “Monobook” (Wikipedia) skin, was bolted onto Elgg v0.4, to provide users with wiki functionality.  Further versions of PmWiki were made available during the course of the PROWE project, but there were no technical or other reasons to up-grade from v2.1.11, or to further customise it.

With all the care taken in choosing a system which responded to user requirements as closely as possible, it is very disappointing that the Elgg (v0.4) part of the OU system received considerable criticism from those interviewed once the project finished. One reason for this may be the frequent comparisons made between this system and the FirstClass conferencing system that tutors at the OU already use for similar purposes.

Another reason might be that this was an early version of the system (later versions might have been less problematic for users) The project team, in particular the OU Project Officer (Anne Hewling) felt that Elgg has potential but given the nature of the potential user group it needed ‘packaging’ (not provided in this project) to encourage many tutors to see its potential, particularly in relation to their use of FirstClass and the ways in which they approach sharing in this.

Another problem may have been the lack of familiarity with research projects where the technology may be ‘unsettled’ and where the practice is evolving. Since Elgg (v0.4) was an early version of an open source software it required frequent updating and was still very much under development itself as the project got underway. Part-time tutors for very good reasons are usually ‘task-focussed’ and there was a lack of sympathy for the system and the project from some participants. For example tutors writing in the OU PROWE blog commented:

I’m finding it hard to see why ALs would bother to use this system when there are other similar, more efficient systems available. I’m prepared to consider that I might be missing some vital point here, but at the moment my impression is that this is much too clunky and irritating to be of much use to me and/or my students and I doubt I’d inflict it on them.

I’m no Luddite, and I’m far from opposed to using online resources – but not just because they are there … My initial impression of ‘blogging’ in general is that it’s little more than an online version of vanity publishing and too open to being used for self-indulgent online ramblings. That is by no means a comment on any of the postings here, I hasten to add, but it’s striking me as just an interesting means of wasting time at present.

9.2 Uploading into the OU PROWE blogSeveral OU tutors were very critical of the Elgg system selected, specifically citing the difficulties in uploading files and managing them:

The file handling was appalling. … Once you created a folder you couldn’t edit the name and things like that. You had to delete it and start all over again. And document sharing didn’t work very well with the e-logs. I went on and tried again the other night. Actually to be able to put documents there and then make them available to other people would be really useful but that didn’t work. [OU T2]

The drawbacks were primary technically, things like the difficulty of being able to upload files which caused quite a few things to fall down holes. There were things like simply not being able to pick up a file, you could find a folder but you couldn’t go further down to find the file you wanted. [OU T7]

I think that I am warming to the idea but this particular interface seems rather unwieldy and to some extent counter-intuitive. For example I set up a Linguistic

stuff folder in my area into which I intended to upload 5 files; two of them made it into the folder, the remaining three are hanging around rather untidily as if they’re not sure whether they’ve been properly invited. I know that this is likely to be something that I have done rather than a system fault but can’t for the life of me see what that can be, except that it’s probably something to do with there being apparently two gateways (and I just know that isn’t going to b the right term) into uploading files. (Comment in Elgg v0.434)

A less ICT-sophisticated OU tutor (OU T1) was more forgiving ‘I thought it was good enough, the choices were all there that I needed’.

There were particular problems with the upload system in the OU PROWE blog and the ease with which content could be lost (and difficulty in finding it). Both would appear to argue against its suitability as a repository.

9.3 Help in using the OU PROWE blogElgg (v0.4) did not arrive in the project fully supported, which is common with open source software. Instead a set of ‘Induction Help Materials’ was developed and shared with users in the wiki and advice on particular problems, e.g. where to find a spell checker, were also posted in the wiki and into the Elgg pages.

Several tutors commented on not knowing how to do things and not finding the Help system very useful for reminding them. For part-time tutors, who may not be using the system very frequently (i.e. not on a daily basis and with gaps of several months) this is perhaps particularly important. The error messages were not informative (particularly for less ICT experienced users):

I thought it was quite easy to use until yesterday, until I wanted to change a photograph. I wanted to find how to do one of those things and I couldn’t find out how to do it … and what I was missing. I don’t mean missing in a sense that it is not there, but in ‘Why don’t I know? I looked in ‘How do I?’ Presumably it’s on there somewhere? [OU T1]

There was also some annoyance coming through in this message on the OU PROWE blog about the system’s lack of a facility to make ‘Public’ a default option. ‘It’s a bit frustrating having to remember to select ‘Logged in user’ every time35. It would be nice to have the facility to change the default option’.

In addition to the above it is worth mentioning that the technology did not always work. The OU PROWE blog was down at several points for fairly extended periods, certainly longer than would have been the case if it were a system used for teaching. At one stage, one of the participants was ‘lost’ and had to be reinstated. Anne Hewling smoothed this

34 This user went on to comment that she could not drag and drop or cut and paste which she would have liked as an easy workaround. 35 Other options were ‘Private’ and ‘Public’.

and several other technical glitches as they occurred. It is worth reflecting on what might have happened if tutors were not able to call on her for advice and support. The system did not – in this sense – support a self-sustaining community.

This potential to instability may be another factor which could affect how willing tutors are to invest their time in using systems which are outside the main institutional systems and not maintained by the central student service departments – which have first call on technical resources.

Technology is wonderful when it works, and that’s what I feel about PROWE. This is good when I could get on and then I couldn’t get on and then I could get on but nobody else could.... I’m finding that Google has this facility that I can put documents and stuff up there for private use. So, in a way, I’m using Google whereas I would have been using PROWE if it was working. [OU T3]

9.4 Navigating the OU PROWE blogNavigation was another problem with the OU system: ‘I remember that I didn’t get the hang of how it all hung together so it took a bit to understand that I think. Yeah there was no interactive help, I think if you wanted help you had to go the user guide.’ [OU T3]

One tutor pointed out that she looked at everything that she could look at but did not have a good sense of what she might be missing:

I don’t know what I don’t know you know, there might have been fantastic match stuff and I didn’t know how to find it, I did explore most of the things I came across. [OU T3]

For one tutor this difficulty in finding things – the lack of structure – was his biggest criticism:

My dislike was finding things, I suppose. You had to put in a search term and see who was interested in it and then willy nilly poke about until you found something that looked interesting. I didn’t find a way that would have searched the entire spectrum that the entire grouping it and then say there are these interests that have mentioned these this word, and I found you the most interesting ones. It found them all but it found them indiscriminately so it was unstructured, but that was perhaps because I wasn’t putting a structure to it. [OU T4]

9.5 RSS feeds and finding contentIt was one of the disappointments in using the OU PROWE blog, particularly for the more internet-savvy users, that the alerts – flagging new postings using RSS – did not function properly:

The other issue was awareness, a method for being aware of what other people had uploaded and when new stuff was uploaded seemed to be missing if I remember correctly, so I didn’t find it from a technical side of things a good tool to use. The idea’s great but it needs a better tool. [OU T6]

It was not that easy for users to get to and identify new postings. Because there as not a great deal of content within the system they tended to browse everything, read postings and from there follow their inclination:

My method of checking through Elgg was to go to all new blogs and have a flick through. That’s another thing that’s unfriendly about it you have to click everything at least twice to get there, it would have been nice to just have a button to click that takes you to all new postings. 36 [OU T6]

I like the way, this almost anarchic way in which you link together blog postings. I’ve never worked out how they fit together so some of the ways you can read is quite surreal because they seem to be delivered in a skip, mixed up and plonked on a page. I think that’s also part of its weakness, I found that the interface counterintuitive, I just couldn’t find the things that I wanted. [OU T7]

9.6 Offline fallbacks? As identified in the surveys and during the focus group meetings, USBs are used by many of the participants as a way of passing information from machine to machine and transporting it from location to location. This has continued.

I’ve got memory sticks, two are on my key ring that I’ve always got with me. I know what’s on both, the core stuff that’s on both of them and then I use them for moving stuff around. But generally they’ve both got things that I know I’m going to need frequently on them. [OU T7]

The idea of a personal and informal online repository should to some extent replace this, but one of the interesting aspects of this project was the distinct separation in most tutors minds between resources which were largely private and for personal use (or use within a small closed community of colleagues) and that which was shared more widely and publicly (see Different types of publishing space below).

9.7 UoL PROWE blog/wiki (Plone v 2.1.3) and the University of Leicester users

The UoL PROWE blog/wiki was based on Plone was implemented at UoL for a variety of purposes in late 2005 and trialled by the PROWE Project until August, 2007. The version evaluated for accessibility in August 2007 was 2.1.3.

36 This refers to Elgg v0.4.

At UoL the comparisons were not so harsh. Plone was seen as generally less restrictive than Blackboard, although many UoL users were also quite satisfied with Blackboard. There was uncertainty about whether Plone would be (could be) used for teaching instead of Blackboard, while at the OU PROWE blog and wiki were compared with – but not seen as replacing the FirstClass conferencing system.

One problem mentioned about Plone by one of the heavier users (the team based at Leicester Royal Infirmary) was the current memory restrictions. Plone would not accept the attachments of >2Mb (although there was an understanding that this might change) which restricts some of the uses that they wish to put it to. There were also some issues with navigation which made uploading awkward – echoing here some of the comments about the OU PROWE blog, but with rather less vigour.

To some extent this was overcome by one of the users at LRI doing most of the work of retrieving and uploading files within the system on behalf of colleagues. This is how she described it:

The other thing was I found that if I did load something up I’d have to click out of the software and click back on again or else it wouldn’t save it. So it was like if I did pull, or add an attachment I’d have to click back out again just so that it would update it which was a pain. [UoL FG Female].

For other purposes Plone was largely considered user-friendly enough, once you had been shown what to do. Again the LRI users pointed out that the method of doing certain tasks, in this case uploading a presentation, was sufficiently complicated that the user recorded all the steps so she could refer to them if she later forgot them. This suggests that this was not an instinctively or obviously easy to navigate system at a level where uploads were required.

For some users there were real or perceived problems of access. These limited their confidence in the use of any online repository as their only route to accessing the resource:

If you’re stuck in an airport somewhere and wifi’s no good or whatever then you’re stuck. So for that reason most of the stuff that I have I also copy onto my personal space. But what I would like is nationwide wifi 100% of the time access to the University server so that I can put everything on there. … So for me the biggest problem is ensuring that I’ve got access all the time whenever I need it, wherever I am. And I have got that with a portable hard drive or a computer. [UoL FC Male 1]

9.8 Preparing content offlineAlthough some users were writing directly into the wikis and blogs others were preparing content offline first. For example a UoL user: ‘I’d probably prepare them on a Word file first. The two entries I’ve made, I prepared them in Word and then entered them. So they

[a version] stayed in Word I think’. [UoL T1] Some OU users were frustrated with the problems with cutting and pasting from their existing Word documents, suggesting that this – rather than writing directly online - may be a common approach.

10.0 Theme one: Informal repository useAlthough we talk in this project about ‘informal’ use of blogs and wikis as a form of repository, or perhaps as a supporting frame for a particularly personal and informal repository-sharing community, we might be thinking of ‘non-formal’ rather than informal. Erault (2000) distinguishes between non-formal, formal and informal in talking about work-based learning activity. While non-formal and formal learning can be connected with a shared and to some extent predictable objective, it is more difficult to do this with informal use, which could imply no connection with the workplace at all and may be simply a pastime. The term ‘non-formal’ therefore suggests something which is outside the normal controls and most of the ‘informal’ activity noted within this project was of the non-formal rather than informal variety. Where informal activity (e.g. discussion of tai chi or digital camera purchases or holidays) occurred, or where participants were playful in their selection of pictures to represent themselves this resulted in some other users being concerned about wasting time – a very real concern for part-time tutors.

Tutors could see the need for and difference between informal, non-formal and formal spaces for discussion. For example:

I moderate quite a few conferences on FirstClass and actually taking a too formal approach to things can actually drive people away, I think some of the relaxed lessons and stuff is more for a relaxed environment and works in your favour. I would want to have some of the less formal [OU T2] argues for something akin to Elton’s non-formal learning, but is not really informal.

It would be utter bedlam wouldn’t it? Everyone with their own repositories and their own little hobby horse to ride … you wouldn’t be able to get a word in edgeways. [OU T4] suggests that informality in the sense of absence of control, is not desirable.

I use the FirstClass stuff to communicate about my course and Openings and … I do keep it to work related stuff. Whereas this one I found that I was using to……to do anything, like I’ve just discovered this or I’ve just got a new digital camera and I’m playing around with that and it’s all a bit too much ... somebody came back and said have you got this bit of software it’s really good, you know that sort of thing so it’s not specifically work related. [OU T3] This suggests that the PROWE system at the OU was treated in a more playful and informal fashion than this user would adopt within her working (FirstClass) environment.

The OU PROWE blog lent itself to spontaneous publishing of comments and requests for comments. Being a project there was also an absence of the usual restrictions (institutional or self-imposed) on what to use the system for. This was seen as a virtue by one OU tutor who felt that they might use the OU PROWE blog and wiki for work-related but non-formal sharing, if it continued to be available. She felt that it was good for sharing fleeting ideas or spur of the moment ideas, making this comment in the OU

PROWE blog: ‘you know if I had a brilliant idea and I’m afraid I’m very good at brilliant ideas but they often just last a day so if I don’t have anyone to share them with I sometimes I feel frustrated so yeah to share some of my brilliant ideas with people and to ask for ideas on how to tackle particular things’. This does however assume that tutors may be motivated to spend time responding to what may be fleeting ideas.

10.1 When it’s ‘informal’ will there be high quality?There was clearly an issue with how participants could judge the quality of material which has been passed around within an ‘informal’ context where there is no clear agreement on what should be posted and what should not. The range of resources which could be posted within an ‘informal’ repository is infinite and the potential for users to find material that they are interested in becomes more challenging. Particularly so if they are looking for material for a specific purpose which requires high quality as well as easy discoverability.

One way in which the high quality materials might be spotted in a more formal repository would be their connection with known institutions. As one OU tutor puts it:

‘Has it got the OU logo on it?’ Because I tend to trust – not blindly and not uncritically – but if something has the OU badge then at least I have some idea of the quality that I should expect from it. [OU T7]

10.2 Provenance and personal profilesDuring the course of the interviews questions were asked about the use made of personal profiles in discovering information about others, and whether that information played any part in deciding whether to share content with them, or whether to refer to the content that particular participants had posted.

If I found a particular resource from a particular person that I found really useful I might go off and then search for that particular person and see what else they’ve done rather than looking for a specific subject all the time. [OU T2]

I don’t believe that everyone’s opinion is the same worth as anybody else’s because in this world it isn’t. If you’re looking for the answers to some problems it’s nice to know that the person who has provided a possible way forward has actually been here before and is not just spouting nonsense. So I liked the profile, I liked the idea of profile [OU T5]

The interesting thing about profiling that it lets you do was actually it picked up on tags and things so you could go and find the ‘other’ things that you were interested in and you would say ‘Oh look, its highlighted this. I wonder if everyone else is thinking about this’ and you could go off and chase it. [OU T2]

On the other hand, there was some concern that it was not so easy to know how to describe your content so that it was useful to others. It may be easier to describe your content at the point of the request, or within a small group of people whose interests and tastes you know.

If another tutor asks me for a resource that I’ve mentioned that’s fine, but it’s not the sort of thing I’d send out because everybody has their own personalised view of what they are producing and another tutor might think something different about it. But I would want control I think and to keep it very localised to me and my groups. I don’t think I’d want everything to be widely available to everybody. [OU T1]

It is perhaps significant that the same tutor was more comfortable in sharing research findings with unknown others than she was in sharing her teaching material. There is a distinction here between research dissemination and teaching resource dissemination with the former being much more public and widely disseminated. Research dissemination also brings clear rewards, recognition and opportunities in a way that teaching content dissemination or sharing for CPD generally does not.

I do think it does depends on what it is … I wrote an article a few weeks ago for the NCET (National Centre for Excellence in Teaching) on the Diploma Maths courses and that I tried to publicise as much as possible, not just to students but to the whole OU community and even outside if possible. So that to me is something [that] can go as far as possible on a national level. But most of the stuff I do is low key and for me and my group. [OU T1]

If a tutor were looking for a resource to reuse they might look for things like the institutional logo or other broad context identifier from the wider environment to give them a starting point. Then they might look towards people who they know are active in this area, or who are likely to have experience in this area, perhaps those whose work they have used or rated or heard of before.

Several of the OU participants interviewed suggested that they would look at names, profiles and reputations of those posting the information in order to determine whether to read or trust a resource, and they would expect to apply this to formal, non-formal and informal systems. This tutor offers a very formal explanation:

… again from FirstClass – there are some people that I will read and other people I just won’t bother with. It’s this idea that we build up a history. We need a first level filtering system that helps us with determining where to look, but beyond that I think that we build up a mental bank of credibility for people. … It’s that people that I find credible are the people that I tend to go back to and look for their thoughts on ‘x y and z’ and if they were generating material I would look for their material on that. And not uncritically, I work closely with people like [NAME] and [NAME] and people around here, and we are very critical about

each other’s work and we are explicitly critical of each other’s work because have that trust set up. But equally I would go and look at their work before I look outside that group because I trust what they do. [OU T7]

10.3 Informal tagging systems – the attraction of taxonomiesThere was some interest in taxonomies and more informal tagging systems, largely because of a perception of this taking less time and effort. However there was appreciation also of the problems of imprecise use of terms: ‘There are so many words that I can look up and each will give me a subtly different set of results won’t they?’ [OU T4]

Working within an informal repository, and hence within a necessarily less obviously structured system, highlights the importance of how items are tagged so that they can be found by others (or even ourselves) when looking for them later on. The idea of whether to formally or informally tag content can excite strong opinions. As the PROWE metadata report (Whitelaw, 2007) points out, there was initial interest in formal tagging which waned as the project progressed, perhaps as the variety and informality of the content became clearer.

Some of the more ICT-experienced OU tutors showed an interest in a looser user-defined folksonomy approach to describing their content;

…I like the folksonomy for looking at things and for looking at how someone else has done something based on how we have both spelt a particular word the same way … and let us see whether I can make something out of the pair of them. [OU T7]

For this tutor this view was expressed almost as a philosophy about not trying to pin things down too much. He is perhaps thinking of the informality (non-formality) of the content when he talks about its fuzziness:

[OU T6] and I share stuff and we share it on the level that you can imagine that my stuff will be which is fuzzy and his stuff is fuzzy. If we have to beat it into a shape that will fit into a specific box which will fit with any institution, discipline or at a national level then I think that we would probably change what we are doing and if there is any value in what we are doing we will lose it – we run the risk of losing it. [OU T7]

Once you start to look at informal and personal repositories then the content necessarily becomes different in variety and in finish and perhaps in attitudes to authorship than that found in a more conventional repository. This has implications for sharing, both in expectations of what participants will find in such a system, and what might be appropriate to share there. For part-time staff who see this as an additional activity additional to and only loosely connected with their paid work there may be a limit to their willing to maintain or improve on what they have deposited or exposed there. Being

expected to take on-going responsibility for something offered for sharing, or being required to polish it up is not something which tutor OU T7 wanted. It would be an extension of the already optional role of ‘unpaid contributor’ identified in 13.0 Theme four: CPD and sharing resources.

If there is any value in that then go away and do what you want with it but don’t expect me to carry on doing any more with it’. I’ve done what I’m going to do with it because its got to the place where I can use it then I can live with that. [OU T7]

10.4 Non-formal tagging: FolksonomiesInformality and less formal approaches to tagging could arguably create barriers to sharing. For example, if sharing across institutions there may be an issue with folksonomies (user-defined tagging) because other tutors at different institutions may not understand the OU/UoL acronyms or course codes that tutors habitually use.

One of the things about tagging is people don’t use tagging consistently because we’re not all automata and I suppose it goes back to things like social book marking. I’m beginning to use other people’s links because I know I can trust them but I’m still on a bit of a learning curve. [OU T6]

However other participants felt just as strongly that requiring the use of formal tagging systems would reduce their interest in sharing widely.

…….One of the things that would really get in my face if we started doing it in a more formal way is that I don’t like being pushed down lines that make things formal if I lose the other side of it. If I have to start generating metadata for every piece of material that I make available to other people I’m more likely to step back from making it available to other people. [OU T7]

This is a real concern as many tutors are comfortable with their current level of direct sharing – sending out a file in response to a specific request for help or information rather than offering it in advance of any request.

11.0 Theme two: Personal repository use

There is increasing interest in the idea of Personal Resource Management Strategies (PRMS) both within this project and within others within the JISC repository cluster (notably RepoMMan and CD-LOR). Whether this refers to the organisation of resources online (e.g. in Google docs) or offline (e.g. on portable storage devices), several of the same issues are raised. In terms of the potential for developing an online personal repository, it has already been noted that since the onset of this set of projects (PROWE, SPIRE, RepoMMan and CD-LOR) there has been an explosion in Web 2.0 services which offer options outside the institution (Franklin, 2007). Over the same period there has also been increasing attention paid to the idea of Personal Learning Environments (PLEs) (e.g. Milligan, 2006) as viable and perhaps desirable alternatives to the VLE.

The extension in use of blogs and wikis – themselves Web 2.0 tools – was at the heart of this project, but tutors often see any online publication as necessarily ‘public’, drawing a clear distinction between online resources and ‘private’ resources. I’ve always been wary of publishing anything I don’t want people to see’ [OU T1]

Part-time staff can be very directed to searching for what they need now and only if they have time might they look more broadly. This is may be a common constraint for any part-time employees.

Stuff that interested me that second and depend on how much time I’ve got and what I’m trying to do in that very moment and it’s going to vary. I might be trying to create something new in a certain context and because I’ve got lots to do and other things that are nothing to do with the Open University, so it might be that I’ve got an issue that I’m trying to think about or trying to get some ideas on so I might go into PROWE and put a few search terms and have a look and see what comes out to see if it fits that. [OU T3]

Personal (private to the poster) resources were more likely to be held independently of the institution, perhaps on USB pens or laptop drives rather than within the institutional system. One tutor offered this definition of a personal repository as a fall-back for his personal and portable devices: ‘It’s where I go to get the stuff that had I been relying on me putting it in my bag before driving somewhere I would have forgotten (smile)’. [OU T7]

11.1 Different types of publishing spaceMany of the users pointed out that they felt – from the start – that what they were making available on these systems was prima facie public. They would not have uploaded content which they wished to retain purely for personal use and keep ‘private’37. Typical comments were:

37 This was one of the settings offered in Elgg. Others were ‘public’ and ‘logged on users’.

I think if it was something private that I wanted to keep private forever I just wouldn’t put it there … I’d store it in other ways, I’ve got my laptop, I’ve got an external hard drive for long term storage so I’d do that. Partly because why waste the bandwidth on the web if you’re not wanting to share it with anybody at anytime and if you changed your mind you could upload it later. [OU T2]

I did [make it private] at first. And then thought there’s no point writing a blog if nobody’s going to read it. But if there’s something I don’t want to publish widely I could actually use draft. I wouldn’t put it on rather than make it private. [OU T1]

If I’m going to store them online I put them on Blackboard. But I would only put things online so that other people could access it. Mainly things are stored on my own personal files within relevant programs. [UoL T3]

The personal repository is a much broader set of resources than the ‘published’ repository. It contains things that were never intended for publication. The UoL tutor talked about adopting ‘a “squirrel” or “magpie” strategy of “making private” against a known existing or unknown future need’. [UoL T5] He saw a clear division between the personal and broader (more public) uses:

As a portfolio worker, I would see such a personal repository, in the first instance at least, as being for my personal use. Unless mandated or suggested otherwise by the terms of an external engagement or involvement, I would not see much wider use. [UoL T5]

There was some interest in the potential of using an online system for storage of work-in-progress (particularly co-authored work), which could then be made public within the same system. This was offered as a suggestion and several tutors offered ideas about its potential advantages – although none had tried this yet.

I can see situations in work and with study where I might want things to be private to start with and then once I’d reached a certain point with them I might make them available to perhaps a small group. Because then it’s all in one place anyway and I’m not having to develop it in one place and then upload it later when I was ready to do it, I could just develop it in the one place [OU T2]

Even for quite advanced technology users there was concern about keeping things private and public if they were on the same system.

No. I wouldn’t have trusted myself to put stuff in the same place as stuff that was going on as ‘logged on’ or ‘public’ and make sure I’d pressed the right button. So I decided that if I did have stuff I wanted to keep private I would actually keep it

private by writing it on my computer and only put it on Elgg38 when I thought it was in publishing form. [OU T6]

There had also been some experience of involuntary public publishing:

… the last time I did, I’d put it in the public sector where everyone could read it and I thought I’d put inside where you needed special access to get it. So in that sense I’d done it wrong… None of these things are straightforward. [UoL T1]

Interestingly, only one interviewee (at UoL) made a connection between personal repositories and eportfolios even though this might appear an obvious similarity (Stefani et al, 2007). In both, the selection and arrangement of resources, and access to them, can be personalised and used to support development, reflection, or – if required – showcasing. As suggested in previous outputs of the project (Hewling, 2006c) there was a clear interest having a more finely grained means of sharing content than public/private, particularly if working in groups.

Being able to share blog entries was quite useful. I think that’s quite nice being able to do that. But I think it will be much better if we were able to search in a selective way. So that we could make certain entries available to a small group of people whereas on the current system it was either private or is was made available to everyone who could log on. Which I didn’t think was selective enough for some the sort of work that I’d like to be able to do. [OU T2]

Some OU tutors felt that PROWE did not need so many controls because the group of users was quite small. Also it seemed unnecessarily selective to leave out people without the same interests and within the same small project.

I think I may of started off saying I’ll do this only to people interested in languages or something but most of the people seemed to be interested in languages and it didn’t seem to be fair to the two that weren’t. I think if there had been thousands of people out there and a community of people actually interested in languages than I think I might well of started to say this is for you language guys or this is for you techie guys … [OU T3]

… I left all of mine public because I thought there’s no point making it private then no-one can see what’s happening. Let’s expose it all and muck about with it and see if anything goes wrong. If it were being used for real work then yes, I would want private to me as well and private to particular groups as well. Private to TT281 tutors. Private to M150 tutors. Private to R04 Maths tutors loads and loads of privileges and access rights. [OU T4]

It was not clear that the same views would be held were the system much bigger and not simply a project.

38 This refers to Elgg v0.4.

One thing you could do in both Elgg (v0.4) and Plone (v2.1.3) was to share with a small named group which you had yourself specified. Tutors were interested in the idea of personal controls at a more finely tuned level, rather than simply making postings public to everybody or private to everybody.

I’d be interested in the small named group. On FirstClass there’s a conference set up which is basically a bunch of friends who talk to each other from time to time. If the facility to do that on a more ad hoc basis, I’m pretty sure I would. [OU T6]

No I didn’t [make anything private]. The only stuff that would be private would be the marking guides and things like that and even then I shared them with people. I just shared them with a very small sub set. [OU T7] Email is how this tutor keeps things private.

Users were also clear that they would like to personally control access and control what was published (as long as the setting up of groups was not too onerous a task, in which case they might be prepared to compromise fine detail of group building in order to retain control):

It has to be under your control. It would be hopeless if you had to phone the helpdesk because they would just drown in requests. If we have groups of users, because say let this group in, lock those people out that would be fine. [OU T4]

Must be able to delete things in case it’s absolutely a dismal failure. [OU T2]

The team at Leicester Royal Infirmary were clear that they need different levels of access for different kinds of users if the UoL PROWE blog/wiki was to work for them. For example, the ability to keep some things hidden for the wider public at pre-publication stage, but allow some colleagues to see this work and comment on it39. Selecting a group who can view and comment rather than choosing from the list of currently limited options such as ‘Authored but not published’.

Well the way we set it up initially, well what we said was let’s just keep it so that everyone in the group can access it and then people outside can’t access it at this stage until we‘re happy with it. And we’ve never really evolved beyond that. Because the next sort of stage is to make it public and it becomes more than just an internal resource. [OU FG Male 1]

A further user at UoL [UoL T1] made the point that some information has to be kept relatively private because of data protection requirements. He felt that this was one

39 This comment was from UoL T5, explaining the Plone system: ‘Users have their own personal space and site owners/managers can control access to shared areas for individuals or groups of user. However, the need for CFS web author status [i.e. a login to the central IT system] does potentially limit utility, unless the site is completely open to read only (i.e. public) which is clearly a possible disadvantage for most closed community sharing applications’.

reason for preferring an institutional system rather than (say) Google docs for storage of certain resources.

11.2 Versions and back-upsOne way in which a personal repository could be used is as a way of keeping things safe so that they do not get lost, and as a means of keeping track of different versions of documents. At the start of the project it was felt that this might appeal particularly to part-time tutoring staff who:

Were tutoring more than one cohort, or presentation for the same course. Wished to adapt (customise) information for each new presentation or cohort. An

example would be an icebreaker, introductory letter, or revision advice which would contain slightly different information (e.g. different dates) each time used.

Were tutoring the same sorts of courses over more than one institution (e.g. at the OU and UoL) with some materials used in both places.

Did not have one set location for using a computer, so might need to access resources, including resources under development, from more than one machine/location.

Might be sharing and working on documents collaboratively, or sharing experiences, observations and advice in some form of working document or guide.

There were a variety of approaches to saving and storing content, particularly different versions in different locations.

I’m paranoid about actually saving stuff on my computer and everything I do I save on my laptop, save to my pc and save to hard drive as well. If it was something really important that I couldn’t save I would probably put it on a website like this. … I’ve got handouts going back to 2000 on my computer that I use as my own sort of database file thing that has resources that I can change and adapt. If it was something for a current course I’d want them to be safe and then just change them. [OU T1]

There was a general awareness of the problem of ensuring that there were clear differences between versions when sharing with others. There is an assumption of an extra stage of work on the part of the person sharing which may be off-putting to part-time tutors who are calculating hours worked against different courses.

I guess with the resources I do share, they do have dates on so you can tell which is the latest one and which is the most appropriate, I guess if I was doing it....publishing it on a website I’d probably want to remove the early one and then add on current versions and then change any descriptions, change any key words I’ve made. [OU T1]

I can see that you would want three different ones. One would be the public one – public to those people who are paying for the course. One would be an archive so that you could go there to find things that would be useful. For instance if you thought ‘This is the same sort of TMA as we had two years ago and that one I remember caused grief’ what did we do about it then?. And then you would want a third one which would be where you were working on the next iteration of the course. [OU T4]

I only keep the one that I need. I don’t keep old versions. I always overwrite. … coming back to the memory sticks I make sure that I only have one version on the memory sticks. [OU T7]

11.3 Personal profiles

On the OU PROWE blog it was possible to put up a Personal Profile which can contain quite detailed information about the participant40. The participants included substantial information in their profiles (Whitelaw, 2007) and also included photos which appeared against each of their postings or comments. These photos were often of the participant but also included photos of a kitten, a new grandchild, a rose, and cartoons. These did give participants a better idea of talking to a specific person and made postings by that person easier to recognise. As one user noted in the OU PROWE blog (again making the comparisons with FirstClass ‘It is easier to ‘recognise’ people in this way, whereas in FC the small symbol that can be personalised is rather limited’.

40 There was no equivalent profile within the UoL PROWE blog/wiki.

12.0 Theme three: Wikis and blogs as repositories

12.1 ‘Wiki-published’ An interesting distinction was made by one interviewee between ‘wiki-published’ and ‘published published’ suggesting that this might be a stage at which content could be withdrawn from view.

Having worked on the wiki it’s really a good development tool but at some point, if it’s going to be a document that is going to be ‘published’ published someone has got to take it away and refine it. They need it makes a link private having made it public for while and having collected people’s ideas on developments and things they need to take it away and tidy it up and things. [OU T2]

This offers an interesting insight into what are considered the norms of public publication at most HEIs and – perhaps – distance teaching institutions in particular. Distance learning materials are habitually subject to stricter quality checks than resources supplied as part of face-to-face instruction. This is because they carry so much of the teaching content and so need to be correct and unambiguous, and also because the quality of the materials is linked directly to the quality of the teaching and the institution. It is a physical manifestation of what might in other ways, to its students, operate as a virtual institution. The idea of a wiki as a work-in-progress document that is inherently unstable and not signed off as appropriate quality and then ‘fixed in place’ is perhaps more unusual a concept within distance learning institutions than it is within others. The same sort of tension already exists at the OU, with existing routes to individual tutors publishing materials online – whether on personal websites or on within a FirstClass tutor conference.

Wikis were more readily appreciated by these users as useful for specific work projects, or work-in-progress discussions, particularly where participants were geographically separated. This led to continued use of wikis after the active phase of the OU project ended.

[PROWE] was part of the experience leading up to me using both a blog and a Wiki. I’ve been contracted as one of the people rewriting the tutor tool kits and my bit was a sort of broad overview of teaching and learning. In order to do that I set up a blog in order to get people to take part in that and was vaguely successful. I think the success looked at in these terms, part of the success was that I put on the Wiki quite a bit of the stuff that I’d written myself and left it open to see it and amend it if they wanted to. [OU T6]

I have taken some of the resources that were created as part of PROWE and used them elsewhere. On the Sussex Learning Network wiki I’ve used the page that Anne put together about wikis, blogs and vlogs and everything else … Certainly some of the ideas that we talked about in PROWE and the idea of having stuff in

one place we’ve talked about and the MFB stuff that were doing at the moment will go in the same direction. Where we are trying to get people to generate resources which can then be used across different disciplines. It fits in…. so many things that are happening out there at the moment. It fits in with the i-skills programme as well. [OU T7]

Balanced against this are the potential technical barriers to amending stuff on the wiki. Tutors may not have sufficient ICT confidence to do this, and may also feel that there is a social barrier in working on another’s document in front of an audience. They may prefer to go back through the original author, emailing changes rather than making these themselves – perhaps seeing this as a precaution or courtesy.

Nobody’s yet taken up the option to amend it, I think there’s quite a barrier to overcome in terms of doing the techniques involved in actually editing a Wiki. But quite a few people have looked and quite a few people have emailed me stuff to add to it. [OU T6]

One tutor [OU T2] mentioned the practical problem of comparing versions as the wiki only shows one version at a time – although she did like the sense of history:

I think of things like controlling the different versions … you can only have one version on show at once even though most wiki’s have got the idea of the history so you can go back and do the restore. But when you do the restore you get back to back one version, they tend not to have several things around. By a repository I might possibly want to have different versions and possibly versions that other people have had. [OU T3]

12.2 Affordances of blogs and wikisSeveral users pointed out that you can use blogs and wikis in different ways:

You know you can use blogs to kind of outline thinking and to promote discussions amongst people if people are actually interested in reading your pearls of wisdom and you can use the wiki as a more sedimentary process for keeping things and perhaps updating them from time to time. I think they complement each other. [OU T6]

Blog postings tended to be more personal in tone and wiki entries were geared towards group construction or group communication and much lengthier and more complicated advice. The wiki pages created in the OU PROWE wiki (linked to the OU PROWE blog) during the project included:

The OU in prisons Things students need to know about tutors Assignment marking: One tutor’s method for marking assignments Plagarism – An approach to teaching practice

The variability of quality in blog postings was mentioned by participants at both institutions. Talking about blogging in general rather than the project blogs specifically terms like ‘vanity publishing’ were used – suggesting a low opinion of the quality of the blog content. As one lecturer, put it ‘you go from some incredibly expert insider type knowledge on certain blogs to people posting pictures of their cats’ [UoL T4]. This lecturer is himself an established blogger, using WordPress rather than Plone.

There were issues highlighted by various UoL participants about the effect of physical proximity on the usefulness of blogs and wikis.

Our wiki is only for internal use for those of us in the research team and because we sit in the same room there’s not much point in using it. It would have more of a purpose if we were spread over more universities or the Open University situation. Then I’d be much more likely to use it. [UoL T1]

I think maybe because it’s a small group41 we tend to talk to each other than to actually blog. I think in more of a bigger group you’d get more of response because the communication values are there but because we are always communicating anyway so we kind of get over the problems as quickly as possible. [UoL FG Female1]

However the LRI team do use Plone (the UoL blog/wiki) as a formal repository, without the commenting element. This use of Plone as a group repository worked well for them.

I think it’s valuable even in a small group because we don’t want to pass papers round the office. To be able to just flick it up on the screen and get the information you want is I think extremely valuable. And the other thing that it does is because it increases with time is, sometimes there are so many versions of it, you do have a record of how things are developing as well. [UoL FG Male 1]

For me it’s a bit more efficient than just emailing people. You can put it on there, just one copy or whatever. [UoL FG Male 2]

Everyone can access it. It’s never going to be deleted. … they can always refer back so if we’ve done a paper that is relevant to their research now we might have done that paper two, three weeks ago so they can just pull it back up again. So in that way it works really well [UoL FG Female 1]

41 The LRI team has 12 members although they may also have visiting students.

13.0 Theme four: CPD and sharing resources

The idea of communities of practice, including online communities of practice, playing an increasing role in continuing professional development is currently significant throughout work-based learning and not just within the university or education sector. Blogs and wikis have been identified as a way of disseminating and co-creating information within professional communities. Their advantage is that they can offer the chance for many members of that community to publish simultaneously and also collaboratively without obvious central institutional control of the content.

The idea that part-time tutors interviewed within the OU and UoL are members of a community of practice (or perhaps several) within their institutions is difficult to determine (Wenger, 1999). One problem is that a community of practice is defined by what it does rather than what it calls itself. Here we have a researcher at the University of Leicester who is also a part-time tutor at the OU commenting on the norm at the OU as she sees it:

Well with the Open University at the moment there’s not a lot of purpose for communicating with the other tutors because we don’t have a genuine – I know you talk about a community of practice, but we really don’t have a community of practice. The course material is sent and we’re not developing course material and were not developing ideas, wiki is for my purposes for writing and developing ideas within a group and we’re not doing that as tutors. The limited amount we do is served absolutely adequately by the [OU FirstClass] conference. I could see lots of other purposes for it [the wiki] in teaching but not for me as a tutor. [UoL T1]

While she was not involved in the OU PROWE blog or wiki, and only made a very low level of use of Plone (the UoL PROWE blog/wiki), she is expressing a common view across both institutions, that the blogs and wikis are much more obviously tools for developing ideas with peers, than they appear to be for sharing ideas and resources as part of staff development.

One reason for this could be that part-time tutors will quite naturally be asking themselves at each stage ‘Who is paying for me to do this?’ and ‘How does doing this help me?’. This not only varies between institutions, it can also vary between departments at some institutions. It is quite unusual for tutors to be allocated paid staff development days as a matter of routine as part of their contract (although this happens at the OU). Any staff development time might be directed at agreed activities relevant to that time period or could be less formally determined. In both of the institutions within PROWE tutors might receive additional funding to become contributors to staff development (CPD) activity across the university, or within specific courses – for examples see Contributing as paid activity in the figure below. At the University of Leicester at least some of the tutors (Roger Dence is an example) may take on roles

which involve delivery of formal continuing professional development activity for other staff.

Another way of looking at how part-time tutors might be participating in staff development is in terms of their roles as consumers or contributors. While the role of ‘consumer’ will exist for every member of tutoring staff, it will be more obviously structured and more urgent for newer members of staff who are ‘learning the ropes’ and staff who are changing courses or altering their existing practice. Sharing as a consumer may be foregrounded at this stage, while sharing as a contributor is more likely to occur at later stages in professional development where the tutor has gained confidence and skill.

Consuming Contributing

As paid activity: As unpaid activity: As paid activity: As unpaid activity:

Examples would be learning new skills when directed/ expected to do this. For example when starting to tutor for the first time, when changing courses, or when changing existing practice – e.g. moving to an online marking system, or when responding to the needs of a specific student (e.g. dyslexia), particularly where an additional support payment is paid.

Following up institutional and other CPD sources in order to improve practice or for personal interest related to professional practice. Reading forums, blogs and wikis of colleagues, browsing subject centre sites and bookmarking potentially useful resources could be examples. When working for more than one course/HEI the benefits of this activity may be shared across these. PLUS any of the ‘paid activity’ on an unpaid basis (depending on tutor contract).

This could take the form of moderating an online forum (FirstClass Conference) for their other tutors, supporting or delivering training to peers in use of new technologies (e.g. a new VLE), or it could be as part of a contract to produce material in print or online (e.g. a tutor ‘toolkit’).

Answering questions or responding to requests for help from other tutors. Sharing resources or bookmarks which may help others professionally (i.e. not just ‘cat pictures’). Reviewing resources, commenting on or giving helpful feedback to colleagues on their performance, or their contributions – e.g. via their blog.

There is some overhead in terms of extra effort in posting content beyond simple conversational messages. So there needs to be some clear motivation for sharing as contributor:

I would be thinking very much if I was sharing a resource do I think it would be something somebody else would find useful? Or am I trying to get ideas from other people? In which case I would be sharing it. If not I probably wouldn’t be. [OU T2]

The PROWE project provided staff at both institutions with opportunities to contribute in the ways which are normally associated with ‘unpaid contributing activity’ (see figure above), but in this case the OU tutors were paid to undertake the project.

Even so, there were some concerns expressed about whether to invest time in this type of collaboration, where the results may be uncertain (unlike more formal development activities). Talking about activity where three TT281 tutors in PROWE tried to work together one of them concluded: ‘… it just sort of fizzled out really. I don’t know why. Because is only so many hours on the day I suppose. It just becomes another resource that I have to manage. If we could get rid of everything and just have one … but to have that as well was too much I think.’ [OU T4]

13.1 How sharing currently happensThe OU tutors have a range of existing strategies for sharing content, with students, fellow tutors or other colleagues:

… by email there’s a sharing. Email goes with the attachment [and] would just describe what they were and offer any problems they may come across that they don’t understand, but you know they’re only shared between a specific group of people and they’re shared privately and I always bcc everyone as I don’t know half of who it’s going to anyway. In a sense it’s very private like a one-to-one if you can imagine with the information, where as if it was on a website it wouldn’t be would it? [OU T1]

What I would suggest now is this is happening without being formalised into this project in areas where for instance the T175 tutor conference is, where people bring things, it’s a full gift economy where people will come along and say these are the resources that I’ve put together for my next face to face tutorial, they’re here if anyone wants to use them, edit them, take them whatever and this happens quite a lot and again because of this common purpose. … That’s a cultural thing that’s carried on from 17142. [OU T7]

42 T171, You Your Computer and the Net was the precursor of T175 and was the OU’s first wholly online undergraduate course. Because there were 12,000 students recruited in the first year several hundred new tutors were also recruited. This mass of new tutors all teaching online and within the same course led to some innovative approaches to online staff development, including tutors conferences where resources were shared, commented on and adapted. This practice has continued into other online courses, sometimes initiated in those courses by past tutors of T171.

I know that one of the courses I tutor, M150, we actually have a big resources book conference that we store things in and lots of people come by other peoples recourses and take them away and adapt them and bring them back again. So we do have different versions, you have some groups that have activities and Java Script and that people can work through and find which ones they like, which version might actually suit them. [OU T2]

I’ve got no way of sharing anything. If somebody in FC says didn’t we have the same problem as this two years ago then I can go away, rummage in the CDs from two years ago on and stick it in a CD and make a copy and send it to him. And that is sharing after a fashion isn’t it? [OU T4]

There is an issue in CPD terms on how the repository is structured (or regarded) in terms of size and specificity. One of the OU participants commented that ‘At the moment it feels as though we are all in the same bucket together’ [OU T4]. This tutor felt that it would be more useful to have named and specific groups along the lines of current courses or at least disciplines as he saw sharing within an undifferentiated community as increasingly problematic the larger that community became. He claimed he would be happy to belong to several different (separate) repositories to avoid obtaining too many ‘false hits’ when searching within a general repository. This tutor recognises the potential advantage of cross fertilization, but suggested that this was more relevant for courses outside his area (Computing).

However, another tutor [OU T6] (himself a part-time paid contributor to staff development) comments in the OU PROWE blog:

My instant reaction to wondering what the benefits are over tutor conferences in FC is that you get to talk to all sorts of people who are not involved in your particular course. Cross pollination. I organise ALDAP for R13,43 and one of the major features we see with every cycle we run is that people really enjoy and benefit from talking to tutors from other faculties.

Why this difference in opinion? Perhaps a distinction has to be made here between what works well in a synchronous and an asynchronous environment. A cross boundary face-to-face session for tutors will have a limited, predictable duration following only one strand of the many discussions that may be taking place in the same physical space. Its online (blog) equivalent would be a number of overlapping blogs capturing all of the information and growing over time, much of the content of which will be irrelevant depending on the user.

43 ALDAP is the formal AL CPD programme and Region 13 is the South East.

13.2 Discipline differencesIt has already been noted that the OU project participants tended towards a high degree of online competence and were likely to be tutoring courses online, often courses based in the more technical faculties (i.e. Maths and Computing or Technology).

One interviewee from the Maths and Computing faculty who teaches Maths and Maths Education (rather than Computing) courses made the point that present methods of sharing information with students and receiving it from them were often difficult because of the requirements of that discipline. Students increasingly wish to use their own email accounts rather than the FirstClass account provided by the OU. However, while the OU account will display attachments which contain mathematics formulae their own preferred system may not be able to cope:

But the email provided [by them], often doesn’t take attachments and often won’t print things. Like if I send equations to the group it’s gobbledy gook when it gets to anyone. So if I could put something onto my website and afterwards blogged it and say look at this and they would be able to see it. [OU T1]

Another Computing tutor made a point about sharing with different ‘velocities’ – that is sharing when the course content turns over very quickly and when it is fairly static.

Well, but it would be … How do you do it then ? Sorry it’s not my problem how you do it. But how would you cope with different people archiving at different velocities. I can well imagine someone in the A[rts] or E[ducation] courses might be using the same material for 20 years but with the computing courses after three years they are out of date and an embarrassment. So we have very frequent archives. We have a fine grained archive that does not last very long but other people would need a courser grained archive which will last the length of their career with the OU. [OU T4]

13.3 Researchers use of personal and informal repositoriesSeveral of the UoL staff interviewed were not only central academic staff on full-time contracts but also saw themselves as primarily interested in PROWE as a tool for sharing or facilitating research. The three UoL staff located in the Leicester Royal Infirmary fell into this category. Their uses were quite formal – to support research activity and more general admin. All team members had to access it on a regular basis because it contained standard documents and forms which they needed to use:

So what we’ve used it principally for in our group is essentially a common space that we can post things and keep information and distribute reminders and provide up to date stuff and that’s kind of going on and is relevant to a research committee that’s varies of a range between about five to ten or twelve people. [UoL FG Male 1]

This user went on to suggest that Plone (the UoL PROWE blog/wiki) could have been very useful when working on a joint research project with overseas partners three years previously. There were clear opportunities for using this to supplement the email and phone calls that they had relied upon then. The convenience of communicating with external research project partners was attractive to them. In contrast there were many more options for discussing face-to-face with colleagues within their group:

In terms of things as a group, we meet at least once a week. We should have a formal meeting every two weeks but we all meet every Monday to sort of chat to each other and things. So our use of the Plone in that respect is really just to have information, like papers and stuff like that, we can’t deal with. So it really is just a repository of information we need and it’s not used in any sort of dialogue because we don’t need it in such a small group.44 [UoL FG Male 1]

44 This refers to Plone v2.1.3.

14.0 Sustainability – what happens next?

With any work-based project, however informal and personal its scope, there should be concern about how to ensure that any positive effects can be sustained once the project funding and support has been removed. This informed the rationale for designing this project as one with an active first year with Project Officer support and a second year or very low support with a focus on evaluating the effects, including any continuation activity. The slippage in timing, particularly at the UoL meant that this plan was not as cleanly executed as first envisaged, but this section looks at what happened after the first stage and what seems likely to happen in the future. Both projects have had effects on the participants and changed behaviour of at least some of these.

The difference between the perceptions of and experience of sustainability across the two sites was marked. The OU system was clearly presented to participants as part of a project with a finite term – which has had various effects beyond that term. At UoL the system was part of on-going activity which was much broader in scope than PROWE and which will be supported as part of on-going ICT development activity there (Mobbs, 2007).

JISC’s proposed seven models of sustainability at the Homerton College Meeting in Cambridge (Lock and Gambles, 2005):

1. Types of output that require little or no action2. JISC pays for ongoing product/service3. Product/Service is taken over by another organisation4. Product/Service is absorbed into the institutional system5. Product/Service becomes a subscription service6. Product/Service is taken onto a commercial basis7. Product/Service is sustained by an active user community

In PROWE the OU was exploring sustainability in the sense of the first and seventh of these models with some interest in the fourth approach to sustainability in terms of the technical system. At UoL the fourth model of sustainability appears to have been achieved in terms of the system, and there is also interest in the first and seventh models in terms of how content will be managed and added or interacted with.

14.1 The Open UniversityHere the approach was to listen to and be guided by the tutors. Options about the system to be used, and the way in which that use within the project should develop were presented and agreed at an early stage. This led to a stronger sense within that group that the project was about the participants and their needs. The Project Officer, Anne Hewling was commended as being very good at getting things going and supporting the community. Her postings into the OU PROWE blog were frequent and helpful and smoothed the project though various technical glitches.

However the use of the OU PROWE blog and wiki was recognised by all the participants as having a finite term. Tutors were paid for their involvement only for the first year and there was no certain future for the platform after the two years had passed. Participants were aware of this. In at least some cases it affected their decisions on what content to place into the repository. For example one tutor who was considering using the system as a way of keeping their student records handy for online access concluded:

Oh well it wasn’t going to last very long so don’t use it as the only system of record keeping so I don’t think in the end I did because I didn’t want to risk it. [OU T3]

This tutor then went on to use Google docs although she was concerned about the implications on relying on a system outside the institution:

I mean I’ve always got in the back of my mind that Google is owned by somebody who’s owned by somebody whose probably not adverse to intruding and monitoring what you’re doing even though technically they’re not doing it so that is always an issue. So if there had been a system organised by the university then I would of felt more confident on that grounds. [OU T3]

Although there was the potential that Elgg (the system underlying OU PROWE blog) could be incorporated into the institutional VLE (Moodle) then in development, there was no firm undertaking and no assurance of sustained access could be given to tutors. The freedom to choose the most suitable system was tempered by a lack of up-front institutional commitment to retaining this particular system as part of on-going provision to these and other tutors.

In terms of its ongoing impact, the Personalised Integrated Learning Support (PILS) CETL – which also works with tutors at the OU – have used outcomes from the PROWE project in planning its online facilities. The CETL is adding a community building area for ALs into the new programme-level websites they are developing. They are also including information about what people do (personal profiles). They said that they had used information sent to them from the PROWE project when planning this facility.

More directly, and immediately, two of the tutors interviewed were using wikis and blogs and in ways which they see as having been influenced by the PROWE project:

Part of the thing we were doing with this other project is collecting case studies that would go online and I have suggested to the project team that the online version should be an permanent Wiki and they seemed to be quite attracted to that idea. So that was good. So looking in terms of what I was doing with Elgg45 in terms of personal repository and what I’ve done with the Wiki there definitely is a connection between the two. So it’s been very useful. [OU T6]

If I didn’t have another alternative then I’d still be building it up as a repository and then would be pointing it out to my students, But now that I have another

45 This refers to Elgg v0.4.

alternative46 I’m more likely to develop that, than to continue working with PROWE. I’d be happy to continue to work on PROWE as an experimental basis. [OU T6]

14.2 The University of LeicesterHere the choice of system was made very pragmatically and centrally, by the project team. The start of the PROWE project coincided with uptake of Plone as a system for use by teachers and researchers at UoL for a wide variety of purposes. Members of the UoL PROWE team (notably Tony Churchill and Roger Dence) were also actively involved in staff development which explored use of Plone and the impact of the project has been widely reported by the UoL team47.

Most Plone use has been directed at formal rather than informal sharing (e.g. its use to disseminate on PROWE within and beyond UoL and its use for surveys within UoL). Plone entries more often relate to public and group entries rather than personal reflective or development activity. However some use of Plone – specifically the wiki at a personal and informal level – was facilitated by Tony Churchill through the UoL PG Certificate in Academic Practice programme48. In the latest presentation (the third during the project period) the use of the Learning Object wiki (which is integrated into the UoL Blackboard learning environment) has replaced use of Plone. This changeover occurred in Spring 2007 (LO wiki became available at UoL in February 2007) and appears to have sparked significantly greater use and discussion than Plone use did. The reasons why this occurred are not clear (see PROWE Case Study 2: University of Leicester), although one possibility is that being embedded within Blackboard may have made the product more ‘trusted’ by participants in the course. The majority of the PGCert participants did not go on to implement it in their teaching although a number of innovative uses of wikis in assessment have emerged from these workshops.

Overall, as Mobbs has reported (Mobbs, 2007), there has been a substantial interest in Plone for a variety of purposes. While part-time tutors may prefer a solution which is integrated into the standard UoL VLE, the flexibility of Plone is appreciated for uses which are primarily communicative, administrative or developmental rather than related to teaching delivery.

14.3 What level of moderation is required?General consensus across both institutions was that there has to be some sort of moderation even though it’s a staff community initiative. Underlying this is a concern about minimising unintentional as well as intentional misuse.

There’s got to be some control even though we hope that we’re all professional adults who wouldn’t do anything that was inappropriate but obviously there does

46 He now has alternative access to a blog and wiki.47 See http://prowe.ac.uk/dissemination.htm for a list of documents, presentations and URLs.48 UoL staff working with distance learners form a significant minority within this programme.

need to be a moderator to keep a watch on that. … Occasionally people do something fairly quickly not thinking of the consequences and why you shouldn’t be doing this. So yes I would want a moderator. [OU T1]

I think it needs an authority. But not an absolute authority … that can take things out or ban things, but one that can say ‘Yes this has been quality assured, this is Kosher, this is ok’. Then things that aren’t approved we’ll have arguments and discussions about [OU T4]

… if things start to be unpleasant and uncomfortable then it’s great that there is somebody there to monitor that and observe it and to whom recourse could be made so people who are not behaving in a dignified or respectful manner can be asked and it’s not always easy for the person who’s suffering to do that so it can be that somebody else is doing it. [OU T3]

There is an assumption that less formal moderation is needed for users who know each other well and work together already, so the size and specificity of the system is a factor:

I suspect that quite a lot of the groups would be able to be self moderating because you’ve got people working on a particular project together, there shouldn’t be any need for moderating stuff that they’re keeping within the project but I think if you’re going to start opening this up to the students you will need to moderate in some way, especially if they’re going to be able to make contributions and things. [OU T2]

So much about the use/usefulness of this type of system depends on who is in the system. OU T7 makes this point in suggesting that his plans for future use would depend on who was using the system. He talks about personalities driving the splitting off of groups:

I think it’s difficult to know [whether I would continue to use the system] and that would relate more to the people involved and what we had in common, I think that others may of said the problem I had was there weren’t enough commonalities for me to be able to understand what we were going to share. Although interestingly some groups did start to split off but I think that was more based on personality than on anything beyond that. [OU T7]

This reservation is central to the sustainability of any community based resource. It can only be as good as its content, which is based on the community itself (the willingness to contribute and be to some extent a ‘gift economy’). Of course the community itself is often fluid, particularly within the informal context fostered at the OU for PROWE. Without someone formally adding or quality assuring content the participants will have to work hard at building and sustaining the repository. Without reward or recognition it is unlikely that many part-time tutors would do this. Of course, within a very large tutor community (such as the OU or UoL has) these could provide sufficient active participants to develop the repository as a key resource. The less formal the ground rules the less certain all this is.

References

Colwell, C and Pegler, C (2007)PROWE project accessibility reportAvailable at http://www.prowe.ac.uk (from 1st Sept 2007)

Dence, R. and Mobbs, R. (May 2007)Perceptions on the nature of communities and their needs – conceptualising and researching wikis @ UoL. Beyond Distance Research Alliance Working Paper 2007/01 Report

Eraut, M. (2000) ‘Non-formal learning and tacit knowledge in professional work’, British Journal of Educational Psychology, vol. 70, no. 1, pp. 113–36.

Franklin, T (2007) Web 2.0 for sharing content. Presentation at ‘Beyond the Digital Deluge’ JISC conference, Manchester, UK. 5 June

Hewling, A. (2006a)PROWE: Understanding the OU user perspective http://prowe.ac.uk/documents/PROWEUnderstandingtheOUUserPerspective.pdf

Hewling, A. (2006b) Some wiki-related issues and resources of relevance to the PROWE project: a digest of resources and questionshttp://prowe.ac.uk/documents/DigestPublicVersion.doc

Hewling, A. (2006c)Personal Resource Management Strategies – some first thoughts for the Warwick DRP meeting 27-28 March 2006

Hewling, A. and Dence, R. (2006)‘Developing Personal Repositories to Support the Development of Part-Time Tutor Communities - a first view of the PROWE project’. Presentation at Beyond Distance Research Alliance Conference, Januaryhttp://www.prowe.ac.uk/documents/BDRAPtBPROWE20060110final.ppt

Lock, S. and Gambles, A (2005)Report from the JISC Joint Programmes Meeting, Homerton College Cambridge, 7-8th July 2005

Macdonald, J. and Hewling, A. (in press)Exploring the potential of online communities of practice for distance tutors. In Communities of practice: Creating learning environments for educators. Chris Kimble and Paul Hildreth (Eds). Information Age Publishing.

Mobbs, R (2007)A Brief Comparison between the University of Leicester Plone PROWE Systemand the Open University elgg/PmWiki Solutionhttp://prowe.ac.uk/documents/UoLPloneandelggComparisonNote200702.pdf

Milligan, C. (2006)‘What is a PLE? The future or just another buzz word’eLearning Focus site: http://www.elearning.ac.uk/news_folder/ple%20event. Summary of the PLE event at Manchester Univerisity, 7 June 2006.

Stefani, L., Mason, R., and Pegler, C. (2007)The Educational Potential of e-portfolios: Supporting personal development and reflective learning. Abingdon, UK. Routledge.

Wenger, E. (1999)Communities of Practice: Learning, Meaning and Identity, Cambridge, Cambridge University Press.

Whitelaw, L. (2007)PROWE Metadata Report, July 2007 http://prowe.ac.uk/PROWEmetadatareportv6final.doc

APPENDIX: Evaluation plan

Factor to Evaluate

Questions to Address

Method(s) Measure of Success

The potential of informal and personal repository tools with part-time campus-based tutors and part-time (distance and campus-taught) students. (REF: Section 4)

What alternatives currently exist and how are these used?

How do part-time tutors use these repository tools in practice? What improvements or enhancements would increase/improve their experience so far?

Would users find benefit in having the repository tools adapted so they could fit into their local / community portal?

Which support or curriculum-based factors influence activity or enthusiasm in using these repositories?

Were expected benefits delivered? (E.g. enhanced peer to peer community for sharing ideas and experience and quicker access to material to support their teaching). (REF Section 5)

Online observation and tracking, focus group meetings, online surveys and factor analysis to capture current use and support deeper analysis of use in practice across the institutions and discipline areas.

1. Individual tutor buy-in and use of the tools to replace or enhance previous practice.

2. Creation of a shared document repository with onward and outward use of same.

3. Understanding (tools for predicting) uptake of these tools across specific disciplines and within specific educational contexts.

4. Recommendations for/Strong interest in further development of tools to support specific purposes.

Usability and accessibility implications for demonstrator and toolkit (REF Section 1)

How appropriate is the demonstrator for novice through to expert users across both institutions? (REF Section 5)

What improvements or options should be considered?

Analysis of interviews and data capture (audio visual and computer screen) records of different tools in use by representative sub-set of users.

Expert evaluator (Accessible Educational Media project) reporting on accessibility of the demonstrator across a range of disabilities.

1. Tools were observed to be easy to use for both experienced and inexperienced users of ICT.

2. Tools were used appropriately and effectively within a range of contexts.

3. Evidence of portability and accessibility improvements through use of the demonstrator.

4. Clear recommendations for any improvements of enhancements.

Technical adaptability and robustness, including any relevant interoperability concerns with regard to existing OU and UoL systems. (REF Section 2, 3)

What lessons can be learnt about how to integrate informal repositories with the more formal repository systems based on enterprise solutions? (REF Section 4)

Technical testing, analysis of records of user requests for technical support (incl. outcomes), interviews with technical experts at both institutions.

1. Full interoperability of the collaborative communication tool with other toolkit resources

2. Technology fit to the requirements of the tutors with minimal reported problems.

3. Technology fit to the requirements of the institutions with minimal reported problems.

Use, re-use and re-versioning of content held in personal and informal repositories. (REF Section 3)

What alternatives currently exist and how are these used?

Has there been an improved or altered uptake of shared resources?

Online observation and surveys to capture examples and awareness of re-use.

Using an adapted version of IET’s CURVE (CoUrse Reuse and VErsioning) model, what types of re-use and re-versioning have occurred?

1. Improved uptake of shared resources

2. Enthusiasm and interest in re-using and developing (collaboratively and independently) resources for sharing.

3. Evidence (records) of re-use and re-versioning or the creation/identification/publication of objects

which support this.

Long term future of the project and its sustainability. (REF Sections 3, 19)

Is the community self-sustaining and self-regulating?

What resourcing (e.g. moderation, facilitation and staff development support) is required?

Is the regular monitoring and removal of inappropriate material sustainable or scalable?

Interviews with range of users at OU and UoL.

Observation and tracking of system use.

Review of use of project tools in second year.

1. Demonstration that sustainable (e.g. economic) models for staff development and facilitation are effective in practice across both sites.

2. Effective use of the PROWE interfaces and interest and enthusiasm by users in investing time in developing these for the future.

3. Continued use and development of tools in second year.

4. Continued institutional support for the project and wider community interest in using the tools/models.

APPENDIX: Survey questions

Different questions were used at each of the two institutions. The UoL questions being derived from the Open University questions. A detailed analysis of the answers to the OU questions was included in the project report PROWE: Understanding the OU user perspective (Anne Hewling, 2006a).

Open University survey questions

1. Course code(s) for all courses on which you are currently teaching2. OU region3. Time as Associate Lecturer4. Courses taught for OU in last 3 years5. Type of internet access e.g. dial-up, broadband, etc.6. Do you use any other technologies e.g. i-Pod, Palm, Blackberry, etc? (If yes,

please give details)7. How do you organise any digital resources you may use in your teaching?8. Do you have any experience of learning repositories? (If yes, please give details).9. Do you transfer digital resources across teaching contexts, e.g. between OU and

any other institutions? If so, when and how do you do this?10. Do you like to reuse teaching materials? If so, when and how do you do this?11. Who do you share ideas and resources with right now? With other ALs? With

your students?12. How do you like to do this?13. In an ideal world what would most help you do this?14. What kinds of materials would you like to share with other tutors?15. Would you like others to be able to edit your work/contribute to collective

writing?16. What kinds of files, such as Word documents, PowerPoint presentations, etc.

would you want to upload?17. If you had access to an AL community wiki or blog – what would you consider

the most important things that it should be able to do?18. What would you expect to be contained in the community policy/ground rules for

use of the wiki/blog?19. What is the most important aspect of interacting with other tutors?20. How would you expect to access the wiki/blog?

University of Leicester survey questions

As with the OU survey each respondent answered some basic background questions: the name of the Department/Area where they worked, programmes that they work on; their

role (e.g. tutor, assessor, lecturer); whether employed full-time or part-time49 ; whether working on distance learning (DL) courses50; and the number of years worked at the University of Leicester.

1. What type of Internet access do you use, e.g. dial-up, broadband etc? 2. Do you use any other technologies, e.g. i-Pod; Palm; Blackberry etc? 3. How do you organise any digital resources you may use in your teaching? 4. Do you have any experience of learning repositories? 5. Do you transfer digital resources across teaching contexts, e.g. between UoL and any

other academic institutions or teaching/training environments? If so, when and how do you do this?51

6. Do you like to reuse teaching materials? If so, when and how do you do this?7. Who do you share ideas and resources with right now? For example, with other UoL

teaching associates?52   With your students8. How do you like to do this? 9. In an ideal world, what would most help you do this? 10. What kinds of materials would you most like to share with other academic staff

and/or tutors?53

11. Would you like others to be able to edit your work/contribute to collective writing?12. What formats of files would you most want to upload? For example, Word

documents, PowerPoint presentations, etc.13. If you had access to a wiki or blog for a teaching community, what would you

consider the most important things that it should be able to do?14. What would you expect to be contained in the community policy/ground rules for use

of the wiki/blog? 15. What is the most important aspect of interacting with other academic staff and

tutors?54

16. How would you expect to access the wiki/blog?

The following additional questions were included in the UoL survey:

17. Would you be interested in participating in a UoL online community using wiki type technology to share resources and discuss common interests in teaching and learning?

18. What specific themes/topics/subjects would you like to see discussed within such a community?

19. What would be your priority ‘needs’ to receive from such a community?20. What would be your premier ‘offers’ to contribute to such a community?21. Willing for unattributed responses to be used in research? Yes / No?55

49 All OU respondents were part-time.50 All OU respondents were teaching distance learning courses.51 This question was slightly modified for the UoL version52 The term ‘teaching associates’ is used here as an equivalent to the OU AL. 53 This question was slightly modified for the UoL version54 This question was slightly modified for the UoL version55 One of the UoL respondents answered ‘No’ to this question and their comments and answers have therefore not been used.

APPENDIX: Focus Group questions

The trigger questions used for the OU focus group meeting were as follows:

1. How do you share ideas and resources with others right now? How would you like to do this? In an ideal world what would you like?

2. If you had access to a community wiki – what would you consider the most important aspects (please tick):

o Sharing information and ideaso Staff development resourceso Creating and editing web pages on topical subjectso Communication with other Associate Lecturerso Collect teaching and learning resources, e.g. articles, e-bookso eLearning topics – VLE, eportfolios, pedagogical application of wikis in

education, etc.o Address bookso Calendarso Noticeso Updates on what’s new on the wikio Newsfeed from, for example, BBC Education Onlineo Reference lists on topical subjectso Brainstormingo Relevant University and related projects and strategies e.g. younger

studentso Private spaceo Subject discipline resourceso Email notification of new updates to the wikio Other (please specify)

3. Have you tried or seen a blog – this is a tool for personal comments and reflections?

4. Would you like a blog in your wiki? 5. Should there be a community policy/best practice for a use of a wiki/blog – what

would you expect them to be? o Keep it legal – follow best copyright practiceo Considerate of others’ opinions, be politeo Keep it short and simpleo Don’t high behind jargon or excessive wordinesso Editing by other contributorso Password protectiono Moderationo Avoid making changes to others’ textso Other (please specify)

6. What is the most important aspect of meeting with other tutors? 7. Would you like to share materials with other tutors? 8. Would you want to upload files such as Word documents, Powerpoint

presentations?9. Would you like others to be able to edit your work/contribute to collective

writing? 10. Do you use conferencing or online discussion forum and if so what for? What is

the easiest aspect of using conferencing/discussion? what is the most difficult?11. Would you like the wiki to be accessible from Tutorhome/Other (please specify)?

APPENDIX: Interview Questions

Similar questions were asked at both OU and UoL. The interviewer was Chris Pegler.

Open University version

1. What do you understand by the term ‘personal and informal online repository’?

2. Having participated in this project, what do you feel is the potential for an online personal repository? (This need not be in the form (e.g. Elgg v0.4) used here).

3. What did you feel were the benefits and drawbacks of the system chosen?

4. What did you see as the purpose of the personal profile56? How did you feel about filling this in?

5. If this project were to continue to be available, what uses would you (personally) put it to?

6. The system allowed you to make some of your content private. What would motivate you to do this? Did you need any other personal controls on access?

7. What information about resources and postings – other than the profile – might you want/need in order to decide whether to view or use or share a resource? How could we capture that information?

8. How would you expect a personal and informal repository to adapt to your personal requirements? For example how do you currently control different versions of teaching resources? How do you share or describe them?

9. To what extent do you see your personal repository requirements, including your own vocabulary (personal tags or ‘personal folksonomy’), as separate from the wider teaching community (at institutional, discipline or national levels)? To what extent are they – should they – be integrated?

10. Finally, what level of moderation does this sort of initiative require? Where should any control or regulation come from?

11. Any other questions or points that you would wish to make?

56 This was only available on Elgg so only answered by OU participants

University of Leicester version

1. What do you understand by the term ‘personal and informal online repository’?

2. What do you feel is the potential for developing online personal repositories using wiki/blog technologies in general?

3. What did you feel were the particular benefits and drawbacks of the systems that you have so far used. [If not clear] Please tell me which systems you are referring to, eg within Plone or Blackboard, or other?

4. For personal and informal use do you prefer a system which you control or one which is provided for you by the institution?

5. Did you fill in any metadata fields in creating a personal profile for use within this product? If so, what did you see as the main purpose/s of such a profile? What sort of details did you provide? How did you ‘feel’ about providing these details?

6. If wiki/blog facilities continue to be made available to you, what uses for them would you (personally) anticipate?

7. Did the system allow you to make some of your content private? If so, what would motivate you to do this? Did/do you need any other personal controls on access (e.g. the facility to read only, to comment but not post, or to contribute)?

8. What information about resources and postings by others – other than the user profile – might you want/need in order to decide whether to view or use or share a resource? How could that information be captured?

9. How would you expect a personal and informal repository to adapt to your personal requirements? For example, how do you currently control different versions of teaching resources? How do you share or describe them?

10. To what extent do you see your personal repository requirements, including your own vocabulary (personal tags or ‘personal folksonomy’), as being separate from the wider teaching community (at departmental, institutional, discipline or national levels)? To what extent are they – or should they – be integrated?

11. Finally, what level of moderation, if any, does this sort of initiative require? Where should any control or regulation come from and what form should it take?

12. Any other questions or points that you would wish to make?

APPENDIX Accessibility assessment

This assessment takes the form of an expert assessment of and demonstration of accessibility problems encountered in typical navigation and use of the OU PROWE blog and wiki (Elgg v0.4 and PmWiki v2.1.11) and the UoL PROWE blog/wiki (Plone v2.1.3, Simple Blog v1.2.1 and Zwiki v0.58) systems.

The evaluator (Dr Chetz Colwell) is an accessibility expert employed within the Accessible Educational Media project at The Open University. She will engage with the two separate systems as a potential user would if they had disabilities which impacted on sight (blind or partially sighted), if they were dyslexic, or if they had mobility problems which prevented typical use of the mouse. As there is no sound involved in the resources held she will not be assessing the sites for hearing impaired user challenges. Where this reflects typical use she will use assistive technologies such as screen magnification and screen reader software.

Within both systems she will carry out a number of simple tasks relevant to use of the systems as a repository (i.e. to deposit or retrieve documents). Using a guest account she will:

1. Login2. Conduct a search for documents posted on a specific subject3. Browse for a document posted by a specific user4. Open and read three different documents posted by three different users5. Add a comment, including use of any preview facility6. Access personal profiles on users if available7. Upload a document from her own hard disk 8. Modify and re-deposit this document9. Navigate and explore the site more generally, e.g. clicking links and

traversing levels10. Exit

Any key issues on accessibility which Dr Colwell feels are significant and generic enough to merit demonstration will be recorded using the Institute of Educational Technology Data Capture Suite. The recording (on DVD) and a separate short report of her findings form part of this Evaluation Report.