Microsoft SharePoint Portal Server at Michigan State ...winna/documents/sharePoint/report.pdfused by businesses. As wikipedia.org says, “The primary reason for using SharePoint is

Microsoft SharePoint Portal Server at

Michigan State University Department of Radiology

SI 689 Final Report 10 December 2006

by

Derek Cooper Farah Faisal

Messeret Gebre-Kristos Michael Schleif

1

1. Executive Summary

1.1. Overview In July of 2003, the information technology group at the Michigan State

University, Department of Radiology began exploring the potential of Microsoft SharePoint Portal Server to help with information sharing and coordination issues. In January of 2004, the first implementation was rolled out. Since then, the SharePoint implementation has continued to grow as a major resource used by the group; however, it has not developed without problems. To help address these issues: from October to December of 2006, a group of graduate students studying Computer-Supported Cooperative Work at the University of Michigan, School of Information performed an evaluation of the SharePoint implementation.

1.2. Evaluation The evaluation aimed to discover how SharePoint was used within the group, and

what perceived and actual issues existed with the implementation. Rather than conduct a thorough evaluation of the entire SharePoint implementation due to time constraints, the evaluation focused on the original – and generally most frequently used – SharePoint features, including: the In/Out Board, Key Documents, Vacation Calendar, Tasks, and Announcements. In performing the evaluation, several techniques were used, including a heuristic evaluation, a user survey, and user testing. The findings from these evaluation methods were analyzed both individually and in aggregate to reveal a number of issues in the SharePoint implementation.

1.3. Findings and Recommendations There were a number of both positive and negative findings. Generally,

SharePoint is perceived to be relatively useful, primarily as a result of its flexibility, high accessibility and availability, and its role in improving group awareness. However, there are a number of negative findings as well, including:

• A general lack of documentation and help; • A perceived disparity in work versus benefit for some SharePoint features; • A perception that some SharePoint behavior is unintuitive; • The result that finding key documents can be prohibitively expensive.

Based on these findings, it is clear that SharePoint is a valuable asset to the group.

However, to improve the implementation, two major recommendations are made: 1. The SharePoint search engine must be fixed. It was shown to be a critical factor

in finding key documents. Without a reliable search engine, the value of SharePoint as a shared document repository is greatly diminished;

2. The group would benefit greatly from developing documentation on help, guidelines, and policies governing the use of SharePoint. Such documentation may help combat many of the issues discovered in the evaluation.

2

2. Introduction We evaluated the use of Microsoft SharePoint, a web portal for groups to share

documents via the internet. The goal of our evaluation was to analyze usability of the groupware and level of actual use by a particular group in a real work setting.

2.1. Microsoft SharePoint According to the Microsoft SharePoint website, “Microsoft Windows SharePoint

Services is a versatile technology that organizations and business units of all sizes can use to increase the efficiency of business processes and improve team productivity.” The system is easily customizable. It helps put into one location the daunting amount of data used by businesses. As wikipedia.org says, “The primary reason for using SharePoint is to provide taxonomy of corporate data. SharePoint provides a structure to otherwise disparate data.” Our user group uses SharePoint as an In/Out Board for keeping track of people, storage for key documents, a vacation calendar, a to-do list and a community bulletin board, among other things.

Figure 1: SharePoint

2.2. Users We decided to conduct our evaluation in the context of an actual work setting. We

felt this would allow us to make a realistic assessment of the groupware. We also

3

intended to solicit comments from users that have dealt with Microsoft SharePoint in the social and technological setting of a typical work place.

To that end, we focused on a group of IT professionals at the Radiology Engineering Group at Michigan State University. The group develops and maintains software for clinical information and digital imaging systems for the university’s radiology department. It also supports the general networking and computing infrastructure of the department. The group consists of approximately 20 employees all working in one location. They use SharePoint to coordinate their software development, technical support and documentation responsibilities.

2.3. Methodology Considering the abundance of customized features available in SharePoint, we

decided to focus our evaluation on five features that are most commonly used by the group: In/Out Board, Key Documents, Vacation Calendar, My Tasks, and Announcements. In/Out Board is a message board specifically devoted to displaying the comings and goings of the members on a daily basis. Key Documents serves as a repository of important documentation ranging from quick how-to references to detailed technical manuals. The rest of the features in the list are self-explanatory. This list is meant to cover a reasonable breadth of the system while allowing for sufficient depth of analysis.

We utilized several analysis tools for our evaluation. Using usability heuristics developed by Judy Olson each of us evaluated the Radiology Group’s SharePoint site, and subsequently evaluated is together as a group. We used Jakob Nielsen’s severity ranking for the sake of uniform evaluation parameters. The heuristic evaluation gave us an opportunity to gain a quick grasp of the system’s usability. We also conducted a survey to evaluate what the users at the Radiology Group think of the system. For a more intensive analysis, we additionally conducted user task analysis at the group’s work place. The results from each of these analysis tools are discussed in depth below.

3. Literature Review Discretionary Adoption of Group Support Software - Leysia Palen & Jonathan Grudin (2002) This article describes the adoption of calendaring software within large companies. One thing the study found which was particularly relevant to our project was the idea of peer pressure. Palen and Grudin found that after a certain number of users adopted the calendaring software, it became very difficult to schedule in non-users, thus leading to “substantial peer pressure” developing over time. We also found this in our application in the In/Out Board. This feature has now been almost universally adopted mainly due to peer pressure. When people declare where they are/where they will be on the Board, it is easy to find them should a problem arise, but when they do not use the Board, it creates problems for a number of people (similar to the people not using the calendaring software), which leads to peer pressure towards using the In/Out Board.

4

How Oversight Improves Member-Maintained Communities – Cosley, D., Frankowski, D., Kiesler, S., Terveen, L., & Riedl, J. (2005) This article describes a look at MovieLens as a member-maintained community. Overall, the article found that oversight improves outcomes and increases contributions, there was no difference between peer and expert oversight in quality or quantity of contributions, certain individual contributions are of higher quality than others, telling people about oversight may increase their motivation to contribute and opportunities to contribute must be obvious. This applies to our project in that we would like to recommend that the MSU SharePoint site be made into more of a member-maintained community. This would lighten the extremely heavy load on the current organizers of the site as well as make the site easier to navigate for everybody (assuming Cosley et al’s finding applies to this situation). Learning from Notes: Organizational issues in groupware implementation - Orlikowski, W.J. (1992) Orlikowski’s article discusses the adoption of Lotus Notes within a group environment. One very relevant result she found was that:

“[I]n the early adoption of a technology, cognitive and structural elements play an important role in influencing how people think about and asses the value of the technology. And these significantly influence how they choose to use the technology” (p.368).

This applies to our project in that we found many framing effects throughout the product. One example was during the user testing when we asked users to delete a document, and instead of simply going to a drop-down menu, most users switched to “Explorer” mode within SharePoint, which framed the SharePoint file system into a normal Windows file system, making it easier for the users to delete the document. These framing effects help to make the system more intuitive but, at the same time, bring up the idea that users may not find the default view mode as intuitive as “Explorer” mode. More research may be needed. Groupwork close up: A comparison of the group design process with and without a simple group editor - Olson, J.S., Olson, G.M., Storrøsten, M., & Carter, M. (1993) This seminal paper was discussed several times during class and involves observations of group work with and without group editing software called ShrEdit. ShrEdit allowed groups to produce higher quality ideas, even though they disliked the process a bit less (probably from having to learn new technology). Additionally, the researchers found an increased focus on core ideas and a decrease in focus on auxiliary, nonproductive ones. We feel this is relevant to our product in that it closely parallels how this group worked before and after SharePoint. (NN)

5

Proactive Support for the Organization of Shared Workspaces Using Activity Patterns and Content Analysis – Prinz, Wolfgang and Zaman, Baber (2005) Prinz’s article analyzes user requirements, system architecture, ontology, activity pattern building, of a Semantic Workspace Organizer (SMO) very similar to SharePoint. This technology also had an embedded “textual similarity suggestions algorithm” that suggested not only where users should look for documents, but where they should upload them as well. These suggestions were successful and were accepted and seen as useful by the products’ users. We feel this is important to our product because there are some fairly major organizational issues, and an algorithm similar to this SMO might help the MSU Radiology IT group quite a bit. Additionally, we took this paper as an example while we were developing our methodology. Supporting Informality: Team Working and Integrated Care Records – Hardstone, Gillian, Hartswood, Mark, Procter, Rob, Slack, Roger, Voss, Alex and Rees, Gwyneth (2004) This article describes an in-depth look at an integrated care records (ICR) system. Although this is more of an evaluation of a particular ICR system, the researchers do come up with a fairly interesting suggestion:

“We argue that instead of the cscw community developing horizontal apps, which are not linked to specifi working practices and sold as ‘groupware,’ there should be a shift of focus to what might be seen as CSCW middleware. This would comprise flexible, customizable software components that aim to support informal and collaborative working practices, but that can be made part of large scale (inter-)organizational systems” (p.149).

This applies directly to our project in that this is exactly the functionality SharePoint was intended to perform in the MSU Radiology IT group’s setting. What Happened to our Document in the Shared Workspace? The Need for Groupware Conventions – Mark, G. and Prinz, W. (1997)

The study undertaken by the authors examines a large groupware application built to establish a shared workspace for German federal agencies in Bonn and Berlin. The authors argue the importance of establishing conventions (e.g. rules governing document naming, versioning, access, storage, private vs. public work, etc.) in shared workspaces, based on findings from a series of workshops held with the federal agencies. Further, the authors outline challenges in establishing group conventions and suggestions on how to overcome of these challenges. Given the nature of the Microsoft SharePoint Portal Server application as a shared workspace for the MSU Radiology IT group and the prevalence of its use for shared documents, the study is particularly applicable to our evaluation.

6

4. Heuristic Analysis

4.1. Methodology

4.1.1. Heuristics and Ranking System A heuristic evaluation provides a quick and low cost usability evaluation. This

evaluation is quick because the list of heuristics is divided between group members and evaluated using a pre-existing persona. A heuristic evaluation is low cost because it does not involve any direct user testing. Heuristics allow the evaluator to find the major usability problems of a system without in-depth testing.

The heuristics used were developed by Judy Olson. We then used the groups these heuristics fit into to describe the broad problem areas of Microsoft SharePoint. A complete list of heuristics along with their grouping can be found in Appendix A. The eight general categories evaluated are given below: # Heuristic Category 1 Correspondence to the way the user thinks about the task 2 Consistency 3 Visual Display 4 Menu/Command Structure 5 System Response Time 6 Error Recovery 7 Help 8 Documentation and Training

In order to get a better understanding of the severity of the heuristic problem, we

utilized Jakob Nielsen’s rating scale for usability problems. Additionally, the ranking system was used to help ensure consistency in assessing severity of problems. These rankings are:

Ranking Definition

0 I don’t agree that this is a usability problem at all. 1 Cosmetic problem only; need not be fixed unless extra time is available

on project. 2 Minor usability problem: fixing this should be given low priority. 3 Major usability problem: important to fix, so should be given high

priority. 4 Usability catastrophe: imperative to fix this before the product can be

released.

This system allowed evaluators to compare problems within and outside of their heuristic grouping. For an original copy of Jakob Nielsen’s ratings, see http://www.useit.com/papers/heuristic/severityrating.html.

7

The four evaluators investigated the site separately, and later met together to

compare notes and consolidate findings.

4.2. Findings Overall the Microsoft SharePoint site did well on the heuristic evaluation.

However, we found several areas with significant shortcomings.

4.2.1. Summary The following summary table lists the aspects of the Microsoft SharePoint Site

with major flaws:

Nielsen Ranking Issues

Heuristic Category

Specific heuristics violated

3

'Delete' action is performed differently inside of Shared Documents section than in every other section and was not found at all by at least 1 group member

Consistency

#4

3

There is no static menu on the bottom of the page like the menu at the top. This frustrates users when the pages get very long and users have to scroll all the way back to the top to get the command buttons.

Menu/Command Structure

#15

3 The ‘help’ documentation is almost completely without context, making it very difficult to use.

Help #25

A complete list of heuristics with our rankings can be found in Appendix A.

4.2.2. Details


Heuristic Category


3

'Delete' action is performed differently inside of Shared Documents section than in every other section and was not found at all by at least 1 group member

Consistency

#4

Problem Details:

8

We found that deleting a document is done differently within the Shared

Documents section than in every other section of SharePoint. Within most sections of SharePoint, deleting is done by clicking on the “Delete Item” icon and then confirming that action (see Figure 2). This was given a three out of four because most of our evaluators were completely unable to find the delete function in the drop down menu, including one of our evaluators who had used SharePoint many times in the past.

Figure 2: Normal Delete

However, when the user is in the Shared Documents section, the user must choose the delete option from a hard-to-see drop-down menu (see Figure 3 below).

Figure 3: Location of delete drop down menu within Shared

Proposed change:

The delete command should be moved from the drop down menu to the upper section in order to be more consistent. The drop down menu may have been used as an extra level of security (“Security by Obscurity”) to keep users from deleting important documents, but we feel that the pop up notice given before deleting anything in SharePoint is sufficient enough to keep users safe.

9


Heuristic Category


3

There is no static menu on the bottom of the page like the menu at the top. This frustrates users when the pages get very long and users have to scroll all the way back to the top to get the command buttons.

Menu/Command

Structure #15

Problem Details: At the top of SharePoint, there is a static menu that provides links to the most important sections of the site (see Figure 4 below). However, at the bottom of the web page, there is nothing. This was given a three because pages can be quite long in SharePoint, and not having these links can be extremely frustrating to users.

Figure 4: Static Menu

Proposed change: We feel that the addition of a static menu at the bottom of the page would greatly help users, especially when they are navigating the long pages normally seen within SharePoint.


Heuristic Category


3 The ‘Help’ documentation is almost completely without context, making it very difficult to use.

Help #25

10

Problem Details: Help documentation within SharePoint is almost completely without context. Seeing as how important help is within any product, we gave this a three out of four. The reason this was not given a four out of four was because the help can be navigated if the user is an expert and because the interface is simple enough that help would not be needed for the majority of users. Proposed change: Unfortunately for Microsoft, we feel that the ‘Help’ feature in SharePoint needs to be largely rewritten in order to be more useful. We would also like to see help similar to CTools, where question marks appear in the headers for certain sections (see Figure 5 below) instead of a static help icon at the top of every page that links to a single document.

Figure 5: Location of contextual help in CTools

5. Survey

5.1. Methodology The goal of our survey is to gauge the general feelings of expert users in regards

to SharePoint’s overall usability and effectiveness. We would also like to know what functionality of SharePoint is well received and which needs further improvements.

Since we are mainly focusing on the MSU Radiology IT group (RIT), the survey

was only sent out to our target population. They consist of 20 people who are either full time employees or student interns. We created an online survey using SurveyMonkey.com and sent it to the email list.

11

We used SurveyMonkey.com and developed an 11-question survey. We decided to do an online survey as opposed to physical paper survey because all members of the target population have access to the internet at least during their working days, and because it is easier to tabulate and analyze responses using SurveyMonkey’s built in capabilities. This survey made use of open- and closed-ended questions and was designed in order to elicit the respondent’s feelings towards SharePoint. The full survey is given in Appendix B.

The survey was sent to 20 people through the email list and received 13 responses by the end of the survey period. Therefore, about 65% of the total email recipients responded to our survey. Since we have a small target population, we believe that we have received enough responses for each of the target group that we have chosen to evaluate so that we are able to provide some meaningful data in our analysis.

5.2. Findings Overall, 84.6% of the respondents found SharePoint to be useful, with 54.6% of

the respondents rating SharePoint to be “very useful”. More than half of the of the total respondents uses SharePoint on a daily basis, showing significant interaction with the web application, though only 3 people (21.3%) consider themselves to be expert users. We asked the respondents to rate the different implementations of features in SharePoint, which are:

• In/Out Board • Key Documents • Vacation Calendar • Announcements • My Tasks

Each of the implementation received high “very useful” ratings, leading us to believe that since the implementations are created and maintained by members of the group, they are tailored more towards the group’s best practices.

Figure 6: Survey Results

12

We also asked respondents to rate the frequency of use for each of the features. The results are that the In/Out Board is used daily (and thus receives the most frequent usage), while Announcements and My Tasks are used least frequent (both monthly). The Vacation Calendar and Key Documents are both used weekly.

We also posed a few open-ended questions to allow users to comment on what they like and don’t like about SharePoint, as well as giving additional comments about the overall usability and functionality of the application. We found that respondents liked the fact that SharePoint is always accessible as long as there is a computer and an internet connection. Therefore, they are able to access their work and account without having to physically be in their work environment. SharePoint also allows some measure of transparency in regards of other team member’s availability, location and also projects that they are working on. The flexibility of SharePoint also received positive reviews as it allows the site to be tailored toward the group’s best practices.

However, we also received some negative comments regarding the implementation of SharePoint or the functionality of SharePoint itself. We received comments regarding the search feature of SharePoint, which, during our evaluation time period, was not functioning. This makes it extremely difficult for users to find documents and lists in the site, especially since the site contains a lot of documents that are stored in various folders. Another comment was the lack of documentation available for SharePoint. This may be due to the fact that SharePoint is meant to be customizable; however the lack of resources, such as books, on how to use SharePoint, proves to be an issue for some users.

Some of the features also proved unintuitive. For instance, one respondent commented that he has to choose his own name when he wants to create a new entry either in the In/Out Board or the My Task list. Not only is this a hassle, but it made him prone to making mistakes, such as selecting the wrong name from the list or forgetting to make the selection completely. His suggestion was to have the system recognize his information immediately from login to reduce error and also redundant actions. We found that by making the user select a name when creating a new entry, this would allow a user to create an entry for another user, such as an assistant creating an entry for her boss. However, this concept does not map well with the MSU RIT group culture, where users are more often than not make their own entries. Another frequent comment that also pertains to lack of intuitiveness of the SharePoint site is regarding the Vacation Calendar. A user would navigate to the date that he would like to make an entry in and click on the date link, where a form would pop-out. However, instead of having the date selected as the entry date, the form would revert back to the date of the day the entry was made instead of what was selected. This proved to be unintuitive to most, if not all, of the users and made them prone to mistakes.

There was also a lack of perceived benefits from using the features of SharePoint. This implementation of SharePoint is fairly new, and some users are more comfortable with using other applications to perform certain tasks. For instance, some users are more

13

comfortable in using email instead of the Announcement section in SharePoint, as evident from this comment:

“Announcements could be very useful, but we've never developed a mentality for actually using them, which is my fault. I think we are all just more comfortable with email.”

Our results from the survey are used as basis for the tasks we developed for our usability test, which will be explained in a later section.

6. Usability Testing

6.1. Methodology As part of the SharePoint analysis, a user task evaluation was conducted with the

aim of observing how SharePoint is used and ascertaining usability issues. User task evaluations are well known to be tools for effectively performing usability tests. This method was primarily chosen due to SharePoint's asynchronous usage pattern, primarily as a reference. Given a relatively small evaluation group and a limited amount of time, observation would have been infeasible, particularly as the user group was located at a remote site. As a result, a user evaluation was conducted where the users performed a script of tasks created to simulate every-day activities. While a compromise between context and convenience was made in this manner, as noted, it is arguable that the trade-off is justified.

Four users were solicited for conducting the evaluation. Each evaluation consisted of a pre-test survey, eight tasks, and a post-test survey respectively (see Appendix C). Further, the evaluations were designed to take approximately 15 minutes from start to finish. With the scripted tasks based on every-day activities, this would allow us to approximate the working environment with minimal interruption to the participants. The testing was completed in a one-on-one fashion, with a proctor available to provide guidance to the participant as necessary. The evaluation was performed on a laptop PC with the ability to synchronously record the visual display, audio from the user, and video of the user during testing.

After the evaluations were completed, they were analyzed independently by each member of the group. After the individual analyses were completed, the group reconvened to discuss the findings, address any emerging patterns, and summarize the overall findings.

6.2. Findings A number of interesting findings emerged from the results. First, the users were

asked to respond to a pre-test questionnaire. When asked to self-select their level of expertise, three users considered themselves average users, while one user considered herself an expert. Additionally, the expert user found SharePoint to greatly improve her productivity, while another user found it to only somewhat improve her productivity, and

14

two users found it to have no affect on their productivity. This might suggest that there may be some correlation between the level of expertise with SharePoint and its perceived benefit on productivity. With the small sample size, further testing would have to be completed to be certain. Furthermore, all users perceived SharePoint to somewhat improve collaboration within the group. In addition, each user was asked to rate the ease-of-use for each of the functions being evaluated: the In/Out Board, Tasks, Vacation Calendar, Announcements, and Key Documents. Although the In/Out Board consistently received an easy to use rating, the other responses varied.

After completing the pre-test questionnaire, the users began the usability tasks. While a number of interesting findings emerged from the results of the usability tasks, three major findings stood out:

1. evidence of emerging group conventions in some areas; 2. evidence of mental models and framing effects; 3. and evidence that finding particular documents may be a prohibitively expensive

task. It was interesting to note that while users tended to perform sub-tasks in different ways (e.g. which buttons they click, how they categorize an entry, etc.), they generally seemed to converge on the end result of the task. For instance, the second task asked the users to enter a full work day with a lunch break and an afternoon meeting into the In/Out Board. While one user made a single entry in the In/Out Board with a note referring to the lunch break and meeting, the other three users made separate entries for the full work day, the lunch, and the meeting. Additionally, when asked to make an entry in the Vacation Calendar in task seven, each user followed the same convention, ensuring the user’s initials appeared as part of the title in the calendar entry. Two users did not add their initials originally, but upon verifying the entry and seeing other entries, both users explicitly corrected their entries to include their initials.

Perhaps one of the more interesting findings was evidence of mental models and technological frames (Orlikowski 2002), which were unmistakable in task 8, deleting a document. When asked to delete a document, only the self-selected expert user went immediately to the document and directly to the delete menu on the default view (see Figure 7). The remaining users spent a fair amount of time trying to determine how to delete the document before switching to the Explorer view (see Figure 8). The Explorer view is a paradigm consistent with the view of the file system commonly used in Microsoft Windows (see Figure 9). Additionally, the Explorer view allows the user to use a right-click context menu to delete the document, similar to the behavior found in Microsoft Windows. As noted earlier in the report (see Heuristic Evaluation), deleting from the default view of SharePoint is problematic. Again, it is arguable that the expert user’s advanced level of understanding SharePoint enabled her to quickly and easily delete a document without switching to the Explorer view. On the other hand, the other users were somewhat unfamiliar with the SharePoint functionality and therefore moved to a more familiar environment.

15

Figure 7: Default SharePoint file system view for deleting a document.

Figure 8: Explorer view of SharePoint file system for deleting a document.

16

Figure 9: A common view of the Windows file system that can be used to delete a document.

The most important finding may be the limitations of SharePoint as a shared document repository. While it appeared to effectively store documents, finding particular documents proved very challenging and prohibitively expensive at times. In task five, users were asked to find a particular document. While the time to complete the other tasks in the usability evaluation tended vary somewhat, none of them varied as widely as task five. The time to complete task five ranged from approximately 27 seconds to nearly seven minutes. Clearly, taking seven minutes just to find a document is problematic.

Essentially, two major problems with the document repository were found: 1) the search engine did not work and 2) the repository lacked organization. While one user was able to quickly navigate directly to the target document, the other three users started their search with the search engine. One of these users then noted that the search engine was broken and then proceeded to browse to the target document in reasonable time. However, the other two users spent several minutes attempting to locate the target document, switching frequently between browsing and attempting to use the search engine. One of these users ultimately ended up skipping task five, and the other was eventually shown where to find the document. Clearly the broken search engine was crippling to the search process.

The organization of the share document repository proved problematic as well. While one user was able to quickly browse directly to the document, it was later

17

discovered through conversation that he used the document frequently and knew exactly where it was located beforehand. However, none of the other users were able to navigate directly to the target document. Further, two of the users were never able to find the document without help from the proctor (this was necessary as later tasks relied on users finding the document). At one point, a user commented on the shared document repository: “The biggest problem with this is the organization of it”. Again, the disorganization of the shared document repository is damaging to the search process.

After the usability tasks, the users were presented with a post-test questionnaire, in part to help verify the difficulty of the tasks. Not surprisingly, while the responses suggest that most of the tasks were relatively easy, task five – finding a particular document – was consistently rated fairly difficult. Additionally, it is interesting to note how user perceptions changed related to SharePoint’s impact on productivity after performing the usability tasks. The perceptions of two users were unaffected. However, of the two users who originally stated that SharePoint had no affect on their productivity, after testing, one user found SharePoint to actually somewhat decrease productivity while the other found it to somewhat increase productivity.

6.3. Recommendations While the framing affects were not prohibitive, their presence suggests that more

user training may prove beneficial for understanding SharePoint’s entire set of features and its full potential as a collaborative tool. Additionally, while some user conventions seem to have emerged, helping with consistent use within the group, they are clearly not always employed. As a result, it may help to develop some policies surrounding the use of SharePoint to help enforce consistency. Furthermore, and more importantly, SharePoint’s document repository was shown to prohibit use in some instances, partially as a result of disorganization. Research suggests that establishing policies in this area would be beneficial as well (Mark and Prinz 1997). On a similar note, given heavy reliance on the SharePoint search feature for finding documents, fixing the search engine should be of the highest importance.

6.4. Limitations It is difficult to deterministically measure variances in tasks, particularly since

there are a number of different ways to complete the same task. For instance, in a couple of cases there seemed to be some uncontrollable slowness issues that may have impacted timing. Additionally, unexpected events occurred from time to time as well. For instance, the Key Documents feature was expanded after we started our project, providing multiple storage locations for the same document. One of the users went to the alternate location during the usability testing.

7. Results and Recommendations Through our evaluations, we found that SharePoint is generally a useful

application in enabling users to share documents and perform other collaboration activities. The flexibility of being able to tailor the components according to the group’s

18

needs and best practices is useful in improving efficiency by allowing group-specific components to be created.

However, we found some issues with the application. Its flexibility means that some policies needed to be implemented regarding its usage. Constant monitoring is also needed to ensure that documents are ordered and can be found relatively easily. To ensure that the site continues to run smoothly, a small dedicated team is needed to frequently maintain the site.

Since the site holds a large amount of documents and lists which are important to be readily accessible to users, the search feature needs to be improved. Currently, it is prone to breakdowns, which makes the task of finding documents a lot harder than it should be. It has been reported that the search feature has been improved in the next implementation of SharePoint, which we hope would resolve the issues that both users and administrators are currently facing.

We also found that users faced problems when it came time to deleting documents. Some even resorted to changing the list view to an explorer view so that they can perform the delete task as they would on a Windows desktop. This poses unnecessary burden on the user by making them take the extra steps to change the view. One recommendation is to have improved consistency when it comes to deleting entries and documents from the system.

8. Conclusion Due to the highly customizable nature of the Microsoft SharePoint portal, it can

be argued the system has two aspects that can be evaluated: First, the process of building different custom features to set up the portal for a particular work setting, and Second, the actual usage of the basic and custom features once the system is set up. Our evaluation focused on the latter aspect. Our aim has been to evaluate the system usage in an actual work context.

We analyzed the Microsoft SharePoint portal from various perspectives. Our project group built a SharePoint site and used it to coordinate our work throughout the semester. Additionally, most of our analysis involved evaluating a site that has been operational with a group of users for a few years. We conducted heuristic evaluation to gauge its usability. We surveyed our user group to gain quantitative and qualitative data on their experience with the site. We also conducted user test for in-depth look at the way the site is utilized by a representative sample of our group.

Our findings generally point out that SharePoint is used constantly. More than half of the users surveyed rated it ‘very useful.’ It has even influenced the work habits in some ways. For example there is a tacit expectation that everyone would update the In/Out Board to keep colleagues aware of one’s whereabouts. The ease of customization helpfully enables users to tailor their use of the site according to their established work habits. It is also usable mainly because it has features that users are likely to be familiar

19

with from exposure to other popular Microsoft products.

However, we have also found some flaws. The most glaring problem is its dysfunctional search utility. Our users rely on the site as the main repository for their documents. Subsequently, there are many documents placed in folders and subfolders which lack easily discernible organization logic. The problem of disorganization is exacerbated because users looking for documents can’t search for them. They are compelled to browse, a process that usually ends in frustration because documents are not found where they are generally expected.

The lack of organization itself is the negative effect of customization. Customization is generally desirable in single-user applications because it affords flexibility. Flexibility has its value in groupware as well. At the same time, usually a few people in a group setting take the initiative to customize the system (organizing the folder system, for example) in a manner that makes sense only to them. The rest of the group is then confounded when trying to use the customized features.

Another deficiency of the SharePoint site, lack of documentation, compounds the problems related to search and file system organization. There is little documentation on how to use the site. This is critical especially in relation to the parts that are customized for the particular group. If the individual who did the customization doesn’t explain how to use the feature, it is difficult for the rest of the group to leverage the flexibility of the system.

Our evaluation focused on the use of Microsoft SharePoint usage in a particular setting of twenty users. While this served our purpose of studying the system in a real setting, we realize there may be some limitations in terms of how much we can generalize our finding to other contexts of SharePoint usage. We feel confident that we got a comprehensive perspective on the system with the number of users we observed in our evaluation. For example, the need for a functional search utility is indisputable without looking for a larger user group. But our user group has its peculiarities that may not be extended to other work groups. If the user group were not a tightly knit set of IT professionals, would they customize the site so much? Would they put so many documents in the site that it becomes such an important repository of files? These are some caveats that need to be borne in mind when trying to apply the findings to other groups.

This brings up a good point about the need for feature research that could look at usage of SharePoint in various settings and determine if a set of best practices have emerged. Another possible work in the future is to look at the influence of different work settings on the use of SharePoint. It would be interesting to see if there is some correlation with the types of work settings and the patterns of using SharePoint in those settings.

20

9. Works Cited Cosley, D., Frankowski, D., Kiesler, S., Terveen, L., & Riedl, J. (2005) How oversight

improves member-maintained communities. In Proceedings of CHI 2005. New York: ACM. Pp. 11-20.

Couper, Mick (February 2006). Survey Research and the Web: An Introduction. Lecture

presented in School of Information 622. University of Michigan, Ann Arbor, MI.

Grudin, J. (1994) Groupware and social dynamics: Eight challenges for developers. Communications of the ACM, 37(1), 93-104.

Hardstone, Gillian, Hartswood, Mark, Procter, Rob, Slack, Roger, Voss, Alex and Rees,

Gwyneth (2004). Supporting Informality: Team Working and Integrated Care Records. In Proceedings of 2004 ACM Conference on CSCW. New York: ACM. Pp. 142-151.

Kuniavsky, Mike (2003). Observing the User Experience: A Practitioner’s Guide to User

Research. San Francisco: Elsevier Science. Mark, G. and Prinz, W. (1997). What Happened to our Document in the Shared

Workspace? The Need for Groupware Conventions. In Proceedings of the IFIP Tc13 interantional Conference on Human-Computer interaction (July 14 - 18, 1997). S. Howard, J. Hammond, and G. Lindgaard, Eds. IFIP Conference Proceedings, vol. 96. Chapman & Hall Ltd., London, UK, 413-420.

Olson, J.S., Olson, G.M., Storrøsten, M., & Carter, M. (1993) Groupwork close up: A

comparison of the group design process with and without a simple group editor. ACM Transactions on Information Systems, 11, 321-348.

Orlikowski, W.J. (1992) Learning from Notes: Organizational issues in groupware

implementation. In Proceedings of CSCW 92. Pp. 362-369. Palen, L. & Grudin, J. (2002) Discretionary adoption of group support software. In B.E.

Munkvold (Ed.), Organizational implementation of collaboration technology. Prinz, Wolfgang and Zaman, Baber (2005). Proactive Support for the Organization of

Shared Workspaces Using Activity Patterns and Content Analysis. Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work. New York: ACM. Pp. 246-255.

Rubin, Jeffrey (1994). Handbook of Usability Testing: How to Plan, Design, and

Conduct Effective Tests. New York, NY: John Wiley & Sons, Inc. http://www.microsoft.com/technet/windowsserver/sharepoint/techinfo/overview.mspx

21

http://en.wikipedia.org/wiki/Microsoft_Office_SharePoint_Portal_Server

22

10. Appendix A: Olson Heuristics and Nielson Rankings

Correspondence to the way the user thinks about the task 1. The terms the software application uses (what it calls the objects and what it calls the actions) should be as close as possible to the words users normally think of in conducting the task RANK: 0 2. If new words must be used, they should be metaphorical and concrete. RANK: 0 3. The order in which the user performs the subtasks should correspond closely with the order the software requires them to be performed. RANK: 2 Consistency 4. Actions should be performed in similar ways on the objects in the application (e.g., all objects should be deleted in the same general way). RANK: 3 5. The way the user "backs up" or cancels an action should be the same, no matter where the user is in the interaction. RANK: 2 6. The closer the command structure is to the structure of commands in other well-known applications, the easier the new one is to learn. RANK: 0 Visual Display 7. Readability of fonts increases if they are presented in both upper and lower case, serif fonts, variable width strokes, and with appropriate separation. RANK: 0 8. Related items should be grouped spatially; unrelated items should be separated. Gestalt principle of Proximity. RANK: 2 9. Color, highlighting, and special fonts such as bold and italics should be used to indicate similar meaning for those things that cannot be spatially grouped. Gestalt principle of Similarity. RANK: 0 10. Screen areas should be used consistently for similar functions. e.g., error messages always displayed in the same, easy to see place. RANK: 0 11. Display familiar items in familiar ways. e.g., music is shown as musical notation, spatially related items are shown in a spatial display. RANK: 0 12. Error messages should be presented in way that captures the user's attention. e.g., flashing, reverse video, color, accompanied with a tone. RANK: 0 Menu/Command Structure 13. For new or casual users, commands presented on menus are easier to learn and remember than commands that have to be typed in without a prompt. For experienced, dedicated users who know the command set well, commands entered with quick hand motions are preferred. RANK: 2 14. Commands should be formulated with a consistent grammar: The order most closely associated with commands in English is Verb-Object-Modifier ("put-that-there"). [This is a very controversial point; many good, consistent systems have a grammar that is Object-Verb-Modifier; users find it easy to learn and easy to operate.] RANK: 0

23

15. Frequently used commands should be easily accessible. e.g., frequent menu selections should be near the top of the menu. RANK: 3 16. Related commands should be grouped on menus. RANK: 0 17. If there are more than one way to do something, instructions should be clear about when each method should be used for maximum efficiency. RANK: 1 System Response Time 18. Responses to keypresses or mouse movements should be within 100 msec of the act. RANK: 0 19. Normal computed responses (e.g., responses to a request for a font change of the document) should take place within 2 sec, if they are part of an ongoing task. RANK: 0 20. The only times when system responses can be longer than 2 sec and not disrupt the user's cognition is when they occur at the end of a task, and the user is alerted that the time will be long. Showing how long that delay will be is helpful to the user to decide whether to engage in a secondary task during the time. RANK: 2 Error Recovery 21. Error messages should be clear statements of both the problem and how to recover from it. RANK: 1 22. All actions should have a natural reverse, accessed through an "undo" command. RANK: 1 23. Actions that have severe consequences should be prompted with "are you sure?" e.g., actions that delete an entire file should be prompted. RANK: 2 Help 24. Help messages should be accessible from any point in the dialog. RANK: 0 25. Help should help. It should be written specific to the context in which the help was requested. It should be written sensitive to the task the user was engaged in. RANK: 3 Documentation and Training 26. Training should be specific to the tasks the user is most likely to be engaged in. RANK: 2 27. Documentation explains both what the functions are in the system and how to use them in common tasks. RANK: 2 28. Documentation should be accessible through a good keyword index (in the user's vocabulary) and through tabs, headers, and a table of context that makes the grouping of functions obvious. RANK: 2

24

11. Appendix B: Survey Feel free to answer as many or as few questions as you want. This is a completely anonymous survey. Your answers will be analyzed and then destroyed by an irate survey monkey. Please answer questions to the best of your knowledge. Thank you for your participation!

1. How do you rate your expertise with SharePoint?

Novice user

Average user

Expert user

N/A

Other (please specify)

2. How useful do you find SharePoint overall?

Not useful at all

Somewhat not useful

Neutral

Somewhat useful

Very useful

N/A

3. How often do you use the following SharePoint features:

Never Monthly Weekly DailyMultiple

times per day

N/A

In/Out Board Key Documents

25

Vacation Calendar

My Tasks Announcements

4. How useful do you find the following SharePoint features:

Not

useful at all

Somewhat not useful

NeutralSomewhat

useful Very

useful N/A

In/Out Board Key Documents

Vacation Calendar

My Tasks Announcements

5. Do you have any additional comments about the features listed above?

6. What do you like about SharePoint?

7. What do you dislike about SharePoint?

8. Are there any changes you'd like to see made to SharePoint?

26

9. Are there any other services you use that are similar to SharePoint?

Yes

No

N/A

10. If you answered yes to the above question, which services do you use?

11. Do you have any additional comments you would like to share?

Thanks for your time!

27

12. Appendix C: Usability Test Materials

12.1. Pre-Test Questionnaire 1. Gender

Male Female

2. What age group are you in?

Under 18 18-24 25-34 35-49 50-64 65+

3. How do you rate your expertise with SharePoint?

Expert Average Novice Other

4. Please rate the SharePoint functionalities according to ease of use

1 - Very easy 2 - Somewhat Easy 3 – Neutral 4 - Somewhat Hard 5 - Very hard

____ In/Out Board ____ My Tasks ____ Vacation Calendar ____ Announcements ____ Key Documents ____ Other functionality we missed (e.g. pictures, discussion feature).

Please specify: __________________________________

5. Based on your current knowledge of SharePoint, what effect do you think it has on your productivity?

Greatly improves Somewhat improves No effect Somewhat decreases Greatly decreases

28

6. Based on your current knowledge of SharePoint, how do you think it effects collaboration with co-workers?

Greatly improves Somewhat improves No effect Somewhat decreases Greatly decreases

7. Based on your current knowledge of SharePoint, how likely are you to

recommend it to other people? Very likely Somewhat likely Neutral Somewhat unlikely Very unlikely

12.2. Post-Test Questionnaire For each of the tasks you have performed, please rate the ease of performing the task.

1. Logging into SharePoint (Task 1):

N/A …..Very easy ….. Somewhat Easy ….. Neutral….. Somewhat Hard ….. Very hard

2. Checking into the In/Out Board for an 8a-5p shift with a 12p-1p lunch break and a

3p-4p meeting with GE (Task 2):


3. Adding Dr. Potchen’s task to the task list (Task 3):


4. Identifying who is currently at the office (Task 4):


5. Finding ''How to Restart DAS Services on PACS SE 2_0.doc ' (Task 5):

29


6. Uploading the newly edited 'How to Restart DAS Services on PACS SE 2_0.doc’

back to the site (Task 6): N/A …..Very easy ….. Somewhat Easy ….. Neutral….. Somewhat Hard …..

Very hard

7. Specifying that you will be on vacation between Christmas and New Year’s (Task 7): N/A …..Very easy ….. Somewhat Easy ….. Neutral….. Somewhat Hard …..

Very hard

8. Deleting 'How to Restart DAS Services on PACS SE 2_0.doc’ (Task 8):


9. Based on your current knowledge of SharePoint, what effect do you think it has

on your productivity? [ ] Greatly improves [ ] Somewhat improves [ ] No effect [ ] Somewhat decreases [ ] Greatly decreases

10. Do you have any additional comments?

12.3. Usability Tasks 1. You just sat down at your desk for the day. Log-in to Windows, open the

‘SharePoint’ icon on the desktop, and log-in to SharePoint. 2. First, be sure to check into the In/Out Board. You are planning to work an 8a-5p

shift with a 12p-1p lunch break and a 3p-4p meeting with GE.

3. The morning is pretty quiet until ~9a when you receive a call from Dr. Potchen demanding that you ensure his home PC is upgraded to Windows Vista by tomorrow morning: add this to your task list.

4. At ~2p, you receive a call from Tracey that she is having difficulty sending exams

to PACS. Before calling Robert, you check the In/Out Board to see if he is in.

30

5. Also, before you call Robert, you examine the PACS yourself and discover that the DAS services do not appear to be running. You seem to recall a document illustrating how to re-start DAS services: find and open 'How to Restart DAS Services on PACS SE 2_0.doc'.

6. Upon opening the document, you realize there is a mistake: the title reads ‘How to

Restat DAS Services on PACS SE 2.0 (w/o rebooting)’. Correct the spelling error and ensure your changes are permanently reflected in the document.

7. You found the information for which you were looking and the problem is

resolved. The rest of the day is relatively uneventful until your meeting with GE. After working with them for the past year on implementing a new PACS, only to find out that very little progress has been made, you become frustrated and decide that it’s about time for a vacation. Find some time in the upcoming holiday season – between Christmas and New Year’s – to take a vacation. Be sure to make an entry in the Vacation Calendar.

8. Afterwards, you sit around thinking about the problem with the DAS services you

encountered earlier in the day. You realize that the documentation is relatively useless – that the ROC at GE could better handle such problems. Delete the 'How to Restart DAS Services on PACS SE 2_0.doc' document.