QA Focus – Supporting JISC's 5/99 Programme The QA Focus Perspective Marieke Guy UKOLN [email protected]

QA Focus – Supporting JISC's 5/99 Programme

The QA Focus Perspective

Marieke Guy

UKOLN

[email protected]

http://www.ukoln.ac.uk/qa-focus/http://www.ukoln.ac.uk/qa-focus/


The QA Perspective

If you want to sort out the problems on your Web site you can do so by developing a Quality Assurance Framework

This involves:• Testing• Problem tracking and corrective action• Policies• Procedures• Documentation


What Can You Test?

• Functional testing • Compatibility testing• Load/performance testing• Stress testing • Usability testing• Security testing• Integration of unit testing• Link testing• HTML and CSS validation• Reliability testing• Regression testing• …


Automated Testing• Should be scalable• Only suitable approach for large sites• Can push (email) information about

problems• Many free testing tools available

but:• Will not spot all usability/accessibility

problems• Can provide too much information• Comprehensive and configurable

testing tools can be expensive


Manual Testing• Can spot usability/accessibility problems• Can make use of one’s community/user

base/peers• Can use of key pages which will identify

problems which occur elsewhere • Useful for testing new Web sites• “5 users can spot most of your problems”

but:• Not suitable for testing large numbers of

pages• Quality of feedback may be variable• Often not useful for testing

established Web sites


Approaches to Automated Testing

Automated testing can make use of:• Web-based tools• Desktop tools

UKOLN has made use of Web-based testing tools:

• Monitor accessibility, HTML, CSS, compliance, page size, links quality, nos. of links, etc.

• Benchmarking approach provides comparison with one’s peers

http://www.ukoln.ac.uk/qa-focus/surveys/http://www.ukoln.ac.uk/qa-focus/surveys/


Peer Review

• “Give me comments on my Web site and I’ll also test yours”

• Can help in community-building• Particularly useful for programmes in

which project Web sites being developed at same time

But:• Can be difficult in processing

unstructured responses• Need to manage process


Problem Tracking

There are three stages in a corrective action system:

• The short term fix• The cure• Prevention of errors – identify potential

causes of error by analysing data to setect trends which if allowed to continue would result in errors


Documentation

3 Levels of documentation:• Policy e.g. Quality Manual• System e.g. Processes and procedures• Work practices e.g. instructions

More detailed, more specific, applies to

fewer people


Policy

A plan or course of action intended to influence and determine decisions

ProcedureA way of performing or effecting something

ProcessA series of actions, changes, or functions bringing about a result

Definitions


Documentation

• Good documentation• Policies, procedures and audits• Define procedures for all processes• Requirements• Specifications• Mission statements or statement of direction

• Define your audience • Document the testing suite and tools• Document authoring tools etc.• Document problems tracking• A Quality Management System

(QMS)??


Issues I

• Goals for testing will be related to the kind of site you have

• Testing time is limited – use automated tools

• Documentation is critical – to make sure practices are repeatable

• Implement your QA procedures slowly over time to avoid stifling productivity

• QA planning at the start of development means less time fixing things at the end


Issues II

• Motivating/persuading people to implement a QA Culture can be difficult – run workshops to show people what you are aiming for

• Make sure you have a feedback mechanism for the procedures you are implementing

• Having a great team/management is crucial


The QA Focus Perspective

QA Focus:• Funded by JISC• Currently provided by UKOLN and AHDS• Currently supports JISC's 5/99 programme• Staff:

• Brian Kelly, Project manager• Marieke Guy, QA Focus officer, UKOLN• Hamish James, QA Focus officer, AHDS• Gareth Knight, QA Focus officer, AHDS


Approach TakenTwo possible approaches to ensuring compliance with standards and best practices:Enforce

• Inspect all project’s work• Strict auditing, with penalties for no-compliance

Encourage• Training for project staff• Developmental, explaining reasons for

compliance, documenting examples of best practices and providing advice on implementation and monitoring

The latter approach is preferable, especially in a HE context

The latter approach is preferable, especially in a HE context


QA Focus Work

• Documentation and Advice• Briefing Papers• Case Studies • FAQs• Descriptions of tools and architectures for

projects to implement best practices• In-house QA

• Surveys• Providing Motivation

• Workshops• Liasing

• The QA Focus Toolkit


Documentation: Briefing Papers

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys


Documentation: Case Studies

Case Studies are being commissionedThese are:

• Written by projects themselves

• Describe the solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!

Case Studies are being commissionedThese are:

• Written by projects themselves

• Describe the solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!


Web Site Testing Tools

http://www.ukoln.ac.uk/qa-focus/qa-projects/http://www.ukoln.ac.uk/qa-focus/qa-projects/

• Full list of Web site testing tools available from database

• Surveys run with a number of the tools

• In the future there will be reviews of tools

• Full list of Web site testing tools available from database

• Surveys run with a number of the tools

• In the future there will be reviews of tools


Describing Tools and Approaches

As an example of approach which provides easier access to testing tools, see the ,tools approach deployed on UKOLN Web site.

As an example of approach which provides easier access to testing tools, see the ,tools approach deployed on UKOLN Web site.

This approach:• Applies to all resources on Web site• Covers HTML and CSS validation and

various other tests• Some recursive apps (,rvalidate)• Easily implemented with single line redirect

This approach:• Applies to all resources on Web site• Covers HTML and CSS validation and

various other tests• Some recursive apps (,rvalidate)• Easily implemented with single line redirect


In-house QA

• Policies• Procedures• Audits

To date:• Web Standards - XHTML, CSS • Linking - links to and from • Accessibility • Usage Statistic• Metadata


Documentation and Advice IWe have:

• Surveyed project Web sites and identified areas of lack of compliance with standards and best practices

• Demonstrated examples of the potential importance of compliance for repurposing resources


Documentation and Advice IIIn addition we need to provide:

• Brief focussed advice on the standards• Information on how to monitor compliance• More Case studies on solutions deployed

by projects themselves• Guidance on dealing with implementation

difficulties and what to do when strict compliance is difficult to achieve

• Reviews of testing tools


Surveying The Community

Surveys of project Web sites have been carried out in order to:

• Obtain a profile for the community• Identify examples of best practices• Identify areas in which further advice is

needed

Surveys included:• Accessibility• 404 error pages• HTML & CSS compliance


Survey Philosophy I

The surveys made use of freely-available Web-based tools:

• Methodology is open • No software needs to be installed locally

(apart from Web browser)• Findings can be reproduced• Latest results can be obtained by clicking

on link to testing service


Survey Philosophy II

The surveys typically examined project entry points and not entire Web site as:

• This page has the highest profile• The aim is to validate a methodology

which can be deployed by projects themselves, not to test every page on behalf of the projects


Survey Findings

Findings available from

http://www.ukoln.ac.uk/qa-focus/surveys/http://www.ukoln.ac.uk/qa-focus/surveys/


Providing Motivation

There is a need to explain why compliance with standards is important and to provide motivation for projects to update their tools, authoring procedures, etc.This may involve

• More workshops• Evaluating project's QA plans• Liaising with projects over the requirements of the

key QA procedures applicable to their work • Liaising with the Advisory Services over their

provision of advice and address maintenance and feedback mechanisms

• Overseeing QA work


Self Assessment Toolkit

We will be developing a self-assessment toolkit for projects to use, by individual projects or across project clustersThe toolkit will consist of:

• Examples of QA procedures• Documented examples of use of testing

tools• Self-assessment questionnaires• Advice on standards and best practices• Case studies, FAQs etc.

http://www.ukoln.ac.uk/qa-focus/toolkit/http://www.ukoln.ac.uk/qa-focus/toolkit/


QA for Digitisation

Do it once…..do it right:• Project is fundamentally dependent upon

the quality of original product

• Quality is the pre-requisite to preservation

• Quality expectations will only grow

• Delivery problems can be fixed, but capture problems normally can’t


QA For Web Sites • The Web is the main delivery mechanism for

projects and services• An increasing awareness of the importance

of accessibility, use of new devices (PDAs, WAP, e-books, …), repurposing of Web content (e.g. archiving)

• But invalid HTML is still the norm• Look to standards – XHTML, XML, CSS, WAI• If proprietary formats need to be used, flag

them and use in most open way• QA Focus is trying to explain why and how


Next StepsExtended Coverage

We will be moving on from Web and digitisation to include other areas including:

• Metadata• Multimedia• Software development• Deployment into service• …

Nex

t S

tep

s


Next Steps – Metadata

Thoughts on QA for metadata:Syntax Check syntax on embedded DCCataloguing Rules Ensure projects have appropriate

cataloguing rulesInteroperability Ensure metadata can

interoperate with third partiesFitness for purpose Ensure metadata is appropriate

for its purpose Change control Ensure architecture for managing

metadata can cope with change control (cf Exploit Case Study)


Next Steps – Software

Thoughts on QA for software:Specification Process for developing systems

spec, agreeing spec with stakeholders, etc.

Methodology Documentation of systems development methodology (cf UML case study)

Testing Documentation of testing methodology

Deployment Awareness of deployment challenges


Limitations

There are a number of limitations to the work we have carried out so far:

• Project Web sites have different purposes (information about the project; communications with project partners; project deliverables themselves; etc.)

• Projects have different levels of funding, resources, expertise, etc.

• Projects are at different stages of development (and some have finished)


QA Focus And You!

What relevance has QA Focus for you?• You can deploy QA Focus practices for your

projects, etc• You can write case studies and advisory

documents:• Seen to be working with JISC• We can promote your approaches and work• This will help instigate best practices within your

group

• You can develop QA procedures in innovative areas (e.g. OAI, thesaurii, ePrints, e-Books, etc.) which QA Focus can use


Questions

Any questions?

Documents

QA Focus – Supporting JISC's 5/99 Programme The QA Focus Perspective Marieke Guy UKOLN [email protected]