Training Needs Analysis Student Systems Matthew Taylor Training & Communications Manager Student...

Training Needs AnalysisStudent Systems

Matthew TaylorTraining & Communications Manager

Student Systems

Training Needs Analysis Process

Timelines• Phase 1 (Oct):

– Draft survey questions– Staff interviews– Student meetings– Analysis of Event booking statistics– Analysis of staff requesting new roles– Analysis of e:Vision role attribution– Final Draft Survey questionnaires

• Phase 2 (Nov):– Issue surveys– Gather responses– TNA survey analysis & conclusions

• Phase 3 (Dec)– Design training approach

Overall Web analytics (April to Oct ‘13)

86, 609 Unique views113, 753 Repeat Page views

Students; 72.8%

Staff; 27.2%

Unique Views

StudentsStaff

Student Unique Views

62, 818 Unique views79, 834 Repeat Page views

Student Index Page 29.59%

Support 25.30%

FAQ 22.60%

Self Service 12.40%

Mini-Portfolio

7.20%Personal Tutors 2.90% Other 0.01%

Overall Student Page Unique Views

Student Index PageSupportFAQSelf ServiceMini-PortfolioPersonal TutorsOther

Student Unique Views

62, 818 Unique views79, 834 Repeat Page views

Scotland; 23.8%

England; 15.1%

China; 7.0%US; 4.3%Greece; 3.4%India; 3.3%

Germany; 2.2%Italy; 1.8%

Malaysia; 1.7%France; 1.7%

Other; 35.7%

Unique Views by Country

ScotlandEnglandChinaUSGreeceIndiaGermanyItalyMalaysiaFranceOther

Student Web Analytics – Key Observations1. The Student pages are heavily viewed

– Many pages need review & enhancement e.g. EUCLID Portal & Online PG Application

2. The top 5 student pages and sub-pages make up approx. 72.5% of the total unique views.– 47.9% of these pages are made up of Student/FAQ & Student/Support based pages– A third of all FAQ unique hits related to:

• Completing the PG Online application• What happens after application submission?

– The Support pages are divided between the following subjects:• PG Online Application (26.5% of all support views)• The EUCLID portal (35.5% of all support views)

– These areas require review and enhancement

Staff Unique Views

23, 528 Unique views33, 852 Repeat Page views

Admissions 13.4%

Curricula Man 47.9%

Student Admin 24.5%

Staff Index Page 6.9%

Staff Training 6.3% Other 1.0%

Overall Staff Unique Page Views

AdmissionsCurricula ManStudent AdminStaff Index PageStaff TrainingOther

Staff Unique Views

23, 528 Unique views33, 852 Repeat Page views

CCAM, Teach-ing & Learn-

ing 15.1%

CCAM, Assess-ment

Methods 11.5%

Staff User Guide Index

6.9%

Personal Tutors 14.1%

Mark Entry 12.4%

Rest 40.0%

Top Staff pages - Unique View

CCAM, Teaching & LearningCCAM, Assessment MethodsStaff User Guide IndexPersonal TutorsMark EntryRest

Staff Web Analytics – Key Observations1. Curricula Management makes up to 47.9% of all staff unique page views, 65.5%

contains CCAM pages.– These pages need review to reduce size & enhance navigation for staff use

2. There are a large number of SACS web pages with <10 unique views– These should be reviewed for archiving

3. The Mini-Portfolio, PPMD, Reporting & Reference Material web pages are not flagged for SACS Google Analytics.– These need to be flagged as soon as possible.

Web Analytics (All) – Key Observations1. End page content with anchored links to FAQ within the content are not useful for

statistical analysis– This can be remedied by having FAQs linked to unique page content, however, is

this good web design? Will it inhibit the web experience?2. Analysing the Google statistics is hindered by the variations of URL capitalisation for

the same web page. For example, student/support, will have a set of view statistics, however, Student/support will have a unique set of statistics,. – If it is an issue that can be resolved here, this should be resolved to aid analysis.

However, it might be the end user typing error?3. Great care needs to be taken on reading too much into Web analytics, especially

when dealing with less that 1 full academic year!

Student Surveys• Student Focus Group (30 attendees, 3 sessions)

– Web Comparison (UoE vs Stanford)– Web Usability– Learning Styles Analysis

• Student Questionnaire (11/12/13 – 20/12/13)– Draft sign off (Student Survey Unit/EUSA/School/College feedback)– Pilot (handful students)– School of Education– 1737 recipients– 2.5% response rate (43 students)

Student Forum Observations• Web Comparison

– UoE vs. Stanford• UoE rated higher than Stanford for navigation• UoE site considered ‘Simple’• UoE Video tutorial considered ‘Long’• Stanford Video Tutorial narration considered ‘Clearer’• UoE Video narration considered ‘too fast’

• Web Usability– Student Self Help pages

• ‘Simple & Easy’, ‘Very Clear’– Help Request Form

• ‘hard to locate’ , ‘not able to find’• Learning Styles analysis

– Heard of SACS? 4/16 Yes– Heard of EUCLID? 16/16 Yes– Required help with EUCLID? 7/16 Yes

• Online Application 3• EUCLID Portal 4

– Blended learning?• Shorter videos, shorter text guidance• “the absolutely best thing about online help is the direct chatbox”

Student Survey Observations• Timing

– National Student Survey (NSS) & Edinburgh Student Experience Survey (ESES) due to be issued in January

– PG Survey in February– Ethics committee sign off required for survey the crosses schools (2 weeks turnaround)– Personal Tutors, Courses, Induction surveys (August-Sept)

• Section 1: Your details:– 81.4 Full-time, 14% Part time, Other 4.7%– 81.4% Year 1– 86% Mature students (over 21)

PGT; 69.8%

UG; 20.9%

PGR; 9.3%

Study

PGTUGPGR

UK; 55.8%Outside EU; 34.9%

EU; 9.3%

Nationality

UKOutside EUEU

Student Survey Observations cont.• Section 2: Previous Experience

– 23.3% had heard of Student, Admissions & Curricula (SACS)– 95.3% heard of EUCLID (SITS e:Vision)– 58.1% did not recall an issue with software– Of 23.3% who had a training issue , 69.2% searched for online assistance with issue

• 44.4% found it difficult to find online guidance• 70% found the online guidance resolved the issue• 71.4% satisfied with online help

Student Survey Observations cont.• Section 3: Preference for online help

– Most effective (i.e. rated effective to very effective):1. User guide (text/screen shots)

– Effective to Very Effective: 25 (59.5%), Not effective to Satisfactory: 11 (26.2%)

2. FAQs– Effective to Very Effective: 25 (58.1%), Not very effective to Satisfactory: 13 (30.3%)

3. Forum – Effective to Very Effective: 19 (46.3%), Not very effective to Satisfactory: 16 (39%)

4. Narrated Video – Effective to Very Effective: 18 (42.8%), Not effective to Satisfactory: 15 (35.7%)

5. Online Course – Effective to Very Effective: 16 (40%), Not effective to Satisfactory: 11 (27.5%)

6. Webinar – Effective to Very Effective: 15 (35.7%), Not effective to Satisfactory: 9 (21.5%)

7. Wiki – Effective to Very Effective: 13 (31.8%), Not very effective to Satisfactory: 15 (36.6%)

Can you think of any ways to improve online help?

Redesign your pages/portals to be more friendly

& intuitive.

Redesign your pages/portals to be more friendly

& intuitive.

Have people upload videos of their own

issues, that others can comment on

Live operator to provide ‘chat’ help

Quick, easy & clear help on the

main page is very helpful

Staff Survey• Staff Interviews (Oct-Dec 13)

– 2-3 nominated staff from each school & central service– 1 hour interview, with questionnaire

• Staff Questionnaire (07/01/14 to 17/01/14)– Draft sign off (Student Survey Unit/HR/School/College feedback)– Pilot (training contacts: 14)– All staff using Student System software– Bristol Online Survey tool– 3010 recipients– 9.3% response rate (279 submitted)

Staff Survey - Observations• Staff Interviews (Oct-Dec 13)

– Key training contact sign up– Peer review training material– Identify silent partners– Encourage engagement

• Section 1: Your Details– 37.3% Managers– 80% School/College, 20% Corporate Services (Registry, SRA, etc.)– Most staff utilising Student Administration, Reporting software (non-SITS) &

Admissions

Support; 59.1%

Academic; 35.5%

Other; 5.4%

Staff Response

SupportAcademicOther

Staff Survey - Observations• Section 2: Your student system use

– 62% using e:Vision daily, 26.5% weekly– 35.8% used e:Vision 1-3 year, 50.5% for more than 3 years

Confident/Very confident;

66.7%

Could be more confident;

25.1%

Not very confident; 4.7%

Not confident at all; 3.6%

Confidence

Confident/Very confidentCould be more confidentNot very confidentNot confident at all

Col-league; 34.7%

Online user guide; 22.7%

System Support; 17.0%

Don't recall; 15.0%

Other; 10.6%

1st contact for help

ColleagueOnline user guideSystem SupportDon't recallOther

Staff Survey - Observations• Section 3: Experience & Preferences

– 24.4% attended classroom training in last year, 73.5% did not

Effective/Very Ef-fective; 53.0%Satisfactory;

38.2%

Not effective/Not very; 8.8%

Classroom training effectiveness

Effective/Very EffectiveSatisfactoryNot effective/Not very

Effective/Very effective;

34.7%

Satis-factory; 37.6%

Not effective/Not very; 27.7%

Online training effectiveness

Effective/Very effectiveSatisfactoryNot effective/Not very

Where to start;

28%

Couldn't find; 25%

Too much detail; 19%

Too techni-

cal; 14%

Out of date; 12% Other; 2%

Online training problems

Where to startCouldn't findToo much detailToo technicalOut of dateOther

Staff Survey – Observations cont.• Section 3: Experience & Preferences

– In office training:• 34.1% Yes• 63.1% No

– Training provision preference:• 49.5% local• 9.3% central• 41.2% both

– Induction training?• 67% No, 28% Yes, 5% Don’t know

User guide; 23.0%

FAQs; 18.0%

Narrated Video; 14.4%Webinar; 13.7%

Online Course; 12.0%

Wiki; 10.2%

Forum; 8.7%

Online material preferences

User guideFAQsNarrated VideoWebinarOnline CourseWikiForum

Taught; 28%

Remote; 7%

Mixed; 42%

1:1; 23%

Learning preferences

TaughtRemoteMixed1:1

Staff feedback?

What is the SACS online user guide?

Never heard of it.

What is the SACS online user guide?

Never heard of it.

Everyone use systems in different ways - it is difficult to streamline

the training

Training too basic with no follow up

courses

If I need training, then the software is poorly designed (Academic)

Quick wins• New student ‘Self Help’ web pages• Embedding web help into student e:Vision software (existing software)• Creating standards for Developers to embed web help in e:Vision (ongoing)• Staff peer testing of online training materials• Brief, FAQ based online tutorials developed for staff & students

TNA Space Invaders

Training Needs Analysis Process

New Timelines• Phase 1 (March):

– Analysis of Event booking statistics– Analysis of staff requesting new roles– Analysis of e:Vision role attribution

• Phase 2 (March):– TNA survey analysis & conclusions

• Phase 3 (April)– Design training approach

What’s next?• Statistical Analysis• Publish findings• Consultation with business in proposed training approach• Training Design• Resources estimate• Develop materials• *Gamification of eLearning• *Learning Management System creation (Learning & Development Forum)• * Follow up training needs analysis (6 months?)

– Student/Staff Surveys– Web analytics, etc.

Questions?