The Known Unknowns: Discovering what our Bb users are doing

through monitoring with EesyAnalytics

Cardiff University

• 25,000 FTE• Russell Group & research oriented• 27 schools• 3 colleges

– Previously substantial autonomy in IT use and L&T at School level

• Our TEE platform is branded ‘Learning Central’

Learning Central…

• Blackboard 9.1 SP9– Locally hosted– Course delivery system– Content management system– Community engagement system– Blackboard Mobile– Turnitin & Grademark– Learning Objects– Questionmark– Wimba Voice tools– A few custom building blocks

Some (approximate) numbers

• 25,000 FTE equivalent• 6000 staff• 27 schools• 6,000 modules a year• 20,000 visits a day

But what are they all doing?

• Which tools are being used?• And which are not…• Who is using them?• When are they using them?

Assessing our effectiveness

• We implement tools for learning• We deliver staff development• We write support material• We visit schools & talk to staff

• How can we tell if this activity has changed anything?

Our challenges

• Limited supply of resources• Plenty of demand

– For increased administrative efficiency– For improved educational quality / NSS

• Diversity of needs and practice• Always something new

– New features in Service Packs– New tools from suppliers– Emerging topics & changing practice

We need more data!

• Prioritising activities– Avoid spending time on low impact areas

• Identifying areas of low/high uptake– And discovering why

• Assess whether our activity affects things– Evidence for senior managers

• Building business cases– For additional tools or resources– Evidence for deprecating obsolete tools

Google Analytics

• Data on pages visited and when• Page load time• Failure to load• Dwell time• Access to individual tools

• An excellent overview, but usage is amalgamated

Manual reporting

• Auditing every module & assessing against benchmarks

• Extremely flexible in what it reported• Can not be done in real time; a snapshot• Expensive in staff time

Not a practical reporting solution, except as infrequent benchmarking

Blackboard reports

• Excellent tool for assessing the learning activity of individual users

• Data on student’s use of tests, discussion etc• Valuable tool for academics to check progress

of students

Not useful for our purposes as system admins or learning technologists

BIRT

• Uses:– Bb databases– SQL– Eclipse– Building blocks

• Create custom reports run through Building Blocks

• Recently made available

Fairly technical process and runs against live database

Eesysoft Analytics

• Detects where user is in application– Page loads– Mouse movement over HTML elements

• Records these, and associated user role• Reports on

– What different roles have been doing– Trends in use of different tools– Comparisons between tools

How it works

• Admin takes HTML snapshot of page– Define HTML elements

• JavaScript embedded in Bb hotspot• Detects what mouse pointer is over or which

page is loaded• Takes data from user profile• Sent to Eesysoft server & recorded

Contextual help & Eesysoft

• Our entry point to Eesysoft• Provides support to users on the basis of what

they doing at the time– Proactive pop ups for hints/issues– Via Support tab for support materials

• Makes it easier for users to get relevant support in a timely manner

• Multi-language support– Based on language pack in use

Context sensitive help

Pop up help

• Alert users to new functions or problems• Only seen by those who it is relevant to• Only seen at the moment it is relevant to them

EesyAnalytics

• Only in use a short time• Currently for SysAdmin or Learning Techs• Aiming to produce tailored reports for various

people in university• I’ve excluded some data (to improve clarity)• Graphs show %age user group

– Those who’ve accessed, not numbers

What are students using in Schools

• Profile of tool usage in different Schools

What are staff using in Schools…

Undergraduates & postgraduates

Uptake of new tools• Money well spent!

Trendlines

• Pattern of tools over a year• Assessment deadline profile• Change in behaviour after training• Change in use after engagement or awareness

raising

Pattern of student access to Turnitin in two Schools

Language pack & other metrics

• Eesysoft can collect data from user profile• Useful for Welsh language support material• Demographic comparisons… potentially

– Do language pack users use different tools?– Does age make a difference?– Does prior education change what you do?

• Use evidence to build better courses

Roles Roll Call

• Has identified a profusion of roles– 346 in total!– We have a nested structure– And a variety of ad hoc roles

• New Roles– Year of study (staff only)– Year started at University (staff & students)– Will allow us to see how use of changes

What do these allow us to do?

• Focus our efforts better– Training– Support– Engagement, promotion

• Identify areas of good/bad practice– And share lessons– Help academics understand & visualise their students

use of the VLE

Assessing our effectiveness

• Promotions events• Awareness raising• Staff engagement• Using roles to evidence the subsequent

actions after training

Evidence for business cases

• ‘Hard’ evidence to go alongside focus groups and reported needs

• Comparing use of two tools– Use of native and externally supplied

• Highlighting use of a tool– In order to retain it– In order to demonstrate the need for support– To show when something is no longer used

Conclusions

• Beginning of our journey• Benchmarking & strategising• Many data sources are available to us• We can build up a picture of activity in VLE• We can use that to improve our work and

demonstrate our value

• More work to be done and more possibilities to explore

Pete Obee: obeepc@cf.ac.uk