1

EEVS project updateM. Cubric & A. Jefferies

A Strand (iTEAM) cluster meeting 20-01-2012

EEVS project factsheetTheme: Assessment and Feedback

Strand: B (Evaluation)

Duration: Sept11-Sept12

Principal investigators: Amanda Jefferies (AJ) & Marija Cubric (MC)

Management: weekly progress meetings

Aims and Objectives: identify a  set of critical success factors for introducing and maintaining the use of EVS in support of an institutional assessment and feedback strategy, with specific focus on scale of use and subject specifics.

Approach: Grounded theory research combined with multi-layered case-study approach using data from students and staff from 8 different schools relative to EVS deployment in Sem B 2010/2011 and Sem A 2011/12, supplemented with data from other stakeholders (e.g. leaders of the assessment and feedback project).

Links to other projects: feeding (results) into iTEAM evaluation; feeding (data) from iTEAM project participants

Website: http://jiscdesignstudio.pbworks.com/w/page/48734953/EEVS%20Project

2

Progress so far…

3

Data collection Literature review

Planned

Actual

Data Analysis

Data collection Staff reports (2010-11)

– 7 schools: Business, Languages, Education, Computer Science, PAM, Psychology, Life Sciences

– 3500+ students

Staff interviews (2010-11)– Business, Languages, Education, Law

Students’ ‘blogs’ (2011-12)– 16 students reflected over a period of 3-4 weeks in Sem A, resulting

in 24 blog entries, 35 Word documents, 1 podcast and 8 vlogs. – 14 more students due to complete their reflections by end of

January– Good spread across gender, levels and schools.

4

Literature reviewGoogle scholar search produced total of 759 results for search terms:

PRS OR "response system" OR “electronic voting” OR e-voting OR EVS

AND

(learning OR assessment OR feedback)

---

Very little on institutional deployment and differences across disciplines, mainly pedagogical implications (e.g. improved in-class interactions, engagement in large groups, curriculum design and delivery)

Students’ questionnaire design based on:– Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan,

B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp.189–194).

– Venkatesh et al (2003) User Acceptance of Information Technology Towards a Unified View, MISQ 2003

– Draper’s material (ref tbd)5

Data analysis Preliminary data analysis of staff reports and interviews

Benefits outweigh the issuesLittle or no impact on students’ performance, attendance More caution regarding use for summative assessment

Preliminary data analysis of students’ reflectionsAnxiety about using EVS for summative assessmentTechnical issues sometimes outweigh benefitsUnderstanding of benefits for formative testing and in-class

engagementNo issues with handing-out/collection procedures

No indications yet of non-obvious subject-specific differences in use

e.g. handset type (text function important for some subjects)6

NextData collection

Remaining students’ reflections (end of January)

Students’ questionnaire deployment (4000+ students)

Students’ focus groups

Further staff interviews and focus groups

…

Literature review

More in-depth analysis of literature

…

Data analysis

Further memoing of existing data

Identification of categories

Axial coding

…

JISC project interim progress meeting (March 2012) 7