When Student Confidence Clicks - Using Student Response Systems

1

When Student Confidence Clicks

Academic Self-Efficacy

and Learning in HE

Fabio R. Aricò

Using SRS (clickers)

methods, analysis,

and results

OUTLINE

1. Student Response Systems (clickers)

2. Use of clickers in “Introductory Economics”learning environment & learning technology

3. Collecting data and analysing results

• demographic variables, data coding, diagrams• Result 1: assessing self-assessment the role of clickers• Result 2: peer-instruction and learning-effects

2

ETHICAL REMARK

You will be presented with data collected during teaching sessions.

Students involved have given informed consent for me to analyse their responses and present the results of this analysis.

I can assist with ethical queries as well, please ask me.

3

1. Using Student Response Systems (clickers)

4

Do you know clickers? Do you know how to use them?

A. Yes, I use them in my teaching.

B. Yes, but I do not currently use them in my teaching.

C. I have an idea of what clickers do, but a very basic one.

D. I have no clue about what clickers do and how they can be used.

5

Yes, I u

se th

em in

my te

a...

Yes, but I

do n

ot curre

ntl...

I hav

e an idea o

f what c

li...

I hav

e no cl

ue about what..

.

0% 0%0%0%

6

SRS interaction with feedback to the students

CLICKER MANAGEMENT

School of Economics Protocol:

1. Each student receives a clicker during orientation week.Information sheet. Clicker collection is not compulsory.

2. Clicker ID associated to Student IDEthical procedure followed – Informed consent to use data.

3. Students use clickers across their 3 years of study and thenreturn their clicker at the end of their studies to the School.

4. If a student loses/breaks his/her clicker, s/he has to pay a replacement cost (but can obtain a new one).

7

CLICKER MANAGEMENT

Statistics for 2013-14 - Year 1

180 students enrolled

13 students did not collect their clickers(majority of overseas students)

1 clicker lost paid for and replaced

1 clicker returned (after chasing)

1 clicker missing

8

CORE IDEAS ABOUT CLICKERS

• Clickers increase engagement and student satisfaction Yes, it works and it is tested.

• The ‘clicker’ novelty wears out quickly Possibly, but (majority) students keep on using them and enjoy them.

• Cannot be over-ambitious in what you teach If you switch to clickers you will need to cut some of the material.

• Use a counter to get responses in faster Recent research suggests 80% response rate is the cut-point.

• It is not all about technology, it is still about good teaching See Nielsen et. al. (2013) , “Research in Learning Technology”, ALT

9

2. “Introductory Economics” The learning environment

10

TEACHING PROTOCOL – the module

Introductory Macroeconomics Year 1 – compulsory year-long module - 180 students

Lectures traditional frontal-teaching (10 per sem.)

Seminars small group, pre-assigned problem sets (4 per sem.)

Workshops large group, problem-solving sessions (4 per sem.)

Support Sessions non-compulsory drop-in sessions (4 per sem.)

11

TEACHING PROTOCOL – lectures

Lectures interaction via clicker technology

Seminars revision questions + understanding questions

Workshops closing questions:

was the lecture enjoyable interesting?

was the material difficult?

Support Sessions online report of clicking session + feedback

12

13

Lecture difficulty indicator -66% (+8%).Please look out for additional resources coming online very shortly. Video tutorials about the IS-LM will be available shortly.

I would like you to reflect on the feedback asked on the IS-LM model and try to identify what are your OWN difficulties. If many of you are confident about understanding and mastering the material, we need to make this belief becoming a reality. For those of you who are not confident. Why is this the case? Come and discuss this with me.

TEACHING PROTOCOL – lectures

14

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Lecture Week

Longitudinal Lecture Assessment

easy

enjoy

attended

revision

TEACHING PROTOCOL – seminars

Seminars preliminary Seminar Quizzes (paper-based)

Seminars 3-4 revision/understanding questions

Workshops 2 confidence/self-assessment questions

Sessions open-answer comments

Support Sessions online report of Seminar Quiz

- solutions and overall performance- individual performance available- response to open-answer comments

15

TEACHING PROTOCOL – seminars

16

TEACHING PROTOCOL – workshop

Workshops peer-instructed flipped classroom approach

Seminars

17

If the bargaining power of trade unions increases…

A. Unemployment will increase.

B. Unemployment will decrease.

C. Unemployment will stay the same.

18

How confident do you feel about your answer?

A. Very confident.

B. Confident.

C. Somewhat Confident.

D. Not Confident.

19

A. B. C. D.

27%

33%

13%

27%

So did you get the right answer?

Compare your answer with the delegates sitting next to you for a minute and then we will try to re-poll the question.

20


A. Unemployment will increase.

B. Unemployment will decrease.

C. Unemployment will stay the same.

21

A. B. C.

30% 30%

40%


TEACHING PROTOCOL – workshop

Workshops standard algorithm:

1. Preliminary preparation question2. Quiz questions + Confidence questions (no solution)2. Peer-instruction learning3. Quiz questions + solutions4. Problem-set questions (no clicking)4. Feedback questions:

- what was the cause of mistakes/problems?- did you enjoy using clickers?- were clickers useful to your learning?

Support Sessions online report of clicking session + feedback

23

TEACHING PROTOCOL – extra-curricular

Extra-Curricular Activities to promote engagement and Self-Efficacy

Seminars Module Facebook Page + Blackboard pages

- ‘challenges’ to encourage further study- interaction and participation

Seminars Voluntary in-lecture presentations (5 minutes)

- to exploit demonstration effects

Support Sessions Campus Vouchers (for engagement, not attainment)

24

3. Collecting data andanalysing results

25

TEACHING PROTOCOL – the methodology

Targets attainment, engagement, academic self-efficacy role of the SRS (clicker) technology

Learning analytics rich dataset = clicker and paper-based responsesSeminars matched demographics from student records

uncover correlation patterns

Qualitative data focus group and individual interviewsSessions feedback from students

Support Sessions provide the narrative to interpret the analytics

26

EVIDENCE-BASED PRACTICE

27

ATTAINMENT

ENGAGEMENT CONFIDENCE

Gender

Satisfaction

Previous attainmentDomicilePOLAR

Perceived difficulties

DATASETS

Student Q1 Q2 Q3 …

1 0 1 1

2 1 0 0

3 1 1 …

…

performance per questionconfidence by question

pe

rform

ance

pe

r stud

en

tco

nfid

en

ce by stu

de

nt

28

longitudinal study- across all lectures- across all seminars- across all workshops

Intermediate and final attainment outcomes- course test- final exam


Result 1: Peer-instruction and learning

29

RESULT 1: Peer-instruction and learning

30

We have analysed the teaching protocol adopted in workshops

1. Ask a question (do not show distribution of answer)and poll it through clickers.

2. Ask students to self-assess the correctness of their answer:how confident are they about having given the right answer?

3. Ask the same question again, show the distribution of answers.Reveal the correct answer and teach why this is the case.

The difference between % correct answers in the two roundsrepresents the learning gain generated by peer-instruction.

RESULT 1: Some ideas to represent data

31

Week 5 (2012-13)

% correct responses■ 1st round ■ 2nd round

You can represent results by question…


% correct responses 1st round

% co

rrect respo

nse

s 2n

dro

un

d

32

Week 5 (2012-13)

You can represent results by student…


33

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4 5 6 7

Workshop Number

prepared

Q.Score

R.Score

Workshop Longitudinal data 2013-14


34

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

1 2 3 4 5 6 7

Workshop Number

Q.Score

R.Score

SelfAss

CorrConf



35


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7

MYFAULT

ENJOY

LEARN

RESULT 1: Peer-instruction and learning

36

We found that the learning gains from peer-instruction:

• are higher in the group of low-performing students so peer-instruction works in getting everybody at the same level.

• are not associated to self-efficacy and self-assessment skills so everybody has a chance to learn despite being poor at self-assessing.

• are positively correlated to exam performance so it seems that peer-instruction can display some long-run effects. This result has to be investigated with further attention.


Result 2: Assessing Self-Assessment

37

RESULT 2: Assessing Self-Assessment

38

What is the relationship between attainment and confidence?Are students able to self-assess their performance?

Compare two self-assessment set-ups:

• Seminars paper-based self-assessmentseminar quizzes + confidence questions

• Workshops clicker-based self-assessmentpeer-instructed re-iterated algorithm

RESULT 2: Seminars

What is the relationship between attainment and confidence?

39

RESULT 2: Seminars


40

RESULT 2: Workshops


41

RESULT 2: Workshops


42

RESULT 2: Summary

• In Seminar Quizzes:

high-attainment students display higher confidence

low-attainment students not able to self-assess their performance.

• In Workshop sessions:

high-attainment students display higher confidence

low-attainment display lower confidence.

• How to interpret this asymmetry?

43

RESULT 2: Summary

• In Seminar Quizzes:

3 or 4 questions, paper-based quiz, 5-6 minutes, not anonymous

1 confidence assessment for overall performance.

• In Workshop sessions:

5-10 questions, clicker response, slower pace, quasi-anonymous

1 confidence assessment for each question asked.

44

RESULT 2: Conclusion

• Low-attainment students encounter more difficulties in

self-assessing their performance in an environment where:

they self-assess their ‘overall’ performance on a composite task

they are exposed to questions for a shorter period of time

they are exposed to fewer questions, not anonymously.

• Focus group interviews (differentiated by attainment groups)

confirm that low-attainment students display lower

self-assessment skills important finding for intervention!

45

Education

When Student Confidence Clicks - Using Student Response Systems