Exploring Tools to Promote Teacher Efficacy with mLearning

Robert PowerCollege of the North Atlantic-Qatar

Dean Cristol & Belinda GimbertOhio State University

@xPat_Letters @drdcristol & @BelindaGimbert

The Question:“What is the single greatest barrier to the widespread

adoption of mobile learning strategies in K12 and higher education institutions?”

The Response:“Teachers’ confidence in the technology and their

ability to use mobile learning in their own practice.”

Paraphrased exchange between Robert Power (moderator) and Dr. Mohamed Ally (panelist) at the Panel Discussion on Tablet Deployment Initiatives at the 12th World Conference on Mobile and

Contextual Learning (mLearn 2013), Doha, Qatar

The Problem with Teacher Training

The current educational model is outdated because it was developed before the advent of information and communication technologies. The current model, based on classroom-based face-to-face delivery, is geared towards educating a certain segment of the population. Also, teachers are being trained for the current model of education, and will therefore continue using the model when they become teachers. Teacher training must be re-invented to prepare teachers for the technology-enhanced educational system.

(Ally & Prieto-Blazquez, 2014)

The Problem with Our Understanding of Teacher Efficacy

Lack of training in the pedagogical considerations for the integration of a specific type of technology can have a negative impact upon teachers’ perceptions of self-efficacy (Kenny, et al, 2010). However, Kenny et al. (2010) note that:

While a significant body of research exists on learners’ feelings of self-efficacy concerning computer technology, online learning, and even podcasting… this concept does not yet appear to have been examined in any detail in a mobile learning context (p. 2).

The Essential Issues Addressed by CSAM and the mTSES

• Put the focus on pedagogical decision-making.– Tool #1: The CSAM Framework

• Determine if that approach has an impact on teachers’ perceptions of self-efficacy with mobile learning.– Tool #2: The mTSES Instrument

Tool #1: The CSAM Framework

What is CSAM?

• CSAM is:– A summarization of the key pedagogical elements present in

recent case studies of the use of mobile RLOs to facilitate collaborative learning.

– A framework to guide instructional design decision-making.– Consistent with Activity Theory, the zone of proximal

development, Transactional Distance Theory, and FLOW Theory.

• CSAM is not:– A new learning theory.– A new model of instructional design.

Supporting CSAM

1. Qualitative Literature Review

2. Qualitative Meta-Analysis

3. Supporting Theories, Models & Frameworks

Qualitative Meta-AnalysisDetailed meta-analysis of 403 chapters, journal articles, and conference proceedings papers between 2009-2014

ARCS (Learner

Motivation)

Attention

Relevance

Confidence

Satisfaction

The ARCS Model (Keller, 1987, 2013)

The FRAME Model (Koole, 2009)

The TPACK Framework (Koehler & Mishra,

2006, 2008)

Activity Theory and ZPD

Relationships Between CSAM,

Learning Theory, and Supporting Models

Conceptual Framework

Research Questions

1. Does the Collaborative Situated Active Mobile (CSAM) learning design framework provide teachers with an increased sense of self-efficacy in the use of mobile reusable learning objects (RLOs) to facilitate or enhance collaborative learner interactions?

a) Do teachers perceive greater self-efficacy when using the CSAM framework?

b) Do teachers perceive their use of mobile RLOs be more effective when using the CSAM framework?

Methodology

How to Measure Impact on Perceptions of Self-Efficacy?

• Use the CSAM Learning Design Framework as the focus of pedagogical decision making and self-reflective practice in a short, online professional development course on creating mobile reusable learning objects.

• Measure the impact of that training on participants’ perceptions of self-efficacy.

• Get feedback from participants on their perceptions of self-efficacy and the use of CSAM to help make instructional design decisions.

A Snapshot of the InterventionOnline Professional Development course called Creating Mobile Reusable Learning Objects Using Collaborative Situated Active Mobile (CSAM) Learning Strategies.

• Hosted on the Canvas open LMS• Accessed via computer or mobile device.• Five modules, run over ten days (two days per

module).• Research survey instruments were embedded

as learning activities (to reduce extra time commitments for participants).

Participation:

• May 2014.• Four institutions in Canada, the US, and Qatar.• 72 participants enrolled for the course.• 42 participants completed the Informed

Consent, and participated in the research study.

Research Methodology

Measuring effects on perceptions of self-efficacy?

• Mixed-methods research:– Mix of quantitative survey data and qualitative feedback

from follow-up interviews

• Design-Based Research:– This study constitutes the first phase of a longer-term DBR

project. Subsequent phases will build upon this research to inform iterative improvements to the professional development course, and the eventual development of an OER RLO (Anderson & Shattuck, 2012; Cohen et al., 2011; Design-Based Research Collective [DBRC], 2003)

Research Instruments

• Combined Teacher’s Sense of Efficacy Scale (TSES) and Mobile Teacher’s Sense of Efficacy Scale (mTSES) survey instruments (Benton-Borghi, 2006; Tschannen-Moran & Woolfolk Hoy, 2001a, 2001b)

– Pre and post-intervention surveys

– General sense of self-efficacy vs self-efficacy with using mobile RLOs before and after the training

• Follow-up Interviews

– Qualitative feedback regarding the training, and participants’ perceptions of self-efficacy with mobile RLOs

Tool #2: The mTSES

What is the mTSES?

• Modified version of the Ohio State Teacher’s Sense of Efficacy Scale (TSES)– Included slight alterations to question

wording to contextualize the survey for mobile learning

– Followed procedures outlined by Benton-Borghi (2006)

• Original TSES and mTSES questions were combined and administered together

Data Analysis and Results

Construct Validity

TSES, I-TSES and mTSES reliabilities

Cronbach's alpha

SCALES TSES I-TSES 1st TSES 2nd TSES 1st mTSES 2nd mTSES

Efficacy for Student Engagement 0.85 0.86 0.86 0.91 0.88 0.9

Efficacy for Instructional Strategies 0.89 0.89 0.87 0.87 0.84 0.89

Efficacy for Classroom Management 0.91 0.88 0.78 0.93 0.77 0.91

Total Scale Efficacy 0.93 0.93 0.93 0.95 0.92 0.96

Overall TrendsChanges in TSES and mTSES subdomain scores between 1st and 2nd administrations

SCALES 1st Admin 2nd Admin MChange

TSES Scoring MmTSES1 MmTSES2 MChange

Efficacy in Student Engagement: 6.04 6.23 0.19

Efficacy in Instructional Strategies: 6.94 7.25 0.31

Efficacy in Classroom Management: 6.86 6.87 0.01

mTSES Scoring MmTSES1 MmTSES2 MChange

Efficacy in Student Engagement with mLearning: 5.9 6.48 0.57

Efficacy in Instructional Strategies with mLearning: 6.59 7.27 0.68

Efficacy in Classroom Management with mLearning: 6.78 6.89 0.11

Accounting for Maturation(mTSES2 – mTSES1) – (TSES2 – TSES1) = Net Change(Intervention Effect)

Net change (intervention effect)

Domain

Net Change

(mTSES2 – mTSES1) – (TSES2 – TSES1)

Student Engagement 0.38

Instructional Strategies 0.37

Classroom Management 0.11

(Kirk, 2004)

Survey and Interview Analysis (Qualitative)

Inter Rater ReliabilityInter-rater reliabilities for the interview transcript coding

Percent Agreement Scott's Pi Cohen's Kappa Krippendorff's

Alpha#

Agree # Disagree

Primary Coding

Sample 1 96.72 0.96 0.96 0.96 59 2

Sample 2 96 0.95 0.95 0.95 24 1

Simultaneous Coding

Sample 1 91.8 0.9 0.9 0.9 56 5

Sample 2 84 0.81 0.81 0.81 21 4

Notes: nSample1 = 61 units of analysis; nSample2 = 25 units of analysis

Freelon (2011), http://dfreelon.org/utils/recalfront/recal2/

Qualitative CodingFrequency counts of primary comment codes

Primary Codes Descriptions nSurvey nInterviews nTotal

0 Not Coded 7 37 44

100 Framework Strengths 21 35 56

200 Framework Weaknesses 1 12 13

300 Course Strengths 1 50 51

400 Course Weaknesses 6 12 18

500 Self-Efficacy 0 0 0

600 Interest 11 31 42

700 Other Barriers 1 22 23

800 Other Supports 6 25 31

Totals

54 224 278

Trends from the Qualitative CodingFrequency counts of primary comment codes

Primary Codes Descriptions nSurvey nInterviews nTotal Most Common Sub-Theme

100 Framework Strengths 21 35 56 Guidance (n = 26)

200 Framework Weaknesses 1 12 13 Too narrow in scope (n = 6)

300 Course Strengths 1 50 51 Interaction / Feedback (n = 19),Multiple learning resources (n = 11)

400 Course Weaknesses 6 12 18 Development tools (i.e Winksite™) (n = 8)

500 Self-Efficacy 0 0 0

600 Interest 11 31 42 May use mRLOs if appropriate opportunity arises (n = 18)

700 Other Barriers 1 22 23 Lack of institutional interest (n = 5)

800 Other Supports 6 25 31 Informal community of practitioners (n = 8)

Summary of Findings• Participants enjoyed the professional development course.

• Participants expressed a perception that the CSAM framework and the professional development course had increased their understanding of, and confidence with, designing and using mobile RLOs in their teaching practice.

• The mTSES results demonstrate increases in participants’ perceptions of self efficacy with the design and use of mobile RLOs.

• Participants expressed increased interest in integrating mobile RLOs into their teaching practice.

Limitations• Voluntary self-enrollment:

• Results may be limited to individuals with a pre-existing interest in educational technology, mobile learning, and professional development.

• Four partner institutions:• Results may not be generalizable beyond North American

teacher populations.

• Participant demographics:• Undergraduate education students and unemployed

teachers were not included in this study.

• Interview participation:• Random or stratified-random sampling was not possible.• Results may not be generalizable to the entire participant

population

Recommendations for the CSAM Course• Longer duration.

• Incorporate a practicum.

• Incorporate a module or resource section on classroom management considerations for mobile learning.

• Alternative development tools.

• Standalone mTSES tool.

• Multimedia tutorials.

• Community of practitioners.

Recommendations for Future Research• First phase of a DBR project.

• Future phases should:

• Develop and seek feedback on recommended course refinements.

• Verify the findings from this study, and the applicability of those findings to wider subsets of the teacher population.

• Redevelop survey instruments and interview questions to include questions pertaining the reasons why participants perceived changes in their perceptions of self-efficacy.

• Provide future offerings of the PD course in partnership with the four collaborating institutions.

• Seek PD partnerships with additional institutions, and with a wider range of participants.

Significance of This Research• Demonstrated the utility of the CSAM framework.

• Explored the potential for CSAM-focused PD to increase teachers’ perceptions of self-efficacy.

• Development of an instrument to gauge teachers’ perceptions of self-efficacy with respect to designing and using mobile RLOs.

• mTSES can be used to compare effects of other training interventions on perceptions of self-efficacy with mobile learning.

• Explored issues of instructional design competency and perceptions of self-efficacy with the use of educational technologies that are becoming increasingly pervasive in all sectors of education (F2F, blended, distance…).

• Contributed to the body of knowledge about how to better prepare teachers to integrate mobile learning strategies and resources into instructional design at any level of the education system.

ReferencesAlly, M., Farias, G., Gitsaki, C., Jones, V., MacLeod, C., Power, R., & Stein, A. (2013). Tablet deployment in higher education: Lessons learned and best practices. Panel discussion at the 12th World Conference on Mobile and Contextual Learning (mLearn 2013), 22-24 October, 2013, Doha, Qatar

Ally, M. & Prieto-Blázquez, J. (2014). What is the future of mobile learning in education? Mobile Learning Applications in Higher Education [Special Section]. Revista de Universidad y Sociedad del Conocimiento (RUSC), 11(1), 142-151. doi http://doi.dx.org/10.7238/rusc.v11i1.2033

Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16-25. DOI: 10.3102/0013189X11428813. Retrieved from http://edr.sagepub.com/content/41/1/16.full

Benton-Borghi, B. (2006). Teaching every student in the 21st century: Teacher efficacy and technology (Doctoral dissertation, Ohio State University). Retrieved from http://www.pucrs.br/famat/viali/tic_literatura/teses/BentonBorghi%20Beatrice%20Hope.pdf

Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed). New York: Routledge.

The Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Retrieved from http://www.designbasedresearch.org/reppubs/DBRC2003.pdf

Freelon, D. (2011). ReCal2: Reliability for 2 coders. Retrieved from http://dfreelon.org/utils/recalfront/recal2/

ReferencesKeller, J. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Design, 10(3), 2-10. Retrieved from http://download.springer.com/static/pdf/67/art%253A10.1007%252FBF02905780.pdf?auth66=1395208839_f16a62cb46b48a70cc08b9166706ffce&ext=.pdf

Keller, J. (2013, September 17). ARCS explained. Retrieved from http://www.arcsmodel.com

Kenny, R.F., Park, C.L., Van Neste-Kenny, J.M.C., & Burton, P.A. (2010). Mobile self-efficacy in Canadian nursing education programs. In M. Montebello, V. Camilleri and A. Dingli (Eds.), Proceedings of mLearn 2010, the 9th World Conference on Mobile Learning, Valletta, Malta.

Kirk, R. (2004). Maturation effect. In M. Lewis-Black, A. Bryman, & T. Liao (Eds.), The Sage encyclopedia of social science research methods. Research Methods. DOI: 10.4135/9781412950589. Retrieved from http://srmo.sagepub.com/view/the-sage-encyclopedia-of-social-science-research-methods/n534.xml

Koehler, M., & Mishra, P. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 109(6), 1017-1054. Retrieved from http://punya.educ.msu.edu/publications/journal_articles/mishra-koehler-tcr2006.pdf

Koehler, M., & Mishra, P. (2008). Introducing TPCK. In AACTE Committee on Innovation and Technology (Ed.), The handbook of technological pedagogical content knowledge (TPCK) for educators (pp. 3-29). American Association of Colleges of Teacher Education and Routledge, NY, New York.

Koole, M. L., (2009). A model for framing mobile learning. In M. Ally (Ed.), Mobile learning: Transforming the delivery of education and training, 25-47. Edmonton, AB: AU Press. Retrieved from http://www.aupress.ca/index.php/books/120155

ReferencesPower, R. (2012a). Effective learning strategies with mobile devices: Collaborative situated active mobile learning. Unpublished manuscript, Center for Distance Education, Athabasca University, Athabasca, Canada.

Power, R. (2012b). QR Cache: Connecting mLearning practice with theory. In M. Specht, M. Sharples, & J. Multisilta (Eds.), Proceedings of the 11th Annual World Conference on Mobile and Contextual Learning (mLearn 2012) held in Helsinki, Finland, 16-18 October 2012 (pp. 346-349). Retrieved from http://ceur-ws.org/Vol-955/

Power, R. (2012c, October). QR Cache: Linking mLearning theory to practice in Qatar. Qatar Foundation Annual Research Forum Proceedings, 2012(CSP31). DOI: 10.5339/qfarf.2012.CPS31. Retrieved from http://www.qscience.com/doi/abs/10.5339/qfarf.2012.CSP31

Power, R. (2013a). Collaborative situated active mobile (CSAM) learning strategies: A new perspective on effective mobile learning. Learning and Teaching in Higher Education: Gulf Perspectives, 10(2). Retrieved from http://lthe.zu.ac.ae/index.php/lthehome/article/view/137

Power, R. (2013b, April). Collaborative Situated Active Mobile (CSAM) learning strategies: A new perspective on effective mobile learning. Presentation at the Mobile Learning: Gulf Perspectives Research Symposium, Abu Dhabi, United Arab Emirates, 25 April 2013.

Power, R. (2013c, April). Create your own mobile RLOs (reusable learning objects) for situated active learning. Workshop presentation at Technology in Higher Education 2013, 16-17 April, 2013, Doha, Qatar.

Power, R. (2013d). Create your own mobile RLOs RLO. Retrieved from http://winksite.mobi/robpower/mrlos

ReferencesQuality Matters (2012). Quality matters self-review form. Retrieved from http://www.qualitymatters.org

Quality Matters (2013a). About us. Retrieved from https://www.qualitymatters.org/about

Quality Matters (2013b). Introduction to the Quality Matters program. Retrieved from https://www.qualitymatters.org/sites/default/files/Introduction%20to%20the%20Quality%20Matters%20Program%20HyperlinkedFinal2014.pdf

tpack.org (2012). The TPACK image. Retrieved from http://www.tpck.org/

Tschannen-Moran, M., & Woolfolk Hoy, A. (2001a). Teacher efficacy: Capturing and elusive construct. Teaching and Teacher Education, 17(7), 783-805.

Tschannen-Moran, M., & Woolfolk Hoy, A. (2001b). Teacher’s sense of efficacy scale. Retrieved from http://people.ehe.osu.edu/ahoy/files/2009/02/tses.pdf