Dr Andrea Baldwin

Creative Industries Faculty

QUT

Evaluating

a Train the Trainer program

in Papua New Guinea

• Sexual health – focus on STIs and HIV• Behaviour change communication

using applied theatre• Experiential learning, not one-way

dissemination of health information• Train the trainer model• Ultimate beneficiaries are young people

(15-25)• Developing hybrid forms of applied theatre

for education/ health promotion

WHAT’S THE PROJECT?

• Professor Brad HasemanCreative Industries, QUT

• Associate Professor Anne Hickling HudsonEducation, QUT

• Dr Andrea BaldwinCreative Industries, QUT

• Ms Hayley Linthwaite

Creative Industries, QUT

WHO ARE THE TEAM?

• Ms Jane Awi Creative Industries, QUT

• Mr Martin Tonny Research and Administrative Assistant

• Ms Jackie Kauli Creative Industries, QUT

• Research Advisory Groups and Participants in Field Sites

• Australian Research Council – Linkage Grant

• Queensland University of Technology• Griffith University • National AIDS Council Secretariat PNG• University of Goroka• University of Papua New Guinea

WHO ARE THE PARTNERS?

Evaluation Framework

Quantitative Qualitative Performative

Context

Inputs

Processes

Outcomes

Haseman, 2006

Stufflebeam, 2003

THE MAP IS NOT THE TERRITORY

Spiral In:

- what’s the bottom line?

- did implementation go to plan?

- did the project create the desired change?

- what are the take-home learnings?

- yes/no, definitive answers

- quantitative

Spiral Out:

- who learnt what from this project?

- what impacts did the project have – positive and negative, expected and unexpected?

- what does this mean for the future?

- descriptive, expansive answers

- qualitative, performative

RESEARCH/EVALUATION METHODOLOGIES MUST:

• Be culturally appropriate, meaningful and relevant to the participants and other stakeholders

• Be sensitive to issues of advantage/ disadvantage for individuals and groups of stakeholders

• Complement and advance the aims and approach of the overall project

A CONCEPTUAL METAPHOR

PROGRAM

Is the seed viable?

PROJECT

What have we learnt about propagation?

EVALUATION RESEARCH

• Involves determining the worth, merit, or quality of an evaluation object, such as an educational program

• Formative evaluation – helps in the design, implementation and improvement of the program

• Summative evaluation – helps make decisions about whether a program should be supported into the future

Johnson & Christensen, 2008

MAGIC PILLS?

• Drug trial model

• Laboratory

• Quantitative, use of inferential statistics

• Clear distinctions between the agent (drug) and conditions of administration (dosage, interactions, etc.)

PARTICIPATORY ACTION RESEARCH

• Much more organic – can’t make such clear distinctions between the agent (program, action, education materials) and the context (community or organisation)

• Requires more holistic approach to collecting and analysing data, interpreting findings, making recommendations for future

IMPORTANT TO:

• Capture negative/undesirable impacts of the project as well as positive/desirable ones

• Capture and assess unexpected impacts of the project as well as intended/expected ones

NEW PHASE, NEW APPROACH

The Life Drama Program

• Development highly participatory – Tari, Madang, Karkar Island

• Quantitative, qualitative and performative data collected

• Triangulating and synthesising vast quantity of data from participants and other stakeholders

The Train the Trainer Program

• Life Drama program content established

• Train the Trainer program content and structure established – Handbook and Bilum

• Collect qualitative and performative data from two sites

WHAT MORE HAVE WE LEARNT ABOUT THE SEED?

1. Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV?

• Interview/focus group data

• Performative data

2. Do the educators believe their students will acquire/have acquired new knowledge and attitudes through the training?

• Post and follow-up interview data

WHAT MORE HAVE WE LEARNT ABOUT PROPAGATION?

1. What factors enhance or impede

a) the delivery and effectiveness of training Life Drama trainers?

b) the formation and functioning of a network of trainers?

2. What are the similarities and differences between training community educators and teacher educators, which are likely to impact differently on the effectiveness of these two groups as trainers?

3. What are the similarities and differences that are likely to impact on their ability to support one another as a functioning network?

GENERALISABILITY?

• These questions being investigated in PNG – two specific sites/groups – capture realistic picture of this specific scenario

• Interpretation of data will include the attempt to draw out general principles that might apply in other settings

“BASIC”

EVALUATION

• PROGRESS REPORTS – is the project being implemented as designed? Is the research team doing what it said it would do? Are budget milestones and timeframes being met? If not, why not? (emphasis on Context, Inputs and Processes)

• FINAL REPORT– did the project achieve intended outcomes hoped for? What factors enhanced or hindered the achievement of these outcomes? (Emphasis on Context and Products)

OBJECTIVE 1: TRAINEE GROUP

1. To train between 20 and 40 Life Drama trainers, with an equal balance of male and female trainers, through two training hubs:

• University of Papua New Guinea, Port Moresby, National Capital District

• University of Goroka, Goroka, Eastern Highlands

EVALUATION

Quantitative data – how many males and how many females completed training?

Performative data – how many trainees passed requirements for certification?

REQUIREMENTS FOR CERTIFICATION

By completion of the training, trainees will demonstrate -

•A practical knowledge of contextual, communicative practices and their place in society;•A deeper awareness of community issues, in particular sexually transmitted illnesses and HIV;•Specific skills in using drama techniques confidently to help community groups explore community issues, in particular sexually transmitted illnesses and HIV;•Leadership skills which help community groups find their own solutions to those problems and barriers which are preventing widespread understanding of, and positive response to, the challenges of sexually transmitted illnesses and HIV;•Critical, analytical and creative approaches about connections between purposes, audiences and diverse ways of communication;•Individual and group interaction leading to brain-storming, understanding and action;•Attitudes appropriate to inclusive communication practices for cultural, linguistic, gender differences and collaborative contexts.

OBJECTIVE 2: TRAINER NETWORK

2. To support the trainee group to function as a “trainer network” – sharing practice, exchanging ideas, assisting one another’s professional development, and expanding knowledge of the Life Drama program beyond the original group of trainees

EVALUATION

Qualitative data – post-training and follow-up interview and focus group data on factors that may enhance or hinder/have enhanced or hindered the functioning of the trainer network

OBJECTIVE 3: TRAINER EFFECTIVENESS

3. To assist participants to integrate Life Drama techniques with their existing educational skills and strategies, to enhance their effectiveness as trainers of HIV-related material

EVALUATION

Qualitative data – post-training and follow-up interview and focus group data on how trainees intend to use/ are using their Life Drama skills for HIV education purposes

Performative data – review in-training video footage of trainers’ practice, and follow-up practice

OBJECTIVE 4: RESEARCH CAPACITY

To build research capacity in Papua New Guinea:

•Participatory Action Research project involving senior staff in arts health areas of University of Papua New Guinea and University of Goroka•Develop research skills of PNG national members of the research team•Encourage research participants to gather data as “co-researchers”

EVALUATIONQuantitative data – how many PNG national staff from the two universities are actively involved in the project?

Qualitative data – what is the level of engagement/mutual benefit for the PNG national staff and the QUT research team?

Qualitative data – Quality of reflective journalling and reporting by participants as co-researchers

GROUP 1

UNIVERSITY OF

PAPUA NEW GUINEA

PORT MORESBY

COMMUNITY EDUCATORSPort Moresby organisations:

University of Papua New Guinea•Lecturing staff•HIV peer educators

Anglicare StopAIDS

National Capital District Commission

Centre for Democratic Institutions PNG

Family Health International PNG

Motukoita Community Youth Development

Department of Education

Hela Province Organisations:

Porgera Joint Venture

PNG Police Service

Hela Rural Women’s Foundation

OilSearch

GROUP 2

UNIVERSITY OF GOROKA

GOROKA, EASTERN HIGHLANDS

TEACHER EDUCATORS

University of Goroka:

•U100 (mandatory HIV course)•Language and Literature Department•Social Sciences Department•Expressive Arts Department•Science Faculty•Education Faculty•Early Childhood Department•Student Services

Madang Teachers College

Oxfam PNG

Independent Community Theatre Practitioners

OBJECTIVE 1: TRAINEE GROUP

Port Moresby

•6 females

•14 males

•20 TOTAL

Goroka

•15 females

•11 males

•26 TOTAL

TOTAL = 46 TRAINEES, 25 MALE AND 21 FEMALE

OBJECTIVE 2: TRAINER NETWORK

• Only post-training focus group data available so far (follow-up focus group data to be collected in November 2011)

• Only in-training video performative data available so far (follow-up video of trainees’ practice to be collected in November 2011)

Factors Enhancing

• Membership of same organisation (eg. university staff)

• Ability to access communication technology (internet, email, phone)

• Organisational support (esp. U of G)

• Ongoing relationship with QUT

• Ongoing resourcing

Factors Impeding

• Membership of different organisations

• Working with different client groups – different needs

• Communication barriers (few trainees have internet access; internet access often unreliable in PNG; phone access ditto; phone communication expensive)

• Lack of resourcing

OBJECTIVE 3: TRAINER EFFECTIVENESS

Qualitative Data:

•Only post-training focus group data available to date. Follow-up focus group/ interview data will be collected in November 2011

Performative Data:

•Only in-training video available for review to date. Follow-up video will be collected in November 2011

•Video to be analysed by members of team expert in:

– Drama-in-Education/ Theatre for Development (Creative Industries Faculty)

– Pedagogy (Education Faculty)

OBJECTIVE 4: RESEARCH CAPACITY

• # PNG senior university staff actively engaged in project, not as trainees: 7 (5 in Goroka, 2 in Moresby)

• “Active engagement” evidenced by: participating in training sessions, observing training sessions, providing additional supports and assistance to the program, contributing to focus groups

• Level of mutual benefit for PNG national research staff:

- 2 PNG national research staff undertaking PhDs with QUT as a result of Life Drama project (assisting with data collection and analysis for Train the Trainer program)

- 1 PNG national research staff member now undertaking Masters with another university, partly as a result of involvement in Life Drama

- QUT and University of Goroka now exploring future partnership opportunities

• Quality of reflective journalling and report provision by participants – yet to be assessed (November 2011)

TWO GROUPS: COMPARE AND

CONTRAST

CONTEXT

Port Moresby

• Primarily community educators

• Significant percentage from outside Moresby or away from the university (eg. settlement areas)

• Large range of organisations represented, sometimes by 1 or 2 people

Goroka

• Primarily teacher educators

• Primarily local (living and working in Goroka)

• Most trainees affiliated with University of Goroka at some level

INPUTSPort Moresby

• Less organisational commitment – less resourcing – more difficult learning environment

• Indoor venue in poor repair

• Several trainees self-funding attendance – resentments over resourcing

Goroka

• Two indoor venues in good repair but not always available; using outdoor spaces without disturbing exams

• More organisational commitment – more resourcing – more condusive learning environment

• Issues over resourcing for non-local attendees

PROCESSES

Port Moresby

• Handbook not available until last day

• Trainees generally less comfortable with English

Goroka

• Handbook available throughout training, used for daily review of lessons

• Trainees all comfortable with English

PRODUCTS

Port Moresby

• Harder to follow up and evaluate

• We expect less effectiveness of trainer network because fewer participants have good access to communication mechanisms and infrastructure

Goroka

• Easier to follow up and evaluate

• We expect more effectiveness of trainer network because most participants have access to communication mechanisms and infrastructure, and/or are geographically co-located

WHAT MORE HAVE WE LEARNT ABOUT THE SEED?

1. Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV?

• Interview/focus group data – yes, eg. difference between HIV and AIDS, how to live healthy life with HIV

• Performative data – yes, eg. condom demonstrations

QUALITATIVE DATA

“I thought AIDS was an instant disease. But I realise now there’s a harmony in the body that tries to fight against this disease. If I’m thinking that, what about the students? We need to teach them all the things you can do to stay healthy”

Female participant, early 50s, lecturer in U100 HIV program at Goroka University. Post-training focus group.

PERFORMATIVE DATA

2. Do the educators believe their students will acquire/have acquired new knowledge and attitudes through the training?•Post and follow-up interview data – yes, particularly Dancing Diseases

WHAT MORE HAVE WE LEARNT ABOUT PROPAGATION?

1. What factors enhance or impede

a) the delivery and effectiveness of training Life Drama trainers?

Shared language, resources, communication, venues, equipment.

Regular debriefing/consolidation periods.

Handbooks.

b) the formation and functioning of a network of trainers?

Shared language, organisational commitment to trainees and their professional development/ongoing peer support, access to communication, resourcing.

2. What are the similarities and differences between training community educators and teacher educators, which are likely to impact differently on the effectiveness of these two groups as trainers?

•Similarities include degree of commitment to education, identification with the educator role

•Differences include foundational skills (in teaching, in drama), organisational support, resourcing, communication

3. What are the similarities and differences that are likely to impact on their ability to support one another as a functioning network?

•Similarities include degree of commitment to education, identification with the educator role.

•Differences include organisational support, resourcing, communication

METHODS

Quantitative

-Individual interviews with rating scale responses

-Observer ratings of skills displayed in workshops (live and on video)

-Recordings of numbers eg. number of participants, sessions attended

-Trainers rating their own participants’ understanding of HIV issues

Qualitative

-Focus groups

-Interviews

-Stakeholder meetings

-Case studies

-Participant journals

-Lead trainer reflective journals and field notes

Performative

-Participants observed and videoed both as participants in training workshops, and conducting training of others

-Participants performing “stories” illustrating “best” and “worst” moments delivering training (these stories videoed)

UNEXPECTED OUTCOMES?

• Teacher educators excited by new approach to pedagogy – keen to apply it in areas apart from HIV (including early childhood teaching)

• One director keen to re-engineer her “office job” and re-engage with teaching

• Trainers identify range of applications for Life Drama in other life roles eg. Sunday School teacher, police officer, pastor

CHALLENGES

PARTICIPANTS

•Communication

•Literacy

•Numeracy

•Languages

•Cultural expectations

•Culturally-appropriate data collection “tools” and methods

PARTNER ORGANISATIONS

•Power relations between organisations

•Organisational and national politics

•Perceived benefits for individuals and groups

•Infrastructure including venues, equipment, communications

•Travel

STUDENTS OF TRAINEES

•Religious and cultural resistance to discussing sex, body parts

•Resistance to condom use

•Gender issues

•Sensitive topics such as sexual activity with children

RESEARCH OUTPUTS

• Life Drama Train the Trainer booklets: Life Drama Handbook and Life Drama Bilum

• Interim Report and Final Report for participating organisations and participants

• Journal articles• Conference presentations• Further applications of Life Drama

• Life Drama website – evaluate separately as a tool for communicating performative outcomes

PUBLICATIONS

Baldwin, A. (2010). Dancing Diseases: An Applied Theatre Response to the Challenge of Conveying EmotionallyContradictory Messages in HIV Education. Applied Theatre Researcher, 11.

Baldwin, A. (2010). Life Drama Papua New Guinea: Contextualising Practice. Applied Theatre Researcher, 11.

Jennings, M. and Baldwin, A. (2010). “Filling out the Forms was a Nightmare”: Project evaluation and the reflective practitioner in community theatre in contemporary Northern Ireland. Music and Arts in Action, 2(2), pp. 72-89.

QUESTIONS?

www.lifedrama.net