Evaluating the Joint Action HA- REACT...Evaluating the Joint Action HA-REACT Evaluation framework, methods and tools Center for Interdisciplinary Addiction Research Challenges of evaluating

Center for Interdisciplinary Addiction Research

Evaluating the Joint Action HA-REACT

Evaluation framework, methods and tools


Challenges of evaluating the JA

Timeframe: October 2015 – September 2018

Overall objectives of the project Zero new HIV cases, reduced HCV and TB among PWID in the EU by 2020

Improved prevention and treatment of blood-borne infections and TB in priority regions and priority groups in the European Union

AIM: To improve the capacity to respond to HIV and co-infection risks and provide harm reduction with specific focus on people who inject drugs (PWID) in priority regions in the EU

6 work packages with – different objectives

– activities either in community or in prison

– implemented in different EU focus countries (Latvia, Lithuania, Hungary, Poland and Czech Republic)

Involvement: 22 partners from 18 countries


Basis for the evaluation

WP2. Dissemination

WP4. Testing and linkage to care

WP5. Scaling up harm reduction

WP6. Harm reduction and continuity of care in prisons

WP7. Integrated care

WP8 Sustainability and long-term funding

Logical framework approach (LFA)





WP2. Dissemination

WP4. Testing and linkage to care

WP5. Scaling up harm reduction

WP6. Harm reduction and continuity of care in prisons

WP7. Integrated care

WP8 Sustainability and long-term funding

Logical framework approach (LFA)

Output and outcome indicators according to Grant Agreement

Assessment of effects among professionals of Focus countries

and in EU Member States


Objectives of the Evaluation

Process evaluation

– Monitor if the project activities in the WPs have been implemented as intended

– Process indicators: accuracy of content, timeline, participants

– Performance indicators: participants satisfaction, increase in skills

Outcome evaluation

– Evaluate if the outputs (materials, guidelines, training manuals) have been achieved, distributed and known to a wider audience

– Indicators: 100% distribution to target group, knowledge to a wider audience in respective Focus Country

Effect evaluation

– Evaluate if overall project objectives and specific objectives of the WP have been achieved

– Indicators: Experts assessment of improvements due to HA-REACT activities


The evaluation framework

• Mission reports, participants lists

• Participants’ satisfaction rating

• Pre- and post-testing of trainings Processes

• Basically deliverables, products developed within the project

• Website, manuals, educational material Outcome

Impact • Effects of the project activities in Focus

Countries • Surveys among different expert groups in EU


Evaluation methods

Standardised questionnaire for

participant satisfaction in 7 languages

Analysis of HA-REACT web server logs

Support of questionnaires for HIV/HCV testing, for the mobile unit in LT and the condom provision in CZ

Materials produced

Yearly online survey among different stakeholders

Focus group interviews in Focus Countries

Individual interviews with policy maker in Focus Countries

Processes

Output

Impact


Selected results

Participant satisfaction

Quality of the HA-REACT products

Visibility of the HA-REACT project

Impact of the HA-REACT project in extended

Focus Countries (HU, LV, LT, PL, CZ)


Results – participants satisfaction

A total of 38 activities with a total of 835 participants were evaluated. The

participant satisfaction questionnaire was answered by 81% (n=676)

89% 90%

92% 86%

94% 83%

90% 98%

90% 91%

WP4: Trainings on HIV/HCV testing

WP 5: Debatte in Riga und Vilnius

WP 5: Study visit in Spanien, Prag, Helsinki

WP5: Workshops zu Harm Reduction, Opioid…

WP 6: Study visit in Luxemburg und Berlin

WP 6: Internationales Seminar in Warschau

WP6: Train the trainer in Popow

WP 7: Study visit nach Portugal

WP 7: Nationale workshops in Tallinn, Vilnius, Prag

WP 7: Internatioaler Workshop in Utrecht

Average satisfaction rate


Yearly Online Survey

Response rate (%)

Target Group 2016 2017 2018

Associated partners (n=21) 95 81 95

Advisory board (n=16) 77 50 23

THL (n=1) 100 100 100

Collaborating partners (n=9-14) 33 29 23

Heads of REITOX Network (n=27) 52 33 37

EHRN (Eurasian HR Network) (n=10-45) 20 40


Quality of the HA-REACT products


Visibility of the project


Impact in Focus Countries


Challenges of the evaluation

Problems in congruence

- Timeline of activities was partly unclear and not always specified in the LFA workshops

- Some discrepancy in indicators specifies in grant agreement documents and log frame workshop

Problems of cooperation with evaluation

- Utilisation of evaluation instrument was difficult in the beginning and for certain activities such as international workshops

- Partly the responsibility for evaluating events was unclear (WP lead or local event organisers)

Problems of output evaluation

– Requires to forward the products which have been developed

Problem of impact evaluation

– Effects often occur later and not within the life of the project

– Multiple influences in countries through changing policy makes it difficult to assess the project effects

Major requirements

Constant own monitoring and communication across all partners involved in activities

Emphasising again and again the importance of evaluation


Thank you for your attention!

[email protected]

Documents

Evaluating the Joint Action HA- REACT...Evaluating the Joint Action HA-REACT Evaluation framework, methods and tools Center for Interdisciplinary Addiction Research Challenges of evaluating