Upload
gustavo-pinto
View
50
Download
0
Embed Size (px)
Citation preview
Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to1
Before we start..
2
Evidence-Based Medicine
EBM is the conscientious, explicit, judicious and reasonable use of modern, best evidence in making decisions about
the care of individual patients.
It integrates the clinical experience and patient values with the best available
research
It aims to increase the use of high quality clinical research in clinical
decision makingActa Informatica Medica’2008 3
Evidence-Based Medicine
One of the greatest achievements of EBM has been the development of systematic
reviews and meta-analyses, methods by which researchers identify multiple studies on a
topic, separate the best ones and then critically analyze them to come up with a summary of the
best available evidence.
Acta Informatica Medica’2008 4
Evidence-Based Practice
5
EB Psychology EB Psychiatry
EBM
EB Education
extends
Evidence-Based Practice
6
EB Psychology EB Psychiatry
EBM
EB Education
extends
EB Software Eng.
Evidence-Based Software Engineering
Evidence-Based Software Engineering (EBSE) is a way
to integrate the best research evidence with practice.
Barbara Kitchenham Keele University
ICSE’20047
1. Convert the need for information into answerable questions
2. Identify the best evidence with which to answer these questions
3. Appraise the evidence critically: assess its validity and usefulness
4. Implement the results of this appraisal in software engineering practice
5. Evaluate the performance of this implementation
The 5 Steps of Evidence-Based Practice
}Systematic Literature Reviews
(SRs)
8
How to find the best evidence?
Software Engineering conferences
Software Engineering journals
9
How to find the best evidence?
X, Y, and Z: A Systematic Literature Review
10
It involves reading the right papers and then changing
behavior in the practice of the discipline.
Trisha Greenhalgh Oxford University
Evidence-Based practice is not only about reading and summarizing papers
11
Fábio Queda CIn @ UFPE
There is a lack of connection between systematic reviews
and software engineering practice
IST’201112
SRs do not provide guidelines for practitioners
Most of SR authors affirmed that they
hadn’t a direct impact on industrial practice
Lack of connection with industry is the
6th top barrier
13
In addition
14
Evidence-Based Medicine (EBM) researchers argue that SRs:
Focus on a narrow question
15
Are time consumingthey usually require
between 6 months and 2 years to complete
(1,139 hours, on average) Policymakers often require access to contextualized resources that
address a broader scope of scientific evidence quickly
On the other hand…
16
Rapid Reviews Briefings and Summaries
56% of the rapid reviews were conducted
in the last 3 years
“Rapid reviews simplify the
process of SRs to produce
information in a timely manner”
“[Briefings] translates existing SRs into actionable
messages in the form of short accessible briefings”
17
EBM Researchers
Health decision-makers
18
EBSE Researchers
Software Industry
How to transfer knowledge from systematic reviews to
software engineering practice?
19
20
Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to21
“Evidence Briefings”
22
An one-page document, extracted from a systematic
review, that contains findings useful for
practitioners
Approach
23
1
Systematic Reviews Selection 2
Systematic Review Data Extraction 3
Evidence Briefings Generation
4
Evidence Briefings Evaluation
Approach
24
1
Systematic Reviews Selection 2
Systematic Review Data Extraction 3
Evidence Briefings Generation
4
Evidence Briefings Evaluation
25
1 Systematic Reviews Selection120
Systematic Reviews
26
1 Systematic Reviews Selection120
Systematic Reviews
Fábio Queda CIn @ UFPE
YAY! citation++
27
1 Systematic Reviews Selection120
Systematic Reviews
32 Systematic Reviews
with guidelinesFábio Queda CIn @ UFPE
YAY! citation++
28
1 Systematic Reviews Selection120
Systematic Reviews
32 Systematic Reviews
with guidelines
24 Systematic Reviews with search strings
Fábio Queda CIn @ UFPE
YAY! citation++
29
1 Systematic Reviews Selection120
Systematic Reviews
32 Systematic Reviews
with guidelines
24 Systematic Reviews with search strings
12 Selected SRs
Fábio Queda CIn @ UFPE
YAY! citation++
# Subject2 Global software development3 Agile software development1 Software testing1 Software requirements1 Model based software development1 Software development productivity1 Cost and effort estimation1 Code duplication1 Software engineering knowledge management
Approach
30
1
Systematic Reviews Selection 2
Systematic Review Data Extraction 3
Evidence Briefings Generation
4
Evidence Briefings Evaluation
31
2 Systematic Reviews Data Extraction
Paper title:Original The effectiveness of pair programing: A meta-analysis
Briefing The effectiveness of pair programming
32
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Original The effectiveness of pair programing: A meta-analysis
Briefing The effectiveness of pair programming
Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review.
Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review.
33
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Research findings:
Original The effectiveness of pair programing: A meta-analysis
Briefing The effectiveness of pair programming
Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review.
Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review.
Finding 1 Pairing up of individuals seems to elevate the junior pairs up to near senior pair performance
Finding 2 If you do not know the seniority or skill levels of your programmers, but do have a feeling for task complexity, then employ pair programing either when task complexity is low and time is of the essence, or when task complexity is high and correctness if important
Approach
34
1
Systematic Reviews Selection 2
Systematic Review Data Extraction 3
Evidence Briefings Generation
4
Evidence Briefings Evaluation
35
3 Evidence Briefings Generation: Principles
Similarity: elements that are similar are more likely to be organized together Proximity: closer elements are more likely to be perceived as a group Continuation: elements will be grouped as a whole if they are co-linear Unity: elements that have a visual connection should belong to a uniform group
36
3 Evidence Briefings Generation
1. The title of the briefing 2. The goal of the briefing 3. The findings extracted
from the original review 4. An informative box
with general information
5. The reference to the original review
6. The logos of our research group and university
37
3 Evidence Briefings Generation
1. The title of the briefing 2. The goal of the briefing 3. The findings extracted
from the original review 4. An informative box
with general information
5. The reference to the original review
6. The logos of our research group and university
The template of Evidence Briefings is licensed under CC-BY license!
Approach
38
1
Systematic Reviews Selection 2
Systematic Review Data Extraction 3
Evidence Briefings Generation
4
Evidence Briefings Evaluation
RQ: How practitioners and researchers perceive the content and format of
Evidence Briefings?
39
4 Evidence Briefings Evaluation
40
4 Evidence Briefings Evaluation: Practitioners
473 StackExchange Users who asked questions
related to the SRs
41
4
1 SRs Search Strings
Queried a StackExchange website2
Removed false-positive questions3
Classified questions using open card sort4
programmers.stackexchange.com
sqa.stackexchange.com
pm.stackexchange.comreverseengineering.stackexchange.com
softwarerecs.stackexchange.com 1,738 related questions
473 related questions
Kappa: 0.72
Evidence Briefings Evaluation: Practitioners
42
4
473 StackExchange Users who asked questions
related to the SRs
Evidence Briefings Evaluation: Practitioners
43
4
473 StackExchange Users who asked questions related
to the SRs
only 146 of them had public profile
(LinkedIn, Github, Twitter, etc)
Evidence Briefings Evaluation: Practitioners
44
4
The 22 authors of the 12 Systematic Reviews
Evidence Briefings Evaluation: Researchers
45
4 Evidence Briefings Evaluation: Survey
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Authority & Credibility (e.g , Ph.D., University professors)
Liking (e.g., personalized emails)
Scarcity: (e.g., we defined a deadline)
Brevity (e.g., we asked closed questions as much as possible)
Social Benefit (e.g., 1 USD for the Brazilian Red Cross)
Evidence Briefings Evaluation: Survey
46
4
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Authority & Credibility (e.g , Ph.D., University professors)
Liking (e.g., personalized emails)
Scarcity: (e.g., we defined a deadline)
Brevity (e.g., we asked closed questions as much as possible)
Social Benefit (e.g., 1 USD for the Brazilian Red Cross)
47
4
Survey with Practitioners (17 questions, 2 open-ended)
Survey with Researchers (8 questions, 2 open-ended)
Evidence Briefings Evaluation: Survey
7 respondents (31% of response rate)
32 respondents (22% of response rate)
Results
48
Survey with Practitioners
Survey with Researchers
Results
49
Survey with Practitioners
Survey with Researchers
50
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have in your current position?
51
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have in your current position?
52
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?
53
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?
54
Survey with Practitioners: Acquiring Knowledge
Q5. How often do you use StackExchange websites? Q6. How often do you read software engineering research papers?
55
Q7. Have you ever read a systematic review paper?
Q8. For what reason you read a systematic review paper?
Survey with Practitioners: Acquiring Knowledge
56
Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange?
Survey with Practitioners: Briefings’ Content
57
Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange?
Q11. Why?
Survey with Practitioners: Briefings’ Content
“The question is too specific”
“The question expected more than one answer”
“The question touched a slightly different issue”
“The briefing lacks details”
58
Survey with Practitioners: Briefings’ Content
Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing?
59
Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing?
Survey with Practitioners: Briefings’ Content
Q13. Why?“Agile is not a one size fits all methodology. To make it work you need to see what works for you and your
team. [...] Making bold high level statistical statements about Agile software development will only
hurt it where as it can shine in truly Agile organizations.”
60
Q14. How do you compare the answers from the StackExchange community to the findings presented in the briefing?
Survey with Practitioners: Briefings’ Content
61
Survey with Practitioners: Briefings’ Format
Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable? Q17. Does the briefing look reliable?
62
Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable? Q17. Does the briefing look reliable?
Survey with Practitioners: Briefings’ Format
Results
63
Survey with Practitioners
Survey with Researchers
64
Survey with Researchers: Sharing KnowledgeQ1. How important for you is to share research results to practitioners?
?
65
Survey with Researchers: Sharing KnowledgeQ1. How important for you is to share research results to practitioners?
66
Survey with Researchers: Sharing KnowledgeQ2. How often do you share research results to practitioners?
67
Survey with Researchers: Sharing KnowledgeQ2. How often do you share research results to practitioners?
68
Survey with Researchers: Sharing KnowledgeQ2. How often do you share research results to practitioners?
Q3. How do you do that?
“Teaching”
“Seminars”
“Writing”
“Advisory work”
“Social Networks”
69
Survey with Researchers: Briefings’ ContentQ4. How does the briefing that we sent to you cover the main findings of your paper?
70
Survey with Researchers: Briefings’ ContentQ4. How does the briefing that we sent to you cover the main findings of your paper?
Q5. Why?
YAY! :-)
71
Survey with Researchers: Briefings’ FormatQ6. How easy was to find the information in the briefing? Q7. Is the briefing interface clear and understandable? Q8. Does the briefing look reliable?
Discussions
72
Revisiting Findings
73
Practitioners rarely use research papers as mediums to acquire knowledge.
Software engineering practice still has many beliefs with no evidence basis.
Revisiting Findings
74
Both researchers and practitioners positively evaluated the evidence briefings
Revisiting Findings
75
Both researchers and practitioners positively evaluated the evidence briefings
The briefings well covered the main findings of the original systematic reviews
The Yin-Yang of Research and Practice
76
Researchers want to transfer knowledge. But not all of them do so.
Practitioners want to be more aware of software engineering research. But few of them do so.
Implications
77
Researchers
78
Researchers
79
Practitioners
80
Tool builders
81
Tool builders
82
Educators
83
Conference and Publicity Chairs
84
One last thing..
85
86
Search Strings are your best
87
Assess your search strings on
StackExchange on early stages of a systematic
review planning!
88
Are your search strings well-designed?
“quality + model”, “quality + model driven” and “model driven + experience”
(software AND ((cost OR effort OR productivity) WITH (factors OR indicators
OR drivers OR measure)))
89
Thanks!
Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to90