Expert mining compsac-2014

Expert Mining for Evaluating Risk Indicators Scenarios

Oscar Franco-Bedoya, Dolors Costal, Soraya Hidalgo, Ron Ben-Jacob

Monday 21st July 2014

2

Background

Workshop procedure

Generalization of the approach

Applications and lessons

learned

Related work

Conclusions and further

work

Outline

3

Bac

kgro

un

d

4

Bac

kgro

un

d

5

Bac

kgro

un

d

Managing risk in

open source adoption

6

Identification

Mitigation

methods

Management

Advanced tools

Provides

Platform

Methods

OSS

adoption

projects To supporta

In

Uses

Ecosystem

modelingStatistical

tools

Risk

Management

i.e. i.e.i.e.

Bayesian

Networks

Social

network

analysis

Expert

scenarios

assessment

e.g. e.g. e.g.

Bac

kgro

un

d

IdentificationRisks

Management

Systematic

protocoluses

7

Bac

kgro

un

d

Project Site

Code Version Repository

Bug Tracker IRC Mailing List

Ecosystem hubs

Project indexes

Social Media

Twitter Facebook

I

Raw

DataSNA

Measures

Risk Indicators:• Project

• Community

• ContextualII

IndicatorsScenario-based

Assessment DomainExpert

Business Analysis• Business goals

III

Business

Goals

3-Layered

RISCOSS approach

Number of

downloads

Number of

event

referencesCentrality

Number of

open bugs

8

Ou

tlin

e

Background

Workshop procedure



learned

Related work


work

Outline

9

The RISCOSS tactical workshop is designed to permit

experts to assess risk indicators

Wo

rksh

op

pro

ced

ure

10

Tactical workshop

protocol

Wo

rksh

op

pro

ced

ure

Pre-Tasks

Part II: Community dinamics

Part I : Community data

TacticalWorkshop Organizer

DomainExpert

6.Make presentation of RISCOSS

project summary.

7.Explain the RISCOSS analytics platform.

8.Explain the tactical

workshop Part I and

Part II

9.Explain the risk driver

selection WS Excel File

Excel File: risk driver selection WS

TabsTimelinessActivenessCommunity

10.Study the use case scenario

11.Assess the use case scenario

overall state

12.Determine the level of

the risk indicator



overall state


the community

risk indicator

PDF File:KPA RISCOSS Analytics

[more risk drivers]

[not more risk drivers]

RISCOSS Analytics

Team

1.Determine drivers and

risk indicators

2.Construct Bayesian networks

3.Define scenarios

4.Identify and contact experts

5.Workshops planning and preparation

16.Send the scenarios

judgement

Populate BN nodes

11

Pre-Tasks


RISCOSS Analytics

Team


risk indicators


3.Define scenarios



Correspond to

tasks that must

be done before

conducting the

tactical workshop

Wo

rksh

op

pro

ced

ure

12

Pre-Tasks


RISCOSS Analytics

Team


risk indicators


3.Define scenarios



Wo

rksh

op

pro

ced

ure

Risk Driver

Forum posts per day

Forum messages per thread

Mail per day

Overall community size

Number of developers involved

Number of testers (individuals

providing feedback)

Number of companies using the

software

Companies supporting the project

(adding to code)

Risk Indicator Activeness

13

Pre-Tasks


RISCOSS Analytics

Team


risk indicators


3.Define scenarios



Wo

rksh

op

pro

ced

ure

14

Pre-Tasks


RISCOSS Analytics

Team


risk indicators


3.Define scenarios



Wo

rksh

op

pro

ced

ure

15

Pre-Tasks


RISCOSS Analytics

Team


risk indicators


3.Define scenarios



Wo

rksh

op

pro

ced

ure

16

The tactical

workshops begin

with an exposition

about the main

topics that will be

covered

Wo

rksh

op

pro

ced

ure


project summary.



workshop Part I and

Part II



17

Wo

rksh

op

pro

ced

ure






18

Wo

rksh

op

pro

ced

ure


DomainExpert



overall state


the risk indicator

19

Wo

rksh

op

pro

ced

ure




overall state


the community

risk indicator


judgement

DomainExpert

20

Ou

tlin

e

Outline

Background

Workshop procedure



learned

Related work


work

21

Ge

ne

raliz

atio

n o

f th

e a

pp

roac

h

22

Ou

tlin

e

Outline

Background

Workshop procedure



learned

Related work


work

23

Applications and lessons learnedA

pp

licat

ion

s an

d le

sso

ns

lear

ne

d

We have conducted 10 technical workshops In private and official organizations and academic institutions.

The experts were from different countries.

France, Israel, Italy, Spain, and Netherlands

There are some inconsistencies in the scenarios The scenarios were designed using random number generators

While the domain experts are conducting the tactical workshops, The degree of "calibration" of their judgement improves

24

Ou

tlin

e

Outline

Background

Workshop procedure



learned

Related work


work

25

Re

late

d W

ork

Related Work

Delphi method

(QUELCE)

Quantifying Uncertainty

in Early Cost

Estimation[2]

Reliable consensus of

opinion of a group of

experts [1]

[1]N. Dalkey and O. Helmer, “An experimental application of the Delphi method to the use of experts,”.

[2] R. W. Ferguson, D. Goldenson, J. M. McCurley, R. W. Stoddard, and D. Zubrow, “Quantifying

Uncertainty in Early Lifecycle Cost Estimation ( QUELCE ),”

26

Ou

tlin

e

Outline

Background

Workshop procedure



learned

Related work


work

27

DomainExpert

Conclusions and further workC

on

clu

sio

ns

and

fu

rth

er

wo

rk Pre-Tasks




DomainExpert


project summary.



workshop Part I and

Part II







overall state


the risk indicator



overall state


the community

risk indicator


[more risk drivers]

[not more risk drivers]

RISCOSS Analytics

Team


risk indicators


3.Define scenarios




judgement

Populate BN nodes

DomainExpert SNA

Project Site

Code Version Repository

Bug Tracker IRC Mailing List

Step-by-step

protocol

Data used to

construct

Bayesian

networks

Future work

Combines opinion

of domain experts

with OSS raw data

Empirical

application

&

lessons

28

Ou

tlin

e

Outline

Background

Workshop procedure



learned

Related work


work

29

SEe

D4

FI

Thanks for your attention

Comments and Questions

31

SEe

D4

FI

Engineering

Expert mining compsac-2014