Crowdsourcing the notation of domain specific languages

Brambilla, Cabot, Canovas, Mauri. Better Call the Crowd: Using Crowdsourcing to Shape the Notation of Domain-Specific Languages

Using Crowdsourcing to Shape the Notation of

Domain-Specific Languages

Marco Brambilla, Jordi Cabot, Javier Luis Cánovas Izquierdo, Andrea Mauri

Contact: [email protected] @marcobrambi

Better call the crowdBetter call the crowd


Prologue:

Context and

Motivation


Context - 1

Domain-Specific Languages (DSLs) should be all about

• Domain

• Specificity

And yet, frequently challenged in terms of acceptance and adoption


Context - 2

Software and modeling people: we tend to…

• Rely on tools

• Improve on methods

• Focus on performance, quality


Context - 3

Software and modeling people: we tend to…

• Be anti-social

• Forget we are actually humans

• Reflect this on the research and solutions: not

for humans





Problem Statement

Domain-Specific Languages (DSLs) should be all about

• Domain

• Specificity

And yet, frequently challenged in terms of acceptance and adoption

• We stick to MDE, testing, SwEng tools only


Proposed Solution

Rely on COGNIFICATION for improving process

and quality of MDE practices1

Especially: rely on the human side

1 Jordi Cabot, Robert Clarisò, Marco Brambilla, Sébastien Gerard. Cognifying Model-Driven

Software Engineering. GrandMDE (Grand Challenges in Modeling) at STAF 2017.


Episode 1:

Crowdsourcing and

Crowd Management


Call the Crowd

Crowdsourcing: The process of building a computation

or information collection job

by using computers as organizers

by organizing the work as several tasks

by dealing with dependencies on data and tasks

by assigning tasks to humans


9-1-1, what’s your emergency?

initial hypothesis

Collecthuman

feedback

MDSE methods

Crowd or Social Platform


Crowd Management: Design Process

A simple task design and deployment process, based on specific data

structures

Task Specification Task PlanningTask Execution

& Control

• Task Spec: task operations, objects, and performers Dimension Tables

• Task Planning: work distribution Execution Table for task monitoring

• Control Specification: task control policies Control Mart


Task Design

Which are the input objects of the crowd interaction?

Which operations should the crowd perform?

How should the task be split into micro-tasks assigned to each

person?

How should a specific object be assigned to each person?

How should the results of the micro-tasks be aggregated?

Which execution interface should be used?


Crowd task management problems

Task splitting: the input data collection is too complex

relative to the cognitive capabilities of users.

Task structuring: the query is too complex or too critical

to be executed in one shot.

Task routing: a query can be distributed according to

the values of some attribute of the collection.


Splitting Strategy

Given N objects in the task

Which objects should appear in each MicroTask?

How many objects in each MicroTask?

How often an object should appear in MicroTasks?

Which objects cannot appear together?

Should objects be presented always in the same order?


Crowd Management is hard

Task Type Object Type

AttributeOperation Type

*

Performer

Task Politician*

*

Object Attr.

Output Attr.

⎧

⎫

⎨

Politician: ObjectType

LastName: Attribute

Photo: Attribute

Classify by Political Party:

TaskType

MODEL

TRANSF.

(MT1)

MODEL

TRANSF.

(MT2)

Classify: OperationType

µTask723: µTask

J. F. Kennedy: Politician

J. F. Kennedy: Politician


Task

MO

DE

L L

EV

EL

INS

TA

NC

E L

EV

EL K

enne

dy

JFK.jp

g

LastNam

e

Photo

Party

NULL

3

4 6

Op. Parameter

Republican Party:

OpParameter

Democratic Party:

OpParameter

2

*

*


Task

Luigi: Performer

Mario: Performer

George Bush: Politician

µTask

Performer

TaskPolitician

*

Status

5

*

*

Mario: Performer

George Bush: Politician

Barak Obama: Politician

<<instance-of>>

<<instance-of>>

<<instance-of>>

1

OpT

ypes

Para

met

ers

Name

Platform

Oba

ma

BO.jp

g

NULL

METAMODEL STRUCTURAL MODEL

LastName

PartyPhoto

Non-spammer

*

Status

StartTsEndTs

Status

WORKPLAN MODEL

Bush

GB.jpg

NULL

Party


Crowd Control Mart

µTObjExecution

Performer

TaskPoliticianSta

tus

Sta

rtTs

End

Ts

µTas

kID

Object Control

Performer Control

Task Control

Objec

tID

Com

pObjs

Task

ID

Com

pExe

cs

Name

PlatformStatus

PID

Eval

RightWrong

Eva

l

Dem

RepA

nswer

StatusLastName

Party

Photo

Par

ty


Reactive System

Guaranteed

termination

Extensibility

Control Production Rules

Result Production Rules

Execution Modifier Rules

EXECUTION

OBJECT PERFORMER TASK

OBJECT CONTROL

PERFORMER CONTROL

TASK CONTROL


Episode 2:

Crowdsourcing

DSL Notations


A (Simplistic) Traditional Language Design Process

Different phases

Feedback completely separated


Levels of Crowd Involvement

Incremental role


Our Current Challenge: Level (b)

Abstract Syntax

Concrete Syntax

Concrete Syntax

Concrete Syntax

Language in use


Process in Use


Input

(1) the abstract syntax of the language

(metamodel)

(2) a set of symbols representing the candidate

concrete syntax

(3) the mapping

(4) optional model examples


Task Definition Principles

Set tasks and questions that grant coherency of

the language

Cover the whole language structure

Don’t ask many times the same question

Grant support of agreement


Task Definition

Tasks with many correlated questions

Majority voting on tasks

Expertise and demographics considered

Pattern based approach


Pattern-based Task Design

Patterns in the metamodel are used to create

Tasks and Questions

Tasks:

• Question(s)

• Alternative notations

• Interactive examples


Generation of Tasks: Patterns


Episode 3:

Use Case and

Experiment


BPMN

Standard

Widely adopted

Widely understood (20% notation)


BPMN

Standard

Widely adopted

Widely understood (20% notation)

Widely misinterpreted and underused

(80% notation)


INPUT 1: A Metamodel (Excerpt)


INPUT 2: Challenging the Notation

Can we do better

than the standard?


Generation of Tasks


Generation of Tasks


CrowdUser Interface


Experimental Setting

85 participants, 10 countries

• undergraduate/graduate computer science students

• modeling experts

• IT professionals

Assessed against modeling, language specification and BPMN experience

Answer aggregation strategies compared (effectiveness and affordability ):

• static majority policy

• targeted agreement


Results: Task and Events



Targeted Agreement

minimum number of answers = 10

agreement level = 60%

maximum tasks = 50


Targeted Agreement

minimum number of answers = 10

agreement level = 60%

maximum tasks = 50


And the winner is…


And the winner is…


Is the Solution Good?

Ask the crowd!

On AMT

Compare

1st, 2nd and 3rd

alternative notations


AMT Results

Question: shown two models, which one do you prefer?

• 0.05$ per answer

• 204 answer

Results:

• 1st best notation preferred 60% of the times over the 2nd

best, and 69% of the times over the 3rd best.

• 2nd best preferred 67% of times over the 3rd best


Epilogue:

Past and Future


Conclusions

The past:Best notation actually identified

Problems in coherency of the notation

Eat your own dogfood: We did MDE for MDE

Ugly UI -> crowdsource it?

The future:Combination of crowd + AI for MDE?


Using Crowdsourcing to Shape the Notation of

Domain-Specific Languages

Marco Brambilla, Jordi Cabot, Javier Luis Cánovas Izquierdo, Andrea Mauri

contact: [email protected] @marcobrambi

Thanks for Calling!