ELAIS 2017Primer Encuentro Latinomericanode Ingeniería de Software

• Campus San Joaquín

• Universidad Técnica Federico Santa María

• Santiago, Chile

Clave para adoptar técnicas:Determinar qué le sirve a quién

• Hernán Astudillo

• hernan.astudillo@usm.cl

• Departamento de Informática, Universidad Técnica Federico Santa María

• Santiago & Valparaíso, Chile

Agenda

• Motivación

• Estudios empíricos en Ing.SW

• XP Extendido

• PBEC-OTSS

• Patrones v/s tácticas

• Conclusiones

3

Hernán Astudillo

• Académico, Departamento de Informática, UTFSM

• Ingeniero Civil Informático (UTFSM, 1988)

• Ph.D. Information and Computer Science (Georgia Tech, 1995)

• Director Doctorado en Ingeniería Informática & Director académico BPM Center, UTFSM

• MC equipo Toeska de I+D• arquitectura de software

• mejoramiento de procesos

• sistemas semánticos

4

Hernán Astudillo, antes

• Lead Applications Architect, Object Practice

• MCI Systemhouse

• Senior Applications Architect

• Financial Systems Architects (FSA), NYC

• Director de Contratos Tecnológicos CORFO/USM

5

UML

• ¿Quién conoce UML?

• Unified Modeling Language

• ¿Quién usa UML?

• ¿Quién ha usado UML por 10 años?

• ¿Quién ha usado UML por 20 años?

• ¿Quién conoce OML?

• Open Modeling Language

6

Softgoals

72017-03-29 ISW 2017

Effort

¿Qué hace que adoptemos una técnica y no otra?

8

PSP

IEEE/EIA

12207

Baldrige

ISO/IEC

15504

People CMM

IPD-

CMM*

SECAM

SCE

MIL-STD-

498

DOD-

STD-

2167A

MIL-STD

499B*

ISO/IEC

12207IEEE

1220

SDCE

SE-CMM

EIA

731

EIA/IS

632

ISO 9000

series

Ansi/EIA 632

SSE-

CMM

ISO/IEC 15288

CMMI

SA-

CMM

Q9000

DOD-

STD-

2168

FAA-

iCMM#

RTCA

DO-178B

SW-CMM

TL9000

ISO

15939

PSM

SCAMPI

CBA IPI

SAM

FAM**

Process StdsQuality StdsMaturity or Capability

ModelsAppraisalmethods

Guidelines

Six

Sigma

J-STD

016

DOD-

STD-

7935ATSP

The Frameworks Quagmire

Sarah Sheard, in Evolution of the Frameworks Quagmire (2001/2003, aún válido)

Idea: comparar técnicas, notaciones, etc

10

Idea general

Methods

PatternsPractices

Kernel: universals + language

12

Atributos de calidad en la práctica

• Nivel de satisfacción de los clientes en sus proyectos de software en función de atributos de calidad

• Encuesta a 103 profesionales [Tumyrkin et al., 2016]

Sesión 12 17

Atributos de calidad en la práctica

• Nivel de satisfacción de los clientes con respecto a atributos de calidad para diferentes tipos de industrias

Sesión 12 18

Deuda técnica

• 1800 compañías respondieron la encuesta20

Deuda técnica

• Interpretación: Las herramientas actuales no capturan la deuda técnica

21

¿Cómo estudiar este problema?

• Método usual: casos de estudio

• “Una empresa trató de usar esto, hizo esto, y le costó”

• Método más riguroso: encuestas

• “40% de las empresas encuestadas dicen que trataron de usarlo y les costó”

• Método ideal: experimentos

• “Uds 50 usen este método y Uds 50 este otro, para que veamos quiénes quiebran”

• Normalmente infactible

25

XP: ¿A quién le ayuda permitir diseño?

27

Context: XP and Software Design

• Extreme Programming (XP)

• Best known agile development method

• 12 practices

• Evolutionary design approach

• Implement the simplest solution for the currentrequirements

• Don’t worry about next iteration requirements and theirdesign: “You Ain’t Gonna Need It” (YAGNI)

• Tackle overall software structural complexity only whenneeded

28

Problem & Research

• Adoption problems• Practices are difficult to adopt

• Inability to recognize reusable structures• A priori identification vs. emergence of design patterns

• Resistance to change (people insist in designing in advance)

• Research question• Is there any impact on (product) Quality and (people)

Productivity?

• What would happen if we allowed Design and XP practices?

• Study goal• Compare the impact of XP with/without Design on product

design quality and development process productivity

29

Experimental Design (1)

• Two treatments• “XP”: based on the XP definition (evolutionary design)

• “XP+” (Extended XP): variation of XP that incorporates a planned design session at the start of each iteration

• Designing, coding and testing software for a givenproblem using either design approach (XP or Planned)

• Solution implemented in Java, programming in pair, duringtwo development iterations

• Null hypotheses• Using XP+ yields software with similar design quality than XP

approach

• Using XP+ is as productive as using XP

30

Experimental Design (2)

• Independent variables

• Design approach (factor under study): 2 “levels” (XP & XP+)

• Problem to solve (experimental object)

• Undesirable factors of influence: XP Knowledge and DesignExperience

• Dependent Variables

• Product Internal Quality (Design Quality)

• Decision Count (DC), McCabe Cyclomatic Complexity (MCC), Number of Code Statements (NSTMNT)

• Process Productivity

• Number of Classes/hour (PTNOC), LOC/minute (PTLOC), Number of Methods/minute (PTNOM)

31

First Experiment: Design

• Random blocked design, 1 factor and 2 levels

• 62 undergraduatecomputer science students, with backgrounds in OOA&D and OOP, SwEngand SW Design Patterns

• 31 “subjects”• Teams with equivalent

design experience• Divided in 2 “blocks”

• Design experienced (DE)

• Non-design experienced(NDE)

• 4-hour activity

33

First Experiment: Results

• The PTNOC metric withnovices was the mostsignificant result (p ~ 0.05)

• 1-sided t-Test to test whether XP+ is more productive than XP

• PTLOC resulting p-value isp=0.0260 (p < 0.05)

• XP+ helped novices to produced more classes

• Possible interpretation: theymade an effort for placingthe design complexity in theclass structure rather thanleaving it in the methods’ algorithms

34

• p-values of a two-sided t-Test of difference between means for DE subjects and for NDE subjects

Second Experiment: Design

• 2x2 factorial design withrepeated measures

• 22 senior-level computerscience students, with lowlevels of professionalpractice

• in average 2 years older thanthe first experimentparticipants but in the sameacademic program

• Subjects performed bothDesign Methods

• There are no differences ondesign experience

• Both problems are of equivalent complexity

35

Second Experiment: Design

36

Second Experiment: Results (1/2)

• Null hypotheses can not be rejected

• We cannot prove thatMethod influencesdifferences in qualityand productivitybetween XP and XP+

37

• ANOVA of the 2x2 Factorial with Repeated Measures (Method's effect)

Second Experiment: Results (2/2)

• Marginal means suggest (butnot statistically significant):

• XP+ was more productive thanXP for every productivitymetric

• XP produced less LOC, classesand methods

• XP produced better code in two (out of 3) metrics

• This might mean that XP produced less but better code

• This must be furtherinvestigated

• These subjects similar buthave 2 more years of experience

38

• Estimated Marginal Means for the Method’s Effect

Main Threats to Validity

• XP process adaptation• XP+ was built by modifying XP life, and may have caused a

loss of XP main tenets

• Metrics and problems size• The used toy problems might be too simple to show

significant differences on quality or productivity for thechosen metrics

• The quality metrics may not have adequate granularity to show differences in quality

• Academic subjects• Students may not be representative of professional

developers

39

Study Conclusions

• For Novices:• No significant difference in product quality between XP and

XP+• XP+ is more productive

• For Experienced:• XP+ is more productive but XP yields better quality

• Conjecture (HAR)• Novices do not know what to do and Planning helps them do

more and better code, but experienced people already knowwhat to do and Planning creates trade-off productivity/quality

• Advice• Teach XP+ to novices ALWAYS, but to experienced developers

ONLY if productivity matters more than quality

40

PBEC: ¿A quién le ayuda comparar tecnologías?

41

Context

• Off-The-Shelf Software (OTSS) selection• Essential to minimize project uncertainty and risk

• Benefits of new systems become known only after someuse

• How to identify the most appropriate OTSS for a specific organization need?

• Software selection is a subjective and uncertain decisionprocess

• Most organizations lack a rigorous selection process• Often made under pressure by evaluators who

• may not have time or expertise to plan it

• and/or select using only their experience or intuition

42

PBEC-OTSS: Process-BasedEvaluation & Comparison of

OTSS

43

PBEC-OTSS

• Technique to compare alternative implementationsthat support a given process model

• Possible targets for each implemented system: decrease, maintain, or increase current usage levelof each system

• Steps1. Define scope

2. Refine process models

3. Generate alternative process configurations

4. Generate evaluation criteria measures

5. General evaluation

44

2. Refine process model

45

Refined

Process

Model

3. Generate alternativeprocess configurations

46

4. Identify evaluation criteriameasures

• Coverage• OTSS user’s activities rate

• Automation• Variation of automatic activities

• Implementation• Variation of manual activities

• Cost• Cost qualification

• Collaboration• Average of compliance of implemented systems

• Participation• Average of non-compliance regarding the use of implemented

systems

47

Experimental Study

• Compared PBEC-OTSS and an Ad-Hoc approach

• Measured results quality, effort (time), and usersatisfaction

49

Treatments

1. PBEC-OTSS

• previously described

• criteria: Coverage, Automation, Implementation, Cost, Collaboration, Participation

• yields weighted scores and ranking

2. Ad-Hoc Approach

• systematized from literature

• criteria: Functionality, Reliability, Cost, Ease of customization, Ease of use

• yields a (pre-)weighted score for each OTSS

50

Experimental Design

• Subjects

• practitioners finishing a graduate BPM Diploma

• Object

• problem, process and OTSS taken from case study at Chilean public sector institution (FOSIS)

• Configurations

• Simple case

• One system: PeopleNet Recruitment

• Complex case

• Mixed systems: Email2DB & human roles

51

Study Execution

• Participants: 13 Chilean IT practitioners

• Positions: Software Engineer, Project Manager, ProcessEngineer, Consultant, Area Assistant Manager

• None had previous experience with either approach

• For baseline (“ground truth”), a BPM expertperformed the same study

• same problem, process and OTSS

52

Experts results

• Expert results are similar for both approaches

• perhaps because both techniques embed an expertknowledge base

• PBEC-OTSS fuzzy decision-making systems includes parametersproposed by two BPM specialists (unrelated to the baselineexpert)

• Ad-Hoc approach is based on a survey of MIS managers

53

Novices results

• Effort• PBEC-OTSS took (average) 11 minutes more than Ad-Hoc

• Satisfaction• higher for PBEC-OTSS

• Recommendation rate• higher for PBEC-OTSS

• Quality• Simple case: both approaches yield same results as

expert (1% variation for PBEC-OTSS and 2% for Ad-Hoc)

• Complex case: PBEC-OTSS yields better results (5% variation from expert) than Ad-hoc (25% variation)

54

Study conclusions

• Quality of evaluation of alternatives• For experts, both approaches yield similar results• For novices:

• simple case: both approaches are comparable

• complex case: PBEC-OTSS yields better evaluations (== closer to experts’ baseline)

• Effort• PBEC-OTSS may be more difficult (took longer)

• Conjecture (HAR)• Novices do not know what to do and PBEC-OTSS helps them do

better evaluations, but experienced people already know what to do and PBEC-OTSS takes more effort but has no impact on quality

• Advice• Teach PBEC-OTSS to novices ALWAYS but not to experts

55

Patrones y Tácticas: ¿A quién le sirven?

56

Building Secure Software Systems

• Secure software systems are growing niche

• Great variety of security threats

• Knowledge Reuse (KR) allows to reduce risk and costs

• Principaks argued on two KR approaches

• Security Patterns [Fernandez 2012]

• Security Tactics [Bass+ 2003]

57

Tactics

Catalog58

Security Patterns

• Define solutions to handle threats or to fix a vulnerability

• Detection and mitigation of threats throughpatterns

• Include a solution to a security problem and severalsections that define their use, applicability, advantages, and disadvantages.

62

Results: Effort [minutes]

79

Results: Threat Identification[quantity]

80

Results: Quality [expertevaluation rating]

81

Results: Medians

82

Group QoT Quality Effort

Novice-Pattern 3,00 2,75 89,00

Novice-Tactic 3,00 2,83 78,50

Expert-Pattern 3,50 2,54 82,00

Expert-Tactic 4,00 2,63 82,50

KW p p=0,072 p=0,696 p=1

No No No No statisticallystatisticallystatisticallystatistically significantsignificantsignificantsignificant differencesdifferencesdifferencesdifferences werewerewerewere foundfoundfoundfound((((KruskallKruskallKruskallKruskall----WallisWallisWallisWallis ))))

Results: Experience groupsmedians

83

Experience QoT Quality Effort

Novice 3,00 2,83 85,00

Expert 4,00 2,63 82,50

Total 3,00 2,83 84,50

Results: Experience groupsmedians

• Statistically significant differences for the numberof identified threads

• Median test (p=0,014)

• (experts identified more threats than novices)

84

Validity Analysis

• Internal Validity: Classic threats• Students as subjects

• Previous knowledge differences

• Experience differences

• Construct Validity: A Lot of Threats (and interestingquestions)

• Lack of a method definition of how to apply bothapproaches

• How do architects really represent and assess securitydecisions?

• Threat identification and mitigation, are separable mental processes?

85

Study Conclusions

• Solution quality & effort

• No significant differences experts v/s novices in tacticsv/s patterns

• Security threat identification

• Experts identified more threats than novices

• Several construct validity threats arose when tryingto design the decision making activity

• How do software architects manage the process of hardening systems?

86

DVIA: DVIA: DVIA: DVIA: ¿Cómo toman decisiones ¿Cómo toman decisiones ¿Cómo toman decisiones ¿Cómo toman decisiones los arquitectos?los arquitectos?los arquitectos?los arquitectos?

Recover previous decisions

What do we do to

respond to a new

constraint?

What decisions on data connection

were taken by the architectural team?

� Can we replay the meeting?

� Or, can we index the meeting?

Architects make important architectural

decisions in design meetings

88

Context

• Little empirical evidence about how architects make design decisions (Tang,2006).

• Attempt to implement the applied models in other fields.

• Awareness of the importance and advantages of availing the design reasoning (Tang,2006).

• Design reasoning capture is intrusive, boring and expensive (lots of 90’s data on this)

89

Problem

• Decisions made in software architecture meetings, and their justification, are not documented.

• At most in meeting minutes.

• Anec-data: failed BI meetings project

• Loss of fundamental information to understand the architecture design.

90

Key idea: verbal interventions

• We explored extending analysis of software design activities:

• Use a verbal intervention model to understandhowsoftware architects actually take design decisions

• Attempt to recover architectural decisions using this model

91

Case Study: PUA - Pan-andean spatial Project

92

Design of a Command and Critical mission Control (C3)

• Model rockets, research about new technologies to build and assemble engines and develop efficient fuels.

• Specifically, develop a system of Command and Critical mission Control (C3).

• Design physical infrastructure and software architecture to support online communication among a test vehicle, launch base, and Command and Control Center (C3).

93

C3 Architecture overview

• Some architectural decisions:

• Equipment network structure

• Video cameras location and characteristics

• GPRS Internet router selection (simulated vehicle)

• Network Power Supply

94

Changes in rocket test location

• Test with rocket simulator were made under conditions rocket with GPRS communication (Soacha, Cundinamarca, Colombia).

• However, the prototype will be tested in distant urban centers rather than where it is not possible GPRS communication (Inirida, Guainía, Colombia).

95

Study Design

Preparation

• Participants were informed about the need to keep track of meetings to better support architectural design

• …but not of which aspects would be studied.

Execution

• Carried out over 3 weeks in March-April 2012.

Data Validation

• Data collected using video and audio recording for six (6) participants that conform one architectural team.

• Participant were given separate project responsibilities.96

Intervention timelinedistribution

97

The architectural meeting participants are engaged with the

propose solutions, followed by understanding the problem and

evaluating solution alternatives.

Decision topic identification

98

Designers work in nested, repetitive, and intertwined cycles, which

overlap with the simultaneous resolution of several design topics

Decision topic identificationThe network structure of

the equipment that

support C3

Power supply to

the computer

network

Characteristics and

location of the video

cameras into C3

Select the

GPRS Internet

router

Data transmission

from the launch

platform

Get more gain in

local wifi signal

data

Get more gain in

GPRS Internet

cellular

Test on internet

modem

Some subjects are hierarchically linked, so when a new issue or

problem appears the focus is changed to resolve before it and

thereafter the initial subject.

99

Example: recovered decision on“GPRS Internet router selection”

100

Interpretation

• Mapping interventions and design decisions allows to identify cycles that describe decision making activities.

• In some cases, decisions are developed concurrently, and may overlap in time.

• Architecture designers work in nested, repetitive, and intertwined cycles, which overlap with the simultaneous resolution of several design subjects (Baker, 2010).

• Much of the design process addresses the coevolution of problem understanding and solution framing (Dorst, 2001); see also Twin Peaks.

101

Conclusiones

102

Conclusiones

• Comparación experimental de técnicas es muy iluminadora

• En general, los novatos no saben qué hacer y se benefician de transferencia, pero los expertos ya saben qué hacer y se benefician poco

• Posible trade-off: productividad/calidad

103

Personas y Proyectos Citados

104

Proyecto TacPat4SS

• Comparación sistemática de patrones v/s tácticas

• H. Astudillo, René Nöel, Gilberto Pedraza, Eduardo Fernández, Santiago Matalonga

• Subproducto: catálogo razonado de tácticas

• Experimental Software Engineering

• Financiado por FONDECYT (2014-2017)

105

Proyecto VirtualMarket

• Identificación automática de oportunidades de asociatividad entre PyMEs

• H. Astudillo, Romina Torres (UNAB), Rodrigo Salas (UV)

• Colaboración con ChileCompra

• Subproducto: recomendador de licitaciones

• Financiado por FONDEF (2013-2015)

106

Colaboradores citados

• René Nöel (XP Extendido, Patrones v/s Tácticas)• Magister en Ciencias Ing. Informática 2008• Ahora académico en U.Valparaíso

• Gonzalo Valdés (XP Extendido)• Magister en Ciencias Ing. Informática 2009• Ahora @U. Stanford

• María Jesús Faúndez (PBEC-OTSS)• Magister en Ciencias Ing. Informática 2013

• Gilberto Pedraza (Patrones v/s Tácticas)• Dr. Ing. Sistemas, co-tutela U. Andes (Bogotá) & UTFSM

• Eduardo Fernández Buglioni (Patrones v/s Tácticas)• Profesor, Florida Atlantic University (EEUU)

107

Otros experimentos en Ing.SW

• Claudia López• Extracción automática de decisiones de diseño [Arquitectos de software]

• Magister en Ciencias Ing. Informática 2009; ahora Ph.D. & académica@UTFSM

• Carlos Becerra• Recomendación automática de objetos didácticos digitales [Profs. Historia]

• Doctor en Ing. Informática 2012; ahora académico @U.Valparaíso

• Romina Torres• Multi-agentes p/identificar asociatividad de PyMEs [ChileCompra]

• Doctor en Ing. Informática 2014; ahora académica @UNAB

• Pablo Cruz• Computación basada en confianza [Usarios de buscadores de TIC]

• Magister en Ciencias Ing. Informática 2016

• Sven von Brand• Reusabilidad de software de juegos [Desarrolladores de juegos]

• Magister en Ciencias Ing. Informática 2017 (est.)

108

Haciendo experimentos para saber cómo transferir tecnología

• Hernán Astudillo

• hernan.astudillo@usm.cl

• Departamento de Informática, Universidad Técnica Federico Santa María

• Santiago & Valparaíso, Chile

110