ANNEX A - UPC. Universitat Politècnica de Catalunyasmartinez/files/ist16-attachment.pdfCertificates Yes List of certificates: SCJP, SCWCD, SCEA, SSCSA, SSCA, PMP, OCA, OCP, OCM, CSA,

ANNEX A

“BENEFITS AND DRAWBACKS OF SOFTWARE REFERENCE

ARCHITECTURES: A CASE STUDY”

Index A.1 INTERVIEW GUIDE ................................................................................................................... 2

Interview for software architects .............................................................................................. 2

Online questionnaire for architecture developers .................................................................... 4

Online questionnaire for applications builders ......................................................................... 6

A.2 DATA ANALYSIS: REPRESENTATIVES QUOTES FROM SOFTWARE ARCHITECTS ...................... 9

Table 7. Detailed results for the main benefits of using SRAs for the client organization. ...... 9

Table 8. Detailed results about how shorter time-to-market is achieved. ............................. 11

Table 9. Detailed results for the main benefits of using SRAs for everis. ............................... 12

Table 10. Detailed results for the main drawbacks of using SRAs in client organizations. ..... 13

Table 11. Detailed results about the artifacts used in the training for using the SRA. ........... 14

Table 12. Detailed results for the improvements that stakeholders would do. ..................... 15

A.3 DATA ANALYSIS: CLUSTER ANALYSIS..................................................................................... 16

Cluster analysis 1: Which types of stakeholders have more experience? .......................... 16

Cluster analysis 2: Do SRA projects in banking require more effort than SRA projects in

other application domains? ................................................................................................ 16

Cluster analysis 3: Is maintenance and learning curve more costly in banking SRA projects

than in other SRA projects? ................................................................................................ 17

Cluster analysis 4: Which SRA benefits and drawbacks are present in the public sector

domain? ............................................................................................................................... 17

Cluster analysis 5: What is the relationship among “Change of SRA technologies (Imp-B)”,

“Update to the latest technologies (Ben-D)” and “Wrong decisions (Dra-F)”? .................. 18

Cluster analysis 6: Is standardization more strongly perceived by experienced

stakeholders than stakeholders with less experience? ...................................................... 18

Cluster analysis 7: Is there a correlation between evolution SRA projects and the

perceived need of adding functionality or modules to the SRA (Imp-A)? .......................... 19

Cluster analysis 8: Did the participants who asked for changing technologies (Imp-B)

mention the use of latest technologies (Ben-D)? ............................................................... 19

Cluster analysis 9: SRA benefits as seen by different roles. ................................................ 19

Cluster analysis 10: SRA drawbacks as seen by different roles. .......................................... 20

A.1 INTERVIEW GUIDE To study the benefits and drawbacks of Software Reference Architectures (SRA), we have three

personalised questionnaires for the three different kinds of stakeholders:

1. A face-to-face semi-structured interview for software architects.

2. An online questionnaire for architecture developers.

3. An online questionnaire for applications builders.

Interview for software architects The interview for software architects is executed in two steps:

Step I: Sending the interview script

Prior to the face-to-face interview, a questionnaire is delivered to collect personal information

about the interviewee and to inform him/her about the interview. This questionnaire includes

a template of short questions that must be answered in the form sent (see Table 1), and the

script with the questions that will take place during the interview.

Table 1. Delivery and collection questionnaire for software architects.

Question Mandatory Options

PERSONAL DATA

Name and Surname Yes Text

Email Yes Text (Email)

Phone No Number

Level of education Yes List of levels: secondary school, high school, professional training, bachelor, master, PhD.

Education Area Yes List of education areas: IT, telecommunications, administration and management, industrial, mathematics, physics, economics, chemistry,

statistics, electronics, biology, other.

Certificates Yes List of certificates: SCJP, SCWCD, SCEA, SSCSA, SSCA, PMP, OCA, OCP, OCM, CSA, others.

ABOUT THE PROJECT

Project and SRA of the project

Yes List of projects and the corresponding SRA name

Role(s) in the project Yes Text

Number and role of participants

Yes Text (paragraph)

Project initial development phase duration

Yes Number of months

Project maintenance duration


How many participants of the project had experience in SRAs?

Yes Text (paragraph)

Did you have previous experience in SRAs before the project?

Yes Boolean and text for comments

Total effort of the project (people/month)

Yes Persons-month

EXPERIENCE IN EVERIS

Job position in the IT consulting firm (when you participated in this project)

No List of job positions: junior programmer, program analyst, senior analyst, team/project leader.

Years in this job position (when you participated in this project)

No Number

Years in the IT consulting firm (when you participated in this project)

No Number

Experience in project management (when you participated in this project)

No Free Text

Step II: Interview

The interview is recorded (in audio) for later transcription.

It is required that the respondent has the role of software architect and has

participated as a project technical leader (or who has the knowledge to respond as

such).

Questions of the face-to-face interview are shown in Table 2.

Table 2. Interview for software architects.

RQ Question

OVERVIEW OF THE ARCHITECTURE

- What do you understand by SRA?

- Does your SRA take into account client’s business processes?

- Did the use of your SRA cause any organizational change in the client organisation?

BENEFITS AND DRAWBACKS OF THE ARCHITECTURE

RQ1 Which benefits developers might experience while using your SRA?

RQ1 How does the use of your SRA reduce the time-to-market of SRA-based enterprise applications in the client organisation?

RQ1 To sum up, what conclusions do you draw from the facilities provided by your SRA for the client organisation?

RQ1 To sum up, what conclusions do you draw from the facilities provided by the SRA for the IT consulting firm?

RQ2 Which problems developers might experience while using your SRA?

RQ2 How was conducted the training for the client organisation in order to them use your SRA?

RQ2 What do you think should be replaced, included or updated in prospective versions of the SRA?

In Table 2, we can see that there are questions related to others ones. This is to confirm

findings. For instance, there are three questions about the overview of the architecture that

aim to analyse the type of the reference architecture (e.g., SRA or enterprise reference

architecture).

Furthermore, other example of sanity checks can be found on the questions about the benefits

of SRAs. First, we started asking “Which benefits developers might experience while using your

SRA?”. Later, we recall and summarize with other two questions addressing the same issue

(i.e., the benefits of SRAs): “To sum up, what conclusions do you draw from the facilities

provided by your SRA for the client organisation?” “To sum up, what conclusions do you draw

from the facilities provided by the SRA for the IT consulting firm?”

Online questionnaire for architecture developers Architecture developers are contacted by an online questionnaire. This questionnaire is

divided into two parts. First, there are questions about personal information (see Table 3).

Second, the questions to answer the research questions are conducted (see Table 4).

Table 3. First part of the online questionnaire for architecture developers.


PERSONAL DATA



Phone No Number

Level of education Yes List of levels: secondary school, high school, professional training, bachelor, master, PhD

Education Area Yes List of areas: IT, telecommunications, administration and management, industrial, mathematics, physics, economics, chemistry,

statistics, electronics, biology, other

Certificates Yes List of certificates: SCJP, SCWCD, SCEA, SSCSA, SSCA, PMP, OCA, OCP, OCM, CSA,

other

ABOUT THE PROJECT

Project and SRA of the project. Yes List of projects and their corresponding SRA

Describe briefly this SRA project. Yes Text (paragraph)

Were you focused on the development of some component/module of the SRA? Which one(s)?

Text (paragraph)

Besides architecture developer, did you play another role in the SRA project?

Yes Boolean and comment

With how many participants did you interact during the development of the product?

Yes Number

Duration of the development period (if it existed) of the SRA project.

Yes Numbers

Duration of the maintenance period (if it existed) of the SRA project.


Did you have previous experience in SRAs development before this project?

Yes Likert scale


Job position (developer) in the IT consulting firm (when you participated in this project)

No List of job positions: junior programmer, program analyst, senior analyst,

team/project leader.


No Number


No Number

Table 4. Main part of the online questionnaire for architecture developers.

RQ Question Mandatory Options


- Which of the following definitions express best for you an SRA?

Yes List of definitions:

It is the description of the enterprise architecture, including business aspects, strategy, process, method and components.

It is the structure, function and interaction between the software elements of the system.

It is a software architecture where the structures and respective elements and relations provide templates for concrete architectures in a particular domain or in a family of software systems.


RQ1 Which benefits does the client organization might experience while using your SRA?

Yes List of benefits:

reduced development time,

reduced resources for development,

applications aligned with business need (company mission),

improved decision making,

improved agility and requirements changes,

modularity,

latest technologies,

none,

other


Yes Free text

RQ1 To sum up, what Yes Free text

conclusions do you draw from the facilities provided by the SRA for the IT consulting firm?

RQ2 Which problems does the client organization might experience while using your SRA?

Yes List of problems:

high learning curve,

complexity,

reduced flexibility and innovation because of restrictions,

none,

other

RQ2 How long was the training for the client organisation in order to them use your SRA?

Yes List of training periods:

high (more than 2 months),

medium (between 2 weeks and 2 months),

low (less than 2 weeks), you already knew the SRA,

n/a


Yes Free text

Online questionnaire for applications builders SRA-based enterprise applications developers are contacted by an online questionnaire. This

questionnaire is divided into two parts. First, there are questions about personal information

(see Table 5). Second, the questions to answer the research questions are conducted (see

Table 6).

Table 5. First part of the online questionnaire for applications builders.


PERSONAL DATA



Phone No Number

Level of education Yes List of levels: secondary school, high school, professional training, bachelor, master, PhD

Education Area Yes List of areas: IT, telecommunications, administration and management, industrial, mathematics, physics, economics, chemistry,

statistics, electronics, biology, other

Certificates Yes List of certificates: SCJP, SCWCD, SCEA, SSCSA, SSCA, PMP, OCA, OCP, OCM, CSA, other

ABOUT THE PROJECT

Project and SRA of the project. Describe briefly this SRA-

Yes List of projects and their corresponding SRA

based enterprise application project.

Role(s) in the project Yes List of roles

With how many participants did you interact during the development of the product?

Yes Number

Did you have previous experience in SRA-based enterprise applications development before this project?

Yes Yes or no


Job position (developer) in the IT consulting firm (when you participated in this project)

No List of job positions: junior programmer, program analyst, senior analyst, team/project

leader.


No Number


No Number

Table 6. Main part of the questionnaire for applications builders.

RQ Question Mandatory Options


- Which of the following definitions express best for you an SRA?

Yes List of definitions:

It is the description of the enterprise architecture, including business aspects, strategy, process, method and components.

It is the structure, function and interaction between the software elements of the system.

It is a software architecture where the structures and respective elements and relations provide templates for concrete architectures in a particular domain or in a family of software systems.


RQ1 Which benefits do you experience while using the SRA?

Yes List of benefits:

reduced development time,

reduced resources for development,

applications aligned with business need (company mission),

improved decision making,

improved agility and requirements changes,

modularity,

latest technologies,

none,

other


Yes Free text

RQ2 Which problems do you experience while using the SRA?

Yes List of problems:

high learning curve,

complexity,

reduced flexibility and innovation because of restrictions,

none,

other

RQ2 How long was your training process in order to use the SRA?

Yes List of training periods:

high (more than 2 months),

medium (between 2 weeks and 2 months),

low (less than 2 weeks), you already knew the SRA,

n/a


Yes Free text

A.2 DATA ANALYSIS: REPRESENTATIVES QUOTES FROM SOFTWARE ARCHITECTS

Table 7. Detailed results for the main benefits of using SRAs for the client organization. Code Category name Category description Representative quotes from SA Resp.

SA Freq. SA

Resp. AD Freq. AD

Resp. AB Freq. AB

Ben-A

Reduced development costs.

Mainly due to software component reuse that facilitate functionality and speed up the process, leading to shorter time-to-market.

“If the developers need to use common software, they know that the SRA offers software elements that facilitate functionality and speed up the process” [F]. "The cost of developing a new application is lower when it is based on the architecture than ad hoc." [A] "Application development time is lower." [B] "Common services are covered." [E] "It is a common platform among all the front-ends (i.e., mobile applications, web applications, ATM, desktop applications...)" [B]

A, B, C, E, F, G, I

7

A-AD, B-AD, C-AD, D-AD, E-AD, F-AD, G-

AD, H-AD

8 A-AB, B-AB, D-AB,

E-AB, F-AB1, F-AB2, H-

AB2, I-AB

8

Ben-B

Reduced maintenance costs.

Because of: better understandability of systems derived from the SRA; the fact that SRA common elements have fewer errors. Another reason why maintainability is improved is the modularity of SRAs.

“The cost of maintaining an application based on the SRA is lower because SRA-based applications are more comprehensible and easier to evolve and maintain” [A].

A, D, F, G, I

5

A-AD, B-AD, E-AD, F-AD, G-AD, H-

AD, I-AD

7 A-AB, B-AB, C-AB,

D-AB, E-AB, F-AB1, F-AB2, H-

AB1, H-AB2, I-AB 10

Ben-C

Easier development and increased productivity of application builders.

This is due to the fact that architecturally-significant requirements have been already addressed. Also, some SRA artifacts make easier the development of applications, e.g. by abstracting from most technical problems (EJB, back-ends…).

“The SRA abstracts you from the most technical problems (e.g., commutations, work service, EJB, backends). This is the main benefit.” [B]. “SRA modules can be used independently” [E]. "The benefits are easy-to-use and productivity in the developments, always that you follow the established procedures and methods." (C) "A big amount of code is automatically generated." (H) "Easy development allows hiring people with a very technical profile." (D) "Architecture modules can be used independently. This is the main benefit, its flexibility while using it. It isn't a prescriptive architecture, there are parts that can be used or not." (E)

A, B, C, D, E, H, I

7

C-AD, D-AD, G-AD, H-

AD, I-AD

5 A-AB, B-AB, D-AB,

E-AB, F-AB1, F-AB2, H-AB1, H-

AB2

8

Ben-D

Incorporation of latest technologies.

Incorporation of latest technologies, which among other things facilitates the recruitment of professionals with the required technological skills.

“Technological updates facilitate the recruitment of professionals” [H].

C, F, H

3

A-AD, B-AD, D-AD, G-

AD, H-AD

5 C-AB, D-AB, E-AB, F-AB1, F-

AB2, H-AB2, I-AB 7

Ben- Applications more aligned Applications more aligned with business needs, e.g. by “The business process of reviewing records was C, H, I 3 D-AD, F- 4 C-AB, E- 5

E with business needs. supporting the particular workflow in a process. dramatically improved” [I]. "End users are more satified when usability and performance are improved." (C) "The communication between developers and business users is much more fluent with drawing of business processes, rather than UML." (H)

AD, H-AD, I-AD

AB, F-AB1, H-AB1, H-

AB2

Ben-F

Homogenization of the development and maintenance of a family of applications by defining procedures and methodologies.

For software architects, this homogenization implies other two benefits: it allows a higher control over what it is being done by applications builders (supporting distributed teams in different locations), and it helps creating a corporate style integrating all applications.

“The SRA offers procedures and a methodology about how to make applications” [C]. “Homogeneity helps to have a distributed team in different locations” [E]. "Determine the way of developing applications. Architectural decisions are already made and the architecture indicates how to work." (A) "Procedures and methodology about how to make applications." (C) "We have an unique desktop in which all aplications are integrated. Either applications based on the architecture or legacy applications, are integrated in the main frame as a new window. Therefore, the user has the feeling that he/she is using only one applications, although there are many." (C) "It creates a suite, instead of a set of applications. All applications are integrated with single sign-on." (G)

A, B, C, E, F, G

6

E-AD, G-AD

2 D-AB

1

Ben-G

Increased reliability of SRA’s software elements that are common for a set of applications

The SRA consists of one supportive framework that has been tested and matured, with the reliability that it implies. The respondents also indicated the appropriateness of investing time in common software elements that will be reused. This is especially appreciated in cross-cutting elements (e.g., persistence and logging modules) that appear in all applications and are time-consuming without SRAs.

“SRA software elements have been tested and matured, what implies reliability [F]. "The architecture is constanly evolutioning, improving its performance and updating its technologies." (C)

A, C, E, F, H

5

A-AD, B-AD, D-

AD,

3 B-AB

1

Ben-H

Others benefits. In this category we include the benefits not mentioned by any stakeholder type more than once. Software architects indicated: application of best practices; easy distribution of the architecture through the web; support for application builders in case of problems. Architecture developers indicated: improved decision-making; reduced license costs; ability of incorporating more functionality to applications. Application builders indicated: improved agility when requirements are changed; improved decision-making; better documentation of SRAs.

“We used good practices like remove 'dead code' and wrappers of the SRA” [F]. “The SRA can be found on the Web, what saves distribution costs” [H]. “If the application builder has problems, there is a support team that helps them and solves problems if necessary” [F].

H, F

2

A-AD, G-AD, I-AD

3 D,-AB, E-AB, I-AB

3

Table 8. Detailed results about how shorter time-to-market is achieved. Code Category name Category description Representative quotes Respondents

SA Frequency SA

Tim-A

The reuse of software elements

The reuse of software elements (either cross-cutting modules or services that implement business logic)

"Time-to-market was mainly reduced because of the transverse modules of the architecture (i.e., persistence, security, logging, etc.)." (A) "Common services (such as sms and mail services) were implemented in each application. By implementing them only once for reuse, time-to-market was reduce drastically." (G) "In a SOA, if you achieve to create services that could be reused, you start to experience cost savings. However, in the beginning there is no service with business logic to reuse." (I)

A, E, F, G, I 5

Tim-B

Agile and automatized development

Some tasks of the development are automatically done by the architecture.

"The development is fairly automatized, what increase productivity and agility in the development of applications." (C) "It is easy-to-use, for instance generating a workflow is very simple and fast." (D) "Time-to-market is reduced because of the automatization in developments. Developers do not need to think in neither creation of objects that communicate with the database, services in the business layer are automtically created, the structure of presentation screen are predefined, and so on." (H)

C, D, H, I

4

Tim-C

Improved configuration of software elements

Improved configuration of software elements (e.g., setting up the modules to be reused by the application is fast)

"To use a module or connector, you only need to set it up in one file. It is fast and besides there are standard configuration files in the wiki." (E)

D, E, F

3

Tim-D

Technological and architectural decisions were already taken

Technological and architectural decisions were already taken (i.e., it saves time during the architecture design of every new application and improves reliability)

"Every application has a software architecture. With an architecture-centric approach, you gain the required amount of time to design the architecture and more reliability in the applications. You can invest more time with an architecture-centric approach, what improves reliability and reduce errors." (A)

A, E 2

Table 9. Detailed results for the main benefits of using SRAs for everis. Code Category name Category description Representative quotes Resp.

SA Freq. SA

Resp. AD Freq. AD

Ven-A

The consulting company harvests experience for prospective SRA projects.

The main reason is that requirements are very similar between client organizations. Some respondent estimated a 90% of architectural knowledge reused.

“The reference model receives continuous feedback from SRA projects” [G]. "It is oriented to reuse knowledge. If we know that a security module or availability issues have been previously solved, we are going to ask about them. I think knowledge is the key." (B) "All is gained experience that could be applied in other projects. It is internal knowledge for Everis." (C) "Knowledge from cross-cutting modules is aplicable to other projects. The requirements are very, versy similar for client organizations. Therefore, 90% of architectural knowlegde is reusable. Although we do not need to reinvent the wheel in every project, the reference model is being improved with new technologies" (H)

B, C, F, G, H, I

7

B-AB, C-AB, D-AD, G-

AD, H-AD

5

Ven-B

Reusing architectural knowledge can speed up prospective SRA projects and reduce time-to-market

Reusing architectural knowledge can speed up prospective SRA projects and reduce time-to-market, for instance, by greatly reducing their planning and development time.

“It is a differential factor because it greatly reduces the planning and development times of the SRA, and hence the costs [H]. "It is an accelerator in the development of information systems, reduces costs and errors, facilitates the maintenance, contribute to reuse… and most of it is useful for any business domain. It could be customized with business users." (E)

E, F, H, I

4

C-AD, D-AD

2

Ven-C

They gain reputation for prospective client organizations and gain organizational competence.

It is a good point for everis to be able to announce themselves as SRA providers of a big bank or insurance company.

“It gives prestige to announce yourself as the provider of successful SRAs” [D]. "Once you define the architecture for a client organization, you have more options of developing applications on the top of that architecture, since you have knowledge about that architecture." (H)

D, H, I

3

0

Ven-D

Previous experience reduces the risks in future projects because a “to-be” model exists.

This model can be used in all projects that do not have a very specific architecturally-significant requirement.

“It reduces the risk of projects because a 'to-be' model exists. This model can be used in all projects that do not have a very specific architectually significant requirement” [F]. "It allows us to adjust much more the schedule, the effort that we will charge the client organization for an architectural project and being more efficient." (H) "It gives certain guarantees of success." (I)

F, H, I

3

0

Ven-E

It provides a shared architectural mindset that makes projects less dependent on particular architects.

Share a same "way of thinking" in the software company.

“The reference model allows architects to have the same way of thinking and working in the software company [B]”.

B

1

0

Ven-F

It makes tacit knowledge explicit in the reference model.

Some tool support (e.g., wiki technologies) helps in managing this knowledge.

“Architectural knowledge from reference model is explicitly available in a wiki” [G]. G

1

B-AD 1

Table 10. Detailed results for the main drawbacks of using SRAs in client organizations. Code Category name Category description Representative quotes Resp.

SA Freq. SA

Resp. AD

Freq. AD

Resp. AB Freq. AB

Dra-A

Additional high or medium learning curve for using the SRA.

Stakeholders need to learn to develop with the architecture. Although it is based on standards and de-facto technologies, there are extra features that need to be learnt.

“The SRA is very specific. Although SRA is based on standards and de-facto technologies, there are extra features that need to be learnt” [B]. "Architecture requires the knowledge in all the layers, not only in the business layer." (E) "Although the organization had experts in Oracle Forms, they did not have knowledge about developing Java applications." (H)

B, C, E, F, H

5

C-AD, D-AD, G-AD, H-AD

4 A-AB, B-AB, C-AB,

D-AB, E-AB, F-AB2, H-AB1, H-AB2, I-AB

9

Dra-B

Limited innovation by giving prescriptive guidelines to create applications.

“Rare” applications will seldom be developed. “The SRA homogenizes and standardizes. Therefore, it implies less room for inno-vation. For instance, an application very 'rare' will never be developed” [A]. "The architecture restricts the development, and indicate how to do it. Thus, some developer would prefer not to use some part." (A)

A, D

2

C-AD 1 C-AB, D-AB, E-AB, F-AB1, H-

AB1

5

Dra-C

Applications’ dependency over the SRA.

When applications have requirements that the architecture does not offer yet, applications development is stopped until the SRA implements these requirements.

“When applications have requirements that the SRA do not offer yet, there are dependencies. Until the SRA will satisfy them, applications development is blocked. Also, there could be bugs in these architectural changes that affect the applications. This can be solved with a good methodology and coordination” [C].

B, C, D, E

4

G-AD, I-AD

2 - 0

Dra-D

Complexity. Architecture developers and application builders mentioned that the use of the SRA is complex.

No software architect mentioned this drawback. - 0

D-AD, E-AD

2 C-AB, F-AB1

2

Dra-E

None. Some of the responders indicated that the adoption of SRAs does not present any drawback.

No software architect mentioned this drawback. - 0

A-AD 1 A-AB, H-AB2, I-AB

3

Dra-F

Wrong decisions about the technologies to be used in all the applications.

Wrong decisions about the technologies to be used in all the applications (e.g., adopting technologies that were not mature enough to be productive).

“We included an ESB that was not mature enough to be productive” [G]. "The used tool was very limited for the development of new graphical interfaces." (I)

G, I

2

- 0 - 0

Dra-G

Other drawbacks. In this category we include the drawbacks that were not mentioned more than once by any stakeholder. Software architects indicated: difficulty to measure the time-to-market reduction due to the SRA; time-to-market ultimately depends on the skills of the application builder; initial investment in the SRA. Architecture developers indicated: SRA maintenance; initial investment over the SRA. Application developers indicated: some technologies are not up-to-date; there are conflicts between technologies.

“The SRA allows having the structure of the application and a few screens working in one day, but it always depends on the applications builders and the business logic that they put inside" [F]. "The goal that everyone pursue is to measure how many hours are saved by developing an application with an architecture-centric approarch agains developing it without it. It is a quimerical objective. I believe that it reduces the time to market without any doubt. But I do not know how much, and I doubt that someone does." (A)

A, F

2

D-AD, F-AD

2 F-AB2, H-AB1

2

Table 11. Detailed results about the artifacts used in the training for using the SRA. Code Category name Representative quotes Respondents Frequency

Tra-A

User manuals or documentation about how to use the SRA

"There was a development guide, which was a large document that explained in detail how to develop each part of an application." (H)

B, C, D, F, G, H

6

Tra-B

Practical workshops to the application development team

"When there is a new version of the architecture, we perform presentation session for application developers explaining new features." (B) "We carried out training sessions divided by parts of the architecture (i.e., presentation and services) and focus on how to develop an application since the beginning with demos." (C)

B, C, D, E, F, G

6

Tra-C

Training sessions and follow-up meetings for the project managers of the client organization

"Every week, we presented the progress to the architectuer or developme t managers fo the client." (G) "It includes architectural concepts and an introduction." (D) "It was divided into the different parts of the architecture." (C)

C, D, G, H, I 5

Tra-D

Description document of the architecture "We did .ppt documents from a generic description to a detailed description of the architecture." (C) C, H 2

Tra-E A wiki with material (e.g., how-to guides, configuration files) to support application builders

"After the practical workshop, developers support by wiki (level 1) and vy the support office (level 2)." (E) D, E 2

Tra-F Support office and service "After the practical workshop, developers support by wiki (level 1) and vy the support office (level 2)." (E) "A support office was available in case of incidents." (E)

E, F 2

Tra-G

Continuous training when in the SRA team there were also stakeholders from the client organization

"The client was involved since the beginning by joining two architects of the client to the team. Therefore, there was a continous knowledge transfer." (G)

G, I 2

Table 12. Detailed results for the improvements that stakeholders would do. Code Category name Category description Representative quotes Resp.

SA Freq. SA

Resp. AD

Freq. AD

Resp. AB Freq. AB

Imp-A

Add functionality or modules to the SRA.

For instance, one interviewee suggested developing a visual plugin to make easier the development of components and automatize more the job for application development teams.

“Our aim is to foster a visual plugin that makes easier the development and automatize more the development of components for the application developers” [C].

C, D, E, F, G, H

6

A-AD, B-AD,

G-AD, I-AD

4 A-AB, B-AB, E-AB, F-AB1, F-AB2, H-AB1, H-

AB2, I-AB

8

Imp-B

Change of technology because current one is not enough mature or appropriate, or it needs an upgrade to the latest version.

For example, more up-to-date BPM engines, or migration from JSF to allow for mobile technologies;

“We are limited with JSF, migrating to other framework, like Sencha, would allow offering presentation not only for web browser, but also mobile devices” [G]. "Where we have had more problems is in the selection of the workflow engine. We still need to check the state-of-the-art of BPMs opensource engines and then probably change it." (A) "We are using the BPM of Oracle. When we adopted it, it was in an initial state of maturity. Oracle had another BPM, they have integrated both and now have a much more powerful engine." (I) "We need to migrate to the new versions of JSF and Spring." (D)

A, B, D, G, I

5

D-AD, H-AD

2 D-AB 1

Imp-C

Simplify modules that cover too much functionality.

“There is a module that is more complete that what the project is really asking for. Thus, it is not aligned with business needs and therefore it should be simplified” [E].

E

1

C-AD, F-AD

2 - 0

Imp-D

Add new practices or procedures to the architecture.

For instance, moving to a continuous integration approach.

“The main point is to move to continuous integration” [H].

C, H

2

- 0 - 0

Imp-E

Migrate from legacy systems.

“There are very old software elements that we have inherited. It would be good to update them because you do not know them well since they were inherited” [B].

B

1

- 0 - 0

A.3 DATA ANALYSIS: CLUSTER ANALYSIS Clustering allows a user to make groups of data to determine patterns from the data.

Clustering has its advantages when the data set is defined and a general pattern needs to be

determined from the data. To make clusters, we used the WEKA software

(http://www.cs.waikato.ac.nz/ml/weka/) as it is explained in this IBM developerWorks tutorial:

http://www.ibm.com/developerworks/library/os-weka2/. The interested reader can create

these clusters with WEKA from the raw data available at:

http://www.essi.upc.edu/~smartinez/files/ist16-data.xlsx

Cluster analysis 1: Which types of stakeholders have more experience? We made three clusters for the attributes Role, YearsExperienceRole, ExperienceSRALikert. Each cluster shows us the experience of each type of stakeholder, from which we can begin to draw some conclusions:

Cluster 0: This group has the 10 application builders (AB). ABs have an average of 0.775 years of experience and 2.1 value of experience in a Likert scale from 1 to 5.

Cluster 1: This group has the 9 software architects (SA). SAs have an average of 2.8889 years of experience and 5 value of experience in a Likert scale from 1 to 5.

Cluster 2: This group has the 9 architecture developers (AD). ADs have an average of 1.2778 years of experience and 3.1111 value of experience in a Likert scale from 1 to 5.

Conclusion: Software architects have more experience in SRA projects than architecture developers and application builders.

Cluster#

Attribute Full Data 0 1 2

(28) (10) (9) (9)

=================================================================

Role AB AB SA AD

YearsExperienceRole 1.6161 0.775 2.8889 1.2778

ExperienceSRALikert 3.3571 2.1 5 3.1111

Cluster analysis 2: Do SRA projects in banking require more effort than SRA projects in other application domains? We made two clusters for the attributes Domain, Persons-Month, Years, TotalHours:

Cluster 0: This group has the 6 participants from banking SRA projects. We can see an average of 10 persons-month, 4 years, and 67,000 hours for banking SRA projects.

Cluster 1: This group has the 22 participants that do not belong to banking SRA projects. We can see an average of 4.75 persons-month, 1.26 years, and 11,182 hours for non-banking SRA projects.

Conclusion: SRA projects in banking require more persons-month, years, and hours of work. We could see that in SRA projects from the banking domain the effort invested was higher, both in terms of persons-month and duration of the projects (i.e., years).

Cluster#

Attribute Full Data 0 1

(28) (6) (22)

================================================

Domain Other Banking Other

Persons-Month 5.875 10 4.75

Years 1.8475 4 1.2605

TotalHours 23142.8571 67000 11181.8182

http://www.cs.waikato.ac.nz/ml/weka/

http://www.ibm.com/developerworks/library/os-weka2/

http://www.essi.upc.edu/~smartinez/files/ist16-data.xlsx

Cluster analysis 3: Is maintenance and learning curve more costly in banking SRA projects than in other SRA projects? We made the same two groups as in Cluster analysis 2 above, and we added two attributes: Ben-B and Dra-A. The output is telling us how each cluster comes together, with a "1" meaning everyone in that cluster shares the same value of one, and a "0" meaning everyone in that cluster has a value of zero for that attribute. Numbers are the average value of everyone in the cluster.

Cluster 0: This group has the 6 participants from banking SRA projects. Improved maintainability and reduced maintenance costs (Ben-B) has an average value of 0.5 and Additional high or medium learning curve for using the SRA (Dra-A) has an average value of 0.8333.

Cluster 1: This group has the 22 participants that do NOT belong to banking SRA projects. Improved maintainability and reduced maintenance costs (Ben-B) has an average value of 0.8636 and Additional high or medium learning curve for using the SRA (Dra-A) has an average value of 0.5909.

Conclusion: In the banking domain, maintenance costs (Ben-B) and the learning curve (Dra-A) of SRA projects were higher in comparison to the other application domains.

Cluster#


(28) (6) (22)

================================================

Domain Other Banking Other

Persons-Month 5.875 10 4.75

Years 1.8475 4 1.2605

TotalHours 23142.8571 67000 11181.8182

Ben-B 0.7857 0.5 0.8636

Dra-A 0.6429 0.8333 0.5909

Cluster analysis 4: Which SRA benefits and drawbacks are present in the public sector domain? We made two clusters for the attributes Domain, Ben-C, Ben-D, Ben-F, Dra-B, Dra-C:

Cluster 0: This group has the 17 participants that mostly do not belong to SRA projects

from the public sector.

Cluster 1: This group includes 11 participants from SRA projects from the public sector.

Conclusion: In the public sector domain, standardization (Ben-F) was perceived more often as a benefit whereas easier development (Ben-C), the use of latest technologies (Ben-D), limitation (Dra-B), and dependency over the SRA (Dra-C) was perceived in a higher way in the rest of application domains.

Cluster centroids:

Cluster#


(28) (17) (11)

==========================================================

Domain Other Other Public sector

Ben-C 0.7143 0.8235 0.5455

Ben-D 0.5357 0.6471 0.3636

Ben-F 0.3214 0.1765 0.5455

Dra-B 0.2857 0.4706 0

Dra-C 0.2143 0.1176 0.3636

Cluster analysis 5: What is the relationship among “Change of SRA technologies (Imp-B)”, “Update to the latest technologies (Ben-D)” and “Wrong decisions (Dra-F)”? We made two clusters for the attributes Imp-B, Ben-D, Dra-F:

Cluster 0: This group has the 23 participants that mostly strongly highlighted the

“update to the latest technologies” (Ben-D) as a benefit and did not mention “wrong

decisions” (Dra-F) as a drawback. These participants also indicated in a very low extent

the “change of SRA technologies” (Imp-B) as an improvement.

Cluster 1: This group includes 5 participants that did not mention the “update to the

latest technologies” (Ben-D) as a benefit and 2 of them mentioned “wrong decisions”

(Dra-F) as a drawback. All these participants also indicated the “change of SRA

technologies” (Imp-B) as a desired improvement.

Conclusion: Participants from SRA projects that did not use the latest technologies

tend to see the selection of such technologies as a wrong decision and are willing to

make changes in the technologies recommended by the SRA.

Cluster#


(28) (23) (5)

============================================

Ben-D 0.5357 0.6522 0

Dra-F 0.0714 0 0.4

Imp-B 0.2857 0.1304 1

Cluster analysis 6: Is standardization more strongly perceived by experienced stakeholders than stakeholders with less experience? We made two clusters for the attributes YearsExperienceRole, ExperienceSRALikert, Ben-F.

Cluster 0: This group has the 19 participants that did not mentioned standardization as

a benefit. These participants have an average of 1.0658 years of experience and

2.7368 value of experience in a Likert scale from 1 to 5.

Cluster 1: This group has the 9 participants that mentioned standardization as a

benefit. These participants have an average of 2.7778 years of experience and 4.6667

value of experience in a Likert scale from 1 to 5.

Conclusion: Standardization is more perceived by experienced stakeholders (i.e.,

software architects, see Cluster analysis 1).

Cluster#


(28) (19) (9)

======================================================

YearsExperienceRole 1.6161 1.0658 2.7778

ExperienceSRALikert 3.3571 2.7368 4.6667

Ben-F 0.3214 0 1

Cluster analysis 7: Is there a correlation between evolution SRA projects and the perceived need of adding functionality or modules to the SRA (Imp-A)? We made two clusters for the attributes Evolution, Imp-A.

Cluster 0: This group has the 19 participants from mostly design (no evolution) SRA

projects that asked in a high extent for adding functionality or modules to the SRA

(Imp-A).

Cluster 1: This group has the 9 participants that belong to evolution SRA projects and

asked in a low extent for adding functionality or modules to the SRA (Imp-A).

Conclusion: This means that typically SRAs are not definitive, and they are always

evolving, even after years of evolution.

Cluster#


(28) (19) (9)

============================================

Evolution 0.5714 0.3684 1

Imp-A 0.6429 0.8421 0.2222

Cluster analysis 8: Did the participants who asked for changing technologies (Imp-B) mention the use of latest technologies (Ben-D)? We made two clusters for the attributes Ben-D, Imp-B.

Cluster 0: This group has the 23 participants mostly mentioned as a benefit the use of

latest technologies but rarely asked for changing technologies.

Cluster 1: This group has the 5 participants who did not mention as a benefit the use of

latest technologies but asked for changing technologies.

Conclusion: There is an inverse relationship between Imp-B and Ben-D in our data.

Cluster#


(28) (23) (5)

============================================

Ben-D 0.5357 0.6522 0

Imp-B 0.2857 0.1304 1

Cluster analysis 9: SRA benefits as seen by different roles. We made three clusters for the attributes Role, and all the benefits, in order to see the interests from each type of stakeholders. The output is telling us how each cluster comes together, with a "1" meaning everyone in that cluster shares the same value of one, and a "0" meaning everyone in that cluster has a value of zero for that attribute. Cluster#


(28) (12) (9) (7)

=======================================================

Role AB AB SA AD

Ben-A 0.8214 0.75 0.8889 0.8571

Ben-B 0.7857 0.9167 0.6667 0.7143

Ben-C 0.7143 0.6667 0.6667 0.8571

Ben-D 0.5357 0.8333 0.3333 0.2857

Ben-E 0.4286 0.4167 0.1111 0.8571

Ben-F 0.3214 0.0833 0.8889 0

Ben-G 0.3214 0.3333 0.4444 0.1429

Cluster analysis 10: SRA drawbacks as seen by different roles. We made three clusters for the attributes Role, and all the drawbacks, in order to see the interests from each type of stakeholders. The output is telling us how each cluster comes together, with a "1" meaning everyone in that cluster shares the same value of one, and a "0" meaning everyone in that cluster has a value of zero for that attribute. Cluster#


(28) (8) (9) (11)

=======================================================

Role AB AD SA AB

Dra-A 0.6429 0.375 0.5556 0.9091

Dra-B 0.2857 0 0.2222 0.5455

Dra-C 0.2143 0.25 0.4444 0

Dra-D 0.1429 0.25 0 0.1818

Dra-E 0.1429 0.125 0 0.2727

Dra-F 0.0714 0 0.2222 0

Documents

ANNEX A - UPC. Universitat Politècnica de Catalunyasmartinez/files/ist16-attachment.pdfCertificates Yes List of certificates: SCJP, SCWCD, SCEA, SSCSA, SSCA, PMP, OCA, OCP, OCM, CSA,