Upload
alessandra-bagnato
View
578
Download
2
Tags:
Embed Size (px)
DESCRIPTION
The EU funded FastFix FP7 project aims to provide a monitoring control platform for remote software maintenance. This presentation explains the formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. This process allows for a continuous integration and enhancement of the platform. As a result, the development process is steadily monitored and evaluated, and early feedback is provided to the designers, developers and architects. The feedback came from the two industrial end user partners: Txt e-solutions and Prodevelop and will help to improve the overall FastFix design methodology and the related tool set.
Citation preview
Work Plan FastFix Open-Source Formative Evaluation Process in Remote Software Maintenance Alessandra Bagnato, TXT e-solutions – Corporate Research Javier Cano and Christophe Joubert, Prodevelop
Industrial Track at RCIS 2012, Sixth International Conference on Research Challenges in Information Science 2012 “Be Fit, Fast and Alert when testing, maintaining or solving bugs of your Future Internet applications” FITTEST & FastFIX & ALERT & Io.Test Joint Workshop
May 18th 2012
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
2/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
3/49
1.- PROJECT OVERVIEW
European project FASTFIX (ICT- 258109, June 2010 - February 2013) develops a monitoring control platform for remote software maintenance. Within this project, we have been working on a formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed.
4/49
Project Overview
5/49
WP 1 Project Management and Quality Assurance WP 2 FastFix conceptual framework WP 3 Context observation and User modeling WP 4 Event correlation WP 5 Fault replication WP 6 Patch generation and self-healing WP 7 Performance, security and privacy
WP 8 Trials and validation WP 9 Dissemination and exploitation
Formative Evaluation is part of
WP8
Formative Evaluation
6/49
PURPOSE to ensure that the goals of the
under-development software
are being achieved and to improve the
software, if necessary, by means of
identification and subsequent
remediation of problematic aspects.
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
7/49
2.- FORMATIVE EVALUATION PROCESS
1. Analysis
2. Metrics
3. Tools
4. Testing
5. Monitoring
6. Reporting
8/49
FORMATIVE EVALUATION PROCESS
Analyze the project and
development process.
Select metrics to collect.
Select tools to perform metrics collection.
Define tests to execute. Define which tool will
collect what.
Define who is responsible for
monitoring.
Select the reports to create
from the data.
9/49
Analyze the project
Find the basic natures that define the project. In the case of FastFix :
Strong focus in research
Distributed teams
Development of new
Software Artifacts
Integration into common
platform
10/49
Analyze development process
Analyze the development process to use. Try and find points that are worth monitoring. In the case of FastFix :
Iterative process:
Requirements change as development advances.
Requirements and issues appear at each iteration.
Testing is to start early and happen often.
11/49
Select interesting metrics
Find interesting metrics to be gathered. In the case of FastFix :
tickets opened, solved, closed, overdue, etc.
committers activity, evolution of code base, regular activity.
builds and tests stability.
reports on ranked issues found in source code quality.
12/49
Find ‘invisible’ metrics
There exist metrics that indicate a good development of the project but are not directly visible :
Issues discussed
among partners.
Detection of issues before they promote to a problem.
Responses and
actuations on comments
from others.
Honoring deadlines.
13/49
Select tools that integrate into the development process. In the case of FastFix :
Monitorization target Monitoring tool
Source code quality. Static code analysis.
Components integration to form the
FastFix platform. Continuous integration.
Tests execution. Automated builds and tests
execution.
Platform integration in target
applications.
Integration trials after internal
releases.
Open and resolved issues. Issues statistics reports.
Points to hook the formative evaluation
14/49
Non-invasive tools to gather metrics
Automate the process of metrics gathering as much as possible :
Integration into existing tools.
Periodic jobs in build servers.
Periodic scripts that collect information.
Periodic scripts that generate reports.
Tools invisible until notifications are necessary
Automated tools collect metrics silently
15/49
Start testing early, test often
The best formative evaluation is the one that includes all stakeholders in the development : • Include end users as soon as possible. • Iterative testing with small increments. • Make release cycles short.
Project execution
Testing and Trials
M0. Project starts.
M18. Testing starts.
M33. Project ends. Internal Releases
16/49
Monitor evolution, watch for deviations
Monitor metrics gathered periodically (daily) :
Look for deviations
• I.e: continuous integration failing last week.
Take Corrective
actions
• I.e: review build system logs to correct errors.
Use automated
notifications
• I.e: set up automated notifications to responsibles of latest changes when build fails.
17/49
Create reports
Collect all metrics in a single report
• LOC
• # of tests
• Code quality
Highlight areas to improve
• Build stabilty
• Tests
Highlight good areas
• Code quality
• Dependencies management
18/49
Benefits
The overall benefit of the process is to
correct problems and to include new
requirements before the production
phase of the software project and during
the whole life of the project.
19/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
20/49
Development: sprints from backlog.
Produces an increment of the platform.
3.- AGILE DEVELOPMENT PROCESS (SCRUM)
24 h
Product backlog
Sprint backlog
Sprint
30 days
Product increment
21/49
Development: sprints from backlog.
Produces an increment of the platform.
Testing of produced platform. Gathering
of metrics.
Evaluation: Bug and enhancement issues
stored in backlog. Metrics analyzed.
24 h
Productbacklog
Sprintbacklog
Sprint
30 days
Productincrement
Evaluation definition
FastFixIntegration in target system
Test executionon FastFixed
system
Datacollection
Metrics aquisiton
Feedback
Formativeevaluation
Summativeevaluation
Continuousevaluation
Metricsaquisitionrevision
Changerequests
Newrequirements
Bugs
Development
Integration
Evaluation
24 h
Productbacklog
Sprintbacklog
Sprint
30 days
Productincrement
24 h24 h
Productbacklog
Sprintbacklog
ProductbacklogProductbacklog
SprintbacklogSprint
backlog
Sprint
30 days30 days
Productincrement
Productincrement
Evaluation definition
FastFixIntegration in target system
Test executionon FastFixed
system
Datacollection
Metrics aquisiton
Evaluation definitionEvaluation definition
FastFixIntegration in target system
FastFixIntegration in target system
Test executionon FastFixed
system
Test executionon FastFixed
system
Datacollection
Datacollection
Metrics aquisitonMetrics aquisiton
Feedback
Formativeevaluation
Summativeevaluation
Continuousevaluation
Metricsaquisitionrevision
Changerequests
Newrequirements
Bugs
FeedbackFeedback
FormativeevaluationFormativeevaluation
SummativeevaluationSummativeevaluation
ContinuousevaluationContinuousevaluation
Metricsaquisitionrevision
Metricsaquisitionrevision
ChangerequestsChange
requests
Newrequirements
Newrequirements
BugsBugs
Development
Integration
Evaluation
Agile development process
22/49
Agile development at a glance
A simplified view of the development process followed in FastFix:
Sprint planning
Development
Product increment
Testing
Evaluation
23/49
Specific tools
The formative evaluation hooks into the development process using standard tools:
Sprint planning
Development
Product increment
Testing
Evaluation
Code repository analysis
Code quality analysis
Continuous integration server
Metrics collection
Issues management
Trac Subversion
Sonar
Hudson
Jmeter
24/49
Development: sprints from backlog.
Produces an increment of the platform.
Testing of produced platform. Gathering
of metrics.
Evaluation: Bug and enhancement issues
stored in backlog. Metrics analyzed.
AGILE DEVELOPMENT PROCESS
25/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
26/49
4.- OPEN SOURCE SUPPORT TOOLS
• FOCUS ON OPEN SOURCE
• CRITERIA
• SELECTED TOOLS
27/49
Focus on Open Source
WHY OPEN SOURCE TOOLS ? As a research project with focus on Open Source: • Can deliver industrial level performance. • Free as in beer and speech. • Minimum investment. • Great customization. • Huge integration.
28/49
Criteria
Building and Testing
Communication channels
Quality
Issues
Project Management
Source Code
Alternatives
Consortium selects among all alternatives according to: • Capabilities of each tool • Maturity of each tool • Integration with other tools • Familiarity among partners
29/49
Subversion as source code and Maven as project management
Mature, well known source control management. Integrates with lots of other tools.
Mature, well known project management, specially in Java. Integrates with lots of other tools. Simplifies dependency management and build process.
Source Code
Project Management
http://subversion.tigris.org/
http://maven.apache.org/ 30/49
Hudson as continuous integration
Building and Testing
• Off-the-shelf Integration with Subversion and Maven.
• Notifications for failed builds.
http://hudson-ci.org/ 31/49
Sonar as quality analyzer
• Quality analysis with FindBugs.
• Integration with Hudson.
• Reports generation.
Quality
http://www.sonarsource.org/ 32/49
Trac as issue manager
• Integration with SVN.
• Notifications. • Milestone
planning. • Reports
generation.
Issues
http://trac.edgewall.org/ 33/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
34/49
GOAL
• Usability and performance of FastFixed systems not noticeably affected.
INVESTMENT
• FastFix invests part of the effort in tunning the performance of the platform.
PROCESS
• Each iteration release testing collects metrics to evaluate performance.
5.- PERFORMANCE REQUIREMENTS
35/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
36/49
6.- FASTFIX EVALUATION AT WORK
Samples from the deployed tools: • Hudson
• Sonar
• Trac
37/49
FastFix Formative evaluation at work: HUDSON
• HUDSON continuous integration system
schedules builds every night of the whole platform. Notifications in place for interested parties.
38/49
FastFix Formative evaluation at work: SONAR
• SONAR code analysis system analyses code just after every correct build.
SONAR Components
39/49
FastFix Formative evaluation at work: TRAC
• TRAC issues management system is used as a planning tool, to define milestones and their tasks, to manage responsibles for tasks and internal releases.
40/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
41/49
• Lines Of Code
• Developers
• Continuous integration
• Issues management
7.- METRICS UNTIL 15TH MAY 2012
42/49
Lines Of Code : 879742
Metrics until 15th May 2012 Lines Of Code
43/49
Developers: 18
Continuous integration: • Build stability:
Metrics until 15th May 2012 Continuous integration
44/49
Stable 80%
Failed 20%
Unstable 0%
Common
Stable 85%
Failed 0%
Unstable 15%
Client
Stable 75%
Failed 20%
Unstable 5%
Server
• Test stability:
Correct
Failed
Skipped
Issues management: • 282 tickets managed. • 170 closed tickets. • 112 open tickets.
• 29 tickets in the backlog • 6 tickets from internal testing in industrial partners
Metrics until 15th May 2012 Issues Management
45/49
Closed 60%
Other 28% Backlog
10%
Testing 2%
Active 40%
Metrics until 15th May 2012 Code analysis
• Code quality • 34.4% Sonar Rules Compliance Index
• 2547 violations [ 1 blocker, 147 critical, 1610 major, 732 minor, 57 info]
• 23.4% comments, 8.4% duplications, 100% test success
46/49
OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
47/49
8.- NEXT STEPS
Execution of trials in
industrial partners with
live production
applications.
Collection of all metrics data, both qualitative
and quantitative.
Reports that summarize
the evaluation of the project.
48/49
Thank you for your attention!
Alessandra Bagnato Corporate Research Division - TXT e-solutions S.p.A [email protected] Francisco Javier Cano and Christophe Joubert Prodevelop [email protected], [email protected] FastFix Project Web Site: https://www.fastfixproject.eu