Upload
lionel-briand
View
300
Download
0
Embed Size (px)
Citation preview
Documented Requirements !are not Useless After All! !
!The Case for Supporting Change, !
Configuration, and Testing
Lionel Briand XP 2016, May 27th, 2016 Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg
Fundamentals of Agile Methods
2
• Individuals and Interactions over Process and Tools
• Customer Collaboration over Contracts • Working Software over Documentation • Responding to Change over Planning
Requirements in Agile Methods
3
• “Customer” on site • Whole development team involved in eliciting
requirements • Use of a “common” language, e.g., user stories • Test cases as “executable requirements” (TDD) • Minimize traceability • Prioritize requirements • Accommodate requirements changes
• Like all principles, these are meant to be used thoughtfully and adapted in context
These principles are often used !in a simplistic and dogmatic
fashion
4
Agile development, in practice, tend to underplay the role of documented requirements
5
Objectives of the Talk
6
• In reality, whether and how you write requirements is a trade-off
• When are documented requirements important?
• What are their main applications? • What form do they take in practice? • What are the challenges? • Project examples and lessons learned
7
Challenges
Natural Language
8
• Most requirements in practice are expressed using natural language
• Ambiguity and incompleteness • Trade-off between precision and cost • Current support is insufficient, especially for
large sets of requirements
Domain Knowledge
9
• All requirements depend, more or less explicitly, on domain knowledge
• Common concepts and terminology • Not always consistent among all stakeholders • Software engineers often have a superficial
understanding of the application domain they target
Change
10
• Requirements change frequently • Changes have side-effects on other
requirements, design decisions, test cases … • How do we support such changes in ways
that scale to hundreds of requirements?
Configuring Requirements
11
• Many software systems are part of product families targeting varying needs among multiple customers
• Requirements typically need to be tailored or configured for each customer
• Because of interdependencies among such decisions, this is often error-prone and complex.
Challenges
12
• Dealing effectively with natural language • Capturing domain knowledge • Dealing with frequent changes • Configuring requirements with customers in
product families • These challenges also apply to agile
development
13
Addressing the Challenges
14
Analyzing and Handling Natural Language Requirements
Context
15
Challenges
16
• Large projects in satellite domain (e.g., ESA) • Hundreds of natural language requirements • Three tiers of requirements • Many stakeholders • Requirements capture a contract • Requirements change
Research Objectives
Checking compliance
to boilerplates
Domain Model
extraction
Change Impact
analysis in requirements
17
Research Objectives
Checking compliance
to boilerplates
Domain Model
extraction
Change Impact
analysis in requirements
18
Compliance with Boilerplates
19
• Rupp’s boilerplate:
• As soon as an unplanned outage is detected
the Surveillance and Tracking Component shall notify the SOT operator at the local station.
• Automation required for compliance checking • No glossary, scalable parsing
[<When?><under what
conditions?>]
THE SYSTEM<system name>
SHALL
SHOULD
WILL
PROVIDE <whom?> WITH THE ABILITY TO
<process>
<process>
BE ABLE TO<process>
<object> [<additional details about the object>]
Approach and Results
20
• Natural Language Processing (GATE) • Text chunking • Boilerplates: RUPP and EARS • Boilerplates are expressed as BNF grammars
and then pattern matching rules • Four industrial case studies • Low number of false positives and negatives • Hundreds of requirements in a few minutes
Tool: RETA
21
GATE NLPWorkbench
ConformanceDiagnostics
(within GATE)Requirements
Lists of modals,conditional words,
ambiguous terms, etc.
LSTLSTLSTLSTRules for checking
templateconformance
JAPEJAPEJAPEJAPERules for checking
best practices
Glossary(optional)
JAPEJAPEJAPEJAPE
Requirements Analyst
Requirements Authoring &Management
http://sites.google.com/site/retanlp/
Research Objectives
Checking compliance
to boilerplates
Domain Model
extraction
Change Impact
analysis in requirements
22
Change Impact Analysis
23
• A change in requirements may lead to changes in other requirements
• Hundreds of requirements • No traceability • We propose an approach based on: (1) Natural
Language Processing, (2) Phrase syntactic and semantic similarity measures
• Results: We can accurately pinpoint which requirements should be inspected for potential changes
Example
• R1: The mission operation controller shall transmit satellite status reports to the user help desk.
• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.
• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.
24
Change
• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.
• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.
• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.
25
Challenge #1 Capture Changes Precisely
• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.
• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.
• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.
26
Challenge #2 !Capture Change Rationale
• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.
• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.
• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.
27
• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.
• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.
• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.
Challenge #2 !Change Rationale
Rationale:
R1: We want to globally rename “user help desk” R2: Avoid communication between “mission operation controller” and “user help desk” R3: We no longer want to “transmit satellite status reports” to “user help desk” but instead to “user document repository”
28
Solution Characteristics
• Account for the phrasal structure of requirements The mission operation controller shall transmit satellite
status reports to the user help desk document repository.
user help desk, Deleted user document repository, Added
• Consider semantically-related phrases that are not exact matches and close syntactic variations
29
Tool: Narcia
30 https://sites.google.com/site/svvnarcia/
Research Objectives
Checking compliance
to boilerplates
Domain Model
extraction
Change Impact
analysis in requirements
31
Domain Models A domain model is a representation of conceptual entities or
real-world objects in a domain of interest.
32
Motivation • In practice, domain models are not preceding the elicitation and writing of
requirements
• Representation of important domain concepts and their relations
• Facilitate communication between stakeholders from different backgrounds
• Help identify inconsistencies in terminology, etc.
• Support is required for building domain models
• A lot of information required to build a domain model is automatically extractable
33
Approach
The simulator shall continuously monitor its connection to the SNMP manager and any linked devices.
dobj
advmoddet
NP NP NP
NP
possdet
amod
conj_and
nsubj
prep_to
prep_to
1 2 3 4 5 6 7 8 9 10 11 12
1 1 2 3
4
13 14 15
aux
nndet
prep_toprep_to
wor
dde
pend
enci
es
depe
nden
cies
de
rived
by
Alg
. of F
ig. 8
(Sec
tion
4.2)
14
amod
153 4
advmodaux
VB
R4:
ref_to
6
dobj
nsubj
(identified directly by coreference resolution)
Figure 5: Results of dependency parsing for requirement R4 of Fig. 4(a).
(ROOT (S (NP (DT The) (NN simulator)) (VP (MD shall) (ADVP (RB continuously)) (VP (VB monitor) (NP (PRP$ its) (NN connection)) (PP (TO to) (NP (NP (DT the) (NNP SNMP) (NN manager)) (CC and) (NP (DT any) (VBN linked) (NNS devices)))))) (. .)))
R4: The simulator shall continuously monitor its connection to the SNMP manager and any linked devices.
(b)
(a)
Figure 4: (a) A requirement and (b) its parse tree.
Phrase structure parsing [18] is aimed at inferring thestructural units of sentences. In our work, the units of in-terest are noun phrases and verbs. A noun phrase (NP) isa unit that can be the subject or the object of a verb. Averb (VB) appears in a verb phrase (VP) alongside any di-rect or indirect objects, but not the subject. Verbs can haveauxiliaries and modifiers (typically adverbs) associated withthem. To illustrate, consider requirements statement R4 inFig. 4(a). The structure of R4 is depicted in Fig. 4(b) us-ing what is known as a parse tree. To save space, we donot visualize the tree, and instead show it in a nested-listrepresentation commonly used for parse trees.
Dependency parsing [29] is aimed at finding grammaticaldependencies between the individual words in a sentence.In contrast to phrase structure parsing, which identifies thestructural constituents of a sentence, dependency parsingidentifies the functional constituents, e.g., the subject andthe object. The output of dependency parsing is representedas a directed acyclic graph, with labeled (typed) depen-dency relations between words. The top part of the graphof Fig. 5 shows the output of dependency parsing over re-quirements statement R4. An example typed dependencyhere is nsubj(monitor{5},simulator{2}), stating that “simula-tor” is the subject of the verb “monitor”.
Syntactic parsing is commonly done using the pipeline ofNLP modules shown in Fig. 6. There are alternative imple-mentations available for each of the modules in this pipeline.The pipeline can therefore be instantiated in di↵erent waysusing di↵erent combinations of module implementations. Inour work, we instantiate the pipeline using the module im-plementations provided by the Stanford Parser Toolkit [24].
The first module in the pipeline is the Tokenizer, whichsplits the input text, in our context a requirements docu-ment, into tokens. A token can be a word, a number, or asymbol. The second module, the Sentence Splitter, breaksthe text into sentences. The third module, the POS Tagger,attaches a part-of-speech (POS) tag to each token. POStags represent the syntactic categories of tokens, e.g., nouns,
Tokenizer
SentenceSplitter
POS Tagger
Parser(Phrase Structure
+ Dependency)
NLRequirements
1
2
3
5
Named Entity Recognizer 4
StructureParse Tree NP NPVP
TypedDependencies
nsubjnn dobj
Annotations
CoreferenceResolver 6
Figure 6: Parsing pipeline.
adjectives and verbs.The fourth module,the Named-Entity Rec-ognizer, identifies en-tities belonging topre-defined categories,e.g., proper nouns,dates, and locations.The fifth and mainmodule is the Parser,encompassing bothphrase structure pars-ing and dependencyparsing. The finalmodule is the Coref-erence Resolver. This is an optional module whose functionis to find expressions that refer to the same entity. We con-cern ourselves with pronominal coreference resolution only,which is the task of identifying, for a given pronoun suchas “its” and “their”, the NP that the pronoun refers to.Fig. 5 shows an example of pronominal coreference resolu-tion, where the pronoun “its” is linked to the referenced NP,namely “the simulator”, via a ref to dependency.
4. APPROACH
NPs, VBs Dependencies
Process Requirements Statements
NLRequirements
Lift Dependencies to Semantic Units
nsubjnn dobj
DomainModel
Construct DomainModel
0..1 1..*
relation
ExtractionRules
1
23
Figure 7: Approach Overview.
Fig. 7 presentsan overview of ourdomain model ex-traction approach.The input to theapproach is an NLrequirements doc-ument and the out-put is a UML classdiagram. Below,we elaborate thethree main steps of our approach, marked 1-3 in Fig. 7.
4.1 Processing the Requirements StatementsThe requirements processing step includes the following
activities: (1) detecting the phrasal structure of the require-ments, (2) identifying the dependencies between the words inthe requirements, (3) resolving pronominal coreferences, and(4) performing stopword removal and lemmatization; theseare common NLP tasks, respectively for pruning words thatare unlikely to contribute to text analysis, and for trans-forming words into their base morphological form.Activities (1), (2), and (3) are carried out by the pipeline
of Fig. 6. From the parse tree generated by this pipeline foreach requirements statement, we extract the atomic NPs
4
34
Requirements to Design IA Requirements
Design
Objective: Identifying impact of requirements changes on SysML design
35
Motivating Scenario!
Example Change Requests:
• Change to R11: Change over temperature detection level to 147 C from 110 C
• Change to R12: Temperature range should be extended to -40/150 C from -20/120 C
text = "The CP controller shall provide temperature diagnostics."id = "R1"
«requirement»Temperature Diagnostics
text = "The CP controller shall detect temperatures exceeding 110 ºC."id = "R11"
«requirement»Over-Temperature Detection
text = "The CP controller shall be able to measure temperatures between -20 ºC and 120 ºC."id= "R12"
«requirement»Operational Temperature Range
36
:Digital to Analog Converter
:DC Motor Controller
«satisfy»
:Over-TemperatureMonitor :Diagnostics
Manager
Over-Temperature
Motordrive mode
Error: Diagnostics and Status
Signal Generation
…
Motor position
B1
B2B3
B4
«requirement»Over-Temperature
Detection(R11)
……
«requirement»Operational
Temperature Range(R12)
…
:Temperature Processor
«satisfy»
B5
B6
Temperature
Voltage(digitized)
TemperatureMotor speed
…
Motortorque
…
Block Diagram
37
Actually Impacted Elements
• Both changes are related to ``Temperature Diagnostics’’, but they impact two drastically different parts of the system
• Change to R11 impacts a decision statement in software
• Change to R12 impacts electronic voltage divider circuit (hardware) and a lookup table (software)
38
:Digital to Analog Converter
:DC Motor Controller
«satisfy»
:Over-TemperatureMonitor :Diagnostics
Manager
Over-Temperature
Motordrive mode
Error: Diagnostics and Status
Signal Generation
…
Motor position
B1
B2B3
B4
«requirement»Over-Temperature
Detection(R11)
……
«requirement»Operational
Temperature Range(R12)
…
:Temperature Processor
«satisfy»
B5
B6
Temperature
Voltage(digitized)
TemperatureMotor speed
…
Motortorque
…Impacted by R12
Impacted by R12
Impacted by R12
Impacted by R11
Impacted Blocks …!
39
Approach
• Algorithm to compute estimated impacted elements in SysML designs for a requirements change
• Combination of structural analysis, behavioral analysis, and natural language processing (model labels)
• Number of design elements to inspect (average): 4.8% of design elements that can possibly be impacted
• Only one missed impacted element across all changes
• Frequent changes: Significant savings and better quality assurance
40
41
Supporting Product Lines and Requirements Configuration
Context
International Electronics !& Engineering (IEE)
IEE develops real-time embedded systems: • Automotive safety sensing systems • Automotive comfort & convenience systems,
e.g., Smart Trunk Opener
42
Smart Trunk Opener (STO)
STO Provides automatic and hands-free access to a vehicle’s trunk (based on a keyless entry system)
43
IEE Requirements Engineering Use Case Driven
Development
Use Case !Diagram
Use Case! Specifications
Domain !Model
44
Dealing with Multiple Customers
45
STO Requirements from Customer A
(Use Case Diagram and Specifications, and Domain Model)
Customer Afor STO
modify modify
modify modify
STO Test Cases for Customer A
evolves to
(clone-and-own)
STO Requirements from Customer B
(Use Case Diagram and Specifications, and Domain Model)
Customer Bfor STO
evolves to
(clone-and-own)
STO Test Cases for Customer B
evolves to
(clone-and-own)
STO Requirements from Customer C
(Use Case Diagram and Specifications, and Domain Model)
Customer Cfor STO
evolves to
(clone-and-own)
STO Test Cases for Customer C
Product Line Approach
46
• A Product Line approach was clearly needed • Restricted and analyzable use case specifications
(NLP) • Variability modeling in use case diagrams and
specifications • Automated configuration guidance for configuring
requirements with each customer • Automated generation of product-specific use case
models based on decisions
Use Cases And Domain Model
Customer A for Product X
Product-Line Use Cases And Domain Model
Identify Commonalities and
Variabilities Configurator
Customer B for Product X
Use Cases And Domain Model
Customer C for Product X
Use Cases And Domain Model
configure evolves
reconfigure
evolves
reconfigure
reconfigure
reconfigure
47
Product Line Use Case Diagram for STO (Partial)
48
• RUCM is based on a (1) template, (2) restriction rules, and (3) specific keywords constraining the use of natural language in use case specifications
• RUCM reduces ambiguity and facilitates automated analysis of use cases
Restricted Use Case Modeling: !RUCM
49
• Flow of events is described in restricted natural language
RUCM
Basic Flow
1. INCLUDE USE CASE Identify System Operating Status.2. The system VALIDATES THAT the operating status is OK.3. The system REQUESTS the move capacitance FROM the UpperSensor.4. The system REQUESTS the move capacitance FROM the LowerSensor.5. The system VALIDATES THAT the movement is a valid kick.6. The system VALIDATES THAT the overuse protection feature is enabled.7. The system VALIDATES THAT the Overuse protection status is inactive.8. The system SENDS the valid kick status TO the STOController.Post condition: The gesture has been recognised and the STO Controller hasbeen informed.
50
• Keyword: INCLUDE VARIATION POINT: ... • Inclusion of variation points in basic or alternative flows of
use cases:
Use Case: Identify System Operating StatusBasic Flow1. The system VALIDATES THAT the watchdog reset is valid.2. The system VALIDATES THAT the RAM is valid.3. The system VALIDATES THAT the sensors are valid.4. The system VALIDATES THAT there is no error detected.Specific Alternative FlowRFS 41. INCLUDE VARIATION POINT: Storing Error Status.2. ABORT.
!Example Variability Extension
51
• Tool Support (PUMConf): https://sites.google.com/site/pumconf/
• Positive feedback from engineers, both about the modeling approach and configuration tool
• They confirmed they benefited from:
• Understanding the commonalities and differences across product requirements
• Automated guidance in a configuration that is often complex, i.e., many (interdependent) decisions
Results
52
53
Legal Compliance
Regulations - Requirements • Many (government) systems need to be
compliant with the law and remain so over time
• E.g., Tax systems • How can we help with making sure IT
engineers implement systems that comply with the law?
• How can we verify they do?
54
Solution Overview
Test cases
Actual software system
Traces to
Traces to
Analyzable interpretation
of the law Generates
Results match?
Impact of legal changes
Simulates
55
Domain Model
56
Art. 105bis […]The commuting expenses deduction (FD) is defined as a function over the distance between the principal town of the municipality on whose territory the taxpayer's home is located and the place of taxpayer’s work. The distance is measured in units of distance expressing the kilometric distance between [principal] towns. A ministerial regulation provides these distances.
Interpretation + Traces
Procedures as Activity Diagrams
57
The amount of the deduction is calculated as follows: If the distance exceeds 4 units but is less than 30 units, the deduction is € 99 per unit of distance. The first 4 units does not trigger any deduction and the deduction for a distance exceeding 30 units is limited to € 2,574.
Interpretation + Traces
Discussion • Models understandable by both legal experts and IT
specialists
• We addressed the gap between legal experts and IT specialists
• Modeling effort was considered reasonable given the life span of such eGovernment systems
• Traceability to the law was considered a significant asset given frequent and complex changes in the law
58
59
Requirements-Driven Testing
Context • Context: Automotive, sensor systems
• Traceability between system requirements and test cases
• Mandatory when software must comply with ISO 26262
• Customers also require such compliance
• Use-case-centric development TC4
TC3
TC2
Requirements!!
Test cases
TC1
60
Objectives • Automatically generate test cases from requirements
• Capture and create traceability information between test cases and requirements
• Requirements are captured through use cases
• Use cases are used to communicate with customers and the system test team
• Complete and precise behavioral models are not an option: too expensive (Model-based testing)
61
Strategy
• Analyzable use case specifications
• Automatically extract test model from the use case specifications
• Minimize modeling, domain modeling only
• No behavioral modeling
62
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
Errors.size() == 0 Status != null
t > 0 && t < 50
Generate Scenarios and
Inputs
6
Elicit Use Cases 1
Missing Entities
Specify Constraints 5
OCL constraints
Model the Domain 2
Evaluate Consistency
3 Domain Model RUCM Use Cases
Generate Test Cases
7
Test Cases Object
Diagrams Test
Scenarios Mapping Table
64
https://sites.google.com/site/umtgTestGen/
Toolset integrated with IBM DOORS and Rhapsody
65
Discussion
Applications
66
• Requirements to support a shared understanding among many stakeholders in large projects
• Requirements as contract with customers • Requirements to support communication
between software engineers and domain experts
• Requirements to support compliance with standards, e.g., traceability to tests
• Requirements to support change
Forms of Requirements
67
• Natural language statements, complying or not with boilerplates
• Use case specifications, possibly structured and restricted
• Models, e.g., class and activity diagrams • Agile practice: User stories are a basis for
communicating requirements but are not analyzable or complete – This is not always acceptable
The form of requirements depends on how one intends to apply them and many
contextual factors
68
Contextual Factors
69
• Regulatory compliance, e.g., standards • Project size, team distribution, and number of
stakeholders • Background of stakeholders and communication
challenges • Domain complexity • Presence of product lines with multiple customers • Importance of early agreement on and full compliance
with requirements (e.g., Contract, safety certification) • Frequency and consequences of changes in
requirements
Conclusions
70
• Documented requirements are useful after all! • The extent to which requirements and their traceability are
documented depends on multiple factors • They also come in different forms, mostly depending on
analysis requirements • Many challenges to be addressed by research:
(1) Ambiguity in Natural Language requirements (2) Automation of time consuming tasks: Scalability (3) Change impact
• NLP technology is now mature and provides many opportunities for automation and lowering the documentation overhead
The challenge is to find the right trade-off!in a given context
71
Acknowledgements
72
• Mehrdad Sabetzadeh • Arda Goknil • Shiva Nejati • Chetan Aurora • Ines Hajri • Ghanem Soltana • Chunhui Wang
Documented Requirements !are not Useless After All! !
!The Case for Supporting Change, !
Configuration, and Testing
Lionel Briand XP 2016, May 27th, 2016 Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg
Natural Language Requirements • [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP
Approach
• [TSE 2015] C. Arora et al., Automated Checking of Conformance to Requirements Templates using Natural Language Processing
• [ESEM 2014] C. Arora et al., Improving Requirements Glossary Construction via Clustering
Requirements-Driven Testing • [ISSTA 2015] C. Wang et al., Automatic Generation of System Test Cases from Use Case
Specifications
74
SysML Traceability and Safety Analysis • [TOSEM 2014] L. Briand et al., Traceability and SysML Design Slices to Support Safety
Inspections: A Controlled Experiment
• [IST 2012] S. Nejati et al., A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety Certification: Framework, Tool Support, and Case Studies
Legal Modeling and Analysis • [MODELS 2014] G. Soltana et al., UML for Modeling Procedural Legal Rule
• [SoSYM 2016] G. Soltana et al., Model-Based Simulation of Legal Policies: Framework, Tool Support, and Validation
75
Product Families and Configuration • [MODELS 2015] I. Hajri et al., Applying Product Line Use Case Modeling in an Industrial
Automotive Embedded System: Lessons Learned and a Refined Approach
• [SoSYM 2016] I. Hajri et al., A Requirements Configuration Approach and Tool for Use Case-Driven Development
Impact Analysis • [FSE 2016] S. Nejati et al., Automated Change Impact Analysis between SysML Models of
Requirements and Design
• [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP Approach
76