Improving Requirements Glossary Construction via Clustering

Improving Requirements Glossary Construction via Clustering:

Approach and Industrial Case Studies!

Chetan Arora1 Mehrdad Sabetzadeh1 Lionel Briand1 Frank Zimmer2

1 University of Luxembourg, Luxembourg!2 SES TechCom, Luxembourg

.lusoftware verification & validationVVS

NL Requirements• NL requirements can be vague and ambiguous

• Multiple interpretations of same words - scale

• Inconsistent terminology

• Multiple terms for same concepts

• element / component / object!

• Multiple representations of same keywords

• status of Ground Station Interface component!

• Ground Station Interface component’s status!

• Interface component status

2

• Glossaries help mitigate ambiguities

• consistent terminology

• improves communication - especially in scenarios with multiple stakeholders

Requirement Glossaries

3

Glossary Building Process• Identify / Search for relevant glossary terms

!

• Define these terms, along with related terms

!

• Ideally, glossary should be built along with requirements specification, but reality?

4

Automated Term Extraction

JATE

TextRank

TOPIA

TermRaider

5

“Wait, my colleagues have used so many

variations of the same terms”

“Even I’ve used different variations myself. Let me

correct them all in the document.”

“Let’s identify the glossary terms!!!”

Story Behind

“How about we cluster all of them together”

6

Identification of

Candidate Terms

Similarity Calculation Clustering

Candidate Terms

2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

Similarity Matrix

Identification of

Candidate Terms

Similarity Calculation Clustering

Candidate Terms

2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

Similarity Matrix

Identification of

Candidate Terms

Similarity Calculation Clustering

Candidate Terms

2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

Similarity Matrix

Identification of

Candidate Terms

Similarity Calculation Clustering

Candidate Terms

2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

Similarity Matrix

Approach

Similarity Measure

ClusteringParameter(s)

NL Requirements

Combination and Filtering Heuristics

Clusters

7

Identification of Candidate Terms Similarity Calculation ClusteringIdentification of Candidate Terms Similarity Calculation ClusteringIdentification of Candidate Terms Similarity Calculation Clustering

R1 - STS shall supply GSI monitoring information (GSI input parameters and GSI output parameters) to the STS subcontractor.

R2 - When GSI component’s status changes, STS shall update the progress of development activities.

R1 - STS shall supply GSI monitoring information (GSI input parameters and GSI output parameters) to the STS subcontractor.

R2 - When GSI component’s status changes, STS shall update the progress of development activities.• STS • STS Subcontractor

• GSI • GSI input parameter • GSI output parameter

• GSI component • GSI component’s status • GSI monitoring information

• development activity • progress of development activity

0.85

8

Identification of Candidate Terms Similarity Calculation Clustering

Empirical Evaluation

9

Case StudiesCas

e-A

Case-B

380 Requirements

110 Requirements

10

RQ1 - How accurate is our approach at identifying glossary terms?

Term Extraction Accuracy

11

JATE

TextRank

TOPIA

TermRaider

Clustering Accuracy!

RQ2 - Which similarity measure for candidate terms yields the most accurate clusters?

12

Tuning the Clustering Algorithm!

RQ3 - How can one systematically specify the number of clusters for clustering?

13

Clustering Effectiveness!

• RQ4 - Is clustering effective at grouping clustering terms?

• Comparison of generated clusters against Ideal clusters

14

Inferring Ideal Clusters• Difficult to objectively define ideal clusters

directly

• Systematic inference of ideal clusters using domain models

Initial Modelby Domain Expert

Model Completion by Researchers

Validation by Domain Expert

15

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

From Domain Models to ClustersVar(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Specialization Relation

Aggregation Association

Attributes

16

Overlapping

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Var(status):- status- status of GSI component- GSI component status

Var(availability):- availability- availability of GSI component

Ground Station Interface

(GSI) statusavailability

GSI Component

GSI Monitoring Information

GSI Anomaly

GSI Output Parameter

GSI Input Parameter

Accuracy Metrics for Clusters

Precision = |Common Elements| ___________________ |Generated Cluster|

Recall = |Common Elements| ___________________ |Ideal Cluster|

Precision = 4 / 9 !

Recall = 4 / 412 3

4

5 6 7

8

9

17

Ideal Clusters - Only for Evaluation

Scalability!

RQ5 - Does our approach run in reasonable time?

19

Results

20

How accurate is our approach at identifying glossary terms? (RQ1)

16.6

22.7

12.5 16.1

20.2

30

49

20.8 26

48

72 70.7 73.6

70.7

92.9

49 49

70.6

53

90.2

Case A Case A Case A Case A Case A Case B Case B Case B Case B Case B

JATE TextRank TermRaider Topia Our Approach JATE TextRank TermRaider Topia Our Approach

Precision Recall F-measure

Case-A Case-B

21

Which similarity measure yields the most accurate clusters? (RQ2)

22

SyntacticSoftTFIDFMonge ElkanLevensteinRandom

Fmeasure vs. Num_Clusters

Num_Clusters0 5 10 20 30 40 50

Fmea

sure

0.100.20

0.30

0.40

0.50

0.60

Graph Builder

SyntaticSoftTFIDFMonge ElkanLevensteinRandom

F-measure vs. Number of Clusters

Number of Clusters0 50 100 150 200 250 300

F-m

easu

re

0.05

0.15

0.25

0.35

0.45

Where(100 rows excluded)

Graph Builder

Case-B

Number of ClustersNumber of Clusters

Case-A

Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom

Fmeasure vs. Num_Clusters

Number of Clusters0 5 10 15 20 25 30 35 40 45 50

F-m

easu

re

0.100.150.200.250.300.350.400.450.500.550.600.65

Graph Builder

Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom

Fmeasure vs. Num_Clusters

Number of Clusters0 5 10 15 20 25 30 35 40 45 50

F-m

easu

re

0.100.150.200.250.300.350.400.450.500.550.600.65

Graph Builder

Syntactic measure:SoftTFIDFMonge ElkanLevensteinRandom

Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom

Fmeasure vs. Num_Clusters

Number of Clusters0 5 10 15 20 25 30 35 40 45 50

F-m

easu

re

0.100.150.200.250.300.350.400.450.500.550.600.65

Graph Builder

Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom

Fmeasure vs. Num_Clusters

Number of Clusters0 5 10 15 20 25 30 35 40 45 50

F-m

easu

re

0.100.150.200.250.300.350.400.450.500.550.600.65

Graph Builder

Syntactic measure:SoftTFIDFMonge ElkanLevensteinRandom

F-m

easu

re(%

)

F-m

easu

re(%

)

051525

35

45

10203040

6050

How can one systematically specify the number of clusters for clustering? (RQ3)

23

7000

8000

9000

10000

11000

12000

13000

BIC

0 5 10 20 30 40 50Num_Clusters

Overlay Plot

600000

700000

800000

900000

1000000

1100000

BIC

0 50 100 150 200 250 300Num_Clusters

Overlay PlotCase-A Case-B

Number of Clusters

BIC

Number of Clusters

Chosen # of clusters = 120 Chosen # of clusters = 23

5% marginfrom max BIC

5% marginfrom max BIC

Is clustering effective at grouping candidate terms? (RQ4)

Case-A Case-B90.8

65.5

89.1

66.8 76.2

38.9

70.7

52.5

82.9

48.8

78.8

58.8

Upperbound Actual Uppderbound Actual

Precision Recall F-measure

24

UpperboundUpperbound Actual Actual

RQ5 - Does our approach run in reasonable time?

!

• Case - A : 380 requirements ~ 26 minutes

• Case-B : 110 requirements: < 2 minutes

25

Future Work• Further enhance the accuracy using semantic

analysis

• More empirical studies

!

!

• Identifying relationships between different development artefacts

26

Conclusion

27

16.6

22.7

12.5 16.1

20.2

30

49

20.8 26

48

72 70.7 73.6

70.7

92.9

49 49

70.6

53

90.2

Case A Case A Case A Case A Case A Case B Case B Case B Case B Case B

JATE TextRank TermRaider Topia Our Approach JATE TextRank TermRaider Topia Our Approach

Precision Recall F-measure

Case-A

Case-B

380 Requirements 110 Requirements

7000

8000

9000

10000

11000

12000

13000

BIC

0 5 10 20 30 40 50Num_Clusters

Overlay Plot

600000

700000

800000

900000

1000000

1100000

BIC

0 50 100 150 200 250 300Num_Clusters

Overlay Plot

Case-A Case-B

Number of Clusters

BIC

Number of Clusters

Chosen # of clusters = 120 Chosen # of clusters = 23

5% marginfrom max BIC

5% marginfrom max BIC