Copyright © 2015 Raytheon Company. All rights reserved.

Customer Success Is Our Mission is a registered trademark of Raytheon Company.

Edward Dou

Christian Reascos

August 19, 2015

The Case for Automating System

Level Radar Verification Testing

Agenda Introduction to Complex Systems

Verification of Airborne Radars

Introducing Automation

Benefits / Cost Analysis

Conclusion

8/11/2015 2

Complex System “…how parts of a system give rise to the collective

behaviors of the system, and how the system interacts with

its environment.” [NECSI]

Interactions with complex systems are not

always predictable 8/11/2015 3

Airborne Radar Systems

8/11/2015 4

Parallel Processing

Various Jobs

Priority

Many competing tasks determined

by environmental data

Results Based

Limited Timeline

Pilot Interrupts

Verification Construct

8/11/2015 5

Verification

Test

New

Requirements

Verification

Collateral

Regression

Overall

Functionality

Check

Software

Stress Test

• Built-in Test

• A/A Mode Entry

• A/G Mode Entry

• …

• Race Conditions

• Boundary Bugs

• Exception Handling

• …

• Existing Requirements

• Performance Requirements

• …

• Functional Requirements

• Performance Requirements

• ICD Requirements

• …

Introducing Automation

8/11/2015 6

Automation

Tool

Operator

Action

Radar Response

Radar Performance Data

Pilot’s Control

(optional) Interim Panel Radar/Avionics HW

Simulated Threat

Environment

Real-Time

Statistics Automating Script

Automating Data Analysis

Example

8/11/2015 7

Range Quality

0 nmi 5% Good (>85%)

10 nmi 86%

20 nmi 93% Satisfactory (60%-85%)

30 nmi 89%

40 nmi 78% Bad (<60%)

50 nmi 65%

60 nmi 67%

Automating Script

Automating Data Analysis

Automating Tests

Functionality check – Functional Requirement

– Foundational Test

Software stress test – Stability & Reliability

– Staple Test

New requirements verification – Functional Requirement

– Performance Requirement

– Interface Requirement

Collateral regression – Existing Performance Requirement

8/11/2015 8

Test Characteristics Automation Benefit

Somewhat beneficial /

Potentially impactful

Very beneficial

Beneficial for

Performance Requirement

Very Beneficial

Benefits and Costs

8/11/2015 9

COSTS

BENEFITS

• Testing Speed

• Test Repeatability

• Test Coverage

• Ease of Troubleshooting

• Continuous Coverage

• Instant System Evaluation

• Faster Data Collection

• Consistent Data Analysis

• Material (upgrades)

• Script Development

• Automation Tool Development

• Integration

• Maintenance

• Training

• Time to Implement

Benefit Cost Analysis If B/C ≥ 1.0, accept the project as economically acceptable

If B/C < 1.0, the project is not economically acceptable

8/11/2015 10

Benefit

Est. Value

(1 Build)

Est. Value

(15 Builds)

Testing Time Saving $14,000 $210,000

Troubleshooting Time $3,200 $48,000

Reduced Rework $9,000 $132,000

Data Analsis Time Saving $2,400 $36,000

Reduced Resources $22,000 $54,000

TOTAL $50,600 $480,000

Cost

Est Dollar

Value

Material (including upgrades) $45,000

Script Development $12,000

Automation Tool Development $50,000

Integration $75,000

Maintanance $20,000

Training $8,000

Time to Implement (down time) $10,000

TOTAL $220,000

For the first build

B/C = (Total Benefits) / (Total Costs) = $50,600 / $220,000 = 0.23

For 15 builds

B/C = $480,000 / $220,000 = 2.18

* Example is for illustration purposes only -- values are representative

Programmatic Environment Business Benefits

– Time Saving

– Resource Allocation

– Product Quality

– Money Saving

– Customer Contentment

8/11/2015 11

Conclusion The benefits outweigh the costs

Technical benefits – Improvements over the current verification process

Business benefits – Quality, Time, and Money

Future Considerations – Intelligent simulation of pilot’s actions

– Learning system

8/11/2015 12

Automation saves time and improves the quality

of software updates in complex radar systems

References New England Complex Systems Institute. About Complex Systems. n.d. Web. 16 July

2015. <http://necsi.edu/guide/>

M. E. Davis, "A history of battlefield surveillance radar." Radar Conference (RadarCon),

2015 IEEE. IEEE, 2015.

Raytheon Company. Raytheon's Advanced Combat Radar (RACR). 20 Sep. 2012, Web. 15

Jun. 2015. <https://www.youtube.com/watch?v=RSVziY1Zwn8>

Raytheon Company. Raytheon F 16 Integrated Fighter Mission Scenario. 4 Feb. 2014,

Web. 15 Jun. 2015. <https://www.youtube.com/watch?v=SnDUJfsNrbo>

H. Kaur, S. Ahamad, and G. N. Verma. "Elements of Legacy Program Complexity."

International Journal of Research in Engineering and Technology. IJRET, 2015

J. Ladyman, J. Lambert, and K. Wiesner. "What is a complex system?." European Journal

for Philosophy of Science 3.1 (2013): 33-67.

H. S. Mir and A. Guitouni. "Variable Dwell Time Task Scheduling for Multifunction Radar."

Automation Science and Engineering, IEEE Transactions on 11.2 (2014): 463-472.

SE Handbook Working Group INCOSE, INCOSE Systems Engineering Handbook v. 3.2.2,

INCOSE, 2011.

D. Hoffman, “Cost Benefit Analysis of Test Automation,” Software Quality Methods LLC,

1999.

L. Blank, A. Tarquin, Engineering Economy, 5th ed, McGraw Hill, 2002, pp. 308 - 310.

8/11/2015 13