July 2, 2015 Sam Siewert

SE420 Software Quality Assurance

Lecture 8 – I&T Test Cases

http://www.stsc.hill.af.mil/resources/tech_docs/gsam4.html

Reminders Assignment #4 Posted Today – I&T Focus

Remaining Assignments [Top Down / Bottom-Up] – #5 – Design, Module Unit Tests and Regression Suite – #6 – Complete Code, Refine and Run all V&V Tests and Deliver

Gcov and Lcov Command Summary for Reference and Use in Assignment #6 Follows …

Sam Siewert 2

Summary of Gcov Commands

Sam Siewert 3

%make clean rm -f *.o *.d *.exe sclogic *.gcov *.gcno *.gcda *.info %ls Makefile SC-Logic-2-LCOV-results SC-Logic-2-LCOV-results.zip sclogic.c << ALWAYS MAKE CLEAN FIRST>> %make cc -Wall -O0 -fprofile-arcs -ftest-coverage -g sclogic.c -o sclogic << DO NEW BUILD ON YOUR TEST MACHINE AND NOT PROFILE-ARCS and TEST-COVERAGE GCOV INSTRUMENTATION DIRECTIVES >> %./sclogic function_A function_B do function_C do function_D function_A do function_D … do function_C function_A do function_D function_A function_B do function_D <<RUN OF CODE TO BE TESTED WITH SOME PRINTF DEBUG OUTPUT HERE>>

%gcov sclogic.c File 'sclogic.c' Lines executed:100.00% of 29 sclogic.c:creating 'sclogic.c.gcov' <<RUN POST RUN COVERAGE ANALYSIS ON SOURCE FOR LAST RUN OF INSTRUMENTED CODE>> %gcov sclogic File 'sclogic.c' Lines executed:100.00% of 29 sclogic.c:creating 'sclogic.c.gcov' <<RUN POST RUN COVERAGE ANNOTATED TEXT GENERATION>> %cat sclogic.c.gcov -: 0:Source:sclogic.c -: 0:Graph:sclogic.gcno -: 0:Data:sclogic.gcda -: 0:Runs:1 -: 0:Programs:1 -: 1:#include <stdio.h> … 1: 41:int main(void) -: 42:{ … -: 52: // Test Case #2, Test use in logic 11: 53: for(testIdx=0; testIdx < 10; testIdx++) -: 54: { 10: 55: if((rc=(function_A() && function_B()))) 2: 56: function_C(); -: 57: else 8: 58: function_D(); -: 59: -: 60: } -: 61: 1: 62: return(1); -: 63:}

Summary of Lcov Commands

Sam Siewert 4

%lcov -t 'SC LOGIC 2 REPORT' -o sclogic.info -c -d . Capturing coverage data from . Found gcov version: 4.4.7 geninfo: WARNING: invalid characters removed from testname! Scanning . for .gcda files ... Found 1 data files in . Processing sclogic.gcda Finished .info-file creation <<RUN LCOV TOOL ON GCOV RESULTS>> %genhtml -o result2 sclogic.info Reading data file sclogic.info Found 1 entries. Found common filename prefix "/home/facstaff/siewerts/se420/src" Writing .css and .png files. Generating output. Processing file MCDC2/sclogic.c Writing directory view page. Overall coverage rate: lines......: 100.0% (29 of 29 lines) functions..: 100.0% (5 of 5 functions) % <<GENERATE THE WEB PAGES FROM LCOV RESULTS FOR BROWSING>>

Integration and Test Integrate Software Modules [units] and Hardware Components into Sub-systems Test Focus on Interfaces [Function, Message, Shared Memory, Hardware], Protocols, and Interoperability of Modules

Sam Siewert 5

Test Types Positive Tests – Functional Software Interface Tests

Functions calling Functions – API Message Passing – Local Message Queues, Network, Client-Server Shared Memory – Synchronization, Buffers

– Hardware Interface Tests Drivers and Device Interfaces Firmware [ROM Code, Run out of Reset]

Negative Tests – Software Interface Faults – Hardware Interface Fault Injection

Diagnostics [Built-in Self-Test] Unit Interoperability – Sub-system Resource Testing – Memory, CPU, I/O, Storage, Power – Protocols – Message Acknowledgement, Command/Response, Background

Commands, Peer-to-Peer, etc.

Performance Tests – Profiles and Traces

Sam Siewert 6

Familiarity with Sub-System Test Scenarios

RAID 10, 50 Sub-system [Logical Volume Data Protection] – XOR Encoding – Recovery of Lost Disk Drive (File or Segment) with XOR of Remaining – Normal Write and Read Operations – Read with Recovery on Fly [Degraded Mode] – Write During Degraded Mode – Lost Segment Rebuild

Encryption and Decryption of Messages [Message Encrypt/Decrypt]

– Simple Substitution – Transposition [Block Cypher] – Mathematical Hashing Function for Substitution or Transposition – Combinations of Methods – Reverse Operation to Decrypt

Image Processing Transformation [Frame Transformation]

– Load Image File to Buffer – Apply Pixel Operations to Buffer [X, Y, Channel] – Write Buffer to Image File – Preview Display to User – Computer Vision is Continuous Image Processing

Sam Siewert 7

Outline for Every Integration Test 1. Check out Specific Source Code Test Configuration – CMVC Tools, Git

– Collection of Modules [Units] Tagged by Revision Control – OR Current

2. Build and Link Modules (*.o) and Libraries (*.a) into Sub-system to Test

3. Load / Install Sub-system Code onto Test Hardware Platform of Known

Configuration – Record key hardware configuration parameters – E.g. for I/O HW config - lspci, lsusb, – General config - hwinfo – Linux OS kernel build config - uname –a – cat /proc/meminfo – cat /proc/cpuinfo

4. Run Integrated Test(s) [with Gcov, Lcov, Gprof] 5. Review of Expected Syslogs, Output to Terminal, for Each Feature 6. Review Performance Profiles 7. Track Bugs, Anomalies, and Disposition as Defects

Sam Siewert 8

GitHub and GitHub Desktop Primer Use CMVC from Now On – Install GitHub Desktop on Windows or Mac – Create GitHub Account - https://github.com/ – Code is Public for FREE Use – Clone from siewertserau or PRClab or Desktop

git clone https://github.com/siewertserau/Examples-RAID-Unit-Test.git You need my permission to push changes back, but anyone can pull

Sam Siewert 9

GitHub New Project Create Your Own SE420 Student Project Start GitHub Desktop, Download Code from Web, Drag Folder over to Create new repository Publish for Use by Others [on your team, public] Push to GitHub so Others can Clone on Your Team [Pull]

Sam Siewert 10

#1

#2 #3

PRClab I Requested Git client for Cloning and CMVC on PRClab See Cloning Path on GitHub Desktop / Web – View on GitHub

Sam Siewert 11

Grab clone path here

Clone to Desktop here

Get Archive here

I&T - Positive Tests Focus on Major Functions and Features of Sub-Systems – Validation [Did we Build Right Thing?]

Tracing to Sub-system Requirements Tracing to Sub-system Design [Architecture and Module Levels]

– Verification [Did We Build Thing Right?] Function Point, Feature Point Testing

– Call Every Function – Variation of Input Parameters [Extremes, Zero, Random, Pattern]

Feature Point Testing – Invoke and or Exercise Every Feature with Calls or Workload [E.g. I/O Requests] – Variation of Request or Workload Parameters and Conditions [Light, Medium, High Workload]

Sam Siewert 12

Negative Tests Focus on Faults, Malfunctions, Error Handling and Recovery of Sub-Systems – FMEA – Failure Modes and Effects Analysis (Identification, Impact,

Risk [Probability of Occurrence], Mitigation Costs) – RPO/RTO – Recovery Point and Recovery Time Design Objectives

[System Level, but Drive by Sub-system Recovery] – RAS – Reliability, Availability, Serviceability Design Goals – High Availability, High Reliability Design Goals – FDIR – Fault Detection, Isolation, and Recovery Design and

Implementation – Hardware, Operating System, and Library Interface Error Handling

Design and Implementation

Tests – Hardware Fault Injection – Operating System Fault Injection – Application Layer Fault Injection

Sam Siewert 13

Negative Tests - HA Design Criteria 5 9’s – Out of Service no more than 5 Minutes per Year [Up 99.999% of the Time] Availability = MTTF / [MTTF + MTTR] MTTF ≈ MTBF, for MTTR << MTTF E.g. 0.99999 ≈ (365.25x60x24 minutes) / [(365.25x60x24) + 5 minutes] MTBF=Mean Time Between Failures, MTTR=Mean Time to Recovery

Sam Siewert 14

Big iron lessons, Part 2: Reliability and availability: What’s the difference? Apply RAS architecture lessons to the autonomic Self-CHOP roadmap

Negative Tests – Fault Injection Hardware Layer – E.g. Cable Pull for Storage [SAS, SATA] – Cable Pull for Networking [Ethernet] – Power Loss – Analog Circuit Methods [Over/Under Voltage, Short, Open, Mis-

route]

Software Layer – Buffer Corruption [Shared Memory Corrupt, Message Loss,

Corrupt, … from Test Client] – Network Layer 2, 3, 4 Packet Corruption - Wireshark – Bad Command Generation [Test Client] – Bad Pointer [Buffer]

Sam Siewert 15

Diagnostic Tests Primarily Hardware Tests, Driven by Software Could be OS test, E.g. During Boot of System – CPU – I/O – Network – Memory test – File system test – OS Services

Memory Test – Simple – Walking 1’s,

Address Bus Test, Pattern Tests all Read-after-Write to Address

– Advanced – ECC, SoC Drawer Paper Sam Siewert 16

E.g. Linux Boot-up Process for Centos 6.x

Interoperability Tests Functions calling Functions – E.g. Modules that call Libraries – E.g. OpenCV, Computer Vision Library – C++ Boost Library – Operating System API – POSIX pthreads, aio, shm, mq

Shared Memory Synchronization Message Passing [Queues] Application Layer Command/Response – Start, Header Message, Stop [E.g. 0xA5A5, <header>, <msg>, 0x5A5A] Network Stack [Above Layer 4/5] – Application Layer – Presentation Layer – Session Layer

Sam Siewert 17

Interoperability - Stack Examples 7) Application – VLC, SSH, SFTP, SCP server/client tools (to stream video/audio, copy files and remote terminal access) 6) Presentation – ASCII, MPEG, JPG, PNG (encoding of binary data or text for consumption via a web browser or application) 5) Session – Linux Sockets API (code level interface for network applications) 4) Transport – TCP (Transmission Control Protocol), UDP (User Datagram Protocol) 3) Network – IP (Internet Protocol), IPsec (secure encrypted IP), ICMP (Internet Control Message Protocol), IGMP (Internet Gateway Message Protocol) 2) Data Link – IEEE 802.3 (Ethernet) and 802.11b-n (Wireless) 1) Physical – UTP (Unshielded Twisted Pair) Cat-6, SFP (Small Form Pluggable Optical), RS-422 (AppleTalk)

Sam Siewert 18

Interoperability – File Copy Secure Copy – remote file copy program (application)

Sam Siewert 19

Sockets API

MPEG Program File

SCP Service

Big Buck Bunny MPEG4 Stream File

TCP

transformer.uaa.alaska.edu

IP

IEEE 802.3z - GigE

Cat-6 UTP

Internet

Sockets API

MPEG Program File

SCP Client

TCP

ssiewert-HP-8540w.uaa.alaska.edu

IP

802.11abgn - Wireless

Radio Transmission

IP Router/Switch

L5

L6

L7

L4

L3

L2

L1

Big Buck Bunny MPEG4 Stream File

Interoperability – MPEG Streaming VLC Server/Player – remote MPEG Streaming

Sam Siewert 20

Sockets API

MPEG Program

VLC Stream Service

Big Buck Bunny MPEG4 Stream File

UDP

transformer.uaa.alaska.edu

IP

IEEE 802.3z - GigE

Cat-6 UTP

Internet

Sockets API

MPEG Program

VLC Player

Big Buck Bunny Movie

UDP

ssiewert-HP-8540w.uaa.alaska.edu

IP

802.11abgn - Wireless

Radio Transmission

IP Router/Switch

L5

L6

L7

L4

L3

L2

L1

Fault Injection Methods (IP Network Layer Forwarding)

Sam Siewert 21

Sockets API

MPEG Program

VLC Stream Service

UDP

IP

IEEE 802.3z - GigE

Cat-6 UTP

Sockets API

MPEG Program

VLC Player

UDP

IP

802.11abgn - Wireless

Radio Transmission

L5

L6

L7

L4

L3

L2

L1

IP Forwarding [Enable/Disable]

802.3z - GigE

Cat-6 UTP

802.11abgn - Wireless

Radio Transmission

Fault Injection Methods (Application Layer Message

Forwarding)

Sam Siewert 22

Sockets API

MPEG Program

VLC Stream Service

UDP

IP

IEEE 802.3z - GigE

Cat-6 UTP

Sockets API

MPEG Program

VLC Player

UDP

IP

802.11abgn - Wireless

Radio Transmission

L5

L6

L7

L4

L3

L2

L1

Sockets API

MPEG Program

Store and Forward [Corrupt, Delay, Drop]

UDP

IP

802.3z - GigE

Cat-6 UTP

802.11abgn - Wireless

Radio Transmission

Performance Tests Profiling – Gprof – Open souce tool [similar to Gcov, but for Profiling] – Vtune – Commercial Tool from Intel – Logic Analyzer and HP’s SPA (Statistical Performance Analysis)

Tracing – E.g. Timestamps output to syslog

Statistics – top, htop – iostat – memstat

Workloads – Iometer – stress

Sam Siewert 23

Performance - Sysprof What is Using CPU on my System Rather than Profile of an Application – Sub-System [Service]

Sam Siewert 24

Gprof Simple –pg compile opiton Run, gprof on gmon.out to get analysis

Sam Siewert 25

%make cc -O3 -Wall -pg -msse3 -malign-double -g -c raidtest.c raidtest.c: In function 'main': raidtest.c:99: warning: format '%d' expects type 'int', but argument 2 has type 'long unsigned int' raidtest.c:68: warning: unused variable 'aveRate' raidtest.c:68: warning: unused variable 'totalRate' raidtest.c:66: warning: unused variable 'rc' raidtest.c:212: warning: control reaches end of non-void function cc -O3 -Wall -pg -msse3 -malign-double -g -c raidlib.c cc -O3 -Wall -pg -msse3 -malign-double -g -o raidtest raidtest.o raidlib.o %./raidtest Will default to 1000 iterations Architecture validation: sizeof(unsigned long long)=8 RAID Operations Performance Test Test Done in 453 microsecs for 1000 iterations 2207505.518764 RAID ops computed per second %ls Makefile gmon.out raidlib.h raidlib64.c raidtest raidtest.o Makefile64 raidlib.c raidlib.o raidlib64.h raidtest.c raidtest64 %gprof raidtest gmon.out > raidtest_analysis.txt

Gprof Analysis 1 million iterations of RAID test XOR and Rebuild

Sam Siewert 26

Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ns/call ns/call name 82.13 1.54 1.54 main 15.47 1.83 0.29 2000001 145.38 145.38 xorLBA 2.67 1.88 0.05 2000001 25.07 25.07 rebuildLBA % the percentage of the total running time of the time program used by this function. cumulative a running sum of the number of seconds accounted seconds for by this function and those listed above it. self the number of seconds accounted for by this seconds function alone. … calls the number of times this function was invoked, if this function is profiled, else blank. self the average number of milliseconds spent in this ms/call function per call, … total the average number of milliseconds spent in this ms/call function and its descendents per call, … name the name of the function. …

RAID Operations Performance Test Test Done in 206417 microsecs for 1000000 iterations 4844562.221135 RAID ops computed per second

Call Graph Profile from Gprof

Sam Siewert 27

Call graph (explanation follows) granularity: each sample hit covers 2 byte(s) for 0.53% of 1.88 seconds index % time self children called name <spontaneous> [1] 100.0 1.54 0.34 main [1] 0.29 0.00 2000001/2000001 xorLBA [2] 0.05 0.00 2000001/2000001 rebuildLBA [3] ----------------------------------------------- 0.29 0.00 2000001/2000001 main [1] [2] 15.4 0.29 0.00 2000001 xorLBA [2] ----------------------------------------------- 0.05 0.00 2000001/2000001 main [1] [3] 2.7 0.05 0.00 2000001 rebuildLBA [3] ----------------------------------------------- This table describes the call tree of the program, and was sorted by the total amount of time spent in each function and its children… % time This is the percentage of the `total' time that was spent in this function and its children… self This is the total amount of time spent in this function. children This is the total amount of time propagated into this function by its children. called This is the number of times the function was called…

Q & A Integration of Modules [Units] Configuration of Sub-systems and Systems Sub-systems built from Modules [units] and Components Positive Tests Negative Tests Interoperability of Modules and Components Sam Siewert 28