2005 EPRI PowerPoint Template

1© 2009 Electric Power Research Institute, Inc. All rights reserved.

Developing Software in EPRI

Software Engineering Team (SET)2009


Software Engineering Team Quality Manager and Sector Contacts

Mary McKenna

Rachel Ostraat

ENV, GEN

Tien Luong

NUC

Manuel Morales

PDU


EPRI Software Engineering Team (SET)

Manuel Morales

Oliver Carcallas

Tien Luong

Colette Handy

Mary McKennaVenkat Natarajan

Rachel Ostraat

Adam Wiseman

Dixie Herd

Vu Nguyen

Dinah Carson


Agenda

• EPRI Software Distribution Center

• Developer’s QA Process

• Frequently Asked Questions (FAQs) and Common Problems

• Software Development Requirements Process Table

• SET Guide for Testing Your Software


EPRI Software DistributionCharlotte, NC


EPRI Software Distribution Center

Distribute over 95% of EPRI Software and emedia products to EPRI members and non-members Library houses over 400 software and 850 emedia products We activate product records in APOLLO which captures that deliverable date has been met

• Majority of orders received are processed via EPRI.com and are shipped within 24-48 hours

• Prior to order being shipped, it has been screened for export control, program/project eligibility and licensing Customers can receive hard copies (CDR) of our products, or download from EPRI.com for immediate use

http://rds.yahoo.com/_ylt=A0WTb_0JqxVKYnAANO2jzbkF/SIG=121jkdiik/EXP=1243020425/**http%3A//www.flickr.com/photos/nobbin/10239990/


EPRI Software Distribution Center

Maintains a Nuclear Project Quality Plan and set of Project Quality Instructions Responsible for distribution of EPRI products developed under an Appendix B program.Houses an archive library of more than 550 legacy software products dating back to the early 80’sRetains source code for all active productsFor legacy products being updated, certain steps must be followed to allow access to source codeSince 1997, all documentation for software (e.g. User Manuals) has been maintained The EPRI Software Distribution Center is extremely proud of our dedication to our customers and our #1 goal is to achieve the highest level of customer satisfaction

http://rds.yahoo.com/_ylt=A0WTbx4IrhVKAeEAHLqjzbkF/SIG=13rvt17ct/EXP=1243021192/**http%3A//www.freesignage.com/pdfthumbs/osha_think/our_goal_is_customer_satisfaction_osha_caution_sign.png


Developer’s QA Process


Developer’s QA Process

• Corporate software quality assurance plan is:– A high level plan– A description of processes, procedures, and

guidelines to ensure the production of high-quality, error-free software

– Adaptable to multiple software projects

• Each of the subject points may be handled differently by each project, depending on client’s requirements


Corporate Software QA Plan Contents

• Software standards • Test Plans

– Unit, system, and integration testing– Beta testing

• Bug Reporting and Tracking• Verification and Validation of software

See the minimum list at:http://mydocs.epri.com/docs/SDRWeb/processguide/csqap.html

Developers are expected to have a current software quality assurance plan within the past 18 months

http://mydocs.epri.com/docs/SDRWeb/processguide/csqap.html


Frequently Asked Questions (FAQs) and Common Problems


FAQs

• What do I have to do for software quality approval when first submitting a contract?

– Assemble the Software Contract Package– The developer and the EPRI project manager

should also review the Process Table

• How do I know if my software needs to follow a different process from that outlined in the basic Process Table?

– The key milestones, dates, and document approvals for your software project will be listed on the Software Deliverable Requirements form in your project's Contract Package.

http://mydocs.epri.com/docs/SDRWeb/processguide/swdrf.html

http://mydocs.epri.com/docs/SDRWeb/processguide/table.html


FAQs (cont.)

• What forms are required for EPRI software?– http://mydocs.epri.com/docs/SDRWeb/processguide/forms.html– Software Deliverable Requirements Form (SDRF)– Software Encryption Functions Checklist– Software Life Cycle Management Document– Software Product Description Template

http://mydocs.epri.com/docs/SDRWeb/processguide/forms.html


FAQs (cont.)

• When is beta testing a requirement?

– Beta testing is expected for all EPRI software. Exceptions may be made by the Sector VP.

• What test services does the SET Team provide?

– Alpha, Beta, Prescreen, and Final Acceptance tests


FAQs (cont.)

• Why do I need my application tested?--my funders don't want to spend the extra money

– All EPRI software must deliver the level of usability and reliability that customers and EPRI management expect. The testing process ensures this. Beta testing and final acceptance testing are EPRI requirements.


Common Problems

• Problem: The application crashed because an incorrect input was entered into a data input field.

– Solution: Implement range checking functions in all relevant input fields.

• Problem: An anomaly was fixed in one area of the application, but not in other similar areas of the application

– Solution: Review the SET test report for the use of the phrase, "check for similar occurrences" by the tester.


Common Problems (cont.)

• Problem: The user's manual does not follow EPRI'ssoftware manual guidelines.

– Solution: Before submitting for testing, visit the Software Manual Preparations Guidelineswebpage for guidelines and requirements.

• Problem: The tutorial/solved example problem results did not match exactly with the actual results from the application.

– Solution: Before submitting for testing, the developer needs to check the results in the documentation compared to the actual results generated by the application.

http://mydocs.epri.com/docs/SDRWeb/processguide/manuals.html


Software Development Requirements Resource


Where Is The Software Development Website?



Via EPRI.com



Location:http://mydocs.epri.com/docs/SDRWeb/processguide/index.html

http://mydocs.epri.com/docs/SDRWeb/processguide/index.html


Software Development Website

• Features of the Software Development Requirements Website

– Outlines EPRI software requirements

– Provides guidance and assistance for EPRI software developers, project managers, and software quality managers on the software development life cycle

Please Note:For the Nuclear Sector, these software development requirements are not applicable to software developed under, the EPRI QualityAssurance Program, which complies with 10CFR50 Appendix B, 10CFR21 and ISO 9000-1994. Software developed under the EPRI Quality Assurance Program shall follow the requirements documented in that Program. Your EPRI contract will clearly state if this separate program applies to you.


Software Development Requirements Process Table


Process Table: Homepage & Step 1

Click on the Requirements Process link (circled above) in the websites left-side navigation bar to go directly to the process table.

Step 1. Concept Development


Process Table: Step 2 to Step 7

Step 2. Defining RequirementsStep 3. DesignStep 4. ImplementationStep 5. Alpha & Beta TestStep 6. Final Acceptance TestStep 7. Support & Maintenance


Process Table: Roadmap MajorResponsibility

EPRI Project ManagerSoftware DeveloperSoftware Engineering Team

Step 1: Define User Requirements

RFPIf Needed

Step 2: SW PlanningDocuments

Step 3 and Step 4: Start SW

Development

Step 5: SW Prototype/Alpha

& Review

Complete SWDevelopment

Review Beta Testers’Feedback &Bug Report

Step 5: Beta Testing

& User Manual(for PM & SET)

Customer Beta Testing &User Feedback

Step 6: SW Prescreen or

Final Acceptance Testing

Announcement& Distribution

Step 7: Support &

MaintenanceArchive / Retire

SW ContractPackage


Step 1: Concept Development

Concept development begins when an EPRI customer specifies or confirms the need for a new software system or for modificationsto an existing software product.

What must be included in a Software Contract Package?

• Software Deliverable Requirements Form (SDRF)

• Software Life Cycle Management Document• Software Encryption Functions Checklist• Software Product Description Template• Developer Qualifications Summary• Developer Corporate Software Quality

Assurance Document

For additional contract package details, see: http://mydocs.epri.com/docs/SDRWeb/processguide/swdrf.html

http://mydocs.epri.com/docs/SDRWeb/processguide/swdrf.html


Step 1 (cont’d)

Budgeting for Testing

For typical software, SET has observed that on average:

• Beta test requires 14 testing hours• Final Acceptance test (usually two tests)

requires 34 testing hours

Therefore, planning about 48 testing hours is a safe estimate. Depending on the complexity of the software, some software will require more or less testing time.


Step 2: Defining Requirements

Software planning documents are developed, which address EPRI software product and process requirements according to the developer’s Corporate Software Quality Assurance Document.

What planning documents does EPRI require?

Example planning documents:• Software Requirements Document (SRD) • Software Development Plan (SDP) • Functional Specification (FS)

For planning document information, sample content, and requirements, see: http://mydocs.epri.com/docs/SDRWeb/processguide/reqdes.html

http://mydocs.epri.com/docs/SDRWeb/processguide/reqdes.html


Step 3: Design Documentation

The design step begins when the software requirements have been defined.

The design document provides precise directions to software programmers about how basic control and data structures will be organized.

See the EPRI Software Types chart for requirements specific to the type of software being produced: http://mydocs.epri.com/docs/SDRWeb/processguide/soft_typ.html

http://mydocs.epri.com/docs/SDRWeb/processguide/soft_typ.html


Step 4: Implementation

Software programming begins after defining the problem and designing the solution. The software is built and tested according to the planning and design documentation.

Implementation Tasks are performed by developer:

• Project Plan Status Reviews• Code Software• Create Solved Examples• Unit, System, and Integration Tests• Draft Documentation• Verification & Validation (V&V)


Step 5: Alpha & Beta Testing

The main objective of Alpha and Beta testing is to show with a high level of confidence that the software application meets the following acceptance criteria: function, performance, usability, features, and capabilities.

ALPHA Testing:• Performed in the developer's environment.• Software contains most core functions, but

will not contain all the intended functionality.

BETA Testing:• Performed in customer environments.• Allows users to find errors and provide

functionality feedback before product release.


Step 5 (cont’d) - Beta Testing

BETA Testing Reminders:Testers - EPRI requires at least one customer beta tester. Three or more beta testers are recommended.

Approval – Before a beta software is distributed to users, SET must perform a Beta review. The acceptance for beta distribution is given within 24-Hours (72-Hours in Nov. and Dec.) of physically receiving the software.

Distribution – All beta software must be distributed to users via EPRI.com. Distribution requires a beta splash Screen. For splash screen information and all other requirements, go here: http://mydocs.epri.com/docs/SDRWeb/processguide/betapre.html

http://mydocs.epri.com/docs/SDRWeb/processguide/betapre.html

http://mydocs.epri.com/docs/SDRWeb/processguide/betapre.html


Step 6: Final Acceptance Test

When must I submit my software if it is due on December 31, 2009?

For software due on 12/31/09:– Beta must be submitted by 10/01/09– Final must be submitted by 11/02/09

In the Final Acceptance Step, the software is submitted to EPRI and completed. Software is released for distribution after successfully passing Final Acceptance Testing.


Step 6 (cont’d) - Final Acceptance Testing

What must I submit for the final acceptance test?

Final Acceptance Test submittal package:

• Application CD-ROM• Source Code CD-ROM • Source Code Transmittal Letter• Software Acceptance Form (SAF)• Certificate of Conformance• Developer response to previous SET report

For package details, go: http://mydocs.epri.com/docs/SDRWeb/processguide/achk.html

http://mydocs.epri.com/docs/SDRWeb/processguide/achk.html

http://mydocs.epri.com/docs/SDRWeb/processguide/achk.html


Step 7: Support & Maintenance

After the final software release, support and maintenance (archiving, retiring, and bug-fixing) of the software become important as customers use the software.

Software Support

The EPRI Customer Assistance Center (CAC) provides first-line support. Contact CAC at (800) 313-3774 or email [email protected].

Maintenance – Archiving and Retiring

Software archival and retirement are handled by the EPSC after Project Manager initiation, the archival process has been followed, and approval is obtained.

mailto:[email protected]


Step 7 (cont’d) - Maintenance

Maintenance – Bug Fix Process

What is the bug fix process?

1. Project Manager notifies SET of the bug2. Developer fixes software and submits for testing3. SET tests the updated software 4. EPSC sends software to tester for real world

data test.5. After “OK” from tester, software is distributed.6. Customers are notified by EPSC of update.

Reminder - Always budget for support and maintenance needs.


Process Table: SET’s Role In Process

Step 1 (Approve) - Software Contract Package

Step 3 (Review) - Software Planning Documents

Step 5 (Approve) – Beta Testing

Step 6 (Approve) – Final Acceptance Testing(Responsible) – Distribution by EPSC

Step 7 (Responsible) - Archive/Retire by EPSC


SET Guide for Testing Your Software


Usability Testing Sections

• Installation

• Solved Example Problems (or Tutorial)

• User Documentation

• Graphical User Interface (GUI)

• Stress Testing

• Security Vulnerability Testing


Installation

1. Run a Virus Scan

2. Verify:– User Manual– Instructions for installation

3. If applicable, provide network installation instructions

4. Default setting installation


Installation (cont’d)

5. Uninstall, then reinstall in non-default directory/drive

6. Click on Cancel button(s) during installation process

7. If serial numbers or security keys are required, enter invalid entries to make sure the security works


Installation (cont’d)

8. Change the Program Folder where the shortcut in the Windows Start menu is located.

9. Applications that do not require installation, such as Spreadsheets, still require installation instructions.


Solved Example Problems (or Tutorial)

Reminder: Three solved example problems (or one tutorial) are required

• Run solved example problems (or tutorial) to make sure all inputs and results (i.e., calculations, graphs, screen captures, etc.) in the application match exactly with the inputs and results in the user documentation

Note: If any inputs or results do not match, the software can not be approved to send to customers


Solved Example Problems (or Tutorial)

Additional SET information and Solved Example Problems (or Tutorial): http://mydocs.epri.com/docs/SDRWeb/processguide/testcase.html

http://mydocs.epri.com/docs/SDRWeb/processguide/testcase.html

http://mydocs.epri.com/docs/SDRWeb/processguide/testcase.html


User Documentation

1. Check that EPRI Technical Publications User Manual template was used (or followed)

Note: This ensures title page, disclaimer page, contacts page, copyright and ordering information are all current and that EPRI style guides are used

2. Check headers and footers3. Check for system requirements:

a. Hardware and Software specificationsb. Permissions such as Administrator rights

4. Check application feature descriptions5. Check spelling and grammar


User Documentation

• SET has a Manual template for the required documentation.

• Below is a link for the documentation template: http://mydocs.epri.com/docs/SDRWeb/processguide/swurr.html#WebReq

http://mydocs.epri.com/docs/SDRWeb/processguide/swurr.html#WebReq




Graphical User Interface (GUI)

1. Windows fit in the main application screen and nothing is cut-off if windows are resized

2. Make sure all data/information is accessible

3. Internationalization, check multiple regions

4. Change appearance settings5. Controls on pages must respond

properly to Tab order and hot-keys (alt-keys)

6. Check online Help feature, including buttons to open the Help feature


Stress Testing

1. Range checking – Look for input fields and enter invalid values

2. Make sure that numeric-only fields accept only numeric values

3. Follow the solved example problems, but then skip a step or do them in a different sequence


4. Check print feature5. If there are logins, enter invalid login information6. Check error messages for clarity. Error

messages should appear when the error occurs.

7. Check for spelling within the application

Stress Testing (cont’d)



8. For databases:a. ensure all connections through the application are

valid when accessing datab. ensure single quotes and double quotes are tested

to verify they do not corrupt the databasec. add duplicate recordsd. delete all records to make sure it does not crash the

application9. Modify data files (such as adding an extra comma) to

make sure the application gives a correct error message



10. For application administrative features, make sure only administrators of the application may access those features

11. Check for compatibility with Microsoft Office applications if applicable (such as copy and paste features)

12. Click all buttons to make sure they work

13. Check save feature (does not overwrite existing file without permission, saves to correct directory, creates correct extension, etc.)

Without administrative feature

With administrative feature



14. Check open file feature (correct file extensions, choosing incorrect file type brings up error message, etc.)

15. If there are graphs, check graph features and settings

16. Check options/settings not covered in the sample problems.

17. Check to make sure international units are converted correctly

The International Standard

date notation

DD-MM-YYYY

United States Standard

date Notation

MM-DD-YYYY



18. Maximize, minimize, and resize windows to make sure the application responds correctly.

19. Check keyboard shortcuts 20. Check all menu items, including the

pop-up menus that come up when the user right-mouse clicks an item

21. If there are hardware/software keys, check to see if the application responds when executed with the key(s), then without the key(s)


Security Vulnerability Testing



• OWASP Top Ten Web Application Vulnerabilities – http://www.owasp.org/index.php/OWASP_Top_Ten_Project

1. Cross Site Scripting (XSS)2. Injection Flaws3. Malicious File Execution4. Insecure Direct Object Reference5. Cross Site Request Forgery (CSRF)6. Information Leakage and Improper Error Handling7. Broken Authentication and Session Management8. Insecure Cryptographic Storage9. Insecure Communications10. Failure to Restrict URL Access

http://www.owasp.org/index.php/OWASP_Top_Ten_Project



• 2 examples of vulnerabilities SET will test for:– Cross-Site Scripting– Structured Query Language (SQL) Injection

• The developer is expected to address security vulnerabilities when developing an application


Security Vulnerability Testing (cont.)

• Cross-Site Scripting - Harmful scripts are entered into web sites via querystring or form field

• Example:– Enter in "<script type="text/javascript"> alert(‘hello’);

</script>" into a form field to check whether the form field is validated

• Allows the user to execute scripts that are harmful

• See the following for more information: http://www.owasp.org/index.php/Cross-site-scripting

http://www.owasp.org/index.php/Cross-site-scripting



• SQL Injection – Injection of a SQL Query through input data, such as a querystring or form

• Examples:– In the querystring, enter a SQL Statement, such

as " ‘; Delete from users --’ ", into a querystring variable

– Enter in " ' OR 1=1 " into a form field or querystring variable

• See the following for more information and testing examples: http://www.owasp.org/index.php/SQL_Injection

http://www.owasp.org/index.php/SQL_Injection



• Testing tools:– OWASP’s Web Scarab– Acunetix Web Security Scanner– IBM Rational AppScan

• Reference:– Open Web Application Security Project (OWASP)

http://www.owasp.org/index.php/Main_Page

http://www.owasp.org/index.php/Main_Page


What SET Does Not Do

SET software usability testing does not do:1. V&V (Verification and Validation) testing 2. test or validate real world data (this should be

done by beta testers) 3. exhaustive testing or “white box” (source code)

testing

SET usability testing will not find all errors and is not intended to

All errors are expected to be found by developers


Together…Shaping the Future of Electricity