Embed Size (px)
Citation preview
II..II I 1 II III IID
DOT/FAA/SE-90-1 National Airspace SystemDOT/FAA /CT-TN90/01 (NAS)
NAS System Engineering Service Software Life CycleWashington, D.C. 20591 Management Study
D T IC .. TIC FILE COPYSELE CTES....... 1990 :
00
N March 1990
Final Report
This document is on file at the Technical
Center Library, Atlantic City International-,Airport, New Jefsey 08405
APPtIC'ede for PULIlic iolejeas
... �t-Qf UlJlted
US Depo:tmentof TronpoftotxOnFedro AvlatloinAdmInistration
I J
NOTICE
This document is disseminated under the sponsorshipof the U.S. Department of Transportation in the interest ofinformation exchange. The United States Governmentassumes no liability for the contents or use thereof.
The United States Government does not endorseproducts or manufacturers. Trade or manufacturers'names appear herein solely because they are consideredessential to the objective of this report.
i
Technical Report Oocumentotion Pagl
1. Report No. 9. G" og ment Acceslo,, No. 3. Recipient'a Catalog Ne.
DOT/FMA/SE -90/1
4. TLiO and Subt-tle S. Repoli DottoMarch 1990
NATIONAL AIRSPACE SYSTEM (NAS) 6. Perfmri Orgaei.t,oon Cod.SOFTWARE LIFE CYCLE MANAGEMENT STUDY
8". Petfistmino O,qan, action Recall No.7. A,,thO,' 6)
Technology Planning, Inc. DOT/FAA/CT-TN9O/019. Pe6,6to',.n organOlot. 0 Name and Address 10. Wlk t Unt No. (TRAIS)
Technology Planning, Incorporated II Co.ntact ofG,O, rno. -51 Monroe Street, Suite 102 DTOS59-88-C-00064, T.O. 0011Rockville, Maryland 20850 13. TYP Of Report and Peood Co-.°.d
12. Sponsoringl Agency Nmame and AddressU.S. Department of Transportation Final ReportFederal Aviation AdministrationSystem Engineering Service 14. Spene.,.g Ag.n4, Cod. ASE-200Washington, DC 20591 transferred to ASE-500
I5. SupfPItieetery Notes
Contract Technical Monitor: Betty K. Falato, FAA Technical Center, ACN-140Atlantic City International Airport, NJ 08405
14. Abstract
In implementing FAA-STD-026 (based on DOD-STD-2167A) for acquiring NAS softwareand adopting the Ada Programming Language as the single high order language forNAS, several issues were raised pertaining to the differences in softwaredevelopnent, testing, and maintenance practices between the NAS acquisition andoperaticns support organizations. This report identifies and documents some ofthe deficiencies with respect to the existing process and guidelines used toacquire and maintain NAS software and recommends an action plan to address theidentified deficiencies.
In addition, it presents an overview (list of products and activities) of theNAS System Life Cycle management process within the context of a major systemsacquisition, as described by FAA Orders 1810.1 and 1800.8. The process isapplicable to non-major systems as well. -/-
17. Key Words Oi, mt-bu,,., Sttemen,
\ NAS System Life Cycle Document is on file at the TechnicalSoftware Engineerijng,1 . Center Library, Atlantic CityNAS Software Engineering. - I Internasiocal Airport, NJ 084-05Software Life Cycle .
Uncla.|qfi -d Unclassified 113
Form DOT F 1700.7 (8-72) Re•adctef of completed page owtherlsed
E=CUTIVE SUMMARY
Acceslorn For
NTIS CRiA&iOTIC TAB 3U nan ro., (ced0Jujs if icajtO'i ________
By ____
AvdII~l~Codes
A, ~ i-i i of
TABLE OF CONTENTS
Page
1.0 INTRODUCTION ............ ................... Page 1
1.1 BACKGROUND .............. .................... Page 1
1.2 SCOPE, OBJECTIVES AND GOALS ....... ............ Page 1
1.3 TERMINOLOGY ............. ................... Page 2
1.4 METHODOLOGY ............. ................... Page 2
2.0 OVERALL ASSESSMENT .......... ................ Page 3
3.0 RECOMMENDED COURSE OF ACTION .... ........... Page 4
3.1 ESTABLISHING AN ADEQUATE SOFTWARE COMMITMENT_. . . Page 4
3.1.1 Software Policy Statement ..... ........... .. Page 5
3.1.2 Establishment of Technical Support Groups . . . Page 5
3.2 IN-DEPTH TECHNICAL ASSESSMENT OF FAA SOFTWARECAPABILITIES ............ ................... Page 6
3.2.1 Software Process Assessment ......... .......... Page 6
3.2.2 Software Tecbnoloav Assessment ... ......... .. Page 7
3.3 OVERCOMING ORGANIZATIONAL BARRIERS .......... Page 7
3.4 DEVELOPMENT AND IMPLEMENTATION OF A STRATEGICPLAN FOR IMPROVING THE NAS SYSTEM LIFE CYCLE . . Page 8
3.4.1 Software Technoloy Transfer .. .......... ... Page 8
3.4.2 ImprovinQ the Software Process ThrouQh Standards,Orders, and Guidelines ...... ............. .. Page 9
3.4.3 Evaluation . . . .................... Page 10
4.0 REFERENCES .............. .................... Page 11
1.0 INTRODUCTION
1.1 BACKGROUND
The Federal Aviation Administration (FAA) is involved in a majormodernization and expansion of the National Airspace System (NAS)in order to meet future requirements and demands. The AdvancedAutomation System (AAS) is estimated to contain two million linesof code. The primary AAS language is Ada. While the AAS is thelargest of the NAS subsystems, many other subsystems such as DataLink, Mode-S, NADIN II, CWP, and AERA are software intensive aswell. More and more NAS functions will be automated and softwarewill be increasingly depended upon.
Key to the NAS modernization initiative is the acquisition ofsoftware to support the intended services. Considering thecomplexity of the effort, it is imperative that the FAA methodologyused to acquire and maintain NAS software be documented to providea common perspective for planning, requirements analysis, systemdesign, program development and implementation, test anuevaluation, deployment, and maintenance.
1.2 SCOPE, OBJECTIVES AND GOALS
The System Engineering and Program Management Office, SystemDesign and Configuration Management Division (ASE-200), isresponsible for developing and maintaining the technical standardsused in acquiring NAS subsystems. To date, FAA has standardized ona software development standard, FAA-STD-026 (based on DoD-STD-2167A), and on the Ada Programming language. In implementingthese standards several issues have arisen pertaining to thedifferences between software development, testing, and maintenancepractices between the NAS acquisition and maintenanceorganizations.
To clarify and address these issues the FAA tasked TechnologyPlanning Incorporated (TPI) to review the existing FAA softwarepolicies, orders, standards, process, and procedures which are usedby the FAA for NAS software acquisition, maintenance, data anddocumentation management. The objectives were to:
1. Identify current deficiencies, omissions, and conflicts withrespect to these guidelines;
2. Review specific NAS Plan subsystems in terms of the proposedend-state software, data engineering environment, softwaredevelopment methodology, general implementation andmaintenance strategy;
Page 1
3. Document deficiencies and make recommendations; and
4. Assist in the preparation of a NAS System-level Plan toaddress the identified deficiencies.
As a result of accomplishing the above objectives, the followinglonger range goals can be initiated.
Goal 1
Improve the process of constructing qualitysoftware.
Goal 2
Reduce the risk factors associated with buildingsystems. The risk factors include technical,schedule, cost, operational and support areas.
Goal 3
Heighten the awareness and increase the involvementof management and other appropriate staff with thesoftware acquisition process.
Goal 4
Cultivate the development of software engineeringstrength within FAA.
1.3 TERMINOLOGY
Throughout this report the word "guidelines" is used to indicateany of the standards, orders, policies or procedures used by theFAA for NAS software acquisition.
1.4 METHODOLOGY
In order to adequately address Software Acquisition by the FAA, theacquisition of software must be looked at in the larger context ofthe NAS System Life Cycle and must consider the system levelaspects of the life cycle which directly influence or areinfluenced by software.
The methodology used in performance of this task includes foursteps:
1. Development of a Data Collection Instrument which was usedduring interviews with FAA personnel who are responsible forsome aspect of software within some phase of the NAS System
Page 2
Life Cycle;
2. Data collection which included the personnel interviews andreviews of selected policies, orders, standards andguidelines;
3. Analysis of the data collected;
4. Development of conclusions and recommendations which haveresulted in a plan of action for the FAA.
The following organizations participated in the interviews for thistask:
AAF-4, AAP-120, AAP-220, AAP-320, AAP-400, AAT-14, ACD-340,ACD-350, ACN-1I0, ACN-130, ACN-210, ACN-310, ACS-320, ADS-120,AHT-400, AHT-500, ALG-410, AMC-300, AOR-ll0, APS-300, APS-500,ASA-6, ASA-130, ASA-210, ASM-140, ASM-160, ATR-210, ATR-250,LOGICON.
2.0 OVERALL ASSESSMENT
One of the most common problems facing government and privateorganizations is the transition to a modern and effective softwareengineering methodology for creation of software systems,acquisition of software systems, or both. This problem isaddressed in depth in the literature ([AFSB89], [AFSC89], (NASA89],and [PRESS88]) but still remains a problem. In these and otherstudies the fundamental approach has been a four stage approachconsisting of:
1. Assessment of current practices;
2. Development of a strategic plan for the transition to improvedpractices;
3. Implementation of the plan;
4. Evaluation of the success or failure of the plan.
TPI has performed the first portion of step 1 by performing aqualitative analysis of the FAA's software acquisition process. Thefindings of this analysis can serve as the basis for identifyingthe remaining activities required to improve the FAA's softwareacquisiticn process.
The results of TPI's assessment are summarized as follows:
'. There is a lack of adeqk14t-e commitment to the software aspectsof FAA projects;
Page 3
2. There is not enough up to date software expertise within theFAA;
3. Some ptoblems in software acquisition are due to the FAAorganizational structure which artificially separateshardware, software and systems engineering activities;
4. There are problems that are the result of inadequate, missing,or poorly understood FAA standards, guidelines and ordersassociated with the software acquisition process.
5. There is a lack of involvement of software maintenance peopleduring the program definition and software acquisition phases.
Although the report that TPI produced, detailing the findings ofthe assessment, reports several other problems, analysis shows thatalmost all the problems can be traced back to these four basicpoints.
3.0 RECObMMNDED COURSE OF ACTION
To continue with the approach outlined in Section 2.0 above, TPIrecomuends the tollowing course of action. Although the activitiesdetailed imply a sequential order, there is room for parallelactivities to occur. Some of these activiLies are required to beperformed in the near term, while others may be performed in therange from medium to long term.
While a complete implementation of all the recommendations isencouraged, budget considerations may dictate that selected partsof the recommendations be deferred to a later time. This isentirely possible due to the nature of the problems. There willbe a loss of the synergistic effect of implementing allrecommendations as one program plan. Due to the complexity of thesoftware acquisition process, it is expected that a completeprogram to improve the process will take several years toaccomplish.
The following activities have been identified to address theapproach and the problems cited in the preceding section. Eachactivity supports the longer range goals described in Section 1.2.The predominant goal for the activity is listed along with a briefexplanation.
3.1 ESTABLISHING AN ADEQUATE SOFTWARE COMMITMENT
The FAA needs to establish a clear and firm commitment to dealingwith software issues in FAA projects. Such a conmitment consistsof: (1) allocation of resources and funds to improve the process
Page 4
and staff capabilities: (2) establishment of additional andenhanced software policies; (3) participation with otherorganizations dealing with the same issues; and (4) encouraging theinvolvement of senior management. This activity supports Qoal 3 byelevating the level and degree of attention given to softwareprojects.
3.1.1 Software Policy Statement
The FAA needs to develop a comprehensive software policy statementoutlining the goals, commitment, and rules pertaining to softwaresystems acquisition. This software policy statement will be usedto perform evaluations of FAA projects and also to evaluatetechnologies, tools, and FAA guidelines pertaining to softwareprojects. Such a policy will be coupled with improved guidelines,standards, and orders.
Because of poor experiences with past FAA projects, most projectsthat have a significant software component should be consideredhigh risk projects. The overall policy statement will require asoftware risk assessment and risk management plan as an integralpart of these projects. This activity supports Goal 2 by providinga method for evaluating high risk projects during the early phasesof the software acquisition life cycle.
3.1.2 Establishment of Technical Support Groups
The FAA needs to establish agency-wide support groups dealing withmajor software issues. Such groups will both support projects andalso serve as review and evaluation groups for project activities.They will be available upon request to assist a project withproject definition and initiation activities. These groupr willserve as the focal points for infusing software technology iz. .o theFAA. They will interact with industry (especially contractors),academia, and other government agencies dealing with the sameissues (e.g., DoD, NASA) . These groups will support the infusionof software skills into the FAA strff and projects through suchactivities as: training, workshops, symposia, and forums.
It is anticipated that these groups will be small in size, batwe'ýn3 and 7 persons, and that these groups will report to a high enoughmanagement level to firmly establish tht validity of theirmissions.
3.1.2.1 Software Process Sup~ort Working Groups - These groupswill deal with the overall software process including guidelines,standards, and orders as well as with software project management.These groups deal primarily with sELiware project management ratherthan specific technologies. Software quality assurance and
Page 5
software productivity also fall under these groups. These groupsare intended to be temporar'4 in nature and to focus on one softwaretask at a time in order to resolve immediate issues or concerns.This activity supports Goal 1 by providing a feedback loop duringthe software construction process.
3.1.2.2 Software TechnoloQv SuDnort Group - This group will dealwith specific software technologies and tools, as well as provideoverall guidance to the working groups discussed above. Thisgroup's charter is to identify those technologies, both availableand emerging, that are applicable to the FAA's environment. Theservices of this group would be provided on a request basis andwould be available as an on-going support function. Further, thisgroup will be responsible for the eventual incorporation of theappropriate technologies into the FAA environment. This activitysupports Goal I by providing the technologies and tools necessaryfor the development and maintenance process.
3.2 IN-DEPTH TECHNICAL ASSSSMENT OF FAA SOFTWARE CAPABILITIES
Although a qualitative assessment of the FAA's softwareacquisition process and maintenance has already been performed byTPI, an in-depth quantitative assessment needs to be performed toguide any corrective action. For example, an assessment similar tothose presented in the literature ([HUMP87] and [PRESS88]), whichaddresses the software engineering environment and softwaremanagement activities, needs to be tailored for the FAA. Such anassessment will evaluate both: (1) the technology present withinthe FAA and also (2) the software process itself. This study willinvolve a comprehensive survey/skills profile inventory ofsoftware engineering skills present within FAA personnel. TPI'sinitial study identified a lack of up to date software skillswithin the FAA as a serious problem. This follow-on assessmentwill quantify the skills currently present within the FAA andidentify precisely those areas where skills are lacking.
3.2.1 Software Process Assessment
A software process assessment evaluates the current practices withrespect to those tools, methods, standards, and processes used bythe software engineers in performance of these assignments.
Performing a software process assessment will evaluate the FAA'ssoftware process for acquisitions and maintenance against therecommended practices established within the software engineeringcommunity. Areas lacking adequate emphasis along with the skillsnecessary to pro-vide that emphasis will be identified. Althoughthe FAA's software acquisition and maintenance ii•,ds aie similar toother organizations, some deviations from the normal practices are
Page 6
expected and acceptable. The framework for evaluation of the FAA'sprocess should be extended to also evaluate the various contrz ctorssupporting the FAA's projects. This activity supports iQ...1 byestablishing and refocusing the software process for acquisitionsand Goal 2 by reducing the exposure for FAA by identifying risksearly in contractor's projects.
3.2.2 Software TechnoloQy Assessment
Unlike the previous software process assessment that deals withsteps and phases of a software project life cycle, this softwaretechnology assessment deals with evaluating the details of eachstep in the life cycle. For example, the previous assessmentconcerns itself with the existence of a coding standard while thisstep will evaluate the quality of such a standard.
This step will also perform an inventory of the software skills ofthe FAA staff and compare them against the skills identified by theprocess assessment as necessary for the various softwareengineering roles (i.e. manager, analyst, designer, coder, tester).This activity supports Goal 4 by providing a skill inventory forsoftware engineers. The skill inventory would provide the basisfor a professional development plan for each person.
3.3 OVERCOMING ORGANIZATIONAL BARRIERS
The management within the FAA needs to recognize and acknowledgethat the NAS life cycle software process has difficulties whichare associated with the current organizational structure. Thehardware, software, and systems engineering activities are, in mostcases, separated due to organizational decisions. This separationcauses daily coordination problems for those personnel involved inthese engineering activities. Lack of appropriate coordinationleads to incorrect solutions with respect to requirementsdefinition, project implementation, system integration and post-deployment support.
Policies and procedures which overcome the organizational barriersneed to be implemented and rigorously enforced by all levels of FAAmanagement. Such policies will consider: (1) increasing awarenessof FAA personnel with respect to the entire NAS life cycle'sproducts, reviews, and activities; (2) containment of life cyclecosts as the responsibility of all organizations who have any levelof involvement in the NAS system life cycle; (3) smooth transitionprocedures; (4) oarly consideration of the needs of allorganizations (during project definition); and (5) high levelcommitment to enforcement of these policies. This activitysuppuiLs Gog! . 1- prGoiJ .J JbyJ. I C a -'t ti cost andschedule risks for each project.
Page 7
3.4 DEVELOPMENT AND IMPLEMENTATION OF A STRATEGICPLAN FOR IMPROVINg THE NAS SYSTEM LIFE CYCLE
This plan will identify the steps required to improve theacquisition of software systems within the FAA. It will identifyactivities such as: (1) improvements to FAA standards, orders, andguidelines; (2) establishment of a software project and systemengineering curriculum; (3) improvements to the requirementsdetermination phase of the life cycle; and (4) improvements to thethe phase transitions during the life cycle. The Strategic Planfor Improving The NAS System Life Cycle will consist of three majoractivities:
1. Software technology transfer into the FAA;
2. Improvements to the software process through revisions ofFAA standards, guidelines, and orders;
3. Continued evaluation of the software acquisitions processand software productivity.
When the plan is completed, a cost-benefit analysis will beperformed to determine the implementation priority for the variousaspects of the plan. This activity supports Goal 1 by focusing,monitoring and re-tuning the software acquisition process.
3.4.1 Software Technology Transfer
The transfer of software technology into the FAA involves twoareas: training and support systems.
3.4.1.1 TraininQ - The strategic plan will first develop asoftware engineering education curriculum that covers all aspectsof software education. Such education will consist of inhousecourses and seminars, off-site commercial offerings, educationalsupport for staff members, and sponsoring of relevant workshops andsymposia. The separate training needs of management, technicalstaff, and project specific requirements will be identified andaccommodated by the proposed curriculum. The various roles(analyst, designer, coder, tester) identified in the softwareprocess assessment will serve as one of the inputs to thisprocess. This activity supports Goal 4 by assisting with thedevelopment of a curriculum to support th. technical growth ofstaff.
3.4.1.2 Tool and SUipprt Systems - Available technologicaladvances include software tools as well as techniques. Thestrategic plan will identify a means of evaluating software toolsand environments that would prove beneficial to the FAA and then
Page 8
provide a plan for making these tools and environments availablewithin the FAA. In some cases, evaluation experiments, perhapsinvolving real projecLt, will be performed to assess the value ofthese tools. This activity supports Goal 4 by evaluating the toolsneeded for staff involved with the software acquisition process.
3.4.2 ImprovinQ the Software Process Through Standards.Ordcrs, and Guidelines
While many of the FAA's standards, orders, and guidelines arecurrently adequate, several problems were evident during TPI'sinitial assessment. The strategic plan will outline how themissing and inadequate standards are to be improved. The softwareprocess assessment may identify additional missing standards,orders, and guidelines. The plan will also address how appropriateinformation concerning FAA standards, orders and guidelines is tobe disseminated to the FAA staff. The plan will also address bywhat process the standards, orders, and guidelines will be keptcurrent. This activity supports Goal 1 by supporting theimprovement, specification and use of standards, orders andguidelines and Goal 3 by elevating the software process to theattention of management.
3.4.2.1 Improvement to Standards, Orders, and Guidelines - SeveralFAA and other standards, orders, and guidelines were identified ascausing difficulties with NAS software acquisition. FAA-STD-026and DoD-STD-2167A were foremost in this area. The plan willidentify the strategy for updating or replacing some standards withmore effective standards. DoD-STD-2167A is now an acceptedstandard throughout the government. However, no consistent set ofguidelines for tailoring 2167A and for using 2167A for maintenanceoperations are available. The plan will include an activity toprovide support and guidance for the use of DoD-STD-2167A withinthe FAA framework. The plan will also direct the development of astrategy for the retrofitting of standards, for purposes ofmaintenance, where appropriate. Others in the softwareengineering field have reported success with retrofitting atailored version of DoD-STD-2167A for purposes of softwaremaintenance [KEMP88] to projects not originally using 2167A. Thestrategic plan will address such issues since maintenance wasidentified as a problem area within the FAA.
3.4.2.2 Specification of FAA Standards, Orders, and Guidelines -The plan will address the improvement and dissemination ofguidelines for using the standards, orders, and guidelines withinthe FAA projects. It should be clear to the staff which standards,orders, and guidelines must be applied to their project and theserules should be enforced. Of course waivers are possible, but therules should be consistently applied across FAA projects. One of
Page 9
the recurring themes found in the use of standards outside the FAAis the use of tailoring and waivers. Properly used, this canprevent the misapplication of standards.
3.4.2.3 Improvement of the Use of FAA Standards, Orders, andGuidelines - After the FAA's standards, orders, and guidelineshave been strengthened, there must be some mechanism fordistributing this information to the FAA staff. The strategic planwill identify methods of disseminating info-:mation about FAAstandards, orders, and guidelines to appropriate FAA staffmembers.
Possible methods for providing such distribu-cion might include:training courses; creation of an on-li.ne database syste.a re2atingthe stages of the NAS life cycle to appropri.ate FAA standards,orders, and guidelines; development of quick reference charts; anddevelopment of a software manager's handbook pLoviding furtherexplanation concerning the use of FAA stancdards, orders, andguidelines in relation to the NAS life cycle.
Another way to improve tf.e effective use of the standards is toensure that the procurement quidelines for software acquisitionincorporate these new standards.
3.4.3 Evaluation
Finally, the strategic plan will direct the evaluation of currentand existing software practices within the FAA. This involves theestablishment and specification of software program management andengineering metrics based on the FAA Software Policy Statementmentioned in Section 3.1.1 of this summary. That will provide a wayof measuring whether or not the expectations are being met. As theplan is implemented, the degree of success or failure can bedetermined by re-evaluating the FAA's software activities andanalyzing the actual results against the expected results.
Page 10
4.0 REFERENCES
[AFSB89] Air Force Studies Board, Committee on AdaptingSoftware Development Policies to Modern Technology,Commission on Engineering and Technical Systems,and National Research Council, "Adapting SoftwareDevelopment Policies to Modern Technology",National Academy Press, Washington, D.C. 1989.
[AFSC89] Air Force Systems Command, "Software ManagementInitiatives Implementaticn Plan - ChangingPerspectives for SoftwarE Development", Draft 23June 1989.
(HUMP87] Humphrey, W.S. and W.L. Sweet, "A Method forAssessing the Software Capability of Contractors",Technical Report CMU/SET-87-TR-23, September 1987.
(KEMP88] Kempton, S., C. Sobell, and C. Withrow, "DoD-STD-2167A Applied to Software Maintenance",Proceedings of the 1988 Conference on SoftwareMaintenance, Phoenix, Arizona, p 159-164.
[NASA89] Documents from the "Ada and Software Management inNASA: Symposium/Forum", June 1989.
[PRESS88) Pressman, Roger S., "Making Software EngineeringHappen", Prentice-Hall, 1988.
[SHERE88] Shere, Kenneth D., "Software Engineering andManagement", Prentice-Hall, 1988.
Page 11
DETAILED REPORT
TABLE OF CONTENTS
1.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . 2
1.1 BACKGROUND.......................................... 2
1.2 SCOPE, OBJECTIVES AND GOALS ............. .............. 2
1.3 TERMINOLOGY ..................... ...................... 3
2.0 METHODOLOGY .................. ....................... 3
2.1 STEP 1 - DATA COLLECTION INSTRUMENT ......... .......... 4
2.1.1 NAS System Life Cycle ............... ................ 4
2.2 STEP 2 - DATA COLLECTION .............. ................ 5
2.2.1 Documentation Review ............... ................ 5
2.2.2 Personnel Interviews .................. .............. 6
2.3 STEP 3 - DATA ANALYSIS ................ ................. 7
2.4 STEP 4 - CONCLUSIONS AND RECOMMENDATIONS .... ........ 7
3.0 FINDINGS ....................... ........................ 9
3.1 DOCUMENTATION REVIEW ................ .................. 9
3.1.1 Summary .... ....................... 9
3.1.2 Detailed Findings . ......... ................ .. 11
3.2 INTERVIEWS .................. ..................... 11
3.2,1 Summar. . ... .................... 11
3.2.2 Detailed Findings ................ ................. 14
4.0 CONCLUSIONS AND RECOMMNDATIONS ..... ........... 14
4.1 CONCLUSIONS ............... ..................... 14
4.2 RECOMMENDATIONS ............. ................... 16
5.0 PROPOSED ACTION PLAN .......... ................. 20
5.1 NEAR-TERN TASKS ............. ................... 20
5.2 MID-TERM TASKS....... . .. .................... 21
i
5.3 LONG-TERMTASKS.....................................23
APPENDIXES
Appendix A ................................................ A-i
Appendix B . . .............................................. B-1
App•ondix C .................. ......................... ... C-i
App•ndix D .................. ......................... ... D-1
Appendix E .................. ......................... ... E-1
Appendix F ................................................ F-i
Appendix G .................. ......................... ... G-1
Appendix H .................. ......................... ... H-i
FIGURES
Figure 1-1 NAS SYSTEM LIFE CYCLE ............ .............. 8
ii
1.0 INTRODUCTION
1.1 BACKGROUND
The Federal Aviation Administration (FAA) is involved in a majormodernization and expansion of the National Airspace System (NAS)in order to meet future requirements and demands. The AdvancedAutomation System (AAS) is estimated to contain two million linesof code. The primary AAS language is Ada. While the AAS is thelargest of the NAS subsystems, many other projects such as DataLink-s, NADIN II, CWP and AERA III are software intensive, and moreand more NAS functions will be automated via software.
Key to the NAS modernization initiative is the acquisition ofsoftware to support the intended services. Considering thecomplexity of the effort, it is imperative that the FAA methodologyused to acquire and maintain NAS software be documented to providea common perspective for planning, requirements analysis, systemdesign, program development and implementation, test andevaluation, deployment, and maintenance.
1.2 SCOPE. OBJECTIVES AND GOALS
The System Engineering and Program Management office, System Designand Configuration Management Division (ASE-200), is responsible fordeveloping and maintaining the technical standards used inacquiring NAS subsystems. To date, FAA has standardized on asoftware development standard, FAA-STD-026 (based on DoD-STD-2167A), and on the Ada programming language. In implementing thesestandards several issues have arisen pertaining to the differencesbetween software development, and maintenance practices between theNAS acquisition and maintenance organizations.
To clarify and address these issues, the FAA has tasked TechnologyPlanning Incorporated (TPI) to review the existing FAA softwareengineering process, standards, policies, procedures and orderswhich are used by the FAA for NAS software acquisition,maintenance, data and documentation management. The objectives areto: (1) identify current deficiencies, omissions, and conflictswith respect to these standards, policies, procedures and orders;(2) review specific NAS Plan subsystems in terms of the proposedend-state software, data engineering environment, softwaredevelopment methodology, general implementation and maintenancestrategy; (3) document deficiencies and make recommendations; and(4) assist in the preparation of a NAS System-level Plan to addressthe identified deficiencies.
As a result of accomplishing the above objectives, the followinglong range goals can be initiated.
Page 1
Goal I
Improve the process of constructing qualitysoftware.
Goal 2
Reduce the risk factors associated with buildingsystems. The risk factors include technical,schedule, cost, operational and support areas.
Goal 3
Heighten the awareness and increase the involvementof management and other appropriate staff with thesoftware acquisition process.
Goal 4
Cultivate the development of software engineeringstrength within FAA.
1.3 TERMINOLOGY
Throughout this report the use of the word "guidelines" is used toindicate any of the standards, orders, policies or procedures usedby the FAA for NAS software acquisition.
Page 2
2.0 METHODOLOGY
In order to adequately address Software Acquisition by the FAA, theacquisition of software must be looked at in the larger context ofthe NAS System Life Cycle and must consider the system levelaspects of the life cycle which directly influence or areinfluenced by software.
The methodology used during Task Order 0011 is described in thefollowing paragraphs and includes four steps:
(1) development of a Data Collection Instrument,
(2) data collection,
(3) analysis of the data collected,
(4) development of conclusions and recommendations whichresult in a plan of action for the FAA.
2.1 STEP 1 - DATA COLLECTION INSTRUMENT
The Data Collection Instrument contains a list of questions to beused during interviews, a NAS System Life Cycle definition, and alist of the current FAA standards, FAA orders, MIL standards, andDoD standards believed to be in use by the FAA for softwareacquisition. This instrument will be updated and modified asrequired during the interview process. The life cycle definitionhas been updated since delivery of the Data Collection Instrumentand is shown in Figure 1-1 and expanded in Appendix E.
Also as part of step one, a memo was written and sent to allconcerned service level managers within the FAA, requesting namesof persons to be interviewed during step two.
2.1.1 NAS System Life Cycle
A brief description of the various phases of the NAS System LifeCycle as defined for this report is as follows.
(1) Reouirements Determination - This consists of twodistinct phases, Concept Definition and Validation andProgram Definition. The Concept Definition andValidation phase encompasses all of the FAA R,E&Dactivities and results in initial product specificationswhich are input to the Program Definition phase. DuringProgram Definition, the Statement of Work, theSystem/Pro ject/Piroyiam Specifications and the Recuest forProposal are finalized.
Page 3
(2) Acquisition - This consists of eight sub-phases. TheProject Initiation phase incorporates the initialactivities of a program. These include contractnegotiations, finalization of the requirementsdefinitions, and production of initial management andplanning docunintation by the contractor. Activitiesduring this phase must set the stage for the follow-onphases with respect to how the FAA and contractor willmanage and control the program.The other seven phases ofacquisition are the standard analysis, preliminary anddetailed design, development, and the three testingphases.
(3) Operational Support - This phase covers the transition ofthe product from development into operational use andincludes two sub-phases, Operational Transition and PostDeployment Support. The Operational Transition phaseincludes those testing activities conducted by both theFAA end users and the contractor which assure thereadiness of the product for deployment. The PostDeployment Support includes the on-going maintenance andenhancement activities which assure continuing operationof a product over many years of service.
2.2 STEP 2 - DATA COLLECTION
Data collection involved reviewing the documentation and conductinginterviews with FAA personnel.
2.2.1 Documentation Review
The documentation reviewed and reported on is listed in Appendix Aof this report. Other references used in performance of this taskare listed in Appendix F.
In developing the list of guidelines for review, a broad list ofFAA guidelines were initially reviewed to determine whether theywere used for NAS software acquisition and software management. Thenumber of guidelines was reduced to those listed in Appendix A.Because of time constraints associated with performing this task,the selected guidelines were categorized as to whether they wereconsidered major or not major in terms of their importance to NASsoftware acquisition and software management. Next the guidelineswere reviewed according to a set of criteria, as discussed below,with more emphasis given to the major guidelines.
To properly perform the quideline evaluation process, a set ofevaluation criteria were established. These criteria wereprioritized to reflect their importance to the review process. Thefollowing evaluation criteria were used in the review of the FAA
Page 4
standards and orders. They are listed in order of decreasingpriority.
EvaluationCriteria Comment
Completeness Are all required aspects of thesoftware development processcovered?
Correctness Are the aspects of the softwaredevelopment process inconformance to guidelines?
Consistency Are the aspects of the softwaredevelopment process covered inthe same way within eachdocument and throuchout thevarious documents?
Clarity Are the guidelines andinstructions concise,understandable, and self-contained?
Traceability Does the guideline support thetraceability of the quality,function, and characteristic ofan item throughout the lifecycle?
Effectiveness Is the guideline feasible forsupporting implementation onFAA projects? Will the processproduce the products needed tomanage the acquisition andassure the quality andcorrectness of the software?
Flexibility Is the guideline adaptable tothe variety of projects andapproaches that will beencountered by the FAA?
Currency Does the guideline reflectmodern software engineeringpractice"?
The review was performed ubing a muilti~pass review process. During
pass 1, the effort concentrated on reviewing each guideline on an
Page 5
individual (i.e., stand-alone) basis, During pass 2, the documentswere reviewed to evaluate the consistency and completeness betweenthe various guidelines.
The first output of the reviews is the summary report in Paragraph3.1.1 of this report. The summary provides a general evaluation ofthe guidelines and an overview of the consistency and completenessbetween the guidelines. The second output of the reviews isAppendix B which provides more detailed information concerning theindividual guidelines that were reviewed.
2.2.2 Personnel Interviews
The FAA personnel being interviewed are from a variety oforganizations which have responsibilities within the phases of theNAS System Life Cycle. The organizations included AAP, ASA, ADS,APS, ASM, ATR, ALG, AHT, ACD, AOR, ACN, AAT, AMC, ACS, AAF, andLOGICON.
The Data Collection Instrument from Step I was used during theinterviews to guide the process and to enable organization of theanswers. As the interviews progressed, the instrument was modifiedas some questions were found to be inadequate or required moredetailed information than the interviewees were prepared to answergiven the nature of the interview process.
The results from the interviews were then summarized and thedetails were collected in Appendix C.
2.3 STEP 3 - DATA ANALYSIS
The data analysis involves coordination of the informationcollected during the document reviews and the interviews. Some ofthis information will be organized using the NAS system life cycleas the guiding tool. That is, the information will be categorizedas to the life cycle phase, product, action, or review that itaffects. This categorization will then lead to the last step ofdrawing conclusions and making recommendations.
Appendix D is a table which lists the life cycle products, actions,reviews and other relevant software engineering topics and showsthe guideline which is the governing document with respect to thattopic. There may be more than one governing document as theAppendix shows. The Appendix does not attempt to list everydocument which references that topic. Creation of such a databaseis a recommended action for the FAA.
Appendix E is the Expanded NAS System Life Cycle which itemizes allproducts, reviews and actions which may apply to each of the phasesand sub-phases of the life cycle. The items within this Appendix
Page 6
are the result of the Cocument reviews by TPI. This expanded lifecycle represents the picture found within the current FAA NASpolicy and process. That is, all of tie documents, reviews, andactions itemized were found listed in some FAA Order or standard.This life cycle musL be tailored for each NAS project.
2.4 STEP 4 - CONCLUSIONS AND RE•COMMENDATIONS
The conclusions were then drawn and itemized. The finalrecommendations will be in the form of an action plan for the FAA.
Page 7
1.0 REQUU4D4ZNTS DETZIUINA'TION
1.1 CONCEPT DEFINITION AND VERIFICATION (R,E&D)
1.2 PROGRAM DEFINITION (MSA: REQUIREMENTS DEFINITION)
2.0 ACQUISITION
2.1 PROJECT INITIATION (MSA: CONCEPT ANALYSIS)
2.2 REQUIREMENTS DEFINITION
2.3 PRELIMINARY DESIGN
2.4 DETAILED DESIGN (MSA: DEMO PHASE)
2.5 IMPLEMENTATION (MSA: DEMO PHASE)
INTEGRATION AND TEST (MSA: DEMO PHASE)
2,6.i . ZSC INTEGRATION AND TESTING2.6.2 CSCI TESTING2.6.3 SYSTEM INTEGRATION AND TESTING
2.7 DEVELOPMENTAL TEST AND EVALUATION (DT&E)
2.8 FACTORY ACCEPTANCE TESTING
3.0 OPERATIONAL SUPPORT
3.1 OPERATIONAL TRANSITION
3.1.1 OPERATIONAL TEST AND EVALUATION/INTEGRATION TESTING3.1.2 OPERATIONAL LEST AND EVALUATION/SHAKEDOWN TESTING3.1.3 PRODUCTION ACCEPTANCE TEST AND EVALUATION3.1.4 SITE FIELD SHAKEDOWN TEST AND EVALUATION
3.2 POST DEPLOYMENT SUPPORT
NAS SYSTEM LIFE CYCLE
Figure 1-1
Page 8
3.0 FINDINGS
This section documents the findings from ..e documentation reviewsand the interviews.
3.1 DOCUMENTATION REVIEW
3.1.1 Summary
The FAA has in place today a baseline set of standards and orderswhich reflect the current policy and process for software lifecycle management. This set of orders and standards must all worktogether as an integrated process and policy. Further, whenevernew standards or orders are developed, they must be totallyintegrated into the existing set.
This set includes as a foundation the following standards andorders:
FAA-STD-005 FAA Order 1800.8FAA-STD-018 FAA Order 1810.1FAA-STD-021 FAA Order 1810.2FAA-STD-024 FAA Order 1810.4FAA-STD-025 FAA Order 4630.9FAA-STD-026
Along with these, a new Action Notice which selectq Ada as the
language of choice for the FAA has been approved.
The results of the documentation review are summariL,, as follows:
(1) In general, there is broad but not specific agreement on thedefinition of the FAA NAS acquisition life cycle within thedocuments reviewed. The terminology used within the standardsand orders is not consistent. Generally, terms and productsare defined but on occasion terms are used which are notdefined.
(2) Some phases of the NAS System Life Cycle are not addressed byany of the reviewed or identified guidelines (see Appendix D).These phases include, but are not limited to, the ConceptDefinition and lerification phase, transitioning fromRequirements Definition into Acquisition and from Acquisitioninto Operational Support.
(3) Certain software engineering topics are not addressed by anyof the reviewed FAA standards or orders. Examples includeComamercial Off-the-Shelf Software (COTS), software metrics,software management, software reuse, documentation management
Page 9
prototyping, and software risk assessment procedures andmethods.Metrics were specifically mentioned in the ISSS TaskForce Report (DOT/FAA-I7) as and area requiring attention." ... a set of metrics...must be developed, documented and usedto unambiguously measure progress on a build by build basis."
(4) The NAS-SS-1000 includes a list of applicable standards an-A'orders for all NAS F&E projects. This implies that R,E&D andthe maintenance organizations are excluded from tl.isrequirement. This approach does not support a full NAS lifecycle view of product development within the FAA andcontributes to the compartmentalization which leads to lack ofcommunication and inefficient development of software productsfor the NAS.
(5) Since the FAA has adopted DoD-STD-2167A as the basis for FAA-STD-026, all other military documents which are referenced byFAA standards and by 2167A itself may be out of date and mustbe reviewed to assess the impact of this situation on theguidelines in use by the FAA. Review of these militarydocuments is outside the scope of this task.
(6) A number of the FAA Standards and Orders do not provide enoughdetailed and specific information to allow a straightforwardand consistent implementation of the stated procedures.Specific examples are listed in Appendix B and are summarizedbelow as lacking in the following areas:
(a) Deliverables - The standards/ orders do not completelyspecify the deliverables that must be generated in accordancewith the procedures. For those deliverables that areidentified, only minimal information is provided concerningthe format or content of the items.
(b) Timeframe/Duration - The standards/order3 do not providesufficient information concerning the timeframe and durationof the activities and deliverables that are addressed in theprocedures.
(c) Activity Details - The standards/orders do not provideenough detailed procedural information to properly supportmeasurement and assurance of conformance to procedures. Theprocedures use activity descriptions such as "coordinate","notify", and "advise" while providing no definitions of whatthese terms mean in the context of the standard/order.
(7) The standards/orders are not consistent with respect to thetreatment of firmware. Some of them state that firmware is tobe treated as software, but most of them do not address thei.sue at all..
(8) "The FAA's maintenance planning tool is the national Airspace
Page 10
Integrated Logistics Support (NAILS) NAILS uses MilitaryStandard 1388 (1A and 2A) to provide Logistic Support Analysis(LSA). Military Standard 1388 and NAILS do not presentlyhandle software, accommodate software support data or enablesoftware support to be analyzed." (WARREN89)
(9) Ada has been adopted by the FAA as the preferred languagewhile allowances are made for using other languages whenappropriate. Language standards which cover each languageallowed should be put in place by the FAA.
3.1.2 Detailed Findings
Akpendix B lists the details of the document reviews by TPI anditemizes any issues which require attention by the FAA.
3.2 INTERVIEWS
3.2.1 Summary
For this task, 31 interviews were conducted, and a survey wasconducted to determine the familiarity of the interviewees withvarious aocuments. The details of the survey results are containedin Appendix H. A summary of the survey's findings is describedbelow.
(1) Of the 81 documents presented to the interviewees, FAA Order1810.4b FAA NAS Test and Evaluation Program received thehighest level of recognition. The second highest level ofrecognition was DoD-STD-2167A Defense System SoftwareDevelopment.
(2) Of the 81 documents, one (1.2% of all documents) received ascore of zero, which indicated a lack of knowledge about theexistence of the document. This document was FAA Order1370.53 Uniform Document Standards.
(3) Of the 81 documents, 9 (11.1% of all documents) received ascore of one, which indicated that the document is known toexist but its usage could not be explained.
(4) Based on the above results, a total of 1- (12.3% of alldocuments) in the survey received either no recognition or l•recognition scores.
Gueral observation s based on the interviews are itemized below.
Page 11
(i) Of the three major life cycle phases (Requirements Definition,Acquisition, and Operational Support), the RequirementsDefinition phase is handled the poorest within the FAA. Lackof adequate requirements definition was stated as the greatestinfluence on the success or lack of success during allsubsequent phases. Few guidelines exist in the form ofstandards or orders which address this phase. For the ConceptDefinition and Validation phase, no FAA guidelines are knownto exist.
The Operational Support phase was said to be the best phasebecause it "cleaned up the messes" and did not allow a systemto be deployed which did not meet requirements (i.e. needs)even though it met specifications. This is more evidence ofa problem with the requirements definition process. Aninadequate job of requirements definition leads to rejectionsduring testing because "it wasn't what was really wanted."Having the vendor build the wrong system, even correctly, isvery expensive for the FAA. These problems should be caughtmuch earlier.
(2) The transitions from Requirements to Acquisition and fromAcquisition to Operational Support were also mentioned ascauses for problems. The transitions are not well defined andno formal guidelines exist to help in this process. A draftaction notice has been prepared by the Software IntegrationWorking Group (SIWG) which addresses parts of this area.Transition is also addressed to some extent in FAA Order1800.8.
In general, the mechanism used to communicate and to obtainagreements about the process during the Acquisition toOperational Support transition is a Memorandum ofUnderstanding. A contributing factor to the difficulties withthe transitions is that different organizational groups areinvolved. Also the requirements tend to be readdressed at theAcquisition to Operational Support transition point eventhough prior agreements and sign-offs occurred during theProgram Definition phase.
(3) One of the points that was mentioned repeatedly during theinterviews was the lack of enough skilled and up to datesoftware engineers within the FAA. This appears to be thecase throughout all phases of the life cycle and at all careerlevels. A lack of well trained prograii managers with softwarebackground or knowledge was also cited as a problem for theFAA.
(4) Interpretation and tailoring of the standards and orders hasbeen difficult for the FAA. Many of the standards nowavailable are new to the FAA, especially DoD-STD-2167A, andexpertise has not yet been developed within the FAA. DoD-
Page 12
STD-2l67A was cited over and over as having problems when usedwithin the FAA environment. The FAA views 2l67A as adequatelyaddressing the Acquisition phase, but 2l67A does not provideenough support for the Post Deployment Support activities forthe FAA.
(5) The guidelines do not stand alone and do not have adequateinstructions for interpretation and use. In general,tailoring guidelines are missing. As illustrated in AppendixA, most of the guidelines reference several other guidelinesand while not shown, these may reference several more. Thelack of tailoring guidelines was reported several times asbeing a problem for the FAA.
(6) The 'culture' within the FAA was mentioned during theinterviews as "getting in the way of success." This may beinterpreted to mean that organizational divisions anddifferent perspectives make agreements difficult to achieve.It also seems to imply that the narrow focus of each group,whether engineering or air traffic, inhibits cooperation andaccommodation.
(7) In general, the personnel do not know how to get copies of theguidelines or what guidelines exist. There may be guidelineswhich are applicable to their area but they are unknown. Insome cases, the people rely upon their manager to giveguidance with respect to standards which are applicable.
8)Another issue which came up during the interviews was theperceived lack of emphasis on minimizing life cycle costs forprojects. Each group appears to only concern itself with itsown phase of the project. There are no apparent incentives toencourage a life cycle view of costs by all organizations.
(9) Some new policies or standards are currently underconsideration. Examples of these include a possible set ofstandards which specify the minimum level of qualificationsfor a vendor who provides training to the FAA and guidelinesfor COTS within the NIAS.
3.2.2 Detailed Findings
Appendix C lists the details of the interviews and provides asummary of the problems addressed and the answers received.
Page 13
4.0 CONCLUSIONS AND RECOMMENDATIONS
4.1 CONCLUSIONS
The FAA has addressed the NAS software acquisition problems overthe years and many guidelines exist as a result of these effortswhich are valuable and which provide the needed support to the FAAProgram Managers and Software Managers. The FAA Order 1800.8f isespecially worth noting for its completeness and detailedguidelines for Configuration Management. Basically, there are manynoteworthy guidelines which need updating in order to become moreuseful in today's environment.
A NAS software acquisition process is in place within the FAA andthis task has attempted to clarify that process and to evaluatewhether or not the process is adequate and effective.
Many of the pieces for an overall NAS software acquisition policyand guidelines exist and what is needed is the thread to pull themall together into a cohesive strategic plan. As a result of thedocument reviews and the interviews the follouing conclusions weredeveloped. They are not listed with regard to any priority.
(1) The Requirements Definition phase of the life cycle isinadequately performed within the FAA. Clearly a seriousproblem, not unique to the FAA, the lack of a sound approachto requirements determination/specification is a source oflife cycle cost expenses to the FAA due to the large amount ofrework required to develop a system acceptable to the customer(AT). This phase contributes directly to the success orfailure of NAS programs and must be given priority attentionin terms of resources, including equipment, personnel andbudgets. There is no evidence that incentives exist whichwould encourage AT to define needs early and adequately.
(2) Requirements are a moving target within the air traffic domainand this is an accepted way of life within the FAA to almosteveryone. However, it is not accepted by the Program Managerswho are attempting to finish a program on schedule and withinbudget against the SOW and Requirement Specificationsinitially bid on by the contractors.
(3) There is no overall software engineering policy statementwithin the FAA which provides the system level guidance forall other software engineering activities within the FAA.Such a policy statement should address critical areas such asorganizational commitments and accountability and containmentof cost along with specific software technology and managementguidelines.
Page 14
(4) A consistent system-level definition of the NAS System LifeCycle is missing around which the FAA can structure itspolicies, procedures, standards and orders.
(5) A glossary of terms specific to the FAA environment, lifecycle and business methods is not available.
(6) Centralized guidelines missing within the FAA for severalsoftware engineering topics. The matrix in Appendix Dillustrates which software topics are not addressed byguidelines. Examples include COTS, software risk analysis andrisk analysis and risk management, software metrics, andsoftware management. COTS guidelines have become especiallycrucial since current NAS projects, such as AAS, have animportant COTS content.
(7) A centralized guideline for evaluation of vendor standards forsoftware design and code is not currently available within theFAA. The FAA requires guidelines as to what is a minimumacceptable standard when evaluating a vendor's proposedsoftware design and code standards. In general, guidelinesare not available which address software technology and itsmanagement in this rapidly changing environment.
(8) Some transitions between phases are addressed in FAA Order1800.8. However, the transition from one phase to another inthe life cycle was mentioned as a problem area during theinterviews. It is clearly recognized by the FAA as such andsteps are being taken to address it. There is an ActionNotice which addresses the hand-off process from acquisitionto operational support which is a start in addressing some ofthe issues.
(9) The use of DoD-STD-2167 (and now FAA-STD-026/DoD-STD-2167A)has been difficult and the need for expertise in thesestandards has been raised several times during the interviews.There is general lack of understanding within the FAAconcerning DoD-STD-2167A, especially in the area of tailoring.Additionally, it is an ineffective mechanism formaintenance/change specification since: (a) the customer (AT)does not understand it; (b) its organization hinders a "clean"presentation of the proposed changes; (c) it does not specifyMMIs well; and (d) it lacks a Computer Program FunctionalSpecification (CPFS).
(10) There is an insufficient number of strong software engineeringexperts within the FAA who could provide the leadershiprequired to manage this complex technology for the FAA.Furthermore, there is a lack of systems engineers whounderstand both software and the air traffic controlenvironment. There is also a lack of Quality Assuranccpersonnel trained in software engineering.
Page 15
(11) Other than an informal network of contacts, there is nomechanism for sharing lessons learned and for knowing what isavailable in the software engineering area within the FAA.Software personnel do not know where or how to get copies ofthe existing guidelines.
(12) There is a lack of incentives which encourage and support thenotion that life cycle cost containment is the responsibilityof each organization in the FAA.
4.2 RECOMMENDATIONS
(1) The FAA should implement new approaches to definingrequirements. The distinction between needs, requirements,and specifications should be addressed and understood. Needsare what AT has, these needs are expressed throughrequirements definitions and requirements are documented inspecifications to be used by the builders of the products.Part of the difficulties experienced by the FAA withrequirements determination may be that these distinctions arenot made and understood by either the user (AT) or thebuilders (engineering). Rather than expecting AT to definetheir requirements, AT should be asked to define and discusstheir needs (i.e. what problem are they trying to solve) andthen the system analysts (i.e. engineers) should define therequirements that will satisfy the needs and then documentthese requirements in a specification.
At the lower levels of requirements determination, such as theMan-Machine Interfaces, a distinction between functionalrequirements and build-to specifications is not made withinthe FAA. That is, the FAA tends to think of build-tospecifications as functional requirements. Thus, whileprototyping is used to define requirements according to theFAA, it often results in build-to specifications rather thanfunctional specifications.
This distinction is important when a program is defined sincethe contractor has more flexibility with design andimplementation when given functional specifications; this maythen require more stringent management and technical controlsin the form of standards being imposed in the contract. Inother words, the tailoring requirements for a program arequite different when the program has functional rather thanbuild-to specifications. Build-to specifications may be moredesirable in the FAA environment for satisfying someoperational needs.
TPT recommends that the FAA analyze all aspects of the wayrequirements are currently defined including who defines them,who is responsible and accountable for definition of
Page 16
requirements, and what is the process used. Although there areguidelines in place for writing System RequirementSpecifications, they are incomplete and ineffective asdemonstrated by uncontrolled changing requirements and thedisagreements about whether requirements have been met duringthe transition from acquisition to operational support.
(2) Changing requirements cannot be prevented within the FAAoperational domain. As with any other type of change, thereshould be a conscious, well- reasoned decision to pursuechanges during the current acquisition, or alternatively todefer them to the future. "Management guidance shouldencourage and support this deferral and accept theconsequences of doing so." [MIL-DOC-11
Within the FAA, changed requirements are often not identifieduntil the transition from acquisition to operational support.To alleviate this problem of late identification of changedrequirements, new approaches to management during acquisitionneed to be tried which will seriously involve the end users atall steps of the acquisition. While the end users are nowexpected to participate in reviews and to review alldocumentation during the acquisition phase, this participationis apparently not effective or not occurring and needs to bestudied as to why it is failing. Guidelines for managingchanging requirements may prove useful. A review process suchas that used for Deployment Readiness Review is suggested.
(3) There is a need for an overall FAA policy statement withrespect to NAS Software and the NAS System Life Cycle. Thispolicy statement should be the guide for decision making andprudent management of software within the FAA. Currently thereare several individual efforts which concern softwareengineering taking place in the FAA and there is nocoordination between these efforts to prevent duplication orconflict. A policy to guide these individual efforts and toprovide a vehicle for communication and coordination isrequired.
The policy statement is not the panacea for problems withinthe FAA but can set the tone and system level guidance for thesoftware standards and orders within the FAA. Furthermore, apolicy statement will not make up for poor software managementor for ill-equipped and poorly trained managers.
The types of items which should be considered by the policystatement include:
(a) A "software first" approach to NAS System Acquisitions.'Inat is, address user needs and functional requirementsand the software approach to satisfying them first
before specifying the hardware.
Page 17
(b) The role of state-of-the-art software technology and
software production techniques
(c) Grandfathering existing programs once under contract.
(d) Introduce the concept of "reasonable judgement" inmanagement of software.
(e) The need to tailor standards for each program.
(f) Organizational and management commitments to softwareengineering.
(g) Cost containment.
(4) Establish temporary working groups which involve members fromacross the FAA organization. These working groups would havespecific mission charters which address very specific andnarrowly focused software engineering needs within the FAA.The objectives of the working group should be well focused onspecific products to be developed by the working group.Products should include:
(a) a NAS software policy statement,
(b) development of a consistent definition of the NASSystem Life Cycle,
(c) development of an FAA Glossary of Terms,
(d) new or revised standards or orders,
(e) Guidelines for evaluation of vendor standards forsoftware design and production.
(f) Guidelines for conduct of program reviews.
Once the product is delivered and has gone through anacceptance process, the working group should be disbanded orassigned another task.
(5) There is a scarcity of systems engineers with softwarecredentials, a scarcity of software engineers with systemperspectives, and few of either with air trafficunderstanding. Thus TPI recommends the establishment of asystem level software engineering group with expertiseavailable to work with the FAA Program Managers and SoftwareManagers. This group of software/systems/air traffic advisorscan observe many programs over a relatively short time span,enjoy a rapid learning curve, and apply lessons learnedimmediately.
Page 18
This group should be available as in-house consilitants to anyorganization within the FAA which requires a..sistance withsystems and software engineering technology. This groupshould be staffed by people with strong qualifications insystems, software, and air traffic control. Over time, thesepeople would become well versed in each others disciplines andprovide the much needed expertise across all of thesedisciplines.
(6) Establish better communications vehicles for sharing "lessonslearned" and for continuing education of the softwaremanagement in the FAA. Some possibilities include an on-lineconferencing system, an on-line database of all standards,orders, etc., an on-line database of software topics pointingto relevant guidelines, quarterly presentations of updatedguidelines and quarterly "lessons learned" brown bag sessions.
(7) Provide a software managers handbook. This handbook shouldinclude the NAS Software Acquisition policýy statement. Thishandbook should cover the what, where, when, and who ofsoftware management within the FAA:
(a) What guidelines the managers need,
(b) Where to get the guidelines,
(c) When to apply the guidelines,
(d) Who to see for assistance with the guidelines.
The guidelines themselves should provide the 'how' part of thesoftware managers handbook.
(8) Prepare a technology transfer plan for software engineeringwithin the FAA. Address the management of technology changeand the mechanisms for keeping personnel current, as well asrelated topics.
(9) One or two pilot projects are recommended by TPI as a way toapply some of the new approaches to software engineeringwithin the FAA. That is, bring together the expert group(system, software, air traffic) to tailor the guidelines fora project. Try other approaches to requirement definition andsoftware management. Other suggestions should be discussedand applied as found appropriate by the FAA.
Page 19
5.0 PROPOSED ACTION PLAN
The action plan proposed by TPI is presented in this section. Theaction plan is the result of all document reviews, personnelinterviews, and feedback received on the earlier TPI reports. Theplan is divided into near, mid, and long-term tasks. The near-termtasks are to be accomplished within the next one to three months.The mid-term tasks are to be accomplished within the next three totwelve months, and the long-term tasks are to be completed beyondtwelve months from now. The phased-in approach of the action plansupports the incremental attainment of the goals identified inSection 1.2.
5.1 NEAR-TERM TASKS
In general, the near-term tasks are intended to fill in gaps whichare immediate in nature and which further define the softwareengineering process within the NAS System Life Cycle. Attainmentof these objectives then enables the mid-term tasks to commence.
(1) Give briefings on this work (Taskll) to all of the FAAorganizations who have participated.
(2) Develop a "strawman" FAA Software Policy statement andtransition this into an official policy statement.
(3) Develop quick reference charts for use by the FAA softwarecommunity showing the life cycle and applicable standards andorders.
(4) Develop a database of NAS System Life Cycle vs GuidelinesMatrix.
(5) Develop a strategic Software Process Improvement Plan for theFAA including plans for:
a. Identifying and resolving conflicts among existingstandards,
b. Creating a framework for practical use and tailoring of
2167A,
c. Creating a software engineering training curriculum,
d. Transitioning development among phases (e.g., entryand exit criteria, traceability relationships,consistency/completeness criteria),
e. Expanding Action Notice 1370.9,
Page 20
f. Providing guidelines for dealing with COTS software,
g. Assessing current FAA software engineering skills,
h. Promoting effective requirements development,
i. Addressing Technology Transfer issues.
5.2 MID-TERM TAS1KS
The mid-term tasks continue the software engineering processdefinition in more detail and implement some of the new processes.
(1) Implement the FAA Software Process Improvement Plan. Basedupon work completed during Task 11, implementation is likelyto include the tasks discussed below.
(2) Expand the Action Notice 1370.9 to address all of the issuesof concern with respect to Ada:
(a) What systems should be excluded from the Adarequirements, if any,
(b) What Ada coding standards are needed,
(c) What Ada metrics are appropriate,
(d) Address any issues with using other languages with Ada,
(e) Address Ada and the R,E&D activities,
(f) Address the Ada 9X impact.
(3) Develop a Software Managers/Engineers Handbook; develop anoutline with the initial emphasis on what guidelines areneeded, where to get the guidelines, when to apply theguidelines, and who to see for assistance with the guidelines.
(4) Update the existing guidelines to reflect the current FAAorganization and current technical terminology. Incorporatefixes for the problems indicated in Appendix C of this report.
(5) Develop a NAS System Life Cycle (expanded version) with theaddition of roles information; indicate which FAAorganizations have responsibilities at each phase of the lifecycle and for which products and activities they areresponsible.
(6) Develup and offer a softwarc engineering training curriculumfor management and technical software engineering personnel inthe FAA, including such topics as:
age 21
(a) Use of Software Managers/Engineers Handbook,
(b) Software Engineering Environments,
(C) Software reuse,
(d) Software risk management,
(e) Software metrics,
(f) Software Project Management,
(g) Software Design Methods,
(h) Style for the Professional Ada Programmer,
(i) Conducting Effective program reviews,
(j) Conducting an assessment of vendor software developmentprocesses,
(k) Tailoring of guidelines, especially 2167A,
(1) Effective Software Quality Assurance.
(7) Develop missing guidelines for:
(a) Conducting PDRs and CDRs,
(b) Evaluation of vendors standards for software design andcode,
(c) Risk Management,
(d) Tailoring of the guidelines, especially 2167A.
(8) Initiate investigation of what the DoD tailoring activitiesare with respect to DoD-STD-2167A and DoD-HNBK-287. DoD hasan automated tool for tailoring of 2167A, and the FAA shouldinvestigate the feasibility of bring that tool into the FAAenvironment. The FAA was a test site for this tool, but itdoes not appear to be in use by the FAA.
(9) Develop a new approach for managing changing requirements;this would first involve understanding the current approach interms of who defines requirements, who is accountable forrequirements definition and who controls budgets which areaffected by changes to requirements.
(10) Investigate the current practice within the FAA with respectto the use of software metrics for both management andtechnical control and tracking of software development for NAS
Page 22
systems. Select specific projects now in progress as samplesto be used in this assessment. Compare these projects againsta set of software metrics which are considered valid bycurrent software engineering practitioners.
5.3 LONG-TERM TASKS
The long-term tasks fully implement and provide evaluation of thenew guidelines and the new software engineering processes.
(1) Establish one or two pilot projects for application of the newsoftware engineering approaches and guidelines and assessimpact on such projects.
(2) Develop an on-line database of NAS System Life Cycle itemsversus the applicable guidelines for these items. This is toautomate the matrix in Appendix D and enable rapid retrievalof this information by the software community in order toassist them with their technical or managerial tasks.
(3) Develop an FAA specific glossary of terms to supplement theIEEE STD 729 which is currently in use by the FAA.
(4) Define and put into practice a procedure for conductingindependent process assessments of bidders and contractors aspart of the FAA risk assessment approach. Establish a minimumset of requirements with respect to a Software DevelopmentEnvironment for various types of contracts.
(5) Develop a NAS project database which provides information suchas what languages were used, what standards were applies, whatdocumentation was required, what tailoring was applied andother pertinent software data. This database could provideguidelines to new projects.
Page 23
APPENDIX A
DOCUMENT LIST
%0000
In
cc N coGo0%0wgr w Vw
00
* 400
1-
4 '1rAU, 4D 44.o c
(D CD q*4m ww .4 fn 00
(Aj
1.4N
A, L
I - -
N) 1 40 w 4cN4 .4) :1 a a4 Q1
0 01
$4 u
C 41
40 41 Is u 0L, 14A . 4 0
li cnC) 0
0I 0.ty 14 A
U0 0 W 41 c " 0 1- *L
V4 fn -4
41'Uc CD. 0 0~i 'JC~~~ 0 C4U) 1.40
44 u M 10w 0
4. ~4 -4 A
0..4 L 0 (Ai L4 w4 14 g.-4 4 II T a0 0 )
0 -~40 04. 40C;001
144. 412 4 >,iJ
4 4 -4
0 0o-
IA0
41
0
0 p w-4 W4
aa
004 C0
1144
~t4~O0 0~t 0 0 0 0 0 e e IV n.- e~~rn fnM~ 'r m~ (0 0w(( ( ( ( ( (0(WO(G
r41.. 14
I~e 0
- ~ ~ ~ ~ q - 0
CD. Q L 0
ca .4 IN IL
0 0 rl
N '4 11)~ 1. 1 4
0in 0 v
4m co c~ CD w~ C a
zN~ - n04 -4
-O~ l 000 - - -
O(QO
000 ))N n 00 ocr oO D 50
_ __ _ -W - - -D %a %aW-4e
0' 44J 4l 44i
V 0N 0N ~4 4 .
0.o 00 0 0 0 a0
a -1
Ian-qN s-- *NV 4 U.14
0 C 40 00
J .14 &J r- -4 14Aj '.
'0 -)~ 441
0s C ~4 s S
4) ~ 0) 0Os 414 44 ~ 4 0C 14W
CL4 Ll 0 i -
M -.4. 4.14 4.
f4.
c, , 1 . C u)
'0~0 % 4 0C 0 --
0 -44.004 41 0 .e-4
ICU -IO -0
4C14 14 w. $4 4.
00 0 0 0 00 0n5
-4 rol-l - -
cn -w I I I I m 4
-J ItwinOC 0
coo Ir 0.D M4 a%
Wa ~ c r- ow N' 01W
Lnl~ Go m NAI af~ w 11 r w0Muin 1-a , c c4 - o 0 _4 4N 0 -, N N -4' .
Q 0 loc Q 0 In oD D o
401.1 10 i Ii!QQ1. -D
4 %D -4 r4. o .
r.~ E 0 -4
441 41 W- 0''-
N. 0
r. 0
00 00 c
44 4.4 u 4 If-
4., C 11c -0cc *
6. 4j 4 4 1
0 0 w wI LI .0 ~~~ aI 0 ~ ' . 1 1 0
4~~ ~ - Q1
I-. 1 0- .4 1.
-~0 0 -LI in 4n
Cc) 0'N-4N .0 0 40
03W W0 *.-
C4 w 0 kl I IA 1 4
'-4q0n 0,(C vmmr( 4 co
d ~ ~ ~ i 0~~. . 0ý . ý4l -'v
* .f0000 a 0 0tC 03 aZU $4 v qW 4ji0
000000 ,* 0000 m 0"0 ooooaa5o0wzzz UQoOoXZ
-40 N
00 a a -o oC 0 0 14
,4 " 0in r-0 E0 IquIn Na
44
jj44 4 U 44 Iwo 0
0(% NC l 0 NN'NN 4 464
41
000 4000 0 00dO 41l0 0.q0u 0 N
4541 L%,q L.00 00 04
0 0
4 C C I0 0 00 0
44.4 'A4.4 .
14 t0% w
w4 0
0 -40(A~
x
0 I C '.0 CjJ
m O WOec g O N~' 0"CI4~
SA
C30-
U C
I - E -i-C C "
APPZNDIX B
DOCUMENTAT ION REVIEW
DZTAIILED FINDINGS
'U C04 c 0G) 14.44 C 04 C.M r-).C 'a %64 0' 40 4 ) .0 0 4 C .4
4)0 4 " ~ 4 c.6 04 F) 41n 14 4j0 45 0 J~.04 4- ,-044. in M G 4) 10. 01 HO Om 44
0r 0 a) 4)ý 039cL. 4)~ X IC 1 .4> 0 * 0 cO 0 14) ' 04* W 0>,01 p 0C 412 0 ?A' 1 a 4 II 31"'c r_ n 44- 4E1) 0 -.- 0 '1 4)w' 4) ,.3. 901'0U.
4) V v .w4 00 U-4v 4r 6 100 .3j c))0 m~.0 Q 4 - 4) .C &1 v r.4).. v) 10 44 j 44 46:
c u 0460 .4 .1 41 - 40 c4 00 4141 -1 -40V ~ 4.3 ~ >.4~- 444 toc I) >4 Cb' 0 r-4 0 W' * 4)u01 ~4 04' .- 4> 1- c) A* 4.44 Uý0 114 1
aj.0~) 4)4).... 0 0 C4 W1'4 LC a-4 tr44 _ . 0 4)14.40.44)1 -44) 4) 0 4 01 4)4 40 0 M0 V~ 0 A. 0 U
0.0 0.-40 .4 r . C4) 4 4140 C 44 14. () 04cC~~~q10~~ 440, 10. U 0 4 )0 - o - 04 430 4 4C 00- >0' isO )4 00 0 .1
44 w U.4 V csJ 0 >.1) 44 -.-.4)C 00- 4. )414 . $4~04 '1- .01- O4.4. 4 1-04 04c .. 4 0 4.4 4) 0) 4 6-
0 )0- .0 w IV 0 *. 4)-4 ' 0 4) 4)W ý 04 1 41 0.4) Om 40 4 E 4) 11 0.0 0% It C.. G.0 .0 .-4 . 4 2 44 4OX . '4
044N Mr.44L, 0.4400'' -4 .0 42 -.0 c ) 0 0 44m N 4 C W-0'4 V4 4)c -410.' -. 0 .40 0 4) 41 .0 30 04) C , 44 a. M
044 .. 004) 44C)440 C 30' 4)14) 44.i '0 4 0 V 0)4 .24 0044Q01 4 4c4C0. 4 ~' Q.4 C) .0 0.00 414 04 41.0 C .4.40
0.0.406)04)~~A 0' 4 .4 >4
4) 440 044 41 x0 AU v. 4)0 0 6 0 0 4)'- 4)- A4 c) 0 3i V C u .. .4140
t i3 1 .404) . .. n 4)-.4 4) 01 00 C W04 'a 4) V 0 p.4 40> j W.0000 M .CX 4)- V0. 4> c0. .0 0 0 1
6-.44.4UI4>-.- 1 4) 4'A "4 1- -4 00' 43 0.0 344 3 * c) 64.) 00
.4C44 u i ' 4 > ow. V. C W .1 & 0 : 4' (1c In C :n C 40 ig E
00 . ul 4)0 v c w4 - (UU- 04) U 4 IV. 0 4, "' 'U . 4 x ~ t) (- j C 4
6. 0)4 c) . &' .(4 4) w -W.044 -
1.2 4) 0 4)0 ' 4 0CA L 0 0 c0 '0 '0 0..In n u0 j c4 4 - 4 12. c2 0. W4) 44
.0U4)L.04) W M .5C r. 5 -%6 &;c4) .0a(L0 0v04 7 1 - 1Wjl y
4 .4 Ic
0
L44-
0 A'In
61
v
0
0
0.
0- 44S0
S0i74 j-
Ci U 0 01 w 'w
0 0 U- 0 .4 &J V4 19V ) 0 00 040 40 54 04 4 0 x0 0N1 -0* x( too4 0 0 0 siJ .i 4.44.
0 I 0 *0 4) jLl V0 13 Aji C . 044 0 0D040 4 "I -i Q'0 IV 0 4 0
-I %41 0 CN 0 EAV' .- u 41 0% 2-. 9 ' v 0V 0 c ScJ 0 '1'4) V4 w) r. E 20 41 -0 4) -0. 0ol 0) -0'1)-' 0 C, Li 4 4 0-i 04.4O L.) *0 .10 ct w 0 -C (0_ uI 41 0
u .0 C ' 0, '4 0 0 ' 0 W ri 4 0 '0 0 V461 ~ C cO 0I( 1 0 ~ ) 2 '4 0
-l 0 CO 3- M4 ) 0 0r-0 '4 -I2*4w 4~ 0 "~ -, V40 C 00- 00 u
04~4 00 .4 0 ai & 0 4 '0 0 0.. )~ ~~~~ 40 Uj LI.-'4 v...0 4 9 ... 4) 0
CL 0 4r 4)0 44-CCuL~1. ~ 0 4 4)a .O 2i 4 'A0 40. -.1 g o V a0 15I
m) a.- tr- w.0 .4 O 0 04.0 -4C C 4)4 44le 4 0 II '0 4 044 ('40 - 4 U.40 LI. 0 .4 0w00
A. 4 0 !.i4i &0 10 ~ 0 04g :gou
c CI C 4 0.0 -, 4 '"0 000 cL V' C- a Uý .4 &j 00 0 02 0') 2 WUL 0b l 0 00 -.44 A1 0c40 4.4m LI 4 4q 0- Li H .4i0 4. C L4 0-I4)
V. 4 41- 41.V 0 4 0 44 0 V0
m M >1 ci-. a Q 4 41 31u4 vi6 m3w04.40 -j 0.4 a c14s
4) . 0i Q.L 0 .4.4 4 '0L 0 E O I . 0 04. .4 L4 4 .i 3 0 4 0 '0 4). LI 4 4 , %4~ 40 ( AI C '
.0 0' 0. '00 04) 4 00 24 0. 000 -44idAa 0 L04C '0 0m 0 a 4 C LI 20. 4 IQ 44
'A 42 .'044 61 0 .1(0 2', 20-4 a i) w/ 004) 9j 00 0'5 4 1. 0 00> v .0 50' 4 4 U' 04-
* 1 0/ 42' 4 004 C-02 j L .ý -. 0" CW'-4 '. 4 00 0t"V ) .. 04LIi .00 v 1 (04 LI u 0 -i 11 n /4. 0 4 -.-ilI39 4) 01 0 0. 0 00 %0 c 0 '0 C4P 0. 4' 140 C 9 4404 04 91 4.)0 0''04'0 4 4 E 00 04'L I >L 04 00. 41 w 4 3
0-' 00 Ca 4.2, ) 0 0 0 40 40. 04)06.4 . 0 4l- 0 4) 0404 IsaL- i0 ' 0'0 04 0'0.. C v0 0 OC'04 nU 0 4) 00 0 .
0.40 M 91 00.- 44 .-.L 0) 0.- ":' 0.0 04 . W A0'c44 064 c2~* Oi A 04 00C V ~ 3C 0 U4i >~ 0Cj
AJ 4JO .C' 0 04) .04 w- > v 0 9 :U ~ -0. 41 ~-4 4W14" . C. V0 (A0 &21 04 U Wwk
0 c i4i> 4 - 6 $4 10 -. 0. .-1 4) W0. 0 0 0m MA .0 r1 r-i .4 0 .00
= ,44 .01 J 1 ) J>4U 40 i-I M i0 >I i-) 1 1@ 1 1 V(U)> ) C 0>
(4 Q4- a 3, (A V .n09 A A u *
1- 04
(14 -W -
41
C.
0
'-4 0
4-
0 0
00w
A0 *040
4
(n(04)
(00
01ýo j(A U
0 .0 .0 - .4
4) 11 ,-4 -O.OC 0 co C. 0 -4 .4.44J 0 V)
U J 41- '44 444 -4) 0 CO 44 wU0 UCU' 04
C4) CU c) 04 4 04 4 4 m.00 0: 4- 414 L C )
4 0 a1 .1-4 1Q -0 li4441C C V) C:4 41 0 c 004 W C
0'44 1 4 0 0 )4 01 0 ) 4 .
441.4.'4 04 U . . ~ 4 4 " 0
U0' w4 rC -I 0)~4 W 0 '9 P'4) ..- E1 .0 -. 01 *U V)'4 44.4
2. .. 41 4 ~.4 4 -. 44 C 44C 4144
14'n m (A -40 0 4) 11 2 4a, 4) 04 10 CO V: 2c 0 %W
4)0 49' -~~~~ v U-4. 0. I C (1 1. 4 44
C-I 04 4) M0~ 0 4%4 44 4) -4 C 4 C
c m.00 b) 041444" 41 a) 4)4-4 4) '44 4 4.*U) U0 W (110 (A W.2)f CL C -U p- La 04)101 C>. aO -4n w -C I's . 1) 4b0 C0 fa 14 ~ Z V4 %A4
0 34 G C u1 4)00 =.- M44 C) Ir41 .4V U 0)
CA. C 044 1 4 U -C 04)0 2 0 4 w4 c' cu .-a1 E-. .00 c4 $4EOl 0.U~ ul M- ..- G)4 4 . 0
0..4 c $4 r-C ,4 4 .41C4). 4) .4 C 4) -4.
0) 4.- 49 W4 -,4U E 1 V4 .14244C 4 14CO 4 )
04 C 0- 0 0004 E~1) .442 .*0'441 4444 ýC4 a4 4 04.4 IA 0.)42 M. UV 0 4 44 40
Cr 20'. u 040 -. 0 444 4 Co La r 4) G40 V ) v )
a-0 14 0 u44a .1- ON444 4) 0 '40414 x 6 46 0 0O4 644*ý )14 a441 4 E4)0 4)0 Im C4400 V 4)4 V ) 44 .4-
01 C6 IL - Cn a))04 c14 4 c 440.44 4) wV I 0. 0 w
0.4 .9' ~i .. D' 4 0 0 r4 0 1., )0.4 j 4- 21 )) U
044ti 40 04 4)0 .4 0x 4. 0 44 44 4C 4)C E )
.C'4 o 0-)- 4)44 :3 4)4)'0W ) 4 )-4 0 . .014 ))) CCiL a a ) 0 44 00 0.4 0. 4
0.- 'A V)- 4) 4444C> C0 . 4) 4))44 v 44u.4 '.4 a4 C'4 '. U j4 0a 0 9' 41V9 4.0 4 in 'O W0 I1)14 44 04 4, 0) =0 mo. 4, 0
4z C E rMV 044 4. u344 uOC4 -a a44)3 c)4 w4- c4- 64 6 .11
'n.- a 0 :3 >. Q 44)441 4) -" . 0 L4 Wp411 '0 4 .cc 0 4, 00 CL4 4)44 90Z
v) V4 41 w w114 m c '444 4144Q 410 4) 4) W W 14144 V 4 44 0- 4 CC 04 C u1
w1c a4) 4) I I 0) V44 O O 4 ) 9' 04 0 4:4 000 . 44 . 4 U * ,1 4) aC 10 XU)) .4 4) 9 4 ) a 44 44 4)> 44 414 544424) 4
C~~ ~ W)4 V. V 14 C 4)- 44p t46 4)4 )0J
01 'a )9 0- 01 1 c4). 1 1 1 U4 4)- V P. '0 . 4
v10 .4 )0 044 0c )0 1 00l 00) 44. C. 4) 444)U 4 4)1 : 44 S4 144
U4) . 4 /144 4/10 Q 0 3V .4 :; ZA 04 > L. 04 0.4 0 w 4) 00.
"O W4.)1:- W 111 cwý V 0ý 4 )
o 0- a, 04 'a 0 IN) - 40 4 0 a v I44 0 r. cI U) Id 0 1 o w4 p ww w
444
!. 0.
.- 4) . 4
40 4
44 U
c40
-4 4U 41
00
'-424
%o 4 u u j 4, L4In 42 2 4 21 2.2
54 IC SI 4"2 0.5 w1 a,300
'D 0 SC ý 0- , C .. @C,S'A 14 02 - "~ CC S Aj 0~ E 1-4~
V491. CS 1%4. q S
14W~ 4240 14U vi a I dhv 42 OC 8C I r-OS wJ "C
SI~ ~~~ SI .- ' OS .0 64 CS b40S SU C2 CI .42SI 4j '4 C41 00 CC CBc4 42' ce w i "
L.' SI - 08 mu o. t gSI SI 012C C42 S
U L4 0I 4 1. W SIj CO 0C4 . '4S -P1 pI CS) to 4)0 2) 0-4 42.24 '14X 12 I Ij 4j 42j4S1 o-/S U' w 0 4 SI
4j C AS M ' 164-4 93 M to 2 C 42 4m Q- c1 0' 12 A2 SI W~ > VI 4C 14 0., 42
c 4)1 . 4'40 BiSb. -0 -'S 4SI SI.- 1 1 ., u L0oIS -o O4.8 o * j c -46 S
:3ý. ý VC VQ 4 0 54) C1
4 r.2-V .L140 4- CC SI 41 4 w a 9. 1 40
SI C SIIm 0SI~ 0.0 24) -0 0a4 C .21 3t w SI42
02 1. %SI %.4 o4 0) 404 a C o'0 w2 j0 01 3. SISI SI 02I JS V2 b- a"4 ='0~ ~ 0.. S. c S -SC SII.1S 0 e 22 0 04~
C 0 0. 124 42SI4 V0IS 4 1)0 1442C 41 - 0- QSI 12 124 -4:0.44 'Ell 'C 0S 0 -4 .1 I
%2 4j0 C > 21 0 2 4 I 4 44.1 042 0I- 14 0 4UC4C 122 6 I- 424 L.~. '0~ " 4z 42
'0 C' C 0 C0 VS SII OSSI2 c. C 4) SICSI 4). a- SS -C 64 V SI3 C .f V O -.4SI CI @0 .4 c m 0 c
0 L M2 4) &j2' _S SI4 12 w2. C-S OWwC SI SIS $4 v1 ý 514 0% 0- o2 1 C. 4 c
o 1244 W04 3C12 1 124 fl 0.~ *V 044 -. 144. WCS -442 0S -014 0 Z 0 0 00 S4 Cj >u 0' a1 SIv 31~
0. 4 S- SI SI 4 .- SI-4 j - '0'4 4jSI.-4 I 120 -012W c2 342 c21- 1244.q c2 C r c
0 ~ 4 021 02S C0. CIA 2112 24 04 14 40O 00 so4 f~ . ~ 0. 020 W ) 0'04
U. 81- s .u s 0 O
C L: 2 22
U ~~~ 14 - 0 0 ~ C SIC u C
4/I 14C) 4U)
00- 0
MI- cS
201 '0
1214C12S
C a 0 In 044
Z24 4 4 v/IC0140.10r SIf
SI120 D.I 0
0. w2 01 -. 0.c 42C C 0 f
InU~ 421."
I -6
-r 11 la.- Ea o~ o4' 4120. ) ~ 40 c 41 N~ 4 4@j 8111 C
04m 01 W, -4 00%~40 .. ~ 54 4
44 ~~~~V 0 2 Q X .. *-C40 4U~ ~441 4 10 C'- 0. 04 u 40 log, 0 - ' 0 14 8
'o 444 4D W) 04 a4- 0. *-C QL
44 .0 . V w -. 410. 0 U4 0 .4. 4.. 0 .
.4 V 14 0 0~ 4.)4 4 0. - C 0 04)8 IC ~ ~ >). v 0ý 4 I- w- - 44 03 44
4-.0 4.40 -4 C0 vC ' - )0 .4 54)U.4 0. Cc a C -4 0)I4 UU .
3. f 4)4 0a 4C4 9A4 %4 I 44C (1 o'0-4 0 -4481 4E. 44 c 5 m 4 C: 440 C:44 4 .444 .0
(A 41 It 0 4
44 IV 4 4) W AJ 14044'0 4) .4 ' 4 4 C 4-- 4) 081 10 ~ 04100'C) J - $8cr1)- 0.00 0 0 40 o 1. C 4 )
j4 f.) 00 0. a 0 .- 0 4c'.' 44 44jinV 81 4 C. II 4 4)4 814 Cl 41.C 1
14 z4 I 0 (d .S 0 V".4 041:)4J bi4 E ~ 0.0 $44 U14 - J 40 14w
L. C 04 "-. C1 L- 0.4 0. 4 H -4-404C t.. -044 41 M a44. 4:00C J -i II 0 %4 $4 c -M .
00 cc 40 44 C0 C6 0 0 C3 c- 0UC 44 .C 0 1 41 11 4C 81 tU 4)-4 A .0 r .10-IC-C81 0. . 4 0- 4) 44 -44 A) C44- CC. UO
c 9C.g 0. M 440. 4) 4 -4 4) C14.4 C afl41"4#I4 M1 L44 0.! - 4 81 41. CC55. 00 1
0.'- M4 0 0 04 4) 0. 4041 x C;4in 10C -0ICIVw0 41 4 4 44.0 .i5 C V 0 -444 O
0'.A W4 V 0 Q.04 & -4 > 4 w1 ") > 04 Q0444 1 4) m10 C Q1 4 0 0) 04 z 0 - &) 4
r.4 4. u -u w) 0( c4 4C4- w41 414 m v 44 L. 93c C V-2.' 0 cI .204 ' )Q O O 4 0 0 0.40 v-4OW
f - 11EX4) C.4 01 84) 'C) 4o ()4-a) 04'1 $4. 4 . 44. 81 ('
%v81aC 44 .Uý4 41 44V 814 aCC OD4-- W.- 0.V -0.C -4 4J " W 0B0r4 4 ) ~0, UC- 40 :3 0 * 0 2- 0 4.-'x 0'0 28. w 81 (D
44 104 814 1 81 81.4 045 41 1 04a1OU '
40U . 0. O 84) 4) 4)0.' 0 vi -0 0 4444c " 0
44 '00 004 Al m0 w -40 0'4 a ~ 0 40 w05 $4 40UCa.4 4U W4 0-40>. 6 (A W. 01 4 0. 41 004 U O.4 w u
2i 4) 1 41 0) w :)Iv 4)1 D0A ' 4)~ wC . a C>4V -1NC0 V 0 4V 1 440 0
Q)J 00. 0. 0. 0.0mm 0- nw z 0j 0e ý
CL z C
-4 04J0
;44 cU 01 1 ,1- L 4k1 CY d 40 1 4)4 v v0.C
4 0 00O
41415 0 U4
'~1C0 (44 -A 0C 4 )4 ý4 ~4 4. %4,
). 4)) 4)0 ' . 0.4.. 0 ~ 4; a
0 81 1 0 0.
200 IV 04 V 0. ' 0.
0 0040L
w Q 0 0 0 14 0t 104 4 -1
N 4)
-4 0 $4 014 C -4 0 -H
c 0 -0 c 4. ju
41 0 c v v v -4u w a, c c 0
4) ty, 31 14E
Q w VM 1) 1 0 n
M 7a E 4.4 0 Aj 10 " 0 w 0'A 4. > v C) E C3 0c 4, F 0 c 0 0) w -0 " " I0 4) m Li 4 0-0 41 C j Iz -
%. Q. w -0 W) 6. &ý 0 ý aw oj r- 0 c 0 0 4) 0 0 C4
F W Q0 c 1
u a'M C CIO U -4 %4
W fa
.c 91 j 4n Q4) 0
co CL V 0% 0 4 _(n1, 0 15 64 41 %4
914 M C. 0 "r-CA E fo W Q 0 Ajw 0 V to 4 wav % %W a =
0 Im ý c u w x a c 4) a 0 a Im%W X 0 r- %4 440 C 10 0 0 UN Aj
DIX MV Aj 41 4w 0 r 0 0 fn2 41 V 0 a V c L, 4.1 -4 u V 4 -
U l c to 41.0 10 -5 L4 31 AJ 'a
u 4) M c m 0 4) IN & c 0 v 40 c0 00 -1 0%1u V 4) 0 tr 4 c a, 10 c V., 4 . . 1 0 41v 4) v 1 14 0 m L4 0 -0 14 ý W W
0) c 'a c 4) 4) 4) 0 x Q. = 0 31 Q 0 :3 44 Sý 'ý4n 0 T oc C ý 0 '1 E j j V.0 6. 0 u 4) 4) -4
4jV - c U 41 W W u 4) 4) 41 44 4) tn 0 0 $444 0 U0 , , > 0 Q (A 0 X v
0 0 in 41 0 & 00 04j 31 w 41 v 41 010 9 o- $1 0 W AJ 4 a o " w 0
0 c 4) CL c > c Q. .0.0 1. 0 w it a in L, 16 In >1 4) .6 -4 C ý4 cr Q 0.13 V4 0 4) z U 4 4) C6 4) IJ 14 L. 41 Aj
0) li E u.C 41 t) 0.0 4) 0 %W 0. $4 $4 3 4J 0 CS v r- - - - 'I Q C'a 0 0 0.11 v 0 c j - 46 0 W 0 a Ic lo- 10 44 06 CU %4 -. 1 a., v .
c - c 10 Z 41 0 0 C444 VI 4) m a fj 4 W 0 4) v 000 0.0. 'A 0 0. E 'A 0 0 r 0 L. 4 W 1., 4 W In 14 WV fý 0 U
r0.14 E - 'A c 4) 0 V 0 4) m >W 'A 4) In v c 3 V -0, 0 0 T 10 0 4; 'A 4)1410 a 0 4) 4) 4) 0 4) 0 co 0 m V li E - ý4 La0. ej -0 r-- u u - c V M w 14 IA 41 %. ;1 0 c A c eq 4) 4 4 E E 4
Q .0 c r c m w 0 a 0 41 0 G C 0 0 0 a a 41 0 It'A 0 0 .0 m V A
14 n c 0 ý w '. , . E .1 0. c n 4' 1.4 44 '1 a 914414 0. .0 W W M IV M I.. - 10 0 0 0 4) 0 0 E-- V 0 0 'a I" >. 0-A *0 v 0 u -0 iv tn c -a .0 p, tr = m ý zwou 0 w 000 IA 0n ýj rj c 0 0 c 'a c , u C6 0 j loý a)
M C 10 ý 1 7 10 t3,14 ýj 'A CS j co :." 0 C) v 10 1- a 2 F- i 11 li 0 41 t.., cn c: tA 4) 1 1 a I i 0 1 1 6 0 9 1 1
M 41 .0 in u z V -C 40 W 4) 4A 0 0 w 0 13 OVA U -4 0 U T r. r
zdc cn ff.
z m
c,4 w Ll 4) L,40 & 1 4) E lu Q v
u c c c c c r- E . Iw 4) 0 4) 0 0 0 > uCA 0 L) U u u 00 u ov uo u u 0 u
E'a
0
4) 10
0 cow M L4
2160 126
L4 x0
0 S.CA c 0
E E w 64 u0 41 ii 4
c 00 W 0 u
CP w u m u
(A3
Al
M 0
a, 23
4.1 ' 14.114 )1 41 4-44
4 1 ' '0 41.44.1 414 C 40 0 r -. 4 C1.- 04a C
.4 0 414 4U~ 0 IV4 441 c4 cC 0-0 vi . C. 0 4' 0j. >
I.- CC.. AW1 cf .21. r0C .4 40,11. S9aC.~ IA4 4 0 0 -
.4~ 41 - 14.41- 00. n '.' 00 (A.4 V- W.4 4 0~ v C 0 44 CO4.1
-. ~~A C4 C '-4 40 44f -0 n b4 t 4.4
14JJ- 0441. in4 c34 40 .0 0 M4414. 4. . 0 ~ ~
p M4 bi0 Q. - 44 0 40 C a -4 r a 0.0 0- 0 4.1C14 .41 C4 4; %40 401 001 20 0.ui 1410 06
0. 0 .06 004 4 4 1 C) %D-4 00 2c cc,0 40 4-1-1-. C0 0 ' 0 C %& -4 40
@--- 04) 40 .0 w4 0V I 0 440 c01 OD CE .1*- 0 ~1., b U Q 0 04 0 4 4
2~~~~~ ~~ 4) .4n 0.4 44-4A1 C 0 O~44) C. N'C' '" -.4 C4 0 o1 -1
u W 4. ".1. CDf0n (A41 A 4 .. 0.
04 1 c0 0.-f0 41 44'J 4, 4 0 4 O'A.0 ., 0 .- 0041 .11 1 'a'a;- 41 - .' - 9.4f .0-
C1) 4 W 1 4.. T, 1§ 19 u41 0 04 '0 . '%414 O4 M1 4 f
1f -I 4) I .0 "
'004 r4 . re~. 0. wt) 44- 0 4 U1.44440. "41-4 It4 44 20 41.. 44
C4 43 04 - 1.345. 41 .04 . 4)4 C C w4-4 414 1) 1 40 C140 -44 %.0 AJ 4) 4
A4 4 4A U- 44 .441 2 aJO04 0'00 410404 4. 0 u L IO 44 4 0
4 -4 C. Id V44 4 w0 AC. 4C .1. . 0044 .. t0 . 4 4) 14 0 1 (4 .4 -4 'M4-12c zu 204.
444 4 1. r0 4 A 6-. 10 U 440 >44404 00I .04 W 04 Cf t) 41 WI
(NC 0 0 04.4 m1 (4OI 044 4j >41C A04 0 4 44 I'C'u .41 40 0 0C c cO
04 000 44 V4 0 0 41 4 c. -. 4 4 .0 W%40a- 0-C 4 ) 1 .0 1- a 44 '04. 0~0 4 0 0..
'4'a 42 C-1 CCO41 44- .- )0 CUr. -'A Q44 4 D04 40.I 04 G0o00 4"'410 44 .0104. 244. .0 41n o 0 0 .4u0w
000 4 44 44 otn 01 -f J 1 4 l4 u0
Q -C
o1 40 C 4C 40c 40 40 r .. -
0) CE E C ECW4 4) 0 40 40 40 40In us 00 00 00 00
01.v4
C0 4
41W
- 000 20 .4
00 0 a
C 44 4.1
01 1)0ý
-. ~00
0 4
a- Cý Q4
LO0)
)1'PZNDIX C
INTERVIZWS - %ETAILZD FINDINGS
C.1 INTRODUCTION
This section presents the results of the interview processes.There have been 31 separate interviews for this task. This sectionpresents the data gathered during the interview process along withanalysis of that information. Because of the organization of thissection the same issues appear in more than one section. In manysections, a summary of the issues identified during the interviewsare presented along with the total number of subjects raising eachissue.
C.2 PROFILES OF SUBJECTS
The subjects interviewed during this study represented all aspectsof the FAA's development cycle. The subjects had between one and20 years of experience with the FAA and were located at both FAAHeadquarters and the Technical Center in Atlantic City.
The following personnel were interviewed:
Chuck Bolling AAT-14Ralph Caprio ACN-310Jerry Champion APS-410Ken Clark APS-500James Clinton ATR-250Steve Coulombe ACN-130Lconi Czekalski AMC-300Vern Edwards ADS-120Dennis Emerik ASM-160Robert Erikson ACN-210Don Espinosa AHT-400Mary Ann Farrell LOGICONJohn Hamilton ASA-130Joan Hannan AAP-120John Horrocks AAP-320Willie Hunter ASM-140Harry Kane ASA-210Rick Lay APS-300Garry Long AHT-500Jim Minsterl ASA-6Jim Monnie AAP-400Harriet Neuman AAP-220Jacques Press ACN-110Bill Riehl ASM-160Steve Smith ACS-320John Timmerman ATR-210Gonzalo Tornell ALG-410Robert Ulanch ACD-340Jim Warner AAF-4John Wiley ACD-350Alice Wong AOR-110
Page C-I
C.3 POLICIES, STANDARDS, AND ORDERS
C.3.1 SUMMARY OF SURVEY ATTITUDES TOWARD EXISTING STANDARDS
In this section the results of the interviews regarding existingstandards are documented. Where there were a small number ofopinions concerning a standard, it was felt that this implied thatthe standard does not pose a major problem. The interviewees werequite emphatic when discussing a standard that caused significanLproblems.
C.3.1.1 Positive Opinions
The following standards were mentioned in a positive sense by thesubjects. The number of subjects that mentioned the standard inthis sense is included.
FAA-STD-016a (1)FAA-STD-018a (2)FAA-STD-028 - good, but hard to understand (1)DoD-STD-2167A - for development (1)DoD-STD-2167A - for maintenance/replacement of
systems (1)FAA Order 1100 series (1)FAA Order 6100 series (1)
C.3.1.2 Negative Opinions
The following standards were mentioned in a negative sense by thesubjects. The number of subjects that mentioned the standard inthis sense is included.
FAA-STD-013 (1)FAA-STD-018 (1)FAA-STD-026 (3)FAA-STD-028 (2)FAA Order 1810.4 - too difficult, need help (1)DoD-STD-2167A - too general (5)DoD-STD-2167A - vendor problemsDoD-STD-2167A - for maintenanceDoD-STD-2167A - produced too much documentation (5)DoD-STD-2167A - for development; assumes good, solid
requirements (1)
C.3.2 CONFLICTS
One subject thought there might be some conflicts with some FAAstandards and DoD-STD-2167A.
C.3.3 DISCUSSION OF SPECIFIC STANDARDS
Page C-2
This section presents the issues pertaining specifically tostandards. Included are the number of subjects associated witheach issue.
C.3.3.1 FAA-STD-013a Quality Control Program Recruirements (1)
It was reported that FAA-STD-013a has virtually been superseded byFAA-STDs 016a and 018a and should be either updated or eliminated.
C.3.3.2 FAA-STD-016a Quality Control Program Recruirements (1)
It was reported that FAA-STD-106a is up to date, with the possibleexception of some terminology, and is effective.
C.3.3.3 FAA-STD-018a Quality Control Program Requirements (3)
Three surveyed had an opinion on FAA-STD-018a but there was notgeneral agreement. FAA-STD-018a was viewed by some as moreeffective than DoD-STD-2168, and in good shape and up to date (withthe exception of some terminology) . However: another subjectthought it was too vague and not detailed enough. No one describedthe standard as poor, but some feel that it could use more detail.
C.3.3.4 FAA-STD-026 NAS Software Development (3)
FAA-STD-026 was viewed as an obstacle and source of trouble. Itrefers to DoD-STD-2167A and includes so many cross references thatit makes tailoring DoD-STD-2167A difficult. It does not standalone and provides little assistance in the development process.It is out of date and should be reworked or replaced.
C.3.3.5 FAA-ZTD-028 Contract Training Programs (3)
Opinion was divided on FAA-STD-028. However, all thought it tooka great deal of effort to usefully interpret FAA-STD-028. In spiteof this, one subject thought it was a good standard while the othertwo thought it was not good because of the difficulty ininterpretation. Differences in interpretation within the FAA occurbetween headquarters and FAA Academy training personnel. It seemsthe common denominator is that FAA-STD-028 is not clear enough andpossibly lacking in sufficient detail.
C.3.3.6 FAA Order 1810.4b FAA NAS Test and Evaluation Program(1)
One subject reported trouble with FAA Order 1810.4a. It was statedthat this order was not understood by staff in both the TechnicalCenter and FAA Headquarters.
C.3.3.7 DoD-STD-2167A Defense System Software Development (13)
Section C.4 below addresses DOD-STD-2167A in detail as a specificproblem because it was the most widely discussed standard and the
Page C-3
one that presented the most concern to the subjects. Based on theamount and strength of the responses, DoD-STD-2167A seriousattention by the FAA.
C.3.3.8 FAA Order 1100 (1)
One subject reported that the 1100 "series" was very useful.
C.3.3.9 FAA Order 6100 Quality Control (1)
One subject reported that the 6100 "series" was very useful.
C.4 SPECITIC PROBLEMS
C.4.1 DoD-STD-2167A
DoD-STD-2167A was the most commonly mentioned source of problemsmentioned in the interview activity. Ten out of twelve subjectsmentioned DoD-STD-2167A specifically in conjunction with some formof problem. Specific problems cited with respect to DoD-STD-2167Awere:
Too hard to use;
Need help in understanding DoD-STD-2167A;
Needed help in tailoring, and FAA-STD-026 got in theway during this process;
DoD-STD-2167A was misapplied;
DoD-STD-2167A documentation delivered, but not used;
DoD-STD-2167A resulted in too much documentation toreview;
Although good for development, DoD-STD-2167A documentsare not suited for maintenance - too hard to specifychanges;DoD-STD-2167A not adequate for interactive, real-time;and
DoD-STD-2167A loses the Computer Program FunctionalSpecification (CPFS) concept, and the CPFS is importantto the FAA.
The majority of those interviewed did not understand DoD-STD-2167A.They hcd little or no training in this area, They longed fortailoring guidelines or a group to help them tailor it. Tailoringwas viewed as critical, and without it the standard would be, andwas, misapplied. FAA-STD-026 hindered tailoring efforts because of
Page C-4
the numerous cross references to DoD-STD-2167A and other documents.
DoD-STD-2167A was viewed as producing a set of developmentdocuments rather than a set of maintenance documents. In additionto tailoring, the second major problem with DoD-STD-2167A was inusing it as a maintenance document. It is hard to specify changeswith DoD-STD-2167A since they are often scattered throughout thedocuments. The end-user (AT) cannot understand changes specifiedusing DoD-STD-2167A. Most interviewed preferred using a CPFS forchange specification. It was also stated that specifying Man-Machine Interfaces (MMIs) was difficult with DoD-STD-2167A. SinceDoD-STD-2167A documents are not maintained by the FAA, they getincreasingly out of date with time.
However, most users felt that with training, guidelines, and amaintenance CPFS, the use of DoD-STD-2167A would be effective.
Note: FAA-STD-018 was viewed as more effective than DoD-STD-
2167A's companion SoS-STD-2168.
C.4.2 REQUIREMENTS
The problem of Air Traffic (AT) not accurately specifying oragreeing to requirements was the other dominant theme that surfacedduring the interviews. The problems cited were:
Cannot get an firm requirements from AT;
AT does not pay attention until they can see it;
Lack of adequate guidelines for requirementsproduction;
AT does not follow the rules with respect torequirements specification; and
Projects fail at the testing phase due to vague or"changed" requirements.
Virtually all interviewed, that had an opinion, felt that AT, theend user/customer, did an inadequate job in specifying or agreeingto requirements for FAA projects. Various reasons were offered forthis problem: too much turnover of user personnel; lack ofguidelines; too little time and budget allocated to the task;congressional specified deadlines; and lack of enforcement of FAArules. Some stated that AT is incapable of developingrequirements. The result of all this is that projects fail duringtesting due to requirements that are either vague or no longer whatthe customer wdiiLs. Some felt that prototyping might help,especially in the areas of MMIs. If done early and even separatelyfrom the project's main contract, it might serve as an aid to
Page C-5
requirements specification, This might solve the "AT doesn't knowwhat it wants till it sees it" problem.
C.4.3 PROBLEMS WITH STANDARDS, ORDERS, AND GUIDELINES
In addition to the problems stated with DoD-STD-2167A above,several other problems with standards, orders, and guidelinessurfaced during the interviews. Specifically mentioned were:
DoD-STD-2167A - addressed separately above; (13)
A lack of guidelines for dealing with COTS software;(5)
The high cost of retrofitting new standards to projectsalready underway; (2)
Some orders are out of date with respect to the FAA
organization; (1)
FAA Order 1810.4 is hard to understand; (1)
A lack of coding standards for contractors; (1)
R,E&D organization doesn't follow any FAA Orders orStandards, thus mismatched equipment and systems in theNAS (1);
Difficulty in keeping up to date with FAA standards;(1)
There are no policies or standards with respect to airtraffic controller training on new enhancements, thusquality varies greatly (1).
The DoD procurement approach caused problems withgetting test plans too early; and (1)
Management was resistive to change - hard to getstandards and orders updated. (1)
After DoD-STD-2167A, the most frequently mentioned problem withstandards was missing guidelines for dealing with COTS software.Issues mentioned were: definition of a COTS software, how to dealwith modification COTS software, and what documentationrequirements are needed for COTS software.
Some problems with the management, maintenance, and distribution ofstandards were mentioned. Some ordeis were not kept up to datewith changes in the FAA organization. Others cited managementresistance to change as a reason for not keeping standards up to
Page C-6
date with the technology. Some of the interviewees citeddifficulty in finding out which FAA standards exist and areapplicable to their tasks. Keeping up to date with changes wasalso cited as difficult.
Two areas were identified where guidelines were missing. Codingstandards exist within the FAA but are not applies to vendors.This results in code that is difficult to maintain.
Faa Order 1810.4 was cited as difficult and complex.
A case was cited with the DoD procurement policies which resultedin test plans being developed far too early.
C.4.4 FAA ORGANIZATION
Some problems with the FAA organizational staffing and resourceswere noted:
Too compartmentalized - lack of communication betweengroups within the FAA;
No group to handle project interfaces - lack of overallsystem engineering;
Difficulties with lack of trained software people inthe project and other offices and difficulty inmaintaining software project skills in the projectoffice;
Lack of emphasis for minimizing life cycle project
costs;
Lack of software skills within the QA staff; and
Wrong group is contacted by Program Manager whendetermining requirements for standards documents on aproject.
Two issues concerning the FAA's organization and staffing profileshowed up. A lack of competent software engineering staff in theproject office, software support and QA function was specificallymentioned with some mention of the same problem in most all otherareas. The software background discussed covers: life cyclesoftware project management and software engineering.
Even if the problem were solved, it was believed that individualsin these positions would either lose these skills or leave thesepositions. The possibility of a "rotation shift" was mentioned.There was some mention of training, but some felt that hiring inthe necessary skills would be more effective than training existingstaff members.
Page C-7
The other issue concerns the life cycle cost of FAA projects. Itwas mentioned that not enough concern for the life cycle cost ofthe project was shown in budgeting effort allocations to thevarious phases of the development cycle. A lack of concern for thelife cycle cost, especially in the earlier phases of thedevelopment effort, was also cited. The Faa was described as toocompartmentalized, which may correspond to the cited lack ofconcern for life cycle cost.
Another issue raised was the FAA's apparent approach to projectmanagement by letting standards, rather than direct involvement,control the project.
C.4.5 QUALITY ASSURANCE (QA)
QA was mentioned, but no solid concerns were evident. The lack ofsoftware engineering skills by the QA staff, as well as not enoughearly involvement in project, were both mentioned.
C.5 LIFE CYCLZ ANALYSIS
This section examines the various phases of the NAS life cycle. Insome cases, references will be made to the life cycle stagespresented in Figure 1-1 of this report.
C.5.1 REQUIREMENTS DETERMINATION
C.5.1.l Strengths
Only one person stated that the requirements definition processworks well and then it was qualified with the statement "if therequirements are worked out and an NCP is generated".
C.5.1.2 Weaknesses
The requirements determination phase of the NAS life cycle wasunanimously named the worst phase of the life cycle by thosesubjects who had an opinion. The major problems that were citedare:
A general lack of standards defining the requirementsdetermination phase;
Little or no participation by AT during therequirements determination phase;
Not enough early involvement during the requirementsphase by QA and maintenance; and
ATO reacts to daily situations which keeps changing therequirement s;
Page C-8
The real air traffic controllers are not defining therequirements;
Not enough time allocated for a thorough job ofrequirements determination.
Most of the subjects put the blame on AT for the problems in thisphase. It was stated that AT does not obey the rules and letsengineering develop the requirements. Then AT approves themwithout adequate review. The other approach is that AT states verygeneral and vague requirements. Finally, when AT can "see" theproduct (typically during testing during phase 2.6, DT&E) . it thendefines the real requirements for the project.
No one denied that the requirements definition phase is verydifficult.
There seems to be no clear transition from phase 1 to phase 2,which allows poor requirements to leak through to the next phase.One subject suggested some form of Procurement Readiness Review tohelp verify the quality of the requirements statement.
Human factors (MMIs) were cited as very difcicult to specify in arequirement document. prototyping was mentioned as one useful tohelp this difficult task. See Section C.7 for more details onprototyping.
C.5.2 ACQUISITION
C.5.2.1 Strenqths
The acquisition phase was named as the best phase in the NAS lifecycle. However, it was cited as adequate, but not outstanding.
One subject noted that programming standards are not applied to thecontractors during this phase which hurts system maintenance.
Demonstrations were mentioned as a good way of determining thesoftware's condition.
C.5.2.2 weaknesses
The testing guidelines were cited by several subjects as being toovague and wordy.
One subject noted the differences between the rules and actualpractice but did not cite any specific instances. Other subjectsnoted that often the guidelines were not followed for severalreasons. Sometimes the orders were out of date and in other cases,staldaLds and guidelinAes were simply ignored.
It was noted that the standards and orders applicable during this
Page C-9
phase sometimes get out of date with respect to the FAA's
organizational structure.
C.5.3 OPERATIONAL SUPPORT
C.5.3.1 Strenqths
This phase appears to function adequately with a few exceptions.Most of the problems expressed were a result of problems with theearlier phases and not inherent problems with this phase. Onesubject stated that this was the best phase because it stopsprojects until the customer is happy - even though the system maymeet the specifications.
C.5.3.2 Weaknesses
Maintenance was viewed as hard for a number of reasons, most ofthem originating with the use of DoD-STD-2167A documentation andthe lack of a CPFS in the acquisition phase. It was noted that,due to the use of separate organizations to support hardware andsoftware, that maintenance became political and somewhat difficult.
C.5.4 PHASE TRANSITIONS
Both phase transitions were viewed as weak points within the NASlife cycle. There are not clearly understood rules for definingthese transitions.
Apparently, there are no standards, orders, etc., for definingthese transitions, nor formal procedural methods for insuring theadequacy of one phase before moving to the next. It was suggestedthat an approach similar to that used for a Deployment ReadinessReview (DRR) would be useful in evaluating the transition from onephase to another.
C.S.4.1 Phase Transition: Requirements Determination (i) toAcquisition (2)
This transition, although scheduled, never really occurred. Phase2 begins before phase 1 is completed. Then the two phases bothcontinue until they both merge into phase 3. A requirementsdetermination is completed before phase 2 begins; however, therequirements determination is performed by engineering, not thecustomer (AT). At does have to approve the requirements documents,but apparently does not thoroughly review the document; they havenot decided themselves what is necessary. When testing begins, the"real requirements" come out and, most often, the real requirementsare different than those used to design and build the system.Obdviously, this results in a large amount of costly rework.
There are rules concerning the development of a requirements
document, but they are apparently not followed by At during various
Page C-10
stages of phase 1 and 2.
C.5.4.2 Phase Transition: Acquisition (2) to- Operational Support
The p'- e ' - 3 transition, although not as much a problem as thephase •- 2 -,ansition, does cause some problems. The testingoperation -iided by a vague set of rules and the exact criteriafor gcirg 'o thase 3 is not defined.
(NOTE: The jof' dare Integration Working Group (SIWG) is in theprocess of preparing a "Procured Software Hand-off Procedure"Action Notice which addresses the transition from phase 2.0 to 3.0]
C.6 DOCUMENTATION
C.6.1 PROBLEMS ENCOUNTERED
The majority of problems cited concerning documentation wererelated to the use of DoD-STD-2167A. This standard, oftenmisapplied, results in documentation that is not appropriate forsystem maintenance and is often of such volume that it is notthoroughly reviewed. The use of DoD-STD-2167A also causes troublefor the FAA when .,: tries to tailor the standard. Section C.4 ofthis document contains a thorough discussion of the problems withDoD-STD-2167A.
C.6.2 gTHER TOPICS
The Government Printing Office (GPO) often reformats and publishesdocuments produced by vendors. This is obviously expensive andtime-consuming. It would be desirable if the vendors could delivertheir documentation in machine readable form (Interleaf wasmentioned).
It was not clear how to handle COTS documentation. The subjectswere riot clear concerning which COTS documentation is required ona project.
Sometimes it is cheaper not to require a full set of documentationbut rather to obtain needed documents on a case basis. 2167A seemsgood for development documentation but not maintenance. Since theDoD-STD-2167A documents are not kept up to date, some documentationmust be maintained in order to support maintenance. Many subjectsrecommended that the CPFS, used before DoD-STD-2167A appeared,would be the right candidate for a maintenance document. The largeamount of documentation produced by DoD-STD-2167A created more workthan the FAA could handle and often was ignored by the FAA.
Many of these DoD-STD-2167A problems can be solved by tailoring the
standard, but few subjects interviewed knew how to tailor DoD-STD-
Page C-lI
2167A.
Guidelines are clearly needed to help the FAA staff deal with COTSsoftware documentation. This was discussed in the standardssection above.
C.7 PROTOTYPING
Prototyping was general.y viewed as a useful and desirableactivity. It was most often cited for use in MMIs and once forreal-time signal processing.
Three advantages to prototyping were cited: (1) it is useful fornew systems where the activity borders on R&D; (2) it helps "firm-up" requirements; and (3) demonstrations of prototypes may help getAT more interested and involved since they can actually see theproduct.
The downside of prototyping is the cost of prototyping in anaccurate simulated environment of AT. A critical evaluation isdifficult to do without a simulated environment.
C.8 METHODOLOGY SPZCIFICATION
There seems to be no demand for the FAA to require a specificmethodology of the vendors. There was concern that thedocumentation and other standards would provide sufficientassurance of this.
C.9 SPECIFIC RECOMMENDATIONS BY THE INTERVIEWEZS
This section presents the specific recommendations that were madeduring the interview process by the interviewees. Only therecommendations explicitly made are presented here, therefore thissection is not a summai:v of the interview process.
The specific recommendations were:
Standards, especially DoD-STD-2167A, should betailored;
There should be a support group within the FAA toassist in tailoring;
The CPFS concept should be brought back;
A draft CPFS shoula be done very early in the project.
I'. helps the requirements andlysis phase;
Do not retrofit new guidelines to existing projects;
Page C-12
Enforce existing standards and orders;
Test in an integrated environment in phase 2.7 (seeFigure 1-1), rather than just during phase 3;
Have QA and maintenance become involved early inprojects;
Put incentives in place for minimizing life cyclecosting;
Get more qualified software people in project office,QA, and elsewhere;
Develop standards for dealing with COTS and non-
developmental software (NDS);
Apply FAA coding standards to contractors;
Develop firm criteria (standards) for moving from phaseto phase; and
Do away with or heavily revise FAA-STD-026.
Do more tailoring of standards.
Provide software engineering concepts training.
Each of these issues have been discussed in other sections of thisreport.
Page C-13
APPVNDIX D
LITZ CYCLZ ELEMENTS
VERSUS
GOVERNING DOCUMENTS
SOITIIU NGIU.G ITIN GOVD IQ I0 I a |0 0073I I I0_AK-STD- W"L QRDUR OTBE
Acceptance Testing _ _ _ 1810.4b
Acqui sit ion -- _ , 1800.8f
Allocated Baseline 021a 1800.8f
Allocated Configuration Identification: ACI 021a
Air Traffic Configuration Control Board: AT CC8 1800.8f
Audits 016a 1800.8f018a
Automated Tools
Baseline(s) 021a
Case Files 1800.8f Form 1800-151100.134a Form 1800-17
Change Status Report 021a
Cluster CCBs (see Division CCB) 1800.8f
Code Standards 026
Commercial Off-The-Shelf (COTS) Software 1800.8f
Compilers A.N.1370.9
Computer Program Functional Specification 026
Computer Resource Integrated Support Document 026
Computer Security 1600.54b
Computer Software Component; CSC 026
Computer Software Configuration Index 021a
Computer Software Configration Item; CSCI 026
Computer Software Quality Program 018a 4630.9
Computer Software Quality Program Plan; CSQPP 0l8a . 4630.9
Computer Software Unit; CSU 026
Computer System Operator's Manual 026
Concept Analysis , --
Concept Definition and Verification 1800.8
Configuration Audits _ 1800.8f MIL-STD-1521B
Configuration Control 021a 1800.8f MIL-STD-4801800.25 MIL-STD-481
Configuration Control Board __.__ 1800.Bf
Configuration Control Decisions 021a, 026 1900.8f Form 1800-49
Configuration Control Procedures 1800.8f
Configuration Control Support Facility ____ 1800.25
Configurat-on Management Audits 021a
IConfiguration Management Plan 021a
Configuration Management Procedures 021a 1800.8f
Page D-I
SOI ITRI IINGII1,1 1 I IT I GOVIK1NG GOI JIUO I OOYI N G
VMk-STD- Ikk ORDUR OM
Corrective Action Process 026
Cost Modeling
Cost/Schedule Report 026
Critical Design Review 026 1800.8f MIL-STD-1521
CSC Integration and Testing 026 .
CSCI Testing 026
CSU Testing 026
Database Design Document 021a
Database Management -
Deployment Readiness Review; DRR 1800.8f- I 810.4b ________
Design Baseline 021a , ]VU.sf
Design Configuraticn Identification; DCI 021a
Design Standards 026
Detailed Design 026
Developmental Configuration .... ...
Developmental Test and Evaluation Plan 1810.4b
Developmental Test and Evaluation Procedures 1810.4b
Developmental Test and Evaluation Test Report 1810.4b
Discrepancy Reports 026
Division Configuration Control Board 1800.8f
Documentation 005
Documentation Management ..
ORR Memorandum ... . 1800.,
DRR Monthly Status Report ,_ , 1800.
DRR Report 1800.
Engineering Change Proposals 026, O2a .
Engineering Release 021a _
FCA/PCA Plan 1800.8f
Field Shakedown Testing 1810.4b
Firmware Support Manual 026 ,
Formal Qualification Review; FQR 021a 1800.8f ,
Formal Qualification Testing 026
Functional Baseline 021a 1800.8f
Functional Configuration Audit; FCA 026 1800.8f MIL-STD-1521
Functional Configuration Identification; FtI .O2ia &_ .
Hand-off Package 1800.8f
Page D-2
8OTWIM KwGMIU• X 1TIN GOVBINIUG QOVIZRIrlu'G OOVEIrz GF__-SITD- via OR.DK OTM
Implementation 026 _
Independent OT&E for MSA -- 1810.3 --
Independent Verification and Validation; IV&V 026, ol1a 4630.9
Integration and Testing 026 -- -
Interface Control Document; ICD 025 . -- .
Interface Design Document 1 026 1810.4b
Interface Management .... _1810.4b NAS-SS-1000
Interface Requirements Document; IRD 025 1810.4b
Interface Requirements Specification 026, 005d | •_• _MIL-STD-490A
Key Decision Memorandum .1810.Id
Key Decision Point 1810.1d OMB CircularA-109
Languages 026 A.N.1370.9
Logistics 034 1800.58
Maintenance Engineering Configuration Control Board; ME 1800.8fCCB
Management 026
Major System Acquisition 1810.1d Order4200.14b
Master Test Plan; MTP 024 1810.4b
Memorandum of Understanding; MOU . ... 1810.4b
Mission Analysis ,__
Monthly Management Review -- .
Monthly Progress Reports 026
NAS Change Proposals -- _._1800.8f Form 1800-2
NAS Configuration Control Board; NAS CCB 1800.8f
NAS Life Cycle .... 1800.8f.
NAS System Reruirements Specification 005d
Non-Developmental Software; NDS 026
Operating Systems ,___,
Operational Support 1800.8f
Operational Test and Evaluation; OTE _ 024 1810.
Operational Test and Evaluation Integration Test Report 1810.4b
Operational Test and Evaluation Plan 024 1810.4b
Operational Test and Evaluation Procedures 024 1810.4b
Operational Test and Evaluation Shakedown Test Report 1810.4b
Operational Transition
Physical Configuration Audit; PCA 026 l800.8f MIL-STD-1521B
Page D-3
$O1TWAUZ EMNGZIXQ ITEW GOVDI1WI= GoVtIRMZWG QOV3RlQ_AX-STD- FAA& ORDER OM
Portable Software
Portability
Post Deployment Support 1100.134a1100.1451800.8f
Preliminary Design 026 • ,
Preliminary Design Review 026 1800.8f, MIL-STD-1521
Problem/Change Report 026
Problem Technical Report ,__ _1100.134a
Problem Tracking and Reporting 026 _
Procurement Request 030 1800.
Product Baseline 021a 1800.8f
Product Configuration Identification; PCI 021a
Product Specificatiou 005d
Production Acceptance Test and Evaluation Plan 1810.4b
Production Acceptance Test and Evaluation Procedures 1810.4b
Production Acceptance Test and Evaluation Test Report 1810.4b
Program Authorizations; PA
Program Definition
Program Directives; PD 1810.1d1810.4b
Program Management Plan ,_ 1810.4b
Program Master Plan; PMP 1810.1d
Program Plan
Program Technical Report; PTR 1800.8f Form 6100-1
Programmatic Baseline 1800.8f
Project Implementation Plan 036
Project Initiation
Project Management Plan ..... _ 180i.4b . .. .
Pro ect Plan
Prototyping
Quality Assurtnce Ole& 4630.9016a 4630.8
Quality AssuranceReport 018a 4630.9
Quality Control -=016a08a 4630.9
Quality Control Procedures
Quality Control Program Plan
Quality Control System Plan 016a 4453.2a NO. 00-41A
Quarterly Review 1810.1d
Page D-4
8OITTHIMM UUGfI IP"aG X!ZT OVEPSM aOVMNmIuG QOVI.IING__Am-SmD- FAA Opim OTR
Quarterly Status Reports ,_ __ 1810.ld
Rapid Prototyping
Records Ola 4630.9
Regional Configuration Control Board _ _ 1800.8f
Reports 018a 4630.9
Request For Proposal 030
Requirements Definition 026 1810.1d NAS-SR-1000
Requirements Determination 1800.8f .
Requirements Traceability matrix 1800.8f
Research, Engineering and Development
Resource Performance Analysis
Reviews 026 1800.8f MIL-STD-1521
Risk Management 026
Risk Analysis
Risk Management Plan 026
Safety Analysis 026
Security 1600.54b
Shake-Down Testing 1810.4b
Site Adaptation ,,_ 1800.8f
Software Acquisition Plan
Software Code Standards
Software Configuration Management
Software Configuration Management Plan 021a, 026 1800.8f
Software Cost Estimates
Software Estimating ,_--_.
Software Design Document 026
Software Design Standards __
Software Detailed Design Ducument 026
Software Development Environment
Software Development Fj e 026
Software Development Fold' r 026
Software Development Library 026
Software Development Management 026
Software Development Methods 026 MIL-STD-.1521
Software Development Plan 026
Software Development Tools
Software Documentation Management
Page D-5
souTNAiw mINZ• wo z!E GOOVYIIZUGU GOVEAMI" GOMM=
VU-Srn- r~A ON= TMSoftware Engineering 026
Software Engineering Environment 026
Software Lanquaqes 026
Software Management 026 AFSCP 800-43
Software Management Plan •_026 APSC Pamphlet
Software Mai~ntenance 11.. i00.124
Software Maintenance Plan
.Software Methodolocqr 026
Software Metrics
Software Portability -
Software Product Evaluation 026
Software Product Specification 026,005d -- _ MIL-STD-490A
Software Programmer's Manual 026
Software Quality Assurance Report 018a,016a 4630.9
Software Quality Program Plan 018a,016a 4630.9
Software Quality Control Procedures .018a16a 4630.9
Software Requirements Specification; SRS 026 .
Software Requirements Review 026 • .
Software Reuse
Software Reviews 026 . -- MIL-STD-1521
Software Risk Analysis s ,
Software Schedules
Software Security 026
Software Specification Review 026 1800.8f MIL-STD-1521
Software Standards and Procedures Specification 026
Software Technology
Software Test Descriptions 026 • •
Software Test Environment 026
Software Test Management
Software Test Plan 026
Software Test Procedures
Software Test Reports =_-
Software Tools
Software Top Level Design Document 026 ....
boftware TransiLion
Software Transition Plan
Page D-6
BOTTUM ENGINZRNING ZTZM GOVauRIIG GOVE3n" OOVIPW.. ___F__k-STD- FAA ORDUR OTM
Software User's Manual 026
Source Code 026
Specification Change Notices 021a,026,005d
Specification Review Board 1800.8f
Specifications OO5d 1800.8 NAS-SS-1000
Statement of Work 031
Subcontractor Mana ement 026
System Desian Review; SDR 026 1800.8f MIL-STD-1521b
System Enqineering . _ 1800.8f
System Life Cycle 1800.8f
System Integration and Testing
System Requirements Review 026 1800.8f MIL-STD-15216
System Requirements Specification 026 1800.8f MIL-STD-499_ _MTL-SrD-490A
System Requirements Statement; SRS _ _ _ 1810 1 _
Sistem Security 026
System Specification
Uysten Test Plan
System Test Procedures
Systems Engineering Configuration Control Board: SE CCB 1800.8f
System/Segment Design Document
System/Segment Specification 005d _
Technology Envaluation Reports 024 1810.4b •
Test and Evaluation Reports 024 1810.4b
Test Documents 016a
Test Management 024 1810.4b
Test Readiness Review 026 1800.8f MIL-STD-1521
Tools
Traceability Matrix 026
Training 028 3120.4 .
Training Materials 028
Training Plan 028
Transition Plan
Transition to Software Support 026 .
TSARC Program List 4400.56 Order 4200.9A
Validation & Verification 026
Verification 1 NAS-S5-4000
Page D-7
sOFr'itW UM•G)XUG ZI!3 GOV3PXUIIG GOYWUOM GOOVfRl__A_-STD- FAA ODltZi OTUM
Verification Requirements Traceability M!atrix: VRTM 1810.4b ,
Version Description Document 026
Work Breakdown Structure
Page D-8
APPENDIX E
EXPANDED NAS SYSTEM LIFE CYCLE
EXPANDzD HAS SYSTEM LiFr CYCLE
1.0 REQUIREMENTS DETERMINATION: (all FAA activities)
1.1 CONCEET DEFINITIQN AND VERIFICATION
1.1.1 Products i
System Level Operational Concept DocumentsR,E&D Program PlansR,E&D Project PlansSystem Specification, Program (initial)Functional Specification
1.1.2 Reviews:
Monthly Management Reviews
1.1.3 Actions:
Concept analysesMission analysesTechnology application studies
1.2 PROGRAM DEFINITION (MSA: Requirements Definition)
1.2.1 Product;:
NAS SYSTEM LEVEL:
NAS Requirements Document (initial)NAS System Requirements Specification (baselined)NAS Level I Design (functional baseline)NAS System Specification (allocated baseline)(also called NAS Level II Design)
NAS Transition Plan (initial)(also called NAS Level III Design)
NAS Interface Requirements Document (baselined)
PROGRAM SYSTEM LEVEL:
Major System Acquisition Candidate StatementMission Need Statement (for program not in NAS Plan)System Requirements Statement; SRS (initial)Key Decision Memorandum; KDM (initial)Acquisition Paper; APProgram Direccives (for testing)Clearance Records (T&E)Program Management Plan; PMP
(also called Program Master Plan,Project Management Plan 1810.1d)
.- 0 .- 1
EXWID2D MAS SYSTEM LIFE CCLE
System Specification (baselined)(also called Project Specification,
Program Specification)Master Test Plan; MTP (initial)(includes Verification Requirements Traceability Matrix;-
V1RTM)Request for ProposalWork Breakdown StructureConfiguration Control Decisions (CCD)Case FilesStatement of Wcrk (SOW)NAS Change ProposalsRisk Management PlanContract Training Proposals
SOFTWARE:
Software Acquisition PlanSoftware Management PlanSoftware Configuration Management Plan (initial)Software Maintenance Plan (initial)Software Transition Plan (initial)
1.2.2
NAS SYSTEM LEVEL:
NAS System Requirements ReviewNAS System Engineering Configuration Control Board; CCBQuarterly Review
PROGRAM SYSTEM LEVEL:
Monthly Management ReviewSpecification Review Board (SRB)System Engineering CCBTest Policy & Planning Review Board (TPRB) meetinq(MTP review)
SOFTWARE:
TBO
1.2.1 actiocas A
NAS SYSTE1M LEVEL:
Designate i program as a Major System Rcquisition or nota MSA
Vag& 1-2
i_• m" mI | Im I
EXPANDED HAS SYSTEM LUFZ CYCLI
appoint Program Sponsorappoint Program Managerapprove Program Manager charter
PROGRAM SYSTEM LEVEL:
Identify training requirements (for FAA personnel)Define Operational Test and Evaluation OT&E) Integration
and Shakedown requirementsDefine Production Acceptance Test and Evaluation
(PAT&E) requirementsValidate Deployment Readiness Review (DRR1 items in the
solicitation package
SOFTWARE:
Assess software technology requirements
1.2.4 Key Decision Point (KDP) #1 For MSA:
Authorizes program to proceed to Concept Analysis
Page 9-3
F.XMMAIMD HAS SYSTEM LIUZ CYCL3
2.0 ACQUISIT'ON: (contractor activities except items with * areFAA activities)
2.1 PROJECT INITIATION (MSA: Concept Analysis)
2.1.1 Products:
SYSTEM LEVEL:
Program Management PlanConfiguration Management Plan (initial)Quality Control Program Plan; QCPP (initizL)Quality Control System Plan; QCSP (initial!System/Segment Design Document; SSDD (init"Ial;(also called System/Segment Specificatior; TSS,Risk Management Plan (initial)Training Plan (initial)Monthly Progress Reports; Cost/Schedule ReportsEngineering Change Proposals (ECP)(also called Engineering Change Requests - 018a)Specification Change Notices (SCN)Configuration Control Decisions (CCD)
*System Development Contract*Quarterly Status Reports*Test Management Plan
SOFTWARE:
Software Development Plan (initial)(includes Software Engineering Environment Plan)Software Requirements Specification (initial)Interface Requirements Specification (initial)Software Configuration Management Plan (initial)Computer Software Quality Program Plan; CSQPP (initial)Software Standards and Procedures Specification (initial)(includes software design and code stand3rds)Traceability Matrix; System Spec., SOW (initial)
2.1.2 Review
SYSIEM LEVEL:
Monthly Management ReviewSystem Requizements ReviewProgram/project CCB
•Quarter17 Reviews
S XPTWARE :
Wage 5-4
EXPANDED NAS SYSTEM LITZ CYCLE
Software Specification Review; SSR
2.1.3 Actions:
SYSTEM LEVEL:
TBD
SOFTWARE:
Software methodology selectionScftware tools selectionLanguage selectionOperating System selectionBuild versus buy decisionsSoftware metrics selectionSoftware Tools demonstrations
Configuration Control ToolSoftware Development LibraryTraceability Matrix ToolProblem Tracking and Reporting ToolSoftware development tools (compilers, etc)
2.2 REQUIREMENTS DEFINITION (MSA: Concept Analysis)
2.2.1 Products:
SYSTEM LEVEL:
Configuration Management Plan (baselined)Quality Control Program Plan; QCPP (baselined)Quality Control System Plan; QCSP (baselined)System/Segment Design Document; SSDD (baselined)
(or Sy.tem/Segiaent Specification; SSS (baselined))Risk Management Plan (baselined)System Test Plan; STP (initial)
(this is the Development Test and Evaluation (DT&E)Plan)
Monthly Status ReportsContract Training PlanComputer System Diagnostic ManualSystem Allocation DocumentComputer Resource Integrated Support Document; CRISD(initial)
*Quarterly Status Reports
SOFTWARE:
Page 9-5
EXPANDED HAS SYSTEM LIZf CYCLE
Software Development Plan (baselined)Software Requirements Specification (baselined)Software Configuration Management Plan (baselined)Computer Software Qualf.ty Program Plan (baselined)Software Standards and Procedures Specification(baselined)Software Quality Control Procedures (initial)Interface Requirements Specification (baselined)Traceability Matrix (SSDD, SRS..IRS)Software Quality Assurance ReportsComputer Program Functional Specification; CPFS (initial)
2.2.2 Reviews:
SYSTEM LEVEL:
Monthly Management ReviewSystem Requirements Review; SRRSystem Design Review; SDR
*Quarterly Review
SOFTWARE:
Software Specification Review; SSRSoftware Products Evaluations; SDP, SSDD, SRS, IRS
2.2.3 Actions:
SYSTEM LEVEL
Submit Engineering Change Proposals (ECP)Configuration Control Board acts on ECPsEstablish project Problem Tracking and Reporting databaseFinancial Data AnalysisTraining Course Task Analysis
SOFTWARE:
Establish project Software Development LibrarySoftware Metrics Analysis
2.3 PRELIMINARY DESIGN (MSA: Concept Analysis)
2.3.1
SYSTEM LEVEL:
Page 2-6
I"XPAUDD WJU SYST X LIZI CYCLI
Monthly Status ReportsTraining Plan (baselined)System Test Plan; DT&E (baselined)System Test Procedures; DT&E (initial)Training Materials (initial)Contract Training Plan
*Master Test Plan (baselined)*Quarterly Status Reports*Key Decision Memorandum (updated)
SOFTWARE:
Software Quality Control Procedures (baselined)Software Test Plan (initial)
(includes Software Test Environment Plan)Interface Design Document; IDD (initial)Software Top ,Level Design Document; STLDD (initial)Traceability Matrix (updated - STLDD)Software Development Files; STLDD
(also called Software Development Folders - 018a)Software Quality Assurance ReportsComputer Program Functional Specification; CPFS(baselined)
2.3.2 Reviews:
SYSTEM LEVEL:
Monthly Management Reviews*Quarterly Review
Sc TWARE:
Software Product Evaluations; STLDDPreliminary Design Review (PDR)Configuration Management Audits
2.3.3
SYSTEM LEVUL:
Financial Data AnalysisSubmit Engineering Change Proposals (ECP)Configuration Control Board acts on ECPs
SOFTWARE:
Page X-7
ZXPANDED H&S SYSTZX LiFn CyCLE
Resource Performance AnalysisSoftware Metrics Analysis
2.3.4 KDP #2 for MSA:
*Authorizes program to proceed to Uemonstration Phase
2.4 DETAILED DESIGN (MSA: Demonstration Phase)
2.4.1 Products:
SYSTEM LEVEL:
Monthly Status ReportsSystem User's Guide (training)Course Design Guide (training)
*Quarterly Status Reports
SOFTWARE:
Software Test Plan; STP (baselined)Interface Design Document, Top Level; TLIDD (baselined)Software Top Level Design Document; STLDD (baselined)Software Test Descriptions, Cases (initial)Software Detailed Design Document; SDDD (initial)Interface Design Document, Detailed; DIDD (initial)Traceability Matrix (updated - SDDD)Software Quality Assurance ReportsSoftware Development Files (updated - SDDD, CSC)
2.4.2 Reviews:
SYSTEM LEVEL:
Monthly Management Reviews*Quarterly Review
SOFTWARE:
Critical Design Reviews (CDRs)Configuration Management Audits
2.4.3 Actions:
SYSTEM LEVEL:
"Submit Engineering Change Proposals (ECP)
Page Z-8
ZXPAMDID HUA SYSTEII LITZ CYCLE
Configuration Control Board acts on ECPS
SOFTWARE:
Resource Performance AnalysisSoftware Metrics AnalysisConfiguration Control; STP, TLIDD, STLDD
2.5 IML~IENTTION (MSA: Demonstration Phase)
2.5.1 Products:
SYSTEM LEVEL:
System Test Procedures; DT&E (bas.zlined)Production Acceptance Test and Evaluation Plan; PAT&E(initial)Monthly Status Reports*Operational Test and Evaluation Plan; OT&E (initial)*OT&E Procedures (initial)*Quarterly Status Reports
SOFTWARE:
Source CodeComputer System Operator's Manual; CSOMSoftware Programmer's Manual; SPMFirmware Support Manual; FSMSoftware Detailed Design Document; SDDD (baselined)Software User's Manual; SUM (initial)Software Test Descriptions, procedures (initial)Traceability Matrix (updated - source code)Software Quality Assurance ReportsSoftware Development Files (updated - CSU)
2.5.2 Reviews:
Monthly Management Reviews*Quarterly Review
2.5.3 Actions:.
Resource Performance AnalysisSoftware Metrics AnalysisConfiguration Control; SDDD, DIDD, STD, CSUSubmit Engineering Change Proposals (ECP)
Page 1-9
Z"XPMIDD HAS SYSTrI LIZ! CYCLI
Configuration Control Board acts on ECPs
2.6 INTEGRATION AND TESTING (MSA: Demonstration Phase)
2.6.1 CSC Integration and Testing:
2.6.1.1 Products:
SYSTEM LEVEL:
Monthly Status Reports*Quarterly Status Reports
SOFTWARE:
Software Test ReportsTraceability Matrix (updated - CSC tests)Software Quality Assurance ReportsSoftware Development Files (updated - CSC tests)
2.6.1.2 Reviews:
Monthly Management Review*Quarterly Review
2.6.1.3 Actions:
Submit Engineering Change Proposals (ECP)Configuration Control board acts on ECPs
2.6.2 CSCI Testing:
2.6.2.1 Products:
SYSTEM LEVEL:
Monthly Status Reports*Quarterly Status Reports
SOFTWARE:
Software Test ReportsTraceability Matrix (updated - CSCI tests)Software Quality Assurance ReportsSoftware Development Files (updated - CSCI tests)
2.6.2.2 Reviews:
Page 1-10
ZXPANIDD NUS SYSTSH LITZ CYCLS
Monthly Management Review*Quarterly Review
2.6.2.3 Actions:
Submit Engineering Change Proposals (ECP)Configuration Control Board acts on ECPs
2.6.3 System Integration and TestinQ:
2.6.3.1 Products:
SYSTEM LEVEL:
Monthly Status ReportsDT&E Procedures (baselined)PAT&E Plan (baselined)PAT&E Procedures (initial)
*Project Implementation Plan*OT&E Plan (baselined)*OT&E Procedures (baselined)*Quarterly Status Reports*Key Decision Memorandum (updated)
SOFTWARE:
Software Test ReportsSoftware User's Manual; SUM (baselined)Software Test Descriptions, procedures (baselined)Training Materials (baselined)Version Description Documents (initial)Software Product Specifications (initial)Traceability Matrix (updated - system test procedures)Software Quality Assurance Reports
2.6.3.2 Reviews:
Monthly Management Review*Quarterly Review*Test Readiness Review
2.6.3.3 Actio.ns:
Submit Engineering Change Proposals (ECP)Configuration Control Board acts on ECPsFunctional Configuration Audit (FCA)Physical Configuration Audit (PCA)
Pago 2-1I
XXPAMDRf Mks SYSTIX LinZ CCLZ
2.6.4 KDP #3 for MSA:
*Authorizes program to proceed with full scale development and limited-production; return to Step 2.0.
2.7 FACTORY ACCEPTANCE TESTING (DT&E)
2.7.1 Products:
SYSTEM LEVEL:
DT&E Test ReportsMonthly Status Reports
*Quarterly Status Report*Test Support Memorandum of Understanding; MOU (initial)
SOFTWARE:
Updated source codeVersion Description Documents (baselined)Software Product Specifications (baselined)Traceability Matrix (updated - all)Computer Resource Integrated Support Document; CRISD(initial)Software Quality Assurance Reports
2.7.2 Reviews:
Functional Configuration Audit (FCA)Physical Configuration Audit (PCA)Monthly Management Review
*Quarterly Review
2.7.3 Actions:
Formal Qualification Testing (FQT)Configuration Control Board acts on ECPs
Page 1-12
JXPANDrD N&S SYSTUK LZrI CYCLE
3.0 OPEJTIOMAL SUPPORT (contractor and VAL activities)
3.1 OPERATIONAL TASITION
3.1.1 Operational Test and Evaluation/Integration Testing:
3.1.1.1 Products:
Computer Resource Integrated Support Document; CRISD(baselined)
*Program Directives*Memorandum of Understanding; MOU (final)*OT&E Integration Test Report*Quarterly Status Reports*DP• Memorandum (announce DPR Team Meeting)
3.1.1.2 Reviews:
Monthly Management Review*Quarterly Review
3.1.1.3 Actions:
Initial review of DRR checklist
3.1.2 Operational Test and Evaluation/Shakedown Testing:
3.1.2.1 Products:
*Quarterly Status Reports*OT&E Shakedown Test Reports*DRR Team Meeting Report*DBR Monthly Status Report
3.1.2.2 Reviews:
Monthly Management Review*Quarterly Review*Deployment Readiness Review (DRR)
3.1.2.3 Actions:
TBD
3.1.3 Production Acceptance Test and Evaluation:
Page Z-13
ZXPANDID NUAS SYST]H L'Z! CYCLZ
3.1.3.1 Products:
PAT&E Test Reports*Quarterly Status Reports
3.1.3.2 Reviews:
Monthly Management Review*Quarterly Review
3.1.3.3 Actions:
TBD
3.1.4 Site Field Shakedown Test and Evaluation:
3.1.4.1 Products:
*T&E Reports*Quarterly Status Reports*Key Decision Memorandum (updated)*DRR Team Meeting Report*DRR Monthly Status Report
3.1.4.2 Reviews:
Monthly Management Review*Quarterly Review*Deployment Readiness Review (DRR)
3.1.4.3 Actions:
System Commissioned
3.1.5 KDP #4 for MSA:
*Authorizes program to proceed with full production, installation,and operation of the system.
Page Z-14
ZXAKD3D tMS SYSTIN LZfl CYCLZ
3.2 POST DEPLOYMENT SUPPORT
3.2.:1 Product•s;
Problem Technical Report (PTR)Case FileNAS Change ProposalsUpdated source codeUpdated documentation (what documentation?)
3.2.2 Reviews:
Monthly Management Review
3.2.3 Actions:
Live environment shakedown testing
Page 1-15
Azxznzc r
aurzuxZcKu
APPENDIX F
REFERENCES
DOT/FAA DOCUMENTATION
[DOT/FAA-1] (ADL-10), "Systems Acquisition, StandardOperating Procedures (SOPs)", June 4, 1985
[DOT/FAA-2] FAA, "National Airspace System (NAS) FieldImplementation Plan", March 1989
[DOT/FAA-3] FAA Order 4400.56, "Acquisition Review andApproval", ALG-120, 9/19/85
[DOT/FAA-4) FAA Order 9500.4a, "Technical Data Package(TDP) Handoff", ARD-54, 9/27/78
[DOT/FAA-5) FAA Order 9550.3, "Requests for Research,Development and Engineering (R,E&D) Efforts",ARD-54, 16 Jan 73
[DOT/FAA-6] FAA Order 9550.4, "Human Factors Considerationin the Development/Procurement Cycle",ARD-603, 7/11/74
[DOT/FAA-7] FAA Order 9550.5, "Internal AED Procedures forRequest for Research, Development andEngineering Efforts", AED-10, 5/13/81
(DOT/FAA-83 GAO/RCED-87-8, Report to the Chairman,Subcommittee on Transportation, Committee onAppropriations, House of Representatives,"Aviation Acquisition, Improved Process Needsto be Followed", March 1987
[DOT/FAA-9] FAA Technical Center, "NAS Integration TestPlan Preparation Guide", July 1987, DRAFT
[DOT/FAA-10] FAA-CDRL-240-001B, "Software RequirementsSpecification", DI-E-X107
(DOT/FAA-il] No author given, "Configuration ManagementProcurement Guidance", 25 August 1989
Page F-I
[DOT/FAA-12] FAA, "An Evaluation and Analysis of the FAA'sNational Airspace System (NAS) SoftwareDevelopment Management", May 1989
[DOT/FAA-13] FAA, "NAS Transition Plan, System Engineeringand Integration Contract for Implementationof the National Airspace System Plan", VolumeII, Section 4.0 Software Integration &Transition, February 1989
[DOT/FAA-141 FAA, "ATR-250's Basic Acceptance Requirementsfor Delivery of Contractor DevelopedSoftware", no date, Rough Draft
[DOT/FAA-15] FAA, "ATR-250's Software Development andMaintenance Activities", no date
[DOT/FAA-16] Various memos and pieces of reports:
(a) Memo; SIWG Technical Team, "SoftwareMaintenance Issue", 13 September 1989
(b) AAP-310 Comments oin the Draft NASTransition Plan Volume 2, Section 4.0(Software Tran."ition & Tnteenr~tion)
(c) Judith Warren, "Discussion Paper *i,Software Support Planning", June 16,1989, US DOT/TSC
(d) excerpt from NAS-SS-1000, Volume I,December 1986,pages 5-12, 4-2, 64-76
(e) memo; A.Cocanower, "Proposed SoftwareHandback Requirements", 7 December 1988
(f) presentation; Background Information forThe Software Integration Working Group(SIWG), July 1989
(g) memo; A.Cocanower, "TA .j Draft SIWG ActionItem Descriptions", ii April 1989
(h) memo; SIWG Tochnical Team, "SoftwareProblem Categorization/Reporting System(SIWG-175)", May 8, 1989
Page F-2
(i) memo; Manager, Automation SoftwareDivision, ATR-200 or Chairman, SoftwareIntegration Working Group (SIWG), "AdaTraining", April 21, 1989
(j) memo; Acting Director, ASM-1 andDirector, ATR-1, "Software MaintenanceDocumentation Requirement", February 10,1989
(k) SIWG Technical Team, "COTS Software andFirmware", December 15, 1988
(1) briefilnq, SIWG Management Team, "TheSoftware Integration Working Group(SIWG)", January 30, 1989
(m) briefing, SIWG Management Team, "TheSoftware Integration Working Group(SIWG)", July 24, 1989
(n) no author, "Software Coordination Group(3CG) Management and Operating Plan",July 11, 1988
(DOT/IFAA-1TJ IMS '14k Force Report, July 19,1989, FAA-AP-09-14171,Revision 1 Paragraph 3.6.1.2, page
(DOT/Fh.t-18 AdvAticed Automation System AP SoftwareMt.mvarda and Procedures Manual, Book 1,8February 1989; I'AA-AP-1989-0579
[DAT/rAA-19] Panel Jkeport National Airspace System En RouteComputer Software Support Study, Jack Arnow,et.ai. July 1981
Page F-3
MILITARY DOCUMENTATION
[MIL-DOC-1] Committee on Adapting Software DevelopmentPolicies to Modern Technology, Air ForceStudies Board, Commission on Engineering andTechnical Systems, National Research Council,"Adapting Software Development Policies toModern Technology",
[MIL-DOC-2] NAVY-EC, DOD-HDBK-287, "A Tailoring Guide forDOD-STD-2167A, Defense System SoftwareDevelopment", 14 NOV 88 (DRAFT)
[MIL-DOC-31 Air Force Systems Command, Software ManagementInitiatives Implementation Plan, "ChangingPerspectives for Software Development", 23June 1989 DRAFT
[MIL-DOC-4] Air Force Systems Command, Software ActionTeam, "Software Management InitiativesImplementation Plan", 1 August 1989
(MIL-DOC-5] MIL-HDBK-MCCR (Proposed) Military Handbook onMission Critical Computer Resources SoftwareSupport (DRAFT), 15 December 1988
OTHER DOCUMENTATION
(FLETCHER-l] Fletcher J. Buckley, "Do Standards CauseSoftware Problems", IEEE Computer, September1989, pages 72-73
[WARREN89] Judith Warren, "Discussion Paper #1 SoftwareSupport Planning", US DOT/TSC, June 16, 1989,page 1
[SubCom89] Subcommittee on Investigations and Oversight,for Committee on Science, Space, andTechnology U.S. House of Representatives;"Bugs in The Program; Problems in FederalGovernment Computer Software Development andRegulation," September 1989
Page F-4
APPZNDIX G
GL4OSSARY/ACP.ONYNS
APPZNDIX G
GLOSSARY/ACRONYMS
AAF Associate Administrator for Airway FacilitiesAAP Automation ServiceACD Engineering, Research and Development ServiceACI Allocated Configuration IdentificationACN Engineering, Test and Evaluation ServiceADS Advanced System Design ServiceAHT Office of Training and Higher EducationALG Logistics ServiceAMC Management Control ServiceASA Advanced System Acquistion ServiceASE System Engineering and Program Management
OfficeASM Systems Maintenance ServiceAT Air TrafficATC Air Traffic ControlATR Air Traffic Plans and Requirements ServiceCCB Configuration Control BoardCDR Critical Design ReviewCM Configuration ManagementCOTS Commercial Off-the-ShelfCPFS Computer Program Functional SpecificationCRISD Computer Resources Integrated Support DocumentCSC Computer Software ComponentCSCI Computer Software Configuration ItemCSOM Computer Software Operator's ManualCSQPP Computer Software Quality Program PlanCSU Computer Software UnitDBDD Database Design DocumentDCI Design Configuration IdentificationDID Data Item DescriptionDOD Department of DefenseDRR Deployment Readiness ReviewFAA Federal Aviation AdministrationFCA Functional Configuration AuditFCI Functional Configuration IdentificationFQR Formal Qualification ReviewFSM Firmware Support ManualGFS Government Furnished Software
Page G-1
ICD Interface Control DocumentIDD Interface Design DocumentIRD Interface Requirements DocumentIRS Interface Requirements SpecificationIV&V Independent Verification and ValidationLSA Logistics Support AnalysisMOU Memorandum of UnderstandingMTP Master Test PlanNAS National Airspace SystemNDS Non-developmental SuftwareOSD Operation and Support DocumentOT&E Operational Test and EvaluationPCA Physical Configuration AuditPCI Product Configuration IdentificationPD Program DirectivesPDR Preliminary Design ReviewPMP Program Master PlanPTR Program Technical ReportQA Quality AssuranceQC Quality ControlSDD Software Design DocumentSDDD Software Detailed Design DocumentSDF Software Development FilesSDP Software Development PlanSDR System Design ReviewSE System EngineeringSPM Software Programmer's ManualSRS Software Requirements SpecificationSRS System Requirements StatementSSDD System/Segment Design DocumentSSS System/Segment SpecificationSUM Software User's ManualSTD Software Test DescriptionSTLDD Software Top Level Design DocumentSTP Software Test PlanTPI Technology Planning, IncorporatedVVD Version Description DocumentVRTM Verification Requirements Traceability Matrix
Page G-2
APPENDIX H
GUIDELINES SURVEY RESULTS
9i
This appendix contains the results of a survey conducted during theinterview process concerning the familiarity of the intervieweeswith the various standards, orders, and guidelines used by the FAAand in particular by the interviewees. Each person was asked toindicate which documents they had used or were familiar with atsome level and to indicate on a scale from 1 to 5 their level offamiliarity. Most familiar is a 5 and indicates that the personknows the document well enough to explain its use to someone else.Least familiar is a 1 and indicates they know of it but could notexplain its usage. A rating of 0 indicates that they did not knowof the existence of the document.
The raw results are presented here but no conclusions have beendrawn from this data for the following reasons. The sample is toosmall to be of much significance and the survey was not scientificin nature. The first three interviewees were not asked to indicatetheir familiarity on the 1 to 5 scale; this was a change in theinterviewing process. Their results arc; indicated under the'other' column.
If this data appears to be of interest, a more scientific surveyshould be conducted. The outcomes of such a survey would includeindications of:
a. training needs,b. lack of enforcement,c. old, unused, and un-needed guidelines,d. guidelines which are heavily used.
The heavily used guidelines may provide insight as to why someguidelines are successfully used •nd others are not.
The columns show numbers which are the number of interviewees whohad that level of familiarity with the guidelines. For example,the number 4 for DD Form 1423 under column header #5 means that 4persons indicated a level 5 of familiarity with that item. TheTOTAL column indicates the total number of interviewees whoresponded as knowing that item.
Page H-i
GUIDELINUS SURVEY RESULTS
r,,,." MENT NUMBPR TITLE #5 #4 03 02 11 Other TOTAL,
AC 00-41 FAA Quality Control System Certification 2 2_ Program (for guidance and information)
AFSC Pamphlet 800- Air Force Systems Command Software 1 1 1 343 Management Indicators
ANSI Y32.16 American National Standards Institute 1 1 2Reference Designations for Electrical andElectronic Parts
Do Form 1423 Contract Data Requirements List 4 3 1 1 2 11
DD Form 1664 Data Item Description 5 1 3 1 1 11
DoD FAR Supplement 127.410-6
DoD 5000.19-L, Vol. 1 111 AMSDL,
DoD-HDBK-287 A Tailoring Guide for DoD-STD-2167A. 1 1 2Defense System Software Development - -
DoD-STD-2167A Defense System Software Development 6 4 2 1 2 15
DoD-STD-2168 Defense System Software Quality Program 3 2 2 2 1 2 12
iuD-STD-480 ConfiTirAtion Control Re uirements 2 3 3 2 10
DOT/FAA/ES-85/03 NAS Training Plan 4 6
FAA Order ll00.121a Management of Air Traffic Control 2 1 1 4Automation
FAA Order 1100.124 AT/AF Responsibilities at NAS Computer 1 1 1 1 1 5Equipped ARTCCs ..
FAA Order 1100.127b Airway Facilities Sector Configuration 1 2 2 5
FAA Order 1100.134a Maintenance of NAS Automation Subsystem 1 1 1 3 1 7
FAA Order 1100.145b Program Technical Report (PTR) Procedures 7 1 1 2 11
FAA Order 1320.33 Equipment Modification and Facility 2 1 3instruction
FAA order 1320.48b Engineering Field Support Sector 1 1 3Maintenance Program Procedures
FAA Order 1370.52b Information Resources Management -1 3 4Pdlicies and Vrocedures
FAA Order 1370.53 Uniform Document Standards 0
FAA Order 1600.2 National Security Information 1 2 1 5
FAA Order 1600.40 Security for Electrunlcally Transmitted 1 2 1 4Message-
FAA Order 1600.54 Security of FAA Automated Data Processing 1 1 4 2 8Systems and Facilities
FAA Order 1600.8 Communication Security 1 1 2
FAA Order 1800.25 Configuration Control Support Facility_ 1 1
FAA order 1800.58 National Airspace Integrated Logistics 22 1 3 1 10Support Policy 1
Page H-2
GUIDELINES SURVEY RESULTS
DOCUMENT NUMSFR TITLE 15 94 @3 12 #1 Other TOTAL
FAA Order 1800.8f National Airspace System Configuration 3 2 1 1 3 1 11Mandaqement
FAA Order 1810.1d Ma4ur Systems Acquisition Management 3 1 I 1 5 11
FAA Order 1810.2 Independent Operational Test and 2 1 2 5,Evaluation for Major Systems Acquisition I - -
FA Order 1810.3 Cost Estimation Policy and Procedures 1 1 4
FAA Order 1810.4b FAA NAS Test and Evaluation Program 2 2 5 5 1 2 17
FAA Order 3020.1a Use of Computer-Based Instruction 2 2
FAA Order 4405.15 Reprocurement Data Acquisition Policy 1 1
FAA Order 4453.2a FAA Quality Control System Certification 1Program
FAA Order 4630.8 Quality Assurance Policy 1 1 1 2 5
FAA Order 4630.9 FAA Computer Software Quality Program 1 1 3 1 6Requirements
FKA Ordor 6000.10 Airway Facilities Service Maintenance 1 2 2 2 7Program (inactive) -
FAA Order 6000.30a Policy for Maintenance of the NAS 1 2 1 2 1 7
FAA Order 6032.1A Modification to Ground Facilities, 2 3 1 6Systems, and Equipment in the NAS
FAA Urder 6100.1a MaiitjLtiA1e of NAS EnRoutc Stage-A Ar 1 1 7Traffic Control System
FAA Order 6100.9c Quality Control 1 2 3
FAA Order 6120.1a Facility Modifications to ARTS lilA Air 2 1 1 4Traffic Maintained Software
FAA Order 7032.2b Air Traffic Operationpl Requirements 2 2 4
FAA Order 7800.2D Program Technical Report (PTR) Procedure 7 1 1 1 10
FAA Order 7800.7b Costing for Program System Version Updates 1 1
FAA Order 7880.22a Identification of Source Code Change 1 1
FAA-D-2494 Technical Publications 1 3 4
FAA-STO-002 Engineering Drawings 1 1
FAA-STD-005d Preparation of Specification Documents 2 4 6 2 14
FAA-STD-013a Quality Control Program Requirements 2 1 4 2 2 1 12
FAA-STD-016a Quality Control System Requirements 2 2 4 1 1 2 11
FAA-STD-018a Computer Software Quality Program 2 2 4 2 1 3 14Requirements
FAA-STD-021a Configuration Management (contractor: 2 3 3 3 1 1 13requirements) I I _ - -_
"FAA-STD-024a Preparation of Test and Evaluation 2 2 4 2 3 1 14Documentat ion
FAA-STD-025b Preparation of Interface ConLrol 1 3 3 2 1 11L# 9 ,• t cin nd Tnr trfare RmrntirementsDom o entation3
Page H-3
GUIDELINES SURVEY RESULTS
DOCUMENT NUMBER TITLE #5 #4 #3 #2 11 Other TOTAL
FAA-STD-026 NAS Software Development 4 1 3 2 1 3 14
FAA-STD-028 Contract Training Programs 1 3 2 2 1 9
FAA-STD-030 Preparation of Procurement Request 1 1 3 1 1 7Packages
FAA-STD-031 Preparation of Statement of Work 2 1 2 4 2 11
FAA-STD-034 Instructions for the Preparation of 1 2 2 5,_ Logistics Support Analysis (LSA) Data
FAA-STD-035 Commercial Equipment, Market Research for 1 1Preparation of Project ImplementationPlans
FAA-STD-036 Preparation of Project Implementation 1 1 4 1 1 8
FIRMR Federal Information Resources Management 1 2 1 4 8, Regulation ,.
1EEE STD 729 A Glossary of Software Engineering 1 1 1 4 7Terminology ...
manual U.S.Government Printing Office Style Guide 1 4 1 6
MIL-H-46855 Human Engineering Requirements for 2 1 3-- _ _ Military Systems, Equipment and Facilities
MIL-STD-1472C Human Engineering Design Criteria for 2 3 1 6Military Systems
MIL-STD-1521B Technical Reviews and Audits for Systems, 3 1 1 1 1 3 10
Equipments, and Computer Software
ML-STD-1815A Ada Programming Language 1 1 2 4 1 9
MIL-ST_-2076 Automated Test Equipment 3 _ 1 4
M__-$_TD-_077 Test Program Set Development 1 1 2
MIL-STD-2165 Testability Program for Electronic Systems 1 1 2and Equipment
MIL-STD-481A Configuration Control - Engineering 1 1 2 2 2 8Chan•ges, Deviations & Waivers
MIL-STD-482 Configuration Status Accounting 1 _ 1 4
MIL-STD-483 Configuration Management Practices for 1 1 3 2 7Systems, Equipment, Munitions, andComputer Programs; 21 March 1979 ...
MIL-STD-490P. Specification Practices; 4 June 1985 1 1 2 1 1 1 7
MIL-STD-499A Engineering Management 1 1 1 3NAS-MD-110 Terms and Definitions for the NAS 2 4 2 1 9
regulation Federal Procurement Regulations 11.307.1 1 1 2through 11.307.5
WA 0000.4H Washington Headquarters Directives 1Checklist as of February 1, 1989
Page H-4
![[Week 04/IID] Persona Design](https://img.pdfslide.us/doc/110x75/55ab7b7d1a28abb1418b482f/week-04iid-persona-design.jpg)
![iiD]J@ - libarchive.slcc.edu](https://img.pdfslide.us/doc/110x75/618125fde810af6b28728074/iidj-.jpg)
