Upload
maude-greer
View
225
Download
2
Tags:
Embed Size (px)
Citation preview
1
IV&V Facility
Pre-Software Assurance Symposium Facility Initiatives Briefing
Independent Verification & Validation of
Programmable Logic Devices
8 July 2005
NASA IV&V Facility
James Cercone, Ph.D., P.E.,WVU-Tech
Michael Beims, SAIC
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
2
Outline
• Review IV&V of PLD Research Project Objectives and
Framework
• Review of detailed technical findings and VHDL defect
taxonomy
• Provide overview of Work Instruction development
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Pre-Software Assurance Symposium Facility Initiatives Briefing
3
NASA-STD-8739.8
Software V&V is concerned with ensuring that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products.
….. IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization. For NASA, IV&V is performed and/or managed by the NASA IV&V Facility.
…“Software includes programs and operational data contained in hardware (e.g. firmware, programmable logic, and programmable gate arrays).”
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V PLD Status
4
IEEE STD 1012-1998
IEEE Standard for Software Verification and Validation, provides supporting information regarding the integration of IV&V into every step of the Software Development Life Cycle. The IEEE standard, like the NASA Standard, also cites firmware and microcode in its definition of software: “This standard applies to software being developed, maintained, and reused …. The term Software also includes firmware, microcode, and documentation.”
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V PLD Status
5
IEEE Std 1076™-2002
Abstract:VHSIC Hardware Description Language (VHDL) is defined. VHDL is a formal notation intended for use in all phases of the creation of electronic systems. Because it is both machine readable and human readable, it supports the development, verification, synthesis, and testing of hardware designs; the communication of hardware design data; and the maintenance, modification, and procurement of hardware. Its primary audiences are the implementers of tools supporting the language and the advanced users of the language.Keywords:computer languages, electronic systems, hardware, hardware design, VHDL
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V PLD Status
6
IV&V Facility
NESC Project Activities performed by IV&V
From Project Plan (SAIC Document #ISTO-05-98-192), Section 3 – Activities
Completion Date
1.Identify the FPGA design logic faults from:• NASA and industrial sites,• Document Artifacts, and• Comparison of typical FPGA logic design methods with proven software engineering methodologies, including those used for design and peer review
Year 1, 2Q
2. Identify existing software engineering methodologies that can be directly applied to FPGA designs by tracing common defects to their underlying cause
Year 1, 3Q
3. Suggest enhancements to developers’ design and peer review methodologies
Year 2, 1Q
4. Provide field prototyped training materials for performing PL software V&V Year 2, 4Q
5. Successfully complete a pilot project Year 2, 4Q
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Primary Goals:
• Develop an IV&V strategy for PLD’s
• Provide field proven PLD Work Instruction (WI) to the IV&V practitioner
7
IV&V Facility
Activities Thus Far• Better understanding of PLD’s at WVU Tech and SAIC
– Primarily via literature searches and attendance at workshops– Presentation to IV&V CAWG– WVU Tech obtained and learned IDE’s (Integrated development
Environment) for both Actel and Xilinx PLD’s. WVU Tech has also obtained and learned Active HDL
Limited analysis and simulation of NASA project data• IV&V has mapped PLD’s into a better framework for IV&V WI development• Identifying a taxonomy of defects in VHDL Domain
– Via Literature Search– By comparing VHDL releases for the same chip– Evaluating SW code defects that can be “mapped” to PLD VHDL defects
• Had initial discussions with JWST as candidate for VHDL Pilot Project
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Activities/Results thus far (9-1-05 through 8-8-05)
8
IV&V Facility
Activities/Results thus far (9-1-05 through 8-8-05)
Results Thus Far• Scoped PLD IV&V Framework to understand 05 accomplishments and
identify potential future year activity– Based on increased understanding, and– Realization that existing IV&V Code Analysis WI is insufficient for PLD
analysis• Defect taxonomy, in process of refinement, for presentation at MAPLD
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
9
IV&V FacilityDevelopment Environments
Idealized Software/PLD Development
Requirements Design Code Test
• SRS is a CM’d document• Rigorous flowdown is common
• SDD is a CM’d document• Manual or tool generated
• Different types of code (C, C++, etc)• Mature tools avail to aid in development, verification
• Performed on implemented code• Unit, Subsystem, System testing follows req’ts flowdown• Rigorous processCommon PLD Development
Requirements Design/codeand simulateat functional
level
Design/code and simulate
After chip layout
Testing afterPLD is
programmed• Part of subsystem• Hardware artifact (e.g. EQ spec, product functional spec) • Performed on
implemented PLD• Unit, Subsystem, System testing follows req’ts flowdown
• It is at this stage that Idealized/Common development processes diverge
•Design process•IDE•Target
• PLD’s also have timing concerns that are rare in software development, such as
•Synthesized versus native components•Race Conditions•Adequately buffering data
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
10
IV&V Facility
Current Year (2005) Activities against PLD IV&V Framework
Our current WI activity focuses on verification of VHDL Design• Develop Work Instruction• Flesh out Work Instruction with Pilot Project• Deploy Work Instruction at Facility
Requirements Design/codeand simulateat functional
level
Design/code and simulate
After chip layout
Testing afterPLD is
programmed
Common PLD Development
Similar to FSWbut artifactexpectationsneed to bearticulated by IV&V
VerificationTasks
ValidationTasks
tbd
1) Ensure syntax is correct2) Identify typical errors3) Develop/deploy WI: Programming Standards andDefect ID• VHDL• Verilog• Schematics
Traceability of RequirementsIdentify key timing areas and independently simulate
Identify any known issueswith IDE and ensure potential errors not presentin developed product
Re-simulate key timing functions
Verify tests performedby developer (using simulation to generatetest cases)
tbd, independenttesting?
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
11
IV&V Facility
Ideas for Future Year Projects
Future Projects• Develop WI for verification of Verilog or Schematic designs (item “a”)• Provide WI for Requirements Analysis or Test Analysis of PLD’s (item “b”)
• This is probably a straightforward extrapolation of current WIs for FSW• Have the breadth of knowledge on development tools and all target PLD’s (item “c”)• Provide insights on timing/validation aspects of PLD implementation (item “d”)
d
b
b
a c
Requirements Design/codeand simulateat functional
level
Design/code and simulate
After chip layout
Testing afterPLD is
programmed
Common PLD Development
Similar to FSWbut artifactexpectationsneed to bearticulated by IV&V
VerificationTasks
ValidationTasks
tbd
1) Ensure syntax is correct2) Identify typical errors3) Develop/deploy WI: Programming Standards andDefect ID• VHDL• Verilog• Schematics
Traceability of RequirementsIdentify key timing areas and independently simulate
Identify any known issueswith IDE and ensure potential errors not presentin developed product
Re-simulate key timing functions
Verify tests performedby developer (using simulation to generatetest cases)
tbd, independenttesting?
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
12
Complexity is a Challenge for all Design Representations
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V Facility
Functional Trace / Performance Test
Design Trace / Functional Test
?
13
Review of detailed technical findings
and
VHDL defect taxonomy
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Pre-Software Assurance Symposium Facility Initiatives Briefing
14
IV&V Facility
Entity
Are Signals defined in the port list as out type signals given values?
Are Signals that are defined in the port list as inout type signals used for both
– reading and writing?
Are Signals defined in the port list as in type signals used in the architecture?
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Potential VHDL“Hot Spots” visible in semantics
15
IV&V Facility
Process
Is there a series of sequential statements followed by a branching structure?
Is there a branching structure followed by a series of sequential statements?
Is each process sensitive list made up of the signals from the Entity’s port list?
Sample Findings Potential VHDL“Hot Spots” visible in semantics
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
16
IV&V Facility
If Structures
Having elsif and no else statement
Having neither an elsif or else statement
Is there unreachable code inside an else statement?
When using a compound if statement, are all possible conditions covered in subsequent elsif and else statements.
How deep is the nesting of if structure?
Testing Signals in the condition that are not part of the process’s sensitive list
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Potential VHDL“Hot Spots” visible in semantics
17
IV&V Facility
Signal Assignment
Is the same set of Signals assigned values in each of the if-elsif-else sections?
Is the same set Signals assigned values in each of the case structures
when and when others => clauses?
Are all Signals in a component’s port list mapped values during a
Component’s instantiation?
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Potential VHDL“Hot Spots” visible in semantics
18
Number of Lines Number of Blank Lines 311 Number of Comment Lines 1157 Number of Library and Use lines 130 Number of Packages 1 packages 520 Number of Entities 25 entities 525 Number of Architectures 25 architectures 6291 Total Number of Lines Examined 8934
Sample Findings Static Metrics Analysis of Public Code
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
19
Process Summary 129Number of Processes with 0 <= line count < 10 13Number of Processes with 10 <= line count < 20 61Number of Processes with 20 <= line count < 30 22Number of Processes with 30 <= line count < 40 6Number of Processes with 40 <= line count < 50 5Number of Processes with 50 <= line count < 60 2Number of Processes with 60 <= line count < 70 3Number of Processes with 70 <= line count < 80 5Number of Processes with 80 <= line count < 90 2Number of Processes with 90 <= line count < 100 0
Process with line count >=100 Number of Lines101108109113117152166194217927Note !
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
20
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
Overall Comparison of LAT TD docs
Filename: 1880 1881 1882 1883 1885 2147
Version 50-1 50-1 50-1 50-1 50-3 50-1
Number of Files 13 25 20 21 30 5
# of Comment Changes 0 0 0 0 0 0
# of Functionality Changes 0 0 0 0 0 0
# of Lines 3460 9793 5887 7413 9603 813
# of Blank Lines 0 0 0 0 0 0
# of Commented Lines 561 1214 906 981 1269 124
# of Partial Comments 447 1165 555 781 559 49
# of Library 31 53 44 48 57 10
# of Use 44 77 64 70 109 19
# of Package 1 1 1 1 3 1
# of Entity 14 25 21 23 28 4
# of Architecture 14 25 21 23 28 4
# of Component 23 31 26 25 33 4
# of Signal 134 433 242 288 789 27
# of in 203 469 603 666 758 49
# of out 116 261 410 453 545 50
# of inout 37 75 18 24 17 0
# of if 70 321 139 205 293 13
# of elsif 42 283 100 91 137 3
# of else 41 201 78 132 216 4
# of case 7 33 17 18 18 1
…
21
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
Observations related to Code Changes (absolutes for the public code analyzed)
A code change occurred if:
•There were 12 or less files•Average number of lines per file was greater than 400•There were less than 3 “package” statements•There were less than 11 “entity” statements•There were less than 20 “component” statements•There were less than 13 “architecture” statements•There were less than 20 “clk’event” statements•There were any “while” statements•There were any “wait” statements•There were any “after” statements
22
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
Observations related to Code Changes (normalized for the public code analyzed)
A code change occurred if:
•5% or more of the total lines were “signal” statements•5% or less of the total lines were “in” statements•5% or less of the total lines were “out” statements•3½% or more of the total lines were “if” statements•¼% or more of the total lines were “case” statements
23
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
24
The Percent of "If" Statements Versus The Number of Functional Changes
0
100
200
300
400
500
600
700
0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000
Percent of "If" Statements
Num
ber o
f Fun
ctiona
l Cha
nges
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
25
The Percent of "case" Statements Versus The Number of Functional Changes
0
100
200
300
400
500
600
700
0.0000 0.1000 0.2000 0.3000 0.4000 0.5000 0.6000 0.7000
The Percent of "case" Statements
The Number of Functional Chan
ges
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings Static Metrics Analysis of Public Code
26
3b Problem: wait statements with different conditions in the same process.
Example: processbegin wait until rising_edge(clk1); x <= a; wait until falling_edge(clk2); x <= a;end process;
Context: Synthesis: this construct may not be synthesizable
Explanation: Would require flip flops to be sensitive to different clock edges at differenttimes.
Processes with multiple wait statements are turned into finite statemachines. The wait statements denote transitions between states. Thetarget signals in the process are outputs of flip flops. Using different waitconditions would require the flip flops to use different clock signals atdifferent times.Multiple clock signals for a single flip flop would be difficult to synthesize inefficient to build fragile to operate.
Sample Findings
Sample taxonomy of semantic defects visible in VHDLIV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
27
Problem: Attempt to read an output port.
Example: entity DFF is port( D, CLK : in std_logic; Q : out std_logic )end DFF;
architecture badarchitecture of DFF isbegin case Q is when ‘0’ => … …end;
Context: Compilation: This should result in a compilation error.
Explanation: Most compilers flag this as an error, but Xilinx permitted the code abovewhere an out port is used as an argument to the case statement.
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings
Sample taxonomy of semantic defects visible in VHDL
28
Problem: In an asynchronous reset, the test for reset occurs outside of the test forthe clock edge.
Example: process (reset, clk)begin if (reset = ’1’) then q <= ’0’; elsif rising_edge(clk) then q <= d1; end if;end process;
Context: Synthesis: synthesizable, but not desirable
Explanation: Asynchronous resets are bad, because if a reset occurs very close to a clockedge, some parts of the circuit might be reset in one clock cycle and some inthe subsequent clock cycle. This can cause the circuit to be out of sync as itgoes through the reset sequence, potentially causing erroneous internal stateand output values.
Note: NASA experts’ recommended practice prevents an ‘out of sync’ by insuring that resets are never very close to a clock edge. This design is seen in NASA flight software as a D-FF with reset.
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings
Sample taxonomy of semantic defects visible in VHDL
29
Synthesis vs Simulation difference visible in VHDL
Multiplexer with missing sensitivity signal (signal “b”)
process(a,sel) if sel = '1' then out <=1; else out <= b; end if;end process
www.synplicity.com/literature/pdf/HDLDesignMethods.pdf
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Sample Findings
Sample taxonomy of semantic defects visible in VHDL
30
IV&V Facility
Fault Detection Matrix
Actual
Tested
False True
Positive False/PositivePotential Hot Spot identified/ No defect
True/PositivePotential Hot Spot identified/ Design defect exists
Negative False/NegativeNo Hot Spot identified/ Defect exists
True/NegativeNo Hot Spot identified/ No Defect
High Confidence – IV&V success
Less Confidence - Mission Risk
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
31
IV&V Facility
True/Positive Potential Hot Spot identified/ Design defect exists
“Pre-processor directives and usage are often not controlled by coding standards.
1) #ifdef statements can be left active or inactive and permit non-flight code (e.g. test code) to be compiled. Instances of such errors have been found in Mars program code.
2) #define statements can be left in the code from testing leaving test values or conditions active in the flight code. Instances of such errors have been found in Mars program code.”
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Fault Detection MatrixExamples
32
IV&V Facility
Fault Detection MatrixExamples
process (A,B)
begin
if (cond1)
X <= A + B;
elseif (cond2) X <= X – B;
end if;
end process;
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
If neither cond1 nor cond2 is true, then X will retain its value ...basically, X is stored in a latch
In general, latches are not usually recommended in synchronous designs
False/Negative No Hot Spot identified/ Defect exists
33
IV&V Facility
Taxonomy of Common Visible Defects
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Type of Defect Definition Runtime Errors
Arithmetic Exception Dividing by zero, or returning a negative value instead of a positive one.
Logical Errors Includes the errors in state transition, timing, control, and data flow. Initialization or incorrect initialization of a value.
Abnormal Value Variables should not be assigned to signals. Duplicate Object Objects that have been defined more than one time (i.e.
functions, variables and macros). Compile Time Errors Syntax Errors Errors that are problematic in compilation. Scope and Linking Errors
These errors deal with visibility of functions and libraries, and a global declaration versus a declaration seen by a local function.
Other Defects Unused Objects These objects are defined but never used. Coding Standard Violation
This includes duplicate code, giving meaningful names to different variables and code fragments, and keeping track of the names given to different code packages to prevent two or more packages receiving the same name.
Comment and Spacing Errors
Comment and spacing errors can occur when changing formats such a PDF format to DOC format.
34
IV&V Facility
VHDL Code Severity Chart
Severity Description
A FPGA code will not perform desired tasks. Mission is jeopardized.
B Serious hindrance to mission accomplishment. A serious cost effect to project. I.E. (No “quick fix”)
C Adversely affects FPGA code performance. A minimal cost effect to project but a “quick fix” is possible.
D Annoying effect to user but FPGA code is operable.
E Any other effect.
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
35
Overview of Work Instruction
Development
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Pre-Software Assurance Symposium Facility Initiatives Briefing
36
Work Instruction Development
Background Considerations
VHDL specific considerations
Taxonomy of potential “Hot Spots”
Clock and Reset LinesSensitivity ListsFeatures not consistently supported between IDE’sNon-implemented features (i.e some attributes)
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
37
IV&V Facility
VHDL Code Severity Chart
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Defects /Runtime Errors Impact of Error Defect Criticality (dependent on Functional Criticality)
High Medium Low Arithmetic Exception Termination of a function or
produce an erroneous behavior A-B A-B B-C
Logical Errors Cause erroneous behavior that can lead to mission failure
A-C A-C C-D
Abnormal Value Allocation Error
Variables assigned to signals can produce unexpected behavior
A-D A-D B-D
Duplicate Objects Unexpected results are produced if definitions are different.
C-D C-D D-E
Defects/Compile Time Errors
Syntax Errors C-D C-D C-D Scope/Linking Errors
Errors should never be seen, since they should have been fixed in the review and verification stage.
C-D C-D C-D
Other Defects Unused Objects Produce wasted memory space and
harder code to maintain D-E D-E D-E
Coding Standard Violation Lots of error producing potential and difficult to maintain
D-E D-E D-E
Comment and Spacing Errors
Can produce errors that make functional flow difficult to follow
D-E
D-E D-E
38
IV&V Facility
VHDL Code Severity Chart Examples
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Example of High Functional Criticality -
The FPGA with this defect is the only functional capability for the Satellite to deploy the solar panels. If the FPGA does not perform this function, the satellite will run out of power, causing loss of the mission.
39
IV&V Facility
VHDL Code Severity Chart Examples
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Example of Low Functional Criticality -
The FPGA controls one side of a dual redundant path to the telemetry transponder. If the FPGA fails, then the telemetry is routed via CPU control, causing a momentary delay in telemetry, if all telemetry is buffered through on-board storage. (Note: since this is a design (software) defect, if there were two identical FPGA's controlling this functionality, instead of an FPGA and the CPU, then the redundant FPGA can be expected to fail in the same manner and there is no functional redundancy, making this a High Functional Criticality defect.)
40
IV&V Facility
Example of Vendor Specific Degree of Compliance
• “major differences between XVHDL and Express is IEEE VHDL-93 compliance. XVHDL is a fully IEEE VHDL-93 compliant tool. Express supports many of the most commonly used VHDL-93 synthesis constructs, but is not yet fully compliant; it remains officially compliant with the IEEE VHDL-87 standard.”
• http://www.xilinx.com/xlnx/xil_ans_display.jsp?iLanguageID=1&iCountryID=1&getPagePath=5144 (7/21/2005)
Compliance Issues
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
41
IV&V Facility
Example of Vendor Specific Compliance
(two examples)
• Signal Declaration– Supported ("register" or "bus" type signals are not supported)
• Attribute– Only supported for some predefined attributes: HIGH, LOW, LEFT, RIGHT, RANGE, REVERSE_RANGE, LENGTH, POS,
ASCENDING, EVENT, LAST_VALUE– Otherwise, ignored.
• http://www.xilinx.com (7/21/2005)
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Compliance Issues
42
IV&V Facility
Defect Taxonomy (VHDL Verification at Functional Design), and Pilot Deployment
• Draft of VHDL programming standards geared toward defect identification
– Defects commonly detected by compilers are not included– Includes syntax, timing margins, clock boundaries
• This draft is in process of update, peer review– Align defects with known coding defects– Test draft product against actual VHDL text
• Developer places multiple revs of VHDL on website
• The results will be presented at MAPLD in September, 05
Note: GLAST LAT used Actel VHDL for design and this served as basis for IR&D project.MRO project used Xilinx Verilog for design.
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Work Instruction in 05 addressed VHDL verification. Future tasks needed to address:
Verilog and Schematic verification plus validation.
43
IV&V Facility
Preliminary Considerations for Defect Detection in VHDL Based Designs
• Materials needed to start the verification process are:
• Design Documentation to analyze performance• Actual VHDL Code • Code Pedigree
» (Reused modules, designers, level of experience…)• Development and Analysis Tools State diagrams. • Clock Trees• NASA and IEEE Standards
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
44
IV&V Facility
• V&V Process and Procedure at the Code Level
– Static Metric Analysis– Code Coverage (particularly for behavioral level designs)– Verification of Clock and Reset Tree’s (if provided)– Check for compliance to NASA Standards– Check for device resource usage
• (synthesized vs. board components such as MAC’s, SR, and DFF’s)
– Check of IDE specific restrictions – Check VHDL specific “Hot Spots”
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Preliminary Considerations for Defect Detection in VHDL Based Designs
45
Conclusion
• Review IV&V of PLD Research Project Objectives and
Framework
• Review of detailed technical findings and VHDL defect
taxonomy
• Provide overview of Work Instruction development
IV&V Facility
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
Pre-Software Assurance Symposium Facility Initiatives Briefing
46
IV&V Facility
Background Slides
47
IV&V FacilityIndependent• Technical: IV&V prioritizes its own efforts• Managerial: Independent reporting route to NASA Headquarters• Financial: Budget is allocated by NASA to the IV&V Facility such that
IV&V effectiveness is not compromised
Verification (Are we building the product right?)• The process of determining whether or not the products of a given phase of the
software development lifecycle fulfill the requirements established during the previous phase
• The product is internally complete, consistent and correct will support the next phase
Validation (Are we building the right product?) • The process of evaluating software throughout its development process
to ensure compliance with software requirements. This process ensures:– Expected behavior when subjected to anticipated events– No unexpected behavior when subjected to unanticipated events– System performs to the customer's expectations under all
operational conditions
What is IV&V ?
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
48
IV&V Facility
Maximizing Project V&V and IV&V
• The Project V&V goes end-to-end
– Needs sufficient depth to help ensure that they have build the product right and built the right product
• IV&V needs to be the second line of defense
– Select the most critical functionality and then IV&V to the appropriate depth -- not exceeding the IV&V point of diminishing returns (maintaining reasonability)
– The cut-off point should be where we have found the critical defects and also gained enough confidence in the software to support mission assurance requirements and launch recommendations
• Project Teams should compare our criticality rankings to their knowledge of the development as an independent source and explore differences
• Project Teams should look at activity just below the IV&V line to ensure adequate V&V resources
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
49
IV&V FacilityPLD Updated Framework
• Framework to perform IV&V was updated to– Take into consideration that development of PLD’s is different than development
of software,– Address verification and validation tasks explicitly
• PLD’s are developed in an environment that combines design, code and simulation simultaneously
– Timing aspects much more critical in PLD’s
• Major differences needed to be addressed:– PLD development is part of subsystem development – comparable artifacts not
consistently generated– PLD design can occur in many forms
• Schematic (representation similar to chip/board design)• VHDL (representation similar to Ada)• Verilog (representation similar to C/C++)
– Target system matters• Syntax different even when same language used
– Development environment matters• From a syntax standpoint• From a capability standpoint (e.g. software motivated or hardware motivated)• Which revision of IDE (multiple releases for hw motivated IDE’s)July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
50
IV&V Facility
Updated IV&V Framework for PLD Development
Requirements Design/codeand simulateat functional
level
Design/code and simulate
After chip layout
Testing afterPLD is
programmed
Common PLD Development
Similar to FSWbut artifactexpectationsneed to bearticulated by IV&V
VerificationTasks
ValidationTasks
tbd
1) Ensure syntax is correct2) Identify typical errors3) Develop/deploy WI: Programming Standards andDefect ID• VHDL• Verilog• Schematics
Traceability of RequirementsIdentify key timing areas and independently simulate
Identify any known issueswith IDE and ensure potential errors not presentin developed product
Re-simulate key timing functions
Verify tests performedby developer (using simulation to generatetest cases)
tbd, independenttesting?
• The above updated IV&V framework has more detail– Allows us to clearly understand what we have accomplished, and what lies ahead– Strategy is to perform accomplished tasks well– For each task performed,
1. Develop Work Instruction (WI), flesh out internally2. Test on pilot project3. Deploy updated WI
• The fast pace of PLD product evolution requires additional considerations– Important to have cognizance of market trends– Update WI appropriately with trends that will be implemented in spacecraft in the near term
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
51
Simulink Example
http://www.transtech-dsp.com/software/simulink.asp
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V Facility
52
Small Microprocessor Example
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V Facility
53
Examples of Rare Software Defects
July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone
IV&V Facility
•PLD’s also have timing concerns that are rare in software development, such as:
–Synthesized versus native components – Seen in Stinger real time seeker simulations. Debate over whether a native compiler matrix multiplication routine was sufficiently predictable versus a matrix multiply built up in separate Fortran 77 instructions.–Race Conditions – Seen in the “Response to a Setting Satellite Vehicle” scenario in the Space Shuttle GPS’ firmware.–Adequately buffering data – Seen in the Space Shuttle Primary Avionics Software Systems’ Mid Frequency Executive where every variable must be analyzed for time homogeneity and treated accordingly.
•In General, “PLD–like” defects are seen in hard, real time software systems, which in turn are the primary candidates for migration to PLD’s in the near future (recent past?)