Upload
rosanna-cobb
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Verifying AutonomousPlanning Systems
Even the best laid plans need to be verified
Prepared for the 2005Software Assurance
Symposium (SAS)
DS1 MSLEO1
RajeevJoshi
GordonCucullu
GerardHolzmann
BenjaminSmith
MargaretSmith (PI)
Affiliation: Jet Propulsion Laboratory
Importance
This work is pursuing a solution!
Autonomous Planning Systems (APSs) determine what the spacecraft / rover / installation should do.
Compared to conventional software, they are able to determine thisin a wide range of circumstances.
As a result,• no need for continual oversight (save on 24/7 operations staff)• more science is done (avoid delay of calling back to Earth)• improved safety (more proactive than just “safe mode”)
But because APSs must operate in a wide range of circumstances – far too many to test, even if you could predict them all,
how can you trust them to do the right thing???
SAS_05_Verifying_Autonomous_Planners_Smith
How to getfrom A to B
?
Consequences of a bad planWasted Resources
out of resources
SAS_05_Verifying_Autonomous_Planners_Smith
missed science goal
How to getfrom A to B
?
Consequences of a bad plan:Loss of Mission
SAS_05_Verifying_Autonomous_Planners_Smith
Solution
SPIN Model Checker• Logic Model Checker used to formally verify distributed software systems.• Development began in 1980 at Bell Labs
– publicly distributed source code since 1991• Most widely used logic model checker with over 10,000 users. • Recipient of 2002 System Software Award for 2001 from the Association for Computing Machinery (ACM)• Verifies software using a meta language called Promela
– requires that system being verified be expressed in Promela• SPIN flags deadlocks, unspecified receptions, incompleteness, race conditions and unwarranted assumptions
about relative speeds of processes
Challenge:
Assure that all plans generated by the APS are safe for the spacecraft.
The current empirical testing approach is insufficient because it lacks coverage.
Solution:
Replace current empirical testing with model checking.
Model checking offers exhaustive or measurable test coverage leading to greater confidence in correctness.
SAS_05_Verifying_Autonomous_Planners_Smith
Testing
~100 plans
undesirableplan
all desirable plans
Empirical Testing(current approach)
undesirable plan (error trace) no errors
Testing with the SPIN Model Checker(our work)
inputmodel
Manually inspectplans to identify
undesirable plans
endtesting
Adjust modelto exclude
undesirableplan
propertiesof desirable
plans
Adjust modelto exclude
undesirableplan end
testing
Testing
Approach
limited by time
required to
inspect sample
plans
limited only by
memory and
processor
speed
inputmodel
PromelaModel
requirements
requirements
plans analyzesbillionsof plans
SAS_05_Verifying_Autonomous_Planners_Smith
• APS are needed by NASA projects to reduce operations costs and meet science return requirements.
• Our work retires an important class of risks inherent to all missions using APS.– we replace an inadequate testing method with a
method that has greatly improved and measurable test coverage.
Testing methods must keep pace with the highly complex, autonomous systems we need and are developing.
Relevance to NASAtesting software
complexity
SAS_05_Verifying_Autonomous_Planners_Smith
Accomplishments
sample
image
compress data
uplink
oven1
oven2
camera
drill location
power use
memory use
sample1 sample2
image 1 image 2
uplink
compress
off-cool
off-cool
on off-warm off-cool on off-warm off-cool
off on off
hole1 oven1 hole7 oven1
• For DS4 / Champollion APS model, used model checking to find a deadlock error – 10 activities = exploration of ~ 3 million plans
• Selected Earth Observer 1 as a target mission for application of our work. – 100+ activities = more plans than atoms in the universe!!!
• Current empirical method of where ~100 plans are tested is woefully inadequate.
• Our approach: Use model checking to greatly improve testing coverage = billions of plans.
– prune the search space through the use of constraints
sample2
deadlock:
out of memory
• Currently working on a set of automated tools for automatically converting APS for model checking
SAS_05_Verifying_Autonomous_Planners_Smith
• Our goal: to improve APS testing capabilities which have been an impediment to the acceptance of APS for other than experimental use.
• How we will get there:– complete implementation of a set of tools to fully automate
model checking of APS models– improve coverage from hundreds of test cases to billions of test
cases.
Where we are Going
SAS_05_Verifying_Autonomous_Planners_Smith