F:MAPLD 2005
MAPLD 2005
Computer Overload and the Apollo 11 Landing
(An Insider s View )Jack Garman
NASA Retired (now Lockheed Martin Information Technology)
Garman 2 F:MAPLD 2005
Introduction
• A “ring-side seat”
• My scope of knowledge
• Parkinson’s Law– Work expands to fill the time available for its
completion • A “proverb” coined by the twentieth-century British
scholar C. Northcote Parkinson, known as Parkinson’s Law.
• It points out that people usually take all the time allotted (and frequently more) to accomplish any task.
Garman 3 F:MAPLD 2005
Themes• A Story
– “Long, long ago, in galaxies far, far away”
• The Legacy (Observations)– Flexibility
• As in “change” and evolution• Parkinson’s Law Corollary - software requirements
change and expand to exceed all reasonable projections
– Fault Tolerance• Responding to the impossible• Planning for things that just “can’t happen”
– Layers of abstraction are real• No one really sees “the whole”
– Time• Everything is different, but everything is the same
Garman 4 F:MAPLD 2005
Painting an Image of the Times• No “Internet”, no E-mail, no “graphics”• No “personal” computers
– Word and PowerPoint “by hand”,Excel via (GFE) slide rule
– “Live” CRT’s were a marvel– “The first typewriter”
• Embedded computers– Digital flight control (vs. analog)– Guidance and Control computers
• “Apollo Guidance Computer”
– Technology: • speed, core memory, rope memory, no disk
Garman 5 F:MAPLD 2005
Painting an Image of the Times (cont’d)• Simulation Concepts
– in other than real time– Voice and data delays
• IT Expertise (relatively small)– Operating systems or “systems software”– Software Engineering
• Amazing for the day – primitive by today’s standards• esp requirements, testing, code inspection
• Youth and naiveté (and politics)– Obedient, smart, dedicated– Politics largely invisible to the troops
• Protected, limited comm - no internet, no “NASAWatch”, no “cup half empty”
Garman 6 F:MAPLD 2005
Apollo• The Saturn V and IU
• The CSM
• The LM
• The AGC – and CMC and LGC – and AGS
• Software “freeze”
Garman 7 F:MAPLD 2005
Layers of Abstraction (IT viewpoint) The Astronauts
“Piloting” (and training) The flight control system The guidance system (and orbital mechanics) The flight software (and development systems) The onboard computer and associated gear The spacecrafts (or simulators) and subsystems
The C3 links The ground networks The Mission Control Center The mainframe and associated gear The Mission Control Center (MCC) software
(and development systems) The consoles“Mission Operations” (and simulation/training)
The Staff Support rooms The Flight Controllers and “MOCR”
Me
Garman 8 F:MAPLD 2005
Flight Computers and Software• The design of the “avionics” (guidance
computer software)– Approach to process mgmt
• Async vs. Sync• Continue in light of the unknown
– Approach to Fault Recovery• Single string, highly reliable, “failure not an
option”– Vs. Multi-string, FOFS, “quit” on hard fail– Nightmare: All quit
• It meant the software had to be able to “restart”– Continue in light of the unknown– Nightmare: Restart loop
Garman 9 F:MAPLD 2005
The MCC “Consoles”
TV or “digital”
Speaker
Display Select
“Event” lights
“P”-tube
“Voice Loop” Controls
NO KEYBOARDS
Garman 10 F:MAPLD 2005
Training• Mission Control
– Sci-Fi to a young engineer in the late 60’s– Reading the screens in Mission Control
• Crew Trainers– ISAGC– Simulations
• Types of “training”– Simulations and “Integrated Sim’s”
• Playing both sides
– Pad Tests (MCCH tied to the real vehicle)
Garman 11 F:MAPLD 2005
Mission Control Center Houston• Very hierarchical (command structure)
– MOCR• “Board of Governors”• Flight Director• Flight Controllers
– Capcom– Comm– The “Trench”– Systems– Medical– Support
• “Simsup”– SSR’s (“the backroom experts”)– Infrastructure Support
• MCCH, computers, networks…– Engineering Support
• NASA engineers and contractors around the country
Garman 13 F:MAPLD 2005
The Story• “Setting up” a failure that “can’t
happen”– “playing both sides”
• The Results – and preparation
• The “Real Thing”– 85 + 15 = 100– Alarms, master caution and warning
Garman 14 F:MAPLD 2005
The Console “Cheat-Sheet”
In retrospect is it somewhat terrifying to me that this was even permitted as a “tool” or procedure for a 24-year-old engineer sitting at a console in Mission Control during the first landing on the Moon
Garman 15 F:MAPLD 2005
The Story (cont’d)• Two moments of “reality”
– The “echo”– “We’ve got dust…”
• Rich has it all online– http://www.klabs.org/history/apollo_11_alarms/console/
Garman 17 F:MAPLD 2005
Flexibility (change and evolution)• Shuttle Legacy
– was/is fly-by-wire• Software fixed hardware development issues• Vehicle development almost “killed” software
development– used a HOL, but a real battle
• “measured” 15% penalty in size• Qualitative gain in reliability• Infinite gain in change (unexpected)
(Parkinson’s Law: Shuttle flight software was “developed” at least three times)
• General (into today)– Extraordinary tools
(and additional layers of abstraction)• Good News: Change is easier• Bad News: Change is easier
Garman 18 F:MAPLD 2005
Fault Tolerance• Shuttle Avionics Legacy
– FO/FS and “sync points”– The sync/async “wars”– BFS vs. PASS (vs. “BASS”) and S/W fault
tolerance
• Testing– SPF, Crew Trainers, SAIL– Cycle stealing – testing the impossible
• Murphy’s Law– Good: A flight computer failed without impact on
ALT– Bad: The first orbital launch was delayed 48 hrs
due to a PASS/BFS interface 1:67 failure
Garman 19 F:MAPLD 2005
Layers of Abstraction are Real• No one “knows” the whole
– Horizontal vs. vertical knowledge becoming more and more an issue
– Egocentricity• Downward: Engineering dependency on “lower layer”
black boxes ever increasing– The dilemma of “off the shelf”
– Apollo to Shuttle to Station “flight computer” in degree
• Upward: Knowing where my piece, my “black box”, fits in the whole is more and more difficult
• Anecdote– DoD Software Engineers projection: 1988
Garman 20 F:MAPLD 2005
The Legacy (concluded)– Flexibility (“change”)– Fault Tolerance (“can’t happen”)– Layers of abstraction are real
• Time and Change– Everything is different today
• but everything is the same!
– If it works, don’t fix it!• BUT - if it works, it’s obsolete!