Leverage data analytics to reduce human space mission risks Presentation

© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in

the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation

MACH-5 INSIGHT℠Orion High Data Rate Use Case

October 1st 2015

Haden [email protected]

Jason [email protected]

mailto:[email protected]







Agenda Orion Program Overview High Data Rate Ingest Use Case MACH-5 INSIGHT℠ Platform SFDU and Protobuf HBase bulk loading using Spark

Streaming TRACE Visualization



Orion OverviewOrion Multi-Purpose Crew Vehicle, NASA’s first spacecraft

designed for long-duration, human-rated deep space exploration. Orion will transport humans to interplanetary destinations beyond low Earth orbit, such as asteroids, the moon and eventually Mars, and return them safely back to Earth.



The Orion program will enable NASA to:

Carry out a robust human and robotic exploration program that is both sustainable and affordable

Extend human presence across the solar system to our planets, asteroids and other destinations

Develop the innovative technology, knowledge, and infrastructure needed to support more challenging human space exploration missions

Promote international and commercial participation in exploration to further advance U.S. scientific, security and economic interests



Exploration Flight Test-1 (EFT-1) Uncrewed Orion launched from Cape Canaveral Air Force Base on a Delta IV Heavy

rocket Orbited the Earth twice, traveling approximately 3,600 miles above the Earth’s surface

• 15 times farther than the International Space Station. • Generated more than 80% of the return velocity experienced during a reentry from the moon, which allows engineers to model

expected reentries from future missions in deep space. Orion travelled twice through the Van Allen Radiation Belt, a dense radiation field of

highly energetic charged particles surrounding Earth• This exposure will help the team understand and mitigate radiation exposure, and to develop protective solutions before the first

crewed mission. At the conclusion of EFT-1, Orion reentered Earth’s atmosphere at 20,000 mph and

enduring temperatures of up to 4,000° Fahrenheit After the parachutes deployed, the spacecraft slowed to 20 mph splashing down in the

Pacific Ocean, near the Baja Peninsula.





How Big Is Big On Orion?• How much telemetry is there, anyway? Is it “big

data”?• Orion is transitioning to Big Data• LOTS! PB+. On board, three 1 Gb networks are kept

full• Volume, velocity, variety

• What is Spacecraft Telemetry?• Sensor measurements to monitor everything• Microsecond Timestamps• 12 discrete time sources• More complex time series• Human Space Flight requires fault tolerance

• What produces telemetry?• For Orion, the telemetry is produced in a variety of

simulation and test environments which includes at least 7 differ labs across the US.

• How many items are there to measure in Orion?• The XTCE representing the flight software telemetry

has over 3,000,000 elements, of which over 300,000 are related to currently taken measurements. There are many more when considering the ground system and other subsystems.

• How is telemetry analyzed today?• Traditionally, the responsible engineer

watches a handful of the telemetry indicators on the real time monitors as a test progresses, or in review. Certain behavior and reporting will result in further analysis on a few other measurands.

• Most values of most telemetry measurements are ignored if they are not out of limits

• Specific studies are done on some measurements in historical context, usually after detection of anomalous behavior, to determine if the behavior has been observed in the past.

• Automation has been applied to the problems of automatically collecting data, automatically running scripted tests, and automatically detecting out-of-limit values



Test Labs



How Big Is Big for Orion?Orion’s next mission (EM-1) in 2018 2 weeks instead of 4 hours 4 times as many computers Twice as many instruments Subsystems that support Human Flight!

EM-1 will be orders of magnitude bigger and more complex



Big Data Analytics Organization Technologists/Platform Developers

• Networking• Disk storage strategies• Modern languages and techniques: Scala, R• Big Data tools: Hadoop ecosystem, NoSQL databases, Stream and Batch

Processing Analytics /Data Scientist(Ponderers)

• Tools of the trade• Visualization & Communication• Data Science

StatisticsDesign of experimentsStatistical tools: SAS, SPSS, R, MATLABModelingData MiningLinear algebraData cleansing, sampling, warehousing

Subject Matter Experts• System and Subsystem Experts• Program Tools Experts



High Data Rate Problem DefinitionHorizontally scalable, fault tolerant prototype for processing High Data Rate telemetry from Orion Test Rigs including ingest, processing, transform, storage, advanced analytics

Satisfy Data Rate: ~300MB/s (97MB/s X 3 planes) Provide common data wire protocol that is:

Flexible, expressive and compact Programming language agnostic

Provide system to scale telemetry ingest from Test Rigs to Mach-5 within technical performance metrics

Provide solution that is continuously measurable and monitorable Provide Advanced Analytics to identify anomalies on ingest



MACH-5 INSIGHT℠ Architecture



MACH-5 INSIGHT℠ Benefits to Orion Enables Orion to teach us lessons about it’s own experience

leveraging data to tell the story Reduces time to develop and test Orion by identifying failures

early! Uses data points from one mission to help predict if mission

critical components will succeed or fail in specific future mission circumstances

Permits queries to be retrieved across historical batch and live streaming views within seconds

Streams historical data back as if it were live Overlays data across tests and missions Stores and processes much larger data sets than any other space

craft in historyScales to meet the needs of future missions to keep the conversation all the way to Mars

Data Reader/Simulator IngestPacket

Measurands (GPBs)

Kafka Message

Bus

Packet Measurands

(GPBs)Deduplica

tion (Spark)

HBase Writer(Spark)

mach5-sample ObjSplitter +

Decom (GDS)

C++ client Reads Packets and

Decommutates

Tlm Data

Packet Measurands GPB File

(represents a Packet(s) and contains

decommutated measurands)

Header Metadataapid:seqctr:time: value1

…..

apid:seqctr:time: valueN

mach5-sample (Spark)

Packet Measurands

(GPBs)

Lockheed Martin Proprietary Information

StorageAnalytics

HDFS

HFiles (HBase-RDD)

Mach-5 Data Ingest for Orion

HBase

Web/UITomcatGlassfish

Etc.

TraceFOSS

widgets

Aggregation

(Spark)

Alerting(Spark)

Limit Checking(Spark)



• Leverage space standards and wrapping modern data serialization technologies

• SFDU (Standard Formatted Data Unit)• CCSDS Standard

• Consultative Committee for Space Data Systems – Founded 1982

• Delimiting data and providing an internal index

• Transport and archival format• Provides ease of integration with

space systems• Should be usable 25+ years from

now• SFDU wraps serialized data, in our

case Protobuf

SFDU and ProtobufCCSD Z 001L

LMCO U 009L

V V

LMCO S 010

LMCO I 010

L

L

V

V

Header

Body



Wire Protocol: Serialization Protocol Buffers• Compact, compressible• Many language bindings: Java, C++, Python, etc…• Self describing schema, auto-generates code• Compared to other Serialization libraries (i.e. Avro), allows for

greater data type expression. Ex: Can natively support unsigned Long and Unsigned Integer

on C++ side Allows for cleaner generated code bindings

• Widely used within the Hadoop ecosystem• Widely used within Lockheed Martin telemetry processing



Kafka Benefits to Orion

Enables Orion to scale data architecture linearly on FOSS

Allows for multiple test rigs to simultaneously write data to be ingested at high data rates

Provides ability for analytics processing to independently pull and push data

Provides replay capabilities for guaranteed, exactly once processing semantics per partition

Provides fault tolerance and replication



Spark Benefits to Orion Enables Orion to scale data architecture linearly on FOSS Real-time Analytics reduce time to develop and test the Orion by

identifying anomalies early! Allows for analytics to be run in batch and streaming mode Analytics libraries that scale Integration with Kafka to independently pull and push data Runs on YARN to reduce complexity of managing resources Provides guaranteed, exactly once processing semantics per

partition using Kafka Direct Connect Provides fault tolerance and check pointing Integration with HBase (SparkOnHBase)



Using Spark to Bulk Load into HBasePull Data off of Kafka and Transform from SFDU to Protobuf HBase Column Format•Data bucketed to be aligned to appropriate Region. (HBase table is pre-split). Handled by Spark on Yarn

Sort Rows and Partitioning by HBase Regions•Sort happens locally•HFiles are guaranteed to contain all rows for a specific region•This avoids CPU intensive work by HBase

Construct HFiles Write to HDFS•HFiles are written to HDFS•Writing HFiles is much faster than HBase puts

Bulk Load into HBase•Bulk Load is called on each HFile



HBase Benefits to Orion

Data arrives in random order and HBase automatically sorts data by time

Enables Orion to scale data architecture linearly on FOSS

Provides random access to all telemetry with low latency access

Provides data store to perform batch processing using Mapreduce and Spark

Custom filters



Benefits of Single Column HBase schema Single-Column Schema

• Single Colum is packed with every piece of metadata for a row. Numerous serialization formats can be used: json, protobuf, avro, etc…

Benefits• Compact, greatly decreases row footprint• Higher write throughput. Under the hood HBase performs a Put for each column of a row. We

go from ‘N’ puts in a multi column schema to 1• Caching for queries

Detractors• Increased overhead to update the column• If your column schema has the potential to change, could end up with multiple versions of the

serialized column value• Slower reads because requires that all of the data be returned and deserialized• Requires a custom HBase filter to filter on individual members of the column value• Integration with 3rd Party BI tools and the rest of the Hadoop ecosystem



Processing and AnalyticsDetect abnormal activity during a test and notify operators early as possible. (Tests can run for weeks!) Provide limit checking on individual measurements Combine measurements to compute higher level

understanding (i.e. momentum calculation) Aggregation and statistical summaries Requirements validation Supervised and Unsupervised learning



TRACE View of MACH-5 INSIGHT℠ Stored Data



Next Steps Analytics! We need the help of the

community to develop effective data platforms and analytics that can scale with Orion missions.



Documents

Leverage data analytics to reduce human space mission risks Presentation