Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Corporate Technology
Approaches to Ultra Long-TermSystem Maintenance
Embedded Linux Conference Europe 2016
Prof. Dr. Wolfgang MauererSiemens AG, Corporate Research and TechnologiesSmart Embedded SystemsCorporate Competence Centre Embedded Linux
Copyright c© 2016, Siemens AG. All rights reserved.
Page 1 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Overview
1 Introduction
2 Aspects of Long-Term MaintenanceArchitectural CharacteristicsThreats and Risks
3 Technical AspectsPayload SoftwareDeveloping, Building and TestingIn-Field Strategy
4 Backporting & ProcessesBackporting: Conceptual and Technical Issues
Page 2 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Outline
1 Introduction
2 Aspects of Long-Term MaintenanceArchitectural CharacteristicsThreats and Risks
3 Technical AspectsPayload SoftwareDeveloping, Building and TestingIn-Field Strategy
4 Backporting & ProcessesBackporting: Conceptual and Technical Issues
Page 3 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Introduction 0
Disclaimer
Many statements: Extremely obviousRealisation: Quite remote for many problematic appliancesQuantification: Astonishingly hard. . .
Page 4 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Introduction I
Consumer Electronics
Mobile Phones, Notebooks,Tablets, . . .Entertainment systems(Radio, TV, DVD/BlueRay, . . . )Ovens, Washing Machines,Home Control/Automation
Industrial Systems
Medical devicesComputed tomography,X-Ray Imaging,Ultrasound, . . .
InfrastructureGas, Power, Water supplyPowerstations andtransformersTraffic lights, park spacemanagement
MobilityPlanes, trains,automobiles, mars rovers,space stations
. . .Page 5 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Page 6 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Introduction II
Fundamental questions
Is long-term maintenance reasonable/doable?System architecture for LTM?
Page 7 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Innovation Cycles I
Lifespans
Consumer devices: 2-5 yearsMobility: 5-20 yearsIndustrial: 10-30 yearsInfrastructure: 30-80 years (and up!)
All domains: Linux, of course!
Long-life requirements not restricted to industrial appliances!IoT, smart home, connected devices: Longevity requirementspervade everyday devicesShort lifespans: Exception, not rule!
Page 8 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Innovation Cycles II
Page 9 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Innovation Cycle III
Fundamental Questions
Risks and benefits ofupdates?How to restrict updates to(isolated) areas?How to avoid updates?
Beyond components
Questions not addressed bysimply using LTScomponents/distrosLTM: Architectural issueLTM: Mindset issue
Some field observations/bogus assumptions
All components can be upgraded in-field 7
Updates fix more problems than they create 7
Upstream integration always reduces maintenance effort 7
Long-term component versions solve maintenance problems 7
Page 10 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Innovation Cycle III
Fundamental Questions
Risks and benefits ofupdates?How to restrict updates to(isolated) areas?How to avoid updates?
Beyond components
Questions not addressed bysimply using LTScomponents/distrosLTM: Architectural issueLTM: Mindset issue
Some field observations/bogus assumptions
All components can be upgraded in-field 7
Updates fix more problems than they create 7
Upstream integration always reduces maintenance effort 7
Long-term component versions solve maintenance problems 7
Page 10 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Outline
1 Introduction
2 Aspects of Long-Term MaintenanceArchitectural CharacteristicsThreats and Risks
3 Technical AspectsPayload SoftwareDeveloping, Building and TestingIn-Field Strategy
4 Backporting & ProcessesBackporting: Conceptual and Technical Issues
Page 11 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Appliance Architecture
Long-term maintenance vs. periodic re-building
Fixed (trusted) ↔ arbitrary payload softwareIsolated ↔ universally accessibleHardware stability ↔ varianceFixed hardware ↔ extensibility (e.g., USB)Verification and safety requirementsCost sensitivity (core payload inside virtual environments?)
Software aspects
System base softwarePayload software + architecture
Page 12 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Appliance Architecture
Long-term maintenance vs. periodic re-building
Fixed (trusted) ↔ arbitrary payload softwareIsolated ↔ universally accessibleHardware stability ↔ varianceFixed hardware ↔ extensibility (e.g., USB)Verification and safety requirementsCost sensitivity (core payload inside virtual environments?)
Software aspects
System base softwarePayload software + architecture
Page 12 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Appliance Architecture
Long-term maintenance vs. periodic re-building
Fixed (trusted) ↔ arbitrary payload softwareIsolated ↔ universally accessibleHardware stability ↔ varianceFixed hardware ↔ extensibility (e.g., USB)Verification and safety requirementsCost sensitivity (core payload inside virtual environments?)
Software aspects
System base software: Little/no controlPayload software + architecture: Full control ⇒ LTM Focus!
Page 12 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Threats and Risks
What should LTM prevent in your case?
Device stops workingDevice faults cannot be repaired/debuggedDevice can be influenced from outsideDevice does not meet changed expectations (functionality,interoperability, . . . )
Response catalogue
Ignore issues (can be reasonable, on rare occasions)Replace device (HW + SW; component)Modify SW
Page 13 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Threats and Risks
What should LTM prevent in your case?
Device stops workingDevice faults cannot be repaired/debuggedDevice can be influenced from outsideDevice does not meet changed expectations (functionality,interoperability, . . . )
Response catalogue
Ignore issues (can be reasonable, on rare occasions)Replace device (HW + SW; component)Modify SW ⇐ case of interest
Page 13 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Outline
1 Introduction
2 Aspects of Long-Term MaintenanceArchitectural CharacteristicsThreats and Risks
3 Technical AspectsPayload SoftwareDeveloping, Building and TestingIn-Field Strategy
4 Backporting & ProcessesBackporting: Conceptual and Technical Issues
Page 14 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Payload Software
Software Engineering Considerations
Deliver maintainable software/architecture in the first placeMinimise cross-cutting issuesHarmonise technical and social organisationMeaningful and reproducible history
Think three times before connecting systems to networks; thenthink three more times“Translator” with domain and (base component) communityknowledgeMake components (run-time) replaceable; prefer userland tokernel
Page 15 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Payload Software
Software Engineering Considerations
Deliver maintainable software/architecture in the first placeMinimise cross-cutting issuesHarmonise technical and social organisationMeaningful and reproducible history
Think three times before connecting systems to networks; thenthink three more times“Translator” with domain and (base component) communityknowledgeMake components (run-time) replaceable; prefer userland tokernel
Page 15 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building I
Reproducible Builds
Produce binaries. . . 20 years after initial launchPayload application + modifiable system componentsPreserve base component binaries
Documentation of (seemingly trivial) details essentialDocumentation availability (hardcopy is a serious alternative)Avoid custom build systems
Page 16 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building I
Reproducible Builds
Produce binaries. . . 20 years after initial launchPayload application + modifiable system componentsPreserve base component binaries
Documentation of (seemingly trivial) details essentialDocumentation availability (hardcopy is a serious alternative)Avoid custom build systems
Source Code
Availability of source code + history (e.g., Bitkeeper. . . )Component states + local provision of dependenciesIncludes build infrastructure!
Page 16 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building I
Reproducible Builds
Produce binaries. . . 20 years after initial launchPayload application + modifiable system componentsPreserve base component binaries
Documentation of (seemingly trivial) details essentialDocumentation availability (hardcopy is a serious alternative)Avoid custom build systems
Tool Chain
Cross-Building: (subtle) dependencies!Isolate build environment in VM (strict freeze!)Bugs in ancient toolchain: Payload SW workaroundsEclipse etc.: harder. . . 7
Page 16 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building II
Component Selection and Integration
Consider cost of librariesDynamically changing dependencies (version requirement specsoften unreliable)Changes in components⇒ (silent) breakage in library
Distinguish between prototype and deliverableExperiment with 17 machine learning algorithmsDeploy one (+ rewrite)
Page 17 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building II
Development prior to market release
Develop against latest mainline state (rebasing preferred)Avoid vendor BSPs. Board support essential, not BSPs!
Only chance: Prior to purchasing 1.8× 1023 units
System changes: Upstream first policyAvoid component modifications (socio-technical congruence)
Especially for features useless for upstream
Minimise divergence between upstream state and product atrelease time
Page 18 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Development & Building III
Five Recommendations
1 Avoid complex development environments and generated code2 Avoid web technologies3 Use convenience libraries judiciously4 Avoid integration/consolidation; delegate
communication/networking to separate entities5 Document and automate excessively
Page 19 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Page 20 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Options for post-release development
Rolling Development
Continuous updates of(selected) base componentsUncouple progress fromdistribution (e.g., after eol)Detect issues early,re-invent distribution wheel
Distribution schedule
sudo apt-get upgrade
Requires support bydistribution!
Page 21 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Options for post-release development
System schedule
Update in appliance-specificintervals (periodic orirregular)Combine disadvantages ofdistribution and continuousupdates
Invariant base system
Don’t update base systemPayload applicationdevelopment onlyRequires (extremely) smallattack surface/virtualisedbase system
Page 21 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Outline
1 Introduction
2 Aspects of Long-Term MaintenanceArchitectural CharacteristicsThreats and Risks
3 Technical AspectsPayload SoftwareDeveloping, Building and TestingIn-Field Strategy
4 Backporting & ProcessesBackporting: Conceptual and Technical Issues
Page 22 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Backporting I
Leverage LTSI kernel
LTSI support period: Comprehensive coveragePost-LTSI: Backport only
Critical issues re/ attack surfacesOrthogonal drivers/componentsFeature not required during first 5 years⇒ unlikely required in nextdecade(s)Major change required (debugging, tracing etc.): Time for newrelease. . .
Backport patch stack
Organise backports in proper orthogonal patch stackRebase! Living organism, not code dump
Page 23 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Backporting II: What and when to backport
1.) What to backport
Most upstream changes donot require back-portingSelection crucialSelection criteria differdepending on use case
Approaches
Keyword filtering (possiblefor well-tended projects)Content/file/path basedfiltering (tremendous volumereduction)Manual review + long-termexpert involvementnecessary
Page 24 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Backporting II: What and when to backport
2.) When to backport
ProactivelyAfter incidents/bugs
Simple criterion
# incidents > # backportregressionsHistorical data: Noconclusive evidenceExpert assessment required
Page 24 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Backporting III
The human touch
Determine when no action is requiredNotify users/customers
Page 25 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Backporting IV
Goal: Simplify patch selection for everyone
Fully automatic approach: unrealisticAvoid duplicated manual efforts
Wishlist: Improvements
Maintenance classes (for patches), consistent across projectsCurrent schemes dating back to 70ies⇒ survey!
Applicability range (releases) annotationWide-spread use of automated approaches (backwards integration,testing)
Extend use of semantic vs. text-based modifications
Page 26 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Summary
LTM best practices: Similar to proper (OSS-style) softwaredevelopmentSystem and application architecture: crucialLTM: not rocket science, but still more art than science –quantitative data required!
Page 27 11. Oct. 2016 W. Mauerer Siemens Corporate Technology
Thanks for your interest!
Page 28 11. Oct. 2016 W. Mauerer Siemens Corporate Technology