View
9
Download
0
Category
Preview:
Citation preview
1 | 58Embedded and Networked Systems
LecturerMitra Nasri (M.Nasri@tudelft.nl)Appointment by email
TAsGrzegorz (Greg) Krukiewicz-Gacek
(G.M.Krukiewicz-Gacek@student.tudelft.nl)Fanis Floros
(T.Floros@student.tudelft.nl)
Appointment by email
IN4390 Quantitative Evaluation of Embedded SystemsLecture 1
Introduction
2 | 58Embedded and Networked Systems
Agenda
• Course outline
• Definitions and concepts • Embedded systems specifications • Common requirements
• Modeling and model-based design
3 | 58Embedded and Networked Systems
Course outline
• Lectures• Theory, instructions, examples, quizzes
• Exam • Written exam with open-ended questions
• Practicum (lab)• 3 main assignments• Tools: ROS and Petri-net• Requirement: basic knowledge of C++ and Linux
4 | 58Embedded and Networked Systems
Course organization
• Lectures• Tuesday 8:45 – 10:30• Thursday 10:45 – 12:30• No lecture on Dec. 19th
• Practicum (Lab)• Thursday 13:45 – 17:45 (starts from Nov. 21)• Teams of 2
Note 1: Your presence in the lab is not mandatory, however, some assignments require you to demo your implementation to the TAs.
Note 2: Both team members must equally contribute to the assignments and reports. Otherwise it would be a case of plagiarism.
5 | 58Embedded and Networked Systems
http://www.st.ewi.tudelft.nl/koen/in4390/
Where to find information
• Brightspace• Upload assignments• Register your group
• Main webpage of the course• Announcements• Lecture materials• Extra materials• Quiz solutions• Example exams
6 | 58Embedded and Networked Systems
Final grade = min(10, E + C𝟏𝟏𝟏𝟏𝟏𝟏
) * M
Grading
E = Exam
M = {pass=1 | fail=0} (mandatory assignments)
C = Customizable points
• You fail if you have not passed your mandatory assignments, or your final grade is smaller than 6. • Mandatory assignments must be uploaded to Brightspace before their due dates. Some of these assignments
require a live demo.
7 | 58Embedded and Networked Systems
Mandatory assignments
• Assignment 0 • Paper reading followed by questions on Brightspace • Related to Lect. 1 and 2
• Assignment 1 (Lab 1)• Measurement-based performance evaluation of ROS 2.0 communication
mechanisms• Related to Lect. 2, 3 and 4
• Assignment 2 (Lab 2)• Behavior modeling and performance analysis using Petri-nets• Related to Lect. 5 and 6
• Assignment 3 (Lab 3)• Deriving a petri-net model from a given ROS application • Answering questions regarding what to expect from the system if certain
parameters/configurations change• Related to Lect. 1, 2, 5, 6, and 13
Autoware or AirSim(autonomous car or drone)
8 | 58Embedded and Networked Systems
Use cases for Lab 3 (tentative)
Autoware.org
Autoware(an open source framework
for autonomous driving)
AirSim(an open framework for
autonomous cars or drones)
• Both provide simulation framework for the system and the physical world
• Both are based on ROS
9 | 58Embedded and Networked Systems
Grading
Extra lab assignment
Class activities
Tool presentation
…
ROS Implementation
Quiz
Projects
The background image comes from here
You can build your own grading scheme according to your own preferences
We are going to use Gamification!
Trion Worlds
10 | 58Embedded and Networked Systems
Grading: customizable points• Sources of customizable points:
1. Extra (optional) questions in the assignments • 120 points
2. Quiz• 15 points per quiz for those who answer all questions correctly (4~5 quizzes)
3. Questions/answers during the class • 5 points per question (max one answer per student per lecture)
4. Tool demo• 80 points (maximum one per team, FCFS)
5. Project • 100~200 points (maximum one per team, FCFS)
• You may acquire at most 400 customizable points
• Note that teams may do either a demo or a project (but not both)
300 ≤ E ≤ 8000 ≤ C ≤ 400 Final grade = min(10, E + C
𝟏𝟏𝟏𝟏𝟏𝟏) * M
𝑀𝑀 ∈ fail = 0, pass = 1
11 | 58Embedded and Networked Systems
Grading: customizable points
4. Tool demos• Investigate a certain tool from a given list • Teams will coordinate the tool that they want to present with the TAs.• A short report (of up to 5 pages) • Present the tool in the class • Deadline: at the end of the quarter (will be announced soon)
5. Projects • Projects involve implementation and evaluation using existing tools and frameworks (such as Microsoft
AirSim, Autoware, etc.)• A list of topics will be available soon• Teams will submit a 1-page project proposal
• Explain what the project is about, what are the goals, milestones, steps, and deliverables
• Proposals will be evaluated and adjusted• Deadline: at the end of the quarter (will be announced soon)
12 | 58Embedded and Networked Systems
Relations to the study program
System Validation
Functional correctness
QEES
Evaluation of non-functional properties such as performance, dependability, …
Real-Time Systems
Evaluation (and correctness) of timing properties such as response-time
Quarter 1
Quarter 2
Quarter 3
Embedded Systems Lab
Quarter 4
Designing and evaluating an embedded system
13 | 58Embedded and Networked Systems
Goals of QEES
• Use models to design, analyze, and evaluate a system
• Compare alternatives• To provide quantitative information about alternatives
• Determine the impact of a feature on overall system performance
• Pin-point bottlenecks
• System tuning/optimization• To find the best parameters
that produce the best performance
Example: The number of packets lost on two links was measured for our file sizes as shown below:
Which link is better?
14 | 58Embedded and Networked Systems
Lectures’ organization
• Introduction to modeling and model-based design (1 lecture)
• Measurement-based performance evaluation and statistics 101 (1 lecture)
• Design of experiments (2 lectures)
• Petri-nets and data-flow networks (2 lectures)
• Markov models (2 lectures)
• Queueing theory (2 lectures)
• Memory efficiency and optimization (1 lecture, tentative)
• Design-space exploration and optimization (1 lecture)
• Presentations by students, exam examples, FAQ (1 lecture)
15 | 58Embedded and Networked Systems
Organization of the contents
Prof. Koen LangendoenK.G.Langendoen@tudelft.nl• Lect. 9 and 10: operational laws and queuing theory• Lect. 13: design space exploration and optimization• Lect. 14: student presentations, exam preparation, FAQ
Marco ZunigaM.A.ZunigaZamalloa@tudelft.nl• Lect. 7 and 8: Markov models
Mitra NasriM.Nasri@tudelft.nl• Lect. 1: Modeling and model-based design• Lect. 2: Measurement-based performance evaluation
Lect. 5: Petri-nets• Lect. 6: Data-flow networks• Lect. 12 (tentative): Memory efficiency• Lect. 14: student presentations, exam preparation, FAQ
Lydia ChenY.Chen-10@tudelft.nl• Lect. 3 and 4: Experiment design
Lecturers
There is no Lect. 11
16 | 58Embedded and Networked Systems
Sources• Books
• The Art of Computer Systems Performance Analysis• Raj Jain • John Wiley & Sons Canada, Ltd., 1 edition, 1991.
• Embedded System Design • Peter Marwedel• The second edition (feel free to get the third edition)
• Introduction to Embedded Systems: A Cyber-Physical Systems Approach• Edward Lee and Sanjit Seisha (UC Berkeley)• Second Edition, 2017• https://ptolemy.berkeley.edu/books/leeseshia/
• Measuring Computer Performance: A Practitioner’s Guide• David J. Lilja - Hardcover.• Cambridge University Press, 2000.
• Papers• Basic Concepts and Taxonomy of Dependable and Secure Computing
This page will be updated soon.Do not yet buy your book!
17 | 58Embedded and Networked Systems
Agenda
• Course outline
• Definitions and concepts • Embedded systems specifications • Common requirements
• Modeling and model-based design• Modeling• Design process• Model-based design
Sources: • [Book]: Marwedel (chapter 1)• [Paper]: Basic Concepts and Taxonomy of Dependable and Secure Computing
18 | 58Embedded and Networked Systems
Embedded Systems (and Quantitative Properties)
19 | 58Embedded and Networked Systems
What is an embedded system?
It is an information processing system that is embedded into an enclosing
physical product.
Unlike a PC or servers, an embedded system “interacts” with its physical world.
Have you heard about Cyber-physical systems (CPS)?
20 | 58Embedded and Networked Systems
Cyber-physical systems (CPS): A simple rename or a purposeful concept?
[Wiki]: A CPS converts goals defined digitally (the cyber domain) to the
physical world.
[wiki] CPS v.s. ES: A CPS is typically designed as a network of interacting
elements with physical input and output instead of as standalone devices.
Are you convinced?
Mechanical engineer
Software developer
Hardware engineer
Controlengineer
System integratorEvery team tends to assume a perfect output
from the other teams, e.g., zero execution time, perfect controller, perfect hardware, etc.
21 | 58Embedded and Networked Systems
Concepts and definitions
SystemA system is an entity that interacts with other entities, i.e., other systems, including hardware, software, humans, etc.
FunctionThe function of a system is what the system is intended to doand is described by the functional specification in terms of functionality and performance
The boomerang drone’s functions:- Normal mode (receives signals from user and battery
is not low)- Move up, down, left, right- Increase or decrease speed - Land- Take picture- Send picture
- Hold mode (no signal)- Stay still and wait for signal
- Safe-return mode (no signal from user for 2 minutes)- Follow the path back if there is no signal- Detect obstacles on the way- Avoid obstacles
- Low battery mode (battery is low)- Land safely if the battery is low
BehaviorThe behavior of a system is what the system does to implement its function. The behavior, for example, can be described by a sequence of states
StructureThe structure of a system is what enables it to generate the behavior.
Signal
No signal for 2 minutesS
B
NSignalNo signal
Low battery
Low batteryH
Low battery
22 | 58Embedded and Networked Systems
BehaviorThis is a state diagram, which is a “model” that
describes how the system changes modes
Is this enough to describe the behavior?
Decision making
This is a component diagram, which is a “model” to show the dependencies and interactions between SW/HW components
Signal
No signal for 2 minutesS
B
NSignal No signal
Low battery
Low batteryH
Low battery
Flight control
Note: this diagram is symbolic and is not accurately model our prior example.
23 | 58Embedded and Networked Systems
V-Model for system development
customer
Product development
requirementsVerification and validation
End product
24 | 58Embedded and Networked Systems
Specifications
The boomerang drone’s functions:- Normal mode (receives signals from user and battery
is not low)- Move up, down, left, right- Increase or decrease speed - Land- Take picture- Send picture
- Hold mode (no signal)- Stay still and wait for signal
- Safe-return mode (no signal from user for 2 minutes)- Follow the path back if there is no signal- Detect obstacles on the way- Avoid obstacles
- Low battery mode (battery is low)- Land safely if the battery is low
This is a (very informal) functional specification
SpecificationsThey describe the functional and non-functional requirements of the system
25 | 58Embedded and Networked Systems
Specifications
SpecificationsThey describe the functional and non-functional requirements of the system
What examples do you have in mind for non-functional requirements?
[Wiki] A non-functional requirement (NFR) is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors.
Read more here: https://en.wikipedia.org/wiki/Non-functional_requirement
26 | 58Embedded and Networked Systems
Quantitative properties
Quantitative properties are key selling points (next to the functional correctness)
Slide course (Bart Theelen): https://www.win.tue.nl/~pcuijper/QEES/Guest%20Lecture%20Bart%20Theelen.pdf
27 | 58Embedded and Networked Systems
Quantitative properties in industry
Quantitative properties are key selling points (next to the functional correctness)
Slide course (Bart Theelen): https://www.win.tue.nl/~pcuijper/QEES/Guest%20Lecture%20Bart%20Theelen.pdf
28 | 58Embedded and Networked Systems
Examples of quantitative measures
• Extrema (worst/best case)• Maximum occupancy of a memory• Minimum time until next failure• Peak power consumption• Worst-case response time• Worst-case end-to-end delay
• Reachability (expected time until something happens)• Expected time until next failure• Expected time until first output is produced
• Long-run average• Processor load• Throughput of a communication network• Mean time between failures (MTBF)• Average power consumption
Slide course (Bart Theelen): https://www.win.tue.nl/~pcuijper/QEES/Guest%20Lecture%20Bart%20Theelen.pdf
29 | 58Embedded and Networked Systems
Quantitative properties
Response time
Performance
Safety
Security
ReliabilityPrivacy
Availability
PortabilityDurability
Extensibility
Understanding these requirements is what makes you a valuable,
highly-paid, and highly-wantedengineer!
Dependability
Read more here: https://en.wikipedia.org/wiki/Non-functional_requirement
Life time
Energy consumption
Quantitative properties are key selling points (next to the functional correctness)
30 | 58Embedded and Networked Systems
Quantitative properties
Response time
Safety
Security
ReliabilityPrivacy
Availability
PortabilityDurability
Extensibility
Understanding these requirements is what makes you a valuable,
highly-paid, and highly-wantedengineer!
Read more here: https://en.wikipedia.org/wiki/Non-functional_requirement
Life time
Energy consumption
Quantitative properties are key selling points (next to the functional correctness)
Performance
Dependability
31 | 58Embedded and Networked Systems
Common performance measures• Response time
• Response time is the total amount of time it takes to respond to a request for service• Latency
• Latency is a time delay between the cause and the effect of some physical change in the system being observed
• Bandwidth• In computer networking, bandwidth is a measurement of bit-rate of available or consumed data
communication resources, expressed in bits per second or multiples of it (bit/s, kbit/s, Mbit/s, etc.)• Throughput
• the rate of production or the rate at which something can be processed• Scalability
• Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth
• Power consumption• The amount of electricity used by the computer
• Compression ratio• Size and weight• Transistor count
Source: https://en.wikipedia.org/wiki/Computer_performance
32 | 58Embedded and Networked Systems
Dependability
“The dependability and security specification of a system must include the requirements for theattributes in terms of the acceptable frequency and severity of service failures for specified classes offaults and a given use environment. One or more attributes may not be required at all for a givensystem” [TPDS00].
The ability to deliver service that can justifiably be trusted• The stress is on the need for justification (e.g., through rigorous
evaluation or proof) of trust
Definition 1
The ability to avoid service failures that are more frequent and more severe than is acceptable
Definition 2
[TPDS00] “Basic Concepts and Taxonomy of Dependable and Secure Computing”, 2004.
33 | 58Embedded and Networked Systems
Service/function failure
• Fault -- often referred to as Bug [TPDS00] • A static defect in software (incorrect lines of code) or hardware • A fault is the adjudged or hypothesized cause of an error
• Error• An incorrect internal state (unobserved)• An error is the part of total state of the system that may lead to its subsequent
service failure
• Failure• External, incorrect behavior with respect to the expected behavior (observed)
[TPDS00] “Basic Concepts and Taxonomy of Dependable and Secure Computing”, 2004.
Fault Error Failure
34 | 58Embedded and Networked Systems
Fault, error, and failure
First we need to know the desired behavior
What is this? A fault? An error? Or a failure?
image: Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
35 | 58Embedded and Networked Systems
Fault, error, and failure
First we need to know the desired behavior
What is this? A fault? An error? Or a failure?
image: Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
“A design without specifications cannot be right or wrong, it can only be surprising!”
[Lee and Seshia]
Under this usage, this implementation might not be
a fault :-)
36 | 58Embedded and Networked Systems
Fault, error, and failure
Assume it is a rail for a normal train
It is a fault
What is this? A fault? An error? Or a failure?
Fault examples: • Misconfiguration (e.g., misconfigured user permissions)• Hardware faults (a memory cell that is always zero)• Physical faults (caused by the environment)
• At high radiation, memory bits may randomly flip• At high temperature, pressure sensor produces noisy data• In the tunnels, the GPS does not work
image: Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
37 | 58Embedded and Networked Systems
……
…
……
Fault, error, and failureA fault
(like a bug in the code)
It is not activated as long as it is not on the program path
38 | 58Embedded and Networked Systems
……
…
……
Fault, error, and failure
An error(a fault that is on the
program path)
39 | 58Embedded and Networked Systems
……
…
……
Fault, error, and failure
A failure(an error that caused an observable deviation of
correct behavior)
40 | 58Embedded and Networked Systems
Taxonomy of faults
• Development Faults• All fault classes occurs during the development• This includes bugs, misconfigurations, etc.
• Physical Faults• All fault classes that affect hardware or physical system• This includes hardware defects, etc.
• Interaction Faults• All external faults • This includes interface mismatch between components
Read the rest from: “Basic Concepts and Taxonomy of Dependable and Secure Computing”.
41 | 58Embedded and Networked Systems
Dependability and security tree
“Basic Concepts and Taxonomy of Dependable and Secure Computing”
42 | 58Embedded and Networked Systems
Addressing faults at different stages
Patching/fixing Redundancy Testing
Bernd Bruegge & Allen H. Dutoit. Object-Oriented Software Engineering: Using UML, Patters, and Java
Fault prevention
Better design, better tools, ….
Fault detection
Testing, debugging, …
Fault removal
Fixing, patching, …
Fault tolerance
Redundancy, isolation, …
Fault forecasting
Estimating how many faults might
still be there
43 | 58Embedded and Networked Systems
Maintainability• Ability to undergo modifications and repairs• It is the probability that a failing system can be
repaired within a certain interval of time• 𝑴𝑴(𝒅𝒅) = probability of system working correctly 𝑑𝑑 time units after an error occurred.
Dependability attributes
“Basic Concepts and Taxonomy of Dependable and Secure Computing”
Availability• Readiness for correct service• It is the probability that the system is available at
time 𝑡𝑡• High availability requires high maintainability and
reliability• 𝑨𝑨(𝒕𝒕): probability of system working at time 𝑡𝑡
Reliability• Continuity of correct service• It is the probability that the system will not fail if it
was working from the beginning• 𝑹𝑹(𝒕𝒕) = probability of system working correctly at
time 𝑡𝑡 provided that it was working at 𝑡𝑡 = 0
Safety• Absence of catastrophic consequences on the
user(s) and the environment
Integrity• Absence of improper system alterations
44 | 58Embedded and Networked Systems
Assignment 0
• Read the following paper (all chapters)• “Basic Concepts and Taxonomy of Dependable and Secure Computing”• https://ieeexplore.ieee.org/document/1335465
• Answer the questions that will be uploaded on Brightspace• Due date is 8:45am Tuesday Nov. 26
• Notes• The exam will include chapters 1 to 4 of the paper. • Questions that will be asked on Brightspace are similar to what will appear in the exam. • There will be a quiz (with optional points) from the first four chapters of the paper on
Nov. 19th
45 | 58Embedded and Networked Systems
Agenda• Course outline
• Definitions and concepts • Embedded systems specifications • Common requirements
• Modeling and model-based design• Modeling• Design process• Model-based design• Models of computation (MoC)
Sources: • [Book]: Lee and Seshia (chapter 1)• [Book]: Marwedel (chapter 1 and 2)
46 | 58Embedded and Networked Systems
Modeling and Model-Based Design
47 | 58Embedded and Networked Systems
Modeling, design, analysis
Modeling is the process of gaining a deeper understanding of a system through imitation.
Models specify what a system does.
Design is the structured creation of artifacts.
It specifies how a system does what it does. This includes optimization.
Analysis is the process of gaining a deeper understanding of a system through dissection.
It specifies why a system does what it does (or fails to do what a model says it should do).
48 | 58Embedded and Networked Systems
Modeling, design, analysisReal system
http://ctms.engin.umich.edu/CTMS/index.php?example=InvertedPendulum§ion=SimulinkModeling https://www.nomagic.com/mbse/images/whitepapers/SysML_Inverted_Pendulum_System.pdf
Models can be used to describe specification
This mathematical model expresses the physics of the plant
Models can be used to design the system Models can be used to analyze the system
49 | 58Embedded and Networked Systems
Why is modeling important?• Systems getting more complex
• Abstraction is needed to understand the system or to design it
• Partitioning into subsystems is needed
Design space
Specifications
Intermediate models
implementation
Abst
ract
ion
gapDesign step
Abst
ract
ion
leve
l
50 | 58Embedded and Networked Systems
Why is modeling important?• Models are a good base for communication between
engineers• Engineers think in diagrams
• Models are important for documentation• Standardized semantics• Formal language with (hopefully) only one meaning
• Models abstract from the very detailed implementation• Allow focusing on most important aspects
• Models can be used when the system architecture is not yet ready at early design stages in order to:• Evaluate functional and non-functional requirements• Find design faults
51 | 58Embedded and Networked Systems
Model-driven v.s. model-based design
An interesting readIBM Harmony: https://www.ibm.com/support/knowledgecenter/SSB2MU_8.3.0/com.btc.tcatg.user.doc/topics/atgreqcov_SecSysControllerHarmony.htmlOther source: http://www.win.tue.nl/~pcuijper/docs/QEES/DF/QEES%20introcollege%202013-2014.pptx
Model-driven design: use models to automate the arrows
Model-based design: use models to get a grip on the boxes
Example: The IBM Harmony process (for IoT) is a model-driven design.
52 | 58Embedded and Networked Systems
When do we create a model?
To design a given specification To understand, evaluate, analyze, or optimize a given system
Use modeling to design the
system
End user
Development team Use modeling to
understand, analyze, or optimize the system
Quantitative evaluationand optimization
End user
Development team
53 | 58Embedded and Networked Systems
Models of computation (MoC)
MoC categories:
State-based MoCs• Finite state machines• Hierarchical state machines• Process state machines• Pushdown automata• Turing machine
Source: http://users.ece.utexas.edu/~gerstl/ee382v_s14/notes/lecture5.pdf
A model of computation (MoC) is a model that describes how an output of a function is computed given an input
Process-based MoCs• Process networks
• Kahn process network• Data flow networks
• Petri nets• Synchronous data flow
• Process calculi
Signal
No signal for 2 minutesS
B
NSignalNo signal
Low battery
Low batteryH
Low battery
54 | 58Embedded and Networked Systems
Recap: an overview of the courseIntroduction
Measurement-based performance evaluation
Statistical modeling
Modeling concurrent programs
(design and performance analysis)
Modeling state-based programs and queues
(performance analysis)
Modeling and model-based design (1 lecture)
Measurement-based evaluation and statistics 101 (1 lecture)
Design of experiments (2 lectures)
Petri-nets and data-flow networks (2 lectures)
Markov models (2 lectures)Operational laws and queueing theory (2 lectures)
Design-space exploration and optimization (1 lecture)
Memory efficiency and optimization (1 lecture)
Team presentations, exam examples, FAQ (1 lecture)
Model-based design optimization
(tentative)
55 | 58Embedded and Networked Systems
Questions?Don’t forget Assignment 0
Recommended