Copyright © 2019 University of Colorado

Dr. Sam SiewertElectrical, Computer and Energy EngineeringEmbedded Systems Engineering Program

Real-Time challenges and concepts:Predictable response concepts and strategies

2

Segment Outline

– Why is software often the best implementation for HRT services?

– A brief history of CPU overload for real-time systems and the RM LUB

– CPU Scheduling OverviewLinux CFS [Completely Fair Scheduler]Linux CFQ [Completely Fair Queue for I/O]We really want an “unfair” scheduler for Real-Time

– RTOS Schedulers and POSIX RT – Linux, FreeRTOS, VxWorks

– RT Services – How to Define?

– Linux POSIX RT Thread Analysis and DebugSCHED_FIFOSynchronization (next)Sequencing (next)

3

Why is software used for HRT (Hard Real-Time) systems at all?

Reasons not to use software for HRT services– ASIC and FPGA State-Machine Solutions Offer Hardware

Clocked Deterministic Solutions– FPGAs Can be Updated with New Bit-streams (Synthesized

HDL to Reconfigurable Logic Elements)

Reasons to use software for HRT services– Software (Firmware) Remains Simplest for Field Upgrade

(Reconfigurable at Run-time)– FPGAs Can be Costly Per Unit– Cost of Software Engineering vs. Hardware Engineering

4

A brief history of software HRT services

Apollo 11, Launched July 16, 1969– On descent, Lunar Module experienced a

computer overload– Alarm 1201, 1202– CPU overload s.t. Sum(Ci/Ti) > 1.0– Landed July 10, 1969 [By ignoring alarms -

lucky?]– Rate Monotonic Theory - Not yet formalized

(math model)

Wisdom - don’t overload a CPU– https://wehackthemoon.com/people/margare

t-hamilton-computer-got-loaded– But, how much is too much?

Theory for how much loading is ok was needed

The LUB was an answer!Margaret_Hamilton_(scientist)

“Scheduling Algorithmsfor MultiprogrammingIn a Hard-Real-TimeEnvironment”, C.L. Liu[MIT], James W. Layland[NASA JPL], Jan 1, 1973

5

Best Effort Service Utility

6

Hard Real-Time Service Utility

7

Soft Real-Time Service Utility

8

Isochronal Hard Real-Time Utility

9

Isochronal Soft Real-Time Utility

10

Anytime Utility Curve

11

Key Terms for this segment

– Kernel Task– Process– Context Switch– Preemption– Dispatch

– AMP – Asymmetric Multi-Processing – SMP – Symmetric Multi-Processing – NUMA – Non-Uniform Memory Access – SMT – Simultaneous Multi-threading– Flynn’s Taxonomy

{Single Instruction, Multi-Instruction} x {Single Data, Multi-Data}

12

Fair vs. Unfair Scheduling

CFS (Completely Fair Scheduler) in Linux – Fair!– Linux 2.6.23 Introduction of CFS– Fundamental Concepts of Fair Schedulers – “Everyone Get’s a Slice of

Pie (Time), Some Just Get Slices More Often”– Requires PIT (Programmable Interval Timer) to Raise Time Based

Interrupts – The Quantum– Scheduling Changes Can Be Made at Each Quantum “Tick”– All Tasks (Threads) Should Make Progress – Fair– Does Not Provide Predictable Response

Priority Preemptive Run-to-Completion RTOS – Unfair!– Tasks (Threads) Run at Priority Until:

Complete and ExitYield, Sleep, Activate Higher Priority Task or Block on Secondary Resource (System Call)Interrupt Raised

13

General Computing has gone MIMD with SIMD ISA and SIMD Offload (GP-GPU)NUMA vs. UMA (Trend away from UMA to NUMA or Memory Hub vs. I/O Hub)SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling, Process Load Balancing, Multi-User, Multi-Application, CPU Affinity Possible)MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data

Embedded has traditionally been SISD or MIMD AMPNeed for SMP in Embedded?SIMD Offload and Acceleration Quite Useful (DSP, Image Processing, Error Correction Codes, Encryption, etc.)

Single Instruction Multiple InstructionSingle Data SISD (Traditional Uni-

processor)MISD (Voting schemes and active-active controllers)

Multiple Data SIMD (e.g. SSE 4.2, GP-GPU, Vector Processing)

MIMD (Distributed systems (MPMD), Clusters with MPI/PVM (SPMD), AMP/SMP)

Flynn’s Computer Architecture Taxonomy

14

CPU Scheduling Taxonomy

15

RTOS Kernel Tasking / Linux NPTLAn RTOS task or Linux NPTL thread is an Implementation of a Service

– E.g. Anti-Lock Braking Digital Control– Created and Put Into Service, Awaits Event (Data Ready) For a Release– Memory Resident (TCB / Process descriptor (container), stack, code, data,

semaphores, message queues)

RTOS or Linux Kernel Dispatcher Runs…– On Entry Point into Kernel Following Initialization

– Loops (Idle) Until Task(s) in Ready QueueDispatches Highest Priority Task from Ready QueueIdle is NOT a Task Context (Kernel Context)

– Tasks or Threads become Ready and Are Dispatched When …Sequencer or Interrupt Service Routines Release (E.g. semGive(), sem_post())As a Result of a System Call (E.g. mq_send(), taskActivate(), pthread_create())Following Return from an Interrupt Service Routine

Dispatch Forces a Context Switch– Task/Thread State of Preempted is Saved (RTOS VX_FP_TASK for floats)

Coming from Idle, No Preemption, Simply DispatchRegisters, Stack, PC (Program Counter or Instruction Pointer) Saved

– Task State of Dispatched Task is Restored– PC Set to Last Text Address of Dispatched Task

NPTL Documentation - manpage (Red Hat paper on NPTL out of date)

Liu and Layland Paper

Liu, Chung Laung, and James W. Layland. "Scheduling algorithms for multiprogramming in a hard-real-time environment." Journal of the ACM (JACM) 20.1 (1973): 46-61. (11,711 citations)

Describes RM fixed priority

Describes RT Dynamic priority as “Deadline Driven” scheduling

Both are still used today

RM fixed - HRTRT Dynamic priority - SRT

16

POSIX Threads - Real-Time SCHED_FIFO Multi-threading with RT priority– FIFO refers to First-in-First-Out for threads of same priority– Ready queue is a priority queue– All dispatched threads run-to-completion at priority– Unless interfered with by a higher priority thread that enters queue– Or they block on need for resource in addition to CPU core– Example uses Multi-threading, SCHED_FIFO and affinity

http://ecee.colorado.edu/~ecen5623/ecen/ex/Linux/rt_thread_improved/

Multi-threading with synchronization– Synchronization can caus blocking, but this is sometimes unavoidable

http://ecee.colorado.edu/~ecen5623/ecen/ex/Linux/twoprocs/

Sequencing services on a periodic basis– Ultimately we need to sequence services from hardware interrupts, or

an interval timer, here’s a look ahead at one way to do thishttp://ecee.colorado.edu/~ecen5623/ecen/ex/Linux/simplethread-affinity/

17

Issues Beyond Policy and Feasibility

Latency Control– Do We Allow Deadline Over-runs?

Over-runs Use Resources That Could Be Used By Services On Time So FarDo We Need to Execute a Missed Deadline Call-back Handler?Should We Just Immediately Terminate a Service Release That Misses its Deadline?

Dealing with Non-Determinism– Do We Always Have to Assume Worst Case?

WCETShortest Period (Worst Case Release Frequency)

– When Have We Tested Long Enough to Have Verified Worst Case Assumptions?

Overhead– Lump it Into Worst Case?, Ignore it?

18

Summary

– Software use in HRT systems from Apollo 11 to Today– CPU Scheduling Overview

– Linux CFS [Completely Fair Scheduler]– Linux CFQ [Completely Fair Queue for I/O]– We need an “unfair” scheduler for Real-Time

– RTOS Schedulers and POSIX RT – Linux, FreeRTOS, VxWorks

– RT Services - Definition of various types– Linux POSIX RT Threads for Real-Time (SCHED_FIFO

Copyright © 2019 University of Colorado