Enabling Arm® DynamIQ™ supportconnect.linaro.org.s3.amazonaws.com/sfo17/Presentations...Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ

Enabling Arm® DynamIQ™ supportDan Handley (Arm)

Ionela Voinescu (Arm)Vincent Guittot (Linaro)

ENGINEERS AND DEVICES

WORKING TOGETHER

Agenda

● DynamIQ introduction

● DynamIQ and Arm Trusted Firmware

● OS Power Management with DynamIQ

● L3 partial power-down support


WORKING TOGETHER

DynamIQ™ key featuresFrom https://developer.arm.com/technologies/dynamiq

1. A new single-cluster design

2. Intelligent compute capabilities

3. Interfaces for closely coupled accelerators

4. Built-in power-saving features

5. DynamIQ big.LITTLE

6. Advanced RAS and safety features

https://developer.arm.com/technologies/dynamiq


WORKING TOGETHER

DynamIQ™ key featuresFrom https://developer.arm.com/technologies/dynamiq

1. A new single-cluster design

2. Intelligent compute capabilities

3. Interfaces for closely coupled accelerators

4. Built-in power-saving features

5. DynamIQ big.LITTLE

6. Advanced RAS and safety features

https://developer.arm.com/technologies/dynamiq

ENGINEERS AND DEVICESWORKING TOGETHER

DynamIQ Shared Unit (DSU)TRM: http://infocenter.arm.com/help/topic/com.arm.doc.100453_0002_00_en

● Armv8.2+ Cortex-A CPU support○ e.g. Cortex-A55, Cortex-A75

● 2 different CPU types in same cluster○ Maximum 8

● Per-CPU L1+L2 caches and shared L3

● Per-CPU DVFS control

● Partial L3 cache power down

● Hardware assisted power management○ Simplifies power up/down software

http://infocenter.arm.com/help/topic/com.arm.doc.100453_0002_00_en


WORKING TOGETHER

Agenda






DynamIQ Shared Unit (DSU) and Arm TF● DSU enables simpler, faster and more robust software during power up/down

○ Simplified micro-architectural programming sequence

○ Automatic enabling and disabling of coherency with the interconnect

○ Automatic and faster cache flushing at all levels without software intervention

○ Reduced power controller communication via P-channel interface

● TF enables more performant PSCI operations via HW_ASSISTED_COHERENCY

option○ CPU idle, hotplug, secondary CPU boot

○ Will still work without HW_ASSISTED_COHERENCY but won’t get the benefits

○ Allows more aggressive OSPM tuning

○ Warning: Some HW operations will be invisible to SW and may give misleading statistics


CPU idle to power down (Armv8.0 CPUs)

● Validate CPU_SUSPEND arguments

● Acquire locks for non-CPU levels

● PSCI state coordination

● CPU-specific power down handling○ Disable data caches

○ Flush data cache(s)

○ Disable intra-cluster coherency (!SMP_BIT)

● Stack maintenance

● Platform suspend operations

● Release locks for non-CPU levels

● Wait For Interrupt (WFI)

● Minimal SCTLR initialization

● Platform reset handling

● CPU-specific reset handling○ Errata handling○ Enable intra-cluster coherency (SMP_BIT)

● CPU architectural register initialization

● Enable MMU


● Platform suspend-finish operations

● Stack maintenance

● Enable data caches

● Restore OS context

● PSCI bookkeeping


● ERET to OS

Power Down Power UpOS calls SMCCPU_SUSPEND

Reset






● CPU-specific power down handling○ Request CPU power down

(CORE_PWRDN_EN)






● CPU-specific reset handling○ Errata handling (none yet)


● Enable MMU and data caches






● ERET to OS


Reset







(CORE_PWRDN_EN)














● ERET to OS


Reset

D$ enabledmuch earlier

D$ remains

enabled

throughout







(CORE_PWRDN_EN)














● ERET to OS


Reset

No need for explicit

cache flushes or stack

maintenance







(CORE_PWRDN_EN)














● ERET to OS


Reset

Much more efficient spin locks instead of bakery locks(using v8.1 CAS instruction)







(CORE_PWRDN_EN)














● ERET to OS


Reset

No need for explicit interconnect

programming for masters to

enter/exit coherency

(Potentially) reduced

power controller

communication


WORKING TOGETHER

Future TF enhancements● Use per-thread cluster power voting register

○ CLUSTERPWRDN_EL1

○ Automatic cluster power down or memory retention ...

○ ... if the power controller hardware and firmware support it

● Remove cluster level locks○ or at least reduce the time they are held

● Analyze performance on DynamIQ hardware platforms


WORKING TOGETHER

Agenda






OS Power Management with DynamIQ● Finer grained power capabilities

○ Already handled by PM frameworks

● Per-core Frequency/Voltage domain

● DSU Frequency/Voltage domain


Scheduler domains● Current big.LITTLE system

○ Energy model layout matches scheduler domain

● Example of 4 big cores + 4 LITTLE cores:


Scheduler domains● DynamIQ changes domains boundaries

○ Not necessarily congruent

○ Physical / Voltage / Frequency / Architecture

● Change the scheduler topology○ And energy model layout

● Example of 4 big cores + 4 LITTLE cores:


Phantom domains● Add intermediate domain

○ Voltage/Frequency boundary

● Example of 4 big cores + 4 LITTLE cores:○ Per core DVFS


Phantom domains● Example of 4 big cores + 4 LITTLE cores:

○ One frequency domain for big cores and one for LITTLE cores

○ Frequency domain close to current big.LITTLE system

● Enable similar scheduler

topology


WORKING TOGETHER

OSPM next steps● Shared frequency domains

● Shared voltage domains

● Impact on energy model

● Impact on compute capacity

● Getting notified of power domain OPP change

● Multiple DynamIQ clusters

Reference:https://developer.arm.com/-/media/developer/developers/open-

source/energy-aware-scheduling/DynamIQ_design_specification_v1.0.pdf

https://developer.arm.com/-/media/developer/developers/open-source/energy-aware-scheduling/DynamIQ_design_specification_v1.0.pdf


WORKING TOGETHER

Agenda






WORKING TOGETHER

L3 partial power-down

● Arm DynamIQ Shared Unit (DSU) L3 cache○ Implementation specific number of portions controlled through a

power control register

○ Counters for cache misses and cache hits to help drive decisions

● Support in software○ DevFreq driver

○ Control of active portions based on:

■ Cache hit/miss rates

■ Computed power benefit

■ Bias for performance

○ Out of tree reference implementation:https://git.linaro.org/landing-teams/working/arm/kernel-release.git/log/?h=dsu_partial_powerdown_support_v1.0

https://git.linaro.org/landing-teams/working/arm/kernel-release.git/log/?h=dsu_partial_powerdown_support_v1.0


L3 partial power-down: architecture

hit counter

DSU register interfaceLinux Kernel DSU L3 cache

miss counter

control register

DevFreq governor

DevFreqdevice

Target portions

Timer10ms

Update DevFreq

Set target portions


L3 partial power-down: algorithmUpsize: Weigh additional cost in energy of

enabling another portion against potential

savings by decreasing dynamic cost of

accessing DRAM.

● Condition for upsize:

MBW > (1.0 – Tu) * CB

● MBW – miss bandwidth: MiB/sec

● CB – cost bandwidth: MiB/sec○ CB = L / ED

● L – static leakage of single portion: uJ/sec

● ED – dynamic energy of DRAM: uJ/MiB

● Tu – upsizing threshold: fraction 0.00 to 1.00○ Bias for performance

● Compare energy consumption○ Bias for performance

L3 cache static

DRAM dynamic

energy


L3 partial power-down: algorithm - 1Downsize: From an energy trade-off perspective,

to justify a portion to be powered on, requires a

hit bandwidth that pays for its leakage. If that

requirement is not met, it can be powered-off.

● Condition for downsize:

HBW < (N – Td) * CB

● HBW – hit bandwidth: MiB/sec

● N – current number of portions enabled

● CB – cost bandwidth: MiB/sec○ CB = L / ED

● L – static leakage of single portion: uJ/sec

● ED – dynamic energy of DRAM: uJ/MiB

● Td – downsize threshold: fraction 0.00 to 1.00○ Bias for performance

● Compare energy consumption○ Bias for performance

L3 cache static

DRAM dynamic

energy


L3 partial power-down: behaviour

Example:

● 2MB L3 cache

● Memcpy workload with

buffer size of 4MB


L3 partial power-down: behaviour - 1

Expected behaviour:

● CPU intensive workloads

should not have an effect

on the number of active

portions

● I/O intensive loads should

raise portions when the

cache is well used


WORKING TOGETHER

L3 partial power-down

● Limitations of current reference implementation○ Portion is the smallest single unit of the cache that can be powered

up/down

○ Only support for a single DynamIQ Shared Unit

○ Not suitable for use with the simple on-demand governor

● L3 partial power-down in Arm Trusted Firmware?

Reference:https://developer.arm.com/-/media/developer/developers/open-

source/energy-aware-scheduling/DynamIQ_design_specification_v1.0.pdf

https://developer.arm.com/-/media/developer/developers/open-source/energy-aware-scheduling/DynamIQ_design_specification_v1.0.pdf

Thank You

#SFO17

BUD17 keynotes and videos on: connect.linaro.org

For further information: www.linaro.org

http://www.linaro.org/

Documents

Enabling Arm® DynamIQ™ supportconnect.linaro.org.s3.amazonaws.com/sfo17/Presentations...Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ