The Experiment Orchestration Toolkit (ExOT)

Bruno Klopott?§ Philipp Miedl†§ Lothar Thiele‡§

AbstractResearchers are required to support their claims with

experimental evidence and provide results that are reproducible,comparable and exhaustive. However, the effort required toconduct exhaustive experimental analyses, or reproduce andcompare different results, has proven to be high. To tacklethese issues, we present the Experiment Orchestration Toolkit(ExOT).

In this whitepaper, we give a detailed overview of the designand implementation strategies used during the development ofthe first public version of ExOT. ExOT is designed to be easilyextended and can be used to easily include a variety of differentplatforms in a measurement setup. It helps to automate theprocess of setting up, executing and analysing measurements.All components of the ExOT project that were developed atETH Zürich are publicly available under the 3-clause BSDlicense.

1. Introduction

The Experiment Orchestration Toolkit (ExOT) projectemerged from research conducted at the ComputerEngineering Group at the Computer Engineering andNetworks Laboratory at ETH Zürich. Various versions ofExOT were used to conduct the experiment for multiplepublications [2], [3], [13], [14], [24], which allowed theframework to evolve into a flexible measurement toolkit.

The basic design of ExOT assumes a measurementsetup with five components:• The measurement environment consisting of different

zones.• The source application which actively interacts with

the measurement environment.• The sink application which observes the measurement

environment.• The jammer applications, which tamper with the meas-

urement environment to provide the possibility forcontrolled external influences onto the measurement.

• The experiment engine for data processing andexperiment orchestration.

These five components and their relation are illustratedin Figure 1.

ExOT is designed to be easily extendable, allowingit to be applied in a broad range of measurement

*bruno.klopott@alumni.ethz.ch†miedlp@ethz.ch‡thiele@ethz.ch§ETH Zurich, Computer Engineering and Networks Laboratory

(TIK), Gloriastrasse 35, Zurich, Switzerland

environment description file

configuration file

driver

experiment

zone(s) source app sink app jammer app(s)

experiment engine

experimental data

Figure 1: Block diagram of Experiment OrchestrationToolkit (ExOT), illustrating the relation of the differentcomponents.

tasks. The main goals were to reduce the engineeringburden placed on the researchers to conduct extensiveand scalable measurements campaigns, to support thereproducibility, comparability and expressiveness ofmeasurement results.

In this paper, we give detailed insights into thedesign of the first public version of ExOT, which isan extensive software package compromised by anapplication library, an application compilation suite, andthe experiment engine. The application library and thecompilation suite speed up the implementation of source,sink and jammer applications for different processorarchitectures. The experiment engine mimics the layeredstructure of a communications channel, and simplifiesand systematises the information flow and the analyses.

In Section 2 we present implementation details of theapplication library and in Section 3 we illustrate howapplications are build and which testing facilities areincluded in ExOT. Section 4 gives an overview of theAndroid integration of ExOT and in Section 5 we presentthe implementation strategy for the experiment engine.In Section 6 we present small examples of the usage ofExOT, list possible future extensions in Section 7 andgive some concluding remarks in Section 8.

2. The application library

The Kahn Process Networks model was chosen as theconceptual underpinning of the application library. Themodel allows for great extendability and reusability,because the individual nodes of the network are selfcontained, and communicate only through well definedinterfaces. Any number of nodes can be introducedwithout any impact to the existing functionality andthe model is simple yet expressive. Processes in process

network nodes can communicate only via unboundedfirst-in, first-out queues, but can perform computationof any degree of complexity1. The message passingsemantics are rather straightforward, requiring blockingread accesses and non-blocking writes to the queues. Thismeans that a process attempting to read from an emptyqueue will be suspended until a token is available. Sucha model can describe systems that process streams ofdata, which can run sequentially or concurrently. Furtherdetails about the model of computation, its extensions,and implementation requirements are presented byGeilen and Basten [38, ch. 2], Allen [32, ch. 3] and Vrba[35].

2.1. Related work

A number of potential candidates for core process networksupport were identified: (i) Computational ProcessNetworks (CPN) by Allen [32], (ii) RaftLib++ [27],(iii) FastFlow [30], (iv) Yapi [39], and (v) ThreadingBuilding Blocks (TBB) by Intel [31]. Nornir [35] wasalso investigated, but the implementation has not beenmade publicly available. Task-based models were alsobriefly explored; a survey is given in [34].

After reviewing these related libraries we decidedto implement ExOT from scratch due to limitations ofthese libraries or them being outdated or unmaintained.However, a few observations were influential on thesubsequent design process:• Templatable input and output ports. CPN, TBB, RaftLib++,

and Yapi use template parameters to indicate whichdata type is used by the input and output interfaces. InTBB, the parameters are passed to node declarations,e.g., tbb::flow::function_node<int, int>. In RaftLib++the input/output interfaces contain template memberfunctions, e.g. output.addPort<int>(/*...*/). In Yapi andCPN, a separate interface class template is used, e.g.,Out<int>. Such an approach seems certainly superior tocasting a void* argument, and better than configuringdata types at runtime.

• Using strong types. Some of the libraries seemed tobe using code constructs specific to C in their C++codebases. Most notably, some used generic pointersto void for “carrying” data between nodes. Such anapproach provides no type safety, and might resultin unexpected errors at runtime. Stronger typing canhelp prevent many errors at compile time and makesoftware less ambiguous. Moreover, modern C++provides avenues for polymorphic types and genericcontainers.

• Abstracting the creation of queues from the user. Thelibraries that did not require the explicit creation ofqueues seemed much friendlier. For example, in TBB1Data-flow graphs “have computational capability equivalent to a

universal Turing machine” [42, p. 32].

. . .input parser

data nodes

modulegenerator

module

. . .logger modulemeter

module

system

source schedule

meas. data

SRC

SNK

token queuedata I/O interaction

Figure 2: Process Networks model applied to theExperiment Orchestration Toolkit (ExOT) applicationdesign.

the tbb::flow::make_edge(node, node) function is used,FastFlow uses consecutive calls to pipeline.add_stage(),and RaftLib++ provides a rather strange operatoroverloading (map += producer >> consumer;).

• Function nodes. Another aspect that distinguishedTBB from the rest was the ability to quickly definenodes with a function_node class template. This featuremakes it particularly useful for quick exploration andtesting.

• Using pointers and references to objects and settingsstructures instead of string descriptions for configuringthe process network. The libraries which rely on the latterrequired many more steps before the nodes and thenetwork were usable.

2.2. Application design

Figure 2 shows how the process networks concept couldbe applied to realise a measurement. The components ofthe channel model can be naturally expressed as processnetwork nodes and connected with queues that can carryany data type, including heterogeneous or variable-sizecontainers. The ellipses in the diagram indicate that othernodes could be introduced, as long as they conform tothe input data type requirements of the dependent node.

The core of the library aims to provide the buildingblocks necessary for using the process network model.Nodes Initially, the library will only provide supportfor single input and single output, which meets theneeds of all existing covert channel applications. Threecomplementary classes of nodes are defined: consumers,producers, and processors. As the names suggest, theydiffer in the type of interfaces they provide, the lattercombining both input and output interfaces. Similarlyto GNU Radio and the frameworks listed in Section 2.1,the nodes contain a single executable process.Interfaces Since queues are generic data containers,encapsulating interfaces are required to enforce theformalism of the process networks. The library willprovide an abstraction of the underlying queues or

2

communication channels, that ensures that only single-directional access is allowed. To allow for some deadlockavoidance, the interfaces in the library are extended withthe ability to “time out”. If the attempt to access thequeue is not successful after a specified amount of time,it will be given up and indicated with a return status. Ifunsuccessful, there will be no side effects, i.e., the queuewill not be modified.Connectors In order to not burden the programmerwith the creation of queues and bootstrapping theinterfaces, the library will provide facilities to connectnodes together. The task of the connectors will be to verifythe compatibility between nodes, create appropriatequeues, and provide them to the nodes’ interfaces.Executors Executors will abstract the task of runningthe nodes’ processes. These might in the future be usedby bespoke schedulers, but initially will aim to executethe processes on threads that are scheduled by otherentities, like the operating system.

2.3. Software design

From a software engineering perspective, the chosendesign patterns and programming idioms supportextendability, and provide an easy to use applicationprogramming interface (API).

To achieve reusability and extendability, the libraryaims to take the full advantage of the genericprogramming capabilities of C++. The language provideswhat is known as “templates”, which allow creatingclasses and functions that can operate on different datatypes. For example, we may declare a function template:

1 template <typename L, typename R>2 auto function(L arg_l, R arg_r);

An entity with such a declaration does not have any hard-coded types. Instead, the function template is instantiatedwhen needed (implicitly or explicitly), and the compilergenerates the actual function [see 19, temp.spec].For example a call to function(static_cast<int>(1),

static_cast<long>(2))will instantiate a function implicitlywith template parameters L and R being int and long.With modern C++ we can also place constraints on thetemplate arguments. For example, the instantiation of thefunction template above can be restricted to arithmetictypes with:

1 template <typename L, typename R,2 typename =

std::enable_if_t<std::is_arithmetic<L>::value↪→

3 &&4 std::is_arithmetic<R>::value>>5 auto function(L arg_l, R arg_r);

Templates are even more powerful when applied toclasses and their member functions, and combined with

other language facilities, like inheritance. The followinglist provides some of the programming idioms thatguided the development of the library.Policy-based design This design technique makes useof templates to allow “assembling a class with complexbehaviour out of many little classes, each of whichtakes care of only one behavioural or structural aspect”[40, p. 45]. The library will strive to make use of thisidiom whenever some orthogonal functionality can bedecomposed into smaller structures.“Template template parameters” To facilitate usingmultiple template parameters, some of which also beingtemplate entities, the library makes use of “templatetemplate parameters”. These constructs also help withpolicy-based design [40, p. 76]. Examples can be foundthroughout the Standard Template Library (STL); forexample, the std::vector is a class template with atemplate parameter for the value type, but also aparameter for an allocator, which itself is a class template.“Template template parameters” allow propagatingtypes, making the API cleaner. In the library, this idiomwill be used to pass value types to containers, which thenwill be passed to interfaces operating on them. Sincethis idiom is quite difficult to describe, an applicationexample is given in Section 2.5.1 (for the code sampleshown in Listing 1).RAII The behaviour known as “resource acquisition isinitialisation” is used throughout the library to ensurethat access to a particular resource is held during anobject’s lifetime. Notably, the library will use reference-counted smart pointers for managing dynamicallyallocated queues. Thanks to that, there will be no risk ofending up with a “dangling pointer”, since the sharedqueue will not be destroyed as long as there is any entityholding a reference to it.SFINAE “Substitution failure is not an error” is a rulethat applies to function templates. With so-called “typetraits” and compile-time polymorphism it is possible toprovide conditional overloading of function templatesvia std::enable_if (e.g., a single print function that hasdifferent overloads for different types), check for theexistence of specific member functions, or to providestatic checks of matching types. In the library, it will beused to provide generic functionality while avoidingunnecessary abstraction through class hierarchy.Meta-programming & variadic templates Templatescan also be used to “generate” code at compile time, ina much type-safer manner than using preprocessordefinitions. That also includes function and classtemplates that work with variable number of differenttypes that need not share a common base class. Thisidiom can be particularly powerful when combined withinheritance in class templates, allowing the functionalityof multiple smaller classes to be joined together.

3

2.4. Library structure

The software has been arranged in descriptivenamespaces:exot::framework Defines the application library corefunctionality, including the process network nodes(consumers, producers, and processors), input andoutput interfaces to communication queues/channels,concurrent single-producer, single-consumer queues,thread-safe state holder, node and pipeline connectors,and executors.exot::utilities Includes a range of general-purposeand helper functions and classes, including timekeeping facilities, thread attributes, synchronisationprimitives, template meta-programming facilities, file-system utilities, bit manipulation helpers, command lineparsing, logging, type traits, input and output streamoverloads, and workers.exot::components Contains reusable complete processnetwork components, such as load generators, loggers,and input schedule parsers.exot::modules Includes modules used for sink meters,arranged by measured physical quantities (frequency,power, temperature).exot::primitives Contains functions and classes thataccess low-level subsystems, such as model specificregisters, time stamp counters, and platform-specifichelpers.

2.5. Application library core functionality

In this subsection, we present the application librarycore functionality, implemented in the exot::frameworknamespace.2.5.1. Node structureThe most fundamental building blocks of the applicationsare the process network nodes. Three classes of nodesare defined: consumer, producer, and processor. As thenames suggest, the consumer node has the ability to onlyread data (consume tokens) from an input interface,the producer node provides a write-only interface, andthe processor node combines the two. No bidirectionalcommunication over a single interface is possible.

The nodes are realised using class templates with“template template parameters”, using the techniquedescribed earlier in Section 2.3. The declaration of such aclass is shown in Listing 1. The Token type is passed on asa template parameter to the Container, which then bothare passed on to the Reader template parameter.With suchconstruction, one only needs to write IConsumer<int, queue,queue_reader> instead of the more verbose IConsumer<int,queue<int>, queue_reader<int, queue<int>>>. Node classesderive from virtual, empty base classes (e.g. TConsumer),which are necessary to make type traits more usable withthe class templates.

1 template <typename Token, template <typename...>typename Container,↪→

2 template <typename, template <typename...>typename> typename Reader>↪→

3 class IConsumer : public virtual Node, public virtualTConsumer {↪→

4 public:5 using consumer_type = IConsumer<Token, Container,

Reader>;↪→

6 using interface_type = Reader<Token, Container>;7

8 IConsumer() = default;9 explicit IConsumer(typename

interface_type::container_pointer input_queue)↪→

10 : in_(input_queue){};11 virtual ~IConsumer() = default;12

13 void set_input(typenameinterface_type::container_pointer input);↪→

14 typename interface_type::container_pointerget_input() const;↪→

15

16 protected:17 interface_type in_;18 };

Listing 1: Consumer node interface

Each node class template contains an interface object,which can either be initialised in the constructor,or configured after instantiation using set_input andset_output functions. The end user will rarely, if ever,need to use the non-default constructor. Classes derivingpublicly from the these base node templates can thenaccess the inherited interface object thanks to the protectedaccess specifier.

The class template for the processor node has multipleinheritance from both consumer and producer node classes.One important feature of processor classes is their abilityto bridge potentially disparate domains; hypotheticallyspeaking, the consumer side could be connected to anetwork interface, and the producer side to a regularqueue.

A UML class diagram which illustrates the rela-tionships between the various class templates, regular,abstract and interface classes, is presented in Figure 3.Such an organisation of software components allowsfor easy extendability and reuse. Since their most oftenchanged and crucial aspects are template parameters, itis trivial to declare nodes that deal with various tokendata types. If a new container or an interface is designed,to use it with the node, one only needs to pass them astemplate parameters. For further convenience templatealiases are provided for default containers and interfaces,such that the user only needs to supply the token typesused by the node.

The classes above are generic and do not yet define any

4

1 template <typename Token, template <typename...> classContainer> class↪→

2 QueueReader : public IReader<Token>, publicReader<Token, Container>;↪→

3

4 template <typename Token, template <typename...> classContainer> class↪→

5 QueueWriter : public IWriter<Token>, publicWriter<Token, Container>;↪→

Listing 3: Queue interfaces

executable elements. To implement some functionality,a user can derive from the IProcess interface class thatdefines a process-oriented execution model. The interfacecontains a pure virtual member function that needs tobe overwritten by implementing classes. In addition tothe void process() function the interface class adds thecommonly used shared pointer to the global executionstate. This is also the only place where an object, theGLOBAL_STATE, is declared with the extern specifier forexternal linkage.2.5.2. InterfacesCommunication between process network nodes hap-pens through interface class templates. They providethe set of functionality required for satisfying therequirements of the process network model, that isblocking reads, and non-blocking writes (as long as queuesare declared unbounded). The two pure virtual interfaceclass templates shown below define the functionsrequired for input (Figure 4) and output interfaces(Listing 2). The interfaces enforce the process networkformalism and allow for different calling styles of theunderlying containers (some use the combination offront, pop and pushmethods, while others define enqueueand dequeue). Similar layers of indirection are presentin other process network implementations [35, p. 73, 41,p. 19].

Additional Reader and Writer base class templatesallow reuse of constructors and provide commonfunctionality of setting and getting the shared pointers tocontainers used as the ‘transmission medium’ betweennodes. The concrete interfaces, the class templatesQueueReader and QueueWriter, implement the reader andwriter virtual interfaces for queue-like objects, and derivefrom the base classes above (Listing 3).

The extended queue interfaces additionally implementalternative semantics and define member functiontemplates such as try_read_for(token, timeout), as shownin Listing 4. Analogous functions are provided for thewriter interface. The important distinction from the basicinterface is the addition of a return type (bool insteadof void), which indicates whether the operation wassuccessful and allows for less conventional, defensive

1 inline bool try_read(Token &token);2 template <typename Rep, typename Period>3 inline bool try_read_for(Token &token,4 const

std::chrono::duration<Rep,Period> &timeout);

↪→

↪→

5 template <typename Clock, typename Duration>6 inline bool try_read_until(7 Token &token, const std::chrono::time_point<Clock,

Duration> &time);↪→

Listing 4: Extended queue interface

code constructs.2.5.3. QueuesThe underlying communication channel has to abide bythe process network formalism. The queue providedin the C++ standard libraries does not meet therequirements. First of all, std::queue is non-blocking. Anempty queue can be “popped”, and calls to the front()method can return data from uninitialised memorywhen the queue is empty. Moreover, std::queue is notthread-safe. Even with the queues having only two users,concurrent access has to be free of race hazards.

The library provides a concurrent queue suitable forsingle producer-consumer scenarios that mirrors theinterface of std::queue, and its more complex extensionthat additionally implements the extended semantics.The queues are provided as class templates, allowing theuse of different token data types and synchronisationprimitives. Moreover, the queues can have boundedcapacity.

The implementation uses two thread synchronisationconstructs: mutual exclusion locks, and conditionvariables. Their combination allows efficient lockingand monitoring the status of certain boolean conditions.A lock protect access to private class variables in each ofthe public interface functions. Two condition variablesare used for waiting and notifying on empty and fullqueues.

All read operations acquire the lock and wait until thequeue is not empty, typically using the code constructlisted below:

1 std::unique_lock<mutex_type> lock(queue_mutex_);2 queue_empty_cv_.wait(lock, [this] { return !empty_();

});↪→

An analogous mechanism is provided for writeoperations, which will block on a full queue. Waitingusing a condition variable is roughly equivalent towhile(!predicate) lock.lock();.

If the read or pop operation was called on an emptyqueue, the calling thread will efficiently wait untilnotified by another thread performing a write operation.

5

1 1

Consumer

T, C, I

IConsumer

+ set_input(in : pointer) : void+ get_input() : pointer

T, C, I

Node TConsumer

�interface�IProcess

# global_state_ : pointer

+ process() : void

Reader

# in_ptr_ : pointer

T, C

�interface�IReader

+ write (token) : void+ is_writable() : bool

T

Queue

+ front() : T+ pop() : void+ push(token : T) : void

T

Figure 3: UML class diagram showing relationships in core framework’s nodes

1 template <typename Token>2 struct IReader {3 virtual ~IReader() = default;4 virtual void read(Token &) = 0;5 virtual bool is_readable() = 0;6 };

Figure 4: listingReader interface

1 template <typename Token>2 struct IWriter {3 virtual ~IWriter() = default;4 virtual void write(const Token &) = 0;5 virtual void write(Token &&) = 0;6 virtual bool is_writable() = 0;7 };

Listing 2: Writer interface

Notifications are provided by two internal functionsnotify_waiting_on_empty_() and notify_waiting_on_full_().

The extended interface provides try_pop and try_pushfunctions, which do not block if the queue is empty orfull; if the preconditions are not satisfied, the queue willnot be modified and the functions will return false.

The usefulness of the condition variables becomesapparent in functions that attempt to read or write only

1 template <typename Rep, typename Period>2 bool try_pop_for(reference value, const duration<Rep,

Period> &timeout) {↪→

3 std::unique_lock<mutex_type> lock(queue_mutex_);4 if (queue_empty_cv_.wait_for(lock, timeout, [this] {

return !empty_(); })) {↪→

5

6 // ...7

8 lock.unlock();9 notify_waiting_on_full_();

10 return true; // if successful11 } else {12 return false; // if timed out13 }14 }

Listing 5: Accessing the queue with a timeout

for a specified period of time. The Listing 5 shows anexcerpt from the source code, which defines a try_pop_forfunction. It’s a function templates, which can be calledwith any duration object from the standard library, e.g.try_pop_for(token, std::chrono::seconds{2}).2.5.4. State

State objects in the library can be used to track localand global state. They contain a set of atomic boolean

6

variables and functions to access and manipulatethem, for example start() and is_started(). The ob-jects are meant to be used through shared pointers,therefore the State class derives in a CRTP2 fashionfrom std::enable_shared_from_this<State> and provides amember function to get a shared pointer to itself.

The only global variable with static storage durationused in the project is the pointer to a state object:

1 static State::state_pointerGLOBAL_STATE{std::make_shared<State>()};↪→

Thanks to the static storage specifier the pointer can beused in C-style signal handlers. Currently, the librarydefines the following global state signal handlers:

1 static void interrupt_handler(int) {GLOBAL_STATE->stop(); }↪→

2 static void terminate_handler(int) {GLOBAL_STATE->terminate(); }↪→

3 static void start_handler(int) {GLOBAL_STATE->start(); }↪→

2.5.5. ConnectorsOne of the limitations of some of the reviewed processnetwork implementations was the need to manuallycreate and connect queues to nodes. To remove thisburden from the users, a method for connectingcompatible nodes together is provided in the library.

A function template details::connect takes twonodes as arguments, verifies that they have compatibleinterfaces and token types, constructs a queue andpasses the shared pointer to neighbouring nodes. Theverification uses custom type traits and happens atcompile time thanks to static_assert.

Using a variadic function template, shown in Listing 6,any number of nodes can be connected. There is a basefunction that takes only two arguments, and a recursivefunction that takes any number of arguments. Thanks totemplate pack expansion a call to connect(Node1, Node2,Node3) will produce a sequence of calls connect(Node1,Node2); connect(Node2, Node3);. Additionally, the libraryprovides a structure wrapper that allows setting thedesired queue capacity3, and a pipeline function. Thepipeline function uses type traits and a compile-time for-loop to additionally verify that the first passed node is aproducer, and the last one is a consumer.2.5.6. ExecutorsExecutors provide a uniform way to execute processes.The base class provides a function template spawn, whichcan be used to execute any callable object, optionallywith arguments. An example is listed below:

2Curiously Recurring Template Pattern, in which the class derivesfrom a class template, using itself as a template argument.

3With the Connector class one can connect nodes with a queue ofsize 10 by calling Connector(10).connect(Node1, Node2, Node3);.

1 template <typename Left, typename Right>2 void connect(Left &&left, Right &&right) {3 details::connect(std::forward<Left>(left),

std::forward<Right>(right));↪→

4 }5

6 template <typename Left, typename Right, typename...Rest>↪→

7 void connect(Left &&left, Right &&right, Rest &&...rest) {↪→

8 connect(std::forward<Left>(left), right);9 connect(std::forward<Right>(right),

std::forward<Rest>(rest)...);↪→

10 };

Listing 6: Variadic function template for connecting nodes

1 template <typename Callable, typename... Args>2 void spawn(Callable &&callable, Args &&... args);

The concrete executor classes that are provided in thelibrary at the moment are meant for executing callableobjects on system and user-space threads. For example,the ThreadExecutor class will spawn each providedobject in a separate system thread, and is well suitedfor executing process network nodes. It additionallyprovides a way of spawning an object on a specialisedthread, with configurable pinning, scheduling policy andpriority.

The other available executor facilitates the use of user-space threads, fibers, from the project Boost/fiber, andprovides convenience functions for spawning fibers andadding fiber pool worker threads. Both executors alsoprovide functions to query how many worker threadshave been instantiated, and to wait for the completionof and join spawned system/user-space threads.

2.6. Utilities

A wide range of utility functions and classes are providedin the exot::utilities namespace of the application library.These provide both essential functionality, like timekeeping or synchronisation mechanisms, and auxiliaryhelpers, like string formatting and type traits. We describethe most important of those utilities in the remainder ofthis subsection.2.6.1. TimingSleeping and estimating time offsets has been extractedinto a TimeKeeper class template. The class gives astraightforward interface, with member functions begin(),sleep(chrono::duration), and update_offset() providing thebulk of the functionality. Additionally, the class has arun_every function, which can invoke a callable objectwith arguments periodically until some predicate is false.For example, to perform and output some measurementevery 10 milliseconds until the global state is terminated,one can run:

7

1 tk.run_every(std::chrono::milliseconds{10},2 [&]() { return

!GLOBAL_STATE.is_terminated(); },↪→

3 [&]() { std::cout << meter.measure(); })

The time keeper can use different clock sources,different sleep functions, and also provides meanand standard deviation of intervals and offset. The<exot/utilities/timing.h> header also provides astd::chrono-like interface to the POSIX nanosleep

function.The library provides different timing primitives

from literature, the ability to introduce serialisation toany clock type via a fenced_clock Curiously RecurringTemplate Pattern (CRTP) class template, and ExOTclock sources based on system’s monotonic clockand performance counters. Moreover, it significantlysimplifies the timing of any callable type with the help ofa timeit function template. To support faster operation,the TimeKeeper utility has been provided the ability to usebusy-loop sleeping.2.6.2. Barrier synchronisation primitiveThe barrier implements a rendezvous point for aconfigurable thread count. Threads arrive at the barrierand are allowed to pass it only once the specified numberthreads have reached it. The threads wait on the barrierefficiently, i.e. there is no busy waiting, and threads areidle. To allow for greater usability and portability, theolder barrier primitive from the POSIX thread library hasbeen replaced with a solution inspired by Boost librariesand C++ standard proposals [10], and uses just the C++standard library. The advantage of the ExOT barrier isthat it can be reused multiple times and can be used onthe Android system with API support below level 244.

The ExOT barriers are implemented with a combina-tion of locks and condition variables, both of which aretemplate parameters, allowing for the use of differentprimitives. The barrier is initialised with a desired threadcount. As threads arrive at the barrier, they incrementan internal current thread count. If the count is less thenthe desired one, the entering thread waits on a conditionvariable. If the entering thread is the last to enter thebarrier (incremented current count is equal to the desiredcount), it uses the condition variable to notify all waitingthreads.2.6.3. JSON interfaceThe application library has introduced generic supportfor JavaScript Object Notation (JSON)-based configura-tion. The format was chosen due to the simplicity of themarkup language and very good software support witha library by Lohmann [6] that awaits standardisation.

4The functions pthread_barrier_init andpthread_barrier_wait are marked __INTRODUCED_IN(24) inthe <pthread.h> header file in the Android’s system root. API level 24is not yet publicly available.

1 struct MyConfigurableClass :configurable<MyConfigurableClass> {↪→

2 std::vector<std::string> MyStrings; //! A vectorof strings↪→

3 std::map<std::string, double> MyMap; //! A map ofpairs string->double↪→

4 const char* name() const { return"MyConfigurableClass"; }↪→

5 void configure() {6 bind_json_to_data("MyStrings", MyStrings);7 bind_json_to_data("MyMap", MyMap);8 }9 };

10

11 /* Later used as... */12 MyConfigurableClass instance;13 instance.set_json(the_json); instance.configure();14 /* Or using a helper function... */15 configure(the_json, instance);

Listing 7: Demonstration of configurable classes

It features strongly-typed support for all fundamentaltypes and most data structures from the STL, as well asease of providing overloads for functions performingserialisation and de-serialisation of user-defined types.

The support for JSON-configurable classes has beenprovided using the programming paradigm of CRTP. Ina more common class hierarchy, we would use a baseclass with virtual or pure virtual functions, which arethen implemented in derived classes. With this paradigm,however, the base class is a class template that takes thederiving class as a template argument, and inherits fromit. Such structure allows a base class to access the derivedclass and simplifies multiple inheritance from otherconfigurable classes. The simplicity of this solution isdemonstrated in Listing 7. The bind_json_to_data instructswhich key is to be mapped to which member variable.2.6.4. Command line parsing

The command line interfaces in the library rely heavilyon the very idiomatic header-only library clipp by‘muellan/clipp’ [16]. One of the most helpful features ofthat library is the automatic generation of help messages,allowing for good extendability of applications andreducing the programmer’s burden.

A separate class CLI builds upon the parsing facilitiesprovided in the clipp library, and allows easily addingindividual components configurations to a masterconfiguration, adding description and example sectionsto the printed help message5, and takes care of parsingthe command line arguments and notifying aboutparsing errors. Applications using the JSON interfacefor configuration will provide a command line interfacebased on clipp as outlined in 8.

5Additional sections are automatically wrapped to 80 columns.

8

1 $ ./generator_utilisation_mt2 Parsing error: 4 missing/incorrect, 0 blocked, 0 conflicts3

4 SYNOPSIS5 ./x86_64/generator_utilisation_mt ((--json_file <json file>) | (--json_string <json6 string>))7

8 ./x86_64/generator_utilisation_mt -h9

10 OPTIONS11 (--json_file <json file>) | (--json_string <json string>)12 JSON-based configuration13

14 -h, --help print help message15

16 CONFIGURATION17 [logging]18 | provide_platform_identification should provide platform info? |bool|19 | debug_log_filename the debug log filename |str|, optional20 | app_log_filename the app log filename |str|, optional21 | log_level the log level |str|, one of "trace", "debug", "info", "warn",

"err", "critical", "off"↪→

22 | async_size async logger buffer size |uint|, power of 223 | async use the async logger? |bool|24 | async_thread_count async logger thread count |uint|25 | timestamp_files should timestamp the log files? |bool|26 | rotating_logs should rotate the logs? |bool|27 | append_governor_to_files should append the frequency governor to filenames? |bool|28 | rotating_logs_size the size of rotating logs in MiB |uint|29 | rotating_logs_count the maximum number of rotating logs to keep |uint|30 [schedule_reader]31 | input_file the input schedule file |string|, e.g. "input.sched"32 | reading_from_file reading from file? |bool|, reads from stdin if false33 | read_as_hex read hexadecimal values? |bool|34 | cpu_to_pin schedule reader pinning |uint|, e.g. 735 [generator]36 | cores cores to pin workers to |uint[]|, e.g. [0, 2]37 | should_pin_workers should pin the workers? |bool|, default 'true'38 | worker_policy scheduling policy of the workers |str, policy_type|, e.g.

"round_robin"↪→

39 | worker_priority scheduling priority of the workers |uint|, in range [0, 99], e.g.99↪→

40 | host_pinning generator host core pinning |uint|, e.g. 541 | should_pin_host should pin the host? |bool|, default 'true'42 | host_policy scheduling policy of the host |str, policy_type|, e.g.

"round_robin"↪→

43 | host_priority scheduling priority of the host |uint|, in range [0, 99], e.g.99↪→

44 | start_check_period state change detection update period |uint, µs|, e.g. 10045 | use_busy_sleep should use busy sleep loop? |bool|46 | busy_sleep_yield should yield thread in busy sleep loop? |bool|47 [generator]48 | cores cores to run workers on |uint[]|, e.g. [1, 2, 3]

Listing 8: Example for the command line interface of an application generated with the application library. Only aJSON file or a JSON string can be provided via the commandline. The help text lists all possible JSON configurationparameters of the application.

9

2.6.5. File system utilities

Since many of the application components accessvarious pseudo-files provided by procfs, devfs andsysfs, the library provides functions for listingdirectories and for searching directories usingregular expressions. Both recursive and non-recursivemethods are given. For example, to search forall thermal_zone temperature access pseudo-files,the user needs only a single function invocation,grep_directory_r("/sys/devices/virtual/thermal",

".*zone\\d+/temp$"), which will return a vector ofpaths to the files (e.g., “. . . /thermal_zone0/temp”).2.6.6. Meta-programming helpers and type traits

The library provides several compile-time helpers, whichare used in other parts of the library. These includea compile-time for-loop, which can also be used as areplacement for code generation preprocessor directives,and functions to apply functions to heterogeneousstd::tuple containers. For example, to create 256 in-vocations of a callable object with signature void(size_t),one can use const_for<0, 256>([](auto I) { fun(I); }). Thetemplate will be instantiated at compile time and expandto a sequence of calls fun(0); fun(1); ...; fun(255);.

Another functionality that does not have any directapplication, but is used throughout the library are customtype traits. Type traits, which provide information abouttypes and evaluate predicates at compile time, are heavilyused in the C++ standard library. In the case of this work,they are used to check if a type is iterable or const-iterable, if a type is a tuple, or if a node has an input oroutput interface. These can later be used in static_assertstatements, or to conditionally enable certain templates.2.6.7. Formatting and logging support

To facilitate the process of formatting readings forlogging, the library provides overrides for formatting anyiterable or tuple-like objects to basic_ostream objects. Anexample of a signature of a function that overloads the <<operator is show in Listing 9. What this achieves is that alltypes T, which return true for is_iterable<T>::value, willbe allowed to use this overloaded function. In practicalterms, the user no longer has to write loops to outputvectors. For example, this becomes a valid statement withthe overload above: std::cout << std::vector<int>{1, 2, 3,4};. Possibly the greatest utility comes from an overloadfor tuples, since iterating over tuples is not possible.The overload uses the const_for loop above to accesstuple elements, allowing for printing of heterogeneouscontainers. All individual elements are comma separatedin the output.

Similar overloads are provided for reading tuplesand chrono::duration objects from basic_istream objects.This functionality is used in the schedule reader describedlater (Page 10).

2.6.8. Logging

Since logging is used universally across the wholelibrary and in all applications, it has been extractedinto a separate class, which has a uniform configurationinterface. The Logging class is responsible for creating logfiles, setting log levels (e.g., critical, debug), and makessure that the log files are readable. The functionality relieson the spdlog library [12], which provides a modern andhighly configurable interface, and is also compatiblewith Android logging facilities. The logging library isalso thread-safe and very fast: 4,328,228 messages persecond can be logged to a file in a single-threaded modeon an Intel i7-4770 processor [12]. For convenience, mostcomponents and modules try using loggers defined inthe global logger registry, both for application and debuglogging.2.6.9. Thread parameters

The library also features a ThreadTraits class whichprovides a portable way of setting the affinity andscheduling of threads. Most notably, a solution has beenfound for setting affinity on the Android platform, whichwas not possible in the legacy codebase. The Android-compatible way of pinning threads relies on the sys-callto SYS_gettid and using the POSIX sched_setaffinityfunction from the <sched.h> header.2.6.10. Workers

Several worker classes are available in the library.A worker can wrap some callable object in a loopthat checks the global execution state. Most notably,a worker using policy-based design principles isprovided, which can take different synchronisationand threading policy/mixin classes, allowing easyextendability. For example, Worker<BarrierSynchronisation,SpecialisedThreads> declares a worker that will be pinnedto a CPU and will invoke the callable object between twobarriers.

2.7. Components

In this section, we present the reusable completeprocess network components, implemented in theexot::components namespace.2.7.1. Schedule reader

The schedule reader is a producer class template thatparses values line-by-line either from a file, or from thestandard input (in contrast to standard input only in thelegacy framework). For ease of use and reusability, theschedule reader forms a token from each line of input, aslong as proper overloads were provided. At the momentany tuple type can be formed into a valid token. Thanksto such design, the schedule reader can be adapted todifferent components on the receiving end, with onlyminimal involvement of the user:

10

1 late <typename T, typename CharT, typename CharTraitsT>2 name std::enable_if<exot::utilities::is_iterable<T>::value && !exot::utilities::is_const_iterable<T>::value,

std::basic_ostream<CharT, CharTraitsT> &>::type & operator<<(std::basic_ostream<CharT, CharTraitsT> &ostream,const T &range);

↪→

↪→

Listing 9: Output stream operator overload

1 using loadgen = components::loadgen_mt;2 using reader = components::schedule_reader<typename

loadgen::token_type>;↪→

As this example demonstrates, the developer does noteven need to explicitly declare the token type, and can justuse the internal types of other components. In addition,the schedul reader support reading tokens that containiterable types (such as vectors) and fixed-size arrays.2.7.2. LoggerThe logger node is a very simple, but universal additionto the library. It is a class template that takes a token typeas a template parameter. Thanks to the provided outputstream overloads, it will format the token as a stringand write it to the application log, separating individualvalues with commas.2.7.3. Function nodesThe library also provides nodes that take a callableobject as an argument to the constructor. These canbe conveniently used for simpler operations and testing,and wrap the invocations of the callable in loopthat monitors the global state. Three class templatesare provided: FunctionConsumer, FunctionProducer, andFunctionProcessor, which take callable objects withsignatures void(token), token(void), and token(token)respectively. Unlike C-like function pointers, the callabletypes can be more complex capturing and/or mutablelambdas. For example, to produce a sequence of 10 tokensthe user of the library can write:

1 using token_type = std::tuple<int, std::string>;2 int counter{0};3

4 FunctionProducer<token_type> node([&]() mutable ->token_type {↪→

5 if (++counter > 10) GLOBAL_STATE->stop();6 return token_type{counter, std::string{"Token #"} +

counter};↪→

7 });8

9 node.process();

2.7.4. Platform identificationIn order to keep track of important platform charac-teristics alongside measurement log files the libraryimplements a large number of identification functions.Among others, they are responsible for obtainingprocessor details, processor package topologies andfrequency scaling settings.

2.7.5. Adapter nodesThe last type of components in the library allows bridgingdomains using different concurrency primitives: systemand user-space threads. These nodes simply forwardtokens, in a way that does not cause race issues.2.7.6. Meter hostThe host allows combining individual meter modulesinto a readily usable process network component. First,the MeterHost is class template, where meter modules areprovided as variadic template parameters. The MeterHostthen inherits publicly from each module, thanks totemplate pack expansion, as shown below:

1 template <typename Duration, typename... Meters>2 class meter_host3 : public Meters...,4 public framework::IProcess,5 public

framework::Producer<meter_token_type<Duration,Meters...>>

↪→

↪→

It’s token type is also determined automaticallyfrom the inherited modules, using an alias templatemeter_token_type:

Secondly, all module configuration structures arecombined and appended with host-specific settings, alsousing multiple inheritance and template pack expansion.The host has its own settings class, struct settings :Meters::settings...;, and own configure function, whichincorporates module-specific equivalents. The host’ssettings class is then passed to modules’ constructors;each module only accesses its relevant part of thestructure.

The host also combines all meter module measurementfunctions, and the results of their reported variables’names and units. Thanks to that the meter host’s processis very compact:

1 auto until = [this]() { return!global_state_->is_stopped(); };↪→

2 auto action = [this]() { out_.write(measure()); };3 timer_.run_every(conf_.period, until, action);

2.7.7. Meter modulesTo facilitate the declaration and use of metering facilities,the module class type has been introduced. The UMLclass diagram in Figure 5 shows the relationship betweenthe meter host and the modules. A meter module hasonly a few requirements:

11

• It must have a settings structure, struct settings, whichcan provide defaults and can be empty;

• It must have a constructor that takes the settingsstructure by reference;

• It must have a measure() function;• It should provide a vector<string> names_and_units()

function if the user wants to facilitate the processof adding headers to log files;

• The meter module should follow the ResourceAcquisition Is Initialisation (RAII) idiom, and shouldbe perfectly usable after instantiation.There are no hard requirements imposed on the what

kind of values a meter module can return via the measure()function. It is perfectly valid to have one meter returna std::tuple<std::string, int, double> and another returnstd::vector<long>. Although the meter function can returnvariable length arrays, it might be a better idea to keep thesize constant after initialisation to obtain a well formedoutput.

The use of meter modules is enabled by the meterhost component, described on Page 11. The modules canalso be used independently of the host, for example in areactive load generator.2.7.8. Generator host component and module base classes

The generator host and the generator modules arebuilt in a similar fashion to the meter host and metermodules. The generator host works in conjunction withgenerator modules, which define an interface composedof decompose_subtoken, validate_subtoken, and generate_load.The structure of both relies on type traits rather than classhierarchies, therefore generator modules can choose thetypes of each of the member types at their discretion.Moreover, to further simplify the creation of many similargenerator and metering modules the library introducesbase classes for those that require shared memory orinterpret their input as bitsets.

2.8. Primitives

In addition to generic utilities, the library also providesmore platform-specific utilities, arranged in the separateexot::primitives namespace. At the moment, these mostlyinclude functionality specific to Intel-based platforms.2.8.1. Model specific register access

The library provides a class for reading and writingmodel specific registers (MSR), which provides someimprovements over the methods used in legacy code. Inline with the overall aim of making most of the classesconform to the RAII idiom, the class initialises accessto MSRs upon instantiation. A number of checks areperformed to make sure that the arguments suppliedin the constructor are sensible and that all registerscan be accessed. Moreover, in addition to read andwrite functions, the class provides methods read_first,

read_any, and read_all, the latter returning a vector ofreadings. These functions, which have their write accesscounterparts, are mostly meant for users who use thedefault constructor of the class.2.8.2. Time stamp counter clock

Quite an interesting addition to the library is a clock thatexplicitly uses the time stamp counter (TSC) on Intelprocessors as the time source. The class provides thesame interface like the clocks found in the header of theC++ standard libraries. Thanks to that all the facilities inthe library (operations on and between time point andduration objects), handling of timing measurement withany clock implementing, that implements the standardinterface, is very simple, as shown below:

1 auto begin = tsc_clock::now();2 // ...3 auto elapsed =4 duration_cast<microseconds>(5 begin - tsc_clock::now());

Using the TSC as the clock source can be quite beneficial,but there is always the need to estimate the frequency ofthe monotonic counter. To maintain a clean and usableinterface, in tsc_clock the estimation happens upon thefirst invocation of the tsc_clock::now() function. Everysubsequent invocation can use the estimate immediately,because the relevant function-local variables have havestatic storage duration and are initialised the first timethe control flow passes their declarations.2.8.3. Memory mapping support

On the Linux Operating System (OS) and other unix-based systems shared memory is enabled by, amongothers, mmap. A ExOT MMap class implements a safe andconvenient wrapper that follows the programming idiomof RAII. It allows mapping file-backed and anonymousmemory into the virtual address space of the callingprocess, including anonymous and shared huge pagesallocations. The support for huge pages is necessaryin situations that require contiguous physical memorythat is aligned on the huge page size supported by theprocessor. For example, with 2 MB contiguous memoryone can very easily find addresses that map to the samecache set without the need for allocating a large memorybuffer.2.8.4. Cache parameters discovery

The library provides a CacheInfo class, which performsautomatic detection of specific cache’s properties (suchas ways of associativity or cache line size). A CPUCacheInfoclass discovers all caches of a particular processorcore. Moreover, both can be manually configured withany iterable type holding unordered maps of cacheproperties.

12

1..n

1

1

1..n

Host

+ settings : struct

+ Host(conf : settings &)+ configure(conf : settings &) : cligroup+ process() : void+ names_and_units() : vector<string>

Duration, Meters. . .

Host::settings

Module

+ settings : struct

+ Module(conf : settings &)+ configure(conf : settings &) : cligroup+ measure() : auto+ names_and_units() : vector<string>

Module::settings

Figure 5: UML diagram showing relationship between meter host and modules

2.8.5. Virtual-to-physical address translationThe library features an AddressTranslator functor classthat translates a virtual address into a physical one usingthe pagemap exposed for each process in Linux’es procfs.The pagemap is the Linux kernel’s interface that allowsprograms running in the user space to find out themapping between virtual memory pages and physicalframes.2.8.6. Custom allocators and alignment helpersTo benefit from the ease-of-use of STL containers whilebeing able to control their memory allocation, the libraryprovides an array of custom allocators. For example,the AlignedAllocator can be used to align a containeron a specific boundary and the HugePagesAllocator willallocate anonymous huge page memory to store the data.The allocators are passed as the second template argu-ment to many containers, for example: std::vector<int,HugePagesAllocator<int>. Additionally, the library providesa type wrapper that makes it easier to make types withexplicit alignment. The aligned_t<T, A> class template canbe used to produce a type T that has the desired alignmentA, for example via make_aligned<64>(1ull). In this examplethe resulting type would have an alignment of 64 bytes,and array of such types would have each element locatedin a different cache lines on most platforms. Silarly, onecould define a type that is aligned at virtual memorypage size boundaries of 4 KB.2.8.7. Cache slice hash functionThe library also provides the cache slice selection hashfunction as reverse engineered by Maurice, Le Scouarnec,Neumann et al. [28] for older Intel architectures.

3. Compilation suite and testing

The development criteria for ExOT included readability,support for cross-compilation, the Android platform, andtesting, reproducibility of builds, ease of instrumentationwith analysis tools. The tools that were consideredwere the Python-based SCons and Waf, Meson, Bazel,Buck, and CMake. CMake, Bazel and Meson were givenparticular attention because they allow creating toolchaindescriptions, useful when compiling for different targetsfrom different host machines. Although Bazel and Mesonseem to have a more consistent and friendlier syntax,CMake was chosen as the build system for the project.The primary motivations were more robust support forC and C++ projects, longer history, and official supportin the Android build system. Moreover, CMake does notbuild the software itself, but rather generates other buildsystems’ files, including Unix Makefiles, which can beused on different operating systems and in IntegratedDevelopment Environment (IDE). Compared with GNUMake, CMake provides better convenience, superiorcorrectness and easier scalability [33, p. 146, p. 262].

The library is structured into four directories:include/exot, src, vendor and test. The include/exot directorycontains folders with header files, which reflect thenamespace organisation described earlier. To use or adda piece of library code, a developer can include the files ina meaningful way, without caring about the relative path,e.g. <exot/utilities/barrier.h>, because the includedirectories are exported with the library target. Thesrc folder contains all source files used for buildingobject code. Vendor contains the external third-party

13

dependencies of the library. The need for the vendor folderunfortunately arises from the difficulty of importingexternal CMake-enabled code. The dependencies do nothave to be committed into the repository; they are set upas git submodules, and a call to git submodule init willpull the repositories and point to specific commits. Thisensures that only stable features of vendor libraries areused.

The library build targets are governed by the top-levelCMakeLists.txt file. It defines the project exot-libraryand the main build target, exot, which a static librarylibexot.a.The target is configured with a range ofcompile features and options, depending on the buildconfiguration type: Release, or Debug. The third-partylibraries, most of which include are CMake projects, areincluded in the top-level configuration. Thanks to that,all dependant targets will inherit the compiler and buildsettings from the main project. This avoids the situationin which an imported library, possibly from system paths,has been compiled using a different compiler, standardlibrary or ABI. Having an application linked to differentstandard libraries is strongly discouraged.

Moreover, there are custom targets to auto-formatcode to achieve uniform style, and to enable staticanalysis checks using clang-tidy. When enabled, thestatic analyser will be invoked during each build, anda tidy-all target will be created, which lints all librarycode. In the Debug configuration custom targets are alsoprovided with LLVM sanitisers enabled; for example,to compile the target exto with the memory sanitiser,one can use an auto-generated target exot-san-memory,or exot-san-thread for the thread sanitiser. Moreover,a Developer can use the provided CMake functiontarget_enable_sanitiser(target sanitiser) to enablethose for any target. To improve the library user’sexperience, some helpful messages are printed duringconfiguration. For example, the linked libraries andcompile flags are printed out in the console.

To use the library, a developer can use the library repos-itory in their sources, and importing the library target viathe add_subdirectory CMake function. The submodulemechanism described above can be used to import aspecific version of the library to produce applications.The developer can create separate configurations whichdo not ‘pollute’ the codebase, by running:

1 cmake -DCMAKE_TOOLCHAIN_FILE=path/to/toolchain.cmake \2 -DCMAKE_BUILD_TYPE=Release -B build/Release -H.3 cmake -DCMAKE_TOOLCHAIN_FILE=path/to/toolchain.cmake \4 -DCMAKE_BUILD_TYPE=Debug -B build/Debug -H.

All necessary build files and all build artefacts arecontained in the directories specified with the -B flag.Then, to build separate binaries for a target exot-target,the user can navigate to those folders and use the makeprogram, or from the project’s root run:

1 cmake --build build/Release --target exot-target2 cmake --build build/Debug --target exot-target

An important feature of the ExOT build processis the addition of toolchain files, which describethe essentials required for compilation. For example,x86_64-linux-clang-libcxx.cmake could define a Clangcompiler and use libc++ instead of stdlibc++ as thestandard library. The android.toolchain.cmake from theAndroid’s Native Development Kit can also be usedas a toolchain. Most of the executables produced usingthe compilation suite use the LLVM’s Clang compiler.It seems superior to GCC in terms of ease of cross-compilation [43, 17]6. To make the builds reproducible,the compilation suite and the application library use theNix package manager to isolate the build process fromthe system. Recently the Nix package manager has alsobeen recognised in the scientific community as a tool tofacilitate the reproducibility of experimental software[18].

For testing purposes the modern and fast testing suitedoctest is used in this project [8]. The build file alsoprovides an executable target exot-test, which creates atest runner and combines all individual test suites andtest cases from the test directory. The executable has arich calling interface, which allows listing available testsand running specific cases. The creation of unit tests isstill an ongoing process.

3.1. Cross-compilation and build reproducibility

The Docker-based compilation suite encapsulates all therequired software and toolchains necessary for compilingand cross-compiling the library and the applications builton top of it in a docker container. Regardless of whichhost machine is used the same compiled binaries willbe produced. The container can be used very easily byanyone familiar with a command line driven workflow.A special script is used to spawn the container, whichmounts the current directory on the host inside it andsets file permissions to the same user and group ID asthe calling user. This approach solves the common issuewith container-based workflows, where all commandsrun as the root user, resulting in produced files beinginaccessible on the host computer due to insufficientpermissions. Preliminary support has been providedfor using the environment via an Secure Shell (SSH)connection, for example in an IDE.

4. Android applications

As ExOT is designed to allow the integration of manydifferent devices into the measurement environment, weprovide utilities for Android devices. In this section, we

6The Android project also relies on LLVM’s Clang by default.

14

outline the different applications and wrappers that areimplemented as part of ExOT.

4.1. Intent proxy

We provide the intent proxy service, which acts as aninterface between the ExOT experiment engine and anyAndroid app. The intent proxy will scan the extras ofeach received intent it for keywords such as componentor action and will assemble a new intent based onthis information, which is forwarded to the applicationdefined in the received intent. The intent proxy alsoallows us to control the type and keyword of the extras,appended to the forwarded intent. Detailed informationon the intent structure can be found in the exot wiki.

As the adb intent interface is limited, the intent proxyallows us to sent all kinds of data to any applicationrunning on a smartphone. Therefore, ExOT allows tointegrate any application into a measurement setup,and control it via the intent interface of the respectiveapplication.

4.2. Application wrapper

Based on an Android NDK7 wrapper, we provide anAndroid service that allows to encapsulate functionalityimplemented in the ExOT application library to allowfor a fast measurement application development, seeFigure 6. In addition, this base service defines an intentinterface which allows it to be controlled by the ExOTexperiment engine out-of-the-box. The base service canalso be integrated into applications with a user interface,to allow interactive measurement campaigns.

4.3. Example apps

We provide two example applications, thermalsc andthermalscui to illustrate the functionality of the Androidintegration of ExOT. Both applications measure theutilisation, operating frequency, temperature and currentforeground application of the device. thermalsc isa background service that can be integrated into ameasurement setup using the ExOT experiment engine,while thermalscui provides a simple user interface foran interactive measurement campaign.

5. The experiment engine

In this section we present the experiment engine of ExOT,used for data processing and experiment orchestration.We base our data processing design on a layeredinformation flow model, illustrated in Figure 7. Similarto the well known OSI model, information travels fromthe highest layer to the lowest, and then up to the highestagain.

7https://developer.android.com/ndk/

Figure 6: Android integration workspace of ExOT. TheC++ application library is integrated as a submodule inthe directory “exot-c++”.

3 - Raw Data Processing4 - Line Coding5 - Source Coding6 - Generate/Verify

2 - I/O Module1 - Applications0 - System

output formattingsymbol to tracebits to symbolsgenerate input

write schedule filessystem interaction

raw data to tracetrace to symbolssymbols to bits

calculate metrics

read meas. filessystem observation

system interaction

Layer Name Layer Functions

Figure 7: Complete information flow model. Informationtravels from the highest to the lowest layer, is used tointeract with the system and travels up to the highestlayer.

Layer 6 describes how input data is generated and howmetrics are calculated from the measurement data. Inlayer 5 and 4, the source and line coding is defined, whichis used to compress and shape the data stream dependingon the channel specifications. Layer 3 describes the dataformat required by the applications, while layer 2 definesfile I/O. The two bottom layers describe the source(generator) and sink (meter or observer) applicationsand the system.

Layers 2 to 6 are implemented as Python packages inexperiment engine, which has the following advantages:(i) there is no need for recompilation when a new data

15

processing scheme is tested, (ii) the implementationis platform-independent, and (iii) data checks anddebugging are easy to perform. The different layers canbe combined arbitrarily, increasing the code reusabilityamong different evaluations. Furthermore, experimentscan use pass-through layers to skip processing stepsor do not use the complete information flow stack. Forexample, an experiment may only use the stack up tolayer 2 by taking static data and convert it to source andthe sink application input and configuration files.

5.1. Overview of the experiment engine

The simplified Unified Modelling Language (UML)class diagram in Figure 8 shows some of the majorrelationships in the experiment engine and hints at thefunctionality enabled in the major components. As before,an instance of an Experiment class is responsible for thegeneration of experiments, data and path management,and instantiation of processing layers.

It is driven by a configuration file, described inSection 5.2. The experiment instance can define multipleexperimental phases. Each phase holds a number ofruns. For example, a phase might include experimentalruns for a range of symbol rates. An experimental runis the building block of experiments, and encapsulatesa number of parameters that remain invariant. Theseruns are instances of a Run class, which is responsiblefor ingesting (processing the input bit stream throughthe encode parts of each of the layers) and digesting(processing the measurements from the target platformsthrough the decode parts of each of the layers) the data.They also define how the execution has to be carried outon a target platform.

The experiment engine is typically used as shownin Listing 10. First a config is loaded from a file of theTom’s Obvious, Minimal Language (TOML) format (line1). The PerformanceExperiment instance is created usingthat configuration (line 2). All required Run instancesare instantiated by calling the generate method (line 3).The run objects are arranged in a dictionary with a tree-like structure with experimental phases as first levelchildren. The tree is accessed via the phases property. Toperform the encode path processing, the digestmethodis called on all runs—leaves in the hierarchy accessedvia phases (lines 4-5). The experiment is serialised to diskwith the write method (line 6). Once it is available asfiles, it can be executed on a target platform of choice(line 7). The execution involves sending and fetchingdata as well as orchestrating the execution for eachrun. Once it completes, the data produced on the targetplatform is available locally. To read and analyse the data,we perform the decode path processing with the sametechnique as the encoding (lines 26-27). Ingesting thedata requires providing run-time configuration, stored

1 configuration = toml.load("./My\ config.toml")2 experiment =

PerformanceExperiment(config=configuration)↪→

3 experiment.generate()4 for run in

datapro.util.misc.leaves(experiment.phases):↪→

5 experiment.digest()6 experiment.write()7 experiment.execute_in_environment("My Environment")8 ingest_arguments = dict(9 lne={

10 "decision_device":sklearn.pipeline.make_pipeline(↪→

11 sklearn.preprocessing.StandardScaler(),12 sklearn.svm.LinearSVC(),13 )14 },15 io={16 "env": "My Environment",17 "rep": 0,18 "matcher": datapro.util.wrangle.Matcher(19 "medium",20 "type",21 ["variable"],22 list(range(0, 8)),23 ),24 },25 )26 for run in

datapro.util.misc.leaves(experiment.phases):↪→

27 experiment.ingest(**ingest_arguments)28 experiment.write()29 experiment.backup()

Listing 10: Typical workflow with the experimentalexperiment engine.

in the ingest_arguments variable in this example. Finally,we can serialise the results of our analysis with anothercall to write and upload an archived copy to a backupserver.

An instance of the experiment class might also containready to use analysis procedures, which are implementedin the channel class. Besides simple local data processing,such analysis procedures might include data processingwith machine learning libraries on computing clusters.

5.2. Configuration and interoperability

Configuration files for the experiment engine are writtenin TOML, as it has good support for the required key–value structure, understands a number of well-defineddata types (such as arrays, which before required complexstructures or fault-prone string splitting), and has goodreadability. Listing 11 shows how the TOML-basedconfiguration looks like. The structure can be effortlesslyconverted into other mapping or dictionary-like types,such as JSON or Python’s built-in dict.

16

Figure 8: UML diagram of some of the relationships between software components of the data processing andexperimental orchestration framework. 17

1 name = "Report demo"2 save_path = "data"3 backup_path = "data/_backup"4 experiment_exists_action = "move"5

6 [EXPERIMENT]7 type = "PerformanceExperiment"8 channel = "Cache"9

10 [EXPERIMENT.PHASES]11 train = {bit_count = 10000, symbol_rates = [1000,

10000], repetitions = 5}↪→

12 eval = {bit_count = 10000, symbol_rates = [1000,10000], repetitions = 5}↪→

13

14 [EXPERIMENT.LAYERS]15 src = {name = "Huffman", params = { length = 4 }}16 lne = {name = "GenericLineCoding", params = {

saturated = false, demean = false }}↪→

17 rdp = {name = "DirectActivation", params = {sync_pulse_duration = 0.5, sync_pulse_detection ="falling" }}

↪→

↪→

18 io = {name = "TimeValue", params ={output_timing_multiplier = 1e9,input_timing_multiplier = 1e9}}

↪→

↪→

19

20 [EXPERIMENT.GENERAL]21 latency = 1022 fan = true23 governors = "userspace"24 frequencies = "max"25 sampling_period = 3e-626

27 [APPS]28 snk = {executable = "meter_cache_fr", zone =

"insecure"}↪→

29 src = {executable = "generator_cache_read_st", zone ="insecure"}↪→

Listing 11: An excerpt of a the experiment engineconfiguration file.

5.2.1. Interface to applications

Listing 12 shows an example of an actual sourceapplication configuration. We can clearly see that passingthis number of parameters via command line argumentswould be inconvenient and prone to errors. Eachconfigurable component accesses its options in theJSON object specified by its name (as defined on line4 in Listing 7). In the experiment engine these JSONconfiguration objects are generated directly from valuesspecified in the experiment configuration file, as shown inListings 13 and 14. This allows the master configurationfile to encompass all necessary settings in a single file.

1 {2 "generator": {3 "cores": [0], "worker_policy": "round_robin",

"self_policy": "round_robin",↪→

4 "worker_priority": 98, "self_priority": 97,"use_busy_sleep": true,↪→

5 "busy_sleep_yield": true, "shm_file":"/dev/hugepages/8",↪→

6 "set_count": 2, "set_increment": 647 },8 "schedule_reader": {9 "input_file": null, "read_as_hex": false,

"reading_from_file": false↪→

10 },11 "logging": {12 "append_governor_to_files": false, "async": false,

"async_size": 8192,↪→

13 "log_level": "trace","provide_platform_identification": false,↪→

14 "debug_log_filename": null, "app_log_filename":null,↪→

15 "timestamp_files": true16 }17 }

Listing 12: New JSON-based configuration format

[ENVIRONMENTS."Environment name".src]generator.host_pinning = 3generator.should_pin_host = truegenerator.cores = [ 0 ]generator.should_pin_workers = truegenerator.host_policy = "round_robin"generator.host_priority = 97generator.worker_policy = "round_robin"generator.worker_priority = 98generator.use_busy_sleep = truegenerator.busy_sleep_yield = falsegenerator.use_huge_pages = truegenerator.shm_file = "/dev/hugepages/8"generator.set_count = 64generator.set_increment = 64

logging.debug_log_filename = ""logging.app_log_filename = ""logging.log_level = "info"logging.provide_platform_identification = falselogging.async = false

schedule_reader.input_file = ""schedule_reader.reading_from_file = trueschedule_reader.cpu_to_pin = 1

Listing 13: Source application config

5.3. Experiment engine implementation

In this subsection, we briefly explain the implementationstrategies applied to the experiment engine of ExOT.5.3.1. Python modularity, documentation and typing supportThis allows the experiment engine to be used in a moreconsistent manner and makes it easier to import and use

18

[ENVIRONMENTS."Environment name".snk]logging.append_governor_to_files = falselogging.async = truelogging.async_size = 4096logging.log_level = "debug"logging.provide_platform_identification = truelogging.timestamp_files = falselogging.rotating_logs = falselogging.rotating_logs_count = 10logging.rotating_logs_size = 104857600

meter.host_policy = "round_robin"meter.host_pinning = 7meter.should_pin_host = truemeter.host_priority = 95meter.log_header = truemeter.start_immediately = falsemeter.use_busy_sleep = truemeter.busy_sleep_yield = false

cache.use_huge_pages = truecache.shm_file = "/dev/hugepages/8"cache.set_count = 64cache.set_increment = 64

Listing 14: Sink application config

in an IDE or an interactive environment. ?? and shows anexample of the suggestions displayed by a text editor oran IDE when one tries to import a namespace or modulefrom the top-level datapro package.

Modern Python adds the support for type annotations,which provides well-defined function signatures andallows static type checkers, such as mypy8, to be used.Thanks to the type hints the intent of each function ismuch clearer to the user.

Supplements typing with documentation in thestandard “docstring” format conforming to PythonEnhancement Proposal (PEP) 2579. This style ofdocumentation is much easier to parse by tools suchas text editors, as shown in Figure 9 and Figure 10.

5.3.2. Defensive programming

Multiple value and type checks are used throughoutthe experiment engine, preventing the misuse of theprovided software and helps to ensure the correctness ofthe execution. If wrong values or types are supplied, theuser is notified in a quick and informative manner.

5.3.3. Mix-in classes

These classes are are lightweight classes that are meant tobe inherited from, but which provide limited and genericfunctionality. These improve universality, contain certaingeneric functionality in a single unit, and help reducecode duplication.

8http://mypy-lang.org9https://www.python.org/dev/peps/pep-0257/

5.3.4. Config-driven class instantiation

The experiment engine takes advantage of abstract baseclasses and provides a subclass tracker mix-in and ageneric object factory. The mix-ins are enabled by thesimpler customisation of class creation introduced inPEP 48710, and define custom __init_subclass__ classmethods. The subclass tracker traverses the methodresolution order and allows a base class to keep trackof derived classes. The generic object factory usesthe information provided by the subclass tracker andprovides an interface for creation of non-abstract subclassinstances. The code example below shows how little efforis required to provide a factory for a user-defined classhierarchy:

1 class Base(SubclassTracker,track="customisation_point",metaclass=abc.ABCMeta): pass

↪→

↪→

2 class Derived(Base,customisation_point=SomeEnum.SomeValue): pass↪→

3 class Factory(GenericFactory, klass=Base): pass4

5 instance = Factory()("Derived")

In this example, a base class inherits from SubclassTracker.The GenericFactory is provided the base class in its klassparameter. The derived class is readily available in thefactory, and can be created using its name. These factoriesare used in conjunction with the experiment config file.5.3.5. Dependency management

The experiment engine uses the pyproject.toml file that hasbeen proposed in PEP 51811, which is increasingly usedto specify build system requirements for Python projects.The control over the dependencies is achieved using thePoetry packaging and dependency manager12. Using asimple syntax (e.g. numpy = "^1.16") one can declare whichversion of a dependency needs to be installed. A lockfile, which contains all dependencies and their versions,is created and can be easily versioned. The control overthe Python version is exercised with the popular pyenvproject13.5.3.6. Enhances the inspection and control of platform

parameters

The experiment engine makes it possible to both inspectand set the state of the target platform. Thanks to theability to read platform settings, the experiment enginealso validates whether the provided values are availableor suitable. Moreover, the experiment engine has betterknowledge of the original state and can more easilyrestore it after the experiment is completed.

10https://www.python.org/dev/peps/pep-0487/11https://www.python.org/dev/peps/pep-0518/12https://poetry.eustace.io13https://github.com/pyenv/pyenv

19

Figure 9: Module importing suggestions

Figure 10: Function signature and documentation hints

6. Showcase

In this section, we present small examples of theusage of different ExOT components, that illustrate thecapabilities of the toolkit. For more extensive usage seeMiedl [1].

6.1. Application library

Before presenting an example for using the applicationlibrary, let us look at how an actual application iscomposed using the library. The applications are writtenin quite a declarative style, as the example in Listing 15shows. The typical order is as follows:

(2) Declare type aliases for the sake of convenience,with the help of the using statement. Aliasesare indispensable for class templates, variadic inparticular. In the shown example, one meter modulesis provided to the meter host component. Using asubtype contained in the alias meter_type, we thenprovide the right template parameter (time) to thelogger.

(4) Instantiate the command line wrapper which will initself instantiate the executor and spawn componentprocesses.A number of concrete applications were created

using the application library. They all share the samestructure as the one shown in Listing 15, and include a

1 #include <chrono>2 #include <exot/components/meter_host_logger.h>3 #include <exot/meters/thermal_msr.h>4 #include <exot/utilities/main.h>5

6 using namespace exot;7 using meter_t = components::meter_host_logger< (2)8 std::chrono::nanoseconds,

modules::thermal_msr>; (2.2)↪→

9

10 int main(int argc, char** argv) {11 return utilities::cli_wrapper<meter_t>(argc, argv);

(4.0)↪→

12 }

Listing 15: A sample application

source application, and various meter configurations14.An experimental data set was produced using themultithreaded utilisation generator, which was pinnedto cores 1, 3, 5, and 7 of a Lenovo T440p Laptop, basedon an Intel i7-4700MQ core.

Two examples of logged data are visualised inFigures 11 and 12. The former presents an excerptproduced with a meter that combines thermal and powerinformation accessed via model specific registers, and thelatter shows a combination of multiple available metermodules in a single log output. The red dashed verticallines are produced from the load generator’s applicationlog, and indicate the onset and end of high utilisationstates, respectively. The presented graphs have beenpost-processed with a 5-sample moving average filter, toimprove the clarity of the output15. As the graphs show,the change in measured states coincides with the changesin the input state trace. Moreover, Figure 12 shows howthe meter host (Section 2.7.6) could be used for a moreexploratory analysis, where individual modules can becross-referenced.

6.2. Experimental flow

As an example for the use of the experimental flow,we show a simple experiment with the thermal covertchannel shown by Bartolini, Miedl and Thiele [24].

Figure 13 shows examples of temperature tracesgathered from “Haswell” (Lenovo T440p) and “ARM”(Raspberry Pi 3) platforms, produced by execution tracesto transmit data with bit rates of 50 and 5 bits per second,respectively. The dotted lines show boundaries betweenthe line-coded symbols. Each contains a transition fromeither lower to higher or higher to lower temperature (formessage bits 0 and 1, respectively). Depending on thesymbol rate, each line-coded symbol can be represented

14In the future, a meter factory class may be provided to set up metermodules at runtime.

15In particular, the averaging was performed to alleviate the issuewith MSR access as described by Jón Thoroddsen [21].

20

Figure 11: Showcase of a typical experimental run

by a number of samples. The decoding of the input tracefollows the strategy described by the authors [24, sec. 7.1].A Gaussian naive-Bayes classifier operates on slices of theinput trace transformed into complex plane. The complexplane representation is produced with the help of a0°and 90°phase-shifted carrier signals. The slices are de-meaned to remove long-term temperature variations. Thecorresponding symbol spaces with annotated classifier’sdecision boundaries are shown in Figure 14 for the twoplatforms.

7. Future work

The following sections describe the possible future workclassified into functional enhancements, extensions andimprovements, and software engineering refinements.Future directions for the application library developmentinclude:Exploration Since a large part of the research workinvolves informal exploration of the side-effects ofexecution of programs, it might be helpful to providea more direct, perceptible indication of acquired metermeasurements. A pipeline component or a script couldbe provided to visualise the accumulating readings,using graphical or terminal-based output. Moreover,exploratory visualisation could be provided for theAndroid platform, possibly taking advantage of existingexample code which plots sensor data using OpenGLprimitives.New paradigms Implementing the task-based modelof execution would allow for a broader variety ofprogramming styles and allow for greater sharing ofresources among asynchronous components, which

Figure 12: A meter using multiple meter modulesFrom top to bottom, the displayed variables are: coretemperatures for cores 0, 2, 4, and 6; power in RAPLdomains PKG and PP0; scaling frequencies of cores 0 to7; thermal information from sysfs thermal zones 0 and1.

could be executed by a thread pool and avoid expensivethread context switches.Executors The executors in the current implementationare quite basic, and provide only a thin layer ofabstraction over lower-level system or user-space threads.Further possibilities exist, including loop executors andthread pool executors [see 29].Communication between components At the momentthe queues used for communication are concurrent,but there is likely room for improvement in terms ofperformance. Additional focus could be given to zero-copymechanisms in order to verify that the implemented moveoperations (or copy elision/return value optimisation)actually prevent copies being created. It also might be thecase that there are leaner synchronisation mechanismsthan the currently used combination of locks andcondition variables. Moreover, using a more hand-craftedstorage method (e.g., using a circular buffer) rather than

21

0 16 32 48 64 80 96 112Sample #

42

44

46

48

50

Core

tem

pera

ture

(℃)

0 16 32 48 64 80 96 112Sample #

44

46

48

50

52

Core

tem

pera

ture

(℃)

Figure 13: Example transmissions with the thermal channel for Haswell at 50bps (left) and ARM at 5bps (right).

−10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 7.5 10.0Re

−7.5

−5.0

−2.5

0.0

2.5

5.0

7.5

10.0

Im

0.500

SymbolSymbol 1Symbol 0Misclassified?False

−2 −1 0 1 2Re

−2

−1

0

1

2

Im

0.500

SymbolSymbol 0Symbol 1Misclassified?FalseTrue

Figure 14: Example symbol spaces of the decoded thermal channel transmissions for Haswell at 50bps (left) andARM at 5bps (right).

wrapping a queue class from the STL could deliver betterperformance, or provide bulk/stride access to queueelements.Implementing a control flow in addition to data flowmight also allow for better collaboration betweencomponents; for example, in the GNU Radio projectthe ability to exchange control messages between nodesallows the use of push/pull message passing semantics(instead of the push-only model of classical processnetworks).Deadlocks Currently deadlocks may only arise fromimproper ordering of functional code and state changesinside the process network nodes. In the case of apipeline arrangement of nodes there is little chance fordeadlocks, both global and local (due to insufficientqueue capacities). If the library was to be extendedwith multiple inputs and outputs, a proper deadlockdetection and resolution mechanism would need to beprovided. An observer could monitor and resize thequeues, but the process nodes might also need to be

instrumented with an execution state and thread-safeaccess to it. Successful mechanisms have been developedby Allen, Zucknick and Evans [36] and Geilen andBasten [38]. At the moment, the extended semanticsof try_{read,write}_{for,until} allow for handling oflocal deadlocks without any impact to token ordering,and may even prove sufficient for more complex usecases.Extended read/write interface The interface toqueues/channels that allows the “try” semantics shouldideally exist as an interface class, such that a user canextend the core framework by deriving from it andimplementing the member functions. However, thefunctions are template member functions, which cannotbe made virtual.16 Using a different mechanism thaninheritance might be necessary.Duplicated MSR access objects Each meter that needs

16“Member template functions cannot be declared virtual. Currentcompiler technology expects to be able to determine the size of a class’svirtual function table when the class is parsed.”[37, p. 242]

22

Figure 15: Left: Experiment Orchestration Toolkit(ExOT) logo; Right: QR-Code linking to the ExperimentOrchestration Toolkit (ExOT) website

access to model specific registers has its own instance ofthe MSR class, resulting in duplication of functionality andincreased count of file descriptors opened by the process.It might be a good idea to make the MSR a singleton class,and allow users to limit which registers they want toaccess.State semantics Improving the state management se-mantics, and holding the state in an enumeration insteadof atomic boolean variables.

Future directions for the extension of the experimentengine might include:Plotting and data visualisation The plotting and datavisualisation module of the experiment engine isvery rudimentary. Therefore, an extension of thisfunctionality using modern python plotting librarieswould be recommendable to improve the usability of theexperiment engine.Automated data backup and restore While an auto-mated backup and restore mechanism for experimentaldata was already foreseen in the initial design, it wasnever implemented. This mechanism should allow tobackup and restore experimental data to a (remote)location defined in the experiment configuration file.

8. Concluding Remarks

In this paper we presented the implementation detailsand underlying design decision for the ExperimentOrchestration Toolkit (ExOT). We gave a detailedoverview of the design patterns of the application library,the Android integration and the experiment engine ofExOT.

For further information, consult the ExOT website(exot.ethz.ch) or the ExOT wiki (https://gitlab.ethz.ch/tec/public/exot/wiki/-/wikis/home). ExOT was developedin the Computer Engineering Group, which is part ofthe Computer Engineering and Networks Laboratoryat ETH Zürich. All components of the ExOT project arepublicly available under the 3-clause BSD license.

Acknowledgements

Thanks to all students and colleagues at the ComputerEngineering Laboratory, which have used ExperimentOrchestration Toolkit (ExOT) in their projects and helpedto improve it by providing feedback. Moreover, thanks to

Lukas Sigrist and the RocketLogger Team for providingthe website template and giving advice during theprocess of open sourcing.

Last but not least, we also want to thank Azra Gradincicthe design of the ExOT logo shown in Figure 15.

References[1] P. Miedl, ‘Threat potential assessment of power

management related data leaks’, PhD thesis, ETH Zurich,2020.

[2] P. Miedl, R. Ahmed and L. Thiele, ‘We know what you’redoing! Application detection using thermal data’, LeibnizTransactions on Embedded Systems, vol. Special Issue onEmbedded System Security, no. 1, 2020, Under review.

[3] P. Miedl, B. Klopott and L. Thiele, ‘Increased repro-ducibility and comparability of data leak evaluationsusing ExOT’, in 2020 Design, Automation & Test in EuropeConference & Exhibition (DATE), IEEE, 2020. doi: 10.3929/ethz-b-000377986. [Online]. Available: https://doi.org/10.3929/ethz-b-000377986.

[4] A. Fitsios, ‘Towards Task Inference on Mobile Systemsbased on Thermal Traces’, ETH Zürich, Gloriastrasse35, 8092 Zürich, Switzerland, Tech. Rep. 1, Mar. 2019,Semester Thesis; Supervisors: Philipp Miedl, RehanAhmed and Lothar Thiele.

[5] B. Klopott, ‘How bad are data leaks really?’, Supervisors:Philipp Miedl and Lothar Thiele, Master’s thesis, ETHZürich, Gloriastrasse 35, 8092 Zürich, Switzerland, Jun.2019.

[6] N. Lohmann. (2019). nlohmann/json, [Online]. Available:https://github.com/nlohmann/json.

[7] C. Barth, ‘Come again? Towards repeatable securityexperiments’, ETH Zürich, Gloriastrasse 35, 8092 Zürich,Switzerland, Tech. Rep. 1, Aug. 2018, Semester Thesis;Supervisors: Philipp Miedl and Lothar Thiele.

[8] V. Kirilov, ‘onqtam/doctest’, 2018. [Online]. Available:https://github.com/onqtam/doctest.

[9] B. Klopott, ‘You also want to explore other securityleaks? Building an easily extendable application libraryfor security leak research’, ETH Zürich, Gloriastrasse35, 8092 Zürich, Switzerland, Tech. Rep. 1, Aug. 2018,Semester Thesis; Supervisors: Philipp Miedl and LotharThiele.

[10] A. Mackintosh, C++ Latches and Barriers, ISO/IEC JTC1SC22 WG21, May 2018.

[11] M. Meier, ‘Feature Extraction from Thermal Tracesfor the Thermal Fingerprinting Attack’, ETH Zürich,Gloriastrasse 35, 8092 Zürich, Switzerland, Tech. Rep. 1,May 2018, Semester Thesis; Supervisors: Philipp Miedl,Rehan Ahmed and Lothar Thiele.

23

[12] G. Melman, ‘spdlog’, 2018. [Online]. Available: https://github.com/gabime/spdlog.

[13] P. Miedl, X. He, M. Meyer, D. B. Bartolini and L. Thiele,‘Frequency Scaling as a Security Threat on MulticoreSystems’, IEEE Transactions on Computer-Aided Design ofIntegrated Circuits and Systems, vol. 37, no. 11, pp. 2497–2508, Nov. 2018, issn: 1937-4151. doi: 10.1109/TCAD.2018.2857038. [Online]. Available: https://doi.org/10.1109/TCAD.2018.2857038.

[14] P. Miedl and L. Thiele, ‘The Security Risks of PowerMeasurements in Multicores’, in Proceedings of the 33rdAnnual ACM Symposium on Applied Computing, ser. SAC’18, Pau, France: Association for Computing Machinery,2018, pp. 1585–1592, isbn: 978-1-45035-191-1. doi: 10 .1145/3167132.3167301. [Online]. Available: https://doi.org/10.1145/3167132.3167301.

[15] M. Millen, ‘Analysis and Optimization of FrequencyGovernors’, Supervisors: Rehan Ahmed, Philipp Miedland Lothar Thiele, Master’s thesis, ETH Zürich,Gloriastrasse 35, 8092 Zürich, Switzerland, Apr. 2018.

[16] ‘muellan/clipp’, 2018. [Online]. Available: https://github.com/muellan/clipp.

[17] P. Smith, ‘How to cross compile with LLVM based tools’,in FOSDEM’18, Linaro, Feb. 2018, pp. 1–28. [Online].Available: https : / / fosdem.org /2018 / schedule /event /crosscompile/.

[18] B. Bzeznik, O. Henriot, V. Reis, O. Richard and L. Tavard,‘Nix as HPC package management system’, in HUST2017, Nov. 2017, pp. 1–24.

[19] C++ Technical Committee, ISO/IEC 14882:2017, 2017.[Online]. Available: http://www.eel.is/c++draft/.

[20] X. He, ‘A Smart Attack using the Frequency CovertChannel’, Supervisors: Philipp Miedl, Matthias Meyerand Lothar Thiele, Master’s thesis, ETH Zürich,Gloriastrasse 35, 8092 Zürich, Switzerland, Oct. 2017.

[21] Ólafur Jón Thoroddsen, ‘UnCovert 4: The Power CovertChannel’, ETH Zürich, Gloriastrasse 35, 8092 Zürich,Switzerland, Tech. Rep. 1, Jun. 2017, Semester Thesis;Supervisors: Philipp Miedl and Lothar Thiele.

[22] M. Selber, ‘UnCovert3: Covert Channel Attacks onCommerical Multicore Systems’, Supervisors: PhilippMiedl and Lothar Thiele, Master’s thesis, ETH Zürich,Gloriastrasse 35, 8092 Zürich, Switzerland, Apr. 2017.

[23] R. Strebel, ‘What is my Thermal Fingerprint?’, ETHZürich, Gloriastrasse 35, 8092 Zürich, Switzerland, Tech.Rep. 1, Jul. 2017, Semester Thesis; Supervisors: PhilippMiedl, Rehan Ahmed and Lothar Thiele.

[24] D. B. Bartolini, P. Miedl and L. Thiele, ‘On the Capacity ofThermal Covert Channels in Multicores’, in Proceedingsof the Eleventh European Conference on Computer Systems,ser. EuroSys ’16, London, United Kingdom: ACM, 2016,24:1–24:16, isbn: 978-1-45034-240-7. doi: 10.1145/2901318.2901322. [Online]. Available: http://doi.acm.org/10.1145/2901318.2901322.

[25] M. Selber, ‘UnCovert: Operating Frequency, a SecurityLeak?’, ETH Zürich, Gloriastrasse 35, 8092 Zürich,Switzerland, Tech. Rep. 1, Feb. 2016, Semester Thesis;Supervisors: Philipp Miedl and Lothar Thiele.

[26] P. Wild, ‘UnCovert: Evaluating thermal covert channelson Android systems’, ETH Zürich, Gloriastrasse 35, 8092Zürich, Switzerland, Tech. Rep. 1, Aug. 2016, SemesterThesis; Supervisors: Philipp Miedl and Lothar Thiele.

[27] J. C. Beard, P. Li and R. D. Chamberlain, ‘RaftLib’, inthe Sixth International Workshop, New York, New York,USA: ACM Press, 2015, pp. 96–105, isbn: 978-1-45033-404-4. doi: 10.1145/2712386.2712400. [Online]. Available:http://dl.acm.org/citation.cfm?doid=2712386.2712400.

[28] C. Maurice, N. Le Scouarnec, C. Neumann, O. Heenand A. Francillon, ‘Reverse Engineering Intel Last-Level Cache Complex Addressing Using PerformanceCounters.’, English, RAID, vol. 9404, no. Chapter 3,pp. 48–65, 2015. doi: 10 .1007 /978- 3- 319- 26362- 5_3.[Online]. Available: http://link.springer.com/10.1007/978-3-319-26362-5_3.

[29] C. Mysen, Executors and schedulers, ISO/IEC JTC1 SC22WG21, Apr. 2015.

[30] M. Torquati, Parallel Programming Using FastFlow,September 2015, University of Pisa, Sep. 2015.

[31] A. Kukanov, V. Polin and M. J. Voss, ‘Flow Graphs,Speculative Locks, and Task Arenas in Intel® ThreadingBuilding Blocks’, Intel Corporation, Tech. Rep., Jun. 2014.

[32] G. E. Allen, ‘Computational process networks’, English,PhD thesis, University of Texas at Austin, Austin, May2011. [Online]. Available: https://repositories.lib.utexas.edu/handle/2152/ETD-UT-2011-05-2987.

[33] P. Smith, Software Build Systems, ser. Principles andExperience. Addison-Wesley, 2011.

[34] A. Podobas, M. Brorsson and K.-F. Faxén, ‘A Comparisonof some recent Task-based Parallel ProgrammingModels’, English, in 3rd Workshop on ProgrammabilityIssues for Multi-Core Computers, 2010. [Online]. Available:http://urn.kb.se/resolve?urn=urn:nbn:se:ri:diva-23671.

[35] Z. Vrba, ‘Implementation and performance aspects ofKahn process networks’, PhD thesis, Jul. 2009. [Online].Available: https://dblp.org/rec/phd/basesearch/Vrba09.

[36] G. E. Allen, P. E. Zucknick and B. L. Evans, ‘A DistributedDeadlock Detection and Resolution Algorithm forProcess Networks’, in 2007 IEEE International Conferenceon Acoustics, Speech, and Signal Processing, IEEE, 2007,pp. II–33–II–36, isbn: XXX-1-4244-0727-3. doi: 10.1109/ICASSP . 2007 . 366165. [Online]. Available: http : / /ieeexplore.ieee.org/document/4217338/.

[37] B. Eckel and C. D. Allison, Thinking in C++, English,ser. Practical Programming. Prentice Hall, Dec. 2003,vol. 2, isbn: XXX-0-13-035313-2. [Online]. Available: http://www.cs.ust.hk/~dekai/library/ECKEL_Bruce/.

24

[38] M. Geilen and T. Basten, ‘Requirements on the Executionof Kahn Process Networks.’, ESOP, vol. 2618, 2003. doi:10.1007/3- 540- 36575- 3_22. [Online]. Available: http://link.springer.com/10.1007/3-540-36575-3_22.

[39] H. Van Der Linden, ‘Scheduling distributed Kahn processnetworks in Yapi’, PhD thesis, Technische UniversiteitEindhoven, 2003.

[40] A. Alexandrescu, Modern C++ Design, ser. GenericProgramming and Design Patterns Applied. AddisonWesley, Feb. 2001, isbn: 978-0-20170-431-5. [Online].Available: http://www.worldcat.org/title/c- in-depth/oclc/316330731.

[41] M. Goel, ‘Process Networks in Ptolemy II’, PhD thesis,Berkeley, CA, Dec. 1998.

[42] T. M. Parks, ‘Bounded Scheduling of Process Networks’,PhD thesis, Dec. 1995. [Online]. Available: http://ptolemy.eecs.berkeley.edu/papers/parksThesis.

[43] J. Roelofs, ‘Which targets does Clang support?’, inEuroLLVM 2014, pp. 1–15. [Online]. Available: https ://llvm.org/devmtg/2014-04/.

25