Screening for Factors Affecting Application Performance in ... · profiling tools, a test driver for Android, an R package for finding confidence intervals, and a Ruby “gem”

NIST Technical Note 1855

Screening for factors affecting application performance in profiling

measurements

David Flater

http://dx.doi.org/10.6028/NIST.TN.1855

http://dx.doi.org/10.6028/NIST.TN.1855sspringeTypewritten TextThis publication is available free of charge from:

sspringeTypewritten Text







Screening for factors affecting application performance in profiling

measurements

David Flater Software and Systems Division

Information Technology Laboratory

http://dx.doi.org/10.6028/NIST.TN.1855

October 2014

U.S. Department of Commerce Penny Pritzker, Secretary of Commerce

National Institute of Standards and Technology Willie E. May, Acting Under Secretary of Commerce for Standards and Technology and Acting Director

http://dx.doi.org/10.6028/NIST.TN.1855sspringeTypewritten Text

sspringeTypewritten TextThis publication is available free of charge from:


Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately.

Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the

entities, materials, or equipment are necessarily the best available for the purpose.

National Institute of Standards and Technology Technical Note 1855 Natl. Inst. Stand. Technol. Tech. Note 1855, 31 pages (October 2014)

http://dx.doi.org/10.6028/NIST.TN.1855 CODEN: NTNOEF

http://dx.doi.org/10.6028/NIST.TN.1855sspringeTypewritten Text




sspringeTypewritten TextThis publication is available free of charge from:







1

1


Screening for factors affecting application performance in profiling measurements

David Flater

October 2014

Abstract

This report provides an example application of screening techniques in experimental computer science, including validation and selection of metrics and measures, the screening experiment itself, and supporting statistical methods. Together with related references, it is intended to encourage other computer scientists to make more use of established statistical methods in performance evaluations.

The described screening experiment sought to identify which of three factors (environment, frame pointers, and profiler options) affected the performance of the application under test. The results showed that they all had a measurable impact, but the shapes and ranges of the distributions of measurement results would likely make their impacts immaterial in experiments using smaller samples. The validation phase of this experiment yielded evidence that the ill-behaved distributions are attributable to actual performance variations rather than measurement error.

Introduction

The mission of the Software Performance project is to strengthen the scientific foundations of software performance measurement (“metrology for information technology”). Its goals are:

• Application: Replace unreliable common practices with rigorously-studied methods grounded in design of experiments.

• Research: Solve measurement challenges created by the evolution and increased complexity of commodity hardware.

• Transfer: Feed improved methods back into the community of practice.

Previous work on this pro ject has included a case study of performance degradation attributable to one form of software hardening [1], a validation of profiling tools1 for C and C++ applications under 64-bit Linux [2], and a treatise on estimation of uncertainty in application profiles [3]. A test suite for validating application profiling tools, a test driver for Android, an R package for finding confidence intervals, and a Ruby “gem” for managing experimental data are available as associated software products [4, 5, 6].

The research plan brings us now to the topic of screening for factors affecting application performance in profiling measurements. “Screening” is a specialized type of experimental design. The primary purpose of a screening design is to separate significant factors from insignificant ones [7].

We conducted a screening experiment with the goal of identifying which of three factors (environment, frame pointers,2 and profiler options) affects the performance of an application being profiled. However, as with any experiment, it was important to validate the infrastructure before proceeding to full-scale data collection. Thus, there were two phases: validation and selection of metrics and measures, followed by the main experiment. This report provides details on both phases.

1Profiling tools are the instruments used to measure software performance at the function and application levels. 2Frame pointers are an optional feature/protocol of applications that is used by some profiling options.

2 NIST Technical Note 1855

The remainder of this report is assembled as follows. Section 2 introduces terms used throughout the report. Section 3 covers validation and selection of metrics and measures. Section 4 covers the screening experiment itself. Finally, Section 5 ends with conclusions and future work.

2 Terms

The terms factor and independent variable are synonymous. They refer to a property of the system under test that is deliberately changed within the scope of the experiment. In some communities these are called input variables.

The different values to which a factor is set in an experiment are called levels of that factor. Usually, the levels of factors are mapped onto the same numerical scale to simplify discussion and data management. In this report we use the scale of 0,1 for the “low” and “high” values of 2-levelled factors and integers 0–N for factors with more than 2 levels.

A treatment is a specific combination of factor levels. If there are two factors, each of which has two levels, then there are four possible treatments.

A quantity is an objective property (e.g., of a software artifact or program execution) that has a magnitude that can be expressed as a numeric value with a reference (e.g., an SI unit) that defines what the numeric value indicates. When results are quantitative in nature but not necessarily expressing the magnitude of an objective property, the weaker term metric is used.

A dependent variable is a quantity or metric that is used to determine the effects of different treatments. In some communities these are called response variables or output variables.

The term measures is used variously to refer to the instruments, methods, and units that are candidates for use in obtaining and stating the values of quantities and metrics.

Finally, a controlled variable is any property that could in principle change, but that has been deliberately kept constant within the scope of the experiment.

Any terms that remain unclear should be interpreted in light of the International Vocabulary of Metrology (VIM) [8], the design of experiments literature, the statistics literature, and the terms’ ordinary dictionary definitions.

3 Validation and selection of metrics and measures

3.1 Environment

The hardware used throughout the experiment was a Dell Precision T5400 with dual quad-core Xeon X5450 CPUs and 4 GiB of DDR2-667D ECC RAM.

Testing was conducted under two versions of 64-bit Slackware with replacement kernels. The two environments, hereafter known as os 0 and os 1, are identified in Table 1.

All data were collected while running in single-user mode to reduce interference. The kernel’s HZ value was set to 1000 because the resolution of CPU time results from bash’s time command is limited by this configurable. The value 0 was written to /proc/sys/kernel/perf_cpu_time_max_percent on every boot to disable a new (since 3.11) kernel feature that interfered with data collection.

SpeedStep was disabled in the BIOS configuration so that the CPUs would operate at a fixed frequency.

C and C++ test programs were built using GCC version 4.6.4 at optimization level –O2. GCC 4.6.4 was used instead of a current version because GCC 4.7 introduced a feature that invalidated the assumptions of frame pointer profiling in the 64-bit mode of the x86 architecture (x86 64) [9] and this will become important


Factor Description Level 0 Level 1 os Environment Slackware 14.0 Slackware 14.1

Kernel 3.12.6 Kernel 3.14.3

Table 1: Identification of two operating environments.

in the screening experiment. The test suite originated in [2] was used to regression-test and validate this latest release in the 4.6 series.

3.2 Workloads

The programs used to create CPU-intensive workloads were:

• A single-threaded C++ program, aliquot, that calculates aliquot sequences [10] using a simple but inefficient algorithm.

• A Ruby script, aliquot-par.rb, that distributes the work of the simple algorithm over multiple CPU cores.

• An invocation of the core utility program dd that reads data from /dev/urandom, a computationally expensive pseudorandom number source that is implemented in kernel space.

• A single-threaded C program, 10sec-user, that executes a busy-waiting loop polling the system function gettimeofday until the timer readings indicate that at least 10 seconds have elapsed. gettimeofday is implemented in user space [11].

Workloads 0 through 4 as used in the experiment were as follows:

0. Run aliquot to calculate the 154-term aliquot sequence beginning with 107 .

1. Id. except with the nosmp kernel parameter and dd if=/dev/urandom of=/dev/null running continuously as a competing load.

231 − 12. Run aliquot-par.rb to verify that 230 · is a “perfect number” (an aliquot cycle of length 1) using all 8 CPU cores.

3. Run dd to copy 5 blocks of size 10 MiB from /dev/urandom to /dev/null.

4. Run 10sec-user.

3.3 Ground truth

Even if all of the measures tested proved to be consistent with one another, there would remain the question of whether every possible mechanism on the computer ultimately traced to a clock which could, for example, run 10 % fast or slow without drawing attention.

The purpose of 10sec-user is to realize a quantity close to 10 seconds of elapsed time and incidentally somewhat close to 10 seconds of CPU time (if there is no competing load) to serve as a reference or étalon for the measures being tested. Regardless of variations in the speed at which the polling loop is executed, the program terminates when the clock indicates that 10 s have elapsed.

Still, it remained necessary to validate that 10sec-user realizes the desired quantity. As a primitive validation, the elapsed execution time of 10sec-user was measured 10 times using a digital wristwatch and a script that simply printed “Start” and “Stop” before and after the program ran. Given the uncertainty contributed by human reaction time and gained proficiency in activating the buttons on the wristwatch, the consistency of the results was impressive:

10.07 9.99 9.94 10.00 10.01 10.00 10.01 10.01 10.01 9.99

http:aliquot-par.rbhttp:aliquot-par.rb


Although such a small sample proves nothing with regard to outliers, it does provide some independent if imprecise validation that 10sec-user realizes a quantity of 10 seconds elapsed time. Unfortunately, being entirely a function of internal state, CPU time is not independently observable.

3.4 Metrics

Two (or essentially three, as described below) different instruments were employed to collect a variety of related metrics. Bash is the default command-line shell used by Linux. Perf [12] is a high-powered profiling toolset that is included in the Linux kernel source tree.

For the purposes of this discussion, the names of the metrics have been prefixed by “bash” or “perf ” to identify the source.

3.4.1 bash time

Metrics: bash-elapsed, bash-user, bash-sys.

The bash shell has a built-in time command that provides one measure of elapsed time and two measures of CPU time. The two measures of CPU time indicate “user” time and “system” time respectively. Results are reported to a maximum numerical precision of 0.001 s; however, the resolution of the CPU time metrics is limited by the kernel’s compiled-in HZ configurable, which we set to be 1000 Hz.

3.4.2 perf stat

Metrics: perf-elapsed, perf-cpu-cycles, perf-cpu-clock, perf-task-clock.

perf stat provides elapsed time to 9 decimal places as well as the final values of a user-configurable subset of counters. The supported counters include hardware counters implemented by the CPU and chipset, software counters implemented by the kernel, and tracepoint statistics. For this experiment we chose the counters cpu-cycles, cpu-clock and task-clock.

perf-cpu-cycles is a count of CPU cycles that traces to a hardware counter. If the CPU frequency is fixed (as ours was), the CPU cycle counts can be converted to seconds by dividing by the CPU frequency.

perf-cpu-clock and perf-task-clock are both software counters with results provided in milliseconds.

The practical differences among these counters being unclear, it was a secondary goal of the validation to find which ones would be most appropriate for subsequent uses.

3.4.3 perf record

Metrics: perf-samples.

While perf stat reads counters to provide totals for an entire program execution, perf record uses such counters to drive sampling-based profiling. These distinct functions cannot simultaneously be applied to a given program execution unless one nests the perf invocations.

Although perf record is normally used for function-level profiling, the reports generated include a sample count total that can be used as a measure of the whole program. Without nested measurements, it is the only such measure that is available after a perf record profiling run.

The sample counts can be converted to seconds by dividing by the sampling frequency. However, perf record entails periodic interruption of the workload, so measurements of time that are obtained this way may differ from results that are obtained without such interruptions (e.g., by perf stat).


3.5 Data collection for metric validation

1000 samples were collected for each combination of metric, workload, and os. Since different tools provide different metrics, it was necessary to run the program three times to obtain one sample for each of the metrics without nesting the tool invocations: once for each of bash, perf stat, and perf record.

perf-cpu-cycles was converted to seconds by dividing by the CPU frequency, 2.9925 GHz. The perf record sample counts were converted to seconds by dividing by the sampling frequency, 106 cycles = 2992.5 Hz.

3.6 Results and analysis

Figure 1 shows the means of the validation data for the different metrics with 95 % confidence intervals. Confidence intervals were determined using the bias-corrected and accelerated (BCa) bootstrap method [13, 14]. The numbers of bootstrap replications were determined using the adaptive method described in [15, §7.9.4] to achieve estimated bootstrap simulation precisions that are below the screen resolutions of the plots. The two or three plots pertaining to a given workload are drawn to the same vertical scale albeit with different locations on the Y axis; each workload, however, is scaled independently due to the significant range differences among them.

Figure 2 and Figure 3 visualize the entirety of the collected data with hybrid violin-scatter plots. From these figures one can see that the tails of the distributions are sparsely populated but quite long. Direct examination of the time series shows that the outliers are not always solitary but sometimes cluster around low-frequency secondary modes such as in the example of Figure 4. Distributions like this were previously observed in [3, §4].

3.7 Face validity

The data for the first three workloads include high outliers for most of the metrics. These outliers are typical of what we have seen previously in performance measurements [1, 3] and are not readily dismissed as invalid results.

On the other hand, the data for Workload 2 in os=1 include extreme low outliers for perf-cpu-cycles, perfcpu-clock, perf-task-clock, and perf-samples that are implausible. The perf-elapsed results from affected invocations of perf stat were close to the mean. We speculate that the parallel execution in Workload 2 occasionally triggered some kind of concurrency-related fault in the perf subsystem that prevented most of the time from being accounted for. The absence of this anomaly in os=0 is consistent with the theory that a fault was introduced between one kernel version and the other.

The fact that the elapsed time measures for Workload 2 are less than the CPU time measures is not an anomaly; it simply means that the parallel workload averaged a CPU utilization greater than one by using more than one CPU core.

3.8 Analysis

Our findings for the validation phase are as follows:

1. bash-elapsed and perf-elapsed differ under some workloads but are equally usable measures of elapsed time.

2. perf-cpu-cycles, perf-cpu-clock, perf-task-clock, and perf-samples all reflect the total CPU time used by the application. This experiment failed to illuminate any material differences that exist among the first three. The impact of the different measurement method used for perf-samples is apparent in its divergence from the others.


●

●

●

●

●

●

●

●

●

●

●

● ●

●

xx

bash.elapsedbash.user

perf.elapsedperf.cpu.cycles

perf.cpu.clockperf.task.clock

perf.samples

7.20

7.30

Mea

n (s

), W

orkl

oad

0

●●

xx

bash.sys

0.00

0.10

Mea

n (s

), W

orkl

oad

0

●

●

●

●

xx

bash.elapsedperf.elapsed

14.8

814

.94

15.0

0

Mea

n (s

), W

orkl

oad

1

●

●

●

●

●

●

●

●

●

●

xx

bash.userperf.cpu.cycles


perf.samples

7.42

7.48

7.54

Mea

n (s

), W

orkl

oad

1

●●

xx

bash.sys

0.00

0.06

0.12

Mea

n (s

), W

orkl

oad

1

●

●

●

●

xx


1520

25

Mea

n (s

), W

orkl

oad

2

●

●

●

●

●

●

●

●

●

●

xx

bash.userperf.cpu.cycles


perf.samples

150

160

Mea

n (s

), W

orkl

oad

2

●●

xx

bash.sys

05

1015

Mea

n (s

), W

orkl

oad

2

●

●

●

●

●

●

●

●

●

●

●

●

●

●

xx

bash.elapsedbash.sys



perf.samples

3.38

3.42

3.46

Mea

n (s

), W

orkl

oad

3

●●

xx

bash.user

0.00

0.04

0.08

Mea

n (s

), W

orkl

oad

3

●●●●

●● ●● ●● ●●

●●




perf.samples

9.98

09.

990

10.0

00

Mea

n (s

), W

orkl

oad

4

●●

bash.sys0.0

000.

010

0.02

0

Mea

n (s

), W

orkl

oad

4

Figure 1: Means of 1000 samples for each combination of metric, workload, and os (os 0=blue, os 1=red) with 95 % BCa confidence intervals. A horizontal line is drawn at the perf-cpu-cycles mean to show the close agreement of perf-cpu-cycles, perf-cpu-clock, and perf-task-clock. Intervals are not drawn when the range is vanishingly small.


Index

●●●●●●●●●●●

●●●●●●●●●

●●

●

●●●●●●●●●●●●●●●●

●

●●●●

●

●●

●●●

●

●

●

●

●●●●

●

●

●

●●●●●●●●●●●●●●●●●

●

●●●●●●

●●●

●

●

●

●●●●●●●

●

●

●

●●●●●

●●●●●●

●

●●●●

●

●●●●●

●

●

●

●

●●●

●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●●●

●●●●●

●

●

●

●●●

●

●●

●●●●●●●●●●

●

●●●●●●●●●●

●●●●●●●

●

●●●●●●

●

●●●●

●●●

●●●●●●●●

●●●●●●●

●●●

●●

●

●●●

●●●●

●

●●●

●

●●●●●●

●●●●●●●●●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●

●

●●●●●●

●●●●●●●

●

●●●

●●●●●●●●●●●

●

●●●●

●

●

●

●●

●

●●

●

●●●●●●

●●●●●

●

●●●

●

●●●●

●

●●●

●

●

●

●●●

●

●●●●●●

●

●●●●

●

●●●●

●

●●●●

●

●●●●●●●●

●

●●●

●●

●

●●●●●●●●●●●

●

●●●●

●●●●●●

●

●

●●

●

●

●

●●●

●●●

●●●●●●●●●●●

●●●

●

●

●

●●●●

●

●●●●●●●

●

●●●●●●●●●

●

●●●●

●●●●●●●

●●●

●

●●

●●

●

●●●●●●●●●●●

●●●●●

●●●

●●

●

●

●●●●●●●●●●●●●●

●●●

●

●●

●

●●●●

●

●●●

●●

●

●●●

●

●

●●●●●●

●

●●●●●●●●

●●

●●●

●

●●●●

●●●●

●●

●

●●

●●

●●●●●●●●●●●●●●●●

●●●

●

●

●

●●●

●

●

●●

●

●●

●

●●

●

●

●●

●

●●●

●

●●

●

●

●

●●

●

●●●

●

●●●

●

●

●

●●●

●

●●●●●●●

●●

●●●●●●

●

●●●●●●●●●

●

●●●●●

●

●●●●●●●●●●

●

●●

●

●

●

●●●●

●

●●●

●

●

●

●●●●●●●●●

●●

●●●●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●●●●●●●●●

●

●●

●

●

●●●●

●●●●●●●●●

●

●●●●

●

●

●

●●●●

●

●●●

●

●

●

●●●●●●●

●

●●●●●●●

●

●●●●●●●

●

●

●●●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●●●●●●●

●

●●●●●

●●

●●●●●●●●

●

●

●

●

●●●

●

●●●●●●

●

●●

●

●

●

●

●

●●●●●

●

●●

●

●●

●

●●●●●●●●●●●●

●●●

●

●

●

●●●●

●●●●●●

●

●●●●●

●●●●

●

●●

●

●

●

●●●●

●

●●●

●

●●

●●●

●●

●●●●●●

●

●●●●●●●●

●

●●●●●

●

●●●●●

●●

●

●

●●●●●●●

●

●●●●● ●●●●●●●●●●

●

●●●●●●●●●

●●

●

●●●●●●●●●●●●●●●●

●

●●●●

●

●●

●●●

●

●

●

●

●●●●

●

●

●

●●●●●●●●●●●●●●●●●

●

●●●●●●

●●●

●

●

●

●●●●●●●

●

●

●

●●●●

●

●●●●●●

●

●●●●

●

●●●●●

●

●

●

●

●●●

●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●●●

●●●●●

●

●

●

●●●

●

●●

●●●●●●●●●●

●

●●●●●●●●●●

●●●●●●●

●

●●●●●●

●

●●●●

●●●

●●●●●●●●

●●●●●●●

●●●

●●

●

●●●

●●●●

●

●●●

●

●●●●●●

●

●●●●●●●●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●

●

●●●●●●

●●●●●●●

●

●●●

●●●●●●●●●●●

●

●●●●

●

●

●

●●

●

●●

●

●●●●●●

●

●●●●

●

●●●

●

●●●●

●

●●●

●

●

●

●●●

●

●●●●●●

●

●●●●

●

●●●●

●

●●●●

●

●●●●●●●●

●

●●●

●●

●

●●●●●●●●●●●

●

●●●●

●●●●●●

●

●

●●

●

●

●

●●●

●●●

●●●●●●●●●●●

●●●

●

●

●

●●●●

●

●●●●●●●

●

●●●●●●●●●

●

●●●●

●●●●●●●

●●●

●

●●

●●

●

●●●●●●●●●●●

●●●●●

●●●

●●

●

●

●●●●●●●●●●●●●●

●●●

●

●●

●

●●●●

●

●●●

●●

●

●●●

●

●

●●●●●●

●

●●●●●●●●

●●

●●●

●

●●●●

●●●●

●●

●

●●

●●

●●●●●●●●●●●●●●●●

●●●

●

●

●

●●●

●

●

●●

●

●●

●

●●

●

●

●●

●

●●●

●

●●

●

●

●

●●

●

●●●

●

●●●

●

●

●

●●●

●

●●●●●●●

●●

●●●●●●

●

●●●●●●●●●

●

●●●●●

●

●●●●●●●●●●

●

●●

●

●

●

●●●●

●

●●●

●

●

●

●●●●●●●●●

●●

●●●●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●●●●●●●●●

●

●●

●

●

●●●●

●●●●●●●●●

●

●●●●

●

●

●

●●●●

●

●●●

●

●

●

●●●●●●●

●

●●●●●●●

●

●●●●●●●

●

●

●●●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●●●●●●●

●

●●●●●

●●

●●●●●●●●

●

●

●

●

●●●

●

●●●●●●

●

●●

●

●

●

●

●

●●●●●

●

●●●

●●

●

●●●●●●●●●●●●

●●●

●

●

●

●●●●

●●●●●●

●

●●●●●

●●●●

●

●●

●

●

●

●●●●

●

●●●

●

●●

●●●

●

●

●●●●●●

●

●●●●●●●●

●

●●●●●

●

●●●●●

●●

●

●

●●●●●●●

●

●●●●● ●●●●

●

●●●

●

●●●

●

●

●●●●●

●●

●●●●●●●

●

●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●

●

●●●●●●●●●●

●

●

●

●●●

●

●●●●●●●

●

●

●●●●

●

●●●●●

●●●●

●

●●

●●

●●

●

●●

●

●●

●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●

●●

●

●●

●

●●●

●

●●●●●●●●

●●●●●●●

●

●●●●

●

●

●

●●●●●

●

●

●

●●●●●●

●

●●●●

●

●●●●

●●●●

●●●

●

●

●

●●

●

●●●●●●

●

●●

●●

●●●●

●

●

●●

●●●

●●

●●

●●

●●

●

●●

●

●●●●●●●

●●●

●

●●

●

●●

●

●●●●

●●●●

●

●

●●●

●

●●●●●●●●

●

●●

●

●●●●●●●●

●

●

●●●●●●●●●●●●

●

●

●●●●●●●

●

●●●●●●

●

●

●

●●●●●

●

●●

●

●●●●●●●●●

●

●●

●●●

●

●●●

●

●

●

●●

●●

●

●

●●●●●●●●●

●

●●●●●

●

●●●●●

●●●●●

●

●

●●●

●●●

●●●●●

●●

●●●●

●

●

●●●

●●●●

●

●

●●

●

●●●

●●

●

●●●●●●●

●●

●

●

●

●●●

●

●●

●

●

●●●●●●●

●●

●●

●

●●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●

●

●●●●●

●

●

●

●●

●

●●●●●●●●●

●

●

●

●●

●●●●●●●

●

●●●

●

●●

●

●●●●●●●●

●

●●●●●

●●

●●●

●

●●

●

●

●●

●●●●

●●●●●

●

●

●●

●

●

●

●●●●

●●●●●●●

●●

●●●●

●

●

●●●●●●●●●●

●

●●●

●

●

●

●●●●

●

●

●

●●●

●

●

●●●●●

●

●

●

●●●●

●

●

●●

●●●

●

●●●●●

●

●●

●

●●●●

●

●

●●

●

●●●●●●●

●

●●●●●●●●●

●●●●●●●●●●●●

●

●

●●

●

●●●

●

●●

●●●

●●

●

●●●

●

●●●●

●

●●●●

●●●●●●

●

●●●

●●●●●●●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●

●●

●

●●

●●

●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●●●●●●●●●

●

●●●

●

●●

●●

●

●

●●●●●

●

●●●●●●

●

●●●●●

●

●●●

●

●

●

●●●

●●

●●●●●

●

●●●●●●●●

●●

●●●

●●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●●●●●

●●

●●●

●

●●●●●●●●●●●

●

●●●●●

●

●

●●●●●●●●

●

●●●●

●●

●

●●●●●

●

●●●

●

●●●●●

●

●

●

●●●●●

●

●●

●

●●●

●

●●●●

●

●●●

●

●●●

●

●

●●●●●

●●

●●●●●●●

●

●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●

●

●●●●●●●●●●

●

●

●

●●●

●

●●●●●●●

●

●

●●●●

●

●●●●●

●●●●

●

●●

●●

●●

●

●●

●

●●

●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●

●●

●

●●

●

●●●

●

●●●●●●●●

●●●●●●●

●

●●●●

●

●

●

●●●●●

●

●

●

●●●●●●

●

●●●●

●

●●●●

●●●●

●●●

●

●

●

●●

●

●●●●●●

●

●●

●●

●●●●

●

●

●●

●●●

●●

●●

●●

●●

●

●●

●

●●●●●●●

●●●

●

●●

●

●●

●

●●●●

●●●●

●

●

●●●

●

●●●●●●●●

●

●●

●

●●●●●●●●

●

●

●●●●●●●●●●●●

●

●

●●●●●●●

●

●●●●●●

●

●

●

●●●●●

●

●●

●

●●●●●●●●●

●

●●

●●●

●

●●●

●

●

●

●●

●●

●

●

●●●●●●●●●

●

●●●●●

●

●●●●●

●●●●●

●

●

●●●

●●●

●●●●●

●●

●●●●

●

●

●●●

●●●●

●

●

●●

●

●●●

●●

●

●●●●●●●

●●

●

●

●

●●●

●

●●

●

●

●●●●●●●

●●

●●

●

●●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●

●

●●●●●

●

●

●

●●

●

●●●●●●●●●

●

●

●

●●

●●●●●●●

●

●●●

●

●●

●

●●●●●●●●

●

●●●●●

●●

●●●

●

●●

●

●

●●

●●●●

●●●●●

●

●

●●

●

●

●

●●●●

●●●●●●●

●●

●●●●

●

●●●●●●●●●●●

●

●●●

●

●

●

●●●●

●

●

●

●●●

●

●

●●●●●

●

●

●

●●●●

●

●

●●

●●●

●

●●●●●

●

●●

●

●●●●

●

●

●●

●

●●●●●●●

●

●●●●●●●●●

●●●●●●●●●●●●

●

●

●●

●

●●●

●

●●

●●●

●●

●

●●●

●

●●●●

●

●●●●

●●●●●●

●

●●●

●●●●●●●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●

●●

●

●●

●●

●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●●●●●●●●●

●

●●●

●

●●

●●

●

●

●●●●●

●

●●●●●●

●

●●●●●

●

●●●

●

●

●

●●●

●●

●●●●●

●

●●●●●●●●

●●

●●●

●●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●●●●●

●●

●●●

●

●●●●●●●●●●●

●

●●●●●

●

●

●●●●●●●●

●

●●●●

●●

●

●●●●●

●

●●●

●

●●●●●

●

●

●

●●●●●

●

●●

●

●●●

●

●●●●

●

●●●

●

●●●

●

●

●●●●●

●●

●●●●●●●

●

●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●

●

●●●●●●●●●●

●

●

●

●●●

●

●●●●●●●

●

●

●●●●

●

●●●●●

●●●●

●

●●

●●

●●

●

●●

●

●●

●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●

●●

●

●●

●

●●●

●

●●●●●●●●

●●●●●●●

●

●●●●

●

●

●

●●●●●

●

●

●

●●●●●●

●

●●●●

●

●●●●

●●●●

●●●

●

●

●

●●

●

●●●●●●

●

●●

●●

●●●●

●

●

●●

●●●

●●

●●

●●

●●

●

●●

●

●●●●●●●

●●●

●

●●

●

●●

●

●●●●

●●●●

●

●

●●●

●

●●●●●●●●

●

●●

●

●●●●●●●●

●

●

●●●●●●●●●●●●

●

●

●●●●●●●

●

●●●●●●

●

●

●

●●●●●

●

●●

●

●●●●●●●●●

●

●●

●●●

●

●●●

●

●

●

●●

●●

●

●

●●●●●●●●●

●

●●●●●

●

●●●●●

●●●●●

●

●

●●●

●●●

●●●●●

●●

●●●●

●

●

●●●

●●●●

●

●

●●

●

●●●

●●

●

●●●●●●●

●●

●

●

●

●●●

●

●●

●

●

●●●●●●●

●●

●●

●

●●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●

●

●●●●●

●

●

●

●●

●

●●●●●●●●●

●

●

●

●●

●●●●●●●

●

●●●

●

●●

●

●●●●●●●●

●

●●●●●

●●

●●●

●

●●

●

●

●●

●●●●

●●●●●

●

●

●●

●

●

●

●●●●

●●●●●●●

●●

●●●●

●

●●●●●●●●●●●

●

●●●

●

●

●

●●●●

●

●

●

●●●

●

●

●●●●●

●

●

●

●●●●

●

●

●●

●●●

●

●●●●●

●

●●

●

●●●●

●

●

●●

●

●●●●●●●

●

●●●●●●●●●

●●●●●●●●●●●●

●

●

●●

●

●●●

●

●●

●●●

●●

●

●●●

●

●●●●

●

●●●●

●●●●●●

●

●●●

●●●●●●●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●

●●

●

●●

●●

●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●●●●●●●●●

●

●●●

●

●●

●●

●

●

●●●●●

●

●●●●●●

●

●●●●●

●

●●●

●

●

●

●●●

●●

●●●●●

●

●●●●●●●●

●●

●●●

●●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●●●●●

●●

●●●

●

●●●●●●●●●●●

●

●●●●●

●

●

●●●●●●●●

●

●●●●

●●

●

●●●●●

●

●●●

●

●●●●●

●

●

●

●●●●●

●

●●

●

●●●

●

●●●●

●

●●●

●

●●●

●

●

●●●●●

●●

●●●●●●●

●

●●●●●●●

●●●●●

●

●●●●●●

●

●●●●

●

●

●

●●●●●●●●●●

●

●

●

●●●

●

●●●●●●●

●

●

●●●●

●

●●●●●

●●●●

●

●●

●●

●●

●

●●

●

●●

●●

●

●

●

●●

●

●

●

●●●●●●●●●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●

●●

●

●●

●

●●●

●

●●●●●●●●

●●●●●●●

●

●●●●

●

●

●

●●●●●

●

●

●

●●●●●●

●

●●●●

●

●●●●

●●●●

●●●

●

●

●

●●

●

●●●●●●

●

●●

●●

●●●●

●

●

●●

●●●

●●

●●

●●

●●

●

●●

●

●●●●●●●

●●●

●

●●

●

●●

●

●●●●

●●●●

●

●

●●●

●

●●●●●●●●

●

●●

●

●●●●●●●●

●

●

●●●●●●●●●●●●

●

●

●●●●●●●

●

●●●●●●

●

●

●

●●●●●

●

●●

●

●●●●●●●●●

●

●●

●●●

●

●●●

●

●

●

●●

●●

●

●

●●●●●●●●●

●

●●●●●

●

●●●●●

●●●●●

●

●

●●●

●●●

●●●●●

●●

●●●●

●

●

●●●

●●●●

●

●

●●

●

●●●

●●

●

●●●●●●●

●●

●

●

●

●●●

●

●●

●

●

●●●●●●●

●●

●●

●

●●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●

●

●●●●●

●

●

●

●●

●

●●●●●●●●●

●

●

●

●●

●●●●●●●

●

●●●

●

●●

●

●●●●●●●●

●

●●●●●

●●

●●●

●

●●

●

●

●●

●●●●

●●●●●

●

●

●●

●

●

●

●●●●

●●●●●●●

●●

●●●●

●

●●●●●●●●●●●

●

●●●

●

●

●

●●●●

●

●

●

●●●

●

●

●●●●●

●

●

●

●●●●

●

●

●●

●●●

●

●●●●●

●

●●

●

●●●●

●

●

●●

●

●●●●●●●

●

●●●●●●●●●

●●●●●●●●●●●●

●

●

●●

●

●●●

●

●●

●●●

●●

●

●●●

●

●●●●

●

●●●●

●●●●●●

●

●●●

●●●●●●●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●

●●

●

●●

●●

●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●●●●●●●●●

●

●●●

●

●●

●●

●

●

●●●●●

●

●●●●●●

●

●●●●●

●

●●●

●

●

●

●●●

●●

●●●●●

●

●●●●●●●●

●●

●●●

●●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●●●●●

●●

●●●

●

●●●●●●●●●●●

●

●●●●●

●

●

●●●●●●●●

●

●●●●

●●

●

●●●●●

●

●●●

●

●●●●●

●

●

●

●●●●●

●

●●

●

●●●

●

●●

●

●

●●

●

●●●●●

●

●●

●

●

●●●●●●●●●●●●

●

●●●●●●●●●

●●●●●●●●

●●

●●●

●●

●

●

●●●

●

●●●

●

●●●●●

●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●

●

●●●●

●

●●

●

●●

●

●●●●●●●●●●●●●●●●●

●

●

●

●

●●●●

●

●

●

●●●●

●

●

●●

●●

●

●

●

●●●●●●●●●●●●●●

●

●●

●

●●●

●

●

●

●●●●

●●●●●●●

●

●

●●●●

●

●

●

●●●●●●

●

●●

●

●

●

●●●●

●

●

●●●●

●●

●

●

●

●

●

●●●●●●●●●●●●●

●

●

●

●●●●●●●●●●●

●

●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●

●

●●●

●

●●●

●●

●

●●●●●

●●

●●●●

●●●●●

●●

●

●●●●●●●●●●●●

●●●

●

●●●●●

●●●●●●

●

●●●●

●

●●●●●●●

●

●●

●

●●●●●●●●●

●

●

●

●●●

●

●

●

●●●●●●●●

●

●●●●●●●●●●●●●●●

●

●●●●●

●●●●

●

●●●

●

●

●●●●

●

●●●●●●

●

●●●●

●

●

●●

●

●

●

●●●

●

●●●●

●

●

●●●●●

●

●●

●●

●●●●●●●●●●

●●

●

●●●●

●●

●

●●●●●

●●

●●●●●●●●

●

●●●●

●

●●●●●●●●●●●●●●●●●●

●

●

●

●●●●●

●●●●

●●

●

●●●

●

●●●

●

●●●●●●

●●●

●

●●●●●●●●

●

●

●●●●●●

●

●

●

●●●●●

●●●●●●●●●●●●●●●●

●

●

●●●

●●

●

●●●●●●●

●

●●●

●

●

●

●●

●

●●●

●

●●

●

●●●●●

●●●●●

●●

●●●●●

●

●●

●

●●●●●●●●

●

●●●●

●

●●

●

●

●

●●●●●●●●●●●●●●

●●●●●●●●●

●

●

●●

●●●●

●●

●●●●●●●

●●●●

●

●●●●

●

●

●

●

●

●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●

●●●●●●●●

●

●●●●●●

●

●●●●●●●

●

●●

●

●

●

●●

●●

●

●●●●●●●●●

●

●●●●

●

●●

●●●●●●●●

●●●●

●●●●

●

●●●●

●●

●●●

●

●

●●

●

●●●●●●●●

●

●●●

●

●

●

●●●

●●

●

●●●●●●●

●

●

●

●

●

●●●●●●●

●

●●

●●●●

●

●●●●●●

●●

●●

●●●

●

●

●●●●●

●

●●

●●●●●

●

●

●

●●

●

●●

●●

●

●●

●

●●

●

●

●●●●

●

●●●●●●

●

●●●●●

●●

●

●●●

●

●

●

●●

●

●●●●●●●●

●

●

●

●

●●●●●●●●●●●

●

●

●




perf.samples

7.0

8.0

9.0

Wor

kloa

d 0

(s)

Index

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

bash.sys

0.0

1.0

2.0

Wor

kloa

d 0

(s)

Index

●●●●●●●

●

●●●●●

●●●●

●●●●

●

●●●●●

●●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●●●

●

●●●

●

●●

●

●●●●

●

●●●●

●

●●●●

●

●●●●●●●●●●●●●

●

●●●●●●●●●

●

●●●●●●●●

●

●

●●●●●

●

●●●

●●

●●●

●

●●●●

●

●●●●●●●●●●●●●●●●●

●●●●●●●●

●

●●●●●●●●●●●

●

●

●

●●

●

●●●●●●●●●

●

●●

●●

●

●

●

●●●

●

●●●●●●

●

●

●●●

●●

●

●●●●●●●●●●

●

●

●

●●●●

●

●

●●●●●●●

●

●●●●●

●

●

●

●●●●●●●●●

●

●

●●●●●●

●●

●

●

●●

●

●●●●●●●

●

●

●●

●

●

●

●

●

●●●●●●

●

●●●●●●●

●

●●

●

●●●●●

●

●●●●

●●

●●●

●

●●●

●

●●●●●●●●●●

●

●●●●●●●●

●

●●●●●

●●

●

●●●●●●●●●

●

●●

●●●●●●●

●●●●●●●●

●

●●●●●●

●

●●●●●

●

●

●●●●●

●

●●

●●●●●●●

●

●●●●

●

●●

●●●

●

●●●

●●

●●

●

●

●●●●●●●●●●●●

●

●●●●●●●●●

●

●

●●●●●●●

●●●●

●●

●●●

●

●●●

●

●●●●

●

●

●●●

●

●

●

●●●●●

●

●●●●●●●●●●●

●●

●●●●●

●

●●●●

●●●

●

●●●●●●●●●

●

●●●●●●

●

●●●●

●

●●●●●●●●●●

●

●●●●

●●

●●

●

●●

●

●●

●●●

●

●

●

●●

●

●●●

●●●

●

●●●●●●

●

●

●●●

●●●●●

●

●●●●●●●●●●

●●●●●●

●

●

●

●●●●●●●●

●

●

●

●●●●●●●

●

●●●

●

●●

●●●

●

●

●

●

●

●●●●●●●

●

●●●●

●●

●

●●●

●●

●●●

●

●

●●●●●●●

●

●●●●●●●

●

●●

●

●●●●●●●

●

●●●●●●●

●●●●●●●●●

●●

●

●●●●●

●

●

●

●

●

●●

●

●●

●

●

●

●●●●●●

●●

●

●●●●●●●●●●●●

●

●●●

●●

●

●

●●●●●●●●●●

●●●

●●●●●●●●

●

●

●●

●●●●●●●●

●

●●●●●●●●●

●

●●●●●●●●●

●

●

●

●

●

●

●

●

●●●●●●●●

●

●●

●

●

●●

●

●●●●●●●●●

●

●●●

●

●

●●

●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●

●

●●●●●●●●●●●●●

●

●●●●●

●

●

●●

●

●

●●●

●●

●●●●●

●●●●

●●

●●●●●

●●●●●

●

●

●

●●

●●●●

●

●

●

●

●

●

●●●●●●

●

●●●

●

●●

●

●

●

●●

●●

●●●

●

●

●●●●●●●●●●●●●●●

●

● ●●

●

●●

●

●●●●●●

●●

●

●●

●●

●

●●●●

●●

●●●●●●●

●

●●●●●

●

●●

●

●

●

●●●

●

●

●●●

●

●

●

●●

●

●●

●●●

●

●

●●●●●●

●

●●●●●

●

●●●●●

●

●

●

●●●

●

●

●

●●

●●●

●●

●●●

●●●

●●●

●

●●

●

●

●

●●

●

●●●●●●●●●

●

●●●●●●●●

●

●

●●

●●

●●

●●●●●●

●

●

●

●●

●

●

●

●●●

●

●●●●

●

●●●●●●●●

●

●

●

●●●

●●●●●

●

●

●

●

●

●●

●●●●●●●●

●

●

●

●●●●●●●

●

●

●

●

●

●●

●

●●

●●●●●●

●

●●●●●

●

●●●●●●●

●●●

●

●

●

●

●●●●●

●

●●●●●●●●●●

●

●●●●●●●

●

●

●

●●●

●●●●●●

●

●

●

●

●

●●●●

●

●●

●

●

●●●●●●●●●●●●●

●●

●

●

●●●●●●●●●●●●●●

●

●

●

●●●

●

●●

●

●●●●●●●

●

●●●●

●

●

●●

●●●●

●●

●

●●●●●●●●●●●●

●

●●●●

●●

●●●●

●

●●

●

●●

●●

●●

●

●

●

●●●

●

●●

●

●●

●

●●●●●

●

●●●●●●

●

●●●●●

●

●●●●

●

●

●

●●●●

●●●

●

●●●●●

●

●●●

●●●●

●

●●●●●●

●

●●●●●●

●●●●●●●

●●

●●●●●

●●●●●

●

●

●

●●

●

●●●●●●●

●●

●●●

●

●

●

●

●●●●●●●●

●

●●

●

●●●●●

●

●●●●●●●●●●●

●

●

●●●●●

●

●●●●●

●●●●

●

●●●●●

●

●●●

●

●●

●

●●●●●●

●●●

●●●●●●●●●●

●

●

●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●

●

●

●

●●

●●●

●

●●

●

●

●●

●●●●●●●●●●●●

●

●

●●

●●

●●●

●●●●

●

●●●●●●●●●●●●●●●

●

●●●

●

●●●●

●

●●●●●

●●●●●●

●

●

●

●

●●

●

●

●

●●●

●

●●●●●●●●●●●

●

●●●

●

●●●●●●●

●●

●

●

●

●

●●

●●

●●

●

●●●●

●

●

●●●●

●

●●●●●●●●●●●

●

●●

●

●

●

●

●

●●●

●

●

●●●●●●●●●●●

●

●●●

●

●●

●

●●●●●●●●●

●

●

●●

●●●

●●●●●●●●

●

●●●●

●

●

●●●●●●●

●

●●●●●●

●

●●●●●●

●

●●●●●●

●

●●●●●●●●●

●

●●●●

●

●●

●

●●

●

●●●●●●●●●●●●●●●●●●●

●

●

●

●●●●●●●

●

●●●●●●●●●●●●●

●

●●●●●●●●●

●

●●●●●

●

●●●●

●

●●

●

●●●

●●

●●●●●

●●●●●●

●

●●●●●●

●

●

●

●●●

●●

●

●

●

●

●●●●

●

●●

●

●

●

●

●●●●●●●●


1517

19

Wor

kloa

d 1

(s)

Index

●●●●●●●●

●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●

●

●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●

●

●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●

●

●●●●●●●●●●●●●

●●

●●

●

●

●

●●●

●

●●●●●●●

●

●●●

●●●●●●●●●●●●●

●

●

●

●●●●

●●●●●●●●●

●

●●●●●

●

●

●

●●●●●●●●●

●

●●●●●●●●●●

●●●

●

●●●●●●●●

●

●●

●●●

●

●●●●●●●

●

●●●●●●●●●●

●

●●●●●

●

●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●

●●●●●●●●●●●●●●●●●●

●

●●●●●●

●

●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●

●●

●

●

●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●

●

●●●●●

●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●

●

●●●●●●●●●●

●

●●●●●●●●

●●●

●

●●●●●

●

●

●●●

●

●●●●●●●

●●●●●●

●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●

●

●●●●●●●●

●

●●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●

●●●●●●

●●●●●●

●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●

●

●

●

●●●

●

●

●●●●●●●●●●●●

●

●●●●●●●●●●●●

●

●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●

●

●●●●●●●●●●●●●●●●●●●

●

●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●●●●●●●

●

●

●●●

●●●●●●

●

●●●●●●

●

●

●

●●

●●

●●●

●

●

●●●●●●●●●●●●●●●●● ●●

●

●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●●●●

●

●●●●●

●

●●●●

●

●●●

●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●●●

●

●●●●●

●

●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●

●●

●●●●●●

●

●●●●●

●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●●

●●

●

●●●●●●●●

●

●●●●●

●

●●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●

●

●●●●●●●

●

●●

●●●

●●●●●●

●

●

●●

●

●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●

●●●●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●

●

●●●●●●●

●●●●

●

●●●

●

●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●

●

●

●

●●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●●●●●●●●●

●

●●●●●●●●●●●●●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●●●

●●●●●●●●●●●●●●●●●●●●

●

●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●●●

●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●

●●●●

●

●●●

●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●●

●●●●●●●●

●

●●●●●●

●●●●

●

●●●●●●●●●●●●

●●●●

●●

●

●●●●

●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●

●●

●●●●●●●●●●●

●

●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●

●

●●

●

●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●

●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●

●●

●

●●●●●●●● ●●

●

●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●●●●

●

●●●●●

●

●●●●

●

●●●

●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●●●

●

●●●●●

●

●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●

●●

●●●●●●

●

●●●●●

●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●●

●●

●

●●●●●●●●

●

●●●●●

●

●●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●

●

●●●●●●●

●

●●

●●●

●●●●●●

●

●

●●

●

●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●

●●●●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●

●

●●●●●●●

●●●●

●

●●●

●

●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●

●

●

●

●●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●●●●●●●●●

●

●●●●●●●●●●●●●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●●●

●●●●●●●●●●●●●●●●●●●●

●

●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●●●

●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●

●●●●

●

●●●

●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●●

●●●●●●●●

●

●●●●●●

●●●●

●

●●●●●●●●●●●●

●●●●

●●

●

●●●●

●

●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●

●●

●●●●●●●●●●●

●

●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●

●

●●

●

●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●

●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●

●

●●

●

●●●●●●●● ●●

●

●●●●●●●●●●●

●

●●●●

●

●●●●●●●●●●●●●

●

●●●●●

●

●●●●

●

●●●

●●●●●

●

●

●

●●

●

●●●●●●

●

●●●●●●●●●●●●

●

●●●●●

●

●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●●●

●●

●●●●●●

●

●●●●●

●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●●●●●●●

●

●

●●●

●●

●

●●●●●●●●

●

●●●●●

●

●●●●●●●●●●●

●

●

●●●●●●●●●●●●●●●●●

●

●●●●●●●

●

●●

●●●

●●●●●●

●

●

●●

●

●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●

●●●●●●●

●

●●●●

●

●●●●●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●

●

●●●●●●●

●●●●

●

●●●

●

●●●●●

●

●●●●●●●●●●●●

●

●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●

Documents

Screening for Factors Affecting Application Performance in ... · profiling tools, a test driver for Android, an R package for finding confidence intervals, and a Ruby “gem”