188
STABILIZER Charlie Curtsinger and Emery D. Berger Statistically Sound Performance Evaluation

STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

  • Upload
    others

  • View
    13

  • Download
    1

Embed Size (px)

Citation preview

Page 1: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER

Charlie Curtsinger and Emery D. Berger

Statistically SoundPerformance Evaluation

Page 2: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

We all care about performance evaluation

STABILIZER

Page 3: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

We all care about performance evaluationWe’ve been doing it wrong

STABILIZER

Page 4: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

We all care about performance evaluationWe’ve been doing it wrong

STABILIZER

Repeated runs and error bars are not enough

Page 5: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

We all care about performance evaluationWe’ve been doing it wrong

STABILIZER

Repeated runs and error bars are not enough

We’re not measuring what we thought

Page 6: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERWe’ve been doing it wrong

Page 7: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER

Memory layout affects performance

We’ve been doing it wrong

Page 8: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

We’ve been doing it wrong

Page 9: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

no way to measure effect of change in isolation

We’ve been doing it wrong

Page 10: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

STABILIZER eliminates the effect of layout

no way to measure effect of change in isolation

We’ve been doing it wrong

Page 11: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

STABILIZER eliminates the effect of layout

no way to measure effect of change in isolation

enables sound performance evaluation

We’ve been doing it wrong

Page 12: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

STABILIZER eliminates the effect of layout

no way to measure effect of change in isolation

Case Studies

enables sound performance evaluation

We’ve been doing it wrong

Page 13: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

changing a program will change its layout

STABILIZER

Memory layout affects performance

STABILIZER eliminates the effect of layout

no way to measure effect of change in isolation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

enables sound performance evaluation

We’ve been doing it wrong

Page 14: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERWe’ve been doing it wrong

changing a program will change its layoutMemory layout affects performance

STABILIZER eliminates the effect of layout

no way to measure effect of change in isolation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

enables sound performance evaluation

Page 15: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A

Unsound performance evaluation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 16: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

A

Unsound performance evaluation

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 17: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

A

Unsound performance evaluation

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 18: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

A

Unsound performance evaluation

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 19: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

A

Unsound performance evaluation

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

Page 20: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

A

Unsound performance evaluation

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

Page 21: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

A

Unsound performance evaluation

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

Page 22: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′A

Unsound performance evaluation

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; size_t meaning_of_life=42; for (size_t i = 0; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } for (size_t i = 16; i < size; i += 32) { asm("icbi 0,%0" : : "r"(p)); p += 32; } asm("isync");}

Page 23: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

A′A ×1 ×1

Which is faster?

Page 24: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

Page 25: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

Page 26: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

Page 27: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

Page 28: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

2.8% faster

Page 29: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.00

0.25

0.50

0.75

1.00

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

what about variance?

2.8% faster

Page 30: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0

5

10

15

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Which is faster?

A′A ×30 ×30

Page 31: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0

5

10

15

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

Page 32: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0

5

10

15

85.0 87.5 90.0 92.5 95.0Time (s)

Num

ber o

f run

s

VersionAA'

Is faster than ? A′ A

still2.8% faster

Page 33: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

Page 34: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 35: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Was it the code change?

Page 36: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Was it the code change?

Page 37: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Or was it the new layout?

Page 38: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Or was it the new layout?

Page 39: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Why is faster than ? A′ A

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Mytkowicz et al. (ASPLOS’09)Layout biases measurement

Page 40: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Page 41: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Link OrderChanges function addresses

Page 42: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Link Order

Environment Variable Size

Changes function addresses

Moves the program stack

Page 43: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Link Order

Environment Variable Size

Changes function addresses

Moves the program stack

Larger than the impact of -O3

Page 44: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cache

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

A

Page 45: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

map to same cache set

Page 46: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

conflict

map to same cache set

Page 47: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 48: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

A′

Page 49: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

map to same set

A′

Page 50: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

map to same set

A′

Nothing here

Page 51: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

map to same set

A′

Nothing here

no conflict

Page 52: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

Page 53: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

or branch predictor

Page 54: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

or branch predictor

or TLB

Page 55: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

or branch predictor

or TLB

or branch target predictor

Page 56: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

Blame the cacheA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

or branch predictor

or TLB

or prefetcher

or branch target predictor

Page 57: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Blame the hashA

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; }}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

int main(int argc, char **argv) { topFrame = (void**)__builtin_frame_address(0); setHandler(Trap::TrapSignal, onTrap); setHandler(SIGALRM, onTimer); setHandler(SIGSEGV, onFault); for(Function* f: functions) { f->setTrap(); } setTimer(interval); int r = stabilizer_main(argc, argv); return r;}

void setTimer(int msec) { struct itimerval timer;

timer.it_value.tv_sec = (msec - msec % 1000) / 1000; timer.it_value.tv_usec = 1000 * (msec % 1000); timer.it_interval.tv_sec = 0; timer.it_interval.tv_usec = 0; setitimer(ITIMER_REAL, &timer, 0);}

static void flush_icache(void* begin, size_t size) { uintptr_t p = (uintptr_t)begin & ~15UL; for (size_t i = 0; i < size; i += 16) { asm("icbi 0,%0" : : "r"(p)); p += 16; } asm("isync");}

DataHeapType* getDataHeap() { static char buf[sizeof(DataHeapType)]; static DataHeapType* _theDataHeap = new (buf) DataHeapType; return _theDataHeap;}

A′

Page 58: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ? A′ A

Page 59: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

Is faster than ? A′ A

Let’s do a poll

Page 60: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster it’s faster

it’s faster

it’s faster

Is faster than ? A′ A

Let’s do a poll

Page 61: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Do we trust this?

it’s faster it’s faster

it’s faster

it’s faster

Page 62: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

But it’s faster

Is faster than ? A′ A

Page 63: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster it’s slower they’re the same

But it’s faster

Is faster than ? A′ A

Page 64: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster it’s slower they’re the same

But it ran faster!

Page 65: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster it’s slower they’re the same

But it ran faster!What if we only talk to Bob?

Page 66: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

it’s slower they’re the same

Page 67: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

Page 68: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

Upgrade libc

Page 69: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

Upgrade libcChanges layout

Page 70: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

Change Username

Page 71: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

it’s faster

But it ran faster!What if we only use this layout?

Change UsernameChanges layout

Page 72: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout is Brittle

it’s faster

But it ran faster!What if we only use this layout?

Run in a new directory

Page 73: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout is Brittle

it’s faster

But it ran faster!What if we only use this layout?

Run in a new directory

Changes layout

Page 74: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout is Brittle

But it ran faster!What if we only use this layout?

Page 75: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout is Brittle

But it ran faster!What if we only use this layout?

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Page 76: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout is Brittle

But it ran faster!What if we only use this layout?

Layout biases measurementMytkowicz et al. (ASPLOS’09)

Can we eliminate the effect of layout?

Page 77: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

But it ran faster!What if we only use this layout?

Layout biases measurement

Can we eliminate the effect of layout?

YES

Page 78: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layoutenables sound performance evaluation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

makes performance evaluation difficult

Page 79: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layoutenables sound performance evaluation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

makes performance evaluation difficult

Page 80: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurement

STABILIZERrandomizes layout

Page 81: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurement

STABILIZER

function addresses

randomizes layout

Page 82: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurement

STABILIZER

function addresses stack frame sizes

randomizes layout

Page 83: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Layout biases measurement

STABILIZER

function addresses stack frame sizesheap allocations

randomizes layout

Page 84: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layout

function addresses stack frame sizesheap allocations

repeatedly

Layout biases measurement

Page 85: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layout

function addresses stack frame sizesheap allocations

repeatedly

Layout biases measurement

during execution

Page 86: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layout

function addresses stack frame sizesheap allocations

repeatedly

Layout biases measurement

Page 87: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layout

function addresses stack frame sizesheap allocations

repeatedly

Layout biases measurementa completely random layout

cannot bias results

Page 88: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′A ×30 ×30

Sound Performance Evaluation

Page 89: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A ×30 ×30

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Sound Performance Evaluation

Page 90: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ? A′ A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Page 91: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 92: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 93: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 94: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 95: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A A′

The Statistical Approach

If =

Page 96: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A A′

The Statistical Approach

If =

hypothesis testing

Page 97: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A A′what is the probability of measuring

a speedup this large by chance?

The Statistical Approach

If =

hypothesis testing

Page 98: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

what is the probability of measuring a speedup this large by chance?

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

easy to compute for the normal distribution

A A′If =

Page 99: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

easy to compute for the normal distribution

A A′If =

Page 100: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

what is the probability of measuring a speedup this large by chance?

Page 101: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

what is the probability of measuring a speedup this large by chance?

if there is alow probability

Page 102: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

this speedup is real

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Page 103: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

this speedup is real

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

not due to the effect on memory layout

Page 104: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

this speedup is real

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Sound Performance EvaluationSTABILIZER

randomizes layoutrepeatedly

what does re-randomization do?

not due to the effect on memory layout

Page 105: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0

0.2

0.4

350 360 370 380 390 400Time (s)

Prob

abili

ty D

ensi

tySTABILIZER

randomizes layoutrepeatedly

Page 106: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0

0.2

0.4

350 360 370 380 390 400Time (s)

Prob

abili

ty D

ensi

tySTABILIZER

randomizes layoutrepeatedly

one random layout per-run

Page 107: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

0.0

0.2

0.4

350 360 370 380 390 400Time (s)

Prob

abili

ty D

ensi

ty

one random layout per-run

Page 108: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

0.0

0.2

0.4

350 360 370 380 390 400Time (s)

Prob

abili

ty D

ensi

ty

many random layouts in each run

one random layout per-run

Page 109: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERrandomizes layoutrepeatedly

Page 110: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER generates a new random layout every ½ second

STABILIZERrandomizes layoutrepeatedly

Page 111: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER generates a new random layout every ½ second

Total execution time isthe sum of all periods

STABILIZERrandomizes layoutrepeatedly

Page 112: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER generates a new random layout every ½ second

Total execution time isthe sum of all periods

The sum of a sufficient number of independent, identically distributed random

variables is approximately normally distributed.

STABILIZERrandomizes layoutrepeatedly

Page 113: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER generates a new random layout every ½ second

Total execution time isthe sum of all periods

The sum of a sufficient number of independent, identically distributed random

variables is approximately normally distributed.

STABILIZERrandomizes layoutrepeatedly

Page 114: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER generates a new random layout every ½ second

Total execution time isthe sum of all periods

The sum of a sufficient number of independent, identically distributed random

variables is approximately normally distributed.

STABILIZERrandomizes layoutrepeatedly

Page 115: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Central Limit Theorem

The sum of a sufficient number of independent, identically distributed random

variables is approximately normally distributed.

Page 116: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Central Limit Theoremexecution times are normally distributed

The sum of a sufficient number of independent, identically distributed random

variables is approximately normally distributed.

Page 117: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layoutenables sound performance evaluation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

makes performance evaluation difficult

Page 118: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZER

makes performance evaluation difficultMemory layout affects performance

STABILIZER eliminates the effect of layoutenables sound performance evaluation

evaluation of LLVM’s optimizations with STABILIZERCase Studies

Page 119: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Case Studies

evaluation of LLVM’s optimizations with STABILIZER

Page 120: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Case Studies

on each benchmark

evaluation of LLVM’s optimizations with STABILIZER

Page 121: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Case Studies

on each benchmark

across the whole benchmark suite

evaluation of LLVM’s optimizations with STABILIZER

Page 122: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Case Studies

on each benchmark

across the whole benchmark suite

evaluation of LLVM’s optimizations with STABILIZER

first, build benchmarks with STABILIZER

Page 123: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Build programs with STABILIZER

Page 124: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Build programs with STABILIZER

> szc main.c

Page 125: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

> szc main.c

Build programs with STABILIZER

Page 126: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

> szc main.c-Rcode

Build programs with STABILIZER

Page 127: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

> szc main.c-Rcode -Rheap -Rstack

Build programs with STABILIZER

Page 128: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

> szc main.c-Rcode -Rheap -Rstack

Build programs with STABILIZER

now run the benchmarks

Page 129: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Run benchmarks as usual

A′A ×30 ×30

Page 130: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Run benchmarks as usual

A′A ×30 ×30

drop the results into R

Page 131: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Run benchmarks as usual

A′A ×30 ×30

drop the results into R

Page 132: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 133: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Is faster than ?A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

A′

Page 134: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

If = A

Page 135: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

A

Page 136: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′If = A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

what is the probability of measuring a difference at least this large?

Page 137: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

what is the probability of measuring a difference at least this large?

A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

The Student’s t-test

If = A

Page 138: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

what is the probability of measuring a difference at least this large?

A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

The Student’s t-test

If = A

t.test(times. A′, times.A)

Page 139: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

what is the probability of measuring a difference at least this large?

A′If = A

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

The Student’s t-testIf p-value

Page 140: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

The Student’s t-testIf p-value ≤ 5%

If = A

what is the probability of measuring a difference at least this large?

Page 141: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

The Student’s t-testIf p-value ≤ 5%

If = A

95% Confidence

what is the probability of measuring a difference at least this large?

Page 142: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = A′

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

A

The Student’s t-testIf p-value ≤ 5%

we reject the null hypothesis

what is the probability of measuring a difference at least this large?

Page 143: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

≠ A

The Student’s t-testIf p-value ≤ 5%

we reject the null hypothesis

0%

10%

20%

30%

40%

85.0 87.5 90.0 92.5 95.0Time (s)

Percen

t of Ob

served

Runtim

es

Random chance not responsible for the measured difference

A′

Page 144: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

≠ A

The difference is real

A′

Page 145: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

−10%

0%

10%

20%

libquantum

milc

bzip2

sphinx3

namd lbm

perlbench

hmmer

h264ref

cactusADM wrf

sjeng

gobmk

gromacs

zeusmp mcf

gcc

astar

Speedup Significant

Yes

No

Speedup of -O2 over -O1

Page 146: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

−10%

0%

10%

20%

libquantum

milc

bzip2

sphinx3

namd lbm

perlbench

hmmer

h264ref

cactusADM wrf

sjeng

gobmk

gromacs

zeusmp mcf

gcc

astar

Speedup Significant

Yes

No

Speedup of -O2 over -O1

Page 147: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

−10%

0%

10%

20%

libquantum

milc

bzip2

sphinx3

namd lbm

perlbench

hmmer

h264ref

cactusADM wrf

sjeng

gobmk

gromacs

zeusmp mcf

gcc

astar

Speedup Significant

Yes

No

Speedup of -O2 over -O1

Page 148: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

−10%

0%

10%

20%

libquantum

milc

bzip2

sphinx3

namd lbm

perlbench

hmmer

h264ref

cactusADM wrf

sjeng

gobmk

gromacs

zeusmp mcf

gcc

astar

Speedup Significant

Yes

No

Speedup of -O2 over -O1

Page 149: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

−10%

0%

10%

20%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

Speedup of -O3 over -O2

Page 150: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Speedup of -O3 over -O2

0.0%

0.5%

1.0%

1.5%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

Page 151: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0%

0.5%

1.0%

1.5%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

Speedup of -O3 over -O2

Page 152: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0%

0.5%

1.0%

1.5%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

Speedup of -O3 over -O2

Page 153: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0%

0.5%

1.0%

1.5%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

Speedup of -O3 over -O2

Page 154: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

0.0%

0.5%

1.0%

1.5%

bzip2

gobmk

zeusmp

libquantum wrf

astar mcf

hmmer

milc

namd gcc

lbmgromacs

h264ref

cactusADM

perlbench

sphinx3

sjeng

Speedup Significant

Yes

No

What do the results mean?

Page 155: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Comparing optimizations

-O2 -O3×30 ×30

Page 156: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2 -O3

×30 ×30lbm lbm

×30 ×30

Comparing optimizations

Page 157: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2 -O3

×30 ×30lbm lbm

×30 ×30astar astar

×30 ×30

Comparing optimizations

Page 158: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2 -O3

×30 ×30lbm lbm

×30 ×30astar astar

...

×30 ×30

Comparing optimizations

Page 159: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2 -O3×30 ×30

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Comparing optimizations

Page 160: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Is faster than ?

Page 161: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Is faster than ?

Page 162: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Is faster than ?

Page 163: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Is faster than ?

Page 164: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Page 165: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

Page 166: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Page 167: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Analysis of Variance

Page 168: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Analysis of Varianceaov(time~opt+Error(benchmark/opt), times)

Page 169: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Analysis of VarianceIf p-value

Page 170: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Analysis of VarianceIf p-value ≤ 5%

Page 171: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

If = -O2-O3

0%

1%

2%

3%

4%

0 25 50 75 100Time (s)

Perc

ent o

f Obs

erve

d Ru

ntim

es

Version−O2−O3

what is the probability of measuring these differences?

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

Page 172: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

-O3 -O2vs

Page 173: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

p-value = 26.4%

-O3 -O2vs

Page 174: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

p-value = 26.4%

-O3 -O2vs

are we 73.6% confident?

Page 175: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

p-value = 26.4%

-O3 -O2vs

are we 73.6% confident?one in four experiments will show an

effect that does not exist!

Page 176: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Analysis of VarianceIf p-value ≤ 5%

we reject the null hypothesis

p-value = 26.4%fail to reject the null hypothesis

-O3 -O2vs

Page 177: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O3

Analysis of Variance

we reject the null hypothesis

-O2The effect of over is indistinguishable from noise

If p-value ≤ 5%

Page 178: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

-O3

Analysis of Variance

we reject the null hypothesis

-O2The effect of over is indistinguishable from noise

If p-value ≤ 5%

Did STABILIZER hide the effect?

Page 179: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Runtime with -O2

Runtime with -O3

Runtime with -O1

Runtime with -O0

Execution Time

Did STABILIZER hide the effect?

Page 180: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZ

ER

STABILIZ

ER

STABILIZ

ERSTA

BILIZER

Execution Time

Runtime with -O2

Runtime with -O3

Runtime with -O1

Runtime with -O0

Did STABILIZER hide the effect?

Page 181: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZ

ER

STABILIZ

ER

STABILIZ

ERSTA

BILIZER

Execution Time

Runtime with -O2

Runtime with -O3

Runtime with -O1

Runtime with -O0

speedups

Did STABILIZER hide the effect?

Page 182: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

Runtime with -O2

Runtime with -O3

Runtime with -O1

Runtime with -O0

STABILIZ

ER

STABILIZ

ER

STABILIZ

ERSTA

BILIZER

Execution Time

speedups

Did STABILIZER hide the effect?

Page 183: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

SPECint SPECfp Summary

0%

10%

20%

30%

40%

perlbe

nch gcc

gobm

kh2

64ref

sjeng

astar

bzip2

libqu

antu

mhm

mer mcfca

ctusA

DMze

usmp

wrfgr

omac

ssp

hinx3 lbm milc

namd

hmea

nmed

ian

% In

crea

se in

Exe

cutio

n Ti

me

STABILIZER is fast enough

Page 184: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layout

showed that -O3 does not have a statistically significant effect across our benchmarks

Case Studies

random layout enables sound performance evaluation

makes performance evaluation difficult

Page 185: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layout

showed that -O3 does not have a statistically significant effect across our benchmarks

Case Studies

random layout enables sound performance evaluation

makes performance evaluation difficult

Page 186: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layout

showed that -O3 does not have a statistically significant effect across our benchmarks

Case Studies

random layout enables sound performance evaluation

makes performance evaluation difficult

Page 187: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

STABILIZERMemory layout affects performance

STABILIZER eliminates the effect of layout

showed that -O3 does not have a statistically significant effect across our benchmarks

Case Studies

random layout enables sound performance evaluation

makes performance evaluation difficult

Page 188: STABILIZERcharlie/files/stabilizer... · 2015-01-06 · Memory layout affects performance STABILIZER eliminates the effect of layout no way to measure effect of change in isolation

available at

stabilizer-tool.org

STABILIZER