Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
Linux shared library profiler implementation
Edward Ryabikov
Mark Zaslavskiy
Kirill Krinkin
Open Source & Linux Lab
13th FRUCT Conference, Petrozavodsk, April 24, 2013
Agenda • Project motivation and goals;
• Linux linking mechanisms and tools: o ELF format fundamentals;
o L
• “Non-invasive” shared objects profiling;
• ELFPERF design o Modules;
o Call redirection;
o Wrappers;
o Call tooling;
• Conclusions
2/ 29
Motivation • Popular Linux Profilers (gprof, gcov, GPT, Valgrind)
have restrictions:
o Need to recompile with special options
(gprof, gcov)
o Need to relink program with 3rd-party libraries
(GPT)
o Need to use special environment for profiling
(Valgrind)
o Very long execution;
o Need to use super-user rights
3/ 29
Project goals • System-wide ELF executables performance analysis
• Requirements: o Profile function calls without recompilation and relinking with 3rd-party
libraries;
o Profiling only given set of C/C++ functions from shared libraries in Linux
o Profiling both dynamically linked and dynamically loaded functions
o Profiling without creating of special environment
• Statistics: o number of calls
o Time inside functions
• Perform profiling on x86/x64 platforms
4/ 29
Linux Loader and Linker • Glibc:
o /lib/ld-linux.so
o /lib/libdl.so
• ELF (Executable Linux File)
o Format interpreter
o Symbols
o Dynamic loading calls
5/ 29
Dynamic Linking
anylib.so
anyfunc1()
anyfunc2()
anyfuncn()
…
exe
int main()
{
…
anyfunc1();
…
return 0;
}
ld-linux.so
_dl_fixup
_dl_profile_fixup
_dl_call_pltexit
LD_LIBRARY_PATH
1
2
3
6/ 29
ELF Interpretation
init and shared lib segments
.text segment (r/o)
.data segment (initialized r/w)
.bss segment (uninitialized r/w)
Process image ELF header
Program header table (required for executables)
.text section
.data section
.bss section
.symtab
.rel.text
.dynamic
.debug
Section header table (required for relocatables)
Executable object file
Virtual Address
0x080483e0
0x08048494
0x0804a010
0x0804a3b0
7/ 29
Dependencies • Linking dependency – one or more shared objects
which are explicitly referenced.
• Runtime linker:
o locates and loads the associated shared objects.
o Inspects dependencies of shared objects;
o Continue process to the deep;
Note: if symbols are referenced several times, only first
reference will be used.
8/ 29
Runtime linker (LD)
• LD (dynamic link editor) is a system “service” for
resolving dependencies and caching shared
objects;
• Responsibilities: o Loading shared objects
o Dependency management
o Caching
o [symbol] -> address translation
o Fixing/creating relocation tables
• Corresponded tools:
o ldd – prints dependency tree
o objdump – inspections for ELF binaries
o objcopy – manipulations with ELF binaries 9/ 29
Dynamic loading
• .dynamic segment contains a pointer to string table
• LD
o creates a scope list (set of libraries to be loaded)
o maps the library to the process address space.
o creates linked list of symbols
o Bind the symbols to function addresses
10/ 29
Symbol Table Structure
typedef struct
{ Elf32_Word st_name; /* Symbol name (string tbl index) */
Elf32_Addr st_value; /* Symbol value */
Elf32_Word st_size; /* Symbol size */
unsigned char st_info; /* Symbol type and binding */
unsigned char st_other; /* Symbol visibility */
Elf32_Section st_shndx; /* Section index */
} Elf32_Sym;
11/ 29
Offset tables in ELF
• Dynamic linker primarily uses two processor-specific
tables: o Global Offset Table (GOT)
o Procedure Linkage Table (PLT).
• Dynamic linkers support PIC Code through the GOT
in each shared library.
12/ 29
Tags in .dynamic
• In the .dynamic section items have next tags:
o DT_STRTAB – address of the string table.
o DT_SYMTAB -- address of the symbol table.
o DT_NEEDED -- string table offset of a null-terminated string,
giving the name of a needed library. The offset is an index
into the table recorded in the DT_STRTAB entry.
o DT_HASH: This element holds the address of the symbol
hash table which refers to the symbol table referenced by
the DT_SYMTAB element.
13/ 29
Relocation Process
• Loading dependencies
• Do relocations for each object
• Connecting symbolic references with symbolic
definitions.
• Ex: when a program calls a function, the associated
call instruction must transfer control to the proper
destination address at execution. Relocatable files
must have information how to modify their section
contents.
14/ 29
Dynamic Loading
libdl.so
dlopen
dlsym
dlclose
dlopen
dlsym
dlerror
anylib.so
anyfunc1()
anyfunc2()
anyfuncn()
…
?
ld-linux.so
exe
int main()
{
void* p =
dlopen(“anylib.so”, RTLD_LAZY);
…
void(*f)() =
dlsym(p, “anyfunc1”);
…
f();
…
dlclose(p);
…
}
LD
_LIB
RARY_PATH
1
2
3
15/ 29
“Noninvasive” profiling
o Profiler can not be implemented in the program code
o Profiling should be performed at well-defined points of
function calls
o Profiling process should not corrupt the algorithm of
profiled application
o Profiling process should use minimum amount of system
resources
o Results of profiling should be as accurate as possible
16/ 29
Possible implementations
• Infiltration into the symbol relocation process
• Modification of Linux dynamic linker
(ld-linux.so)
• Modification of dynamic loading library (libdl.so)
17/ 29
Components of ElfPerf
• Shared library libelfperf.so o Call redirection and function wrapping mechanisms
o Collecting of calls statistics
o Memory management
• Modified dynamic linker (ld-linux.so) o Uses libelfperf.so for profiling of dynamically linked functions
o Displays the results of profiling
• Modified dynamic loading library (libdl.so) o Uses libelfperf.so for profiling of dynamically loaded functions
18/ 29
Call Redirection • Calls redirection mechanism (Redirector) is a set of
machine codes for the next assembly instructions:
• All they do is:
o Save address of profiled function in program stack
o Jump to wrapper-function
push $fcnPtr
jmp $wrapper_addr
19/ 29
Working scheme of Redirectors
push $fcnPtr
jmp $wrapper_addr
Redirector
…
any_function
push %ebp
mov %esp,%ebp
fcnPtr
leave
ret
…
void(*f)() = dlsym(p, “any_function”);
…
f();
…
Code 1
2
3
4
Wrapper wrapper_addr
...
call preProfile
...
ret
wrapper_rp:
...
call postProfile
...
5
20/ 29
Features of Redirectors
• Each redirector is created individually for each
profiled function
• Redirectors are placed into data segment of
process virtual memory
• The operating system allows to mark these memory
areas as executable
21/ 29
Functions Wrapping mechanism
• Function Wrapping mechanism (or Wrapper) is a
function that does next things:
o Takes control from redirector
o Performs pre-profile operations
o Performs replacement of return address
o Performs jump into profiled function
o Again takes control after the work of profiled
function
o Performs post-profile operations
o Returns to caller
22/ 29
Wrappers idea
Wrapper Params
Return value
Function Start time
End time
jmp
ret
Function address
Return address
Context
23/ 29
Wrappers at work
Function
leave
retl
...
push %ebp
mov %esp,%ebp
void wrapper()
{
// push %ebp
// movl %esp, %ebp
asm volatile (
"popl %ebp"
"pushal"
"pushl 32(%esp)"
"pushl 40(%esp)"
"call preProfile"
"addl $8, %esp"
"movl $wrapper_rp, 36(%esp)"
"popal"
"retl"
);
asm volatile (
"wrapper_rp:"
"pushl $0"
"pushal"
"call postProfile"
"movl %eax, 32(%esp)"
"popal"
"retl"
);
}
Wrapper
Context
Stack Base Higher Memory
Addresses …
fcn param #n
…
fcn param #0
ret address
fcn address
old %EBP
%ESP
%EBP %EAX, %EBX, %ECX, %EDX, %ESI, %EDI
fcn address
ret address
ret address
fcn address
start time
wrapper_rp
old %EBP %EBP
$0
%EAX, %EBX, %ECX, %EDX, %ESI, %EDI
end time
ret address
Registers
old %EAX
%EBX old
%ECX old
%EDX old
%EDI old
%ESI old
new
new
new
new
new
new
rval
oldr
oldr
rval
oldr
oldr
raddr
new
new
new
new
new
24/ 29
Features
• Wrapper doesn’t corrupt stack content
• Wrapper exists in a single copy for all functions in
each profiler implementation (x86 or x64)
• Saving/Restoring of registers’ state allows to escape
of uncontrollable changes in the program state
• Such implementation of wrapper mechanism allows
to profile wide set of C/C++ functions
25/ 29
Components interaction
elfperf-ld-linux.so
_dl_fixup
_dl_profile_fixup
_dl_call_pltexit
LD_PRELOAD elfperf-libdl.so
dlclose
dlopen
dlsym
dlerror
libelfperf.so
Wrapper
Redirectors
Statistics
LD
_LIB
RARY_PATH
Memory management
Shared memory
ElfPerf Storage
Function Infos
Function Statistics
26/ 29
Conclusion
• «Light» profiler based on «patched» ld-linux.so and
libdl.so
• Support of profiling for C/C++ functions from shared
libraries
(including libs compiled with –fomit-frame-pointer
flag)
• Collecting of information about number and total
duration of function calls
• Support of both x86 and x64 platforms
27/ 29
Links
• Project repository: https://github.com/OSLL/elfperf
• Full description of project (RUS): http://redmine.osll.spb.ru/projects/epat/wiki/_Architecture_description_
28/ 29
Questions & Answers
Edward Ryabikov, [email protected]
Mark Zaslavskiy, [email protected]
Kirill Krinkin [email protected]
Open Source & Linux Lab, http://osll.ru
29/ 29