Hidden Gems: UNIX Shared Libraries Way better than PROBUILD

Hidden Gems:

UNIX Shared LibrariesWay better than PROBUILD.

A Few Words About The Speaker

• Tom Bascom, Roaming DBA & Progress User since 1987

• President, DBAppraise, LLC– Remote Database Management Service.– Simplifying the job of Managing and Monitoring The World’s

Best Business Applications.– tom@dbappraise.com

• VP, White Star Software, LLC– Expert Consulting Services related to all aspects of Progress and

OpenEdge.– tom@wss.com

What is a UNIX Shared Library?

• A collection of executable code and data which is loaded into memory separately from the launch of the main program (_progres).

• Progress 4GL has supported Windows DLLs since (at least) version 8.

• UNIX DLL support arrived with Progress version 9.• I’ll be using Linux as an example but the

principles apply to all UNIX platforms.More Info: http://www.dwheeler.com/program-library/Program-Library-HOWTO/t1.html

Why Would You Use One?

• To share code from 3rd party, non-Progress, sources.

• To leverage proprietary internal libraries.• To gain performance improvements from

routines written in faster languages.• To access low-level OS features that are not

otherwise available.

/lib (and /lib64)/lib> ls *.so*ld-2.4.so libcrypt-2.4.so libnss_dns-2.4.so libresmgr.so.1ld-linux.so.2 libcrypt.so.1 libnss_dns.so.2 libresmgr.so.1.0.0ld-lsb.so.2 libc.so.6 libnss_files-2.4.so libresolv-2.4.sold-lsb.so.3 libdevmapper.so libnss_files.so.2 libresolv.so.2libacl.so.1 libdevmapper.so.1.02 libnss_hesiod-2.4.so librt-2.4.solibacl.so.1.1.0 libdl-2.4.so libnss_hesiod.so.2 librt.so.1libanl-2.4.so libdl.so.2 libnss_nis-2.4.so libSegFault.solibanl.so.1 libe2p.so.2 libnss_nisplus-2.4.so libss.so.2libattr.so.1 libe2p.so.2.3 libnss_nisplus.so.2 libss.so.2.0libattr.so.1.1.0 libext2fs.so.2 libnss_nis.so.2 libsysfs.so.1libaudit.so.0 libext2fs.so.2.4 libnss_winbind.so.2 libsysfs.so.1.0.3libaudit.so.0.0.0 libgcc_s.so.1 libnss_wins.so.2 libthread_db-1.0.solibblkid.so.1 libhistory.so.5 libpamc.so.0 libthread_db.so.1libblkid.so.1.0 libhistory.so.5.1 libpamc.so.0.81.0 libutil-2.4.solibBrokenLocale-2.4.so libm-2.4.so libpam_misc.so.0 libutil.so.1libBrokenLocale.so.1 libmemusage.so libpam_misc.so.0.81.2 libuuid.so.1libbz2.so.1 libm.so.6 libpam.so.0 libuuid.so.1.2libbz2.so.1.0.0 libncurses.so.5 libpam.so.0.81.5 libwrap.so.0libc-2.4.so libncurses.so.5.5 libpcprofile.so libwrap.so.0.7.6libcap.so.1 libnscd.so.1 libpthread-2.4.so libxcrypt.so.1libcap.so.1.92 libnscd.so.1.0.0 libpthread.so.0 libxcrypt.so.1.2.4libcidn-2.4.so libnsl-2.4.so libreadline.so.5 libz.so.1libcidn.so.1 libnsl.so.1 libreadline.so.5.1 libz.so.1.2.3libcom_err.so.2 libnss_compat-2.4.so libresmgr.solibcom_err.so.2.1 libnss_compat.so.2 libresmgr.so.0.9.8

Can You Narrow That Down?

• For backwards compatibility there are several versions of many libraries.

• Many libraries are “niche” libraries and not generally interesting.

• The most interesting (to me) are:– libc, the standard “C” library– libm, the math library– libz, the compression library– libcrypt, the encryption library

What Functions are Inside?

$ nm -DC /lib/libz.so.1.2.3 | grep -v " _" | grep " T "00000000000017b0 T adler320000000000001a70 T adler32_combine0000000000001c10 T compress0000000000001b40 T compress20000000000001b20 T compressBound0000000000001c70 T crc320000000000002330 T crc32_combine0000000000004220 T deflate00000000000058b0 T deflateBound0000000000005f70 T deflateCopy…

A Few Words of Warning

• The slides that you are about to see contain very little in the way of:– Error checking.– Exception handling.– Garbage collection.– And other niceties.

• While this may be customary for sample code it is very dangerous when working with real C code.

• You have been warned.

Calling Functions in libc

Problem: No OS-PUTENV()NAME putenv - change or add an environment variable

SYNOPSIS #include <stdlib.h> int putenv(char *string);

DESCRIPTION The putenv() function adds or changes the value of environment variables. The argument string is of the form name=value. If name does not already exist in the environment, then string is added to the environment. If name does exist, then the value of name in the environment is changed to value. The string pointed to by string becomes part of the environment, so altering the string changes the environment.

RETURN VALUE The putenv() function returns zero on success, or non-zero if an error occurs.

Calling putenv()

define variable x as integer no-undo.

procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long.end.

display os-getenv( "XYZZY" ).pause.

run putenv( "XYZZY=pflugh", output x ).display os-getenv( "XYZZY" ).

os-command value( 'echo "$XYZZY"' ).

return.

Calling putenv()

┌────────────────────┐│? │└────────────────────┘

Press space bar to continue.

┌────────────────────┐│? pflugh │└────────────────────┘

Press space bar to continue.

pflugh

Press space bar to continue.

Problem: No Scientific NotationNAME sprintf - formatted string output

SYNOPSIS int sprintf( char *str, const char *format, ... );

THE CONVERSION SPECIFIER A character that specifies the type of conversion to be applied. The conversion specifiers and their meanings are: … e,E The double argument is rounded and converted in the style [-]d.ddde±dd where there is one digit before the decimal-point character and the number of digits after it is equal to the precision; if the precision is missing, it is taken as 6; if the precision is zero, no decimal-point character appears. An E conversion uses the letter E (rather than e) to introduce the exponent. The exponent always contains at least two digits; if the value is zero, the exponent is 00.

Calling sprintf()define variable x as integer no-undo.define variable c as memptr no-undo.define variable m as memptr no-undo.

procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long.end.

set-size( c ) = 1024.set-size( m ) = 1024.put-string( m, 1 ) = "%1.4e".run sprintf( input-output c, m, 0.0123, output x ).

display get-string( c, 1 ) format “x(20)”.

return.

Calling sprintf()

┌────────────────────┐│1.2300e-02 │└────────────────────┘

Procedure complete. Press space bar to continue.

Data Type Mapping

“C” Data Type Procedure Parameter Definition

char BYTE

short SHORT

unsigned short UNSIGNED SHORT

long LONG

int LONG

float FLOAT

double DOUBLE

char* (read-only) CHARACTER or MEMPTR

char* (writeable) MEMPTR

pointer to … HANDLE TO …

Creating Your Own Shared Library

Overview

• You’ll need some “C” source code.• Compile to object code.• Run the Linker and create a Shared Library.• Test with Progress.

Problem: Sub-second PAUSE

define variable i as integer no-undo.define variable s as integer no-undo.

s = time.

do i = 1 to 1000: display i. etime( yes ). do while etime < 500: end.end.

display string( time - s, “hh:mm:ss" ).

return.

What’s Wrong With That?

• 8 minutes and 20 seconds of CPU time!

$ ps -ef | grep [_]progrestom 21243 27534 1 17:16 pts/0 00:02:30 /usr/pro102a/bin/_progres -1$ pro –p shortsleep.p

┌──────────────┐│ i ││────── ───── ││ 1,000 08:20 │└──────────────┘

$ ps -ef | grep [_]progrestom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

Source Code

#include <time.h>

long shortsleep( unsigned long milliseconds ){ struct timespec tmReq;

tmReq.tv_sec = (time_t)(milliseconds / 1000) ; tmReq.tv_nsec = (time_t)(milliseconds % 1000) *

1000 * 1000 ;

nanosleep( &tmReq, NULL ) ;}

Compile to Object Code

$ cc –c shortsleep.c -fPIC

$ lsshortsleep.c shortleep.o

$ file shortleep.oELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped

Link to Create a Shared Lib

$ ld -shared shortsleep.o -o shortsleep.so

$ ls shortsleep*shortsleep.c shortsleep.o shortsleep.so

$ file shortsleep.soshortsleep.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped

$ nm -DC shortsleep.soshortsleep.so:0000000000100460 A __bss_startshortsleep.so:0000000000100460 A _edatashortsleep.so:0000000000100460 A _endshortsleep.so: U nanosleepshortsleep.so:0000000000000280 T shortsleep

Progress 4GL Test Harnessdefine variable i as integer no-undo.define variable x as integer no-undo.define variable s as integer no-undo.

procedure shortsleep external "./shortsleep.so": define input parameter ms as long. define return parameter x as long.end.

s = time.do i = 1 to 1000: display i. run shortsleep( 500, output x ).end.display string( time - s, “hh:mm:ss" ).

return.

Test With Progress$ pro –p shortsleep.pCould not open Dynamic Library: ./shortsleep.so (8013)DLL Error : ./shortsleep.so: wrong ELF class: ELFCLASS32 (8014)Press space bar to continue.

$ file $DLC/bin/_progres_progres: setuid ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, not stripped

Test Again!

$ ps -ef | grep [_]progrestom 21243 27534 1 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1$ pro –p shortsleep.p

┌──────────────┐│ i ││────── ───── ││ 1,000 08:20 │└──────────────┘

$ ps -ef | grep [_]progrestom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

• No measurable CPU time!

Problem: Regular Expressions

NAME regcomp, regexec, regerror, regfree - POSIX regex functions

SYNOPSIS #include <sys/types.h> #include <regex.h>

int regcomp( regex_t *preg, const char *regex, int cflags );

int regexec( const regex_t *preg, const char *string, size_t nmatch, regmatch_t pmatch[], int eflags );

size_t regerror( int errcode, const regex_t *preg, char *errbuf, size_t errbuf_size );

void regfree( regex_t *preg );

Source Code#include <regex.h>

long regex_match( char *pattern, char *string ) {

regex_t xpattern; size_t matchDepth = 2; regmatch_t pmatch[2]; int offset;

if ( regcomp(&xpattern, pattern, REG_EXTENDED|REG_ICASE ) != 0 ) { offset = -2; } else { if ( regexec(&xpattern, string, matchDepth, pmatch, 0 ) != 0 ) { offset = -1; } else { offset = pmatch[1].rm_so; } } regfree( &xpattern ); return offset;}

Compile and Link…

$ cc –c rx2.c -fPIC$ ld –shared rx2.o –o rx2.so$ file rx2.sorx2.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped$ nm -DC rx2.so00000000001004e8 A __bss_start00000000001004e8 A _edata00000000001004e8 A _end U regcomp U regexec0000000000000318 T regex_match U regfree

Progress 4GL Test Harnessdefine variable pattern as character no-undo format "x(60)".define variable string as character no-undo view-as editor inner-lines 10 inner-chars 60.define variable x as integer no-undo.

procedure regex_match external "./rx2.so": define input parameter pattern as character. define input parameter string as character. define return parameter x as long.end.

update pattern skip string skip with side-labels.

run regex_match( pattern, string, output x ).display x.

return.

Test!

$ pro –p rx.p

┌──────────────────────────────────────────────────────────┐│pattern: .*(Fox).* ││string: The quick brown fox jumped over the lazy dog! ││ ││ ││ ││ ││x: 16 │└──────────────────────────────────────────────────────────┘

Shared LibraryChallenges

Which Library to Use?

• Use a variable? NOT! • Use a pre-processor.• Use symbolic links.• Wait for 10.2B.

procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long.end.

Variable Argument Lists

• Isolate different cases in multiple .p files.• Use wrapper libraries.• Wait for 10.2B.

int sprintf( char *str, const char *format, ... );

Call-Backs?

• Not supported .

void qsort( void *base, size_t nmemb, size_t size, int(*compar)(const void *, const void *));

Persistence

• About 25% faster in crude testing.• RELEASE EXTERNAL “/lib64/libc.so.6”.

procedure putenv external "/lib64/libc.so.6" PERSISTENT: define input parameter env as character. define return parameter x as long.end.

MEMPTRsdefine variable x as integer no-undo.define variable c as memptr no-undo.define variable m as memptr no-undo.

procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long.end.

set-size( c ) = 1024.set-size( m ) = 1024.put-string( m, 1 ) = "%1.4e".run sprintf( input-output c, m, 0.0123, output x ).

display get-string( c, 1 ) format “x(20)”.

set-size( c ) = 0.set-size( m ) = 0.

return.

DynamicShared Library

Call(Special Bonus Slide!)

Dynamic Shared Lib Calldefine variable LIBC as character no-undo initial "/lib64/libc.so.6".

function os-putenv returns integer ( input env as character ):

define variable hCall as handle no-undo. define variable rVal as integer no-undo.

create call hCall. assign hCall:call-name = "putenv" hCall:library = LIBC hCall:call-type = DLL-CALL-TYPE hCall:num-parameters = 1 hCall:return-value-dll-type = "LONG" . hCall:set-parameter( 1, "CHARACTER", "INPUT", env ). hCall:invoke(). rVal = hCall:return-value. delete object hCall. return rVal.

end.

Conclusion

• Share code from 3rd party, non-Progress, sources.

• Gain performance improvements from routines written in faster languages.

• Access low-level OS features that are not otherwise available.

• It beats the heck out of using PROBUILD.

Questions?

Thank-you!

dlopen() and FriendsNAME dladdr, dlclose, dlerror, dlopen, dlsym, dlvsym - programming interface to dynamic linking loader

SYNOPSIS #include <dlfcn.h>

void *dlopen( const char *filename, int flag ); char *dlerror( void ); void *dlsym( void *handle, const char *symbol ); int dlclose( void *handle );

DESCRIPTION The four functions dlopen(), dlsym(), dlclose(), dlerror() implement the interface to the dynamic linking loader.