View
230
Download
2
Tags:
Embed Size (px)
Citation preview
Porting Open Source to OpenVMS
Brad McCuskerOpenVMS Engineering
page 2September 2003 OpenVMS Technical Update
Topics
– UNIX Application Portability Initiative - Overview – Future Plans
• OpenVMS V7.3-2
• OpenVMS V8.2
• Future Releases
– Porting Experiences
page 3September 2003 OpenVMS Technical Update
Unix Portability - Rationale
• Many ISVs develop applications for both OpenVMS and UNIX/Linux platforms
• Applications are (or can be) ported from UNIX/Linux platforms to OpenVMS
• Operators, programmers, users may be more familiar with *NIX-style interfaces, commands, utilities and tools
page 4September 2003 OpenVMS Technical Update
UNIX Portability - Goal
Provide a full set of UNIX interfaces and tools within OpenVMS– In native, integrated fashion – No layered emulator (e.g. old “POSIX for OpenVMS”
product)• No performance issues
• No interoperability issues
page 5September 2003 OpenVMS Technical Update
UNIX Portability - Benefits
• Easy portability of UNIX applications to OpenVMS• Easy development of applications intended to
run on both UNIX and OpenVMS• No need to train UNIX-skilled personnel on
OpenVMS
• OpenVMS will optionally be like a “UNIX flavor”– Cost of porting from UNIX to OpenVMS equal or
comparable to porting from one “UNIX flavor” to another (e.g. from Solaris to Tru64)
page 6September 2003 OpenVMS Technical Update
But – I like VMS the way it is!!!
• Current VMS behavior is preserved– New UNIX Portability features typically need to be
enabled– Defaults preserve existing behavior
• C Run Time Library: UNIX features are enabled via logical name switches – Old behavior is the default
• Legacy behavior is preserved
– Can also enable features via an API
page 7September 2003 OpenVMS Technical Update
Rollout…
• Started already with VMS V7.3-1…– Delivered first set of UNIX “enabling technologies”
• C RTL
• GNV/BASH (Commands & Utilities)
• File system improvements
page 8September 2003 OpenVMS Technical Update
OpenVMS 7.3-1 Enhancements
• File system– Mixed case file names, case sensitive compares– Time of last file access– Hard link improvements– Root directory support
• C Run Time Library– New UNIX APIs– Improved UNIX filename support– API for controlling feature switches
page 9September 2003 OpenVMS Technical Update
OpenVMS 7.3-1 – GNV
•GNV: GNU’s Not VMS – GNU-based, UNIX® environment for OpenVMS– Open source, freeware product
• http://gnv.sourceforge.net/– Implementation of the UNIX shell BASH (Bourne
Again Shell)– Provides an environment for porting and running
UNIX tools and software on OpenVMS
– Updated version included with OpenVMS and at: http://h71000.www7.hp.com/opensource/opensource.html
– Updates include:• ODS-5 file system support• Additional utilities ported and included• Packaged as a HP-branded PCSI kit
page 10
September 2003 OpenVMS Technical Update
Open Source Tools CD
– Open Source Tools CD– Ships with OpenVMS
V7.3-1– Includes GNV – Other contents:
• Stunnel
• VMSTAR
• ZIP
• SSL 0.9.6b Sources
• CDRECORD
• OpenVMS Migration Software
page 11
September 2003 OpenVMS Technical Update
V7.3-1 Contents – GNV (cont,)
• Provides an environment for porting and running UNIX tools and software on OpenVMS
– Implementation of the UNIX shell BASH (Bourne Again Shell)– Many UNIX-shell utilities
• General purpose
• Command manipulation
• Program creation
• User- and system-level administration
• File manipulation
• Text processing
• Printing
• Networking
page 12
September 2003 OpenVMS Technical Update
BASH – Example
bash$ ls –l
total 1
-rwxr-x--- 1 PITCHER 17 72 Apr 5 14:04 hello.c
-rwxr-x--- 1 PITCHER 17 82 Apr 5 14:04 makefile
bash$
bash$ cat makefile
hello : hello.c
gcc -o hello.exe hello.c
clean :
rm -f hello.exe hello.o
bash$ make
gcc -o hello.exe hello.c
bash$
page 13
September 2003 OpenVMS Technical Update
BASH – Example (cont)
bash$ hello
Hello There!
bash$
bash$ dcl dir /date
Directory STP:[TEST.TEST]
HELLO.C;1 5-APR-2002 13:45:56.46
HELLO.EXE;1 5-APR-2002 14:05:20.67
HELLO.O;1 5-APR-2002 14:05:18.35
MAKEFILE.;2 5-APR-2002 14:04:58.85
MAKEFILE.;1 5-APR-2002 13:59:33.72
What does the future hold?
page 15
September 2003 OpenVMS Technical Update
UNIX Portability Roadmap
OpenVMS Alpha V7.3-2 (Opal)
• CRTL•User database functions•Unix style signal functions •UID/GID•poll() •File I/O
•GNV 1.4•Bbzip2 •Gawk•Man•Improved complier interfaces (cc,gcc)
2003 2004 2005 2006
OpenVMS UP future investigations• Full function select()• fork () •Semaphores• Byte range locking• Shared memory APIs• Unix 98 compliance
OpenVMS I64 V8.1 • CRTL• GNV
OpenVMS V8.2 (Topaz)•CRTL
•Binary Tree•Asynch I/O interfaces•Symbolic links•glob() •flock()
•GNV • VI • Configure and make improvements Note: Investigations are
provided solely to inform
what is being considered
and should not be used as
a deliverable commitment.
page 16
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• Support for POSIX style UID/GID– Requires 32 bit UID/GID data structures
• Define _ _USE_LONG_GID_T macro to compile an application for 32-bit UID/GID support
– DECC$POSIX_STYLE_UID controls whether UIDs/GIDs are POSIX style or derived from the process UIC.
– APIs affected: • getegid, geteuid, getgid, getuid, setgid, setuid and
others
page 17
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• New APIs– User database functions
• getpwnam_r, getpwuid_r
– Signal functions • sigwait, sighold, sigrelse, sigignore
– Clock functions • nanosleep, clock_getres, clock_gettime, clock_settime
– Math and conversion functions• rand_r, remainder, rint, a64l, l64a
page 18
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• New APIs – glob(), globfree()– Pattern matching APIs– Extended to allow for VMS style behavior
• Controlled by feature switch – default is VMS behavior
• Use ‘*’, ‘&’, ‘…’ for wildcard, not ‘?’
• No pattern matching
– DECC$GLOB_UNIX_STYLE• Enables UNIX specific behavior
page 19
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• New APIs (cont)– Security/Impersonation Functions:
• endgrent getgrnam getsid setpgrp
• getgrent getgrnam_r seteuid setregid
• getgrgid getpgid setgrent setreuid
• getgrgid_r getpgrp setpgid setsid
• I/O– pread, pwrite, readv
• Formatted output– snprintf, vsnprintf
page 20
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• TCP/IP related enhancements– 64-bit pointer support in: sendmsg, recvmsg,
freeaddrinfo, getaddrinfo• Previously, 32-bit implementation only
• Now, dual implementation, 64 bit and 32 bit– Be careful with new 64 bit data structures
– /POINTER_SIZE=LONG will not provide 64 bit data structure
– Use 64 bit specific structure
– See C RTL Ref Manual 1.10.4.3 “Functions With Two Implementations”
page 21
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents
• TCP/IP related enhancements (2)
– poll () - input/output multiplexing • Limited to sockets only
– > 64K data transfers: recv, send, recvfrom, sendto,• Previously limited to 64K bytes.
• Support is latent – > 64K requires support in underlying TCP/IP stack– Not in TCP/IP Services V5.4
page 22
September 2003 OpenVMS Technical Update
UNIX File-Name Translation EnhancementsPerformance Enhancements
• New cache for logical name translation– DECC$ENABLE_TO_VMS_LOGNAME_CACHE
• Speeds translation of logical names in UNIX file name translation.
– 0 – Cache disables (default)
– 1 – Enable cache with 1 second entry life
– 2 – Enable cache with 2 second entry life
– Etc
– -1 – Enable cache with no entry expiration time
page 23
September 2003 OpenVMS Technical Update
UNIX File-Name Translation EnhancementsPerformance Enhancements (2)
• DECC$EFS_NO_DOTS_IN_DIRNAME
– NAME.EXT can be:• File – []NAME.EXT
• Directory [.NAME^.EXT]
– Checking to see if [.name^.ext] adds overhead.
– Enable DECC$EFS_NO_DOTS_IN_DIRNAME to suppress interpretation of a filename with dots as a directory.
page 24
September 2003 OpenVMS Technical Update
UNIX File-Name Translation Enhancements
• DECC$NO_ROOTED_SEARCH_LISTS – Controls how the to_vms function resolves search-
list logicals.– ENABLE – Assume a search list logical is all non-
rooted logicals– V7.3 behavior - /dev translates to dev:– V7.3-1behavior - /dev translates to:
• dev is rooted logical: dev:[000000]
• dev is non-rooted logical: dev:
• dev is search list: look at first element and translate as above
– ENABLE restores pre-V7.3-1 behavior.
page 25
September 2003 OpenVMS Technical Update
UNIX File-Name Translation Enhancements & bug Fixes (cont)
• DECC$NO_ROOTED_SEARCH_LISTS (cont)– If dev is a search list of mixture of rooted and non-
rooted logicals – translation can break and may not match legacy.
– By assuming non rooted search lists, DECC$NO_ROOTED_SEARCH_LISTS restores legacy behaviors
• Angle brackets now correctly recognized in UNIX name translation. – Previously, we weren’t interpreting ‘<‘ & ‘>’ as
directory delimiters
page 26
September 2003 OpenVMS Technical Update
V7.3-2 C RTL ContentsNew Feature Switches
• DECC$EXEC_FILEATTR_INHERITANCE – Feature logical modified to allow greater choice in inheritance of
file access modes.– Applies to vfork/exec
= 1 child inherits file positions for all file access modes except append.
= 2 child inherits file positions for all file access modes including append.
= 0 (disabled) child process does not inherit the file position.
page 27
September 2003 OpenVMS Technical Update
V7.3-2 C RTL ContentsNew Feature Switches (2)
• DECC$USE_JPI$_CREATOR– Affect processing of getppid()
• (getppid - get parent process ID)
– ENABLE• Use $GETJPI/JPI$_CREATOR to determines parent process ID
• UNIX compliant behavior
– DISABLE• Use $GETJPI/JPI$_OWNER to determines parent process ID
• Traditional VMS behavior
page 28
September 2003 OpenVMS Technical Update
V7.3-2 C RTL ContentsNew Feature Switches (3)
• DECC$ALLOW_REMOVE_OPEN_FILES – remove() of an open file typically fails – Standard compliance dictates that the operation succeed– Enable this feature logical to enable the standard compliant
behavior.
• DECC$ALLOW_UNPRIVILEGED_NICE – nice() – changes process priority– Controls legacy vs. standard complaint behavior– ENABLE - Exhibit legacy behavior
– No privilege check on calling process
– Set to value > MAX_PRIORITY sets to base priority
– DISABLE - Conform to the X/Open standard – Check privilege of the calling process (ALTPRI is needed)
– Set to value > a MAX_PRIORITY, sets to MAX_PRIORITY
page 29
September 2003 OpenVMS Technical Update
V7.3-2 C RTL ContentsNew Feature Switches (4)
• New switch – DECC$RENAME_ALLOW_DIR– rename() to directory is non-UNIX standard
• But, it is VMS standard behavior
• Example:– rename (file.ext,logname)
– Where:
– logname = [dir.subdir]
– Results in:
– [dir]subdir.ext
– This happens because logname gets translated to a file because rename to a directory is not allowed
– This switch restores the VMS behavior– rename (file.ext,logname) [dir.subdir]file.ext
page 30
September 2003 OpenVMS Technical Update
A word about rename()
• DECC$RENAME_NO_INHERIT should have been called RENAME_UNIX_COMPATIBLE
• DECC$RENAME_NO_INHERIT causes UNIX compliant behaviors to be enforced –
• When DECC$RENAME_NO_INHERIT is enabled, DECC$RENAME_ALLOW_DIR is ignored.
page 31
September 2003 OpenVMS Technical Update
V7.3-2 C RTL Contents (tentative)
• Extended command line length– VMS 7.3-2 increases DCL command buffer to 4K– Corresponding changes in C RTL to support larger
command lines
page 32
September 2003 OpenVMS Technical Update
V7.3-2 C RTL ContentsNew API
• decc$set_child_default_dir
– Typically, vfork/exec child processes inherit default directory from the parent.
– decc$set_child_default(default_dir) • Subsequent calls to vfork/exec, child processes
created with default directories set to default_dir
page 33
September 2003 OpenVMS Technical Update
7.3-2 CRTL ContentsEnhanced access()
• access() enhanced to also check ACLs– DECC$ACL_ACCESS_CHECK – Uses $checkpro system service– Eventually need to add similar capability to stat()
and other APIs – not done yet though
page 34
September 2003 OpenVMS Technical Update
GNV Release for V7.3-2
– New utilities• bbzip2
• gawk
• man
• GNU TAR
– Working towards getting “configure” to work for any arbitrary Open Source package.
– IPF Port complete – Negligible effort!• Available in E8.1 time frame
• Native compiler support may lag behind
page 35
September 2003 OpenVMS Technical Update
UNIX Portability Roadmap
OpenVMS Alpha V7.3-2 (Opal)
• CRTL•User database functions•Unix style signal functions •UID/GID•poll() •File I/O
•GNV 1.4•Bbzip2 •Gawk•Man•Improved complier interfaces (cc,gcc)
2003 2004 2005 2006
OpenVMS UP future investigations• Full function select()• fork () •Semaphores• Byte range locking• Shared memory APIs• Unix 98 compliance
OpenVMS I64 V8.1 • CRTL• GNV
OpenVMS V8.2 (Topaz)•CRTL
•Binary Tree•Asynch I/O interfaces•Symbolic links•glob() •flock()
•GNV • VI • Configure and make improvements Note: Investigations are
provided solely to inform
what is being considered
and should not be used as
a deliverable commitment.
page 36
September 2003 OpenVMS Technical Update
V8.2 Release Contents (tentative)
– Semaphores• semctl, semget, semop, sem_close, sem_destroy,
sem_getvalue, sem_init, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait
– Symbolic links• symlink, lstat, lchown, readlink
– Locking • flockfile, ftrylockfile, funlockfile
– Asynch I/O interfaces• aio_cancel, aio_error, aio_fsync, aio_read, aio_return,
aio_suspend, aio_write
– Binary Tree• tdelete, tfind, tsearch, twalk
– And more, to be determined
page 37
September 2003 OpenVMS Technical Update
UNIX Portability Roadmap
OpenVMS Alpha V7.3-2 (Opal)
• CRTL•User database functions•Unix style signal functions •UID/GID•poll() •File I/O
•GNV 1.4•Bbzip2 •Gawk•Man•Improved complier interfaces (cc,gcc)
2003 2004 2005 2006
OpenVMS UP future investigations• Full function select()• fork () •Semaphores• Byte range locking• Shared memory APIs• Unix 98 compliance
OpenVMS I64 V8.1 • CRTL• GNV
OpenVMS V8.2 (Topaz)•CRTL
•Binary Tree•Asynch I/O interfaces•Symbolic links•glob() •flock()
•GNV • VI • Configure and make improvements Note: Investigations are
provided solely to inform
what is being considered
and should not be used as
a deliverable commitment.
page 38
September 2003 OpenVMS Technical Update
Future Releases Contents
• Future releases plan to add APIs to achieve industry standard compliance (UNIX98? POSIX? LINUX?…)
• The following areas will be covered:– fork()– File System– Resource management– Byte Range Locking – Messaging functions– Schedulers functions– TCP– Other UNIX tools– Increase Command Line length– UNIX style shareable images – Shared memory
page 39
September 2003 OpenVMS Technical Update
Future Releases Contents
• Fork()– New system service $CLONE_PROCESS– IR complete, Functional Spec under review– Post Itanium time frame
• UNIX I/O– aka “forkable-IO”– Goes hand-in-hand with fork()
• File System– select() – pause until specified activity is detected
• Full featured (sockets, files, pipes, etc)– fallocate– fstatvfs - get file system information (maybe V7.3-2)– statvfs - get file system information (maybe V7.3-2)– ulimit - set or report file size limit
page 40
September 2003 OpenVMS Technical Update
Future Releases Contents
• Resource management– getrlimit - get maximum resource settings – madvise– setrlimit - set maximum resource consumption– getrusage - get information about resource
utilization• Messaging functions
– msgctl - message control operations– msgget - get the message queue identifier– msgrcv - message receive operation– msgsnd - message send operation
page 41
September 2003 OpenVMS Technical Update
Future Releases Contents
• Schedulers functions– sched_get_priority_max, sched_get_priority_min -
get priority limits(REALTIME)– sched_getparam - get scheduling parameters– sched_getscheduler - get scheduling policy– sched_rr_get_interval - get execution time limits– sched_yield - yield processor
• TCP– socketpair - create a pair of connected sockets– ioctl - control a STREAMS device
page 42
September 2003 OpenVMS Technical Update
Future Releases Contnets
• Miscellaneous– crypt - string encoding function(CRYPT)– lio_listio - list directed I/O (REALTIME)– realpath - resolve a pathname (into an absolute
path)
Porting Experiences
page 44
September 2003 OpenVMS Technical Update
Porting Experiences
• Some partners already using UP features (in V7.3-1) to port their applications to OpenVMS
• Recent Experience– HP and a partner worked in HP lab to determine
level of effort needed to port partner’s application– Summary of that effort follows on next slides
page 45
September 2003 OpenVMS Technical Update
Porting Experience
• Application architected to isolate OS specific features– An OS interface layer– A Network layer
• Source files maintained on partner’s Linux system– NFS served to OpenVMS system
• Team made extensive use of BASH – make and sed used extensively– Some minor changes to partner’s make files, especially in the
area of recursive make– Successfully compiled and linked all modules, except missing
semaphore routines– ar used to populate object libraries– Some difficulty with GNV linker, successfully used OpenVMS
linker
page 46
September 2003 OpenVMS Technical Update
Porting Experiences (cont.)
• Successfully passed all tests– Developer couldn't believe it - rewrote tests to add
verification that it was actually executing properly• Some things were missing, some hiccups
– Semaphore support• Planned for V7.3-2next
– poll(), vsnprintf() • Both planned for V7.3-2
– Some trouble with periods in directory names– file and lex utilities not yet implemented in bash
(planned)• Overall, a positive experience
– Partner feels effort to port will be similar to other UNIX® ports
page 47
September 2003 OpenVMS Technical Update
Porting Experiences #2
• Customer needed a solution for printing barcode labels– Simple application, just print the barcode– Very expensive to purchase
• products included more than customer needed
– Found simple, UNIX Open Source application– Downloaded to OpenVMS 7.3-1 with BASH
• Ran build scripts
• Everything worked
• Minimal effort
– Not all will be this easy, but, this demonstrates the goal
page 48
September 2003 OpenVMS Technical Update
More Porting Experiences
• From the GNV developers list (July 2003):– “GNV is working better and better. I could
"./configure" and "make install" the following packages (sometimes with little hacks): • mktemp 1.5 • hostinfo 2.2 • patch 2.5.4 • yacc 1.9.1 • flex 2.5.4 • bison 1.35”
– “Recently I gave a try at making a few unix tools I had troubles building in the past, under the latest GNV bash, and got surprisingly further along, than in the past”
page 49
September 2003 OpenVMS Technical Update
GNV Tip
• Define DECC$PIPE_BUFFER_SIZE 65535 to maximize pipe capabilities
• New feature/parameter DECC$PIPE_BUFFER_QUOTA– Exploits VMS 7.3-1change that increases mailbox
buffer quotas ($crembx:bufquo)– Be careful – given enough BYTLM, processes can
quickly eat up virtual memory
page 50
September 2003 OpenVMS Technical Update
In-House Porting
• Quick summary of our efforts at porting CPIO -
page 51
September 2003 OpenVMS Technical Update
Porting CPIO
• 1. Download the “tarball”– Go to WWW.GNU.ORG– Go to The Free Software Directory– Search for CPIO– Download the tarball: ( http://
ftp.gnu.org/pub/gnu/cpio/ )
page 52
September 2003 OpenVMS Technical Update
Porting CPIO (2)
• 2. Expand the tarball:– Fixup name
• Rename so it’s a “.gz” file
• Set its attributes to the right format– $ set file/att=rfm:stmlf cpio*.gz;/log
– GUNZIP it.$ BASH
BASH> gunzip cpio
BASH> ls -l cpio*
-rwxr--- 1 PITCHER VMS 731136 Aug 8 08:19 cpio-2.5.tar
– UnTar it:BASH> tar -xf cpio-2.5.tar
BASH> ls -l
total 1434
drwxr-x--x 1 PITCHER VMS 2048 Aug 8 10:24 cpio-2.5
-rwxr----- 1 PITCHER VMS 731136 Aug 8 08:19 cpio-2.5.tar
page 53
September 2003 OpenVMS Technical Update
Porting CPIO (cont)
• 3.Build it:bash$ cd cpio-2.5bash$ ./configure
– HANG!!!• Whoops – BASH needs a big pipe – some asynch commands
run in synch, needing a big pipe.
• Easy fix – Define DECC$PIPE_BUFFER_SIZE to 65535
• Also – The script tries to build with “-g” which is debug – – Debug isn’t needed, edit the script get rid of the “-g”s, and try again.
– Worked! – We ended up with a makefile– Run the makefile
page 54
September 2003 OpenVMS Technical Update
CPIO – Compile Errors
• 3. Build (cont)– Whoops!!! Pages and pages of compiler errors:
%CC-E-NEEDINTEXPR, In this statement, "(in_file_stat.st_dev)" has a pointer type, which is not integral. at line number 147 in file STP:[GNUKITS.CPIO.cpio-2^.5]copypass.c;1
– Problem is st_dev in stat structure is char on VMS, short in UNIX standards• UNIX expects integers to identify files and devices –
VMS doesn’t• VMS returns a string pointer!• This needs to be resolved in the code
– However, for this effort, lets kludge it, to get it to compile• Change:
– #define major(dev) (((dev) >> 8) & 0xff)
• To:– #define major(dev) (((*(short *) dev) >> 8) & 0xff)
page 55
September 2003 OpenVMS Technical Update
That was Ugly!!!
• What’s the point Brad???• Point is:
– Its not seamless, yet.– In many cases, its easy to work through.– st_dev was the only code problem, and it took us
minimal time to determine that.– We just learned of one more update we need to
make in the C RTL• After that code change, it compiles, links, and
runs!
page 56
September 2003 OpenVMS Technical Update
Contacts
• OpenVMS C RTL Project Leader:• [email protected]
• OpenVMS UNIX Portability Program Manager:• [email protected]
• Coming soon: UP Web Site–