Packaging Perl...

London Perl Workshop 2010Packaging Perl and dependencies

Paulo CastroPerl Developer

Net-A-Porter

Introduction

• Paulo Edgar Castro aka (G@SP@R)• Portuguese• System Administrator• Perl Developer ( writing baby perl since 1999 )• Arrived in London in August 2007

• Came to learn and to be closer to technology• No perl jobs in Porto, a lot of proprietary tech• MobileStreams, RedBeeMedia, Net-A-Porter

Brief Summary

• Where we were / Who we were• The problems of such approach• The decision• Investigation and Initial steps• Questions• Tools (part I)• Problems and pitfalls• Tools (part II)• Getting closer• More problems

Brief Summary

• Evil Hacks• Wrapping it up• Pros/Cons• Final thoughts

Where we were / Who we were

• 64 bit perl 5.8.8 compiled for CentOS 5.2• GIT repository of perl binaries and CPAN modules

checked out into standard dir (prefix=/opt/zt)• export PATH=/opt/zt/zt-perl/bin:$PATH• Developers using different OS'es Arch's ( Ubuntu 32,

Ubuntu 64, Fedora, OSX, Gentoo, Windows ?!?!? QUACK !!! )

• cpan My::Evil::Module; git commit -am "Install/Update My::Evil::Module"

• Deployment consisted of App tarball + GIT checkout tarball

The problems of such approach

• Very tricky for developers to update modules which build .so libs

• .so linked against developer libs which were more recent than the libs present in CentOS 5.2

• Only a rough idea of the modules we were currently using• Production/Staging/Testing boxes had to be manually pre-

installed with all the dependencies (libjpg,libpng,libgd,libdb,...)

• Trial and error deployment into new boxes ( "Oh, we forgot to install libpng, that's why the bar-codes aren't being generated!" )

• Only ran tests for modules being installed/updated

The decision

• We needed a way to describe all these related dependencies

• I suggested RPM since I was very familiar with it ( My first job in London consisted of packaging up their perl app in a nice RPM package )

• Our current production system was RedHat• We didn't seem to really have any other viable options to

accomplish this• Our ultimate goal was, to be able to deploy our apps into a

minimal installed box and have the whole process figure out things for us rather than the other way around.

LET'S GO!

Investigation and Initial steps

• We knew more or less what we wanted• Not entirely sure what the end result was going to be• We needed to know what we had (CPAN modules

inventory)• 2 Apps, 2 Makefile.PL with deps, merged together yielded

235 CPAN packages• Grand total of 618 CPAN modules to satisfy the deps of the

235.• Dug through CPAN, PerlMonks, Google for answers ....• cpanspec ?• cpan2rpm ?

Questions

• 1 perl bin rpm package + n perl modules packages ?

• (this is overkill, who's going to maintain 618 rpm builds)• 1 perl bin rpm package + 1 perl modules package ?• 1 perl bin + modules package ?

• How will we handle our 618 sources ?• CPAN mirror ?• Customized CPAN repo of the modules we use ?

• How are we going to manage the unattended install of 618 packages ?!?!? 'Oh dear... :( Frack, let me check jobs.perl.org'

Tools (part I)

• pmtools, to generate us an inventory of what we hadText::CSV_XS (0.76) - comma-separated values manipulation routines

Compress::Bzip2 (2.09) - Interface to Bzip2 compression library

PPI (1.213) - Parse, Analyze and Manipulate Perl (without perl)

XML::Parser::Expat (2.19) - Lowlevel access to James Clark's expat XML parser

XML::Parser (2.19) - A perl module for parsing XML documents

• A couple of bash scripts to do some simple greping and parsing of the results ( I know this is a perl workshop, but bash one-liners seemed good enough for the job )

• Mark Overmeer's CPAN::Site, quote "The cpansite script is used to create your own CPAN server." - GREAT STUFF!

• How the frack are we going to create an unattended install of the CPAN deps ? 618 of them ?

Tools (part I)

• I remembered how we were in some cases installing new modules... ( a way I'd used before I confess ... )

• Add the modules to the project Makefile.PL which is based in Module::Install

• perl Makefile.PL; make; echo “Module installed”• What about we assume perl is a project with a Makefile.PL full of

dependencies ?

Tools (part I)

Typical Makefile.PL layout

use strict; use warnings; use inc::Module::Install 0.98; use Module::Install::AuthorRequires; use Module::Install::ExtraTests; use 5.008003;

name 'Moose';perl_version '5.008003';all_from 'lib/Moose.pm';license 'perl';

requires 'Carp';requires 'Class::MOP' => '1.11';requires 'Data::OptList' => '0';requires 'List::MoreUtils' => '0.12';requires 'Package::DeprecationManager' => '0.10';requires 'Params::Util' => '1.00';requires 'Scalar::Util' => '1.19';requires 'Sub::Exporter' => '0.980';requires 'Sub::Name' => '0';requires 'Task::Weaken' => '0';requires 'Try::Tiny' => '0.02';

test_requires 'Test::More' => '0.88';test_requires 'Test::Fatal' => '0.001';test_requires 'Test::Requires' => '0.05';...........

Problems and pitfalls

• Getting the sources for the modules• wget ${Module}-${Version}.tar.gz from cpan• if (!$found) { wget ${Module}-${Version}.tar.gz from backpan }

else { echo "Damn it, GRRRR, $#$%#$%/#$%#$ " }• Some module versions just couldn't be found anywhere.

• Solution: Install the closest higher version and hope that everything will be OK

• Other modules had been tweaked in house and their version bumped manually to say (0.78_02) not matching anything found in the wild

• This meant, the module was actually a couple versions below (0.78) and it had a in-house patch applied.

Problems and pitfalls

• Some perl modules had deps on third-party libraries like mysql-devel, db4-devel, openssl-devel, postgresql-devel, libxml2-devel, etc...

• Others, Math::Pari, depended on specific tarballs to be installed

• Some modules didn't pass some tests• Others didn't pass any test at all....

• How the frack did they got installed in the first place ???

“Oh Lordy!!! Is it time to check jobserve this time ?”

Tools (part II)

• RPM spec files• They're just a clever wrapper around a tarball of stuff built in

a temporary directory• With this wrapper we can

• Specify build requirements• Specify lib dependencies• Provide information about our package• Describe a recipe for the build/installation process• Changelog of the actions

• Just before creating the final package, foreach existing file, rpmbuild lists:• What the file is providing/requiring

Tools (part II)

Provides Example:

Byte.so()(64bit) ByteLoader.so()(64bit) Bzip2.so()(64bit) C.so()(64bit) Opcode.so()(64bit) POSIX.so()(64bit) mysql.so()(64bit) perl(Algorithm::C3) = 0.07 perl(Algorithm::Diff) = 1.19 perl(Algorithm::Diff::_impl) perl(Catalyst::Log) perl(Catalyst::Log::Log4perl) = 1.00 perl(Catalyst::Model) perl(Catalyst::Model::ActiveMQ) perl(Catalyst::Model::DBIC::Schema) = 0.29 perl(Catalyst::Model::DBIC::Schema::Types) perl(Catalyst::Model::File) = 0.08 perl(DBD::mysql) = 4.007 perl(DBD::mysql::GetInfo) perl(DBD::mysql::db) perl(Moose) >= 0.82 perl(PPI::Token::Pod) perl(PPI::Token::Prototype) .....

Requires Example:

libc.so.6(GLIBC_2.4)(64bit) libcrypt.so.1()(64bit) libgd.so.2()(64bit) libgmp.so.3()(64bit) libjpeg.so.62()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libmysqlclient.so.16()(64bit) libnsl.so.1()(64bit) libpng12.so.0()(64bit) .....

Tools (part II)

Summary: A searchable xml/html based mailinglist archiverName: lurkerVersion: 1.1Release: 1.evo.2Group: Applications/InternetLicense: GPLURL: http://lurker.sourceforge.net/Packager: Jim Perrin <perrin@ohio.edu>Source0: lurker-%{version}.tar.gzSource1: mimelib-3.1.1.tar.gzBuildRequires: zlib-develRequires: libxsltBuildRoot: %{_tmppath}/%{name}-%{version}-root%descriptionLurker is not just another mailing list archiver. Blá blá blá%prep%setup -a 1%build./configure --with-mimelib-local --prefix=$RPM_BUILD_ROOT --localstatedir=$RPM_BUILD_ROOT/make %{?_smp_mflags}

%install%makeinstallinstall -d -m0755 $RPM_BUILD_ROOT/%{_sysconfdir}install -m0755 lurker.conf $RPM_BUILD_ROOT/%{_sysconfdir}/lurker.confinstall -d -m2775 $RPM_BUILD_ROOT/%{_localstatedir}/www/lurkdb%cleanrm -rf $RPM_BUILD_ROOT%files%defattr(-,root,root)%doc AUTHORS ChangeLog COPYING FAQ NEWS README INSTALL%{_mandir}/man1/*%{_bindir}/*%{_libdir}/*%config(noreplace) %{_sysconfdir}/lurker.conf%defattr(-,apache,apache)%{_localstatedir}/www/lurker/%{_localstatedir}/www/lurkdb/%changelog* Fri Jan 2 2004 Jim Perrin <perrin@ohio.edu>- Changed the spec file to better use macros

Example spec file

Tools (part II)

• CPAN Distroprefs (Configuration for individual distributions)• A distropref file per badly behaved package• Define which tests to run, excluding the ones that fail and

with which we're ok• Pass additional arguments to one of the four commands• Set environment variables• Instantiate an Expect object that reads from the console,

waits for some regular expressions and enters some answers• Temporarily override assorted CPAN.pm configuration

variables• Specify dependencies the original maintainer forgot• Disable the installation of an object altogether

• Examples:

Tools (part II)

DB_File.yml---match: distribution: "/DB_File-\d"pl: env: DB_FILE_INCLUDE: "/usr/include" DB_FILE_LIB: "/usr/lib64"

Convert-PEM-0.07.yml---comment: "skip test t/01-readwrite.t, i got fed up of it failing randomly sometimes without any noticeable explanation."

match: distribution: "/Convert-PEM-\d"

test: args: - TEST_FILES="t/00-compile.t t/02-encode.t t/03-ede3.t"

Tools (part II)

Alien-ActiveMQ-0.000003.yml ---comment: Module doesn't explicitly require the dependency below, so it fails miserably when it tried to require it.match: distribution: "/Alien-ActiveMQ-0.00003"depends: requires: File::Copy::Recursive: 0patches: - "MYCOMPANY/patches/Alien-ActiveMQ.patch"

Term-ReadLine-Gnu.yml---comment: "Skip the entire test suite - hudson doesn't run with a real /dev/tty"match: distribution: "/Term-ReadLine-Gnu-1"test: args: - TEST_FILES="" expect: - "Skip the test suite?" - "Yes\n"

Getting Closer

• Merge projects Makefile.PL creating a single Makefile.PL with everything

• Create our CPAN site with our packages, patches, and our distroprefs files

• Create a rigged CPAN/Config.pm based on sane defaults• 'urllist' => [q[file:///home/pecastro/build/CPANSITE]],• 'make_arg' => q[-j 3]• 'connect_to_internet_ok' => q[0]

Info for the next slides....• %new_perl is an RPM variable that means

• %{new_perl_flags} $RPM_BUILD_ROOT/%{perl_bin_dir}/perl• %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE• %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Getting Closer

• Main recipe so far:• Extract perl sources• %perl_base_dir = /opt/zt/zt-perl• Configure -Dprefix=%perl_base_dir; make; • make install DESTDIR=~/rpmbuild/opt/zt/zt-perl ( this will

install us new perl )• export PERL5LIB,PERLLIB,PATH to match temporary

installation dir "~/rpmbuild/opt/zt/zt-perl/{bin,lib}"• Copy our rigged CPAN/Config.pm into the right place• %new_perl -MCPAN -e 'install Bundle::CPAN' ( so we have a

sane CPAN client that can do our bidding )• %new_perl -MCPAN -e 'install Module::Install' ( so we can

use our auto magical unattended installation process outsourcing dependency sorting to CPAN )

Getting Closer

• export PERL_MM_USE_DEFAULT=1 ( This prevents automated processes from blocking on user input. )

• copy Makefile.PL into cwd• %new_perl Makefile.PL --defaultdeps• make

• Installs all the modules defined in Makefile.PL and their dependencies as well auto-magically....

Getting Closer

This is excellent. All we need now is to wrap up all of this in the SPEC

file and presto.... perl RPM with everything we need ! ROCK ON...

More problems

• When deployed, our custom perl, will live in /opt/zt/zt-perl/bin

• What happens when we're building a perl binary & friends in a system which already has a custom perl installed ?

• Kaboom !!! But Why Kaboom ?• our ~/rpmbuild/opt/zt/zt-perl/bin/perl is being built to live

in /opt/zt/zt-perl/bin/perl and even though it's being run from a temporary location, has it's INC paths pointing to it's final location /opt/zt/zt-perl/lib ... hardcoded in the perl binary.

• When we say: • %new_perl Makefile.PL --defaultdeps && make ;

• It replies: Oh, I can see that I already have everything installed. Nothing else to do..... :(

Evil Hacks

• After having built perl and before starting auto magical modules installation

• Temporarily hack Config.pm and Config_heavy.pl

%define archlib $RPM_BUILD_ROOT/%{perl_lib_dir}/%version/%{perl_archname}

%new_perl -pi -e "\

s|%perl_base_dir|$RPM_BUILD_ROOT%perl_base_dir|g" \

$RPM_BUILD_ROOT/%archlib/Config.pm \

$RPM_BUILD_ROOT/%archlib/Config_heavy.pl

And now...........The most EVIL HACK of all hacks......

Evil Hacks

%perl_base_dir = /opt/zt/zt-perl

garbled_base_dir=`\%new_perl -e '$_=”%perl_base_dir”;s|/|%|g;print' `

%new_perl -pi -e "\s|%perl_base_dir|$mess_base_dir|g" \ $RPM_BUILD_ROOT%{perl_bin_dir}/perl

REMEMBER.....●%new_perl is an RPM variable that means

● %{new_perl_flags} $RPM_BUILD_ROOT%{perl_bin_dir}/perl● %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE● %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Evil Hacks

Characteristics of this binary (from libperl): Compile-time options: PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO Built under linux Compiled at Dec 1 2010 20:52:24 %ENV: PERL5LIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8:/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8" PERLLIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8" @INC: /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8/x86_64-linux /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8 /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8/x86_64-linux /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl .[pecastro@brutalix perl-zap-5.8.8]$ new_perl -VTCan't locate Config.pm in @INC (@INC contains: %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl).BEGIN failed--compilation aborted

Wrapping it up

• RPM specifics• BuildRequires: postgresql-devel expat-devel openssl-devel gd-

devel graphviz jdk , etc .... to specify all the third-party dependencies

• %prep stage used to do custom validation and basic unpacking of sources

• Example: mysql -utest -ptest -e 'use test' || echo "echo \"GRANT ALL PRIVILEGES ON test.* to test@localhost identified by 'test'\" |mysql -uroot" && exit 1

• %build stage to configure and build perl • %install stage to install it and do all the magical Makefile.PL bits• Make sure you undo your Evil (Config.pm, Config_heavy.pl, perl)• Amend our rigged CPAN/Config.pm

• o conf build_dir /tmp/.cpan/build ; • o conf make_install_make_command 'sudo make'• o conf urllist http://mirrorservice.org/sites/ftp.cpan.org/pub/CPAN

Pros/Cons

• Pros• Easily deployed package with dependency management• Easy to spot missing packages ( perspective of dependent

package )• Fast spotting of badly behaved modules

• Module install/test failure means no build• Enabled us to replicate environments like if they were

mushrooms• Cons

• Module updates not as straightforward as before• Tricky CPAN module dependency

• Build time tied to CPU (84 min @ Virtual Double-Core Xeon 2.33GHz/4GB mem)

Final Thoughts

• I'd like to thank my colleagues for their input and feedback• Recommended experiment• Most likely to be possible to replicate the same approach in

other Linux/Unix flavors (Debian,Gentoo,...)• The initial process is tedious but worth going through• Once everything is streamlined there's no thinking just

building on top of...• I've heard of other strategies to do deployment but

honestly.....• This is currently the one that makes more sense to me

END {say “QUESTIONS ?”;say “Paulo Castro”;say “<pauloedgarcastro@gmail.com>”;close $SLIDE_FILE;

}