The FusionInventory projectguillomovitch/presentations/fosdem2012.pdf · GLPI PHP plugin multi-platform Perl agent Guillaume Rousse The FusionInventory project. Project Coding challenges

ProjectCoding challengesTesting challenges

The FusionInventory projectA perl hacker perspective

Guillaume Rousse

FOSDEM 2012

Guillaume Rousse The FusionInventory project


Outline

1 ProjectHistoryAgent

2 Coding challengesLegacy codeEndless code traps

3 Testing challengesTesting HTTP server interactionsTesting system interactionsTesting modules compilation everywhere



HistoryAgent

Outline






HistoryAgent

Plan






HistoryAgent

GLPI

OverviewIT assets inventory solution

inventoryhelpdesk

ArchitecturePHP applicationmodular



HistoryAgent

GLPI

OverviewIT assets inventory solution

inventoryhelpdesk

ArchitecturePHP applicationmodular



HistoryAgent

OCS-NG

OverviewAutonomous IT assets inventory solution

inventorysoftware deployment

ArchitectureServer side:

Perl agent interfacePHP user application

Agent side:Windows C agentUnix Perl agent



HistoryAgent

OCS-NG

OverviewAutonomous IT assets inventory solution

inventorysoftware deployment

ArchitectureServer side:

Perl agent interfacePHP user application

Agent side:Windows C agentUnix Perl agent



HistoryAgent

Tracker

OverviewGLPI extension to deal with agentless devices

device discoveryremote inventory

ArchitecturePHP GLPI pluginPerl agent



HistoryAgent

Tracker

OverviewGLPI extension to deal with agentless devices

device discoveryremote inventory

ArchitecturePHP GLPI pluginPerl agent



HistoryAgent

Yesterday



HistoryAgent

FusionInventory

OverviewGeneric GLPI extension

merge of Unix OCS agent with Tracker agentmodular design

ArchitectureGLPI PHP pluginmulti-platform Perl agent



HistoryAgent

FusionInventory

OverviewGeneric GLPI extension

merge of Unix OCS agent with Tracker agentmodular design

ArchitectureGLPI PHP pluginmulti-platform Perl agent



HistoryAgent

Today



HistoryAgent

Plan






HistoryAgent

Presentation

Componentsenginetasks

Task sampleslocal inventorynetwork discoveryremote network inventoryremote vmware inventorywake on lansoftware deployment



HistoryAgent

Presentation

Componentsenginetasks

Task sampleslocal inventorynetwork discoveryremote network inventoryremote vmware inventorywake on lansoftware deployment



HistoryAgent

Local inventory task

Hardware componentsCPU, memory, etc...USB, PCI, SCSI busesconnected devices

Software componentsoperating systeminstalled software

Configuration

network configurationdisplay configurationenvironment variables



HistoryAgent




Configuration




HistoryAgent




Configuration




HistoryAgent

Network discovery

Network scanSNMP scannmap scannetbios scan

Device identificationSNMP fingerprinting



HistoryAgent

Network discovery

Network scanSNMP scannmap scannetbios scan

Device identificationSNMP fingerprinting



HistoryAgent

Network inventory

Printersink levelpage counters

Networking deviceports listvlans listconnected devices



HistoryAgent

Network inventory

Printersink levelpage counters

Networking deviceports listvlans listconnected devices



HistoryAgent

Server communication

Legacy OCS protocol

HTTP POST requestsXML format

FusionInventory protocolHTTP GET requestsJSON format



HistoryAgent

Server communication

Legacy OCS protocol

HTTP POST requestsXML format

FusionInventory protocolHTTP GET requestsJSON format



HistoryAgent

Not just GLPI

Other management solutionsRudderUranosOTRSPulse2



HistoryAgent

Timeline

Versions2.0.x initial release2.1.x windows support2.2.x refactoring

roadmapswitch from XML to JSONswitch to POEmemory footprint reduction



HistoryAgent

Timeline

Versions2.0.x initial release2.1.x windows support2.2.x refactoring

roadmapswitch from XML to JSONswitch to POEmemory footprint reduction



Legacy codeEndless code traps

Outline







Plan







Sysadmin-style code

Look ’ma, I’m a shell guru

# HPUX sof tware l i s t e x t r a c t i o n@sof tL is t = ‘ s w l i s t | grep −v ’ ^ PH ’ | grep −v ’ ^# ’ | t r −s " \ t " " " | t r −s " " ‘

What’s this syslog thing anyway ?

set logsock ( ’ un ix ’ ) ;openlog ( " fus ion inven to ry−agent " , ’ cons , p id ’ , $ENV{ ’USER ’ } ) ;sys log ( ’ debug ’ , ’ sys log backend enabled ’ ) ;c lose log ( ) ;




Sysadmin-style code

Look ’ma, I’m a shell guru

# HPUX sof tware l i s t e x t r a c t i o n@sof tL is t = ‘ s w l i s t | grep −v ’ ^ PH ’ | grep −v ’ ^# ’ | t r −s " \ t " " " | t r −s " " ‘

What’s this syslog thing anyway ?

set logsock ( ’ un ix ’ ) ;openlog ( " fus ion inven to ry−agent " , ’ cons , p id ’ , $ENV{ ’USER ’ } ) ;sys log ( ’ debug ’ , ’ sys log backend enabled ’ ) ;c lose log ( ) ;




Creative hacks

Dynamic function export

my $backendSharedFuncs = {can_run => sub {

. . .} ,

} ;

foreach my $package ( @packages ) {foreach my $func ( keys %{$backendSharedFuncs } ) {

$package−>{$func } = $backendSharedFuncs−>{$func } ;}

}




Plan







Portability concerns

Contextthe agent runs an HTTP listener threadit forks itself to perform its duty

Code

# h t t p server main thread codesub DESTROY {

$_[0]−>{ l i s t e n e r }−> k i l l ( ’ KILL ’ ) i f $_[0]−>{ l i s t e n e r } ;}

# h t t p server l i s t e n e r thread codethreads−>se t_ th read_ex i t_on l y ( 1 ) ;$SIG { ’ KILL ’ } = sub { threads−>ex i t ( ) } ;

Trapfork() is emulated on Windows: the listener thread is lost






Code










Code








Underlying code concerns

Contextthe network discovery task is multithreadedeach thread reports its result to the GLPI server

TrapOpenSSL is not thread-safe: crash with HTTPS




Underlying code concerns

Contextthe network discovery task is multithreadedeach thread reports its result to the GLPI server

TrapOpenSSL is not thread-safe: crash with HTTPS




LWP HTTPS support

Multiple SSL socket implementationsNet::SSL: certification authorityIO::Socket::SSL: certification authority, server hostname

Multiple LWP versionsLWP 5.x: no server certificate checkingLWP 6.x: server certificate checkingLWP 5.x on fedora: LWP 6.x behaviour backport




LWP HTTPS support

Multiple SSL socket implementationsNet::SSL: certification authorityIO::Socket::SSL: certification authority, server hostname

Multiple LWP versionsLWP 5.x: no server certificate checkingLWP 6.x: server certificate checkingLWP 5.x on fedora: LWP 6.x behaviour backport



Testing HTTP server interactionsTesting system interactionsTesting modules compilation everywhere

Outline







Plan







Overview

GoalTo test interactions with an HTTP server

Examplesproxy usageSSL usagecertificates checking




Overview

GoalTo test interactions with an HTTP server

Examplesproxy usageSSL usagecertificates checking




Using a real server

Prosall features available

Conslimited availabilitylimited variability




Using a real server

Prosall features available

Conslimited availabilitylimited variability




Running a test server

Prosfull availabilityfull variability

Conslimited features

Used modulesHTTP::Server::SimpleHTTP::Server::Simple::AuthenIO::Socket::SSLHTTP::Proxy


















Mocking the user agent

Test::MockObjectNot tested




Plan







Testing command output parsing

GoalTesting the code parsing some command output

Initial code

my @output = ‘command ‘ ;foreach my $ l i n e ( @output ) {

i f ( $ l i n e =~ / ^ foo : ( . ∗ ) / ) {$ inventory−>addItem ( $1 ) ;

}}




Testing command output parsing

GoalTesting the code parsing some command output

Initial code

my @output = ‘command ‘ ;foreach my $ l i n e ( @output ) {

i f ( $ l i n e =~ / ^ foo : ( . ∗ ) / ) {$ inventory−>addItem ( $1 ) ;

}}




Abstraction layer

getFileHandle

sub getF i leHandle {my %params = @_;

my $handle ;

SWITCH: {i f ( $params { f i l e } ) {

i f ( ! open $handle , ’ < ’ , $params { f i l e } ) {carp "Can ’ t open f i l e $params { f i l e } : $ERRNO" ;return ;

}l as t SWITCH;

}i f ( $params {command } ) {

i f ( ! open $handle , ’−| ’ , $params {command} . " 2 >/dev / n u l l " ) {carp "Can ’ t run command $params {command } : $ERRNO" ;return ;

}l as t SWITCH;

}die " n e i t h e r command nor f i l e parameter given " ;

}

return $handle ;}




Code adaptation

Intermediate code

$inventory−>addItems ( $_ ) foreach get I tems ( ) ;

sub get I tems {my @output = ‘command ‘ ;my @items ;foreach my $ l i n e ( @output ) {

push @items , $1 i f $ l i n e =~ / ^ foo : ( . ∗ ) / ;}return @items

}




Code adaptation

Final code

$inventory−>addItems ( $_ ) foreach get I tems (command => ’command ’ ) ;

sub get I tems {my $handle = getF i leHandle (@_) ;while (my $ l i n e = <$handle >) {

push @items , $1 i f $ l i n e =~ / ^ foo : ( . ∗ ) / ;}close $handle ;return @items ;

}




Result availability concerns

Exotic files and commandsAIX lsvpd

HPUX machinfo

Linux Alpha /proc/cpuinfo

Samples collecting campaingresources treeorganisational concerns (tracability, ...)

Windows issuesregistryWMI






HPUX machinfo









HPUX machinfo







Testing environment adaptation

GoalTesting the code selecting the command to run

Initial code

my @packages =−x ’ / b in / rpm ’ ? getRPMPackagesList (command => ’ rpm −qa ’ ) :−x ’ / b in / dpkg ’ ? getDPKGPackagesList (command => ’ dpkg − l ’ ) :−x ’ / b in / equery ’ ? getEqueryPackagesList (command => ’ equery l i s t − i ’ ) :

( ) ;}




Testing environment adaptation

GoalTesting the code selecting the command to run

Initial code

my @packages =−x ’ / b in / rpm ’ ? getRPMPackagesList (command => ’ rpm −qa ’ ) :−x ’ / b in / dpkg ’ ? getDPKGPackagesList (command => ’ dpkg − l ’ ) :−x ’ / b in / equery ’ ? getEqueryPackagesList (command => ’ equery l i s t − i ’ ) :

( ) ;}




Abstraction layer

canRun

sub canRun {my ( $wanted ) = @_;

return −x $wanted ;}




Code adaptation

Final code

my @packages =canRun ( ’ / b in / rpm ’ ) ? getRPMPackagesList (command => ’ rpm −qa ’ ) :canRun ( ’ / b in / dpkg ’ ) ? getDPKGPackagesList (command => ’ dpkg − l ’ ) :canRun ( ’ / b in / equery ’ ) ? getEqueryPackagesList (command => ’ equery l i s t − i ’ ) :

( ) ;}




Function hijacking

mockGetRun

sub mockCanRun {my (%params ) = @_;

my $new = sub {my $wanted = $_ [ 0 ] ;return $params {commands}−>{$wanted } ;

} ;

no warnings ’ rede f i ne ’ ;∗Fus ionInventory : : Agent : : Tools : : canRun = $new ;

}




Function hijacking

mockGetFileHandle

sub mockGetFileHandle {my (%params ) = @_;

my $old = \& Fus ionInventory : : Agent : : Tools : : ge tF i leHandle ;

my $new = sub {my (%opt ions ) = @_;

my $ f i l e = $params {commands}−>{$wanted } ;

i f ( $ f i l e ) {pr in t STDERR " f i l e ’ $ f i l e ’ de l i ve red \ n " ;return $old−>(@_, f i l e => $ f i l e ) ;

} else {pr in t STDERR " noth ing de l i ve red \ n " ;return ;

}} ;

no warnings ’ rede f i ne ’ ;∗Fus ionInventory : : Agent : : Tools : : ge tF i leHandle = $new ;

}




Mocking a full environment

Code

package Fus ionInventory : : Test : : MockSystem : : Debian ;

use Fus ionInventory : : Test : : MockSystem ;

mockSystem (commands => {

’ dpkg ’ => ’ resources / packaging / dpkg ’} ,f i l e s => {

’ / e tc / debian_vers ion ’ => ’ resources / re lease / debian ’}

) ;

Usage

$> p e r l −MFusionInventory : : Test : : MockSystem : : Debian f u s i o n i n v e n t o r y




Mocking a full environment

Code

package Fus ionInventory : : Test : : MockSystem : : Debian ;

use Fus ionInventory : : Test : : MockSystem ;

mockSystem (commands => {

’ dpkg ’ => ’ resources / packaging / dpkg ’} ,f i l e s => {

’ / e tc / debian_vers ion ’ => ’ resources / re lease / debian ’}

) ;

Usage

$> p e r l −MFusionInventory : : Test : : MockSystem : : Debian f u s i o n i n v e n t o r y




Plan







Compilation test

Test::More

use Test : : More ;

use_ok ( ’Some : : Module ’ ) ;use_ok ( ’ Another : : Module ’ ) ;

Test::Compile

use Test : : Compile ;

a l l _pm_f i l es_ok ( ) ;




Compilation test

Test::More

use Test : : More ;

use_ok ( ’Some : : Module ’ ) ;use_ok ( ’ Another : : Module ’ ) ;

Test::Compile


a l l _pm_f i l es_ok ( ) ;




Module availability issues

Unix

package Some : : Module ;

use Win32 : : T ieReg is t r y ;

Windows

package Another : : Module ;

use Sys : : Syslog ;




Module availability issues

Unix

package Some : : Module ;

use Win32 : : T ieReg is t r y ;

Windows

package Another : : Module ;

use Sys : : Syslog ;




Workaround

Filtering


my @fi les = $OSNAME eq ’MSWin32 ’ ?grep { ! / Another / } a l l _ p m _ f i l e s ( ’ l i b ’ ) :grep { ! /Some/ } a l l _ p m _ f i l e s ( ’ l i b ’ ) ;

a l l _pm_f i l es_ok ( @f i les ) ;

Loading modules at runtime

use UNIVERSAL : : require ;

Win32 : : T ieReg is t ry−>require ( ) ;Win32 : : T ieReg is t ry−>import ( ) ;




Workaround

Filtering


my @fi les = $OSNAME eq ’MSWin32 ’ ?grep { ! / Another / } a l l _ p m _ f i l e s ( ’ l i b ’ ) :grep { ! /Some/ } a l l _ p m _ f i l e s ( ’ l i b ’ ) ;

a l l _pm_f i l es_ok ( @f i les ) ;

Loading modules at runtime

use UNIVERSAL : : require ;

Win32 : : T ieReg is t ry−>require ( ) ;Win32 : : T ieReg is t ry−>import ( ) ;




Mocking module, trivial case

No exported symbol

use Win32 : : OLE : : Const ;

Testing code

$INC { ’ Win32 /OLE/ Const .pm ’ } = 1 ;




Mocking module, moderate case

Simple export

use Win32 : : OLE qw( i n CP_UTF8 ) ;

Mock module

package Win32 : : OLE;

use base ’ Expor ter ’ ;

our @EXPORT = qw( i n CP_UTF8 ) ;

1




Mocking module, complex case

Complex export

use Win32 : : T ieReg is t r y (D e l i m i t e r => ’ / ’ ,ArrayValues => 0 ,qw /KEY_READ/

) ;

Mock module

package Win32 : : T ieReg is t r y ;

our $Regis t ry ;

sub import {my $ca l lpkg = c a l l e r ( ) ;no s t r i c t ’ r e f s ’ ;

∗{ " $ca l lpkg \ : : Reg is t ry " } = \ $Regis t ry ;∗{ " $ca l lpkg \ : : KEY_READ" } = sub { } ;

}

1




Using mock modules

Testing code

i f ($OSNAME eq ’MSWin32 ’ ) {push @INC, ’ t / fake / un ix ’ ;

} else {push @INC, ’ t / fake / windows ’ ;

}




Conclusion

Contactswww.fusioninventory.org#fusioninventory on [email protected]

Questions ?




Conclusion

Contactswww.fusioninventory.org#fusioninventory on [email protected]

Questions ?


Documents

The FusionInventory projectguillomovitch/presentations/fosdem2012.pdf · GLPI PHP plugin multi-platform Perl agent Guillaume Rousse The FusionInventory project. Project Coding challenges