Upload
jason
View
36
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Software repository replication using the ASIS Local Copy Manager. IT/DIS/OSE, CERN ASIS Team Presented by: German Cancio ([email protected]). Presentation Plan. Introduction to ASIS ASIS structure Transactional model Repository replication using ASISlcm Project status. - PowerPoint PPT Presentation
Citation preview
Software repository replication using the ASIS Local Copy Manager
IT/DIS/OSE, CERN
ASIS TeamPresented by: German Cancio
Presentation PlanPresentation Plan
Introduction to ASIS ASIS structure Transactional model Repository replication using ASISlcm Project status
Software Distribution: Goals and Software Distribution: Goals and RequirementsRequirements
Centralized management of SW applications necessary for: standard and homogeneous user environment better support (less diversity!)
Equally accessible by all supported platforms same availability same configuration
User nodes to see ‘ready to use SW’ trough a distributed file system
Software Distribution at CERN: Software Distribution at CERN: ASISASIS
ASIS: stands for Application Software Installation Service, offering: A multiplatform and multiversion SW repository with ready-to-use
applications A toolsuite for
assisted application compilation and configuration accessing, manipulating and replicating the repository
Contents: > 700 packages for ~8 platforms GNU software (compilers, system tools, editors, desktops) LaTeX X11 and contributions CERN program library Others, like Perl, Tcl/Tk, Java, licensed software
Platforms currently supportedPlatforms currently supported
ASIS (AFS) name Platform
i386_redhat51 RedHat Linux 5.1
sun4x_56 Sun Solaris 2.6
hp_ux102 HP_UX 10.20 ACE3
alpha_dux40 Digital UNIX 4.0b
rs_aix41 IBM AIX 4.1.5
rs_aix43 IBM AIX 4.3.2
sgi_64 SGI IRIX 6.5
Software Distribution at CERN: Software Distribution at CERN: ASIS (II)ASIS (II)
Main tools:
ASIShappi: concurrent software compilation and configuration on multiple platforms
ASISwsm: client workstation access to the repository
ASIStm: checks and validates repository modification requests
ASISlcm: repository update and replication ASIStar: generate tarfiles from ASIS packages
Logical organizationLogical organization
Packages are classified into Families, Products and Versions A family groups together products with similar origin or
functionality (i.e. MAIL, KDE, GNU.EDIT,CERN.LIB) A product may contain one or several versions (emacs-19.34,
emacs-20.2)
Each version is usually available for all supported platforms
‘share’ platform: contains common files (documentation, fonts, scripts)
Logical organization (II)Logical organization (II)
A model has been defined for the software processing from the sources release until delivery to users. Based on a state-transition diagram, 2 phases: generation and
deployment
A deployed package may be on one of the following states: In Production: default version, formally supported Certified: proved to work in the proper environment Under Certification: for evaluation by beta users
ASIS stores InProduction packages in a common area for improved efficiency
Client access to the repositoryClient access to the repository
ASISwsm manages the links from the repository to the workstation (in /usr/local or /cern)
The workstation administrator can customize the access to the packages (i.e. default version, specific, ignore)
Packages can be stored as ‘local copies’
ASIS
ASISwsm
ASISwsm
ASISwsm
ASISwsmASISwsm
ASIS repository
Transaction ModelTransaction Model
In order to group together related repository
modifications, a transactional system is used
A transaction is a sequence of operations... … either
a modification of the state of a package a ‘patch’ operation (modifying a package)
The atomicity of the execution of the transaction is guaranteed.
Transaction: exampleTransaction: example
Description: update emacs & related packages
RemoveFromProduction GNU.EDIT/emacs-19.34 (sun4x_55,alpha_osf32,hp_ux102)
RemoveFromProduction GNU.EDIT/vm-5.96beta (share)
IntroduceInProduction GNU.EDIT/emacs-20.2 (sun4x_55,alpha_osf32,hp_ux102,alpha_dux40)
IntroduceInProduction GNU.EDIT/vm-6.32 (share)
Transaction SystemTransaction System
Based on the cooperation between ASIStm (ASIS
transaction manager) and ASISlcm (ASIS local copy
manager)
A transaction is submitted to ASIStm, which checks its validity and coherence. The transaction is queued if accepted.
Repository modifications are carried out by ASISlcm
ASISlcm: Repository ReplicationASISlcm: Repository Replication
ASISlcm ... Reads transactions which are new since last run Determines what has to be done according to local
configuration Performs locally all selected transactions.
Only new packages are copied
ASISlcm can be run asynchronously, (i.e. at nighttime)
ASISlcm (II)ASISlcm (II)
Fault tolerance: ASISlcm rolls back the currently executed transaction in case of failure (i.e. when network problems arise) Transaction atomicity is always guaranteed.
Star scheme: modifications are propagated to the replicas (master/slave)
ASISlcm runs locally on the replicated site and only accesses the master when necessary
Replication costs proportional to the size of changes
Replication: example Replication: example
ASISASISlcm
CERN AFS master repository
ASIS
ASIS
ASIS
ASISlcm
ASISlcm
ASIStm
CERN NFS replica
Replica 1 on site XXX
Replica 2 on site XXX
WAN file system (AFS)LAN file system (NFS)
LAN file system (NFS)
Full repository replicationFull repository replication
ASISlcm also offers a ‘full’ replication mode State of all packages is checked against the master and updated
accordingly
When to use one mode or the other? Transactional (or incremental) replication:
frequent repository updates, i.e local replicas for load balancing
Full replication: initial repository cloning independent replicas (occasional updates) recovery after crashes
the full replication does not offer atomic properties
ASISlcm: configurationASISlcm: configuration
ASISlcm can be configured to match your site’s needs:
Filtering by architecture list of platforms to be replicated
Filtering by family / product: take all product versions skip a family / product / version ‘freeze’ a family / product / version based on the product’s state (Certified, In Production, Under
Certification)
ASISlcm: configuration exampleASISlcm: configuration example
Architectures:
sun4c_411 sun4x_55 hp700_ux90 hp_ux102 rs_aix41 alpha_osf32 sgi_52 i386_linux2
Products:
<All>/<All> InProduction # Lightweight copy
GNU.LANG/<All> None GNU.LANG/cvs InProduction GNU.LANG/rcs All
GNU.MISC/<All> Frozen GNU.MISC/gzip InProduction
MISC/<All> None MISC/a2ps InProduction MISC/tcsh 6.06 # Specific version nr MISC/arc Frozen
How to set up an ASIS local How to set up an ASIS local copy?copy?
1st. Get the ASIS software by mirroring from the CERN AFS tree or by downloading it from our ftp server
2nd. Configure and run ASISlcm (full) on your server(s)
3rd. Configure and run ASISwsm on your workstations
4th. Decide frequency and update type (full, incremental)
Please contact us for assistance!
ASIS project statusASIS project status
ASIS runs at CERN as part of the standard setup:
> 2000 nodes, > 10000 Users ~ 700 packages on ~ 20 GB
ASISlcm: in production for the CERN NFS server for 18 months on other sites: ???
EPFL (Swiss Federal Institute of Technology, Lausanne): acts as co-developer and owns a private repository
Further documentation and Further documentation and contactscontacts
ASIS has an URL on the web:
http://wwwinfo.cern.ch/dis/asis
with links to:
Guides (not up to date…) Product Documentation Site Usage Statistics Download area
Contacts:
ASIS team at CERN: [email protected]