Embed Size (px)
Citation preview
SUSE ManagerTUT17530 Patching / Troubleshooting / Best Practices
Jack HodgeGTS Backline Engineer
Jeff PricePrincipal Architect
Sascha WeberTechnical Support Engineer
2
Summary – Installation & Patching● Evaluation Keys & ISO download
● Licensing overview
● Installation of SUSE Manager Appliance● /var/log/susemanager_setup.err
/var/log/susemanager_setup.log
● Requirements (RAM, disc space, etc.)
● Installation of patches
● Database schema upgrade
● Installation of PTF (Program Temporary Fix)
● How to run SQL files provided by R&D
● Adding products / repositories
● Custom channels / Clones / Clone-by-date
● Activation keys / bootstrap script
3
Summary – Troubleshooting● Debuggins SUSE Manager (spacewalk-debug)
● Architecture overview / important log files
● Apache2 (access.log / error.log) vs. Tomcat (catalina.out) vs.
rhn (rhn_server_xmlrpc.log)
● /etc/rhn/rhn.conf
● OSAD vs. SSH push
● jabberd
● Repo-sync
● Apache tuning (/etc/apache2/server-tuning.conf)
● Taskomatic
● Configuration Management
● cobbler
4
Summary – Best Practices
● PostgreSQL database backup (smdba backup-hot)
● How to tune PostgreSQL database with pgtune
● Lifecycle Management (patch archive, clone-by-date, etc.)
● SUSE Customer Center migration
● Using multiple mirror credentials
● Service Pack Migration
● Synching channels from SMT
● SUSE Manager Proxy
● SUSE Manager Inter-Server Synchronization
SUSE Manager - Patching
6
Evaluation Key
• https://www.suse.com/products/suse-manager/
• SUSE Customer Center Account needed
• https://scc.suse.com/login
8
Licensing overview
• https://www.suse.com/products/suse-manager/how-to-buy/
9
How to register• During installation in GUI:
• Command line:suse_register -a email=<email> -a regcode-sms=<reg-code>
10
SUSE Manager - Requirements• https://www.suse.com/documentation/suse_manager/book_susemanager_install/data/ch-installation.html
11
SUSE Manager – Ports to open
12
• Stop services via spacewalk-service stop
• via YaST online update
• via zypper: run zypper patch twice
• If the update contained a postgreSQL update it is necessary to restart postgreq before starting the schema update:
rcpostgresql restart
• Perform database schema upgrade via:
spacewalk-schema-upgrade
• Start services via spacewalk-service restart
Please note: If a Kernel update was part of the patching process
a reboot is always neccessary
SUSE ManagerPatching & Database schema upgrade
13
• https://www.suse.com/support/kb/doc.php?id=7016640
• PTF = Program Temporary Fix
• Download the RPMs and install these via rpm -Fvh
• Restart Services via spacewalk-service restart
• Run mgr-sync refresh
• In case R&D has provided you with a SQL script please execute it via spacewalk-sql --select-mode
• If you got a file: spacewalk-sql --select-mode file.sql
• If you have a SQL statement:spacewalk-sql --select-mode - <<< "select *from pg_stat_activity;"
SUSE ManagerPTF installation & SQL execution
Important: do not miss the trailing “-”
14
SUSE ManagerAdding products/channels
• mgr-sync add product or mgr-sync add channel
• Using the Web-Interface → Admin → Setup Wizard → SUSE Products:
15
SUSE ManagerCustom channels / clones / clone-by-date
• spacewalk-manager-channel-lifecycle
• spacecmd softwarechannel_clone
• spacecmd softwarechannel_clonetree
• Web-Interface → Channels → Manage Software Channels
• spacewalk-clone-by-date (part of spacewalk-utils package)
16
SUSE ManagerActivation Keys
• https://wiki.microfocus.com/index.php/SUSE_Manager/System_Activation_Keys
• Avoid using the “SUSE Manager Default” parent channels
• Create one Activation key per distribution & architecture
• Create bootstrap script/Activation Key "pairs" - designed to work together for proper channel, group, and config assignments
• Web Interface → Systems → Activation Keys:
17
SUSE ManagerActivation Keys
• Use something “human readable” for the Key:
18
SUSE ManagerActivation Keys continues
• Select the corresponding “Child Channels”, these should contain (for SLES 12):
SLES-Manager-Tools-Pool
SLE-Manager-Tools-Updats
SLES12-Updates
• Make sure to add additional packages as needed:
19
SUSE ManagerBootstrap
• mgr-create-bootstrap-repo
• Make sure to re-run this command from time to time (or create a cron job) → SUSE Manager 3 will do this itself when there is an update
• Check in path /srv/www/htdocs/pub/repositories/
• mgr-bootstrap to create a new bootstrap script: mgr-bootstrap --activation-keys KEY1,KEY2 \
--gpg-key /srv/www/htdocs/pub/MY_CORPORATE_PUBLIC_KEY \
--allow-config-actions –allow-remote-commands
• Make sure the bootstrap script contains the needed GPG Keys:
ORG_GPG_KEY=suse-9C800ACA.key,res.key,suse-307E3D54.key,suse-39DB7C82.key
• The bootstrap scripts are stored in /srv/www/htdocs/pub/bootstrap
SUSE Manager - Troubleshooting
21
SUSE ManagerDebugging SUSE Manager
• spacewalk-debug This tool collects several pieces of information and stores them in a tarball at /tmp/spacewalk-debug.tar.bz2
For SLES 11 SP3 and greater / SLES 12 the spacewalk-debug is part of the supportconfig utility
• Set debug level in /etc/rhn/rhn.conf:
debug = 5
• To debug spacewalk-repo-sync problems:$> export URLGRABBER_DEBUG=DEBUG
$> /usr/bin/spacewalk-repo-sync --channel <channel-label> --type yum
• To disable the debug mode of spacewalk-repo-sync:$> unset URLGRABBER_DEBUG
22
SUSE ManagerArchitecture overview – important log files
• https://www.suse.com/documentation/suse_manager/singlehtml/book_susemanager_install/book_susemanager_install.html#tb-log-files
• Apache config files: /etc/apache2/conf.d/*.conf
• Apache log files: /var/log/apache2/error.log /var/log/apache2/access.log
• Tomcat config file: /etc/apache2/conf.d/zz-spacewalk-www.conf
• Tomcat log files: /var/log/tomcat6/* /var/log/tomcat6/catalina.out
• Client-Server communication: /var/log/rhn/rhn_server_xmlrpc.log
23
SUSE Managerrhn.conf
• Main SUSE Manager config file is /etc/rhn/rhn.conf
• Set debug levels (e.g. debug=x / osa-dispatcher.debug=5)
• Adjust taskomatic memory (taskomatic.maxmemory=4096)• If you used embedded or external database, rhn.conf will keep the
connection details: db_backend = postgresql db_user = susemanager db_password = *** db_name = susemanager db_host = localhost db_port = 5432
• Forwarding registrations to SCC / NCC:
server.susemanager.forward_registration = 0
24
SUSE ManagerOSAD
• OSAD requires all clients to have different credentials in /etc/sysconfig/rhn/osad-auth.conf
• On a cloned system, make sure to delete the NCCcredentials file first (/etc/zypp/credentials.d)
• Do not install the osad package on the SUSE Manager but only on the client side
osad on the client side
osa-dispatcher on the server side
• Config files are Client: /etc/sysconfig/rhn/up2date & /etc/sysconfig/rhn/osad.conf
Server: osa-dispatcher is configured via /etc/rhn/rhn.conf
25
SUSE Managerjabberd
• If jabberd services fail to start (during spacewalk-service restart):
Starting spacewalk services...
Initializing jabberd processes...
Starting router done
Starting sm startproc: exit status of parent of /usr/bin/sm: 2 failed
Terminating jabberd processes...
Simply delete the jabberd database: spacewalk-service stop rm -Rf /var/lib/jabberd/db/* spacewalk-service start
26
SUSE ManagerSSH Server Push
• For tunneling connections via SSH, two available high port numbers (> 1024) are needed: one is for tunneling HTTP and one for HTTPS (default 1232 and 1233)
• In order to overwrite these, edit values in /etc/rhn/rhn.conf: ssh_push_port_http = <high port 1> ssh_push_port_https = <high port 2>
• By default only 2 simultaneous SSH session are initiated. For larger environments the number of session can be changed with the following option in /etc/rhn/rhn.conf:
taskomatic.ssh_push_workers = 10
27
SUSE ManagerRepo-sync
• spacewalk-repo-sync is scheduled by taskomatic
• Channel sync logs are in /var/log/rhn/reposync
• Taskomatic log is in /var/log/rhn/rhn_taskomatic-daemon.log
• To troubleshoot repo-sync problems it is very helpful to view the HTTP protocol. To achieve this:
$> export URLGRABBER_DEBUG=DEBUG
To manually start repo-sync:
$> /usr/bin/spacewalk-repo-sync --channel <channel-label> --type yum
To disable the debug mode of spacewalk-repo-sync:
$> unset URLGRABBER_DEBUG
28
SUSE ManagerPostgreSQL & Apache tuning
• Only do this if the host has enough memory. Can cause other performance issues!
• Check your # of connections: rhn-db-stats /tmp/db-stats.log | cat /tmp/db-stats.log | grep max_connections
• Increase number of connections with pgtune:
/usr/bin/pgtune -T Mixed -c 200 -i postgresql.conf -o postgresql.conf.pgtune Restart postgres with rcpostgresql restart
• In /etc/apache2/server-tuning.conf:ServerLimit 200 # max number of server processes
MaxClients 200 # max number of requests a server process serves
29
SUSE ManagerPostgreSQL & Apache tuning
• By default taskomatic (the central scheduler for all actions within SUSE Manager) reserves 2 GB of memory.
• In most environments this is simply not enough. As an easy rule: 50% of the available memory should be allocated.
• To allocate more RAM to the taskomatic scheduler edit /etc/rhn/rhn.conf and add:
taskomatic.maxmemory = 4096
• Memory recommendations: Less than 100 systems connecting to SUSE Manager: 8GB Between 100 and 200 systems: 12 GB Above 200 systems: 16 GB When planning to use RedHat Expanded Support: 16 GB
SUSE Manager – Best Practices
31
SUSE ManagerPostgreSQL database backup – restoring backup
● If the backup is running correctly you should now have the following files
in /mnt/backup: ● -rw------- 1 postgres postgres 16777216 Sep 13 09:46 000000010000001000000035
● -rw------- 1 postgres postgres 16777216 Sep 14 01:36 000000010000001000000039
● -rw------- 1 postgres postgres 16777216 Sep 15 05:18 000000010000001000000040
● -rw------- 1 postgres postgres 16777216 Oct 5 02:23 000000010000001000000041
● -rw-r--r-- 1 postgres postgres 1464526433 Jul 22 12:51 base.tar.gz
● drwxr-xr-x 2 postgres postgres 4096 Jul 22 12:35 database
● drwx------ 2 postgres postgres 4096 Jul 22 12:51 tmp
● Use smdba backup-restore to restore to an earlier point in time.
● To restore the backup, proceed as follows:
● Shutdown the database: smbda db-stop
● Start the restore process: smdba backup-restore start
● Restart the database: smbda db-start
● smdba backup-restore force
32
SUSE ManagerPostgreSQL database backup (smdba backup-hot)
● Backup is not automatically active
● Allocate permanent space (on remove storage if needed), e.g.:
● /mnt/backup
● This directory should always be the same, do not change
● Create a directory with the correct permissions:
● sudo -u postgres mkdir /mnt/backup/database
● Alternatively as root: install -d -o postgres /mnt/backup/database
● Or: mkdir /mnt/backup/database & chown postgres:postgres /mnt/backup/database
● For the first time run:● smdba backup-hot –enable=on –backup-dir=/mnt/backup
● Perform hot backup:● smdba backup-hot –backup-dir=/mnt/backup/database
33
SUSE ManagerHow to tune PostgreSQL database with pgtune● We have the tool pgtune installed on SUSE Manager.
Usage: pgtune [options]
Options:
-M TOTALMEMORY, --memory=TOTALMEMORY
Total system memory, will attempt to detect if unspecified
-c CONNECTIONS, --connections=CONNECTIONS
Maximum number of expected connections, default
depends on database type
● So if you want to increated the number of connections:● /usr/bin/pgtune -T Mixed -c 1500 -i postgresql.conf -o postgresql.conf.pgtune
● Recommended values for number of connections are either 200 or 400.
● Stop all spacewalk services and the database, then swap the configuration
files and start everything again:● spacewalk-service stop & rcpostgresql stop
● mv postgresql.conf postgresql.conf.bak
● mv postgresql.conf.pgtune postgresql.conf
● rcpostgresql start & spacewalk-service start
34
SUSE ManagerSUSE Customer Center migration● To be able to get access to the SLES 12 repos you have to migrate from NCC
to SCC (=SUSE Customer Center). There is two options available:
● mgr-ncc-sync –enable-scc
● Web Interface → Admin → SUSE Manager Configuration →
35
SUSE ManagerMultiple Mirror Credentials● With SUSE Manager 2.1 the handling of multiple mirror credentials does not
happy in /etc/rhn/rhn.conf anymore. There is two ways now:
● mgr-sync list credentials
● mgr-sync add credentials
● mgr-sync remove credentials
● Using the Web-Interface → Admin → Setup Wizar → Mirror Credentials:
36
SUSE ManagerService Pack Migration● Supported migration paths are:
● SLES 11 SP1 → SLES 11 SP2 → SLES 11 SP3 → SLES 11 SP4
● SLE 10 SP2 → SLE 10 SP3 → SLE 10 SP4
● Service Pack migration can be done via the Web-Interface:
→ Systems
→ select a system
→ Software
→ SP Migration:
37
SUSE ManagerSynching channels from local SMT ● In the Web-Interfact, create new repository pointing to SMT repo:
● Add this repo to a (custom) channel
38
SUSE ManagerSUSE Manager Proxy● Requirements
● 64bit multicore processor
● 16 GB Memory recommended in production environments
● 20 GB Minimum free disk space
● Additional disk space required depends on amount of channels cached from
SUSE Manager Server
● NTP
● Installation● Installed as an appliance
● Registration to the SUSE Manager Server via bootstrapping
● Proxy Servers can be tiered:
SUSE Manager Server → Proxy 1 (California)→ Proxy 2 (L.A.)→ Proxy 3 (Server Farm 1)
● Proxy 1 bootstrapped to SUSE Manager Server
● Proxy 2 bootstrapped to Proxy 1
● Proxy 3 bootstrapped to Proxy 2
39
SUSE ManagerSUSE Manager Proxy
40
SUSE ManagerSUSE Manager Proxy
41
SUSE ManagerSUSE Manager Proxy● Benefits
● Efficiency and bandwidth ● Offload file updates taking load off of SUSE Manager Server
● Packages are delivered significantly faster over a local area network
● 2000 - 5000 Clients per SUSE Manager Proxy Server
● Scalability
● Proxy Servers can be built from inexpensive hardware and/or Vms
● Troubleshooting● Slow file performance, missing files, etc
● Clear the proxy cache
spacewalk-proxy stop
rm -rf /var/cache/squid/*
squid -z
spacewalk-proxy start
42
SUSE ManagerSUSE Manager Proxy● Troubleshooting (cont.)
● Metadata errors● Disable caching of metadata on proxy:
Edit the file /etc/squid/squid.conf on the proxy and change the following line:
refresh_pattern /XMLRPC/GET-REQ/.*/repodata/.*\.xml.*$ 0 1% 525960
to
refresh_pattern /XMLRPC/GET-REQ/.*/repodata/.*\.xml.*$ 0 0% 0
* Note: This will cause increased network traffic
● Check NTP settings
43
SUSE ManagerInter-Server Synchronization
● https://www.suse.com/documentation/suse_manager/book_susemanager_install/data/s1-sync-iss.html
● Web interface → Admin → ISS Configuration → Master Setup
Slave Fully Qualified Domain Name
● Allow Slave to Sync? ● Choosing this field will allow the slave SUSE Manager to access this master
● Sync all orgs to Slave? ● Checking this field will synchronize all organizations to the slave SUSE Manager.
44
SUSE ManagerInter-Server Synchronization Continued
● Perform the corresponding steps on the Slave
● Use mgr-inter-sync command to sync channels:sumaslave:~ # mgr-inter-sync -l
19:25:35 SUSE Manager - live synchronization
19:25:35 url: https://suma21.weber.dus
19:25:35 debug/output level: 1
19:25:35 db: susemanager/<password>@susemanager
19:25:36 sles12-pool-x86_64:
19:25:36 . sle-manager-tools12-pool-x86_64 34 full import from Wed Sep 16 04:56:22 2015
19:25:36 . sle-manager-tools12-updates-x86_64 77 full import from Wed Sep 16 04:56:23 2015
19:25:36 . sle-sdk12-pool-x86_64 2290 full import from Tue Sep 15 04:56:38 2015
19:25:36 . sle-sdk12-updates-x86_64 536 full import from Wed Sep 16 04:56:21 2015
19:25:36 . sle-we12-pool-x86_64 926 full import from Wed Sep 16 04:56:22 2015
19:25:36 . sle-we12-updates-x86_64 278 full import from Wed Sep 16 04:56:22 2015
19:25:36 . sles12-updates-x86_64 1348 full import from Wed Sep 16 04:56:22 2015
● To add channel run mgr-inter-sync -c <channelname>
● Enable the inter-server synchronization in the /etc/rhn/rhn.conf:
disable_iss=0
● Make sure to restart httpd service: service httpd restart
Advanced Patch Lifecycle Management
46
SUSE Manager – Motivation...Advanced Patch Lifecycle Management
• Most company security policies are based on a public, published, regulation that requires “compliance”
• Compliance normally dictates a process of “proofs” or approved exceptions – aka compensating controls
• The process of proof requires a flexible tool, a documented and repetitive set of procedures, and people to perform them
47
SUSE Manager - Common RequestsAdvanced Patch Lifecycle Management
There are some common requests from our customers.•
“How can I … ???”
...Automatically create and archive “patch sets” by quarter (or any other time period)...
...Leverage a consistent method of patch promotion and delivery through numerous landscapes and environments...
...Develop an exception process for handling patches that need to be excluded from a patch cycle...
...Create an test/lab environment using historical patch sets...
...remove the need of host channel subscription manipulation from cradle-to-grave...
...do service-pack migration using my own custom child channels and current patch sets...
48
SUSE Manager – How do you roll-out?Advanced Patch Lifecycle Management
● Most companies have “landscapes” like DEV, TEST, QA, UAT, Pre-Prod, Sandbox, or similar
● Patch deployment often requires avetting process to validateeffectiveness and reduce risk
● This is typically – deploy, evaluateand promote...
● Rinse & Repeat
49
SUSE Manager – Process FlowAdvanced Patch Lifecycle Management
●
50
SUSE Manager – Channel SetsAdvanced Patch Lifecycle Management
Here is an example “Channel Set”:
Base : SLES 12 Pool for x86_64DEV - Current Patch Set - SLES 12 Updates for x86_64QA - Current Patch Set - SLES 12 Updates for x86_64PROD - Current Patch Set - SLES 12 Updates for x86_64Patch Exceptions - SLES 12 x86_64Security ASAP Exceptions - SLES 12 x86_64
They can be clone-sets of the SUSE Channels – including a prefix- like a company name.
Repeat for each version of SLES you have...
51
SUSE Manager – Channel Sets (cont.)Advanced Patch Lifecycle Management
From the UI:
52
SUSE Manager – The Missing Link...Advanced Patch Lifecycle ManagementThe “Merge” Command:#!/usr/bin/pythonimport xmlrpclibimport sysimport getpass
MANAGER_URL = "https://suma01.chameleoncorp.com/rpc/api"MANAGER_LOGIN = raw_input("Please Enter the SUSE Manager Login Name: ")MANAGER_PASSWORD = getpass.getpass("Please Enter the Password: ")
MERGE_SOURCE = raw_input("Enter the SOURCE channel label to Merge FROM: ")MERGE_TARGET = raw_input("Enter the TARGET channel label to Merge INTO: ")
print("This tool is going to take all packages and errata from the SOURCE")print("Channel : " + MERGE_SOURCE)print("and merge it into the TARGET ")print("Channel : " + MERGE_TARGET)...client = xmlrpclib.Server(MANAGER_URL, verbose=0)key = client.auth.login(MANAGER_LOGIN, MANAGER_PASSWORD)
client.channel.software.mergePackages(key, MERGE_SOURCE, MERGE_TARGET)client.channel.software.mergeErrata(key, MERGE_SOURCE, MERGE_TARGET)
client.auth.logout(key)
53
SUSE Manager – More Info...Advanced Patch Lifecycle Management
Grab a copy of the new document:
“Advanced Patch Lifecycle Management with SUSE Manager”
Full Descriptions, Examples, Scripts, Automation, PICTURES!!, etc.
54
More System Management @SUSECon
• CAS18158: How to use SUSE Manager and CVEs
• FUT20721: SUSE Manager Roadmap
• HO20098: Install and Configure SMT and SUSE Manager for Dummies
• TUT18400: Architecting your SUSE Manager Deployment
• TUT20514: SaltStack and SUSE
• TUT20516: Using SUSE Manager in Heterogeneous Environments
• TUT20829: Implementation of a SUSE-based Solution with SUSE Manager at Apollo-Optik
Thank you.
55
QUESTIONS?
Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany
+49 911 740 53 0 (Worldwide)www.suse.com
Join us on:www.opensuse.org
56
Unpublished Work of SUSE LLC. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.
