Linux Agents Using OpsMgr 2012 - Kevin Holman's System Center Blog - Site Home - TechNet Blogs (3)

Deploying Unix/Linux Agents using OpsMgr 2012

11Kevin Holman 18 Mar 2012 12:41 PM

Microsoft started including Unix and Linux monitoring in OpsMgr directly in OpsMgr 2007 R2, which shipped in 2009. Some significant updates have been made to this for OpsMgr 2012. Primarily these updates are around:

Highly available Monitoring via Resource PoolsSudo elevation support for using a low priv account with elevation rights for specific workflows.ssh key authenticationNew wizards for discovery, agent upgrade, and agent uninstallationAdditional Powershell cmdletsPerformance and scalability improvementsNew monitoring templates for common monitoring tasks

This article will cover the discovery, agent deployment, and monitoring configuration of a Linux server in OpsMgr2012. I am going to run through this as a typical user would – and show some of the pitfalls if you don’t follow theexact order of configuration required.

So what would anyone do first? They’d naturally run a discovery, just like they do for Windows agents. However – thiswill likely end up in frustration. There are several steps that you need to configure FIRST, before deploying Unix/Linuxagents.

High Level Overview:

The high level process is as follows:

Import Management PacksCreate a resource pool for monitoring Unix/Linux serversConfigure the Xplat certificates (export/import) for each management server in the pool.Create and Configure Run As accounts for Unix/Linux.Discover and deploy the agents

Import Management Packs:

The core Unix/Linux libraries are already imported when you install OpsMgr 2012, but not the detailed MP’s for eachOS version. These are on the installation media, in the \ManagementPacks directory. Import the specific ones for theUnix or Linux Operating systems that you plan to monitor.

Create a resource pool for monitoring Unix/Linux servers

The FIRST step is to create a Unix/Linux Monitoring Resource pool. This pool will be used and associated withmanagement servers that are dedicated for monitoring Unix/Linux systems in larger environments, or may includeexisting management servers that also manage Windows agents or Gateways in smaller environments. Regardless, it isa best practice to create a new resource pool for this purpose, and will ease administration, and scalability expansion inthe future.

Under Administration, find Resource Pools in the console:

Deploying Unix/Linux Agents using OpsMgr 2012 - Kevin Holman's Syst... http://blogs.technet.com/b/kevinholman/archive/2012/03/18/deploying-uni...

1 of 20 23/03/2014 3:30 PM

OpsMgr ships 3 resource pools by default:

Let’s create a new one by selecting “Create Resource Pool” from the task pane on the right, and call it “Unix LinuxMonitoring Resource Pool”

Click Add and then click Search to display all management servers. Select the Management servers that you want toperform Unix and Linux Monitoring. If you only have 1 MS, this will be easy. For high availability – you need at leasttwo management servers in the pool.

Add your management servers and create the pool. In the actions pane – select “View Resource Pool Members” toverify membership.


2 of 20 23/03/2014 3:30 PM

Configure the Xplat certificates (export/import) for each management server inthe pool

This process is documented here: http://technet.microsoft.com/en-us/library/hh287152.aspx

Operations Manager uses certificates to authenticate access to the computers it is managing. When the DiscoveryWizard deploys an agent, it retrieves the certificate from the agent, signs the certificate, deploys the certificate back tothe agent, and then restarts the agent.

To configure high availability, each management server in the resource pool must have all the root certificatesthat are used to sign the certificates that are deployed to the agents on the UNIX and Linux computers.Otherwise, if a management server becomes unavailable, the other management servers would not be able to trust thecertificates that were signed by the server that failed.

We provide a tool to handle the certificates, named scxcertconfig.exe. Essentially what you must do, is to log on toEACH management server that will be part of a Unix/Linux monitoring resource pool, and export their SCX (cross plat)certificate to a file share. Then import each others certificates so they are trusted.

If you only have a SINGLE management server, or a single management server in your pool, you can skipthis step, then perform it later if you ever add Management Servers to the Unix/Linux Monitoring resourcepool.

In this example – I have two management servers in my Unix/Linux resource pool, MS1 and MS2. Open a commandprompt on each MS, and export the cert:

On MS1:

C:\Program Files\System Center 2012\Operations Manager\Server>scxcertconfig.exe -export\\servername\sharename\MS1.cer

On MS2:

C:\Program Files\System Center 2012\Operations Manager\Server>scxcertconfig.exe -export\\servername\sharename\MS2.cer

Once all certs are exported, you must IMPORT the other management server’s certificate:

On MS1:

C:\Program Files\System Center 2012\Operations Manager\Server>scxcertconfig.exe –import\\servername\sharename\MS2.cer

On MS2:

C:\Program Files\System Center 2012\Operations Manager\Server>scxcertconfig.exe –import\\servername\sharename\MS1.cer

If you fail to perform the above steps – you will get errors when running the Linux agent deployment wizard later.


3 of 20 23/03/2014 3:30 PM

Create and Configure Run As accounts for Unix/Linux

Next up we need to create our run-as accounts for Linux monitoring. This is documented here: http://technet.microsoft.com/en-us/library/hh212926.aspx

We need to select “UNIX/Linux Accounts” under administration, then “Create Run As Account” from the task pane. Thiskicks off a special wizard for creating these accounts.

Lets create the Monitoring account first. Give the monitoring account a display name, and click Next.


4 of 20 23/03/2014 3:30 PM

On the next screen, type in the credentials that you want to use for monitoring the Linux system(s).

On the above screen – you have two choices. You can provide a privileged account for handling monitoring, or youcan use an existing account on the Linux system(s) that is not privileged. Then – you can specify whether or not youwant this account to be able to leverage sudo elevation. Since I am providing a privileged account in this case – I willtell it to not use elevation.

On the next screen, always choose more secure:

Now – since we chose More Secure – we must choose the distribution of the Run As account. Find your “LinuxMonitoring Account” under the UNIX/Linux Accounts screen, and open the properties. On the Distribution Securityscreen, click Add, then select "Search by resource pool name” and click search. Find your Unix/Linux monitoringresource pool, highlight it, and click Add, then OK. This will distribute this account credential to all Managementservers in our pool:


5 of 20 23/03/2014 3:30 PM

We would repeat the above process, as many times as necessary for the number of different accounts we need. If allour Linux systems use the same credentials, then we need at a minimum, ONE monitoring account that is privileged,and it can be associated to the three Run As Profiles (covered in next section).

However, what would be more typical, if all our systems had the same credentials and passwords, is to use THREE RunAs accounts:

One for for Unprivileged (do not use elevation) monitoringOne for Privileged monitoring using EITHER a priv account (do not use elevation), OR a unpriv account usingsudo (use elevation)One for Agent Maintenance using EITHER a priv account (do not use elevation), OR a unpriv account using sudo(use elevation)

For the purposes of this demo, I am just going to create a SINGLE priv Run As account (root) that I will use for all threescenarios.

Next up – we must configure the Run As profiles. This is covered here: http://technet.microsoft.com/en-us/library/hh212926.aspx

There are three profiles for Unix/Linux accounts:

The agent maintenance account is strictly for agent updates, uninstalls, anything that requires SSH. This will always be


6 of 20 23/03/2014 3:30 PM

associated with a privileged account that has access via SSH, and was created using the Run As account wizard above,but selecting “Agent Maintenance Account” as the account type. We wont go into details on that here.

The other two Profiles are used for Monitoring workflows. These are:

Unix/Linux Privileged account

Unix/Linux Action Account

The Privileged Account Profile will always be associated with a Run As account like we created above, that is Privileged(root or similar) OR a unprivileged account that has been configured with elevation via sudo. This is what anyworkflows that typically require elevated rights will execute as.

The Action account is what all your basic monitoring workflows will run as. This will generally be associated with a RunAs account, like we created above, but would be used with a non-privileged user account on the Linux systems.

***A note on sudo elevated accounts:

sudo elevation must be passwordless.requiredtty must be disabled for the user.

For my example – I am keeping it very simple. I created a single Run As account, of the Monitoring type, which is theprivileged root account and password credential. I will associate this Run As account to BOTH the Privileged andAction account. This will make all my workflows (both normal monitoring and elevated monitoring) run under thiscredential. This is not recommended as the “lowest priv” design, but being leveraged in this example just to keepthings simple. Once we validate it is working, we can go back and change this configuration and experiment using lowpriv and sudo enabled elevation accounts, and associate them independently.

For more information on configuring sudo elevation for OpsMgr monitoring accounts, including somesample configurations for your sudoers files for each OS version: http://social.technet.microsoft.com/wiki/contents/articles/7375.configuring-sudo-elevation-for-unix-and-linux-monitoring-with-system-center-2012-operations-manager.aspx

I will start with the Unix/Linux Action Account profile. Right click it – choose properties, and on the Run AsAccounts screen, click Add, then select our “Linux Monitoring Account”. Leave the default of “All Targeted Objects”and click OK, then save.

Repeat this same process for the Unix/Linux Privileged Account profile.

Repeat this same process for the Unix/Linux Agent Maintenance Account profile.

Discover and deploy the agents

Run the discovery wizard.


7 of 20 23/03/2014 3:30 PM

Click “Add”:

Here you will type in the FQDN of the Linux/Unix agent, its SSH port, and then choose All Computers in the discoverytype. ((We have another option for discovery type – if you were manually installing the Unix/Linux agent (which isreally just a simple provider) and then using a signed certificate to authenticate))

Now – hit “Set Credentials”. If we do not want to provide a root account here, and wanted to use SSH keyauthentication, we support that on this screen now. For this example – I will simply type in my root account in order touse SSH to discover and deploy the Linux agent.


8 of 20 23/03/2014 3:30 PM

Notice above that you can tell the wizard if the account is privileged or not. Here is an explanation:

A privileged account is a user account that has root-level access, including access to security logs and read,write, and execute permissions for the directories in which the Operations Manager agent is installed.An unprivileged account is a normal user account that does not have root-level access or special permissions.However, an unprivileged account allows monitoring of system processes and of performance data.

If you have to discover only UNIX and Linux computers that already have an agent installed, rather than installing anagent, you can use an unprivileged user account on the UNIX or Linux computer. If you have to install an agent, youmust use a privileged account. If you do not have a privileged account, you can elevate an unprivileged account to aprivileged account provided that the su or sudo elevation program has been configured on the UNIX or Linuxcomputer for the user account.

So – if we had pre-installed the agent already – we could simply use an unprivileged account to authenticate anddiscover the system, bringing it into OpsMgr.

Or – we could provide an unprivileged account that was allowed elevation via a pre-existing sudo configuration on theLinux server.


9 of 20 23/03/2014 3:30 PM

Click save. On the next screen – select a resource pool. We will choose the resource pool that we already created.


10 of 20 23/03/2014 3:30 PM

Click Discover, and the results will be displayed:

Check the box next to your discovered system – and deploy the agent.

This will take some time to complete, as the agent is checked for the correct FQDN and SSL certificate, the managementservers are inspected to ensure they all have trusted SCX certificates (that we exported/imported above) and theconnection is made over SSH, the package is copied down, installed, and the final certificate signing occurs. If all ofthese checks pass, we get a success!

There are several things that can fail at this point. See the troubleshooting section at the end of this article.

Monitoring Linux servers:

Assuming we got all the way to this point with a successful discovery and agent installation, we need to verify thatmonitoring is working. After an agent is deployed, the Run As accounts will start being used to run discoveries, andstart monitoring. Once enough time has passed for these, check in the Administration pane, under Unix/LinuxComputers, and verify that the systems are not listed as “Unknown” but discovered as a specific version of the OS:


11 of 20 23/03/2014 3:30 PM

Next – go to the Monitoring pane – and select the “Unix/Linux Computers” view at the top. Look that your systems arepresent and there is a green healthy check mark next to them:

Next – expand the Unix/Linux Computers folder in the left tree (near the bottom) and make sure we have discoveredthe individual objects, like Linux Server State, Linux Disk State, and Network Adapter state:

Run Health explorer on one of the discovered disks. Remove the filter at the top to see all the monitors for the disk:


12 of 20 23/03/2014 3:30 PM

Close health explorer.

Select the Operating System Performance view. Review the performance counters we collect out of the box for eachmonitored OS.

Out of the box – we discover and apply a default monitoring template to the following objects:

Operating SystemLogical diskNetwork Adapters

Optionally, you can enable discoveries for:

Individual Logical ProcessorsPhysical Disks

I don’t recommend enabling additional discoveries unless you are sure that your monitoring requirements cannot bemet without discovering these additional objects, as they will reduce the scalability of your environment.

Out of the box – for an OS like RedHat Enterprise Linux 5 – here is a list of the monitors in place, and the object theytarget:


13 of 20 23/03/2014 3:30 PM

There are also 50 rules enabled out of the box. 46 are performance collection rules for reporting, and 4 rules are eventbased, dealing with security. Two are informational letting you know whenever a direct login is made using rootcredentials via SSH, and when su elevation occurs by a user session. The other two deal with failed attempts for SSH orSU.

To get more out of your monitoring – you might have other services, processes, or log files that you need to monitor. For that, we provide Authoring Templates with wizards to help you add additional monitoring, in the Authoring paneof the console under Management Pack templates:

In the reporting pane – we also offer a large number of reports you can leverage, or you can always create your ownusing our generic report templates, or custom ones designed in Visual Studio for SQL reporting services.


14 of 20 23/03/2014 3:30 PM

As you can see, it is a fairly well rounded solution to include Unix and Linux monitoring into a single pane of glass foryour other systems, from the Hardware, to the Operating System, to the network layer, to the applications.

Partners and 3rd party vendors also supply additional management packs which extend our Unix and Linux monitoring,to discover and provide detailed monitoring on non-Microsoft applications that run on these Unix and Linux systems.

Troubleshooting:

The majority of troubleshooting comes in the form of failed discovery/agent deployments.

Microsoft has written a wiki on this topic, which covers the majority of these, and how to resolve:

http://social.technet.microsoft.com/wiki/contents/articles/4966.aspx

For instance – if your DNS name that you provided does not match the DNS hostname on the Linux server, ormatch it’s SSL certificate, or if you failed to export/import the SCX certificates for multiple management serversin the pool, you might see:


15 of 20 23/03/2014 3:30 PM

Agent verification failed. Error detail: The server certificate on the destination computer(rh5501.opsmgr.net:1270) has the following errors:The SSL certificate could not be checked for revocation. The server used to check for revocation might beunreachable.

The SSL certificate is signed by an unknown certificate authority.It is possible that:1. The destination certificate is signed by another certificate authority not trusted by the managementserver.2. The destination has an invalid certificate, e.g., its common name (CN) does not match the fully qualifieddomain name (FQDN) used for the connection. The FQDN used for the connection is: rh5501.opsmgr.net.3. The servers in the resource pool have not been configured to trust certificates signed by other serversin the pool.

The server certificate on the destination computer (rh5501.opsmgr.net:1270) has the following errors:The SSL certificate could not be checked for revocation. The server used to check for revocation might beunreachable.The SSL certificate is signed by an unknown certificate authority.It is possible that:1. The destination certificate is signed by another certificate authority not trusted by the managementserver.2. The destination has an invalid certificate, e.g., its common name (CN) does not match the fully qualifieddomain name (FQDN) used for the connection. The FQDN used for the connection is: rh5501.opsmgr.net.3. The servers in the resource pool have not been configured to trust certificates signed by other serversin the pool.

The solution to these common issues is covered in the Wiki with links to the product documentation.

Perhaps – you failed to properly configure your Run As accounts and profiles. You might see the following showas “Unknown” under administration:

Or you might see alerts in the console:

Alert: UNIX/Linux Run As profile association error event detected

The account for the UNIX/Linux Action Run As profile associated with the workflow"Microsoft.Unix.AgentVersion.Discovery", running for instance "rh5501.opsmgr.net" with ID {9ADCED3D-B44B-3A82-769D-B0653BFE54F9} is not defined. The workflow has been unloaded. Please associate anaccount with the profile.

This condition may have occurred because no UNIX/Linux Accounts have been configured for the Run Asprofile. The UNIX/Linux Run As profile used by this workflow must be configured to associate a Run Asaccount with the target.


16 of 20 23/03/2014 3:30 PM

Either you failed to configure the Run As accounts, or failed to distribute them, or you chose a low priv account that isnot properly configured for sudo on the Linux system. Go back and double-check your work there.

If you want to check if the agent was deployed to a RedHat system, you can provide the following command in a shellsession:

Comments

Rohit Kochher 19 Mar 2012 4:16 PM

Thanks for the great post.

It is valuable and amazing as always.

cchelten 2 Jul 2012 7:21 PM

Agreed this document helped out so much. I appreciate the share.

Dominique 11 Oct 2012 8:11 AM

Excellent document again....

I do not see any Resouce Pools in my environment even I have about 100 Cross Platform agent runningmonitored and alerting properly, is it normal?

Is it only SCOM 2012?

As I have VMs managed in SCOM through nWorks from Veeam/VMware using the vCenter what is the maindifference in the informtion provided by the two ways to managed the VM/Linux machine?

Thanks,

DOm

barry 4 Mar 2013 10:36 PM

we are getting below error when using a privileged account

Failed during SSH discovery. Exit code: 1

Standard Output: Sudo path: /usr/bin/

Standard Error: sudo: no tty present and no askpass program specified

Exception Message:

Dote78 28 May 2013 5:32 PM

Hello

Thanks for the info. I succesfully configured a number of linux using the same credentials for monitoring.But I'm trying to add now another group of linux boxes with a different set of credentials (as I dont want to


17 of 20 23/03/2014 3:30 PM

share a privileged account between all my servers) and when trying to add the credentials to the profile, the"All targeted objects" option is not available anymore.

Is it possible to monitor several Linux machines using different SUDOer accounts for each one?

Thanks a lot

Fran

Dote78 28 May 2013 5:36 PM

Oh, nevermind my last post. The solution is already out there :)

social.technet.microsoft.com/.../27d1983a-96d2-4900-8730-0a9522d870b4

Again, thanks for the great post :)

Raksha 18 Jul 2013 9:31 AM

HI Kevin,

Could you please help me in including the SCOM R2 agent in our Server Template.

Is it there any power shell script to install manually.

We are planning to automate the SCOM agent installation by adding the agent to Template. But we havedifferent SCOM gateway server for different domain.

P Lease help if there any solution for this.

Thanks,

Raksha

lee cooper 26 Jul 2013 11:43 AM

hi kevin

i followed your instuctions as detailed above but i seem to be getting the following error i dont know if imissed something out what do you think

Failed to sign kit. Exit code: 1

Standard Output: Failed to start child process '/sbin/init.d/scx-cimd' errno=13

RETURN CODE: 1

Standard Error: cp: cannot create /etc/opt/microsoft/scx/ssl/scx.pem: Permission denied

Exception Message:

Geet 17 Aug 2013 5:51 AM

Hi Kevin,

Do you have the Solution to fix the below error:

Failed to sign kit. Exit code: 1

Standard Output: Failed to start child process '/sbin/init.d/scx-cimd' errno=13

RETURN CODE: 1


18 of 20 23/03/2014 3:30 PM

ACS agents Cluster Connectors Database grooming Hotfix Hyper-V Links Management Pack MP-AD

MPAuthoring MP-DNS MP-Exchange MP-SharePoint MP-SQL Notifications QuickStartGuides Reporting ResourcePools Security

Tools TSQL UI Console Web Console

March 2014 (5)

February 2014 (5)

January 2014 (14)

November 2013 (3)

October 2013 (9)

September 2013 (3)

August 2013 (1)

July 2013 (1)

June 2013 (2)

May 2013 (6)

April 2013 (5)

March 2013 (1)

February 2013 (2)

January 2013 (8)

December 2012 (5)

November 2012 (2)

October 2012 (1)

September 2012 (6)

August 2012 (2)

July 2012 (2)

June 2012 (6)

May 2012 (2)

April 2012 (3)

March 2012 (3)

February 2012 (2)

December 2011 (1)

November 2011 (2)

October 2011 (3)

do you need an agent on the AIX system 31 Oct 2013 7:28 PM

I know how to get the .cert to the unix system (aix) but dont they need an agent on their side and where doyou get it from?

Radha 6 Feb 2014 8:38 AM

i am trying to discover and install agent into Linux machines throught SCOM 2012 . Below is the error i amfacing ...please someone help em Failed to sign kit. Exit code: 1 Standard Output: Failed to start childprocess '/etc/init.d/scx-cimd' errno=13 RETURN CODE: 1 Standard Error: cp: cannot create regular file`/etc/opt/microsoft/scx/ssl/scx.pem': Permission denied Exception Message:

Search BlogsOptionsTags

Archive


19 of 20 23/03/2014 3:30 PM

September 2011 (3)

August 2011 (7)

July 2011 (4)

May 2011 (2)

April 2011 (3)

March 2011 (5)

February 2011 (9)

January 2011 (2)

December 2010 (3)

November 2010 (5)

October 2010 (6)

September 2010 (5)

August 2010 (8)

July 2010 (9)

June 2010 (3)

May 2010 (1)

April 2010 (7)

March 2010 (4)

February 2010 (5)

January 2010 (2)

December 2009 (2)

November 2009 (9)

October 2009 (3)

September 2009 (5)

August 2009 (3)

July 2009 (8)

June 2009 (9)

May 2009 (3)

April 2009 (7)

March 2009 (5)

February 2009 (9)

January 2009 (3)

November 2008 (4)

October 2008 (4)

September 2008 (5)

August 2008 (3)

July 2008 (9)

June 2008 (10)

May 2008 (2)

April 2008 (5)

March 2008 (5)

February 2008 (9)

December 2007 (9)

November 2007 (1)

October 2007 (2)


20 of 20 23/03/2014 3:30 PM

Documents

Linux Agents Using OpsMgr 2012 - Kevin Holman's System Center Blog - Site Home - TechNet Blogs (3)