Canonical Rsyslog CentralLogging v4 20090901 03

7/21/2019 Canonical Rsyslog CentralLogging v4 20090901 03

1/33


2/33


3/33

Table o! (ontents

Overview.........................................................................................................2

Introduction....................................................................................................5

Logging models..............................................................................................6

). Sinle system *to #is&+..............................................................................................................,

2. Multiple systems *to #is&+..........................................................................................................,

-. Multiple systems *to #atabase+..................................................................................................

/. Branch o!!ices *remote storae+................................................................................................

Technical considerations to central logging................................................9

1etwor& loin reliability..............................................................................................................9

atabase loin...........................................................................................................................9

T3S connections............................................................................................................................9

Logging software.........................................................................................1

!etting started with rsyslog........................................................................11

%nstallation...................................................................................................................................))

(on!iuration structure................................................................................................................)2

4ules5actions...............................................................................................................................)2

Output !ile syncin.......................................................................................................................)/

Timestamps.................................................................................................................................)/

Templates....................................................................................................................................)/

Property6base# !ilters..................................................................................................................),

7ueue processin.......................................................................................................................)

Central logging scenarios...........................................................................1"

Multiple systems *to #is&+............................................................................................................)

Multiple systems *to #atabase+....................................................................................................)

Branch o!!ices *remote storae+..................................................................................................20

On the (erti!icate 8uthority......................................................................................................20

On the loin server...............................................................................................................2)

On a loin client...................................................................................................................22

#dvanced $syslog features a%%lica&le to central logging.......................2'

BS6style bloc&s.........................................................................................................................2-

3oin "ueues...........................................................................................................................2-is& 7ueues.............................................................................................................................2-

(entralise# loin with rsyslo - www.canonical.com


4/33

%n6Memory 7ueues..................................................................................................................2/

ybri# is&68ssiste# %n6Memory 7ueues................................................................................2/

7ueuein an# #e6"ueuein.........................................................................................................2/

3oin "ueue e:amples............................................................................................................2;

3ocal #is& loin.....................................................................................................................2;

4emote #is& loin.................................................................................................................2;

4emote #atabase loin........................................................................................................2;

iscar# watermar&s.....................................................................................................................2,

#%%endi( #) $eferences and useful Lin*s.................................................2+

#%%endi( ,) rsyslog.conf - syslog.conf diff...............................................2"

#%%endi( C) essage %ro%erties................................................................'

#%%endi( /) 0ro%erty o%tions.....................................................................'2

(entralise# loin with rsyslo / www.canonical.com


5/33

%ntro#uction

ect assumes a whole new #imension. %n lare

oranisations$ where the number o! computer systems can rane in the thousan#s$ there is the

tas& o! manain such loin #ata. ?eoraphically #iverse branch o!!ices brin another

element to the mi:. @inally$ los play a vital role when a system has been compromise# by an

e:ternal *or internal+ hostile aent.

This white paper also tries to a##ress how a company technically manaes the potentially hue

volume o! los its computer systems enerate.

Other "uestions #eservin o! serious consi#eration but which are not covere# by this technical

paper areA

8uthorisation*i.e. whoshoul# have access to such los+

3eally$ how !ar bac& in the past must a company retain its los *particularly when

manain client #ata+

The so!tware that is covere# in this #ocument is rsyslog. Possible alternatives are the stoc&

3inu:5'ni: syslogsystem or syslog-ng. This paper #escribes the reasons !or the choice o!

rsyslo in the section Logging Softwarean# provi#e technical caveats an# bac&roun#

in!ormation in the section Technical Considerations and Historical Background.

This paper is not an intro#uction to the !iel# o! system loin. See Appendix A, "The ns and

!uts of Syste Logging #sing Syslog"!or the basics.

Note$ at the tie of pu%lication, #%untu &'() *karic koala+ is in alpha and uses rsyslog as its

default tool for logging, replacing sysklogd that was the preious default ' The analysis

perfored for this white paper is what triggered this change'

(entralise# loin with rsyslo ; www.canonical.com


6/33

3oin mo#els

This section surveys several typical architectural mo#els o! computer system loin.

1. ingle system to dis*3

%n#ivi#ual computer systems$ by #e!ault$ per!orm loin. Messaes typically et written to the

local har# #rive but 1etwor& 8ttache# Storae *18S+ or Storae 8rea 1etwor& *S81+ are also

vali# storae options !or this mo#el.

2. ulti%le systems to dis*3

Cnown as central logging$ many systems !orwar# their los over the networ& to a central loin

server. 8naloous to the sinle6system mo#el$ on the server6si#e$ messaes et written to the

local har# #rive or to some other available storae.

(entralise# loin with rsyslo , www.canonical.com


7/33

'. ulti%le systems to data&ase3

8 common option is to have the remote messaes store# #irectly into a #atabase on the server

with$ possibly$ a web6base# inter!ace actin as a viewin5"uery tool.

The #atabase nee# not resi#e on the loin server *as shown in the #iaram+D it can be place#

onto a separate system.

(entralise# loin with rsyslo www.canonical.com


8/33

4. ,ranch offices remote storage3

We continue the loical proression where multiple branch o!!ices are each implementin the E2

or E- mo#el. Their central loin servers now relay their los to a secon#6level central loin

architecture *typically resi#in at the company hea# o!!ice or #ata centre+. The !act that

sensitive in!ormation is bein transporte# over a non6truste# networ& *here the internet+ is a vital

!acet that nee#s to be a##resse# by your company=s security team.

(entralise# loin with rsyslo www.canonical.com


9/33

Technical consi#erations to central loin

etwor* logging relia&ility

Tra#itional 'ni: syslo uses the 'P protocol. This is unsuitable !or central5networ&

loin #ue to the protocol=s lossy5unreliable nature. 8lternative so!tware such as

syslo6n an# rsyslo inclu#e support !or the T(P protocol. This is a reat

improvement but there remains nonetheless a reliability issue even with T(P.

Thousan#s o! messaes can be lost i! the networ& connection with the loin server

brea&s as there is no mechanism in T(P that noti!ies the sen#er imme#iately *its sen#

bu!!er continues to !ill up+. The rsyslo pro>ect is currently #evelopin a truly reliable

loin protocolA 4


10/33

3oin so!twareThe rsyslogtool was chosen over the more popular syslog-ng!or the !ollowin reasonsA

). Licensing and software features

Syslo6n is #ual6license#. 8 commercial pro#uct has been !or&e# !rom the open6

source *?P3+ pro>ect an# the more a#vance# !eatures are !oun# only in the

commercial o!!erin. 8!!ecte# !eatures o! import so !ar are i+ native T3S5SS3 support

*i.e. not usin stunnel+ an# ii+ on6#is& spoolin o! messaes. %t=s un&nown how these

!or&s will #ivere in the !uture.

2. Truly relia&le message delivery $7L03

4syslo is con!rontin the unreliability o! T(P in a loin environment throuh the

#evelopment o! the 4


11/33

?ettin starte# with rsyslo

This section coversA

%nstallation

(on!iuration structure

4ules5actions

Timestamps

Templates

Properties6base# !ilters


12/33

Configuration structure

(on!iuration !iles are structure# in the !ollowin mannerA

Mo#ules

?lobal #irectives

@ilter rules

8ll mo#ules an# lobal #irectives nee# to be speci!ie# one per line an# must start with a #ollar6

sin *H+. They a!!ect all rules.

$ules-actions

4ules consist tra#itionally o! =selector action= *where selector consists o! =facility.%riority=+.This metho# has been retaine# !rom reular sys&lo because they are e!!ective but also !or

bac&war# compatibility with sys&lo con!iuration !iles. owever$ rsyslo provi#es other uni"ue

an# power!ul metho#s o! buil#in rules as we=ll see.

The !acility an# priority are #e!ine# in 4@( -),/. ere is a summaryA

acilities

1umerical (o#e Ceywor# @acility

0 &ern Cernel

) user 4eular user processess

2 mail Mail system

- #aemon System #aemons

/ auth Security *authentication an# authorisation+ relate#

comman#s

; syslo Syslo internal messaes

, lpr 3ine printers system

news 11TP subsystem

uucp ''(P subsystem

)0 authpriv Private authorisation messaes

),62- local06 Site speci!ic use

(entralise# loin with rsyslo )2 www.canonical.com


13/33

0riorities

1umerical (o#e Ceywor# @acility

0 emer


14/33

remember thatA

8n action .ueueis create# each time an action is speci!ie#.

8ction "ueue paraetersare reset a!ter an action "ueue has been create# *allowin

the creation o! a new action "ueue an# its correspon#in parameters+.

Out%ut file syncing

ue to per!ormance #era#ation$ rsyslo no loner retains sys&lo=s #e!ault o! !ile syncin )i!

not speci!ie# otherwise *by placin a #ash in !ront o! the output !ile name+.


15/33

Where =!rom(har= an# =to(har= are character a##resses. These enable us to bein an# en# a

property=s value at certain places *e:A )A2 are the !irst two characters in the value o! thespeci!ie# property+. Property options are liste# in 8ppen#i: .

We apply this template to messaes by associatin it with the #e!ault template !or !ile action *we

can #o the same !or !orwar#in5networ& action+A

$5ction#ile'eault(emplate templatename

e!ault !orwar#in templates use# with 'P or T(P are #e!ine# with the !ollowin parameterA

$5ction#or"ard'eault(emplate


16/33

0ro%erty8&ased filters

This type o! !ilter is uni"ue to rsyslo. Property6base# !ilters provi#e the capability to !ilter on

messae properties li&e hostname$ syslota an# ms *!ull list o! properties provi#e# in

8ppen#i: (+.


17/33

ueue %rocessing

8ll incomin messaes are place# in the main messae "ueue where they are then !iltere# by

con!iure# actions *what to #o with certain messaes+ an# assine# to the action=s "ueue an#

processe# accor#inly. This is all applie# serially. The conse"uence o! this is that every

action=s processin is only as !ast as the sum o! all the actions. When even one action is

reularly slow this can become a serious problem. This is true even to the point o! actions

ceasin to be processe#. This can occur$ !or e:ample$ when an action writes to a remote

#atabase an# the #atabase becomes overloa#e# or simply unavailable. The answer here is to

de-couplethe slow action "ueues !rom the main "ueue$ e!!ectively creatin parallel processin.

This is simply accomplishe# with rsyslo.

%n the con!iuration !iles$ the main "ueue is #enote# by 0ain0sgan# a #e6couple# action"ueue is #enote# by Action. %n this #ocument$ "ueue parameters enerically contain the place

name ob>ect to re!er to the "ueue type. So replace that with either o! the two "ueue types

when usin them.


18/33

(entral loin scenarios

This section loo&s at how to implement loin mo#els E2$ E-$ an# E/ encountere# earlier.

ulti%le systems to dis*3


19/33

@irebir#5%nterbase

%nres

mS73

MyS73 an# PostreS73 are supporte# natively *plu6ins provi#e#+ while the rest are supporte#

via li%d%i$ a #atabase abstraction layer. Below we provi#e ui#ance !or MyS73.


20/33

,ranch offices remote storage3

This e:tension to the central loin mo#el involves the use o! a non6truste# networ& such as

the %nternet. Securin the connection over which the syslo #ata is transporte# may be

re"uire#. %n a branch o!!ice environment it is probable that a RP1 is alrea#y in place. %! so$ this

option shoul# be use#. %n the absence o! a company RP1$ however$ you may choose to use

the T3S5SS3 protection that rsyslo natively provi#es.

We will provi#e the basic steps re"uire# to set this up. See 8ppen#i: 8$ LThe ?1' Transport

3ayer Security 3ibraryL !or more on T3S.

On the system where you will be creatin &eys an# sinin certi!icates you will nee# to install

the necessary tools an# create #irectories to manae the various !ilesA

$ sudo aptitude install gnutls-%in$ mkdir -p H/tls/Ica,server,clientJ$ chmod go-r"x H/tls/Ica,server,clientJ

1otesA

Fou nee# to create a separate certi!icate !or each machine *client an# server+.

When eneratin a certi!icate *6c option+ use the proper 1S name o! the machine in

"uestion *#ns1ame #ialoue+ as this is the name use# in the certi!icate. ere$ we

assume the names o! the server an# client are$ respectively$ serer'exaple'coan#

client'exaple'co.

Protect all private &eys.

@or security reasons$ try to &eep the machine actin as (8 not permanently connecte#

to a networ&.

@or simplicity$ create all &eys$ re"uests an# certi!icates on the (8A

On the Certificate #uthority

). Manae the (8A

$ cd H/tls/ca

2. (reate the private (8 &ey *ca8*ey.%em+A

$ certtool -p --outile ca-key.pem

-. Sel!6sin the public (8 certi!icate *ca.%em+A

$ certtool -s --load-privkey ca-key.pem --outile ca.pem

/. Manae the serverA

$ cd H/tls/server

;. (reate the private server &ey *server8*ey.%em+A

(entralise# loin with rsyslo 20 www.canonical.com


21/33

$ certtool -p --outile server-key.pem

,. ?enerate a sinin re"uest *re:uest.%em+A

$ certtool -F --load-privkey server-key.pem 0--outile reFuest.pem

. Sin the re"uest with the (8 private &ey to obtain the server=s certi!icate *server8

cert.%em+A

$ certtool -c --load-reFuest reFuest.pem 0--outile server-cert.pem 0--load-ca-certiicate ../ca/ca.pem 0--load-ca-privkey ../ca/ca-key.pem

. Manae a clientA

$ cd H/tls/client

9. (reate the private client &ey *client8*ey.%em+A

$ certtool -p --outile client-key.pem

)0. ?enerate a sinin re"uest *re:uest.%em+A

$ certtool -F --load-privkey client-key.pem 0--outile reFuest.pem

)). Sin the re"uest with the (8 private &ey to obtain the client=s certi!icate *client8

cert.%em+A

$ certtool -c --load-reFuest reFuest.pem 0--outile client-cert.pem 0--load-ca-certiicate ../ca/ca.pem 0--load-ca-privkey ../ca/ca-key.pem

)2. Securely trans!er the necessary !iles to the server *ca.pem$ server6cert.pem$ server6

&ey.pem+ an# each client *ca.pem$ client6cert.pem$ client6&ey.pem+.

On the logging server

(on!iurationA

$ModLoad imtcp

$'eault9etstream'river gtls

$'eault9etstream'river45#ile ca.pem$'eault9etstream'river4ert#ile server-cert.pem$'eault9etstream'riverKey#ile server-key.pem

$5ction&end&tream'river5uthMode x>EB/name$5ction&end&tream'riverermittedeer client.example.com$5ction&end&tream'riverMode :

$@nput(4&erver)un :E>:?

(entralise# loin with rsyslo 2) www.canonical.com


22/33

On a logging client

(on!iurationA

$'eault9etstream'river gtls

$'eault9etstream'river45#ile ca.pem$'eault9etstream'river4ert#ile client-cert.pem$'eault9etstream'riverKey#ile client-key.pem

$5ction&end&tream'river5uthMode x>EB/name$5ction&end&tream'riverermittedeer server.example.com$5ction&end&tream'riverMode :

*.* AA:B.:CD.E.:3:E>:?



23/33

8#vance# 4syslo !eatures applicable to central

loin4syslo has a number o! interestin an# power!ul a#vance# !eatures. ere are two such

!eatures as applicable to central loinA

BS6style bloc&s

3oin "ueues

iscar# watermar&s

,/8style &loc*s

We can create bloc&s o! rules with each one separate# by the previous by a proram or

hostname label. The bloc& will only process messaes correspon#in to the proram an#5or

hostname iven.

'se =Kproram= or =6proram= to inclu#e or e:clu#e prorams an# =hostname= or =6hostname= to

#o the same !or hostnames. These !eatures are also ta&en !rom the BS sources an# help in a

central loin environment.


24/33


25/33

Logging :ueue e(am%les

ere are some e:amples o! usin "ueues in various situations. 8## the !ollowin lines to your

con!iuration to enable "ueuein !eatures.

Local dis* logging

(reate a #e!ault *@i:e#8rray+ "ueue !or a stan#alone systemA

$Qork'irectory /var/log/Fueue O destination Fueue directory$MainMsg8ueue#ile9ame ilename O set ile name or thisaction; ena%les disk mode

$emote dis* logging

When loin to a remote server there may be times when the #atabase is no loner able to

cope with the tra!!ic volume. We set up a 3in&e#3ist %n6Memory 7ueueD speci!y to save the

"ueue=s memory6resi#ent #ata i! rsyslo ever shuts #ownD an# connect to server )92.),.0.)

over the T(P protocol on port ;)/A

$Qork'irectory /var/log/Fueue O destination Fueue directory$5ction8ueue(ype LinkedList O de-couple this action Fueue$5ction8ueue#ile9ame ilename O set a ile or this action;ena%les disk mode$5ction)esume)etry4ount -: O ininite retries on ailure$5ction8ueue&ave+n&hutdo"n on O save in-memory data i

rsyslog shuts do"n*.* AA:B.:CD.E.:3>:? O connect to remote server

$emote data&ase logging

We use the same setup as above but swap the last line with the !ollowin one. We will access a

MyS73 server at )92.),.0.) containin #atabase =los= with user =rsyslo= an# a passwor# o!

=abc)2-=A

*.* 3ommysFl3:B.:CD.E.:,logs,rsyslog,a%c:G;

(entralise# loin with rsyslo 2; www.canonical.com


26/33

/iscard watermar*s

When loin centrally$ there may be times o! su##en bursts o! tra!!ic. When a "ueue reaches

a threshol# o! a number o! "ueue# elements$ less important messaes can be #iscar#e# to help

alleviate the problem. The threshol# in this conte:t is calle# a =#iscar# watermar&=. The

ob>ective is to save "ueue space !or more important messaes. The alorithm #iscar#s both

incomin messaes an# those currently "ueue#.

The #iscar# watermar& shoul# be set su!!iciently hih to not #iscar# messaes unnecessarily

but low enouh to allow !or lare messae bursts.

$1o%ect28ueue'iscardMark somethreshold O num%er o elements$1o%ect28ueue'iscard&everity someseverity O numerical severity

This #irective accepts both the usual te:tual severity &eywor# as well as a numerical co#e as

#e!ine# in 4@( -),/.

To turn messae #iscar#in o!! simply ma&e the #iscar# watermar& hiher than the "ueue siGe.

8n alternative is to speci!y a #iscar# severity o! . This is the #e!ault settin *to prevent

unintentional messae loss+.


27/33

8ppen#i: 8A 4e!erences an# use!ul 3in&s

4syslo home pae

httpA55www.rsyslo.com

4syslo mailin list *rsyslo6users+

httpA55lists.a#iscon.net5mailman5listin!o5rsyslo

4syslo public !orums

httpA55&b.monitorware.com5rsyslo6!/0.html

The %ns an# Outs o! System 3oin 'sin Syslo

httpA55www.sans.or5rr5whitepapers5loin5)),.php

(omparison between rsyslo an# syslo6n

httpA55www.rsyslo.com5#oc6rsysloncomparison.html

4@( -),/ *The BS Syslo Protocol+

httpA55www.iet!.or5r!c5r!c-),/.t:t

4@( -)9; *4eliable elivery !or Syslo+

httpA55www.iet!.or5r!c5r!c-)9;.t:t

The ?1' Transport 3ayer Security 3ibrary

httpA55www.nu.or5so!tware5nutls5manual5htmlno#e5in#e:.html

3ist o! lo analysershttpA55www.syslo.or5wi&i5Main53o8nalyGers

4syslo main #eveloper blo

httpA55blo.erhar#s.net5

S81S %n!ormation System 8u#it 3oin 4e"uirements *200,+

httpA55www.sans.or5resources5policies5in!osysau#it.#oc

1%ST %n!ormation System 8u#it 3oin 4e"uirements *200,+

httpA55csrc.nist.ov5publications5nistpubs5006925SP00692.p#!

istribute# syslo architectures with syslo6n Premium


28/33

8ppen#i: BA rsyslo.con! 5 syslo.con! #i!!

$ di rsyslog.con syslog.con

- RRR this should %e di -u, only )M& still uses context dis:c:1 O /etc/rsyslog.con 4oniguration ile or rsyslog vG.---2 O /etc/syslog.con 4oniguration ile or syslogd.G,GcG,?1 O #or more inormation see1 O /usr/share/doc/rsyslog-doc/html/rsyslogcon.html111 OOOOOOOOOOOOOOOOO

1 OOOO M+':?111 OOOOOOOOOOOOOOOOOOOOOOOOOOO1 OOOO L+P5L '@)4(@U& OOOO1 OOOOOOOOOOOOOOOOOOOOOOOOOOO11 O1 O GdB1 O1 O @nclude all conig iles in /etc/rsyslog.d/1 O1 $@nclude4onig /etc/rsyslog.d/*.con1

11 OOOOOOOOOOOOOOO1 OOOO )


29/33

1 OOOOOOOOOOOOOOO11 O

1 O #irst some standard log iles. Log %y acility.1 OCDc?1 mail."arn -/var/log/mail."arn---2 mail."arning -/var/log/mail."arnV:,VcV1 O1 O Logging or @99 ne"s system.---2 O Logging or @99 ne"s systemVBcG?1 O &ome catch-all log iles.---2 O &ome Wcatch-allX logiles.D?cGB1 *.=ino;*.=notice;*.="arn;0---2 *.=ino;*.=notice;*.="arning;0:E:c>C1 O *.=notice;*.="arn /dev/ttyD---2 O *.=notice;*.="arning /dev/ttyD::?cCB,VE1 *.=notice;*.="arn Y/dev/xconsole---2 *.=notice;*.="arning Y/dev/xconsole2



30/33

8ppen#i: (A Messae properties

0ro%erty eaning

ms entire messae

rawms entire messae e:actly as it was receive# !rom the soc&et

hostname hostname o! oriinal sen#er

source alias !or hostname property

!romhost hostname o! imme#iate sen#er *may be #i!!erent !rom oriinal sen#er+

!romhost6ip %P a##ress o! =!romhost=

syslota messae Ta *see appen#i: 8D LThe BS Syslo ProtocolL+

proramname name o! reportin proram

pri priority *un#eco#e#+

pri6te:t priority *te:tual !orm+

iut MonitorWare %n!o'nitType 6 use# when tal&in to a MonitorWare

bac&en#

syslo!acility !acility *numerical !orm+

syslo!acility6te:t !acility *te:tual !orm+

sysloseverity severity *numerical !orm+

sysloseverity6te:t severity *te:tual !orm+

syslopriority alias !or sysloseverity property *not pri+

syslopriority6te:t alias !or sysloseverity6te:t property

timeenerate# hih resolution timestamp o! receive# messae

timereporte# messae timestamp

timestamp alias !or timestamp property

protocol6version contents o! the P4OT(O36R


31/33

proci# contents o! the P4O(% !iel# !rom %


32/33

8ppen#i: A Property options

O%tion eaning

uppercase convert property to uppercase

lowercase convert property to lowercase

#rop6last6l! remove last line!ee#

#ate6mys"l !ormat as mys"l #ate

#ate6r!c-),/ !ormat as 4@( -),/ #ate

#ate6r!c---9 !ormat as 4@( ---9 #ate

#ate6subsecon#s subsecon#s o! a timestamp *always 0 !or low precision timestamps+

escape6cc replace control characters *8S(%% value )2 an# values less then -2+

with an escape se"uence. The se"unce is LEcharvalL where charval

is the -6#iit #ecimal value o! the control character. @or e:ample$ a

tabulator woul# be replace# by LE009L.

space6cc replace control characters by spaces

#rop6cc #rop control characters 6 the resultin strin will neither contain control

characters$ escape se"uences nor any other replacement character

li&e space.

sp6i!6no6)st6sp returns either a sinle space character or no character at all. @iel#

content is never returne#. 8 space is returne# i! *an# only i!+ the !irst

character o! the !iel#=s content is 1OT a space. This option is a hac&

to solve a problem roote# in 4@( -),/ which speci!ies no #elimiter

between the syslo ta se"uence an# the actual messae te:t.

8lmost all implementation in !act #elimit the two by a space. 8s o!

4@( -),/$ this space is part o! the messae te:t itsel!.

secpath6#rop rops slashes insi#e the !iel# *e.. La5bL becomes LabL+. 'se!ul !or

secure pathname eneration *with #yna!iles+.

secpath6replace 4eplace slashes insi#e the !iel# by an un#erscore. *e.. La5bL

becomes LabL+. 'se!ul !or secure pathname eneration *with

#yna!iles+.

1oteA options escape6cc$ space6cc$ or #rop6cc re"uire that

H


33/33

Documents

Canonical Rsyslog CentralLogging v4 20090901 03