Upload
truongquynh
View
253
Download
0
Embed Size (px)
Citation preview
syslog-ngOpenSourceEdition3.16
AdministrationGuide
Copyright 2018 One Identity LLC.
ALL RIGHTS RESERVED.
Thisguidecontainsproprietaryinformationprotectedbycopyright.Thesoftwaredescribedinthisguideisfurnishedunderasoftwarelicenseornondisclosureagreement.Thissoftwaremaybeusedorcopiedonlyinaccordancewiththetermsoftheapplicableagreement.Nopartofthisguidemaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,includingphotocopyingandrecordingforanypurposeotherthanthepurchaserspersonalusewithoutthewrittenpermissionofOneIdentityLLC.TheinformationinthisdocumentisprovidedinconnectionwithOneIdentityproducts.Nolicense,expressorimplied,byestoppelorotherwise,toanyintellectualpropertyrightisgrantedbythisdocumentorinconnectionwiththesaleofOneIdentityLLCproducts.EXCEPTASSETFORTHINTHETERMSANDCONDITIONSASSPECIFIEDINTHELICENSEAGREEMENTFORTHISPRODUCT,ONEIDENTITYASSUMESNOLIABILITYWHATSOEVERANDDISCLAIMSANYEXPRESS,IMPLIEDORSTATUTORYWARRANTYRELATINGTOITSPRODUCTSINCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTYOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ORNON-INFRINGEMENT.INNOEVENTSHALLONEIDENTITYBELIABLEFORANYDIRECT,INDIRECT,CONSEQUENTIAL,PUNITIVE,SPECIALORINCIDENTALDAMAGES(INCLUDING,WITHOUTLIMITATION,DAMAGESFORLOSSOFPROFITS,BUSINESSINTERRUPTIONORLOSSOFINFORMATION)ARISINGOUTOFTHEUSEORINABILITYTOUSETHISDOCUMENT,EVENIFONEIDENTITYHASBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.OneIdentitymakesnorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisdocumentandreservestherighttomakechangestospecificationsandproductdescriptionsatanytimewithoutnotice.OneIdentitydoesnotmakeanycommitmenttoupdatetheinformationcontainedinthisdocument.Ifyouhaveanyquestionsregardingyourpotentialuseofthismaterial,contact:
OneIdentityLLC.Attn:LEGALDept4PolarisWayAlisoViejo,CA92656
RefertoourWebsite(http://www.OneIdentity.com)forregionalandinternationalofficeinformation.
Patents
OneIdentityisproudofouradvancedtechnology.Patentsandpendingpatentsmayapplytothisproduct.Forthemostcurrentinformationaboutapplicablepatentsforthisproduct,pleasevisitourwebsiteathttp://www.OneIdentity.com/legal/patents.aspx.
Trademarks
OneIdentityandtheOneIdentitylogoaretrademarksandregisteredtrademarksofOneIdentityLLC.intheU.S.A.andothercountries.ForacompletelistofOneIdentitytrademarks,pleasevisitourwebsiteatwww.OneIdentity.com/legal.Allothertrademarksarethepropertyoftheirrespectiveowners.
Legend
WARNING: A WARNING icon indicates a potential for property damage, personal injury, or death.
CAUTION: A CAUTION icon indicates potential damage to hardware or loss of data if instructions are not followed.
IMPORTANT,NOTE,TIP,MOBILE,orVIDEO:Aninformationiconindicatessupportinginformation.
syslog-ngOSEAdministrationGuideUpdated-July2018Version-3.16
http://www.oneidentity.com/http://www.oneidentity.com/legal/patents.aspxhttp://www.oneidentity.com/legal
Contents
Preface 18
Summaryofcontents 18
Targetaudienceandprerequisites 19
Productscoveredinthisguide 19
Summaryofchanges 20
Version3.15-3.16 20
Version3.14-3.15 20
Version3.13-3.14 21
Version3.12-3.13 21
Version3.11-3.12 22
Version3.10-3.11 23
Version3.9-3.10 24
Version3.8-3.9 25
Version3.7-3.8 25
Version3.6-3.7 26
Version3.5-3.6 28
Feedback 29
Acknowledgments 29
Introduction to syslog-ng 30
Whatsyslog-ngis 30
Secureandreliablelogtransfer 30
Flexibledataextractionandprocessing 31
Bigdataclusters 31
Messagequeuesupport 31
SQL,NoSQL,andmonitoring 32
Wideprotocolandplatformsupport 32
Whatsyslog-ngisnot 32
Whyissyslog-ngneeded? 32
Whatisnewinsyslog-ngOpenSourceEdition3.16? 33
Whousessyslog-ng? 34
Supportedplatforms 34
syslog-ng OSE 3.16 Administration Guide 3
The concepts of syslog-ng 35
Thephilosophyofsyslog-ng 35
Loggingwithsyslog-ng 35
Therouteofalogmessageinsyslog-ng 36
Modesofoperation 37
Clientmode 37
Relaymode 38
Servermode 38
Globalobjects 39
Timezonesanddaylightsaving 40
Howsyslog-ngOSEassignstimezonetothemessage 41
Anoteontimezonesandtimestamps 42
Productlicensing 42
Highavailabilitysupport 42
Thestructureofalogmessage 42
BSD-syslogorlegacy-syslogmessages 43
ThePRImessagepart 43
TheHEADERmessagepart 45
TheMSGmessagepart 45
IETF-syslogmessages 45
ThePRImessagepart 46
TheHEADERmessagepart 47
TheSTRUCTURED-DATAmessagepart 48
TheMSGmessagepart 48
Enterprise-widemessagemodel(EWMM) 48
Messagerepresentationinsyslog-ngOSE 49
Structuringmacros,metadata,andothervalue-pairs 51
Specifyingdatatypesinvalue-pairs 52
value-pairs() 53
Thingstoconsiderwhenforwardingmessagesbetweensyslog-ngOSEhosts 57
Commercialversionofsyslog-ng 59
Installing syslog-ng 62
Compilingsyslog-ngfromsource 62
Compilingoptionsofsyslog-ngOSE 64
Uninstallingsyslog-ngOSE 67
syslog-ng OSE 3.16 Administration Guide 4
ConfiguringMicrosoftSQLServertoacceptlogsfromsyslog-ng 67
The syslog-ng OSE quick-start guide 74
Configuringsyslog-ngonclienthosts 74
Configuringsyslog-ngonserverhosts 77
Configuringsyslog-ngrelays 79
Configuringsyslog-ngonrelayhosts 79
Howrelayinglogmessagesworks 81
The syslog-ng OSE configuration file 83
Locationofthesyslog-ngconfigurationfile 83
Theconfigurationsyntaxindetail 83
Notesabouttheconfigurationsyntax 86
Definingconfigurationobjectsinline 87
Usingchannelsinconfigurationobjects 88
Globalandenvironmentalvariables 90
Modulesinsyslog-ngOSE 91
Loadingmodules 91
Managingcomplexsyslog-ngconfigurations 92
Includingconfigurationfiles 92
Reusingconfigurationblocks 93
Passingargumentstoconfigurationblocks 95
Generatingconfigurationblocksfromascript 96
source: Read, receive, and collect log messages 99
Howsourceswork 99
default-network-drivers:Receiveandparsecommonsyslogmessages 103
default-network-drivers()sourceoptions 105
internal:Collectinginternalmessages 108
internal()sourceoptions 108
file:Collectingmessagesfromtextfiles 110
Notesonreadingkernelmessages 111
file()sourceoptions 111
wildcard-file:Collectingmessagesfrommultipletextfiles 122
wildcard-file()sourceoptions 123
network:CollectingmessagesusingtheRFC3164protocol(network()driver) 136
network()sourceoptions 138
syslog-ng OSE 3.16 Administration Guide 5
nodejs:ReceivingJSONmessagesfromnodejsapplications 149
nodejs()sourceoptions 150
mbox:Convertinglocale-mailmessagestologmessages 152
mbox()sourceoptions 153
osquery:Collectandparseosqueryresultlogs 154
osquery()sourceoptions 157
pipe:Collectingmessagesfromnamedpipes 160
pipe()sourceoptions 160
pacct:CollectingprocessaccountinglogsonLinux 171
pacct()options 171
program:Receivingmessagesfromexternalapplications 173
program()sourceoptions 174
snmptrap:ReadNet-SNMPtraps 181
snmptrap()sourceoptions 184
sun-streams:CollectingmessagesonSunSolaris 187
sun-streams()sourceoptions 187
syslog:CollectingmessagesusingtheIETFsyslogprotocol(syslog()driver) 194
syslog()sourceoptions 195
system:Collectingthesystem-specificlogmessagesofaplatform 207
system()sourceoptions 209
systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstorage 211
systemd-journal()sourceoptions 213
systemd-syslog:Collectingsystemdmessagesusingasocket 218
systemd-syslog()sourceoptions 218
tcp,tcp6,udp,udp6:CollectingmessagesfromremotehostsusingtheBSDsyslogprotocolOBSOLETE 220
tcp(),tcp6(),udp()andudp6()sourceoptions:OBSOLETE 220
Changeanoldsourcedrivertothenetwork()driver 221
unix-stream,unix-dgram:CollectingmessagesfromUNIXdomainsockets 222
UNIXcredentialsandothermetadata 222
unix-stream()andunix-dgram()sourceoptions 223
stdin:Collectingmessagesfromthestandardinputstream 232
stdin()sourceoptions 233
destination: Forward, send, and store log messages 244
amqp:PublishingmessagesusingAMQP 246
syslog-ng OSE 3.16 Administration Guide 6
amqp()destinationoptions 247
elasticsearch:SendingmessagesdirectlytoElasticsearchversion1.x 257
Prerequisites 259
Howsyslog-ngOSEinteractswithElasticsearch 260
Clientmodes 261
Elasticsearchdestinationoptions 261
elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher 273
Prerequisites 275
Howsyslog-ngOSEinteractswithElasticsearch 276
Clientmodes 277
SearchGuardandsyslog-ngOSE 278
Elasticsearch2destinationoptions 279
ExampleusecasesofsendinglogstoElasticsearchusingsyslog-ng 299
file:Storingmessagesinplain-textfiles 299
file()destinationoptions 301
graphite:SendingmetricstoGraphite 312
graphite()destinationoptions 313
SendinglogstoGraylog 315
graylog2()destinationoptions 317
hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS) 318
Prerequisites 320
Howsyslog-ngOSEinteractswithHDFS 321
StoringmessageswithMapR-FS 322
Kerberosauthenticationwithsyslog-nghdfs()destination 323
HDFSdestinationoptions 324
PostingmessagesoverHTTP 334
HTTPdestinationoptions 335
http:PostingmessagesoverHTTPwithoutJava 339
HTTPdestinationoptions 340
kafka:PublishingmessagestoApacheKafka 354
Prerequisites 355
Howsyslog-ngOSEinteractswithApacheKafka 356
Kafkadestinationoptions 357
loggly:UsingLoggly 363
loggly()destinationoptions 364
syslog-ng OSE 3.16 Administration Guide 7
logmatic:UsingLogmatic.io 366
logmatic()destinationoptions 367
mongodb:StoringmessagesinaMongoDBdatabase 369
Howsyslog-ngOSEconnectstheMongoDBserver 370
mongodb()destinationoptions 371
network:SendingmessagestoaremotelogserverusingtheRFC3164protocol(network()driver) 381
network()destinationoptions 382
osquery:Sendinglogmessagestoosquery'ssyslogtable 395
osquery()destinationoptions 396
pipe:Sendingmessagestonamedpipes 399
pipe()destinationoptions 399
program:Sendingmessagestoexternalapplications 406
program()destinationoptions 407
pseudofile() 416
pseudofile()destinationoptions 417
redis:Storingname-valuepairsinRedis 419
redis()destinationoptions 420
riemann:MonitoringyourdatawithRiemann 426
riemann()destinationoptions 427
smtp:GeneratingSMTPmessages(e-mail)fromlogs 438
smtp()destinationoptions 440
Splunk:SendinglogmessagestoSplunk 448
sql:StoringmessagesinanSQLdatabase 448
Usingthesql()driverwithanOracledatabase 450
Usingthesql()driverwithaMicrosoftSQLdatabase 451
Thewaysyslog-nginteractswiththedatabase 453
MySQL-specificinteractionmethods 454
MsSQL-specificinteractionmethods 454
sql()destinationoptions 454
stomp:PublishingmessagesusingSTOMP 466
stomp()destinationoptions 467
syslog:SendingmessagestoaremotelogserverusingtheIETF-syslogprotocol 473
syslog()destinationoptions 474
syslog-ng()destinationoptions 488
syslog-ng OSE 3.16 Administration Guide 8
tcp,tcp6,udp,udp6:SendingmessagestoaremotelogserverusingthelegacyBSD-syslogprotocol(tcp(),udp()drivers) 500
tcp(),tcp6(),udp(),andudp6()destinationoptions 500
Changeanolddestinationdrivertothenetwork()driver 501
Telegram:SendingmessagestoTelegram 502
telegram()destinationoptions 502
unix-stream,unix-dgram:SendingmessagestoUNIXdomainsockets 505
unix-stream()andunix-dgram()destinationoptions 506
usertty:Sendingmessagestoauserterminal:usertty()destination 515
WriteyourowncustomdestinationinJavaorPython 516
log: Filter and route log messages using log paths, flags, and filters 517
Logpaths 517
Embeddedlogstatements 518
Usingembeddedlogstatements 520
if-else-elif:Conditionalexpressions 522
Junctionsandchannels 522
Logpathflags 525
Managingincomingandoutgoingmessageswithflow-control 528
Flow-controlandmultipledestinations 532
Configuringflow-control 532
Usingdisk-basedandmemorybuffering 534
Enablingreliabledisk-basedbuffering 536
Enablingnormaldisk-basedbuffering 537
Enablingmemorybuffering 537
Aboutdiskqueuefiles 538
Filters 539
Usingfilters 539
Combiningfilterswithbooleanoperators 540
Comparingmacrovaluesinfilters 541
Usingwildcards,specialcharacters,andregularexpressionsinfilters 542
Taggingmessages 543
Filterfunctions 544
Droppingmessages 549
Global options of syslog-ng OSE 551
Configuringglobalsyslog-ngoptions 551
syslog-ng OSE 3.16 Administration Guide 9
Globaloptions 551
TLS-encrypted message transfer 569
SecureloggingusingTLS 569
EncryptinglogmessageswithTLS 570
ConfiguringTLSonthesyslog-ngclients 571
ConfiguringTLSonthesyslog-ngserver 572
MutualauthenticationusingTLS 574
ConfiguringTLSonthesyslog-ngclients 575
ConfiguringTLSonthesyslog-ngserver 576
Password-protectedkeys 578
TLSoptions 579
template and rewrite: Format, modify, and manipulate log messages 586
Customizemessageformatusingmacrosandtemplates 586
Formattingmessages,filenames,directories,andtablenames 587
Templatesandmacros 587
Date-relatedmacros 589
Hardvs.softmacros 590
Macrosofsyslog-ngOSE 591
Usingtemplatefunctions 600
Templatefunctionsofsyslog-ngOSE 601
Modifyingtheon-the-wiremessageformat 623
Modifyingmessagesusingrewriterules 623
Replacingmessageparts 624
Settingmessagefieldstospecificvalues 625
Unsettingmessagefields 628
CreatingcustomSDATAfields 629
Settingmultiplemessagefieldstospecificvalues 630
map-value-pairs:Renamevalue-pairstonormalizelogs 631
Conditionalrewrites 632
Howconditionalrewritingworks 632
Addinganddeletingtags 633
Anonymizingcreditcardnumbers 634
Regularexpressions 634
Typesandoptionsofregularexpressions 635
syslog-ng OSE 3.16 Administration Guide 10
Optimizingregularexpressions 637
parser: Parse and segment structured messages 639
Parsingsyslogmessages 640
Optionsofsyslog-parserparsers 642
Parsingmessageswithcomma-separatedandsimilarvalues 644
OptionsofCSVparsers 647
Parsingkey=valuepairs 651
Optionsofkey=valueparsers 654
The JSON parser 655
TheJSONparserTheJSONparser 655
OptionsofJSONparsers 658
TheXMLparser 660
OptionsofXMLparsers 663
Parsingdatesandtimestamps 666
Optionsofdate-parser()parsers 667
TheApacheAccessLogParser 669
Optionsofapache-accesslog-parser()parsers 670
TheCiscoParser 671
TheLinuxAuditParser 673
Optionsoflinux-audit-parser()parsers 675
ThePythonParser 676
Parsingenterprise-widemessagemodel(EWMM)messages 681
Thesudoparser 681
Theiptablesparser 682
db-parser: Process message content with a pattern database (patterndb) 684
Classifyinglogmessages 684
Thestructureofthepatterndatabase 685
Howpatternmatchingworks 686
Artificialignorance 687
Usingpatterndatabases 688
Usingparserresultsinfiltersandtemplates 689
Downloadingsamplepatterndatabases 691
Correlatinglogmessagesusingpatterndatabases 692
Referencingearliermessagesofthecontext 694
syslog-ng OSE 3.16 Administration Guide 11
Triggeringactionsforidentifiedmessages 695
Conditionalactions 697
Externalactions 698
Actionsandmessagecorrelation 699
Creatingpatterndatabases 702
Usingpatternparsers 702
Patternparsersofsyslog-ngOSE 704
What'snewinthesyslog-ngpatterndatabaseformatV5 707
Thesyslog-ngpatterndatabaseformat 707
Element:patterndb 709
Element:ruleset 709
Element:patterns 710
Element:rules 711
Element:rule 712
Element:patterns 714
Element:urls 715
Element:values 716
Element:examples 716
Element:example 717
Element:actions 718
Element:action 720
Element:create-context 722
Element:tags 725
Correlating log messages 726
Correlatingmessagesusingthegrouping-by()parser 726
Referencingearliermessagesofthecontext 730
Optionsofgrouping-byparsers 731
Enriching log messages with external data 735
Addingmetadatafromanexternalfile 735
Usingfiltersasselector 737
Optionsadd-contextual-data() 738
LookingupGeoIPdatafromIPaddresses(DEPRECATED) 740
Optionsofgeoipparsers 742
LookingupGeoIP2datafromIPaddresses 743
syslog-ng OSE 3.16 Administration Guide 12
Referringtopartsofthemessageasamacro 744
UsingtheGeoIP2parser 744
TransferringyourlogstoElasticsearchusingGeoIP2 745
Optionsofgeoip2parsers 746
Statistics of syslog-ng 748
Metricsandcountersofsyslog-ngOSE 748
Logstatisticsfromtheinternal()source 751
Multithreading and scaling in syslog-ng OSE 753
Multithreadingconceptsofsyslog-ngOSE 753
Configuringmultithreading 755
Optimizingmultithreadedperformance 755
Troubleshooting syslog-ng 757
Possiblecausesoflosinglogmessages 757
Creatingsyslog-ngcorefiles 759
Collectingdebugginginformationwithstrace,truss,ortusc 759
Runningafailurescript 760
Stoppingsyslog-ng 761
Reportingbugsandfindinghelp 761
Recoverdatafromorphaneddiskbufferfiles 762
Nolocallogsafterspecifyinganunusualstoragedirectory 762
Nologsafterspecifyinganunusualportnumber 762
Errormessages 763
Best practices and examples 765
Generalrecommendations 765
Handlinglargemessageload 765
Usingnameresolutioninsyslog-ng 766
Resolvinghostnameslocally 767
Collectinglogsfromchroot 767
Configuringlogrotation 768
The syslog-ng manual pages 770
Thedqtooltoolmanualpage 770
Name 770
Synopsis 770
syslog-ng OSE 3.16 Administration Guide 13
Description 770
Thecatcommand 771
Files 772
Seealso 772
Author 772
Copyright 772
Theloggenmanualpage 772
Name 773
Synopsis 773
Description 773
Options 773
Examples 776
Files 776
Seealso 776
Author 777
Copyright 777
Thepdbtoolmanualpage 777
Name 777
Synopsis 777
Description 778
Thedictionarycommand 778
Thedumpcommand 778
Thematchcommand 779
Themergecommand 781
Thepatternizecommand 782
Thetestcommand 783
Files 783
Seealso 783
Author 784
Copyright 784
Thesyslog-ngcontroltoolmanualpage 784
Name 784
Synopsis 784
Description 785
Enablingtroubleshootingmessages 785
syslog-ng OSE 3.16 Administration Guide 14
syslog-ng-ctlquery 786
Thestatscommand 788
Handlingpassword-protectedprivatekeys 789
Reloadingtheconfiguration 790
Files 790
Seealso 791
Author 791
Copyright 791
Thesyslog-ng-debunmanualpage 791
Name 791
Synopsis 792
Description 792
GeneralOptions 792
Debugmodeoptions 792
Systemcalltracing 793
Packetcaptureoptions 793
Examples 793
Files 795
Seealso 795
Author 795
Copyright 795
Thesyslog-ngmanualpage 795
Name 795
Synopsis 796
Description 796
Options 796
Files 799
Seealso 799
Author 799
Copyright 799
Thesyslog-ng.confmanualpage 799
Name 800
Synopsis 800
Description 800
Basicconceptsofsyslog-ngOSE 800
syslog-ng OSE 3.16 Administration Guide 15
Configuringsyslog-ng 801
Files 805
Seealso 805
Author 806
Copyright 806
Third-party contributions 807
GNUGeneralPublicLicense 807
Preamble 807
TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 808
Section0 808
Section1 809
Section2 809
Section3 810
Section4 810
Section5 810
Section6 811
Section7 811
Section8 811
Section9 812
Section10 812
NOWARRANTYSection11 812
Section12 812
HowtoApplyTheseTermstoYourNewPrograms 813
GNULesserGeneralPublicLicense 814
Preamble 814
TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 816
Section0 816
Section1 816
Section2 817
Section3 817
Section4 818
Section5 818
Section6 819
Section7 820
Section8 820
syslog-ng OSE 3.16 Administration Guide 16
Section9 820
Section10 820
Section11 821
Section12 821
Section13 821
Section14 822
NOWARRANTYSection15 822
NOWARRANTYSection16 822
HowtoApplyTheseTermstoYourNewLibraries 822
Licenseattributions 823
Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License 824
About us 830
Contactingus 830
Technicalsupportresources 830
syslog-ng OSE 3.16 Administration Guide 17
Preface
Welcometothesyslog-ngOpenSourceEdition3.16AdministratorGuide!
Thisdocumentdescribeshowtoconfigureandmanagesyslog-ng.Backgroundinformationforthetechnologyandconceptsusedbytheproductisalsodiscussed.
Summary of contents
Introductiontosyslog-ngdescribesthemainfunctionalityandpurposeofsyslog-ngOSE.
Theconceptsofsyslog-ngdiscussesthetechnicalconceptsandphilosophiesbehindsyslog-ngOSE.
Installingsyslog-ngdescribeshowtoinstallsyslog-ngOSEonvariousUNIX-basedplatformsusingtheprecompiledbinaries.
Thesyslog-ngOSEquick-startguideprovidesabrieflyexplainshowtoperformthemostcommonlogcollectingtaskswithsyslog-ngOSE.
Thesyslog-ngOSEconfigurationfilediscussestheconfigurationfileformatandsyntaxindetail,andexplainshowtomanagelarge-scaleconfigurationsusingincludedfilesandreusableconfigurationsnippets.
source:Read,receive,andcollectlogmessagesexplainshowtocollectandreceivelogmessagesfromvarioussources.
destination:Forward,send,andstorelogmessagesdescribesthedifferentmethodstostoreandforwardlogmessages.
log:Filterandroutelogmessagesusinglogpaths,flags,andfiltersexplainshowtorouteandsortlogmessages,andhowtousefilterstoselectspecificmessages.
Globaloptionsofsyslog-ngOSEliststheglobaloptionsofsyslog-ngOSEandexplainshowtousethem.
TLS-encryptedmessagetransfershowshowtosecureandauthenticatelogtransportusingTLSencryption.
templateandrewrite:Format,modify,andmanipulatelogmessagesdescribeshowtocustomizemessageformatusingtemplatesandmacros,howtorewriteandmodifymessages,andhowtouseregularexpressions.
parser:Parseandsegmentstructuredmessagesdescribeshowtosegmentandprocessstructuredmessageslikecomma-separatedvalues.
db-parser:Processmessagecontentwithapatterndatabase(patterndb)explainshowtoidentifyandprocesslogmessagesusingapatterndatabase.
Correlatinglogmessagesexplainshowtocorrelatelogmessagesthatmatchasetoffiltersorthatareidentifiedusingapatterndatabase.
syslog-ng OSE 3.16 Administration Guide
Preface18
Enrichinglogmessageswithexternaldataexplainshowtoimportdatafromexternalsourcestoincludeinthelogmessages,thusextending,enriching,andcomplementingthedatafoundinthelogmessage.
Statisticsofsyslog-ngdetailstheavailablestatisticsthatsyslog-ngOSEcollectsabouttheprocessedlogmessages.
Multithreadingandscalinginsyslog-ngOSEdescribeshowtoconfiguresyslog-ngOSEtousemultipleprocessors,andhowtooptimizeitsperformance.
Troubleshootingsyslog-ngofferstipstosolvingproblems.
Bestpracticesandexamplesgivesrecommendationstoconfigurespecialfeaturesofsyslog-ngOSE.
Thesyslog-ngmanualpagescontainsthemanualpagesofthesyslog-ngOSEapplication.
Third-partycontributionsincludesthetextofthelicensesapplicabletosyslog-ngOpenSourceEdition.
CreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseincludesthetextoftheCreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseapplicabletoThesyslog-ngOpenSourceEdition3.16AdministratorGuide.
Target audience and prerequisites
Thisguideisintendedforsystemadministratorsandconsultantsresponsiblefordesigningandmaintainingloggingsolutionsandlogcenters.ItisalsousefulforITdecisionmakerslookingforatooltoimplementcentralizedlogginginheterogeneousenvironments.
Thefollowingskillsandknowledgearenecessaryforasuccessfulsyslog-ngadministrator:
l Atleastbasicsystemadministrationknowledge.
l Anunderstandingofnetworks,TCP/IPprotocols,andgeneralnetworkterminology.
l WorkingknowledgeoftheUNIXorLinuxoperatingsystem.
l In-depthknowledgeoftheloggingprocessofvariousplatformsandapplications.
l Anunderstandingofthelegacysyslog(BSD-syslog)protocolandthenewsyslog(IETF-syslog)protocolstandard.
Products covered in this guide
Thisguidedescribestheuseofthefollowingproducts:
l syslog-ngOpenSourceEdition(syslog-ngOSE)3.16.1andlater
syslog-ng OSE 3.16 Administration Guide
Preface19
https://www.ietf.org/rfc/rfc3164.txthttps://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424
Summary of changes
ThissectionliststhechangesofThesyslog-ngOpenSourceEditionAdministratorGuide.
Version 3.15 - 3.16
Changes in product:
l Anewdestinationdriver,telegram(),hasbeenadded.Thetelegram()destinationsendslogmessagestoTelegram,whichisasecure,cloud-basedmobileanddesktopmessagingapp.Formoreinformation,seeTelegram:SendingmessagestoTelegram.
l Anewtemplatefunction,urlencode,hasbeenadded.Youcanusetheurlencodetemplatefunctiontogetherwiththetelegram()destinationtosendsyslogmessagestoTelegram.Formoreinformation,seeurlencode.
l Toensurethatamoduleisloaded,[email protected],seeLoadingmodules.
l Theadd-contextual-data()hasbeenextendedwiththeignore-case()option.Formoreinformation,seeOptionsadd-contextual-data().
l Thehook-commands()hasbeenadded,whichmakesitpossibletoexecuteexternalprogramswhentheyareinitializedortorndown.Thehook-commands()canbeusedforbothsourceanddestinationdrivers.Formoreinformation,seehook-commands().
Version 3.14 - 3.15
Changes in product:
l Itisnowpossibletouseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeif-else-elif:Conditionalexpressions.
l Anewlogpathflag,drop-unmatched,hasbeenadded.Thenewflagcausesmessagestobedroppedalongalogpathwhentheydonotmatchafilterorarediscardedbyaparser.Fordetails,seeLogpathflags.
l SupportforElasticsearch'sShieldhasbeenremoved.
l SupportforPOSIXregularexpressionshasbeenremoved.
syslog-ng OSE 3.16 Administration Guide
Preface20
https://core.telegram.org/https://core.telegram.org/
Version 3.13 - 3.14
Changes in product:
l Youcanusepassword-protectedprivatekeysinthenetwork()andsyslog()sourceanddestinationdrivers.Fordetails,seePassword-protectedkeys.
l Tobettercontroltowhichlogmessagesyouaddcontextualdata,youcanusefiltersasselectors.Inthiscase,thefirstcolumnoftheCSVdatabasefilemustcontainthenameofafilter.Foreachmessage,syslog-ngOSEevaluatesthefiltersintheordertheyappearinthedatabasefile.Ifafiltermatchesthemessage,syslog-ngOSEaddsthename-valuepairrelatedtothefilter.Fordetails,seeUsingfiltersasselector.
Version 3.12 - 3.13
Changes in product:
l Anewsourcedriver,stdin(),hasbeenadded.Thestdin()drivercollectsmessagesfromthestandardinputstream.Formoreinformation,seestdin:Collectingmessagesfromthestandardinputstream.
l Anewdestination,SendinglogstoGraylog,andatemplatetosendsyslogmessagestoGraylog,format-gelf,hasbeenadded.
l Anewtemplatefunction,getent,hasbeenadded.YoucanusethegetenttemplatefunctiontolookupentriesfromtheNameServiceSwitchlibraries.Formoreinformation,seegetent.
l Thedefaultvaluesofthe--enable-json,--enable-mongodb,and--with-libmongo-clientcompileparametershavechanged.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.
l Anewcompileoption,--with-module-path,hasbeenadded.Thenewoptionspecifiessyslog-ngOSE'smoduleinstallationdirectory.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.
l Anewdestinationdriver,osquery(),hasbeenadded.Thenewdriversendslogmessagestoosquery'ssyslogtable.Formoreinformation,seeosquery:Sendinglogmessagestoosquery'ssyslogtable.
l ItisnowpossibletospecifyTLSoptionsinatls()block.Formoreinformation,see:
l amqp()destinationoptions
l HTTPdestinationoptions
l riemann()destinationoptions
l SupportformicrosecondsinRiemanndestinationshasbeenintroduced.Formoreinformation,seeevent-time().
syslog-ng OSE 3.16 Administration Guide
Preface21
l Moduleauto-loadingnowalsoworksforthesystem()source.Formoreinformation,see--default-modules.
Changes in documentation:
l Anewsectiondescribingcommonerrormessageshasbeenaddedtothedocument.Formoreinformation,seeErrormessages.
l Severalcorrectionsandeditorialchanges.
Version 3.11 - 3.12
Changes in product:
l Anewsystemd-journal()sourceoption,calledread-old-records(),hasbeenadded.Formoreinformation,seeread-old-records().
l Anoptioncalledjvm-options()hasbeenadded,whichallowsyoutofine-tuneJavaVirtualMachinesettingswhenconfiguringElasticsearch,HDFS,andApacheKafkadestinations,orwebservicestowhichyousendlogmessagesviatheHTTPprotocol.Fordetails,see:
l Elasticsearchdestinationoptions
l Elasticsearch2destinationoptions
l HDFSdestinationoptions
l HTTPdestinationoptions
l Kafkadestinationoptions
l Globaloptions
l AnewHDFSdestinationoption,calledhdfs-append-enabled()hasbeenadded.Forfurtherinformation,seehdfs-append-enabled().
l Macrosarenowsupportedinthehdfs-file()option.Fordetails,seehdfs-file().
l ThefollowingnewTLSoptionshavebeenadded:
l dhparam-file()
l ecdh-curve-list()
l pkcs12-file().
l Anewparser,capableofprocessinginputinXMLformat,hasbeenadded.Formoreinformation,seeTheXMLparser.
Changes in documentation:
l Addedsectionaboutcommercialversionofsyslog-ng.Formoreinformation,seeCommercialversionofsyslog-ng.
l Addedwarningabouttherequirementtodeletethepersistfileoncethedir()option
syslog-ng OSE 3.16 Administration Guide
Preface22
ofdisk-buffer()hasbeenmodifiedoranewonehasbeenadded.Formoreinformation,seedestination:Forward,send,andstorelogmessages.
l ClarifiedinformationaboutthePythonparser'sdeinit()method.Itrunsnotonlyatasyslog-nggracefulstop,butatareloadtoo.Fordetails,seeMethodsofthepython()parser.
l Severalcorrectionsandeditorialchanges.
Version 3.10 - 3.11
Changes in product:
l LookingupGeoIP2datafromIPaddresseshasbeenaddedtothedocument.
l http:PostingmessagesoverHTTPwithoutJavahasbeenupgradedwithnewimprovements.
l Thegeoip()parserisnowdeprecated.LookingupGeoIPdatafromIPaddresses(DEPRECATED).
l Thetemplate()optionhasbeenaddedtotheApacheAccessLogParser.Fordetails,see:TheApacheAccessLogParser.
l SSL-relatedoptionshavebeenaddedtoamqp()destination.Fordetails,see:amqp()destinationoptions.
l Theprefix()optionhasbeenaddedtotheCiscoparser.Fordetails,see:TheCiscoParser.
l Thedrop-unmatched()optionhasbeenaddedtothedb-parser()statement.Fordetails,see:Usingpatterndatabases.
l Theevent-time()optionhasbeenaddedtotheRiemanndestination.Fordetails,see:riemann:MonitoringyourdatawithRiemann.
Changes in documentation:
l Anewexamplehasbeenaddedtotheosquery()source.Fordetails,see:osquery:Collectandparseosqueryresultlogs.
l Severalcorrectionsandeditorialchanges.
syslog-ng OSE 3.16 Administration Guide
Preface23
Version 3.9 - 3.10
Changes in product:
l wildcard-file: Collectingmessages frommultiple text files has been added tothe document.
l snmptrap:ReadNet-SNMPtrapshasbeenaddedtothedocument.
l osquery:Collectandparseosqueryresultlogshasbeenaddedtothedocument.
l Theelasticsearch2()destinationnowsupportsHTTPSmode,includingencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeelasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher.
l Thehttp()destinationnowsupportsencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeHTTPdestinationoptions.
l Thehdfs()destinationnowsupportsKerberosauthentication.Fordetails,seeKerberosauthenticationwithsyslog-nghdfs()destination.
l ThePythonParserhasbeenaddedtothedocument.
l TheCiscoParserhasbeenaddedtothedocument.
l map-value-pairs: Rename value-pairs to normalize logs has been added to thedocument.
l Thelist-*templatefunctionsallowyoutomanipulatecomma-separatedlists.Fordetails,seeListmanipulation.
l Thenewbasename()anddirname()templatefunctionsallowyoutoeasilyseparatethepathandfilenames.Fordetails,seeTemplatefunctionsofsyslog-ngOSE.
l stardatehasbeenaddedtothedocument.
l create-statement-append()hasbeenaddedtothedocument.
l Thedefaultvalueofthelog-msg-size()optionhasbeenincreasedto64k.Thatwaysyslog-ngOSEwillnottruncatelonglogmessages,whicharegettingincreasinglycommon.
Changes in documentation:
l Splunk:SendinglogmessagestoSplunkhasbeenaddedtothedocument.
l Aboutdiskqueuefileshasbeenaddedtothedocument.
l AnexamplefailurescripthasbeenaddedtoRunningafailurescript.
l Severalcorrectionsandeditorialchanges.
syslog-ng OSE 3.16 Administration Guide
Preface24
Version 3.8 - 3.9
Changes in product:
l WhenusingTLS-transport,youcannowusecertainfieldsoftheX.509certificatesasmacros.Fordetails,see.TLS.X509.
l Theelastic2()destinationdrivernowsupportsSearchGuard,analternativesecuritysolutionforElasticsearch.Fordetails,seeSearchGuardandsyslog-ngOSE.
l .TLS.X509hasbeenaddedtothedocument.
l Unsettingmessagefieldshasbeenupdatedwithgroupunset().
Changes in documentation:
l Correctionsandeditorialchanges.
Version 3.7 - 3.8
Changes in product:
l Enrichinglogmessageswithexternaldatahasbeenaddedtothedocument.
l Correlatinglogmessageshasbeenaddedtothedocument.
l elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigherhasbeenaddedtothedocument.
l http:PostingmessagesoverHTTPwithoutJavahasbeenaddedtothedocument.
l logmatic:UsingLogmatic.iohasbeenaddedtothedocument.
l loggly:UsingLogglyhasbeenaddedtothedocument.
l Disk-basedbufferinghasbeenaddedtosyslog-ngOSE.Fordetails,seeUsingdisk-basedandmemorybuffering.
l What'snewinthesyslog-ngpatterndatabaseformatV5,,hasbeenaddedtoElement:create-contexthasbeenaddedtodb-parser:Processmessagecontentwithapatterndatabase(patterndb).
l Parsingdatesandtimestampshasbeenaddedtoparser:Parseandsegmentstructuredmessages.
l TheApacheAccessLogParserhasbeenaddedtoparser:Parseandsegmentstructuredmessages.
l Newoptionsoftheset()rewriteoperatorhavebeenaddedtoSettingmessagefieldstospecificvalues.
l ArewriteoperatortounsetfieldshasbeenaddedtoUnsettingmessagefields.
syslog-ng OSE 3.16 Administration Guide
Preface25
https://github.com/floragunncom/search-guard
l Atemplatefunctionthatformatsname-valuepairsasArcSightCommonEventFormatextensionhasbeenaddedtoformat-cef-extension.
l NumericaltemplatefunctionsthatworkonnumericalvaluesofacorrelationcontexthavebeenaddedtoNumericaloperations.
l Theinherit-environment()optionhasbeenaddedtoprogram:Receivingmessagesfromexternalapplicationsandprogram:Sendingmessagestoexternalapplications.
l @NLSTRING@hasbeenaddedtoUsingpatternparsers.
Changes in documentation:
l LookingupGeoIPdatafromIPaddresses(DEPRECATED)hasbeenmovedtoEnrichinglogmessageswithexternaldata.
l Severalcorrectionsandeditorialchanges.
Version 3.6 - 3.7
Changes in product:
l mbox: Converting local e-mailmessages to logmessages has been added tothe document.
l Thekeep-alive()optionhasbeenaddedtotheprogram()destination.
l The Linux Audit Parser has been added to parser: Parse and segmentstructuredmessages.
l pythonhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.
l PostingmessagesoverHTTPhasbeenaddedtothedocument.
l Write your own custom destination in Java or Python has been added to thedocument.
l Looking up GeoIP data from IP addresses (DEPRECATED) has been added tothe document.
l Elasticsearchdestinationoptionshasbeenaddedtothedocument.
l kafka:PublishingmessagestoApacheKafkahasbeenaddedtothedocument.
l hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS)hasbeenaddedtothedocument.
l Parsingkey=valuepairshasbeenaddedtothedocument.
l format-cimhasbeenaddedtothedocument.
l Simpletemplatescanbedefinedwithoutbraces.Templatescanalsoreferenceothertemplates.Fordetails,seeTemplatesandmacros.
l Customtemplatefunctionscanbedefinedinthesyslog-ngOSEconfiguration.For
syslog-ng OSE 3.16 Administration Guide
Preface26
details,seeUsingtemplatefunctions.
l CSV-parserscanusestringsasdelimiters.Fordetails,seedelimiters().
l IPv6addressescanbefilteredusinganewfilter.Fordetails,seenetmask6().
l Theloggenutilitycansendmessagesindefinitelyusingthe--permanentoption.
l Thessl-options()optionhasbeedaddedtoTLSoptions.
l TLS-supporthasbeenaddedtoriemann()destinationoptions.
l Theextract-solaris-msgid()parserhasbeedaddedtosun-streams:CollectingmessagesonSunSolaris.
l Thecontextoptionofinherit-propertieshasbeedaddedtoActionsandmessagecorrelation.
l flush-lines()hasbeenaddedtothedocument.
l Thesanitize-utf8flaghasbeenaddedtothelistofsourceflags.
l Theformat-welffunctionhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.
l The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.
l Theuse-uniqid()optionhasbeenaddedtoGlobaloptionsofsyslog-ngOSE.
l TheUNIQIDmacrohasbeenaddedtoMacrosofsyslog-ngOSE.
l TheJSON-parsernowhandlesspecialcharactersinobjectnames.Fordetails,seeextract-prefix().
l Thesyslog-debuntoolusedtogeneratesyslog-ngOSEdebugbundleshasbeendocumented.Fordetails,seeThesyslog-ng-debunmanualpage.
l The --control option has been added to the The syslog-ngmanual pagemanual page.
l Version3.7andnewerautomaticallyincludestheplugin.conffilesfromthe/scl/*/directories,makingiteasiertouseanddistributeconfigurationblocks.
l The--enable-all-modulescompileroptionhasbeedaddedtoCompilingoptionsofsyslog-ngOSE.
l Thecreate-dirs()optionhasbeenaddedtounix-stream()andunix-dgram()destinationoptions.
Changes in documentation:
l Generatingconfigurationblocksfromascripthasbeenaddedtothedocument.
l Example:Sendingalertwhenaclientdisappearshasbeenaddedtothedocument.
l Thetcp(),tcp6(),udp(),udp6()sourceanddestinationdrivershavebeendeprecated,asalloftheirfunctionalitycanbeachievedwiththenetwork()driver.Forhelponmigratingtothenetwork()driver,seeChangeanoldsourcedrivertothenetwork()driverandChangeanolddestinationdrivertothenetwork()driver.
syslog-ng OSE 3.16 Administration Guide
Preface27
l ThebeginningofTroubleshootingsyslog-nghasbeenextendedwithbasictroubleshootinginformation.
l Thedescriptionofthechain-hostnames()globaloptionhasbeenclarifiedandextended.Fordetails,seechain-hostnames().
l Othereditorialcorrections.
Version 3.5 - 3.6
Changes in product:
Changes in documentation:
l riemann:MonitoringyourdatawithRiemannhasbeenaddedtothedocument.
l nodejs:ReceivingJSONmessages fromnodejsapplicationshasbeenadded tothedocument.
l systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstoragehasbeenaddedtothedocument.
l systemd-syslog:Collectingsystemdmessagesusingasockethasbeenaddedtothedocument.
l use-rcptid()hasbeenaddedtothedocument.
l Settingmultiplemessagefieldstospecificvalueshasbeenaddedtothedocument.
l TheretriesandthrottleoptionsareavailablefortheSMTP,MongoDB,AMQP,andRedisdestinations.
l Thedescriptionofthemulti-line-modeoptionhasbeenupdated.
l UNIXcredentialsandothermetadatahasbeenaddedtothedocument.
l RUNIDhasbeenaddedtoMacrosofsyslog-ngOSE.
l Theextract-prefixoptionhasbeenaddedtoTheJSONparserTheJSONparser.
l Thegraphite-output,orandpaddingtemplatefunctionshavebeenaddedtoTemplatefunctionsofsyslog-ngOSE.
l PCREisnowarequireddependencyofsyslog-ngOSE,andbydefault,syslog-ngOSEusesPCRE-styleregularexpressions.Therefore,the--enable-pcrecompliationoptionhasbeenremoved.
l graphite:SendingmetricstoGraphitehasbeenaddedtothedocument.
l pseudofile()hasbeenaddedtothedocument.
l Thecustom-domain()andstats-lifetime()optionshavebeenaddedtoGlobaloptions.
l Theretry_sql_insertsoptionhasbeenrenamedtoretriestoincreaseconsistency.
l on-error()canbesetlocallyforMongoDBdestinationsaswell.Also,MongoDBdestinationssupporttheusernameandpasswordoptions,andconnectingtotheserver
syslog-ng OSE 3.16 Administration Guide
Preface28
usingUNIXdomainsockets.Fordetails,seemongodb:StoringmessagesinaMongoDBdatabase.
l Howsyslog-ngOSEconnectstheMongoDBserverhasbeenaddedtothedocument.
l Severaltyposandsyntaxerrorsinexampleshavebeencorrected.
Feedback
Anyfeedbackisgreatlyappreciated,especiallyonwhatelsethisdocumentshouldcover.Generalcomments,errorsfoundinthetext,andanysuggestionsabouthowtoimprovethedocumentationisalsowelcomeatdocumentation@balabit.com.
ThesourceofthisguideisavailableonGitHub.Incaseofthesyslog-ngOpenSourceEditionguides,youcanalso:
l Openanissue
Acknowledgments
OneIdentitywouldliketoexpressitsgratitudetothesyslog-ngusersandthesyslog-ngcommunityfortheirinvaluablehelpandsupport.
syslog-ng OSE 3.16 Administration Guide
Preface29
https://github.com/balabit/syslog-ng-ose-guideshttps://github.com/balabit/syslog-ng-ose-guides/issues
3
Introduction to syslog-ng
Thischapterintroducesthesyslog-ngOpenSourceEditionapplicationinanon-technicalmanner,discussinghowandwhyisituseful,andthebenefitsitofferstoanexistingITinfrastructure.
What syslog-ng is
Thesyslog-ngapplicationisaflexibleandhighlyscalablesystemloggingapplicationthatisidealforcreatingcentralizedandtrustedloggingsolutions.Amongothers,syslog-ngOSEallowsyouthefollowing.
Secure and reliable log transfer
Thesyslog-ngOSEapplicationenablesyoutosendthelogmessagesofyourhoststoremoteserversusingthelatestprotocolstandards.Youcancollectandstoreyourlogdatacentrallyondedicatedlogservers.TransferlogmessagesusingtheTCPprotocolensuresthatnomessagesarelost.
Disk-based message buffering
Tominimizetheriskoflosingimportantlogmessages,thesyslog-ngOSEapplicationcanstoremessagesonthelocalharddiskifthecentrallogserverorthenetworkconnectionbecomesunavailable.Thesyslog-ngapplicationautomaticallysendsthestoredmessagestotheserverwhentheconnectionisreestablished,inthesameorderthemessageswerereceived.Thediskbufferispersistentnomessagesarelostevenifsyslog-ngisrestarted.
Secure logging using TLS
Logmessagesmaycontainsensitiveinformationthatshouldnotbeaccessedbythirdparties.Therefore,syslog-ngOSEsupportstheTransportLayerSecurity(TLS)protocolto
syslog-ng OSE 3.16 Administration Guide
Introduction to syslog-ng30
encryptthecommunication.TLSalsoallowsyoutoauthenticateyourclientsandthelogserverusingX.509certificates.
Flexible data extraction and processing
Mostlogmessagesareinherentlyunstructured,whichmakesthemdifficulttoprocess.Toovercomethisproblem,syslog-ngOSEcomeswithasetofbuilt-inparsers,whichyoucancombinetobuildverycomplexthings.
Filter and classify
Thesyslog-ngOSEapplicationcansorttheincominglogmessagesbasedontheircontentandvariousparameterslikethesourcehost,application,andpriority.Youcancreatedirectories,files,anddatabasetablesdynamicallyusingmacros.Complexfilteringusingregularexpressionsandbooleanoperatorsoffersalmostunlimitedflexibilitytoforwardonlytheimportantlogmessagestotheselecteddestinations.
Parse and rewrite
Thesyslog-ngOSEapplicationcansegmentlogmessagestonamedfieldsorcolumns,andalsomodifythevaluesofthesefields.YoucanprocessJSONmessages,key-valuepairs,andmore.
Togetthemostinformationoutofyourlogdata,syslog-ngOSEallowsyoutocorrelatelogmessagesandaggregatetheextractedinformationintoasinglemessage.Youcanalsouseexternalinformationtoenrichyourlogdata.
Big data clusters
Thelogdatathatyourorganizationhastoprocess,store,andreviewincreasesdaily,somanyorganizationsusebigdatasolutionsfortheirlogs.Toaccomodatethishugeamountofdata,syslog-ngOSEnativelysupportsstoringlogmessagesinHDFSfilesandElasticsearchclusters.
Message queue support
Largeorganizationsincreasinglyrelyonqueuinginfrastructuretotransfertheirdata.syslog-ngOSEsupportsApacheKafka,theAdvancedMessageQueuingProtocol(AMQP),andtheSimpleTextOrientedMessagingProtocol(STOMP).
syslog-ng OSE 3.16 Administration Guide
Introduction to syslog-ng31
SQL, NoSQL, and monitoring
Storing your log messages in a database allows you to easily search and query themessages and interoperate with log analyzing applications. The syslog-ng applicationsupports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, andSQLite.
syslog-ngOSEalsoallowsyoutoextracttheinformationyouneedfromyourlogdata,anddirectlysendittoyourGraphite,Redis,orRiemannmonitoringsystem.
Wide protocol and platform support
syslog protocol standards
syslog-ngnotonlysupportslegacyBSDsyslog(RFC3164)andtheenhancedRFC5424protocolsbutalsoJavaScriptObjectNotation(JSON)andjournaldmessageformats.
Heterogeneous environments
Thesyslog-ngOSEapplicationistheidealchoicetocollectlogsinmassivelyheterogeneousenvironmentsusingseveraldifferentoperatingsystemsandhardwareplatforms,includingLinux,Unix,BSD,SunSolaris,HP-UX,Tru64,andAIX.
IPv4 and IPv6 support
Thesyslog-ngapplicationcanoperateinbothIPv4andIPv6networkenvironments,andcanreceiveandsendmessagestobothtypesofnetworks.
What syslog-ng is not
Thesyslog-ngapplicationisnotloganalysissoftware.Itcanfilterlogmessagesandselectonlytheonesmatchingcertaincriteria.Itcanevenconvertthemessagesandrestructurethemtoapredefinedformat,orparsethemessagesandsegmentthemintodifferentfields.Butsyslog-ngcannotinterpretandanalyzethemeaningbehindthemessages,orrecognizepatternsintheoccurrenceofdifferentmessages.
Why is syslog-ng needed?
Logmessagescontaininformationabouttheeventshappeningonthehosts.Monitoringsystemeventsisessentialforsecurityandsystemhealthmonitoringreasons.
syslog-ng OSE 3.16 Administration Guide
Introduction to syslog-ng32
Theoriginalsyslogprotocolseparatesmessagesbasedonthepriorityofthemessageandthefacilitysendingthemessage.Thesetwoparametersaloneareofteninadequatetoconsistentlyclassifymessages,asmanyapplicationsmightusethesamefacility,andthefacilityitselfisnotevenincludedinthelogmessage.Tomakethingsworse,manylogmessagescontainunimportantinformation.Thesyslog-ngapplicationhelpsyoutoselectonlythereallyinterestingmessages,andforwardthemtoacentralserver.
Companypoliciesorotherregulationsoftenrequirelogmessagestobearchived.Storingtheimportantmessagesinacentrallocationgreatlysimplifiesthisprocess.
What is new in syslog-ng Open Source Edition 3.16?
Version3.16ofsyslog-ngOpenSourceEditionincludesthefollowingmainfeatures.
Easily receive and parse messages from remote hosts
Thedefault-network-drivers()sourceisaspecialsourcethatusesmultiplesourcedriverstoreceiveandparseseveraldifferenttypesofsyslogmessagesfromthenetwork.Fordetails,see"default-network-drivers()sourceoptions"intheAdministrationGuide.
Transfer log messages and their key-value pairs between syslog-ng nodes
TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,throughanynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.
Clearer configuration using if, else, elif conditions
Youcanuseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeAdministrationGuide.
Message parsing
syslog-ngOSEversion3.16includesparsersforthesudoandiptablesapplications.
Foramoredetailedlist,seeVersion3.14-3.15andthesyslog-ngReleasespage.
syslog-ng OSE 3.16 Administration Guide
Introduction to syslog-ng33
https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/https://github.com/balabit/syslog-ng/releases
Execute external programs during startup
Thehook-commands()optionmakesitpossibletoexecuteexternalprogramswhentherelevantdriverisinitializedortorndown.Itcanbeusedwithallsourceanddestinationdriverswiththeexceptionoftheusertty()andinternal()drivers.Fordetails,seetheoptiondescriptionsintheAdministrationGuide,andthehook-commandsblogpost.
Who uses syslog-ng?
Thesyslog-ngapplicationisusedworldwidebycompaniesandinstitutionswhocollectandmanagethelogsofseveralhosts,andwanttostoretheminacentralized,organizedway.Usingsyslog-ngisparticularlyadvantageousfor:
l InternetServiceProviders
l Financialinstitutionsandcompaniesrequiringpolicycompliance
l Server,web,andapplicationhostingcompanies
l Datacenters
l Wideareanetwork(WAN)operators
l Serverfarmadministrators.
Supported platformsThesyslog-ngOpenSourceEditionapplicationishighlyportableandisknowntorunonawiderangeofhardwarearchitectures(x86,x86_64,SUNSparc,PowerPC32and64,Alpha)andoperatingsystems,includingLinux,BSD,Solaris,IBMAIX,HP-UX,MacOSX,Cygwin,Tru64,andothers.
l Thesourcecodeofsyslog-ngOpenSourceEditionisreleasedundertheGPLv2licenseandisavailableonGitHub.
l Seethelistofprecompiledsyslog-ngOSEbinarypackages.
syslog-ng OSE 3.16 Administration Guide
Introduction to syslog-ng34
https://www.syslog-ng.com/community/b/blog/posts/hook-commands-easy-driver-setuphttps://github.com/balabit/syslog-nghttps://syslog-ng.org/3rd-party-binaries/
4
The concepts of syslog-ng
Thischapterdiscussesthetechnicalconceptsofsyslog-ng.
The philosophy of syslog-ng
Typically,syslog-ngisusedtomanagelogmessagesandimplementcentralizedlogging,wheretheaimistocollectthelogmessagesofseveraldevicesonasingle,centrallogserver.Thedifferentdevicescalledsyslog-ngclientsallrunsyslog-ng,andcollectthelogmessagesfromthevariousapplications,files,andothersources.Theclientssendallimportantlogmessagestotheremotesyslog-ngserver,whichsortsandstoresthem.
Logging with syslog-ngThesyslog-ngapplicationreadsincomingmessagesandforwardsthemtotheselecteddestinations.Thesyslog-ngapplicationcanreceivemessagesfromfiles,remotehosts,andothersources.
Logmessagesentersyslog-nginoneofthedefinedsources,andaresenttooneormoredestinations.
Sourcesanddestinationsareindependentobjects,log pathsdefinewhatsyslog-ngdoeswithamessage,connectingthesourcestothedestinations.Alogpathconsistsofoneormoresourcesandoneormoredestinations:messagesarrivingfromasourcearesenttoeverydestinationlistedinthelogpath.Alogpathdefinedinsyslog-ngiscalledalog statement.
Optionally,logpathscanincludefilters.Filtersarerulesthatselectonlycertainmessages,forexample,selectingonlymessagessentbyaspecificapplication.Ifalogpathincludesfilters,syslog-ngsendsonlythemessagessatisfyingthefilterrulestothedestinationssetinthelogpath.
Otheroptionalelements thatcanappear in logstatementsareparsersand rewriting rules.Parserssegmentmessages intodifferent fields tohelpprocessing themessages,while rewrite rulesmodify themessagesbyadding, replacing,or removingpartsofthemessages.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng35
The route of a log message in syslog-ng
Purpose:
Thefollowingprocedureillustratestherouteofalogmessagefromitssourceonthesyslog-ngclienttoitsfinaldestinationonthecentralsyslog-ngserver.
Figure 1: The route of a log message
Steps:
1. Adeviceorapplicationsendsalogmessagetoasourceonthesyslog-ngclient.Forexample,anApachewebserverrunningonLinuxentersamessageintothe/var/log/apachefile.
2. Thesyslog-ngclientrunningonthewebserverreadsthemessagefromits/var/log/apachesource.
3. Thesyslog-ngclientprocessesthefirstlogstatementthatincludesthe/var/log/apachesource.
4. Thesyslog-ngclientperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement,forexample,totheremotesyslog-ngserver.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng36
CAUTION:
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
NOTE:
Thesyslog-ngclientsendsamessagetoallmatchingdestinationsbydefault.Asaresult,amessagemaybesenttoadestinationmorethanonce,ifthedestinationisusedinmultiplelogstatements.Topreventsuchsituations,usethefinalflaginthedestinationstatements.Fordetails,seeLogstatementflags.
5. Thesyslog-ngclientprocessesthenextlogstatementthatincludesthe/var/log/apachesource,repeatingSteps3-4.
6. Themessagesentby thesyslog-ngclientarrives fromasourceset in thesyslog-ngserver.
7. Thesyslog-ngserverreadsthemessagefromitssourceandprocessesthefirstlogstatementthatincludesthatsource.
8. Thesyslog-ngserverperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement.
CAUTION:
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
9. Thesyslog-ngserverprocessesthenextlogstatement,repeatingSteps7-9.
NOTE:
Thesyslog-ngapplicationcanstopreadingmessagesfromitssourcesifthedestinationscannotprocessthesentmessages.Thisfeatureiscalledflow-controlandisdetailedinManagingincomingandoutgoingmessageswithflow-control.
Modes of operationThesyslog-ngOpenSourceEditionapplicationhasthreetypicaloperationscenarios:Client,Server,andRelay.
Client mode
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng37
Figure 2: Client-mode operation
Inclientmode,syslog-ngcollectsthelocallogsgeneratedbythehostandforwardsthemthroughanetworkconnectiontothecentralsyslog-ngserverortoarelay.Clientsoftenalsologthemessageslocallyintofiles.
Relay modeFigure 3: Relay-mode operation
Inrelaymode,syslog-ngreceiveslogsthroughthenetworkfromsyslog-ngclientsandforwardsthemtothecentralsyslog-ngserverusinganetworkconnection.Relaysalsologthemessagesfromtherelayhostintoalocalfile,orforwardthesemessagestothecentralsyslog-ngserver.
Server mode
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng38
Figure 4: Server-mode operation
Inservermode,syslog-ngactsasacentrallog-collectingserver.Itreceivesmessagesfromsyslog-ngclientsandrelaysoverthenetwork,andstoresthemlocallyinfiles,orpassesthemtootherapplications,forexampleloganalyzers.
Global objectsThesyslog-ngapplicationusesthefollowingobjects:
l Source driver:Acommunicationmethodusedtoreceivelogmessages.Forexample,syslog-ngcanreceivemessagesfromaremotehostviaTCP/IP,orreadthemessagesofalocalapplicationfromafile.Fordetailsonsourcedrivers,seesource:Read,receive,andcollectlogmessages.
l Source:Anamedcollectionofconfiguredsourcedrivers.
l Destination driver:Acommunicationmethodusedtosendlogmessages.Forexample,syslog-ngcansendmessagestoaremotehostviaTCP/IP,orwritethemessagesintoafileordatabase.Fordetailsondestinationdrivers,seedestination:Forward,send,andstorelogmessages.
l Destination:Anamedcollectionofconfigureddestinationdrivers.
l Filter:Anexpressiontoselectmessages.Forexample,asimplefiltercanselectthe
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng39
messagesreceivedfromaspecifichost.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Macro:Anidentifierthatreferstoapartofthelogmessage.Forexample,the${HOST}macroreturnsthenameofthehostthatsentthemessage.Macrosareoftenusedintemplatesandfilenames.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Parser:Parsersareobjectsthatparsetheincomingmessages,orpartsofamessage.Forexample,thecsv-parser()cansegmentmessagesintoseparatecolumnsatapredefinedseparatorcharacter(forexampleacomma).Everycolumnhasauniquenamethatcanbeusedasamacro.Fordetails,seeparser:Parseandsegmentstructuredmessagesanddb-parser:Processmessagecontentwithapatterndatabase(patterndb).
l Rewrite rule:Arulemodifiesapartofthemessage,forexample,replacesastring,orsetsafieldtoaspecifiedvalue.Fordetails,seeModifyingmessagesusingrewriterules.
l Log paths:Acombinationofsources,destinations,andotherobjectslikefilters,parsers,andrewriterules.Thesyslog-ngapplicationsendsmessagesarrivingfromthesourcesofthelogpathstothedefineddestinations,andperformsfiltering,parsing,andrewritingofthemessages.Logpathsarealsocalledlogstatements.Logstatementscanincludeother(embedded)logstatementsandjunctionstocreatecomplexlogpaths.Fordetails,seelog:Filterandroutelogmessagesusinglogpaths,flags,andfilters.
l Template:Atemplateisasetofmacrosthatcanbeusedtorestructurelogmessagesorautomaticallygeneratefilenames.Forexample,atemplatecanaddthehostnameandthedatetothebeginningofeverylogmessage.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.
l Option:Optionssetglobalparametersofsyslog-ng,liketheparametersofnameresolutionandtimezonehandling.Fordetails,seeGlobaloptionsofsyslog-ngOSE.
Fordetailsontheaboveobjects,seeTheconfigurationsyntaxindetail.
Timezones and daylight savingThesyslog-ngapplicationreceivesthetimezoneanddaylightsavinginformationfromtheoperatingsystemitisinstalledon.Iftheoperatingsystemhandlesdaylightsavingcorrectly,sodoessyslog-ng.
Thesyslog-ngapplicationsupportsmessagesoriginatingfromdifferenttimezones.Theoriginalsyslogprotocol(RFC3164)doesnotincludetimezoneinformation,butsyslog-ngprovidesasolutionbyextendingthesyslogprotocoltoincludethetimezoneinthelogmessages.Thesyslog-ngapplicationalsoenablesadministratorstosupplytimezoneinformationforlegacydeviceswhichdonotsupporttheprotocolextension.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng40
How syslog-ng OSE assigns timezone to the message
Whensyslog-ngOSEreceivesamessage,itassignstimezoneinformationtothemessageusingthefollowingalgorithm.
1. Thesenderapplication(forexamplethesyslog-ngclient)orhostspecifiesthetimezoneofthemessages.Iftheincomingmessageincludesatimezoneitisassociatedwiththemessage.Otherwise,thelocaltimezoneisassumed.
2. Specifythetime-zone()parameterforthesourcedriverthatreadsthemessage.Thistimezonewillbeassociatedwiththemessagesonlyifnotimezoneisspecifiedwithinthemessageitself.Eachsourcedefaultstothevalueoftherecv-time-zone()globaloption.Itisnotpossibletooverrideonlythetimezoneinformationoftheincomingmessage,butsettingthekeep-timestamp()optiontonoallowssyslog-ngOSEtoreplacethefulltimestamp(timezoneincluded)withthetimethemessagewasreceived.
NOTE:
Whenprocessingamessagethatdoesnotcontaintimezoneinformation,thesyslog-ngOSEapplicationwillusethetimezoneanddaylight-savingthatwaseffectivewhenthetimestampwasgenerated.Forexample,thecurrenttimeis2011-03-11(March11,2011)intheEU/Budapesttimezone.Whendaylight-savingisactive(summertime),theoffsetis+02:00.Whendaylight-savingisinactive(wintertime)thetimezoneoffsetis+01:00.Ifthetimestampofanincomingmessageis2011-01-01,thetimezoneassociatedwiththemessagewillbe+01:00,butthetimestampwillbeconverted,because2011-01-01meantwintertimewhendaylightsavingisnotactivebutthecurrenttimezoneis+02:00.
3. Specifythetimezoneinthedestinationdriverusingthetime-zone()parameter.Eachdestinationdrivermighthaveanassociatedtimezonevalue:syslog-ngconvertsmessagetimestampstothistimezonebeforesendingthemessagetoitsdestination(fileornetworksocket).Eachdestinationdefaultstothevalueofthesend-time-zone()globaloption.
NOTE:
Amessagecanbesenttomultipledestinationzones.Thesyslog-ngapplicationconvertsthetimezoneinformationproperlyforeveryindividualdestinationzone.
CAUTION:
If syslog-ng OSE sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so syslog-ng OSE will convert the hour:min values based on the explicitly specified timezone.
4. Ifthetimezoneisnotspecified,localtimezoneisused.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng41
5. Whenmacroexpansionsareusedinthedestinationfilenames,thelocaltimezoneisused.(Also,ifthetimestampofthereceivedmessagedoesnotcontaintheyearofthemessage,syslog-ngOSEusesthelocalyear.)
A note on timezones and timestampsIf the clients run syslog-ng, then use the ISO timestamp, because it includestimezone information. Thatway you do not need to adjust the recv-time-zone()parameter of syslog-ng.
Ifyouwantsyslog-ngtooutputtimestampsinUnix(POSIX)timeformat,usetheS_UNIXTIMEandR_UNIXTIMEmacros.Youdonotneedtochangeanyofthetimezonerelatedparameters,becausethetimestampinformationofincomingmessagesisconvertedtoUnixtimeinternally,andUnixtimeisatimezone-independenttimerepresentation.(Actually,UnixtimemeasuresthenumberofsecondselapsedsincemidnightofCoordinatedUniversalTime(UTC)January1,1970,butdoesnotcountleapseconds.)
Product licensing
Startingwithversion3.2,thesyslog-ngOpenSourceEditionapplicationislicensedunderacombinedLGPL+GPLlicense.Thecoreofsyslog-ngOSEislicensedundertheGNULesserGeneralPublicLicenseVersion2.1license,whiletherestofthecodebaseislicensedundertheGNUGeneralPublicLicenseVersion2license.
NOTE:
Practically,thecodestoredunderthelibdirectoryofthesourcecodepackageisunderLGPL,therestisGPL.
FordetailsabouttheLGPLandGPLlicenses,seeGNULesserGeneralPublicLicenseandGNUGeneralPublicLicense,respectively.
High availability supportMultiplesyslog-ngserverscanberuninfail-overmode.Thesyslog-ngapplicationdoesnotincludeanyinternalsupportforthis,asclusteringsupportmustbeimplementedontheoperatingsystemlevel.AtoolthatcanbeusedtocreateUNIXclustersisHeartbeat(fordetails,seethispage).
The structure of a log messageThefollowingsectionsdescribethestructureoflogmessages.Currentlytherearetwostandardsyslogmessageformats:
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng42
http://www.linux-ha.org/wiki/Main_Page/
l TheoldstandarddescribedinRFC3164(alsocalledtheBSD-syslogorthelegacy-syslogprotocol):seeBSD-syslogorlegacy-syslogmessages
l ThenewstandarddescribedinRFC5424(alsocalledtheIETF-syslogprotocol):seeIETF-syslogmessages
l TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesbetweensyslog-ngnodes:seeEnterprise-widemessagemodel(EWMM)
l Howmessagesarerepresentedinsyslog-ngOSE:seeMessagerepresentationinsyslog-ngOSE.
BSD-syslog or legacy-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtothelegacy-syslogorBSD-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:
l PRI
l HEADER
l MSG
Thetotalmessagecannotbelongerthan1024bytes.
Thefollowingisasamplesyslogmessage:
Feb 25 14:09:07 webserver syslogd: restart
Themessagecorrespondstothefollowingformat:
timestamp hostname application: message
Thedifferentpartsofthemessageareexplainedinthefollowingsections.
NOTE:
Thesyslog-ngapplicationsupportslongermessagesaswell.Fordetails,seethelog-msg-size()optioninGlobaloptions.However,itisnotrecommendedtoenablemessageslargerthanthepacketsizewhenusingUDPdestinations.
The PRI message part
ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.
NOTE:
Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng43
https://tools.ietf.org/search/rfc3164https://tools.ietf.org/search/rfc3164
Numerical Code Facility
0 kernelmessages
1 user-levelmessages
2 mailsystem
3 systemdaemons
4 security/authorizationmessages
5 messagesgeneratedinternallybysyslogd
6 lineprintersubsystem
7 networknewssubsystem
8 UUCPsubsystem
9 clockdaemon
10 security/authorizationmessages
11 FTPdaemon
12 NTPsubsystem
13 logaudit
14 logalert
15 clockdaemon
16-23 locallyusedfacilities(local0-local7)
Table 1: syslog Message Facilities
Thefollowingtableliststheseverityvalues.
Numerical Code Severity
0 Emergency:systemisunusable
1 Alert:actionmustbetakenimmediately
2 Critical:criticalconditions
3 Error:errorconditions
4 Warning:warningconditions
5 Notice:normalbutsignificantcondition
6 Informational:informationalmessages
7 Debug:debug-levelmessages
Table 2: syslog Message Severities
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng44
The HEADER message partTheHEADERpartcontainsatimestampandthehostname(withoutthedomainname)ortheIPaddressofthedevice.ThetimestampfieldisthelocaltimeintheMmm dd hh:mm:ssformat,where:
l MmmistheEnglishabbreviationofthemonth:Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec.
l ddisthedayofthemonthontwodigits.Ifthedayofthemonthislessthan10,thefirstdigitisreplacedwithaspace.(ForexampleAug 7.)
l hh:mm:ssisthelocaltime.Thehour(hh)isrepresentedina24-hourformat.Validentriesarebetween00and23,inclusive.Theminute(mm)andsecond(ss)entriesarebetween00and59inclusive.
NOTE:
Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.Fordetails,seethets-format()optioninGlobaloptions.
The MSG message part
TheMSGpartcontainsthenameoftheprogramorprocessthatgeneratedthemessage,andthetextofthemessageitself.TheMSGpartisusuallyinthefollowingformat:program[pid]: message text.
IETF-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtotheIETF-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:
l HEADER(includesthePRIaswell)
l STRUCTURED-DATA
l MSG
Thefollowingisasamplesyslogmessage(source:https://tools.ietf.org/html/rfc5424):
1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
Themessagecorrespondstothefollowingformat:
VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
Inthisexample,theFacilityhasthevalueof4,severityis2,soPRIis34.TheVERSIONis1.Themessagewascreatedon11October2003at10:14:15pmUTC,3millisecondsintothenextsecond.Themessageoriginatedfromahostthatidentifiesitselfas"mymachine.example.com".TheAPP-NAMEis"su"andthePROCIDisunknown.The
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng45
https://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424
MSGIDis"ID47".TheMSGis"'suroot'failedforlonvick...",encodedinUTF-8.TheencodingisdefinedbytheBOM:
Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.
ThereisnoSTRUCTURED-DATApresentinthemessage,thisisindicatedby"-"intheSTRUCTURED-DATAfield.TheMSGis"'suroot'failedforlonvick...".
TheHEADERpartofthemessagemustbeinplainASCIIformat,theparametervaluesoftheSTRUCTURED-DATApartmustbeinUTF-8,whiletheMSGpartshouldbeinUTF-8.Thedifferentpartsofthemessageareexplainedinthefollowingsections.
The PRI message part
ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.
NOTE:
Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.
Numerical Code Facility
0 kernelmessages
1 user-levelmessages
2 mailsystem
3 systemdaemons
4 security/authorizationmessages
5 messagesgeneratedinternallybysyslogd
6 lineprintersubsystem
7 networknewssubsystem
8 UUCPsubsystem
9 clockdaemon
10 security/authorizationmessages
11 FTPdaemon
Table 3: syslog Message Facilities
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng46
Numerical Code Facility
12 NTPsubsystem
13 logaudit
14 logalert
15 clockdaemon
16-23 locallyusedfacilities(local0-local7)
Thefollowingtableliststheseverityvalues.
Numerical Code Severity
0 Emergency:systemisunusable
1 Alert:actionmustbetakenimmediately
2 Critical:criticalconditions
3 Error:errorconditions
4 Warning:warningconditions
5 Notice:normalbutsignificantcondition
6 Informational:informationalmessages
7 Debug:debug-levelmessages
Table 4: syslog Message Severities
The HEADER message partTheHEADERpartcontainsthefollowingelements:
l VERSION: Version number of the syslog protocol standard. Currently this canonly be 1.
l ISOTIMESTAMP:ThetimewhenthemessagewasgeneratedintheISO8601compatiblestandardtimestampformat(yyyy-mm-ddThh:mm:ss+-ZONE),forexample:2006-06-13T15:58:00.123+01:00.
l HOSTNAME:Themachinethatoriginallysentthemessage.
l APPLICATION:Thedeviceorapplicationthatgeneratedthemessage
l PID:TheprocessnameorprocessIDofthesyslogapplicationthatsentthemessage.It isnotnecessarilytheprocessIDoftheapplicationthatgeneratedthemessage.
l MESSAGEID:TheIDnumberofthemessage.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng47
NOTE:
Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.ThetimestampusedintheIETF-syslogprotocolisderivedfromRFC3339,whichisbasedonISO8601.Fordetails,seethets-format()optioninGlobaloptions.
Thesyslog-ngOSEapplicationwilltruncatethefollowingfields:
l IfAPP-NAMEislongerthan48charactersitwillbetruncatedto48characters.
l IfPROC-IDislongerthan128charactersitwillbetruncatedto128characters.
l IfMSGIDislongerthan32charactersitwillbetruncatedto32characters.
l IfHOSTNAMEislongerthan255charactersitwillbetruncatedto255characters.
The STRUCTURED-DATA message part
TheSTRUCTURED-DATAmessagepartmaycontainmeta-informationaboutthesyslogmessage,orapplication-specificinformationsuchastrafficcountersorIPaddresses.STRUCTURED-DATAconsistsofdatablocksenclosedinbrackets([]).EveryblockincludestheIDoftheblock,andoneormorename=valuepairs.Thesyslog-ngapplicationautomaticallyparsestheSTRUCTURED-DATApartofsyslogmessages,whichcanbereferencedinmacros(fordetails,seeMacrosofsyslog-ngOSE).AnexampleSTRUCTURED-DATAblocklookslike:
[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
The MSG message part
TheMSGpartcontainsthetextofthemessageitself.TheencodingofthetextmustbeUTF-8iftheBOM1characterispresentinthemessage.IfthemessagedoesnotcontaintheBOMcharacter,theencodingistreatedasunknown.UsuallymessagesarrivingfromlegacysourcesdonotincludetheBOMcharacter.CRLFcharacterswillnotberemovedfromthemessage.
Enterprise-wide message model (EWMM)ThefollowingsectiondescribesthestructureoflogmessagesusingtheEnterprise-widemessagemodelorEWMMmessageformat.
TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,through
1Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng48
anynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.
ThefollwingisasamplelogmessageinEWMMformat.
1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
Themessagehasthefollowingparts.
l TheheaderofthecomplieswiththeRFC5424messageformat,wherethePROGRAMfieldissetto@syslog-ng,andtheSDATAfieldisempty.
l TheMESSAGEpartisinJSONformat,andcontainstheactualmessage,aswellasanyname-valuepairsthatsyslog-ngOSEhasattachedtoorextractedfromthemessage.The${._TAGS}fieldcontainstheidentifierofthesyslog-ngsourcethathasoriginallyreceivedthemessageonthefirstsyslog-ngnode.
TosendamessageinEWMMformat,youcanusethesyslog-ng()destinationdriver,ortheformat-ewmm()templatefunction.
ToreceiveamessageinEWMMformat,youcanusethedefault-destination-drivers()sourcedriver,ortheewmm-parser()parser.
Message representation in syslog-ng OSEWhenthesyslog-ngOSEapplicationreceivesamessage,itautomaticallyparsesthemessage.Thesyslog-ngOSEapplicationcanautomaticallyparselogmessagesthatconformtotheRFC3164(BSDorlegacy-syslog)ortheRFC5424(IETF-syslog)messageformats.Ifsyslog-ngOSEcannotparseamessage,itresultsinanerror.
TIP:
Incaseyouneedtorelaymessagesthatcannotbeparsedwithoutanymodificationsorchanges,usetheflags(no-parse)optioninthesourcedefinition,andatemplatecontainingonlythe${MESSAGE}macrointhedestinationdefinition.
Toparsenon-syslogmessages,forexample,JSON,CSV,orothermessages,youcanusethebuilt-inparsersofsyslog-ngOSE.Fordetails,seeparser:Parseandsegmentstructuredmessages.
Aparsedsyslogmessagehasthefollowingparts.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng49
l Timestamps
Twotimestampsareassociatedwitheverymessage:oneisthetimestampcontainedwithinthemessage(thatis,whenthesendersentthemessage),theotheristhetimewhensyslog-ngOSEhasactuallyreceivedthemessage.
l Severity
Theseverityofthemessage.
l Facility
Thefacilitythatsentthemessage.
l Tags
Customtextlabelsaddedtothemessagethataremainlyusedforfiltering.Noneofthecurrentmessagetransportprotocolsaddstagstothelogmessages.Tagscanbeaddedtothelogmessageonlywithinsyslog-ngOSE.Thesyslog-ngOSEapplicationautomaticallyaddstheidofthesourceasatagtotheincomingmessages.Othertagscanbeaddedtothemessagebythepatterndatabase,orusingthetags()optionofthesource.
l IP address of the sender
TheIPaddressofthehostthatsentthemessage.NotethattheIPaddressofthesenderisahardmacroandcannotbemodifiedwithinsyslog-ngOSEbuttheassociatedhostnamecanbemodified,forexample,usingrewriterules.
l Hard macrosHardmacroscontaindatathatisdirectlyderivedfromthelogmessage,forexample,the${MONTH}macroderivesitsvaluefromthetimestamp.Themostimportantconsiderationwithhardmacrosisthattheyareread-only,meaningtheycannotbemodifiedusingrewriterulesorothermeans.
l Soft macrosSoftmacros(sometimesalsocalledname-valuepairs)areeitherbuilt-inmacrosautomaticallygeneratedfromthelogmessage(forexample,${HOST}),orcustomuser-createdmacrosgeneratedbyusingthesyslog-ngpatterndatabaseoraCSV-parser.TheSDATAfieldsofRFC5424-formattedlogmessagesbecomesoftmacrosaswell.Incontrastwithhardmacros,softmacrosarewritableandcanbemodifiedwithinsyslog-ngOSE,forexample,usingrewriterules.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng50
NOTE:
Itisalsopossibletosetthevalueofbuilt-insoftmacrosusingparsers,forexample,tosetthe${HOST}macrofromthemessageusingacolumnofaCSV-parser.
Thedataextractedfromthelogmessagesusingnamedpatternparsersinthepatterndatabasearealsosoftmacros.
TIP:
Forthelistofhardandsoftmacros,seeHardvs.softmacros.
Message size and encodingInternally,syslog-ngOSErepresentseverymessageasUTF-8.Themaximallengthofthelogmessagesislimitedbythelog-msg-size()option:ifamessageislongerthanthisvalue,syslog-ngOSEtruncatesthemessageatthelocationitreachesthelog-msg-size()value,anddiscardstherestofthemessage.
Whenencodingissetinasource(usingtheencoding()option)andthemessageislonger(inbytes)thanlog-msg-size()inUTF-8representation,syslog-ngOSEsplitsthemessageatanundefinedlocation(becausetheconversionbetweendifferentencodingsisnottrivial).
Structuring macros, metadata, and other value-pairs
Available in syslog-ng OSE 3.3 and later.
Thesyslog-ngOSEapplicationallowsyoutoselectandconstructname-valuepairsfromanyinformationalreadyavailableaboutthelogmessage,orextractedfromthemessageitself.Youcandirectlyusethisstructuredinformation,forexample,inthefollowingplaces:
l amqp()destination
l format-welf()templatefunction
l mongodb()destination
l stomp()destination
l orinotherdestinationsusingtheformat-json()templatefunction.
Whenusingvalue-pairs,therearethreewaystospecifywhichinformation(thatis,macrosorothername-valuepairs)toincludeintheselection.
l Selectgroupsofmacrosusingthescope()parameter,andoptionallyremovecertainmacrosfromthegroupusingtheexclude()parameter.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng51
l Listspecificmacrostoincludeusingthekey()parameter.
l Definenewname-valuepairstoincludeusingthepair()parameter.
Theseparametersaredetailedinvalue-pairs().
Specifying data types in value-pairsBydefault,syslog-ngOSEhandleseverydataasstrings.However,certaindestinationsanddataformats(forexample,SQL,MongoDB,JSON,AMQP)supportothertypesofdataaswell,forexample,numbersordates.Thesyslog-ngOSEapplicationallowsyoutospecifythedatatypeintemplates(thisisalsocalledtype-hinting).Ifthedestinationdriversupportsdatatypes,itconvertstheincomingdatatothespecifieddatatype.Forexample,thisallowsyoutostoreintegernumbersasnumbersinMongoDB,insteadofstrings.
CAUTION:
Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error().
Tousetype-hinting,enclosethemacroortemplatecontainingthedatawiththetype:(""),forexample:int("$PID").
Currently the mongodb() destination and the format-json template function supportsdata types.
Example: Using type-hintingThefollowingexamplestorestheMESSAGE,PID,DATE,andPROGRAMfieldsofalogmessageinaMongoDBdatabase.TheDATEandPIDpartsarestoredasnumbersinsteadofstrings.
mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );
ThefollowingexampleformatsthesamefieldsintoJSON.
$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)
Thesyslog-ngOSEapplicationcurrentlysupportsthefollowingdata-types.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng52
l boolean:Convertsthedatatoabooleanvalue.Anythingthatbeginswithator1isconvertedtotrue,anythingthatbeginswithanfor0isconvertedtofalse.
l datetime: Use it onlywithUNIX timestamps, anything elsewill likely result inan error. Thismeans that currently you can use only the $UNIXTIMEmacro forthis purpose.
l double:Afloating-pointnumber.
l literal:Thedataasaliteralstring,withoutaddinganyquotesorescapecharacters.
l intorint32:32-bitinteger.
l int64:64-bitinteger.
l string:Thedataasastring.
value-pairs()
Type: parameterlistofthevalue-pairs()option
Default: empty string
Description:Thevalue-pairs()optionallowsyoutoselectspecificinformationaboutamessageeasilyusingpredefinedmacrogroups.Theselectedinformationisrepresentedasname-valuepairsandcanbeusedformattedtoJSONformat,ordirectlyusedinamongodb()destination.
Example: Using the value-pairs() optionThefollowingexampleselectseveryavailableinformationaboutthelogmessage,exceptforthedate-relatedmacros(R_*andS_*),selectsthe.SDATA.meta.sequenceIdmacro,anddefinesanewvalue-paircalledMSGHDRthatcontainstheprogramnameandPIDoftheapplicationthatsentthelogmessage.
value-pairs( scope(nv_pairs core syslog all_macros selected_macros everything) exclude("R_*") exclude("S_*") key(".SDATA.meta.sequenceId") pair("MSGHDR" "$PROGRAM[$PID]: ") )
Thefollowingexampleselectsthesameinformationasthepreviousexample,butconvertsitintoJSONformat.
$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng53
--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \ --pair MSGHDR="$PROGRAM[$PID]: ")
NOTE:
Everymacroisincludedintheselectiononlyonce,butredundantinformationmayappearifmultiplemacrosincludethesameinformation(forexample,includingseveraldate-relatedmacrosintheselection).
Thevalue-pairs()optionhasthefollowingparameters.Theparametersareevaluatedinthefollowingorder:
1. scope()
2. exclude()
3. key()
4. pair()
exclude()
Type: Space-separatedlistofmacrostoremovefromtheselectioncreatedusingthescope()option.
Default: emptystring
Description:Thisoptionremovesthespecifiedmacrosfromtheselection.Useittoremoveunneededmacrosselectedusingthescope()parameter.
Forexample,thefollowingexampleremovestheSDATAmacrosfromtheselection.
value-pairs( scope(rfc5424 selected_macros) exclude(".SDATA*") )
Thenameofthemacrotoremovecanincludewildcards(*, ?).Regularexpressionsarenotsupported.
key()
Type: Space-separatedlistofmacrostobeincludedinselection
Default: emptystring
Description:Thisoptionselectsthespecifiedmacros.TheselectedmacroswillbeincludedasMACRONAME = MACROVALUE,thatisusingkey("HOST")willresultinHOST = $HOST.Youcanusewildcards(*, ?)toselectmultiplemacros.Forexample:
value-pairs(
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng54
scope(rfc3164) key("HOST") )
value-pairs( scope(rfc3164) key("HOST", "PROGRAM") )
pair()
Type: namevaluepairsin"" ""format
Default: emptystring
Description:Thisoptiondefinesanewname-valuepairtobeincludedinthemessage.Thevaluepartcanincludemacros,templates,andtemplatefunctionsaswell.Forexample:
value-pairs( scope(rfc3164) pair("TIME" "$HOUR:$MIN") pair("MSGHDR" "$PROGRAM[$PID]: ") )
rekey()
Type: ,
Default: emptystring
Description:Thisoptionallowsyoutomanipulateandmodifythenameofthevalue-pairs.Youcandefinetransformations,whichareareappliedtotheselectedname-valuepairs.Thefirstparameteroftherekey()optionisaglobpatternthatselectsthename-valuepairstomodify.Ifyouomitthepattern,thetransformationsareappliedtoeverykeyofthescope.Fordetailsonglobs,seeglob.
Ifyouwanttomodifythenamesofseveralmessagefields,seealsomap-value-pairs:Renamevalue-pairstonormalizelogs.
l Ifrekey()isusedwithinakey()option,thename-valuepairsspecifiedintheglobofthekey()optionaretransformed.
l Ifrekey()isusedoutsidethekey()option,everyname-valuepairofthescope()istransformed.
Thefollowingtransformationsareavailable:
l add-prefix("")
l Addsthespecifiedprefixtoeveryname.Forexample,rekey( add-prefix("my-
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng55
prefix."))
l replace-prefix("", "")
l Replacesasubstringatthebeginningofthekeywithanotherstring.Onlyprefixescanbereplaced.Forexample,replace-prefix(".class", ".patterndb")changesthebeginningtag.classto.patterndb
Thisoptionwascalledreplace()insyslog-ngOSEversion3.4.
l shift("")
l Cutsthespecifiednumberofcharactersfromthebeginningofthename.
Example: Using the rekey() option
Thefollowingsampleselectseveryvalue-pairthatbeginswith.cee.,deletesthisprefixbycutting4charactersfromthenames,andaddsanewprefix(events.).
value-pairs( key(".cee.*" rekey( shift(4) add-prefix("events.") ) ) )
Therekey()optioncanbeusedwiththeformat-jsontemplate-functionaswell,usingthefollowingsyntax:
$(format-json --rekey .cee.* --add-prefix events.)
scope()
Type: space-separatedlistofmacrogroupstoincludeinselection
Default: emptystring
Description:Thisoptionselectspredefinedgroupsofmacros.Thefollowinggroupsareavailable:
l nv-pairs:Everysoftmacro(name-valuepair)associatedwiththemessage,excepttheonesthatstartwithadot(.)character.Macrosstartingwithadotcharacteraregeneratedwithinsyslog-ngOSEandarenotoriginallypartofthemessage,thereforearenotincludedinthisgroup.
l dot-nv-pairs:Everysoftmacro(name-valuepair)associatedwiththemessagewhichstartswithadot(.)character.Forexample,.classifier.rule_idand.sdata.*.Macrosstartingwithadotcharacteraregeneratedwithinsyslog-ngOSEandarenotoriginallypartofthemessage.
syslog-ng OSE 3.16 Administration Guide
The concepts of syslog-ng56
l all-nv-pairs:Includeeverysoftmacro(name-valuepair).Equivalenttousingbothnv-pairsanddot-nv-pairs.
l rfc3164:ThemacrosthatcorrespondtotheRFC3164(legacyorBSD-syslog)messageformat:$FACILITY,$PRIORITY,$HOST,$PROGRAM,$PID,$MESSAGE,and$DATE.
l rfc5424:ThemacrosthatcorrespondtotheRFC5424(IETF-syslog)messageformat:$FACILITY,$PRIORITY,$HOST,$PROGRAM,$PID,$MESSAGE,$MSGID,$R_DATE,andthemetadatafromthestructured-data(SDATA)partofRFC5424-formattedmessages,thatis,everymacrothatstartswith.SDATA..
Therfc5424groupalsohasthefollowingalias:syslog-proto.Notethatthevalueof$R_DATEwillbelistedundertheDATEkey.
Therfc5424groupdoesnotcontainanymetadataaboutthemessage,onlyinformationthatwaspresentintheoriginalmessage.Toincludethemostcommonlyusedmetadata(forexample,the$SOURCEIPmacro),usetheselected-macrosgroupinstead.
l all-macros:Includeeveryhardmacro.Thisgroupismainlyusefulfordebugging,asitcontainsredundantinformation(forexample,thedate-relatedmacrosincludethedate-relatedinformationseveraltimesinvariousformats).
l selected-macros:Includethemacrosoftherfc3164groups,andthemostcommonlyusedmetadataaboutthelogmessage:the$TAGS,$SOURCEIP,and$SEQNUMmacros.
l sdata:Themetadatafromthestructured-data(SDATA)partofRFC5424-formattedmessages,thatis,everymacrothatstartswith.SDATA.
l everything:Includeeveryhardandsoftmacros.Thisgroupismainlyusefulfordebugging,asitcontainsredundantinformation(forexample,thedate-relatedmacrosincludethedate-relatedinformationseveraltimesinvariousformats).
Forexample:
value-pairs( scope(rfc3164 selected-macros) )
Things to consider when forwarding messages between syslog-ng OSE hosts
Whenyousendyourlogmessagesfromasyslog-ngOSEclientthroughthenetworktoasyslog-ngOSEserver,youcanusedifferentprotocolsandoptions.Everycombinationhasitsadvantagesanddisadvantages.Themostimportantthingistousematchingprotocolsandoptions,sotheserverhandlestheincominglogmessagesproperly.
Insyslog-ngOSEyoucanchangemanyaspectsofthenetworkcommunication.Firstofall,thereisthestructureofthemessagesitself.Currently,syslog-ngOSEsupportstwostandardsyslogprotocols:theBSD(RFC3164)andthesyslog(RFC5424)messageformat.
syslog-ng OSE 3.16 Administration Guide
The co