Trends in Database Log Management by Anton Chuvakin

8/14/2019 Trends in Database Log Management by Anton Chuvakin

1/5

Trends in Database Log Management

Anton Chuvakin, Ph.D.

WRITTEN: 2007

DISCLAIMER:Security is a rapidly changing field of human endeavor. Threats we face literallychange every day; moreover, many security professionals consider the rate ofchange to be accelerating. On top of that, to be able to stay in touch with suchever-changing reality, one has to evolve with the space as well. Thus, eventhough I hope that this document will be useful for to my readers, please keep inmind that is was possibly written years ago. Also, keep in mind that some of theURL might have gone 404, please Google around.

Introduction: Why Database Logs Fall In The Realm of Database Security

Buried deep within enterprise IT infrastructures, databases can be said tohold the crown jewels of an organization. Unfortunately, database security isoften lacking, leaving sensitive, business-critical information such as customerdata, financial details, and more, vulnerable to hackers. Dept of VA, TJ Maxx, TDAmeritrade these are just a few of the many organizations that have driven themedia wild over data security breaches in the last year.

It is common that database administrators (DBAs) are assigned the task ofdatabase security, but this is an issue that should be of utmost importance to anybusiness that wants to stay in business. TJ Maxx reported at least 45.7 million

credit and debit card numbers stolen over a period of several years, costing thecompany an estimated $168 million [or whatever other large random number].Proper security measures may not have stopped the initial hack-in, but perpetualdata theft could have been avoided through careful log collection and analysis.This article will not only discuss the importance, challenges and benefits todatabase logging, but will also offer a few forward-looking trends to managingyour database logs.

About Logs and Database Logging

Databases are now becoming one of the most voluminous log generatorsin the enterprise rivaling firewalls for the top spot. Most databases (ie: Oracle,Microsoft SQL Server, IBM DB2, MySQL, etc.) will log system starts, stops andrestarts by default, but database logging isnt merely about keeping the systemrunning, particularly when your databases contain sensitive, private information.Security and compliance requirements must therefore be considered whenconfiguring your database and managing your logs. In fact, regulations such as


2/5

PCI, HIPAA, and FISMA all mandate log monitoring, with SOX stronglyrecommending it as a best practice.

Database logging thereby becomes an essential (and required)component of database security and it makes sense to not only focus onkeeping the bad guys out, but also take a whats going on in here? approach.

After all, you may not know who the bad guys are. Logs can provide acontinuous fingerprint of everything that happens in your IT systems and withyour data and will point you to the who, what, when, where information of anybreach whether the malicious behavior comes from outside hackers, adisgruntled employee, or another source.

Database security is a task often assigned to DBAs, not because theyresecurity experts, but because they know the ins and outs of databases. Ifconfigured properly, databases may be logging overwhelming amounts of files,perhaps up to gigabytes of data per day. Typical database log events mayinclude:

User logins and logouts Database system starts, stops and restarts

Various system failures and errors

User privilege changes

Database structure (metadata) changes

Most other DBA actions

Select or all database data access (if configured to be so)

As we know, hackers are always looking for new ways to break throughsecurity barriers to access your sensitive information and all preventative securitymeasures fail at some point. Thus, since you are not able to guard against every

malicious hacker, logs will at least allow you to detect such security breaches aswell as actually figure out how it was done during the incident investigation. At aminimal level, logs must be collected and archived, but log analysis does makethe data significantly more useful. In more explicit terms, log monitoring andmanagement should include:

Collection: Gathering log data where it is being generated via an agent orremotely

Transfer: Securely transmitting log data to a central server for analysisand storage

Alerts: Issuing real-time alerts to database administrators if needed

Reporting and Analysis: Providing reports and analytics based on log data Storage: Securely storing logs as long as prescribed by your retention

policy and then, just as safely, destroying them

The above examples for managing your log data will help you keep tabson the activities occurring in your business. Regularly collecting log data is a bestpractice for incident response and can save you during crunch time after a servercrash, data theft, or surprise visit by your friendly auditor. Alternatively, if


3/5

someone is downloading an entire table or changing a database schema whilebeing logged on from a remote connection, a real-time alert will catch yourattention. Further, reports may help you track and analyze login failures andsuccesses, or after hours access, to better evaluate insider privilege abuse. Inother words, database logs can help you catch unusual behavior before a

problem gets out of hand and into headline news.

Database Log Management: Where to Begin

If you are just beginning to set up a method for managing your databaselog data, be ready for a large volume of log records as well as issues pertainingto log availability and log format complexity. Log formats can be verbose andobscure, particularly in cases where a single message spans multiple lines,making it difficult to extract useful, actionable information via automated tools.

Other challenges to database logging include decreased performance and

storage restrictions. Unlike other situations where logging has a minimal impacton system performance, database audit logging slows down the database,sometimes significantly. High-performance databases are built to providethousands of data transactions per second, logging all of these presents achallenge to system IO as well as CPU and disk storage resources. Since manyregulations specify a 3-12 month period for log retention, plus a longer period forlog retention on tape or another dedicated storage tool, database logging istypically getting a bad rap among DBAs already spread thin for time andresources.

Because the difficulties associated with database log management canseem overwhelming, its best to take things one step at a time. Start slowly andbuild up your system from there. Youll want to collect logs from multiple serversat one central location to facilitate incident analysis and response. This will alsohelp prevent loss of log data during routine log rotations. To gain insight intoWhats going on internally, conduct periodic reviews of DBA activity logs youcan then keep tabs on people entrusted with sensitive and/or private informationsuch as customer data or product inventory information.

When beginning to organize and manage database log data, also keep inmind that manual log analysis can cost a lot in terms of time, human resources,etc. Popular database solutions such as Oracle, Microsoft SQL Server, MySQL,and more, tend to offer various basic logging options, but none comprehensiveenough to really capture a continuous feed of database activity. By contrast, anautomated log management tool will not only free up DBA time for otherimportant database performance and security tasks, but can also be morereliable and efficient than manually managing log data.

Further, with an automated LMI tool, you can schedule log collection to occur atoff hours when other database service operations, such as backups, arehappening so that database performance is undeterred during the workday.


4/5

Trends in Database Log Management

Database logging often presents a new frontier for many securitypractitioners one that must be conquered. Given that historically many

databases were running without any data access and data change loggingdisabled (as by default), the key trend is that this is finally beginning to change.Why is it happening? There are two main drivers for this trend: PCI DSScompliance requirements that apply to those who handle credit card data and theproliferation of data breaches and data loss discussed above. The cost of dataloss investigations in the absence of detailed access logs is absurdly high!

What would happen next? As more people enable logging, the challenge of whatto do with all that data? will emerge. Handling log storage and controlling logretention so that logs will be there when needed for the investigations will be thenext trend.

After that, database log analysis will become all the rage. Analyzing logs foranomalies, suspicious user activities, unsafe administrator actions, privilegeabuse as well as good old hacking will require deployment of log managementtools with database-specific intelligence.

Further, logging guidance from IT best practices such as ITIL and ISO willbecome the norm and we will reach the database logging nirvana, when loggingis enabled because it is the right thing and not only due to compliancepressures or the latest data breach. Log collection and automated analysis willbecome the norm as new a log management and intelligence (LMI) technologiessimplify managing the log flood as well as making sense of logs.

So, enabling logging is a good start, and taking small progressive steps towardsmore in-depth log analysis can be greatly enhanced by deploying LMI platform.

LMI tools are becoming increasingly advanced and are now typically able to takea deep dive in to log analysis and management. A good log managementsolution will not only automate log data analysis, but also the whole lifecycle oflog management.

There is also an increasing trend in logging across the entire IT infrastructure firewalls, servers, network devices, applications, operating systems, othersources of log data all produce logs! all of which can be managed by an LMIplatform. In other words, jump on the logging bandwagon with your other systemadministrators and balance your IT infrastructure with a log management systemthat will work across multiple database servers, across various database types,and with all other log producers in your organization. You will not only improveperformance and business continuity, but also be able to put database log data in


5/5

the context of other organizational log data to correlate IT activities with eventsoccurring in your business.

Conclusion

Database log management is becoming a best practice for databasesecurity you should be aware of who is accessing or changing your data, whenthey are accessing it, and where they are accessing it. Luckily, you can combinedatabase log management with other similar projects (such as firewall or Unixserver syslog management) and use a single automated LMI platform to enableefficient and reliable log collection, reporting analysis, and retention. As long asyou grow your LMI deployment in phases, rather than trying to cover all logs onday one, you will be on the path to overall greater IT security within yourorganization.

ABOUT THE AUTHOR:

This is an updated author bio, added to the paper at the time of reposting in2009.

Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert inthe field of log management and PCI DSS compliance. He is an author of books"Security Warrior" and "PCI Compliance" and a contributor to "Know Your EnemyII", "Information Security Management Handbook" and others. Anton haspublished dozens of papers on log management, correlation, data analysis, PCIDSS, security management (see list www.info-secure.org) . His bloghttp://www.securitywarrior.org is one of the most popular in the industry.

In addition, Anton teaches classes and presents at many security conferencesacross the world; he recently addressed audiences in United States, UK,Singapore, Spain, Russia and other countries. He works on emerging securitystandards and serves on the advisory boards of several security start-ups.

Currently, Anton is developing his security consulting practice, focusing onlogging and PCI DSS compliance for security vendors and Fortune 500organizations. Dr. Anton Chuvakin was formerly a Director of PCI ComplianceSolutions at Qualys. Previously, Anton worked at LogLogic as a Chief LoggingEvangelist, tasked with educating the world about the importance of logging forsecurity, compliance and operations. Before LogLogic, Anton was employed by asecurity vendor in a strategic product management role. Anton earned his Ph.D.degree from Stony Brook University.
http://www.chuvakin.org/http://www.info-secure.org/http://www.securitywarrior.org/http://www.chuvakin.org/http://www.info-secure.org/http://www.securitywarrior.org/

Documents

Trends in Database Log Management by Anton Chuvakin