Log Blindspots: A review of cases where System Logs are insufficient

8/3/2019 Log Blindspots: A review of cases where System Logs are insufficient

1/7

Log Blindspots: A review of cases where system logs are insufficient Copyright 2011 ObserveIT Ltd. | www.observeit-sys.com

Executive Summary

If you spend a few minutes browsing the websites of Log Management and SIEM tool vendors, you might come away with

the conclusion that all your system audit and compliance problems are solved. Unfortunately, this rosy picture seems to

ignore the ever-present problem of blindspots in audit reports: If your apps dont log it, your audit report wont show it.

Audit report tools may do a good job at interpreting and presenting log info, but we can no longer overlook two key facts:

1. Hundreds of critical security event types are not logged at all2. Those events that are logged typically do not show what was done. Instead, the logs only show obscure technical

details of the resulting system changes.

In this whitepaper, Ill highlight examples of where these blindspots occur, by showing a number of very common and basic

system activities that one might think should generate auditable log entries, but in actuality they do not. These non-audited

actions include:

On a Windows server:

Adding and Deleting IP Address Setting a Service to run as administrator Change Web server config file Change port usage for an active service

On a Linux or Unix server:

chmod * or chown * Assign user to an admin rights group Add/Delete IP address in hosts file Give sudo rights to non-admin user

One possible way to eliminate blindspots is to implement custom log utilities or WMI-based tools. But to do this, the

burden remains on you to know what you are looking for. For the examples listed above, adding an IP Address change

monitor wont help with web config file changes, and vice versa. And more importantly, adding 4 different monitors for

each of those issues wont help capture the hundreds of actions that youll never be able to predict. As the well-worn yet

valuable expression states, Expect the unexpected.

User Activity Monitoring follows through on this philosophy. In the context of IT audit logs, perhaps the best way to expect

the unexpected is to drop the paradigm oflisting the actions that should be logged, and instead simply monitor all useractions.

Log Blindspots:

A review of cases where System Logs are insufficient

An ObserveIT Whitepaper | Brad Young


2/7


Scenario 1: Changing a Windows systems IP address

What the User Did:A privileged user logged onto a Win 2003 Server (via RDP in this case, but for the sake of discussion

it could be any local or remote connection protocol). After logging in, this admin user opened the Advanced TCP/IP

Settings (via Start > Settings > Network Connections > Local Area Connection > Properties > Internet Protocol >

Properties > Advanced). Once there, he removed an IP address

(10.1.200.178), and then added a different IP address

(10.1.200.179).

Advanced TCP/IP Settings

Security and Audit Implications of this Action: Adding and IP

address might allow bypassing of firewall settings and may also interfere with proper execution of critical services.

What shows up in system event logs: With full auditing enabled, a total over 11,000 log events were triggered

during the 30 seconds it took the user to delete and add an IP address. Almost all the log entries were of Object

Access category. Searching within the logs for the terms TCP, IP or 179 (last 3 digits of the IP address added)

brought back numerous search, but all were false hits. (ex: IP appears in the filename wshtcpip.dll within one

log entry, another log entry having Operation ID 74312179.) No log entry refers explicitly to the action taken.

It may be possible for a highly-trained system

security expert to piece together the log entries and

determine what actions took place. But it would

involve a time-intensive forensic analysis by a sparse

and expensive resource. Do you have highly-trained

security experts that are bored with nothing better

to do than piece together log entries?

Event Viewer: 11,000 log entries in 30 seconds,

dozens of false hits, no clear picture

What User Activity Monitoring shows you: A

user-oriented textual audit log shows that brad logged on as administrator, and the list of actions tells the story

of what he did: Network Connections > Properties > TCP/IP Properties > TCP/IP Address. This already is much more

than information than what is accessible in the system logs. Adding video replay of the session then shows even

more details.

ObserveIT Audit Log: A Table of Contents of the user session ObserveIT video replay of user changing the IP Address


3/7


Scenario 2: Adding sudo rights for non-authorized users in Linux

What the User Did: A non-privileged user tried running the snmpd service, but did not have permissions. He then tried

running it using sudo, but did not have sudo rights either. So instead, he asked a root user to log on and grant him

sudo rights, using visudo.

Add sudo rights for

a non-authorized user

Security and Audit Implications of this Action:

Giving sudo rights allows a user to run many sensitive commands or services.

What shows up in system event logs: Using auditctl

and ausearch, we can see that the visudo command

was run. However, this logging is almost entirely of a

technical nature. We can see the working directory

from which it was launched, its process id, and the fact

that it finished with a success return value. No

indication shows what rights were granted, or what

the user did once he got those rights.

Technical details only in ausearch

What User Activity Monitoring shows you: With ObserveIT in place, we are able to see exactly what took place.

The textual metadata log shows the commands that were run.

ObserveIT Audit Log, including underlying system calls ObserveIT video replay of CLI activities


4/7


Scenario 3: Setting a Windows service to run as administrator

What the User Did:An admin user changed the

properties of a Service (via Start > Settings >

Control Panel > Administrative Tools > Services).

Once there, he selected the Cryptographic

Servicesservice and marked it to run as

administrator.

Run a service as Administrator

Security and Audit Implications of this Action: Enabling a service that is not secure to run as administrator can

enable remote hacking and can cause the service improperly affect sensitive system configuration and data.

What shows up in system event logs: Over 24,000 log events were triggered during the 40 seconds it took the user

to change the Run As credentials. Despite the sheer volume, no log entries included the word Cryptographic (the

name of the service)!

Again, a full-throttle investigation by system experts

might unearth the true actions, but this task makes

biblical archaeology look easy.

Event Viewer: 24,000 log entries in 40 seconds,

no indication of the Service that was modified

What User Activity Monitoring shows you: As in the

previous example, ObserveIT shows a clear chronological

timeline of what the user actually did: open Control

Panel and then go to Cryptographic Services Properties.

And again, video replay shows even more.

ObserveIT Audit Log Video replay of Service Run As credentials


5/7


Scenario 4: Change web.config (IIS webserver configuration file)

What the User Did:Via Windows Explorer

and Notepad, the user made a simple

change to an XML attribute in the file

web.config, changing a 0 (false)

value to 1 (true).

Editing web.config with Notepad

Security and Audit Implications of this

Action: Changes to this file will affect

how the web server runs, in numerous

different ways. This can expose security

risks, and can also affect proper operations.

What shows up in system event logs: 6,000 log entries cover the 20 seconds it took to make the change. One log

entry indicates that Notepad was launched. Another log entry indicates that web.config was added to the

Recent Files list in Windows. A third log entry seems

to show (not convincingly) that it was Notepad that

edited the filw web.config. But even with this info, we

cannot tell what was actually changed within the file!

(Was it a harmless addition of an application

extension? Or did the user modify an important entry

within the file?)

To know what was changed, we would now have to

access a file server backup, and perform a file compare

on the old and new versions. Doable, but thats a

heavy burden to answer a pretty straightforward

question: What did the user change???

Event Viewer: But what was changed?

What User Activity Monitoring shows you: ObserveITs log shows what the user did, in a concise and descriptive

manner. And again, video replay shows what took place within the file.


6/7


Scenario 5: Changing the port used by IIS

What the User Did:An admin user changed IIS to

listen to port 8080 instead of the default 80. This

was done via Start > Settings > Control Panel >

Administrative Tools > IIS Manager, and once

there editing the Properties for Default web

site.

Set IIS to listen to port 8080

Security and Audit Implications of this Action:

Modifying the port of a service accessible from

outside the DMZ can open a huge hole in the firewall security.

What shows up in system event logs: Among the 5,500 log entries, there is one entry that adds IIS Manager to the

Recent Items list in Windows. This is

timestamped when the app was closed,

which might mislead the investigator, and

alsow wouldnt even occur if the user left

the window open. Earlier, there is an

obscure log entry indicating a DLL that was

loaded to memory. This is the true

indication that IIS Manager was launched,

but it is very difficult to find this in a

reasonable level of effort!

Event Viewer: Obscure log entry of DLL.

It turns out that this is the culprit!

What User Activity Monitoring shows you: Once again, ObserveIT gives us the whole picture.


7/7


Platform Considerations

The Windows experiments were performed on a Windows 2003 server. Windows 2008R2 has added additional audit policy

granularity. However, these updates do not mean that additional knowledge can be gleaned from the logs; Only that the

logs can be filtered a bit better. The bottom line remains that many high-risk, security-impacting actions, including those

highlighted in this paper, are not logged.

The Linux experiments were performed on RedHat RHEL. Similar audit logging is found in other Linux flavors, as well as in

Solaris Unix, with similar focus on technical aspects of each command (pid, cwd, success).

Conclusion

Security audits that rely on existing system logs have large holes in them due to the fact that system logs simply do not

capture the relevant information necessary.

For issues that are known a priori, the blindspot can be eliminated with a custom utility targeted at that specific issue. But

this only solves this one specific issue.

The easiest way to eliminate these blindspots in their entirety is by adding User Activity Monitoring such as ObserveIT,which augments the existing system and database logs by showing precisely what the user did (as opposed to the technical

results of what he did.)

About ObserveIT

ObserveIT User Activity Monitoring software meets the complex compliance and security challenges related to user activity

auditing, one of the key issues that IT, Security and Compliance officers are facing today.

ObserveIT acts like a security camera on your servers, generating audit logs and video recording of every action the user

performs. ObserveIT captures all activity, even for applications that do not produce their own internal logs. Every actionperformed by remote vendors, developers, sysadmins and business users is tied to a video recording, providing bulletproof

forensic evidence.

ObserveIT is the ideal solution for 3rd Party Vendor Monitoring, and PCI/HIPAA/SOX/ISO Compliance Accountability.

Founded in 2006, ObserveIT has a worldwide customer base of Global 2000 companies that spans many industry segments

including finance, healthcare, manufacturing, telecom, government and IT services.

For more information, please contact ObserveIT at:www.observeit-sys.com

[email protected]

US Phone: 1-800-687-0137

Intl Phone: +972-3-648-0614

Documents

Log Blindspots: A review of cases where System Logs are insufficient