View
2
Download
0
Category
Preview:
Citation preview
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Leveraging super-indexed searches Jason Stoops, Software Designer #HPProtect
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Super-index basics
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
What do super-indexes do?
Track columns to rule out ranges of CORR-engine event data This can improve performance in: • Search queries
– Local and Peer search • Reports • Event export
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
Super-index on event ingestion
• Super-indexing occurs after Indexer • Two tiers:
– Master super-index tracks all event data
– Data-range super-index tracks 1-2 hours
Master super-index
Data-range super-index
CORR-engine storage
Indexer
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
Super-index on search
• Master super-index is checked – If no results, search is done.
• Data-range indexes covering search time range are checked – Data-ranges with no results can be ignored
• Finally, ROS search data-ranges with hits in CORR-engine storage
Master super-index
CORR-engine storage
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
What types of queries benefit most?
Queries with no results Master super-index can return in 1-2 seconds regardless of time range searched
Queries with very few results Ranges of event data are ruled out to significantly improve search speed
Needle-in-haystack style searches!
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Super-index use cases
Queries expecting few or no results • Forensic searches for malicious hosts • Find a particular class of event • Determine if a device was down for some period of
time
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Using super-indexes
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
Super-indexed fields
Host fields • destinationHostName • destinationPort • destinationAddress • sourceHostName • sourcePort • sourceAddress • deviceAddress • deviceHostName
User fields • destinationUserId • destinationUserName • sourceUserId • sourceUserName
Event source fields • deviceEventClassId • deviceProduct • deviceVendor
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11
Super-index operators
Queries with equal to (=) are accelerated No performance impact on: • Greater than (>) • Less than (<) • Not equal to (!=) • Other operators
– (CONTAINS, STARTSWITH, BETWEEN, etc.)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Example queries
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
Super-indexed single field-based search
Using equal to (=) operator sourceAddress = 192.0.2.0 • Quick response if address does not exist • Reduced search time if address exists. • Only time-ranges of CORR-engine data containing
this source address are searched.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14
Super-indexed fields with AND / OR
AND sourceAddress = 192.0.2.0 AND destinationAddress = 192.0.2.2
• Quick response if either does not exist • Reduced search time if addresses exist
OR sourceAddress = 192.0.2.0 OR sourceAddress = 192.0.2.2
• Quick response if both do not exist • Reduced search time if addresses exist
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15
NON-Super-indexed full text search
Full text search "arcsight" • Super-indexes not available on full-text search • No negative impact on existing search performance
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
Super-indexed fields AND / OR (cont’d)
AND sourceAddress = 192.0.2.0 AND "arcsight" requestMethod = GET AND sourceAddress = 192.0.2.0
• Quick response if address does not exist • Reduced search time if address exists
OR sourceAddress = 192.0.2.0 OR "arcsight" requestMethod = GET OR sourceAddress = 192.0.2.0
• No super-index speed improvement • Knowing if address exists doesn’t assist with OR
condition • Non-super-indexed field can still make “OR”
condition true even if address does not exist
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Ensuring super-indexes are utilized
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18
Ensuring super-indexes are used
Data before Logger 5.5 upgrade is not super-indexed • Ensure your searches are against data received
after Logger 5.5 upgrade
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
Ensure super-indexes are used
Case-sensitive searches only • Ensure search case-sensitivity is enabled
ROS searches only Super-index benefits cannot be applied to WOS searches • Keep indexing is up-to-date • Only combine super-indexed fields with indexed
fields
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
For more information
Attend these sessions
• TT3132, Army of loggers: Searching for that needle at billions of EPS
• TT3154, Renovation of the HP Arcsight Logger user interface
Visit these demos
• HP ArcSight Logger demo station
After the event
• Contact your sales rep
• Visit the website at: www.hp.com/go/logger
Your feedback is important to us. Please take a few minutes to complete the session survey.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
Please fill out a survey. Hand it to the door monitor on your way out.
Thank you for providing your feedback, which helps us enhance content for future events.
Session TT3099 Speaker Jason Stoops
Please give me your feedback
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Recommended