© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
An introduction to Logger and ESM Web Services APIs Shivdev Kalambi Principal developer and manager correlation team
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
Agenda
Logger Web Services APIs • Look at the Login Service, Report Service and Search Service • Look at some use cases ESM Web Services APIs • Look at the Login Service, Query Viewer Service, and Report Service • REST & SOAP Examples Q&A
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Logger Web Services APIs
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
Web Service APIs What are web services?
Wikipedia definition • Web services are typically
application programming interfaces (API) or web APIs that can be accessed over a network, such as the Internet, and executed on a remote system hosting the requested services.
External client
Internal client
Server that exposes Web Services (e.g. Logger)
Web
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
So what are we doing here?
The idea is simple Fetch data from Logger and apply it to your use case
Do some analytics? Show the source Address on a map
Write a simple client? For the command line folks
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
What must I know?
Got data?
Prerequisites • Logger Search Syntax • Logger Reporting (SQL) • Knowhow of Web Services in general
• Write code for simple SOAP clients
100110011010101001101001101010 ?
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
Logger Web Service APIs APIs available in Logger v5.3
APIs Available • Services
• LoginService • ReportService • SearchService
• WSDL Location • https://<LoggerHost><:Port>/soap/services/<ServiceName>/<ServiceName>.wsdl • https://192.168.35.9/soap/services/ReportService/ReportService.wsdl
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Methods Arguments Return Description
login username, password,timeout cookie Login to a logger and establishes a cookie
logout cookie Ends a session identified by the cookie
getVersion String Returns the version of the web service
extendSession cookie Extends the session identified by the specified cookie
LoginService
Example String cookie = loginService.login(“admin”, “password”, 120);
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
Methods Arguments Return Description
getReportGroups cookie Group[] Get the Report Groups (Categories)
getReportsInGroup groupID, cookie Report[] Get the Reports in a Group
runReport
reportId, startTime, endTime, scanLimit, rowLimit, devices, deviceGroups, storageGroups, reportParams, reportFormat, cookie
String Base64 encoded Report Result (for eg. in CSV format).
getDevices cookie String[] Get a list of devices
ReportService
Example String report = reportService.runReport(report_ID, System.currentTimeMillis() – 2 * 60 * 60 * 1000, System.currentTimeMillis(), 10000, 100, null, null, null, null, “csv”, cookie);
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
Methods Arguments Return Description
startSearch queryString, startTime, endTime, cookie
Starts the search
endSearch cookie Ends a search session identified by the cookie
getNextTuples count, timeout, cookie Tuple[] Get an array of Tuples that matched the Search
hasMoreTuples cookie boolean Find out whether the search has more tuples
getHeader cookie String[] The Format of the Data in the Tuples
SearchService
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11
Sample Java Code Simple Search using the Iterator Pattern // Login String cookie = loginService.login("admin", "password", 60); // Start the Search searchService.startSearch("ERROR“, System.currentTimeMillis() - 10 * 60 * 1000, System.currentTimeMillis(), cookie); // Loop over and find resulting rows while (searchService.hasMoreTuples(cookie)) { Tuple [] tuples = searchService.getNextTuples(500, 1000, cookie); if (tuples != null) { for (Tuple tuple : tuples) { String [] arr = tuple.getData(); // Custom Processing of the data } } } // End the Search searchService.endSearch(cookie); loginService.logout(cookie);
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
Use case 1
Command Line Utility • Run a search (or report) from the Logger
Web UI – Logger | cef name | top 5 name
• Simulate the same search using Web Services from a command line utility
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
Use case 1 Command line utility
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14
Use case 1
ESM/Logger Integration • When a rule fires in ESM • Execute the utility to query logger for
events • Send an email with events
INtegrate the command line utility with ESM
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15
Use case 2
Run a search CEF | cef sourceAddress | top 5
sourceAddress Run a report using this SQL
SELECT arc_sourceAddress, COUNT(arc_eventId) FROM events GROUP BY arc_sourceAddress ORDER BY COUNT (arc_eventID) DESC LIMIT 5
Feed the results to a mapping tool
Plot geo locations for top 5 source IP addresses on a map
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
Best practices
Some points to keep in mind • Login Service
– Always logout – Sessions get purged but will take a while to clean up • Search Service
– Always endSearch – The Searcher will be instantly cleaned up – From v5.3, you will see performance improvements in the getNextTuples(count, timeout, cookie) – by
changing the count • Report Service
– Data returned from runReport(…) call is Base64 encoded – so you would need to decode it
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17
Writing clients
Please note • Users are expected to write their own SOAP Clients • We provide a reference JAVA implementation and have provided some sample JAVA code to serve as an
example • Documentation is available on the customer support site
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ESM Web Services APIs
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
ESM Web Services APIs
Introduction • Login Service • Query Viewer Service
– Fetching data using REST – Fetching data using SOAP
• Java Code Snippets • Report Service • Required Libraries & Interesting Observations
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
Use cases
Fetch query viewer data • Sample Query Viewer • Fetch data
– Using REST – Using SOAP
Top 10 most common events
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
Use case 1 REST call to login service
https://localhost:8443/www/core-service/rest/LoginService/login?login=admin&password=password
https://localhost:8443/www/manager-service/rest/QueryViewerService/getMatrixData?authToken=__&id=__
Test the REST • From the browser • Invoke LoginService
– Copy the authToken • Invoke QueryViewerService
– Pass the QueryViewer ID
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 22
Use case 1 REST call to QueryViewer service
https://localhost:8443/www/manager-service/rest/QueryViewerService/getMatrixData?authToken=__&id=__
QueryViewerService using REST
QueryViewerService using SOAP
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 23
Use case 2
SOAP Call to login service // Set the Base URL System.setProperty("com.arcsight.coma.client.ws.baseURL", "https://” + host + "/www/"); // localhost:8443 // Get the LoginService and login LoginServiceClientFactory factory = new LoginServiceClientFactory(); LoginService service = factory.createClient(); String authToken = service.login(null, “admin”, “password”); // This authToken is required in subsequent calls
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24
Use case 2
SOAP call to query viewer service // Get the QueryViewerService and get the data QueryViewerServiceClientFactory factory = new QueryViewerServiceClientFactory (); QueryViewerService service = factory.createClient(); MatrixData md = service.getMatrixData(authToken, "cwswTlzgBABCKipuKVcyzlg=="); // Get the Column Names List<String> headers = md.getColumnHeaders(); int col = 0; for (String header : headers) { System.out.printf((col++%2 == 0 ? "%60s" : "%20s\n"), header); } // Get the Data List<ListWrapper> rows = md.getRows(); for (ListWrapper row : rows) { List value = row.getValue(); for (Object obj : value) { System.out.printf((col++%2 == 0 ? "%60s" : "%20s\n"), obj); } }
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25
Use case 2
I found that I needed the following static block to trust the hostname // Static Block static { HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { // Make sure that hostname is valid return true; } }); }
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 26
Use Case 3 REST Call to Report Service
Start the Report Generation
https://localhost:8443/www/manager-service/rest/ArchiveReportService/initDefaultArchiveReportDownloadById?authToken=_&reportId=_&reportType=Manual
Get the Download ID and download the report
https://localhost:8443/www/manager-service/fileservlet?file.command=download&file.id=DOWNID
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27
Required libraries and interesting observations
Tips from an end user • Even though it’s SOAP under the covers AXIS2 libraries didn’t work
– manager-ws-client-1.2.0.release.107.jar – core-ws-client-1.5.0.release.51.jar – coma-infrastructure-1.4.0.release.240.jar
• For now, the SOAP APIs can only be written in Java and using these libraries • I was not able to get it to work with AXIS2 libraries in the CLASSPATH
– Marshalling Errors • Don’t forget to implement a HostnameVerifier (by default it will NOT be a verified hostname) • Documentation is available on the Support Site
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 28
Recap Key Takeaways • Logger Web Services APIs
– Supports only SOAP – Login Service – Local Authentication – Search Service – Start Search and Iterator Pattern to go over resulting Tuples – Report Service – Run a Report and get back a Base64 encoded result
• ESM Web Services APIs – REST – for simple use cases – SOAP – For now, Java clients using the provided libraries – GWT-RPC is also used by our UI team
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29
After the event Meet these folks
Find out more
Attend these sessions
• Session ID: 1257 Title: Gain quick and relevant solutions made possible by the HP ArcSight Logger API.
• Logger PMs: Roopak Patel & Suresh Venkatraman
• ESM PMs: Monica Jain & Saran Selvaraj
• ESM, CORRE, Logger Dev: Anurag Singla, David Wiser, & Vivek Vallachira
• Contact your sales rep
• Visit the Social Pages at: http://www.facebook.com/HPSecure Twitter: @HPSecure
• Download the slides at: http://protect724.arcsight.com/
Your feedback is important to us. Please take a few minutes to complete the session survey.
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Security for the new reality