Database Competence Centre Lorena Lobato Pardavila Andrei Dumitru Ignacio Coterillo Coz Antonio Romero Marin Manuel Martin Marquez Luis Rodriguez Fernandez

Database Competence Centre Replication Technologies Virtualization Monitoring Oracle Database 12c Data Analytics Java Enterprise Edition In-Database Physics Analysis 2 CERN openlab Major Review October 2014 15/10/2014

Summer Students IT-DB Sneha Senha: SQLPlotter: Web-Interface for Physics Analysis Database Anti Asko: Veridata for Data Consistency in WLCG Sofia Danko: PaaS solutions evaluation Noe Fernandez Alvarez: Web application component mapping Benjamin Lipp: OpenStack Trove: Evaluation and Interfacing with the CERN Database on Demand Service Jakub Zitny: Oracle TimesTen in-memory database integration Binathi Kulynych: Integration of IT-DB Monitoring tools into GNI: the IT General Notification Infrastucture Check reports: http://openlab.web.cern.ch/education/summer-students/reports 3 CERN openlab Major Review October 2014 15/10/2014

Replication Technologies Lorena Lobato Pardavila

Replication Technologies Evolution 2012 CERN part of Oracle Database beta program Q1-Q2 2013 As part of beta program, several tests were done to compare Streams and Oracle GoldenGate advantages Q3-Q4 2013 Oracle GoldenGate 12c released (12.1.2) OGG better than Streams: Migration to Streams to Oracle GoldenGate decision taken Testing monitoring(OGG Monitor, OGG Director OGG EM plugin) and Veridata to measure data consistency 2014 Streams to Oracle GoldenGate migration planning Testing migration procedure from Streams to Oracle GoldenGate Oracle GoldenGate Production Databases running at CERN 5 15/10/2014 CERN openlab Major Review October 2014

Recent Activities Replication Technology Evolution for ATLAS Data Workshop Replication technologies and requirements review. Replications roadmap established for ATLAS. GoldenGate administration, monitoring and troubleshooting. Data consistency verification with Veridata Testing Veridata 12c (12.1.3) at WLCG Keep evaluating GoldenGate 12c and the new features Finally.. Streams to Golden Gate migration completed! Done July September 2014 Improved scalability performance better than Streams ATLAS ONLINE OFFLINE (condDB) ATLAS OFFLINE TIER1s RAL IN2P3 TRIUMF LHCb ONLINE-OFFLINE 6 CERN openlab Major Review October 2014 15/10/2014

Centralised configuration at CERN 7 CERN openlab Major Review October 2014 15/10/2014 Source databases Central GG servers Source databases A A B C A C B NAS storage with configuration and trail files Replica databases -GoldenGate processes -Monitoring agents -GoldenGate processes -Monitoring agents

Measure data consistency: Veridata After migration, verify data consistency measured by Oracle GG Veridata Veridata is a potential and solid data comparison solution More than 120 schemas inspected between CERN and Tier1s +1.87 TB of data checked +34,888 of tables were processed 8 CERN openlab Major Review October 2014 15/10/2014

Next steps Evaluation of zero-downtime database upgrades/migrations using Oracle GoldenGate Evaluating the use of OGG as an ETL solution for data warehousing and DaaS. Oracle GoldenGate monitoring Evaluating of new release of OGG 12c (12.1.2.1) Oracle GoldenGate at WLCG Feedback to Oracle GoldenGate development from running our production setups with the new technology. Support and maintenance of OGG 9 CERN openlab Major Review October 2014 15/10/2014

Monitoring Oracle Database 12c Andrei Dumitru

EM12c R4 EM12c R3 EM12c R2 EM12c R1 EM12c beta Timeline EM12c R4 - current production service Increasing target base (over 5400) EM CAB @ Oracle HQ (May 2014) Oracle Enterprise Manager 12c 11 CERN openlab Major Review October 2014 15/10/2014

Oracle Enterprise Manager 12c Outlook Automatic Workload Repository (AWR) Warehouse Business intelligence (BI) Publisher for reports Integrate with CERN Agile Monitoring Migrate to RHEL6 12 CERN openlab Major Review October 2014 15/10/2014

Oracle Database 12c 2012 July CERN part of Oracle Database beta program On-site testing at Oracle HQ, USA Testing the new set of features at CERN 2013 June Oracle Database 12c released (12.1.0.1) 2013 July Oracle Database 12c Development Service available at CERN 2014 April Oracle Database 12c Production Databases running at CERN 1315/10/2014 CERN openlab Major Review October 2014

Oracle Database 12c Production service at CERN (12.1.0.1) Continuing work on version 12.1.0.2 Evaluating In-Memory Column Store 1415/10/2014 CERN openlab Major Review October 2014

Virtualization Ignacio Coterillo Coz

Oracle VM as OpenStack Hypervisor 2014 February Custom Hypervisor: manually compiled XEN and libvirt 2014 March Oracle VM 3.3.1 beta release solved the situation of having to work with custom hypervisors 2014 May Oracle OpenStack tech preview release solved having to recompile libvirt to enable XEN support201 2014 Updates on both Oracle VM and OpenStack eventually solved the problems with the networking components 1615/10/2014 CERN openlab Major Review October 2014

Oracle VM as OpenStack Hypervisor Current procedure 1.Install Oracle VM hypervisor from iso 2.Install libvirt from Oracle OpenStack Repository 3.Add hypervisor host as a nova compute node on your OpenStack RDO installation 4.Configure Nova to use the XEN driver 15/10/201417 CERN openlab Major Review October 2014

18 CERN openlab Major Review October 2014 15/10/2014

CERN AI Monitoring integration As mentioned before, one of the biggest issues to the Oracle VM 3 was the impossibility to install 3rd party tools on the hypervisors This made impossible to use CERN monitoring tools with Oracle VM3 hypervisors With the release of Oracle VM 3.3.1 beta, this changed: Possible to install both Lemon and Apache Flume agents for collecting metrics and log 19 CERN openlab Major Review October 2014 15/10/2014

CERN AI Monitoring integration 20 CERN openlab Major Review October 2014 15/10/2014

Next steps Trial production deployment Adding several Oracle VM hypervisors to the IT-DB OpenStack production pools OpenStack IceHouse Database workload testing and evaluation Automatize Oracle VM installation in our setup Installation Puppet integration 21 CERN openlab Major Review October 2014 15/10/2014

Data Analytics Antonio Romero Marin Manuel Martin Marquez

Data Analytics with Oracle R Enterprise R for Data Analytics Powerful and extensible Great user community, active development Standard de facto in data analytics Oracle R Enterprise benefits Allows in-database data analysis Execute R scripts in database server machine Providing scalability and performance Eliminating memory constraint 23 CERN openlab Major Review October 2014 15/10/2014

Data Discovery with Oracle Endeca Information Discovery Interactive and visual analytics by end users Flexible and easy to use but powerful Analyze information of any type and any source Get new insights from data Several PoC for different use cases are being carried out Improve Controls and Operations Accelerator Fault Tracking Diagnostics and Monitoring Accelerator Logbook HR surveys analysis IT Infrastructure Monitoring Server logs analysis Database latency 24 CERN openlab Major Review October 2014 15/10/2014

25 Provisioning Service Endeca Server Cluster Accelerator Operators HR Endeca Studio (user group dedicated ) System Administrators Controls and Operations System Configuration Logging and Monitoring Engineering IT Wide-Ranging Data Sources Logs HR Surveys Oracle Database Engine Engine Engine R ENTERPRISE Infrastructure CERN openlab Major Review October 2014 15/10/2014

Java Enterprise Edition Luis Rodrguez Fernndez

Java Enterprise Edition Goals: Increase developers productivity Make devops people life easier How? Transforming current CERN Java Hosting Service in a true Platform as a Service 27 CERN openlab Major Review October 2014 15/10/2014

Java Enterprise Edition Achievements: The new Java PaaS, Middleware On Demand, in production since September Users feedback: It is good! 28 CERN openlab Major Review October 2014 15/10/2014

Java Enterprise Edition Work in progress: Integration of Oracle Weblogic in the MW On Demand Integration of CERN Oracle APEX in the MW On Demand How to set up multiple JAVA_HOME in a Weblogic Domain Test the Oracle Weblogic maven support 29 CERN openlab Major Review October 2014 15/10/2014

In-Database Physics Analysis Maaike Limper

Physics Analysis: In-Memory Columnar Testing the DB12c In-Memory Columnar 4 different tests Electron counter, Electron Filter TTbar Cutflow Higgs+Z 31 IMC vs SSD Direct Path 7.54x faster! 15/10/2014 CERN openlab Major Review October 2014

Outreach 32 CERN openlab Major Review October 2014 Replication Technology Evolution for ATLAS data Workshop JINR/CERN School on Gried and Advanced Information Systems Enterprise Manager In-Person Customer Advisory Board EMEA Industry Council for Education for Education and Research http://cern.ch/db-blog/ http://cern.ch/db-blog/ SOUG Business Intelligence Intel IoT Data Analytics in Munich 15/10/2014

Outreach 33 CERN openlab Major Review October 2014 OpenWorld 2014, San Francisco(USA) Manage the Manager: Tips on How to Best Manage Oracle Enterprise Manager 12c Integrating Oracle VM into an Enterprise-Grade OpenStack Cloud: CERN Case Study CERN Private Java Cloud: Deep Dive into On-Demand PaaS internals Strengthen Research Abilities on Oracle Cloud Database Storage 101 Planning and Monitoring: Performance and Reliability Managing OpenStack in the Enterprise: Top Tips for DBAs Using the In-Memory Columnar Store to Perform Real-Time Analysis of CERN Data 15/10/2014

Acknowledgements 34 CERN openlab Major Review October 2014 Specially to: Monica Marinucci Greg Doherty Ronen Kofman (Oracle) Joseph Debuzna (Oracle) Patricia McElroy (Oracle) Andrew Bulloch (Oracle) Angeline Dhanarani (Oracle) Zbigniew Baranowski (CERN) Emil Pilecki (CERN) Eric Grancher (CERN) Luca Canali (CERN) Ruben Gaspar Aparicio (CERN) 15/10/2014

www.cern.ch/openlab 35 CERN openlab Major Review October 2014 15/10/2014

Documents

Database Competence Centre Lorena Lobato Pardavila Andrei Dumitru Ignacio Coterillo Coz Antonio Romero Marin Manuel Martin Marquez Luis Rodriguez Fernandez