FaultDetectionusingAdvancedAnalyticsatCERN'sLargeHadronCollider
AntonioRomeroMarínManuelMartinMarquez
USA- 27/01/2016 BIWA16 1
What’sCERN
USA- 27/01/2016 BIWA16 2
What’sCERN• EuropeanLaboratoryforParticlePhysics• FundamentalResearch
• WorldwideInternationalCollaboration• USisanimportantcontributortoCERNandtheexperiments
• Education&Training• PushFrontiersofTechnology
USA- 27/01/2016 BIWA16 3
CERNopenlab• Public-privatepartnershipbetweenCERNandleadingICTcompaniesandresearchinstitutes• Acceleratecutting-edgesolutionsfortheworldwideLHCcommunityandwiderscientificresearch.• Designedtocreateanddisseminateknowledge• Publicationofreportsandarticles• Workshopsorseminars• CERNopenlab StudentProgramme
USA- 27/01/2016 BIWA16 4
CERNAcceleratorComplex
USA- 27/01/2016 BIWA16 5
DataAnalyticsChallenges• Somefaultscannotbeavoided
• Decreasetheavailabilityforrunning physics
• Preventivemaintenanceisnotenough• Doesnottakeintoaccounttheconditionoftheequipment
• LHCupgradeswillfurtherincreaseluminosity• Computing resourcesneedswillbehigher• Datageneratedwillincreasedrastically
USA- 27/01/2016 BIWA16 6
DataAnalyticsObjective
USA- 27/01/2016 BIWA16 7
Areasofinvestigation• Predictivemaintenanceandsystemoptimization• Dataextraction,transformationandloading(ETL)• DataVisualizationandDiscovery
USA- 27/01/2016 BIWA16 8
Rfordataanalytics• Languageforstatisticalcomputingandgraphics• Powerfulintegratedsuiteofsoftwarefacilities
• Datamanipulation• Calculation• Graphicaldisplay,plotting
• FreeandOpenSource• Extensible
• Over7800CRANpackagesextendingbasefunctionality• Interactiveandprogrammaticenvironment• Greatusercommunity
• Activedevelopment,frequentlyupdated
USA- 27/01/2016 BIWA16 9
ParallelProcessingApproachesinR• ParallelprocessingwithR• Foreach• Snow• Rmpi• BatchExperiments package(BatchJobs)
• SparkR• PackagedinSparkfrom1.4.0
• OracleREnterprise(ORE)
USA- 27/01/2016 BIWA16 10
WhyORE?- OREbenefits• Adatabase-centricenvironmentforanalyticalprocessesinR• AllowstousethedatabaseservertorunRscripts(scalability&performance)• EliminatememoryconstraintofclientRengine• Rworkingondatadirectlyinthedatabase,nodatatransfer• Integration withtheSQLlanguage• Allowsin-databasedataanalysis• Providedataparallelism
• TransparencyLayer• TransparentlyanalyzeandusedatainOracleDatabasethroughR• NoneedtoknowSQL
USA- 27/01/2016 BIWA16 11
CERNDatabaseactivity
USA- 27/01/2016 BIWA16 12
October2012 December2015Max size ACCLOG 136TB ACCLOG352TB
Maxredo ACCMEAS27TB/month QPSR115TB/month
OREdeployment
USA- 27/01/2016 BIWA16 13
4xXeonE7-8895v2(15coreseach)2TBRAM
4.8TBFlash+6x1.2TB10KHDD
R ENTERPRISE
RAC– RealApplicationCluster
+
-EquipmentExperts-Operators
OracleDatabase
Engine
OracleDatabase
Engine
CERNControlSystems• IoT andControlSystem
• Cryogenics• Vacuum• MachineProtection• PowerConverters• QPS
• AcceleratorLoggingService• ~275GB/day• Storingmorethan50TB/year• Dataacquisition
• CERNacceleratorcomplex• Relatedsubsystems• Experiments
• Around1millionsignals
USA- 27/01/2016 BIWA16 14
LargestCryogenicsInstallation• 50kI/O,11kactuators, ~5k control loops• Control:
• ~100PLCs(Siemens,Schneider)• ~40FECs(industrial PCs)
• Supervision:26 SCADAservers
USA- 27/01/2016 BIWA16 15
Instrument/Actuators TotalTemperature[1.6– 300 K]Pressure[0– 20bar]LevelFlowControl valvesOn/Off valvesManualvalvesVirtual flowmetersControllers (PID)
10361230092326333692183519163254833
UseCase:AnomalyDetectiononBeamScreen
USA- 27/01/2016 BIWA16 16
the resistive wall power losses dissipated by the beam and it intercepts syn-chrotron radiation and molecules in order to increase the quality of the beamvacuum [8]. Each beam screen is cooled by conduction through two smallcooling pipes, supplied by supercritical helium coming from the CryogenicDistribution Line of the LHC (QRL) at 3 bar and 4.6 K to avoid two-phaseflow regime. One beam screen is installed on each beam pipe in a half-cell ofthe LHC corresponding to three superconducting dipoles and one supercon-ducting quadrupole for a total length of 53 m.
Beam Screen
Cooling tube
Beam Pipe
Figure 2: LHC beam screen cooled by its two small cooling tubes
The initial cryogenic library provides only 0-D pipe models or very simpli-fied 1-D models where the ideal gas law is used. Hence, to model this specificsupercritical helium flow cooling the LHC beam screen, a new model had tobe developed reusing the facilities of the Ecosimpro cryogenic library. Thiskind of approach was already used previously to simulate a heat wave prop-agation along the LHC cryogenic distribution line after a quench [9] but inthis last paper, the model was based on simplifications for very low pressurehelium flows which are not valid for supercritical helium flows.
The following section describes the model of a supercritical helium flowthough small cooling pipes which will be then embedded in a larger systemwith other components from the cryogenic library in order to simulate thecomplete beam screen cooling circuits.
4.1. Modeling of a supercritical helium flow
A one dimensional compressible flow according to the x axis has to beconsidered as the isothermal compressibility is very high for supercritical
5
Source:EN-ICE(BenjaminBradu,EnriqueBlanco)
• PIDoutput(timeseries)segmentation• Characterization+Featureextraction• Featuresbasedclassification
UseCase:AnomalyDetectiononBeamScreen
USA- 27/01/2016 BIWA16 17
STABLEBEAM BEAMDUMPSETUPINJECTBEAM
UseCase:FaultyCryogenicsValveDetection• Whatistheobjective?
• Predictfaultyvalvesbeforetheyactuallyfail
• How?• Valvereceiveanapertureordervalue(apertureorder)• Effectiveaperturerealizedbythevalve(aperturemeasured)• Analyzing thedifferencebetweenboth(S =apertureorder- aperturemeasured)
13/08/2015 openlabSummerStudentsLecture13:DataMining,DataAnalyticsandBigData
18
UseCase:FaultyCryogenicsValveDetection• Signalsused:
• S =apertureorder- aperturemeasured• FeaturesextractionsbasedonS
• Variance• Percentile99.9• Ropedistance– R(S)• NoiseBand – B(S)
• AutomaticFaultyValvesDetectionSystem• SVM- SupportVectorMachine
• Thelearningset44valves• apertureorder(%)• aperturemeasured (%)
USA- 27/01/2016 BIWA16 19
Cryo Valves– ExtractFeaturesinORE
• Valvesdatasetwith6features• 10.5Mrows• 42Mrows• 168Mrows
USA- 27/01/2016 BIWA16 20
Ourexperience• Movethedataisveryexpensive• ItisfastertoprocessitintheDBwithOREusingtheappropriatedegreeofparallelism
• DBnodesalreadypreparedfortheworkload• Memoryisaproblemforsmallclients• Simplifiestheinfrastructure
• Write/adaptRcodeisstraightforward• ThankstotransparencylayerandembeddedRexecution
USA- 27/01/2016 BIWA16 21
Nextsteps• TestwithDBin-memoryfeatures• Benefitfromthehybridecosystem• GetthebestfromRDBMS+Hadoop
• Don’tforgetaboutcurrentapplicationlayerworkingwiththeDB• Deployinstreaming
• PredictiveModelMarkupLanguage(PMML)• Compatibleacrossmultiplelanguages(R,Python,Spark,Java,etc.)
USA- 27/01/2016 BIWA16 22
Conclusions• RecosystemisverygoodforDataAnalytics• OREoffersagoodsolutiontogetmorefromyourDBdeployment• Nopainadaptingcode• FollowsRstandards
• Avoidmovingdatawhenpossible• Hybridsystemslookquitepromisingtocovermultiplescenarios
USA- 27/01/2016 BIWA16 23