DRIVEN - Red Hat€¦ · Hadoop + HP Vertica+ JBoss +HP IDOL + HP Enterprise Services Data source:...

‹#›

DRIVEN

Rob Mueller, Ben MurrayJune 24th 2015

TOINNOVATE

‹#›

NASCAR StoryArchitectureHP HavenIntegrationQ&A

NASCAR StoryArchitectureHP HavenIntegrationQ&A

Agenda

‹#›

NASCAR Environment

1948 – Bill France Sr. Creates NASCAR, the National Association for Stock Car Auto Racing

“Stock car racing has distinct possibilities for Sunday shows and we do not know how big it can be if it’s handled properly….”

- Bill France Sr.

The National Association for Stock Car Auto Racing

‹#›

…and Red Hat

Technology…

‹#›

‹#›

NASCAR Objectives

To be the first of its kind in the industry (i.e. Social and Traditional Media Monitoring), acting as a “state of the art” technical showpiece to garner attention from the technology sector and global sports properties

Establishing a more informed picture of both fan demographic and engagement with the sport of NASCAR

Near-time fan engagement though the monitoring and collation of data from social media, traditional media, broadcast, and in-house sources

Management of social profile and perception, delivering actionable insights and initiatives, to better engage fans, and meet strategic business objectives

‹#›

Event ManagementRaceTV spotPress conference

Incident ManagementRace – unforeseen circumstancesUnscheduled celebrity appearanceBreaking news

AnalyticsEventIncidentEntity

Near time

Retrospective

SentimentShare of voiceTrendsMessage volumesInfluenceGeographic profilingHot and breaking

Sentiment profilingMessage pull-throughComparative profiling

+

NASCAR Use Case

‹#›

Unstructured to structured

Near Time

‹#›

Source dataConstantly evolving and loose use of languageIndustry specific languageDisparate datasets and nomenclatureImplied contextAbbreviations, nicknames, mnemonics etc.SlangSarcasm and IronyProfanityEmoticonsEtc…

NormalizationSentiment Grammars

‹#›

Actionable analytics

Better ROI for Sponsors

Increased Fan Engagement

Increased Sales

Understanding

‹#›

NASCAR StoryArchitectureHP HAVEnIntegrationQ&A

Agenda

NASCAR StoryArchitectureHP HAVEnIntegrationQ&A

‹#›

EIP

‹#›

NASCAR logical

‹#›

System PathReference to Datasift needs to be removed

‹#›

Distributionremove datasift reference

‹#›

Broker View

remove the broker0 arrows pointing to the server2

‹#›

<networkConnector name="Prod1AMQ0toProd2AMQ0-2" networkTTL="1" duplex="false" uri="static:(tcp://10.221.9.32:62700)" conduitSubscriptions="false" >

<networkConnector name="Prod1AMQ0toProd2AMQ0-3" networkTTL="1" duplex="false" uri="static:(nio://10.221.9.32:62800)" conduitSubscriptions="false" >

<dynamicallyIncludedDestinations> <queue physicalName="preQ"/>…

AMQ0/ESB1 Server1 to AMQ0/ESB1 Server2 Load Balancing Network Connector

‹#›

CheckPoint service

‹#›

NASCAR StoryArchitectureHP HavenIntegrationQ&A

Agenda

NASCAR StoryArchitectureHP HavenIntegrationQ&A

‹#› (1) IDC Directions 2013: Why the Datacenter of the Future Will Leverage a Converged Infrastructure, March 2013, Matt Eastwood ; (2) & (3) IDC Predictions 2012: Competing for 2020, Document 231720, December 2011, Frank Gens; (4) http://en.wikipedia.org (a) Data Never Sleeps Domo.com (b) Deloitte Tech, media, and telecoms predictions 2014

Now By 2020

Volume, Velocity, Variety, Value

2,460,000 content shares (a)

277,000 tweets (a)

4,000,000 Google searches (a)

72 hours of video uploaded (a)

216,000 Instagram posts (a)

204,000,000 emails sent (a)

34,700,000 instant messages (b)

Every 60 seconds

40 Trillion(2)

10 Million(3)

… for 8 Billion(4)

30 Billion(1)

DATA

Mobile Apps

Devices

GB

HP Haven

Big Data Platform

‹#›

Relevant Data

Time to decision

1980 1990 2000 2010 2020

HP Haven

“ Current enterprise environments are focused on structured data. However, 90% of digital content created by 2015 will be of mixed data types” 1

¹Source: IDC Predictions 2012: Competing for 2020

‹#›

Contextualsearch

Data Normalization

Image/video analytics Geospatial

analytics

SQL on HadoopAccelerated

analyticsSentimentanalysis

Predictiveanalytics

And more...

On-premise On Demand

Haven Big Data platformCore big data business capabilities

Access Explore Enrich Analyze Predict Serve Act

HP Haven

‹#›

“HP FMEC, takes fan data, collects, it stores, and stitches it together…that helps us understand what is being talked about across the ecosystem of the sport.”-- Senior Director of IT, NASCAR

HP Haven in ActionPowering Real-time, multichannel fan engagementHaven solution:Hadoop + HP Vertica+ JBoss +HP IDOL + HP Enterprise Services

Data source:Twitter, Facebook, Google+, Instagram, Reddit, Tumblr, YouTube, Blogs, RSS, Corporate Data, etc…

Results:Understands sentiment, identifies emerging issues, uncovers trends that helps the NASCAR industry increase value for stakeholders, and enriches the fan and broadcast experience

‹#›

HP Haven in Action

‹#›

HP Haven in Action

Understand Market reactions and brand perceptionIdentify key buying factors and/or customer experiencesSegment populations to customize actionsEnable experimentationPredict outcomesEnable new business models

‹#›

NASCAR StoryArchitectureHP HavenIntegrationQ&A

Agenda

NASCAR StoryArchitectureHP HavenIntegrationQ&A

‹#›

Distributionremove datasift reference

‹#›

Broker View

remove the broker0 arrows pointing to the server2

‹#›

<import resource="file:${jboss.server.base.dir}/configuration/nascarAMQ.xml" />

<camelContext id="DS-HTTP4Consumer" errorHandlerRef="myDeadLetterErrorHandler" useMDCLogging="true“

<propertyPlaceholder id="properties" location="file:/opt/rh/esb/data/property/dataSift.properties, file:/opt/rh/esb/data/property/recordRace.properties"/>

<onException>…</onException>

<route  id="HTTP-­‐Ingest-­‐Route"    autoStartup="false"  streamCache="true"  trace="true">      <from  uri="timer://dsHTTP4Timer?repeatCount=0"/>      <to  uri="http4://api.ingest.com/v1/pull?httpClient.soTimeout=30000&amp;throwExceptionOnFailure=false&amp;      id={{streamId}}…"/>  

<choice>                  <when>  

       <simple>${header.CamelHttpResponseCode}  ==  "200"</simple>          <convertBodyTo  type="java.lang.String"/>                  <to  uri="seda:sedaToQ"/>                  <log  message  =  "      %%%%%%%%%%  HTTP=[${header.CamelHttpResponseCode}]    :::  "  loggingLevel  =  "INFO"/>          <to  uri="log:response200?level=INFO&amp;groupDelay=500&amp;groupSize=100"/>            <delay><constant>500</constant></delay>    

     </when>

Media Ingest

‹#›

<route  id="sedaSendToPreQ">      <from  uri="seda:sedaToQ"/>          <to  uri="activemq:queue:preQ?disableReplyTo=true&amp;allowNullBody=false&amp;        acceptMessagesWhileStopping=true&amp;        disableReplyTo=true"/>          <to  uri="activemq:queue:{{recordRace.fileName}}Q?disableReplyTo=true&amp;        allowNullBody=false&amp;acceptMessagesWhileStopping=true&amp;        disableReplyTo=true"/>    

</route>

Media Ingest - Seda Queue

‹#›

<import resource="file:${jboss.server.base.dir}/configuration/nascarAMQ.xml" />

<camelContext id="DS-HTTP4Consumer" errorHandlerRef=“deadLetterErrorHandler" useMDCLogging="true“

<propertyPlaceholder id="properties" location="file:/opt/rh/esb/data/property/dataSift.properties, file:/opt/rh/esb/data/property/recordRace.properties"/>

<onException>…</onException>  

<route  id="HTTP-­‐Ingest-­‐Route"    autoStartup="false"  streamCache="true"  trace="true">      <from  uri="timer://dsHTTP4Timer?repeatCount=0"/>      <to  uri="http4://api.ingest.com/v1/pull?httpClient.soTimeout=30000&amp;throwExceptionOnFailure=false&amp;      id={{streamId}}…"/>  

<choice>                  <when>  

       <simple>${header.CamelHttpResponseCode}  ==  "200"</simple>          <convertBodyTo  type="java.lang.String"/>                  <to  uri="seda:sedaToQ"/>                  <log  message  =  "      %%%%%%%%%%  HTTP=[${header.CamelHttpResponseCode}]    :::  "  loggingLevel  =  "INFO"/>          <to  uri="log:response200?level=INFO&amp;groupDelay=500&amp;groupSize=100"/>            <delay><constant>500</constant></delay>    

     </when>

Media Ingest - deadLetterErrorHandler

‹#›

NASCAR AMQ Client Configuration - Ingest<bean id="jmsFactory" class="org.apache.activemq.ActiveMQConnectionFactory">

<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">

<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">

<bean id=“deadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder"> <property name="deadLetterUri" value="activemq:queue:dlqIngestQ"/> <property name="redeliveryPolicy" ref="myRedeliveryPolicyConfig"/>

</bean>

<bean id="myRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy"> <property name="maximumRedeliveries" value="0"/> <property name="redeliveryDelay" value="5000"/>

</bean>

‹#›

<import resource="file:${jboss.server.base.dir}/configuration/nascarAMQ.xml" />

<camelContext id="DS-HTTP4Consumer" errorHandlerRef=“deadLetterErrorHandler" useMDCLogging="true“

<propertyPlaceholder id="properties" location="file:/opt/rh/esb/data/property/dataSift.properties, file:/opt/rh/esb/data/property/recordRace.properties"/>

<onException>…</onException>  

<route  id="HTTP-­‐Ingest-­‐Route"    autoStartup="false"  streamCache="true"  trace="true">      <from  uri="timer://dsHTTP4Timer?repeatCount=0"/>      <to  uri="http4://api.ingest.com/v1/pull?httpClient.soTimeout=30000&amp;throwExceptionOnFailure=false&amp;      id={{streamId}}…"/>  

<choice>                  <when>  

       <simple>${header.CamelHttpResponseCode}  ==  "200"</simple>          <convertBodyTo  type="java.lang.String"/>                  <to  uri="seda:sedaToQ"/>                  <log  message  =  "      %%%%%%%%%%  HTTP=[${header.CamelHttpResponseCode}]    :::  "  loggingLevel  =  "INFO"/>          <to  uri="log:response200?level=INFO&amp;groupDelay=500&amp;groupSize=100"/>            <delay><constant>500</constant></delay>    

     </when>

Media Ingest - onException

‹#›

<onException>                  <exception>net.sf.saxon.trans.XPathException</exception>                  <exception>org.apache.camel.language.bean.RuntimeBeanExpressionException</exception>                  <exception>org.apache.camel.builder.xml.InvalidXPathExpression</exception>                  <exception>java.net.SocketTimeoutException</exception>                  <handled>                          <constant>true</constant>                  </handled>                  <pipeline>                                              <setHeader  headerName="goingToJMSQ">  <simple>${routeId}</simple></setHeader>                      <setHeader  headerName="Exception"><simple>${exception.message}  </simple></setHeader>                                  <setHeader  headerName="Exception.Stacktrace">       <simple>${exception.stacktrace}</simple></setHeader>                                                                    <log  message="exception  handled  from  [${routeId}]  with  error  ${header.message}"            loggingLevel="ERROR"/>                      <to  uri="activemq:queue:dlqQ?disableReplyTo=true"/>                    </pipeline>          </onException>

onException- Ingest

‹#›

Audit services

‹#›

Replay Service

‹#›

NASCAR StoryArchitectureHP HAVEnIntegrationQ&A

Agenda

NASCAR StoryArchitectureHP HAVEnIntegrationQ&A

‹#›

Q&A

‹#›

Facts about this projectDuration to Production: < 6 months

Server Types:

JBOSS: EAP6.3, StandAlone

ActiveMQ: AMQ5.9.0, Kahadb

Hawtio: 1.48, The One to Rule Them All

Packaging: War

need the server types

‹#›

Facts about this projectCamel/EIP/Features:

Components: Cache, jdbc, sql, xmlJson, HTTP4, SEDA, JMS, ActiveMQ, Multicast, Enrich, Aggregator, Recipient List, JMSGroupId, JMX, Jsoup, Quartz, log4J, SMTP, RSS/ATOM, Swagger, Saxon, Jackson, Servlet, Xstream,

Languages: Spring DSL, Header Language, File Language, Simple, Groovy, JavaScript, Java Beans, Xquery, regex, Python

Error Handlers: onException, deadLetterErrorHandler

Integration Points: Media Ingest, Vertica, Autonomy, SQL Svr, Hadoop, File, Queues, Media Bin, Streams

need the server types

‹#›

Facts about this projectDuration to Production: < 6 monthsLevel of development expertise at the start:

Camel Architect: Six Months

Spring DSL: Three Months

Integration Architect: 25 Years

SOA Architect: 20 Years

Three Java Developers: 10 years Total

JMS: 20 Years

‹#›

Most irritating moment– Looking online for answers and examples. Invariably you’ll see the dreaded Claus Ibsen quote; “refer to the documentation…”

‹#›

Most irritating moment

- what’s irritating is… he’s usually correct.

‹#›

Facts about this projectMost important moment –

Camel instills a “Route” thought process. It ensures you are thinking of the message at all times.

‹#›

Facts about this projectBest Quote Found while researching.

“The only reason for time is so that everything doesn't happen at once”, Albert Einstein

‹#›46

• Storage  o 3PAR  7200  installed  and  configured  o Brocade  switches  installed  and  configured  in  C7000  enclosures  o Zoned  to  all  HP  server  blades  o FMEC  hosts  defined  on  3PAR  

• Servers  o 9  server  blades  installed  and  configured  in  c7000  enclosures  o VMware  ESXi  5.5  has  been  installed  on  all  9  server  blades  o There  are  11  VMs  and  9  hosts  o 4  RHEL  6.5  physical  servers

Facts about this project