Upload
pal-evensen
View
143
Download
2
Tags:
Embed Size (px)
DESCRIPTION
My PhD thesis defense from 23.04.2013
Citation preview
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Event Processing Applied toStreams of TV Channel Zaps and
Sensor Middleware with VirtualizationPhD Defense
Pal Evensen
Department of Electrical Engineering and Computer ScienceUniversity of Stavanger, Norway
April 23rd, 2013
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
From this
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
To this
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Request/Reply
Client Server
Request
Reply
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
To this
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
and this
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Publish/Subscribe
Publisher Subscriber
PublisherSubscriber
Subscriber
Subscriber
Event
Service
Notificationpublish
notify
subscribe/unsubscribe
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
IS-Home
Altibox at home
INTERNETFrom 10 to 400
Mbit/s, both ways
DIGITAL TVChoose between150 TV channels, film rental and more.
ALARM SERVICESAlarm monitoringcentre. Direct alarm connection to the firebrigade.
MOBILE PHONE SERVICESSame rate, no matter who you call
IP TELEPHONYNo charges to Landlines in Scandinavia
TvFILM RENTALChoose from 1000 titles Now in HD too
FOOTBALLThe whole Premier League and Tippeligaen (Norwegian football pools league)
NEWSUp to date local news
TV-GUIDEA complete overviewfor your TV evening
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Research Contributions
Two main topics:
1. Sensors in smart homes• Middleware for hiding the heterogeneity of devices
2. Event processing applied to TV channel zaps• Architecture for efficient processing of high volumes of events• Evaluating programming models for stateful event processing
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Motivation
Sensorsand
actuators
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
SenseWrap Features
• IP-enabling sensor devices
• Uniform interface to hetereogeneous devices
• A “blueprint” for developers of smart home applications
• Automatic network configuration and service discovery
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
SenseWrap Architecture
VirtualSensor
Physical sensors
Sensewrap
VirtualSensor
Sensor protocol (ZigBee, Bluetooth, etc)
zeroconf
Driver Driver
Gateway
Service
UDP TCP
Service
TCP
Service
HTTP
Client Client Client
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Event-Driven Architecture
Event ProcessingNetwork
EventProcessingAgents
Event ConsumersEvent Producers
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Components
• Esper
• HornetQ
• Custom Java glue
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Event Flow and Interfaces
EventProcessor
Adapter
EventPublisher
EventProcessor
EventPublisher
Message bus
Producer Core
Event
EPL
EPLUpdateListener
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Event Processing Language (EPL)
• Declarative query language derived from SQL
• Operates on stream data as opposed to relational data
• Looks for event patterns that matches the query, andproduces an output event
• Includes additional operators, such as sliding windows
• Part of the Esper framework (Open Source)
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Continuous Queries
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Sliding Windows
win:time(10 sec)
time0 10 20
now
e1 e2 e3 e4 e5
1 select ∗ from ChannelWin.win:time(10 sec)
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Event Patterns
Simple
AEvent
Complex
CEvent
Simple
BEvent
1 select ∗ from pattern [every a=A −> b=B]
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Current State of TV Viewer Statistics
• Sample size is 0.045 % of Norwegian television households
• Only 0.022 % of American households are sampled
• Data collection requires specialized equipment
• Transferred once a day
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Motivation
• IP-based Set Top Boxes (STBs) allows for more accuratestatistics
• Ads can be evaluated on an individual basis
• Additional behavioral markers such as mute and volumechanges allows for better understanding
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
The Future of Media Measurement
• Current measurement methods for TV not in line with newmodels of media consumption
• Gartner: The online/offline division of media will be replacedby measured/unmeasured by 2015
• STB data increasingly being used to augment traditional TVratings
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Altibox Deployment of IPTV
• Over 320,000 STBs deployed in Norway and Denmark
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Our Scenario
• Two-way communication
• Immediate transfer of zap, mute, hdmi and volume events
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Observed Events
• Channel change event (also called a zap event)
• HDMI status event: TV set on/off
• STB audio on/off event (mute)
• STB volume change event
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Number of zap events/day over a 15-day period
31 1 2 3 4 5 6 7 8 9 1011121314
2
2.2
2.4
2.6
·106
Avg=2212097
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Zapping Activity
0 5 10 15 200
0.5
1
1.5
·105
Time (hrs)
30 Minutes15 Minutes5 Minutes
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
EPL Viewer Statistics
Esper Engine
Zaps ChannelWinchannelName,viewersselect(...)
channelName,viewers
1 select channelName, viewers2 from ChannelWin3 output snapshot every 1 sec
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Tru
de
/18
Sy
kk
el:
Gir
o d
`It
ali
a 1
4.
eta
pp
e
Sp
ort
& S
pil
l
Re
ise
må
l
Lø
rda
gsm
ag
asi
ne
tS
po
rtsn
yh
ete
ne
Væ
ret
Vin
n p
å m
inu
tte
t 9
:12
Ny
he
ten
e
Go
d k
ve
ld N
org
e
Sp
ort
sny
he
ten
e
Fil
m:
Mo
rd p
å k
red
itt
Sy
kk
el:
Gir
o d
`It
ali
a -
op
psu
mm
eri
ng
Fa
rme
n 3
0:3
0
Ny
he
ten
e
Ak
va
rie
t
TV
2 h
jelp
er
de
g 1
1
0
10000
20000
30000
40000
50000
60000
12:00 14:00 16:00 18:00 20:00 22:00
Vie
wer
s
Time
Viewers for TV2 Norge + TV2 HD, Saturday 21.05.2011, Separated by Zip Codes
Total viewers0001-39994000-59996000-79998000-9999
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
30000
35000
40000
45000
50000
55000
60000
65000
70000
10:00 12:00 14:00 16:00 18:00 20:00 22:00
See
re
Tidspunkt
Seerstatistikk for TV2 Norge og NRK1, Lørdag 21.05.2011
TV2 Norge + HDNRK1 + HD
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Annoyance Detection
• Aimed at detecting ads/programs that is causing viewers tochange channel
• Triggers an output event on rapid drop in viewers on aparticular channel
• Sliding window algorithm, continously comparing viewernumber with the last minute average
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Annoyance Detector
Annoyance detector
ZapSnap.win.time(1 min)
channelName,viewers,avg(viewers)
avg(viewers)
having(...)
channelName,viewers
1 select channelName, viewers, avg(viewers)2 from ChannelWin(viewers > 2000).win:time(1 min)3 group by channelName4 having viewers < avg(viewers) ∗ 0.85
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Deployment Details
• Processes over 40,000 events/min at peak hours
• Deployed in the Altibox network across four servers:
ZapCollector Messagebus
DatabaseCore
+Manager
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
CPU load - ZapCollector
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
CPU load - Message bus
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
CPU load - EventCaster instance
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
EPL and Java Implementations
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Experimental Setup for Performace Evaluation
• 2,117,897 zap events (one day)
• RHEL6,Quad Core 2.4GHz CPU, 14GB RAM
• Processing capacity tested by loading events from:• Memory• Disk• Network: UDP and HornetQ
• Memory consumption tested by loading events from:• Memory• Disk
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Processing capacity
Memory Disk UDP HornetQ
0
2
4
6
8·105
Eve
nts
/sec
Esper Java
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Memory consumption
From memory From disk
600
800
1,000
1,200
1,400
1,600 1,570
1,401
590 577
Mem
ory
use
d(M
B)
Esper Java
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Performance Evaluation - Observations
• The Java implementation outperforms the Esper version
• Can be accredited to abstraction overhead imposed by theEsper engine
• Performance margin decrease as network overhead isintroduced, but is still significant
• UDP benchmarks offer 45% higher throughput with the Javaversion
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Halstead’s Metrics
• Metrics that describe the complexity and effort required towrite a program
• Sees a program as a series of tokens, classified as eitheroperands or operators
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Complexity Metrics
0 100 200 300 400
Vocabulary (n1+n2)
Program length (N1+N2)
Total operators (N1)
Total operands (N2)
Unique operators (n1)
Unique operands (n2)
Lines of code
(Java)(EPL)
Viewer Statistics
Annoyance Detector
Parsing and Query Setup
Utility Functions
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Complexity Evaluation - Observations
• The EPL implementation scores slightly better for the viewerstatistics application
• EPL scores significantly better for the annoyance detector
• Initial effort was similar for both applications
• Expanding EPL applications is significantly easier than Java
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Summary
• The Java implementation is the most performant
• The Esper implementation is easier to expand
• Can be attributed to different degrees of abstraction
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
OutlineIntroduction
BackgroundProject ContextResearch Contributions
SenseWrapSenseWrap
EventCasterSystem ArchitectureEsper and EPL
Television StatisticsBackgroundImplementationStatisticsDeployment
Paradigm ComparisonPerformanceComplexityConclusions
AdScorerOverviewExperimentsLive Scoring ResultsSummary
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Deployment
TV Network
AdDetectorMute
AdSuccess-Evaluator
STBs
Inp
ut ad
apters
Stats
Ad start
Channel zap events
Filterin
g, tran
sformation
Zap
Ad success score
Historical stats
Processing
QueueingInput
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Defining an advertisement as a context in EPL
Listing 1: Context declaration
1 create context AdBreakCtx as2 initiated by3 AdIdentified(begin=true) as ad4 terminated by5 AdIdentified(detectId=ad.detectId, begin=false) as endAd
Listing 2: Populating the context
1 context AdBreakCtx2 create window STBsnapshots.win:keepall() as STBWin3 insert where4 channel in (context.ad.channel) and hdmi=true and mute=false
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Experimental Setup for AdScorer Evaluation
• 1.5 hour of prime time TV was recorded
• Advertisement times were manually recorded
• 23 days of STB data was sent through the system in order toreach a correct state
• Logged STB data and AdIdentified-events was then pushed tothe system
• Simulation run
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Viewer numbers
Figure: Viewership (in thousands) for the three largest channels, NRK1,TV2, and TVN.
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Viewer numbers
12
13
14
15
16
17
18
19
20
21
S E S E S E
TVN
Figure: Viewership (in thousands) for a single channel (TVN).
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Retained Viewers
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Gained Viewers
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Raw output for a single ad
1 Event: {2 AdId: Mitsubishi ASX,3 channel: TV2 Norge,4 startTime: 22:55:12,5 stopTime: 22:55:32,6 viewersBegin: 83846,7 retained: 79244,8 IAR: 94.51,9 mutes: 141,
10 viewersLost: 4602,11 average volume: 49.6412 }
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Summary
• AdScorer is capable of delivering an unprecedented level ofdetail, not possible through the current measurement regime
• The results indicate that our implementation is capable ofscoring advertisements on multiple channels simultaneously innear real-time with consistent results
Introduction SenseWrap EventCaster Television Statistics Paradigm Comparison AdScorer
Thank You for listening!