Upload
spark-summit
View
899
Download
0
Embed Size (px)
Citation preview
Challenges
•Distributed across sites and firms
•Latency sensitive
•Multiple protocols and data formats
EXCHANGES
Matching Engine Trading
Gateways
Market Data
Validation/ Routing
Consolidation/ Distribution
MARKET DATA SERVICES
BROKERAGE SERVICES
TRADERS
•Transformations:•Change of custody•Different symbologies•Parent/child orders•Variable routing
Platform
Corvil Streaming
DataCorvil
Streaming DataCorvil Streaming
Data
Message BusDistributed Real-Time Processing
StorageData Sources
Exploration and Visualization
Metrics Calculation
Windowing & Correlation
Raw Data•Enriched (correlated data)•Searchable
Custom Metrics•Latencies•Counts•Filterable by key dimensions•Rolled up
Clients MarketsA B C D
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events Correlation
Filter by destination market in point A?
A B C D
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
ID: 1
CLIENT: C1SYMBOL: ABCMARKET: NYSE
ID: APARENTID: 1
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: ZPARENTID: A
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: 10PARENTID: Z
CLIENT: C1SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
Events Windowing
Batch #1 Batch #2 Batch #3
window #1
window #2
A B C D
Spark Streaming window
Reconstruct batch based on original time
Spark Streaming micro-batch
Direct Data Access
Access metrics via SQL / JDBC
BI tools, self-developed application, etc
Calculation done in HBASE
Data Extracts
• Filtering pushed down to HBASE• Aggregation done In SPARK (SparkSQL)
CSV extraction to HDFS