Upload
tomasz-bednarz
View
286
Download
0
Embed Size (px)
DESCRIPTION
Presentation for CSIRO Big Data Workshop 2012. October 2013, North Ryde, Australia.
Citation preview
BIG DATA IN FINANCETomasz Bednarz
���1
OVERVIEW OF THIS TALK
���2
• Flavors of big data in finance • Technology map for big data in finance • Wheel of reincarnation • A touch of history • Role of visualisation • Frameworks • Real Time HFT
BIG DATA AGAIN• Big data is not a “crystal ball”
• The value from big data can only be extracted when there is precise business problem to be addressed
• Business problem has to be understood and well defined at the first place
• Business to data mapping • Better business predictions = difficult process
• Analysing big data • Data scientists to examine the data and extract critical information such as customers buying habits - need for experts
• Catalogue data assets (10% of data may actually mean something)
• Privacy • No clear access controls to access data?
Volume
VelocityVarietyBig Data
SURVEY
http://strata.oreilly.com/2012/01/enterprise-big-data-survey-results.html
FLAVORS OF BIG DATA IN FINANCE
• Structured • Market data feeds
• Big and fast • Order flows • Execution info
• Unstructured • News & • Research
Dad Bailey’s news stand in 1939. Comics are on the top left.
���5
TECHNOLOGY MAP FOR DATA
Information Retrieval Analytics
Loosely Structured Information
Highly Structured Information
Human Computer
Market Interface
= high frequency ticks/trading & order flow
= text and relationships
���6
TECHNOLOGY MAP FOR DATA
Information Retrieval Analytics
Loosely Structured Information
Highly Structured Information
Human Computer
Market Interface
= text and relationships
= high frequency ticks/trading & order flow
fast: trading
���7
TECHNOLOGY MAP FOR DATA
Information Retrieval Analytics
Human Computer
Market Interface
= text and relationshipsLoosely Structured Information
Highly Structured Information = high frequency ticks/trading & order flow
slow: investment research, portfolio management
���8
SKILL ZONES - HUMAN VERSUS MACHINE
���9
Time to exploit
information
Data Subtlety & Complexity. Language & Concept Importance.
ms
months
low highpure HPC (HFT)
collaborative hybrid generation
smart market
monitoringalgo
gurus
Visualizing Marathon 2011CSIRO GPU Cluster
Reinvent tech/self every ~three years!!!
WHEEL OF REINCARNATION
���10
T. H. Myer and I. E. Sutherland. On the design of display processors. Communications of ACM, June 1968, 410-414.
STRUCTURED DATA
���11
���12
NYSENew York Stock Exchange started as very low-tech place. In 1779, the NYSE was a bunch of guys standing around a buttonwood tree at 68 Wall Street shouting at each other on days when it didn’t rain or snow.
In 1794 we see the first big technological solution: the roof.
NYSEEverybody moved inside, to the Tontine Coffee House at the corner of Water and Wall streets. Hands, roofs, chalk (oh technology)!
1n 1823, the Difference Engine was invented by Charles Babbage “I wish to
God these calculations had been executed by steam.”
TECHNOLOGICAL INVASION - SIMPLER ERA
���14
Before telegraphy, in 1850s, the sky over Wall Street was open and clear
It took only a short time for telegraphy’s compression of time and space to transform
the scenery. Everybody had to have it!
BUT you had to know Morse Code to participate in the market! (Morse, 1837)
TICKER TAPEA huge success (as important as the roof, hand signals, and the telegraph). People use jumbo magnifying lenses - people traded faster than the machines, so delay meters were installed.
All that ticker tape also made for nice parades. Here group of specialists
celebrating the one-millionth bagging of a buy-side trader.
AND MORE TECHNOLOGYTicker tapes (an enduring market visualisation) are still with us today.
Type 80 Card Sorter IBM 1925
Our CSIRAC 1 operation per second - 1949
ROLE OF VISUALISATION
Dynamic market view: Oculus Info’s Visible Marketplace - portfolio visualisation. !Not very dynamic on paper. !Another reason to be glad we have web browsers.
MAP OF THE MARKET
Map of the Market , the classic big picture of whole market visualization. It was invented by Marten Wattenberg, now at IBM Many Eyes.
Cultural Finance
Finance Cover Letter
UNSTRUCTURED DATA
���20
TECHNOLOGY MAP FOR DATA
Information Retrieval Analytics
Human Computer
Market Interface
= text and relationshipsLoosely Structured Information
Highly Structured Information = high frequency ticks/trading & order flow
slow: investment research, portfolio management
���21
INFORMATION SOURCES• Specialised Industry Media • Local and International Media • Direct Corporate Communications • Research Labs • Government Agencies • Social Media • Crowd Sourcing !
• HTML / Text Feeds / XML
BIG DATA AT NICTA• Developed to deliver customized business intelligence and advanced data mining to the Enterprise marketplace
• Machine Learning to obtain deep insights in real time
• Build upon a scalable open source software platforms
• See http://www.youtube.com/watch?v=TxQPdUt_x3c • Scoobi - a Scala productivity framework for Hadoop (allows you to write
what you want rather than how to do it). • Source code: https://github.com/nicta/scoobi
• Demo of Scoobi tomorrow at the hands-on afternoon session (by Piotr)
http://www.ambiata.com/index.html
���24
http://www.zdnet.com/how-is-big-data-faring-in-the-enterprise-7000002404/
REAL-TIME REAL-TIME REAL-TIME
• Everything is moving to real time
• Everything is moving towards continuous time
• Everything is moving towards mobility (anywhere, anytime)
• Also GPU for high-frequency trading (HFT)
• GPU alongside FPGAs receiving stream of market data to increase the accuracy of the strategy, or even to suggest change in the algorithm used as market conditions evolve
• GPU Direct enables a GPU to talk directly to the FPGA, receiving data from it into a kernel, which may be preloaded to perform its magic once the data is available
• Hadoop + GPUs
ACKNOWLEDGEMENTS
• Andrew Sheppard, Fontainhead
• David Leinweber, The Lawrance Berkley National Lab, author of Nerds on Wall Street NOWS
• Colleagues from Westac and Commonwealth banks
• Craig Mudge for organising Big Data workshop, and always great discussions
���27
THANK YOU