Embed Size (px)
Citation preview
www.talend.com ©Talend 2018 · En
Becoming a “data trader”
“ In the stock exchange sector, we follow three watchwords: integrity, because it is impossible to lose a single order; permanent availability; and governance in a highly-regulated market. Talend has met these expectations.
CA S E S T U DY
” Abderrahmane Belarfaoui, Chief Data Officer, Euronext
Following its split from the New York Stock Exchange in 2014, Euronext became the first pan-European exchange in the eurozone, fusing together the stock markets of Amsterdam, Brussels, Dublin, Lisbon, and Paris. Euronext comprises close to 1,300 issuers, reporting a total market capitalization of 3,700 billion euros at the end of March 2018.
In 2016, Euronext began the typical process of migrating its data to the cloud. Except that this migration had nothing typical about it at all. First off, the Euronext database contained 100 TB of data– one of the biggest in Europe. Then there was the fact that this was not just a simple transfer of a database to a hosted platform. The idea was to create a governed data lake with self-service access for business units and clients in an effort to monetize new services and generate additional revenues.
Migrating to a governed cloud
“We use Optiq, an incredible trading platform, with systems that practically work in nanoseconds,” explains Abderrahmane Belarfaoui, Chief Data Officer (CDO) at Euronext. The huge Euronext database is the active memory of transactions handled directly by the stock exchange operator (1.5 billion messages per day). “The database is compressed (at a rate of 400%) but some information was not being archived for lack of space,” says Belarfaoui about the problems of the old system.
Before 2016, Euronext stored its data on site, on hardware from one of the big names in the industry. But Euronext’s storage needs continued to grow, especially following several acquisitions, such as the Dublin Stock Exchange and Fast Match in the US.
“Our IT infrastructure had reached the end of its lifecycle in our European operations, where regulators were expecting that Euronext store more and more data,” Belarfaoui recalls. “Moreover, sometimes we had to wait six to twelve hours after market close on days with important events, such as the UK Brexit vote, before we could send the data to business units and clients.”
The situation prompted the CDO to look at moving to a hybrid cloud model. “We still keep trading platform information on an on-site server because lag times are not yet available on the cloud,” explains Belarfaoui. “We also use AWS Managed Services in serverless mode together with Amazon S3 to have access to a data warehouse with unlimited storage capacity. For analysis, we use Amazon Redshift. And taking advantage of the cloud’s great scalability, we can run the whole system while anticipating events that cause high volumes on the markets.”
Still, the transition to a Platform as a Service (PaaS) does require one key condition: remaining independent of the cloud provider.
Why Talend?
Euronext chose Talend Big Data to absorb real-time data in the data lake, including internal data from its own trading platform; and external data, such as from Reuters and Bloomberg.
“The core of the data lake is managed by Talend. It was very important for us to keep this “independence”compared to the layers below Talend. So, if tomorrow Euronext wants to change clouds, they can,” says the CDO, happy about the greater flexibility.
INDUSTRY
• Finance/Stock Market
INFORMATION• Head office: France
• 501-1,000 employees
USE CASE• Stock Trading
CHALLENGE• Migrating to a governed cloud
TALEND PRODUCT USED • Talend Big Data
• Talend Data Catalog
RESULTS
• 100 TB of transaction history: one of the largest databases in Europe, with 1.5 billion messages per day
• From 200 person-days to one click for tracking data
• Ten times more iso-budget data generated
In an ultra-regulated world, Talend has also proven to be highly adept at meeting the challenges of data lake governance and regulatory compliance. Being able to safely open data involves knowing it inside out, keeping track of changes and the history of data feeds, and knowing how to classify them in a granular structure.
“We have an Amazon S3 storage that is shared by everyone. So I have to know who owns data from the start (the data owner), who has access to what, whom to ask, who can use it, and who has priority over whom. Our data stewards protect the organization of our data,” adds Belarfaoui.
This governance strategy is applied in very specific tools, such as the Talend Data Catalog. A dictionary is created together with each technical project for each individual market. These dictionaries are used to find the history of end-to-end data, from the sources to the reporting. “Now I can see when S3 is the data source, I can add value to the data, combine it with other data, and convert it into other data in Redshift,” says the CDO, who is very satisfied with the new process. “I can also add tags. Typically, we add the storage duration. For example, whether data has to be kept for ten years, or five years (per MIFID II), or if it should be archived.”
At the same time, data lineage with Talend drastically reduces impact analysis costs. “One simple example comes to mind: we plan to change the value of an index on the British stock market. Once we integrate it into our systems, it propagates itself pretty much everywhere. Currently we have to figure 200 person-days just to find the index in our different systems. But with the dictionary, we are able to run this data lineage with just one click.”
Monetizing stock market data
Two years after its launch, the governed lake project with Talend and AWS is a success. “The initial returns are more than positive,” says Belarfaoui. “On the technical side, we can manage ten times more iso-budget data.”
Beyond the improved architecture, the migration is also positioning Euronext to become a “data trader.” The stock market operator wanted to be able to refine and add
to its wealth of data in order to monetize it. In fact, the sale of data already brings in 20% of Euronext’s revenues.
“Traders actually sell, buy, and make their investment decisions in milliseconds. They have a huge appetite for aggregated data in real time. Who sells which stock, to whom, at what price and when. We are in the best position to track performance of the CAC 40 or other indexes and sell that information to investors through our Datashop platform,” says Belarfaoui.
In addition to clients, this project also involves giving data scientists and business units self-service access to this data, which they can analyze in data sandboxes for tasks such as market monitoring. Belarfaoui explains: “We can set up an environment for a data scientist in less than one day, compared to the 40 days it used to take, and we have moved from D+1 analytics to real-time analytics. This is fundamental to understanding markets, clients, competitors, and how they interact.”
This is a real turning point for Euronext. “In 2016, we identified the need, but we didn’t have the capacity to do it. At the time, we could only relay the volumes of market activity to market regulators (Mifid II). Today, we can dig deeper. Under the General Data Protection Regulations (GDPR), I have to know where personal data is stored. If I receive requests for modification or deletion, I can find the data, thanks to the dictionary,” elaborates Belarfaoui. “Similarly, a user who searches a transaction can instantly see if it is confidential. Once data is identified as being critical, the data steward can deny user access.”
Euronext is just at the beginning of its digital transformation. A study is currently underway on the deployment of Talend’s Master Data Management (MDM) solution. “We are working on ‘golden sources’ within all of our systems (CRM, trading, billing, finance, various departments, subsidiaries, etc.). The goal is to make all Euronext data even cleaner and of higher quality, such as by being sure that a client is consistently represented across all systems. Such standards will make our data even more usable,” predicts Belarfaoui.
www.talend.com/contact | [email protected] | [email protected] | [email protected]
“Talend is everywhere in our project. We benefit from the most advanced concepts, such as serverless or continuous integration. We also rely on Talend Big Data as well as components of Data Cataloging and Master Data Management.”
