Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Dasgupta and Mayoor Rao,...

REGULATORY REPORTING OF ASSET TRADING USING SPARK

Sudipto Shankar Dasgupta & Mayoor Rao – Infosys Limited

Regulatory Reporting – Current Challenges

2

Financial Institutions need to report asset trade based on defined regulations

Reporting has to comply within a time window from the trade execution

Rules are subject to change and adjustments based on the change in regulations

The transaction volume is high Transactional systems are diverse

Case Study 3

Leading global Financial Institution engaged in around 6 million trades a day in the financial markets

Regulatory requirements necessitates reporting in a specified reporting format within a 15 minute window

The trades are identified for reporting based on steps •  Transformation •  Enrichment •  De-duplication checks •  Business Rules

Report Generation Process Flow Asset

Transactions

Transformation Rules

Transformed Asset Data

Master Data

Enriched Asset Data

Business Rules

Transformation Enrichment Report Generation

4

Architecture Diagram

SQOOP

HDFS Data Lake

Tem

pora

ry S

tagi

ng

Pro

cess

ed O

utpu

t

In Memory Analytic and Processing Zone

Processing Logic

Apps Portal

5

Ingestion throughput >130,000

records/second or 18.22 MB/second (per node)

Report execution time -> 35 seconds for 30,000 trade records

Overall Data Volume -> 596 million trade transaction records

Hardware Configuration -> Spark Installed in 10 node 32 GB RAM , 16 Core m/c

Statistics 6

01

02

03

04

DEMO

7

IIP - Leveraging Power of Spark 8

Open Source / Spark Components

Spark ML

Spark Streaming

HIVE / Spark SQL

Hadoop / FS Storage / Infra Mgmt

Infosys & Partner IP Components Tools | Data Extractors | Algorithms | Packaging &

Support

Customization, Integration & Implementation Services

Data Modeling & Cleansing | Agile App Development Data Science & Analytics | Security & Governance

Custom Data Extractors

Infosys Information Platform - Architecture IIP Data Lake!

Query Interface

BI Applications

Down Stream Apps / Portal

IIP Data Lake!

1"

Rea

l Tim

e –

IIP In

gest

ion

Unstructured Data

RDBMS Data

NO SQL Data

Data Streams

JMS Queue

Social data

Machine data

Machine data

Web URL

Resource Manager

Governance

Security

Data Store

OS /LDAP/Kerberos Authentication Authorization Role Based Access

Metadata Management Data Lineage Data Audit

Cluster Resource Manager Cluster Monitor Disaster Recovery

Spark In Memory Analytic Zone

HDFS / Hive

Natural Language Processing

Machine Learning

Data Views

Data Mining and Models

Data Joins

Data Transformation

Single Click Installer Cluster Maintenance

Deployment Manager

IIP  Data  Explorer  

2"

3"

4"

5"

9

DEMO

10

© 2015 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this documentation nor

any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.

Thank You