When Streaming Becomes Strategic(1)

7/24/2019 When Streaming Becomes Strategic(1)

http://slidepdf.com/reader/full/when-streaming-becomes-strategic1 1/7

Te Bloor Group

WHITE PAPER

WHEN STREAMING BECOMES

STRATEGIC

MapR Provides an Architectural Foundation for Data-Driven Enterprises

Robin Bloor, Ph.D.& Rebecca Jozwiak



WHEN STREAMING BECOMES STRATEGIC

1

A New System of Record?The system of record feeds the collection of applications that run the business, such as an ERPsystem. Traditionally such systems passed information via ETL software to data warehousesystems, which in turn fed data marts for the sake of BI and analytics applications. The deningpoint is that the system of record holds the source data of the organization – the golden record of

the “truth.” Often different contributing applications did not agree perfectly on the denitionsof data, so between the system of record and the data warehouse sat master data management(MDM) software to reconcile such differences and provide business level data denitions forusers to help understand the corporate data universe.

For years, that’s how it was. It worked reasonably well until new technologies and inuencesforced their way into the corporate IT environment. You can think of these as falling into twocategories:

• The growth of data

• The acceleration of data ows

Data didn’t suddenly become “big.” On average, corporate data has been growing at roughly 60percent per year for decades, and it continues to do so. Some of this growth can be attributed tonew applications, particularly web-facing applications and mobile applications. Some is fromexternal sources such as social media or public data feeds. Some is data we never botheredto retain or process, such as that in log les. Some comes from ofce systems (email, instantmessaging, etc.).

Because of these varied formats, a good deal of data did not arrive in a conveniently structuredform. Unstructured though some of it was, if it recorded or inuenced the decision making,then it clearly qualied as part of the system of record. Hadoop and its attendant softwareecosystem emerged at the right time to qualify as a possible repository for such data.

The Data Flow IssueWhile data continued to grow like bamboo, the second disruptive factor came into play.Although we described this as the acceleration of data ows, we could have described it asthe trend to data streaming. The old system of record and data warehouse arrangement wasfounded on the idea that data lived in specic locations, and where there was a need, it wasreplicated to other locations, including the data warehouse.

This data architecture came under pressure with the advent of the internet, which mandated24 x 7 web-facing systems. These applications, in turn, impacted corporate applications viaweb services and a service-oriented architecture, forcing some of them to run 24 x 7. Thisengendered rst generation messaging systems (such as ESBs) to service the ow of data and

messages between programs. Gradually, the time windows during which data could be moved became shorter and shorter, provoking software changes and often more hardware to speedup the process.

It has become increasingly apparent that a streaming (i.e., data ow-oriented) architecture isnecessary. Such an architecture can and should be seen as distinct from the streaming systemsthat were introduced quite early on in the nancial services sector for automated trading.Although these were streaming architectures of a kind, they were focused on a particularapplication rather than built to provide a general purpose streaming environment.











Documents

When Streaming Becomes Strategic(1)