iWay Unites On-Demand Integration, Big Data forReal-Time Analytics

iWay Software, a unit of Information Builders, is bringing together the worlds ofon-demand integration with Big Data. IDN explores how iWay has optimized itslong-standing any-to-any integration framework and connector architecture to make iteasier for IT to design and deploy solutions based on MapReduce, Hadoop and othernew data and analytics technologies.by Vance McCarthyTags: analytics, Big Data, Hadoop, integration, iWay, MapReduce,

iWay Software, a unit ofInformation Builders, is bringing together the worlds of on-demand integration with Big Data.

IDN speaks with iWay’s marketing director Vincent Lam to learn how iWay has optimized its long-standing any-to-anyintegration framework and connector architecture to make it easier for IT to design and deploy solutions based onMapReduce, Hadoop and other new data and analytics technologies.

“We’re definitely seeing among our customers a trend where unstructured data with technologies such as Hadoop andMongoDB are becoming a bigger component of what companies are using, and these customers need those to worktogether with their RDMBS and other resources,” Lam told IDN. “Supporting these solutions within iWay was importantbecause to us, the whole crux of integration is to get all the things a customer needs to work together to worktogether.”

“Our support for MapReduce allows iWay to federate the query [across distributed data

sources] very quickly.”Vincent Lam

Marketing DirectoriWay Software

Supporting Hadoop, MongoDB and other new unstructured data sources was not a major problem for iWay’s corearchitecture, he added.

“iWay has always used an agnostic approach to integration, where endpoints were always separated at the functionallayer. So, integrating with these new types of data only meant that we needed to make sure we had new adapters thatwould talk to these sources,” Lam said.

That said, iWay also wanted to go one step further and make it easier for various IT professionals (from both the dataand the integration side) to design and deploy these types of solutions.

“Because MapReduce and other Hadoop and ‘Big Data’ tools are so new, IT often needs to learn specialized skills.Even beyond training. Sometimes it’s also not clear how IT should address a project or even who should takeresponsibility,” Lam added. “That’s why we added MapReduce-caliber capabilities to its iWay Parallel ServiceManager.”

The iWay Parallel Service Manager 6.1 adds MapReduce-style functionality to provide simplified tooling andout-of-the-box integration support to power Big Data solutions, such as “federated search” across unstructured datastores, as well as many popular ones now used by enterprises including Oracle, MUMPS, and HL7.

Lam added, “For some customers, the ability to search across a large pool of physically distributed data sources andobtain near-real time results can be a monumental issue,” adding MapReduce allows iWay to federate the query veryquickly, supported by agents in the iWay Parallel Service Manager.

The latest Parallel Service Manager also extends ETL with “parallel” extracts, transforms, and loads, as well tobatch-oriented managed file transfer (MFT) where the processing and distribution of files takes place in parallel

Under the covers, the iWay Parallel Service Manager will accelerate the processing of complex queries and/orcomputations by breaking them up and distributing them across multiple technology assets.

It will perform these computing functions in parallel (rather than sequentially) because inside the iWay Parallel ServiceManager, “map” and “reduce” functions are done in a process flow, which is executed in parallel for each item in thelist. So, when a user makes a query that needs to access multiple data stores simultaneously, iWay launches a“parallel control agent,” which will simultaneously execute process flows configured with information for whatconnections are required based on information in the list. The outcome of each query can then be amassed into afinal results document.

This approach, Lam said, will deliver key MapReduce-optimized benefits, without the need for IT complexity. Amongthe benefits are:

● Reduce complexity of setting up and executing MapReduce-type federated queries

● Seamless integration with the Hadoop Distributed File System

● No need to write custom Java code to divide large queries across multiple systems (and thencollect and aggregate the results)

● A drag-and-drop interface to set up and execute parallel processing jobs

● Pre-packaged integration components providing native access to 300+ backend source / targetsystems

This iWay approach to bringing together Big Data and integration comes as Gartner predicts that real-time analytics

will explode for structured and unstructured data across distributed systems.

“Analytics has become a major driving application for data warehousing, with the use of MapReduce outside andinside the DBMS, and the use of self-service data marts, according to Gartner’s just-rleeased report on the Top 10Technologies for 2012. “One major implication of big data is that in the future users will not be able to put all usefulinformation into a single data warehouse. Logical data warehouses bringing together information from multiplesources as needed will replace the single data warehouse model.“