Big Data: Big SQL Application Development with BigInsights

Developing, publishing and deploying your first Big SQL application with InfoSphere BigInsights

Cynthia Saracco

Senior Solution Architect

IBM Silicon Valley Lab

2 © 2013 IBM Corporation

Big Data Application Ecosystem

Eclipse

App library

MapReduce, "

Text Analytics

Query

App Development

• Code application program, and generate

associated App

• Deploy Apps to Enterprise ManagerApp

Development

Publish

Data integration scenario:

Pre-defined work flows simplify

loading data from various

sources

•Work flows can be configured,

deployed, executed and

scheduled

Development tooling:

•Text analytics

•MapReduce

•Query languages

• . . .

Application scenarios (web log,

email, social media, �):

• Samples provide starting

point, speed time to value

Big Data Web Console

3 © 2013 IBM Corporation

BigInsights Applications Catalog (Web Console)

� Browse available applications � Manage and deploy applications (administrators only) � Execute (or schedule execution of ) a deployed application � Monitor job (application) status� Link or chain applications for sequential execution

4 © 2013 IBM Corporation

Overview of Application Development Lifecycle

� Configure your Eclipse environment (one-time set up) � Develop your application using BigInsights tools � Test your application � Package and publish your application � Deploy your application on the cluster

5 © 2013 IBM Corporation

Configure your Eclipse environment

� One-time set up � Download and install BigInsights tools (Eclipse plug-ins)

– Welcome tab of BigInsights Web console includes pre-req info, download &

installation instructions

6 © 2013 IBM Corporation

Develop your application – Big SQL example

� Open the BigInsights perspective in Eclipse� Create a BigInsights project

7 © 2013 IBM Corporation

Develop your application – Big SQL example (cont’d)

� Create a BigInsights program with a SQL script (file)

8 © 2013 IBM Corporation

Develop your application – Big SQL example (cont’d)

� Populate your SQL file with the desired code

create table if not exists media_csv

(id integer not null,

name varchar(50),

url varchar(50),

contactdate string)

row format delimited

fields terminated by ','

stored as textfile;

load hive data local inpath

'/home/biadmin/sampleData/DBMS/RDBMS_data.csv'

-- overwrite

into table media_csv;

9 © 2013 IBM Corporation

Test your application

� Run your application from Eclipse

10 © 2013 IBM Corporation

Publish your application to the BigInsights catalog

� Package and publish your application from Eclipse � Specify application name, workflow requirements, etc.

11 © 2013 IBM Corporation

Deploy your application on the cluster

� Access the Applications tab of the Web console� “Manage” the published applications � Locate your new application and deploy it � Create credentials store file in DFS (if needed) -- see next chart � Optionally, execute the application after it’s been deployed

12 © 2013 IBM Corporation

Run your Big SQL application on the cluster

13 © 2013 IBM Corporation

Upgrade your application (optional)

� Satisfy evolving business requirements, improve flexibility – Example: add input parm(s)

� Modify SQL code and re-package, re-publish, re-deploy

create table if not exists $TABLE(id integer not null,

name varchar(50),

url varchar(50),

contactdate string)

row format delimited

fields terminated by ','

stored as textfile;

load hive data local inpath '$FILE' -- overwrite

into table $TABLE;

14 © 2013 IBM Corporation

Upgrade your application (cont’d)

� Re-publish your application from Eclipse � Adjust workflow specs for input parm(s)

15 © 2013 IBM Corporation

Summary

� Eclipse tools simplify big data application development for BigInsights– Wizards

– Context-sensitive help

– Oozie workflow generation

– Built-in test environment

– Etc.

� Application catalog provides easy way to locate and launch apps of interest– Developers use Eclipse tools to package/publish their applications to this

catalog

– Application upgrades easily managed