- sheepsqueezers.comsheepsqueezers.com/media/documentation/oracle/ore-trng1... · 2015-10-27 · ... The following is intended

1

<Insert Picture Here>

©2011 Oracle – All Rights Reserved

Oracle R Enterprise – Training Sessions

Session 1: Getting Started with Oracle R Enterprise

Mark Hornick, Senior Manager, Development

Oracle Advanced Analytics

3

The following is intended to outline our general product direction.

It is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver

any material, code, or functionality, and should not be relied upon

in making purchasing decisions.

The development, release, and timing of any features or

functionality described for Oracle’s products remain at the sole

discretion of Oracle.

4

Oracle R Enterprise Training Sessions

Date / Time Session Topics

Thursday, December 1, 2:00 PM ET Getting Started with Oracle R Enterprise

(ORE)

Oracle R Enterprise Overview

Installation of R

Installation of Oracle R Enterprise

Connecting to Exadata from R

Tuesday, December 6, 11:00 AM ET Introduction to the R Language and

Environment

R Language Basics

Producing Graphs in R

Thursday, December 8, 2:00 PM ET ORE Transparency Layer Interacting with Database Tables

Manipulating and transforming data through ORE

Tuesday, December 13, 11:00 AM ET ORE Embedded R Script Execution Execution through R interface

Execution through SQL interface

Thursday, December 15, 2:00 PM ET Operationalizing R Scripts From Analyst to Production

Integration with OBIEE

XML graph generation using SQL

Tuesday, December 20, 11:00 AM ET Advanced Topics Base SAS equivalent functionality

ORE support for Hadoop and Map-Reduce

Use of ORE in Exadata and BDA environments


5

Getting Started with Oracle R Enterprise (ORE)

• Oracle R Enterprise Overview

• Installing R

• Installing Oracle R Enterprise

• Connecting to Exadata from R


6

Oracle R Enterprise Overview


7

What is R?

• R is an Open Source language and environment for

statistical computing and graphics

http://www.r-project.org/

• Started in 1994 as an Alternative to SAS, SPSS & Other

proprietary Statistical Environments

• The R environment – R is an integrated suite of software facilities for data manipulation,

calculation and graphical display

• Around 2 million R users worldwide – Widely taught in Universities

– Many Corporate Analysts know and use R

• Hundreds of open sources packages to enhance

productivity such as: – Bioinformatics with R

– Spatial Statistics with R

– Financial Market Analysis with R

– Linear and Non Linear Modeling


http://www.R-project.org/



8

Why statisticians/data analysts use R

R environment is…

• Powerful

• Extensible

• Graphical

• Extensive statistics

• OOTB functionality with

many ‘knobs’ but

smart defaults

• Ease of installation and use

• Free

Statisticians typically are…

• Not SQL literate

• Not familiar with DBA tasks

R is a statistics language similar to Base SAS or SPSS statistics


9

Limitations of R

• R is a client and server bundled together as 1 executable - like Excel

– Single user tool

– Not multi-threaded

– Cannot leverage CPU capacity even on a user's laptop/desktop

• R requires data it operates on to be first loaded into memory

– Loading data may not be a limitation given RAM available on laptops/desktops

– R’s call by value semantics means as data flows into functions, for each function

invocation, many copies of the data are made

– As a result you quickly run into memory limits


10

Why should you be interested in R?

• Emerging trends

– It’s the next “big thing” in advanced analytics

– Colleges and universities use R for statistics classes (replacing more traditional software tools)

– Advanced Analytics as a critical differentiator of the DWH technology stack

• Augment Oracle deployments

– Enhance results with powerful graphics

– Integrate R results and graphics with BI Publisher documents and OBIEE

dashboards

• A scalable R via Oracle R Enterprise

– Leverage Oracle-engineered solutions

– A worthy alternative to SAS


11

Rexer Analytics Survey 2011

http://www.rexeranalytics.com/Data-Miner-Survey-Results-2010.html ©2011 Oracle – All Rights Reserved

http://www.rexeranalytics.com/Data-Miner-Survey-Results-2010.html









12

R Ecosystem Resources

• Many helpful books

– http://www.r-project.org/doc/bib/R-publications.html

• Installation and Administration

– http://cran.r-project.org/doc/manuals/R-admin.html

• User Groups

• UseR Conferences

• Huge libraries of reusable code

– Visit CRAN: http://cran.r-project.org/

• Well-documented packages

– With sample data and code


13

Default R GUI


14

RStudio – Third Party, Open Source IDE


http://www.kdnuggets.com/polls/2011/r-gui-used.html

15

Data Visualization with R

http://addictedtor.free.fr/graphiques/


16

Graph examples…


17

Oracle Strategy for R

Deliver enterprise-level advanced analytics based on R environment

1. Oracle’s Distribution of Open Source R

• Enterprise support for open-source R

• Enhanced performance with Intel MKL libraries for x86 hardware

• Packaged with Oracle Linux distribution

2. Oracle R Enterprise

• Embedded component of the RDBMS

• Eliminates R’s memory constraint by enabling R to work transparently on database resident data

• Brings R users closer to Oracle Database by transparently leveraging in-database analytics via R

• Enables R users to leverage Oracle Database and Big Data Appliance for scalable R Analytics

• Enables integration of R scripts into enterprise production applications and OBIEE dashboards

• Fully leverages the latest R algorithms and models contributed

to the Comprehensive R Archive Network (CRAN)

• Integrated with Hadoop and HDFS – write map-reduce style R scripts

• Enables migration away from Base SAS and helps reduce SA$ Annual Usage Fees


18

Oracle R Distribution

• Features

– Supported by Oracle

– Optimized for high performance analytics using Intel MKL Library

• Platforms

– Windows 32-bit, 64-bit

– Linux 64-bit

• Integrated with

Oracle Enterprise Linux distribution


http://www.r-project.org/index.html




19

R Licensing and Distribution

• Open source R is licensed under GPL-v2

– http://www.gnu.org/licenses/gpl-2.0.html

• Oracle will redistribute open source R providing free support

for Oracle Database and Oracle Linux customers


http://www.gnu.org/licenses/gpl-2.0.html



20

Oracle R Enterprise

Function push-down – data transformation & statistics

R workspace console

Oracle statistics engine

OBIEE, Web Services

No changes to the user

experience

Scale to large data sets

Embed in operational

systems


http://search.oraclecorp.com/search/search


21

Oracle R Enterprise

Transparently leverage Hadoop for High Performance Analytics to Big Data Appliance

Function push-down – data transformation & statistics

R workspace console

Oracle statistics engine

OBIEE, Web Services


http://search.oraclecorp.com/search/search


22

Big Data Appliance

• An engineered system optimized for capturing

and integrating “low density” data into Exadata

• High-performance Hardware • Optimized for Hadoop and NoSQL workloads

• InfiniBand Networking for integration with Exadata

• Software: • Oracle Hadoop

• Oracle R Enterprise

• Oracle NoSQL DB

• Oracle Data Integrator (Hadoop capabilities)

• Oracle Loader for Hadoop

18 Sun X4270 M2 Servers

48 GB memory per node = 864 GB memory

12 Intel cores per node = 216 cores

24 TB storage per node = 432 TB storage

40 Gb p/sec InfiniBand

10 Gb p/sec Ethernet


23

BDA and Exadata

HDFS Hadoop

ORE ORHC

Load Data to Exadata

Analyze Data

Exadata

BDA

Oracle Database

ORE

R Engine


24

What R programmers should know…

Oracle R Enterprise…

• provides a familiar environment to operate on data in Oracle Database

• overloads base R functions for data in Oracle Database to address big

data as a single big data set problem

• enables embedded execution of existing R scripts in Oracle Database to

address big data

• enables embedding advanced analytics into production infrastructure

and business processes / applications

• provides database-controlled data-parallel execution framework

• removes need to manage data outside the database

• no knowledge of SQL required


25

What statisticians should know…


• expands the kinds of problems that can be solved using readily available

technologies through in-database statistical calculations

• Reduces the SQL skills required to get results

• reduces time programmers spend on building infrastructure to manage

data outside of Oracle Database

• reduces the complexity of analytic solutions to be placed into production

• reduces costs by replacing SAS, and its high licensing fees, in the lab

• provides exhaustive support for statistical techniques and functions

found in Base SAS and Base R

• implements select techniques from SAS libraries: STAT, ETS and OR


26

What a SQL user or DBA should know …


• is an advanced analytical engine that can be used in addition to SQL in

Oracle Database

• can be used to put an R script into production through SQL invocation

• reduces or eliminates the need to move data outside Oracle Database

• facilitates auditing of data flows

• reduces the number of LOB help requests for SQL queries to obtain

data of interest

• enables embracing high impact advanced analytic workloads to run

within production database infrastructure

• allows DBAs to control what R code runs within a deployment by

requiring all R code reside in a secure, audited database table


27

What is Oracle R Enterprise?

• R packages, database library, and SQL extensions that bring Oracle Database closer to Advanced Analytics

users in an Enterprise

– Transparency Layer

• Set of packages mapping R data types to Oracle Database objects

• Transparent SQL generation for R expressions on mapped data types

• Enables direct interaction with database-resident data using R language constructs

• Enables R users to work with data larger than desktop system memory

– Statistics Engine

• Set of statistical functions and procedures for commonly-used statistical libraries

• Executes in Oracle Database

– SQL extensions

• Enables database server execution of R code to facilitate embedding R in operational systems

• Eliminates client data loading and result write-back to Oracle Database

• Enables dataflow parallelism, generation of rich XML output, SQL access to R, and parallel simulations capability

– Hadoop Connector

• R packages that enable working directly with an Oracle Hadoop cluster using R

• Data resides in HDFS, Oracle Database ,or local files


28

Oracle R Enterprise Compute Engines

• R-SQL Transparency Layer intercepts R

functions for scalable in-db execution

• Function intercept for data transforms,

statistical functions and advanced analytics

• Interactive display of graphical results and

flow control as in standard R

• Submit entire R scripts for embedded

execution by Oracle Database

• Connect to Oracle Hadoop Cluster and

interact with HDFS data

• Scale to large datasets

• Access tables, views, and external

tables, as well as data through

DB LINKS

• Leverage database SQL parallelism

• Leverage new and existing

in-database statistical and data mining

capabilities

R Engine Other R packages

Oracle R Enterprise packages

User R Engine on desktop

• Database can spawn multiple R

engines for database-managed

parallelism

• Efficient data transfer to spawned

R engines

• Emulate map-reduce style algorithms

and applications

• Enables “lights-out” execution of R

scripts

1

User tables

Oracle Database SQL

Results

Database Compute Engine

2


Oracle R Enterprise packages

R Engine(s) spawned by Oracle DB

R

Results

3


29

Oracle R Enterprise – Data Sources

User tables

Oracle Database

Bulk import

File systems Other databases

Database Links

SQL Results


Oracle R package

R user on desktop

External Tables

File systems

Direct R access

Other databases

Direct R access

RODBC, DBI, etc


OLAP

cubes

30

Installing R


31

Installing R

• Go to http://cran.r-project.org/ and choose

Linux, Mac or Windows

• If Windows

– Click “install R for the first time”

– Click “previous releases”

– Click “Download R 2.13.2 for Windows”

• If Linux

– Click “R Sources” under software

– Click “R-2.13.2.tar.gz” to download

– Build R on your Linux system

• If Mac

– …note ORE not supported on Mac

• Repeat the same process to install R on Oracle Database node(s)

• For more info than you’d want, see http://cran.r-project.org/doc/manuals/R-admin.html


http://cran.r-project.org/



32 ©2011 Oracle – All Rights Reserved

33

Other R GUIs, e.g., installing RStudio

• Go to http://rstudio.org/

• Click

• Click for desktop

• Consider server install if you run R on a Linux server and

want to enable users to remotely

access RStudio using a web browser

• Installation instructions online


http://rstudio.org/

34

Running R on Windows

• Double click the desktop icon

• Run from C:\Program Files\R\R-2.13.2\bin\x64\Rgui.exe

• At the command prompt ‘R>’

– help.start() brings up help in your default browser

– plot(mtcars) get a quick pairs plot of the variables in mtcars

• Quit R

– quit()

– Or use: File Exit

– Save workspace image?


35

Installing R on Linux

• Download R-2.13.2 source tar file from CRAN

– http://cran.r-project.org/

• Untar the file

– tar -zxvf R-2.13.2.tar.gz

• Change directory to R-2.13.2

– cd R-2.13.2

• Configure for shared library generation

– ./configure --with-ICU=no --enable-R-shlib

– make


36

Setting Environment Variables on Linux

• Set R_HOME environment variable to R root installation

– If RPM-based install using Oracle R Distribution setenv R_HOME /usr/lib64/R

– If manual R installation setenv R_HOME <dir-where-ran-make>

• Set PATH environment variable

– Add $R_HOME/bin to PATH

– If RPM-based instlal using Oracle R Distribution

setenv PATH /usr/bin:$PATH

– If manual R installation

setenv PATH <dir-where-ran-make>/bin:$PATH


37

Running R on Linux

• Run R

– $R_HOME/bin/R, or just R

– For the GUI, R --gui=Tk

• At the command prompt ‘R>’

– help.start() brings up help in your default browser

– plot(mtcars) get a quick pairs plot of the variables in mtcars

• Quit R

– quit()

– Or use: Cntrl-D Cntrl-D

– Save workspace image?


38

Installing Oracle R Enterprise


39

Supported Configurations

• Oracle Database 11.2 series on Oracle Linux 5, 64 bit

– Exadata machines

– Oracle Database

• Oracle R Enterprise client

– Windows XP/Vista/7 32 bit or 64 bit

– Oracle Enterprise Linux 64 bit

• Hadoop Connector only deployed on Oracle Hadoop cluster

• See ORE Beta Release Notes


https://stbeehive.oracle.com/content/dav/st/Oracle_R_Enterprise_PM_Workspace/Public Documents/Oracle R Enterprise Software Download/Oracle R Enterprise Installation Setup and BETA Release Notes.pdf

40

ORE Installation steps

• Install Oracle Database 11.2 series

• Install R

– ORE certified with R-2.13.2

• Install ORE

• Set up Rprofile


41

Prerequisites for ORE

• Oracle Database 11.2 series

– Version 11.2.0.3, or

– If using 11.2.0.1 or 11.2.0.2, must include patch that fixes Bug Number 11678127

• R installed on…

– user’s desktop to connect to and work with Oracle Database

– Oracle Database node(s) used as database-embedded R engine

• Two R package must be installed

– Download DBI: http://cran.r-project.org/web/packages/DBI/index.html or R> install.packages(“DBI”)

– Download PNG: http://cran.r-project.org/web/packages/png/index.html or R> install.packages(“png”)


http://cran.r-project.org/web/packages/DBI/index.html



http://cran.r-project.org/web/packages/png/index.html



42

Database Patches

• Download the 11.2.0.3 patch set for Enterprise Edition

from My Oracle Support

OR

• Linux 64-bit

– For Oracle 11.2.0.1.0: Patch number 12598677 for Linux x86-64

– For Oracle 11.2.0.2.0: Patch number 12976544 for Linux x86-64

• Windows XP 32-bit

– For Oracle 11.2.0.2 Patch 11 fixes Bug Number 11678127

• Windows 7 32-bit and 64-bit

– For Oracle 11.2.0.2 Patch 11 fixes Bug Number 11678127


http://support.oracle.com/

43

Applying the Database Patch for Linux

• Install the patch to Oracle Database, if needed

cd DBpatch

setenv PATH $ORACLE_HOME/OPatch:$PATH

unzip -d . p11678127_112010_Linux-x86-64.zip

cd 11678127

opatch apply


44

Installing the ORE Package

• From R command line for both ORE client and database server

– Choose URL appropriate for platform

R

R> remove.packages(c("ORExml", "ORExml", "OREeda",

"OREgraphics", "OREstats", "OREbase", "ORE", "ROracle"))

R> install.packages("ORE", repos =

"http://adc2100203.us.oracle.com:8080/CRAN")

• Check package installation by invoking… R> library(ROracle)

Note: installing ORE package on database server is only

required if planning to use embedded R execution


45

Installing Oracle Instant Client on Linux

• Obtain the RPM files – oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm

– oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

– http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

• Install the RPMs sudo rpm –ivh oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm

sudo rpm –ivh oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm


46

Installing ORE Server Components on Windows

• Download ORE server components from location specified when you

enrolled in the Beta test

– Internal: https://stbeehive.oracle.com/teamcollab/overview/Oracle+R+Enterprise+Beta+Software+Download

rqe_install_win.zip

• Unzip the download into an empty directory, e.g., C:\rqe_install_win


https://stbeehive.oracle.com/teamcollab/overview/Oracle+R+Enterprise+Beta+Software+Download

https://stbeehive.oracle.com/content/dav/st/Oracle R Enterprise Beta Software Download/Documents/Windows Binaries/rqe_install_win.zip

47

Set Windows Environment Variables

for ORE server

• Set R_HOME to the directory where R is installed

– for example, C:\Program Files\R\R-2.13.2, if you installed R in the default location

• Set R_PROFILE_USER to the location where the Rprofile is stored

– Default value is C:\Program Files\R\Rprofile

– This can be changed

• Bounce the database

– As sysdba in SQL*Plus execute…

shutdown immediate

startup


48

Running the Windows install script for ORE server

• In a Command Window, change directory to where downloaded file unzipped

• Change directory to the server folder

• Run install.bat

• Change directory to data folder

• Run install.bat

• The installation scripts automatically create…

– User account rquser (password rquser) in your database

Note: If an account named rquser already exists, rename the account

– Demo tables in the database

– Rprofile file which tells R to connect to a specific database when starting R


49

Location of Windows Installed Files

• Client-side packages in $R_HOME/library

– Database Connectivity packages: DBI, ROracle

– ORE Client: ORE, OREbase, OREeda, OREgraphics, OREstats, ORExml

– ORE Hadoop client: ORHC

• Server-side (database)

– ORE libraries in $ORACLE_HOME/bin

– ore.dll and rqe.dll


50

Start R on Windows with ORE

• Click desktop icon, or

• Click, e.g., C:\Program Files\R\R-2.13.1\bin\i386\Rgui.exe

• Type ore.ls() to see

a list of available tables

Confirms database connectivity


51

.Rprofile

options(prompt='ORE> ')

options(width=132)

.First <- function()

{

# Load Oracle R libraries.

library(ORE)

# Database credentials:

user = “my_schema“

password = “my_password“

sid = “MY_ORACLE_SID"

host = “MY_DB_HOST"

cat("Connecting to ORE\n")

cat(" User:", user, "\n")

cat(" SID: ", sid, "\n")

cat(" Host:", host, "\n")

ore.connect(user, sid, host, password)

ore.sync()

ore.attach()

cat("Connected.\n")

}


52

Installing ORE Server Components on Linux

• Download ORE from the location specified when you

enrolled in the Beta test

• Unzip the download into an empty directory, e.g.,

~/OREbeta

• Install database server-side components

– Log in as user oracle

cd server

./install.sh


53

Location of Linux Installed Files

• Client-side packages in in $R_HOME/library

– Database Connectivity packages: DBI, ROracle

– ORE Client: ORE, OREbase, OREeda, OREgraphics, OREstats, ORExml

– ORE Hadoop client: ORHC

• Server-side (database)

– ORE libraries in $ORACLE_HOME/lib

– ore.so and librqe.so


54

Enable and Check Database Connectivity

• Enable database connectivity on R startup

– Copy provided Rprofile file to the home directory

cp Rprofile ~/.Rprofile

– Set R_PROFILE USER environment variable

setenv R_PROFILE_USER ~/.Rprofile

• Bounce the database, as sysdba execute

shutdown immediate

startup

• Check database connectivity

– Start R

– Execute ore.ls()


55

Sample Code

• ORE shipped with examples illustrating how to use ORE

– A collection of self-contained R scripts

• Scripts located in <unzip-directory>/samples.tar

– Where <unzip-directory> is the directory where you

unzipped the ORE download

– Unzip samples.tar


56

Troubleshooting

• See log.txt file

– created in directory where you executed install program

– View file to review errors

• Install program can be re-executed after correcting errors


57

Uninstalling ORE

Client Components

• Start R and invoke

remove.packages(“OREeda”)

remove.packages(“OREgraphics”)

remove.packages(“OREstats”)

remove.packages(“OREbase”)

remove.packages(“ORExml”)

remove.packages(“ORE”)

remove.packages(“ROracle”)

remove.packages(“DBI”)

• Unset environment variable unsetenv R_PROFILE_USER

Database Server Components

• As sysdba, execute in SQL*Plus

SQL> drop user rqsys cascade;

• On linux command line, execute

rm $ORACLE_HOME/bin/ore.so

rm $ORACLE_HOME/bin/librqe.so


58

Connecting to Exadata (Oracle Database) from R


59

Connecting to Exadata (Oracle Database) from R

• You’ve already done it

by starting R with Rprofile


60

Oracle R Enterprise Initial connection and basic operations

R> ore.connect("rquser", "orcl", "machine-1")

R> ore.sync("rquser")

R> ore.attach ("rquser")

R> ore.ls()

R> names(ONTIME_S)

R> dim(ONTIME_S)

Implicitly invoked through R_PROFILE_USER

Sync database objects with R client, e.g., add/remove tables/views added via SQL*Plus)

List tables visible in schema rquser as ore.frame objects

List columns associated with table ONTIME_S

View the number of rows and columns associated with this ore.frame

Add schema objects in R env search path


61

Working with ORE Connections

ore.connect("RQUSER","SID","HOST","PASSWORD",1521)

ore.create( ONTIME_S, table = "NEW_ONTIME_S")

ore.create( ONTIME_S, view = "NEW_ONTIME_S_VIEW")

ore.drop(table="NEW_ONTIME_S")

ore.drop(view="NEW_ONTIME_S_VIEW")

v <- ore.push(c(1,2,3,4,5))

ore.sync()

ore.sync("RQUSER")

ore.sync(table=c("ONTIME_S", "NARROW"))

ore.sync("RQUSER", table=c("ONTIME_S", "NARROW"))

ore.exists("ONTIME_S", "RQUSER")

Connect to a specific schema and database

One connection active at a time

Create a database table from a data.frame, ore.frame. Create a view from an ore.frame.

Drop table or view in database

Store R object in database as temporary object, returns handle to object. Data frame, matrix, and vector to table, list/model/others to serialized object

Synchronize ORE proxy objects in R with tables/views available in database, on a per schema basis

Returns TRUE if named table or view exists in schema


62

Working with ORE Connections ore.ls()

ore.ls("RQUSER")

ore.ls("RQUSER",all.names=TRUE)

ore.ls("RQUSER",all.names=TRUE, pattern= "NAR")

t <- ore.get("ONTIME_S","RQUSER")

ore.attach("RQUSER")

ore.attach("RQUSER", pos=2)

ore.detach("RQUSER")

ore.rm("DF1")

ore.rm(list("TABLE1","TABLE2"), "RQUSER")

ore.disconnect()

ore.exec("create table F2 as select * from ONTIME_S")

List the objects available in ORE environment mapped to database schema.

All.names=FALSE excludes names starting with a ‘.’

Obtain object to named table/view in schema.

Make database objects visible in R for named schema. Can place corresponding environment in specific position in env path.

Remove schema’s environment from the object search path.

Remove table or view from schema’s R environment.

Disconnect from the database. Clean up all associated R objects and temporary database schema objects


Execute SQL or PL/SQL without return value

63

Demonstration

• Connecting to ORE

• Manipulating database objects


64

Wrapping up…

• What’s Oracle R Enterprise

• Install R and ORE

• Interacting with Oracle Database through R


65 ©2011 Oracle – All Rights Reserved

66