Recipe 14 of Data Warehouse and Business Intelligence - Build a Staging Area for an Oracle Data...

Single meal

Designing, building, loading and managing a Staging Area for an Oracle

Data Warehouse. Meal ready in 20 minutes.

Price: completely free(without loading products and without writing a line of code)

Recipes of Data Warehouse and Business Intelligence

Chef: Massimo Cenci (part 1)

• The data file to be loaded. As an example, we use the data file that describes the world's financial markets downloaded from the site iso15022.org.

• A document describing the file structure. The information on its structure are described in the FAQ on the site.

• The Micro ETL Foundation (MEF), that is, the ecosystem of facilities and processing modules that will connect all the ingredients to obtain the final plate.

• A text editor • Command Prompt to access in SQL*Plus to the Oracle Database

Ingredients

Chef: Massimo Cenci

Preparation

1. Decide a code of three characters for the project (for example dwh), open a DOS prompt and create a folder with the same name as the project (for example d: \ projects\dwh).

2. Download MEF (from https://drive.google.com/open?id=0B2dQ0EtjqAOTQzZSaUlyUmxpT1k) and unpack it in the folder \dwh.

3. Go under the folder \dwh\mef and open the mef_def.sql. In this configuration file you must set some environment variables needed to the Data Warehouse environment that will be created: the ones you see below, are just an example for Windows. For the moment only change the red variables according to your environment. (Part 2 of this article will provide more details)

Chef: Massimo Cenci

Preparation

define prj = 'dwh' -- Project code define usr = 'etl' -- ETL user namedefine pwd = 'etl' -- ETL user passworddefine frm = 20120101 -- Start day for day dimension tabledefine ndy = 3660 -- Num days define dla = 'ITA' -- Default language for descriptionsdefine dev = '?' -- Default value for varchar2define den = 0 -- Default value for numberdefine ded = '99991231' -- Default value for daydefine srv = '10.84.253.61' -- Email server (name or ip)define fss = '\' -- File system separator (Windows)--define fss = '/' -- File system separator (Linux)define prt = 'd:\projects' -- Physical path of the project root (Windows)--define prt = '/u01/projects' -- Physical path of the project root (Linux)

Chef: Massimo Cenci

Preparation4. Open a dos prompt under \dwh\mef, connect to Oracle as sysdba and run the

script mef_install.sql. This installs the MEF ecosystem and will create the etl user

E:\projects\dwh\mef> sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Mon Feb 8 09:29:16 2016Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> @mef_install

Chef: Massimo Cenci

Preparation5. Open with an editor the mef config.sql script and change the email addresses of

Massimo Cenci with yours.6. Connect to the database with the etl user and install the core environment by

running the script mef_core_install.sql

E:\projects\dwh\mef> sqlplus etl/etl

SQL*Plus: Release 12.1.0.1.0 Production on Mon Feb 8 09:29:16 2016Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> @mef_core_install

Chef: Massimo Cenci

Preparation7. Install the staging environment through the script mef_sta_install.sql.

E:\projects\dwh\mef> sqlplus etl/etl

SQL*Plus: Release 12.1.0.1.0 Production on Mon Feb 8 09:29:16 2016Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> @mef_sta_install

Chef: Massimo Cenci

Preparation8. Run the configuration script for the data file that we will code as MIC. The

download of the data file and the information about its structure have already been setted.

E:\projects\dwh\mef> sqlplus etl/etl

SQL*Plus: Release 12.1.0.1.0 Production on Mon Feb 8 09:29:16 2016Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> @sta_conf_io.sql MIC

Chef: Massimo Cenci

Cooking1. Run the job to load the data file

E:\projects\dwh\mef> sqlplus etl/etl

SQL*Plus: Release 12.1.0.1.0 Production on Mon Feb 8 09:29:16 2016Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exec mef_job.p_run('sta_iso_mic');

Chef: Massimo Cenci

Tasting

• Verify that the Staging Area table is loaded. We can use, for example, SQLDeveloper

Chef: Massimo Cenci

Tasting• Check that the loading was successful and to have received the result also via email.

Chef: Massimo Cenci

Conclusion• With the Micro ETL Foundation, you can design, create and manage a Staging Area

using just a text editor and the database internal language.• The loading of a simple data file does not require more than 20 minutes. We verify the

times:

• Time to configure the features of the data file: 5 minutes• Time to configure the structure of the data file: 15 minutes• Running time: 3 seconds (see log)• Total time: 20 minutes

Chef: Massimo Cenci