Accessing and Loading Data

17Copyright © 2004, Oracle. All rights reserved.

Accessing and Loading Data

17-2 Copyright © 2004, Oracle. All rights reserved.

Objectives

After completing this lesson, you should be able to do the following:

• Use Data Pump to import and export data

• Create external tables for data population

• Transport tablespaces across different platforms


Data Pump: Overview

• Server-based facility for high-speed data and metadata movement

• Infrastructure callable via DBMS_DATAPUMP• Provides new-generation export and import tools:

– expdp– impdp– Web-based interface through Enterprise Manager

• Data Pump Export and Import are supersets of the original Export and Import utilities.


Data Pump: General Architecture

External Table API

ORACLELOADER

ORACLEDATAPUMP

Direct Path API Metadata API

expdp impdp Other clients

Data Pump

4DBMS_DATAPUMPData/Metadata Movement Engine

SQLLoader


Data Pump Export and Import: Overview

expdpclient

Dumpfile set

Database

Data Pumpjob

Source

Mastertable

Serverprocess

Dumpfile set

Serverprocess

Target

Data Pumpjob

impdpclient

Database

Mastertable

Databaselink


Data Pump Export and Import: Benefits

• Data access methods:– Direct path– External tables

• Detach from and reattach to long-running jobs

• Restart Data Pump jobs

• Fine-grained object selection

• Explicit database version specification

• Parallel execution (Enterprise Edition only)

• Estimate export job space consumption

• Network mode in a distributed environment

• Remapping capabilities during import


Data Pump Access Methods

Data Pump supports two access methods:

• Direct-path load using the direct-path API

• External tables

Externaltables

Directpath

Database

Database


Data Pump Direct Path Considerations

External tables are used under the following conditions:

• Tables with fine-grained access control enabled in insert and select modes

• Domain index exists for a LOB column.

• Clustered tables are present.

• Tables with active triggers defined

• Global index on partitioned tables with a single-partition load

• BFILE or opaque type columns

• Referential integrity constraint

• VARRAY columns with an embedded opaque type


Data Pump File Locations

• Three types of Data Pump files:– Dump files– Log files– SQL files

• Absolute paths are not supported.

• Oracle directory objects must be used.

• Order of precedence of file locations:– Per-file directory– DIRECTORY parameter– DATA_PUMP_DIR environment variable


Data Pump File Naming and Size

• A dump file set can contain more than one file.

• DUMPFILE determines the list of dump files:– Comma-separated list of files– %U template

• The initial number of dump files depends on:– PARALLEL parameter– DUMPFILE parameter

• The FILESIZE parameter determines the size of each dump file.

• Preexisting files with matching names are not overwritten.


Data Pump Utility: Interfaces and Modes

• Data Pump Export and Import interfaces:– Command-line– Parameter file– Interactive command-line– Database Control

• Data Pump Export and Import modes:– Full– Schema– Table– Tablespace– Transportable tablespace


Data Pump and Filtering

• Fine-grained object selection:– INCLUDE = object_type[:"name_expr"]– EXCLUDE = object_type[:"name_expr"]

• Data selection:– CONTENT = ALL | METADATA_ONLY | DATA_ONLY– QUERY = [schema.][table_name:]"query_clause"

QUERY=hr.employees:"WHERE department_id in (10,20) and salary < 1600 ORDER BY department_id"

EXCLUDE=VIEWEXCLUDE=PACKAGEEXCLUDE=INDEX:"LIKE 'EMP%'"


Data Pump Import Transformations

• You can remap:– Data files using REMAP_DATAFILE– Tablespaces using REMAP_TABLESPACE– Schemas using REMAP_SCHEMA

•

• Using TRANSFORM, you can exclude from tables and indexes:– STORAGE and TABLESPACE clauses– STORAGE clause only

TRANSFORM = SEGMENT_ATTRIBUTES|STORAGE:{y|n}[:TABLE|INDEX]

REMAP_DATAFILE = 'C:\oradata\tbs6.f':'/u1/tbs6.f'


Data Pump Job Monitoring Views

DBA_DATAPUMP_JOBS

DBA_DATAPUMP_SESSIONS

V$SESSION

SADDR

OWNER_NAMEJOB_NAME

V$SESSION_LONGOPS

SIDSERIAL#

Master

Process

Client

processes

USERNAMEOPNAME


Database Control and Data Pump


Parallel Full Export and Import: Example

$ expdp system/manager full = y parallel = 4 dumpfile = DATADIR1:full1%U.dat, DATADIR2:full2%U.dat, DATADIR3:full3%U.dat, DATADIR4:full4%U.dat filesize = 2G

$ impdp system/manager directory = NET_STORAGE_1 parallel = 4 dumpfile = full1%U.dat,full2%U.dat, full3%U.dat,full4%U.dat


Limited Schema Export: Example

$ expdp system/manager schemas = hr,oe directory = USR_DATA dumpfile = schema_hr_oe.dat parfile = exp_par.txt

include = functioninclude = procedureinclude = packageinclude = typeinclude = view:"like 'PRODUCT%'"

$ impdp system/manager directory = USR_DATA dumpfile = schema_hr_oe.dat sqlfile = schema_hr_oe.sql


Network Mode Import: Example

$ impdp system/manager schemas = hr,sh,payroll parfile = imp_par.txt

network_link = finance.hq.com

flashback_time = 2003-09-08 09:00

remap_schema = payroll:finance


Attaching to Existing Job: Example

$ expdp system/manager ATTACH = EXP_TS1 ... Job: EXP_TS1 Owner: SYSTEM ... Mode: TABLESPACE MaxDegree: 2 EXPORT Job Parameters: Parameter Name Parameter Value: CLIENT_COMMAND system/**** estimate=blocks ... DATA_ACCESS_METHOD AUTOMATIC ... State: EXECUTING Job error count: 0 ...Export> STOP_JOBAre you sure you wish to stop this job ([y]/n): Y


Restarting Stopped Job: Example

$ expdp system/manager attach=exp_ts1…Export> parallel = 4Export> start_job…Export> status = 600Export> continue_client


Data-Only Unload: Example

$ expdp hr/hr parfile=exp_par.txt

directory=HRDATAdumpfile=expdat.dmpcontent=data_onlyinclude=table:"in ('DEPARTMENTS','DEPARTMENTS_HIST','EMPLOYEES','EMPLOYEES_HIST')"query="where DEPARTMENT_ID != 30 order by DEPARTMENT_ID"


External Table Population: Overview

• Unload data to external flat files

• Handle complex ETL situations

Flat files(Proprietary format)

CREATE TABLE… AS SELECT

Tables

Unloading

Tables

Loading

INSERT … SELECT


External Table Population Operation

• Uses the ORACLE_DATAPUMP access driver

• Data cannot be modified.

• Resulting files can be read only with the ORACLE_DATAPUMP access driver.

• You can combine generated files from different sources for loading purposes.

ORACLE_DATAPUMP

Oracle database

DPAPIflat files


External Table Parallel Populate Operation

• Multiple files can be created.

• Exactly one parallel execution server per file

• The PARALLEL and LOCATION clauses influence the degree of parallelism.

Coordinator

Parallelexecution

servers

Generatedfiles

emp1.exp emp2.exp emp3.exp


External Table Population: Example

CREATE TABLE emp_ext (first_name, last_name, department_name)ORGANIZATION EXTERNAL ( TYPE ORACLE_DATAPUMP DEFAULT DIRECTORY ext_dir LOCATION ('emp1.exp','emp2.exp','emp3.exp') )PARALLELASSELECT e.first_name,e.last_name,d.department_nameFROM employees e, departments dWHERE e.department_id = d.department_id AND d.department_name in ('Marketing', 'Purchasing');


External Table Projected Columns

2355,1,2289,46,2002355,A,2264,50,100

SELECT COUNT(order_id)FROM order_items_ext;

SELECT COUNT(line_id)FROM order_items_ext;

order_items1.dat

REFERENCED ALLAccessdriver

2 1 1 1


ALTER TABLE order_items_extPROJECT COLUMN {ALL|REFERENCED};

SELECT propertyFROM DBA_EXTERNAL_TABLESWHERE table_name = 'ORDER_ITEMS_EXT';

External Table Projected Column: Examples

• The default value is ALL.

• REFERENCED is useful for performance when data is known to be safe.


Cross-Platform Transportable Tablespaces

• Simplify data distribution between data warehouse and data marts

• Allow database migration from one platform to another

• Supported platforms:

Solaris[tm] OE (32-bit) HP-UX (64-bit) Microsoft Windows IA (64-bit)

Solaris[tm] OE (64-bit) HP Tru64 UNIX IBM zSeries Based Linux

Microsoft Windows IA (32-bit) HP-UX IA (64-bit) Linux 64-bit for AMD

Linux IA (32-bit) Linux IA (64-bit) Apple Mac OS

AIX-Based Systems (64-bit) HP Open VMS Microsoft Windows 64-bit for AMD


Minimum Compatibility Level

• Both source and target databases must have COMPATIBLE set to 10.0.0 or higher.

• Data file headers are platform-aware.

• Before transporting, make sure that all read-only and offline files are platform-aware.

Read-only

Read-only

Read/write

Read-only

COMPATIBLE=10.0.0COMPATIBLE=9.2.0

Read/write

Read/write


Transportable Tablespace Procedure

Make tablespaces read-only.

Yes

Target uses the same endian format?

Use Data Pump to extract metadata.

Convert data files using RMAN.

Ship data files and dump file to target.

Use Data Pumpto import metadata.

Make tablespaces read/write.

No

Source

Target


Determining the Endian Formatof a Platform

SELECT tp.endian_formatFROM v$transportable_platform tp, v$database dWHERE tp.platform_name = d.platform_name;

Source Target


Data File Conversion: Examples

$ rman target=/RMAN> CONVERT TABLESPACE 'FINANCE,HR' TO PLATFORM = 'AIX-Based Systems (64-bit)' DB_FILE_NAME_CONVERT = '/orahome/dbs1', '/orahome/dbs/transport_aix', '/orahome/dbs2', '/orahome/dbs/transport_aix';

$ rman target=/RMAN> CONVERT DATAFILE '/tmp/transport_stage/*' FROM PLATFORM = 'Solaris[tm] OE (32-bit)' DB_FILE_NAME_CONVERT ='/tmp/transport_stage/fin' , '/orahome/dbs1/fin', '/tmp/transport_stage/hr' , '/orahome/dbs2/hr';

Source

or

Target


CLOB Exception

Source Target

CLOB

Convert Transport

CLOB CLOB CLOB

Read

Endian- dependentCLOBs

Automaticendian-independent

conversion


Dynamic Performance View Changes

V$DATABASE

V$TRANSPORTABLE_PLATFORM

PLATFORM_IDPLATFORM_NAME

ENDIAN_FORMAT


Summary

In this lesson, you should have learned how to:

• Use Data Pump to import and export data

• Create external tables for data population

• Transport tablespaces across different platforms


Practice 17 Overview: Using Data Pump

This practice covers using Data Pump to manage data.

Documents

Accessing and Loading Data