Informatica Server Manager

Informatica

Server

Manager

Last Updated : 01/12/00

Center of ExcellenceCenter of ExcellenceData WarehousingData Warehousing

Server Manager ArchitectureServer Manager Architecture

After this session you should be able to describe

How to Configure Server ManagerHow to Configure Server Manager Creating Sessions and BatchesCreating Sessions and Batches Informatica’s session outputInformatica’s session output Difference in Sequential & Concurrent BatchesDifference in Sequential & Concurrent Batches Loading OptionsLoading Options Different Ways to Launch a SessionDifferent Ways to Launch a Session Configure Post- Session EmailConfigure Post- Session Email Monitor Status of Sessions and BatchesMonitor Status of Sessions and Batches How to load Rejected Data with pmrjldrHow to load Rejected Data with pmrjldr How to Recover a Stopped SessionHow to Recover a Stopped Session

Transformation Process

Source DefSource DefMappingMapping

Target Def.Target Def.SessionSession

Source InformationSource InformationTarget InformationTarget Information

MappingMappingSchedulingScheduling

Error handlingError handlingpre/post session scriptspre/post session scripts

Tuning parametersTuning parametersOutput log informationOutput log information

Transformation overridesTransformation overrides

ServerServer

RepositoryRepository

Server Manager

Process Involved

Configure ServerConfigure Server

Create SessionCreate Session

Run SessionRun Session

Monitor SessionMonitor Session

View LogsView Logs

Tune SessionTune Session

Server Configuration

Native database ConnectionNative database Connection– Used to connect to source and target databasesUsed to connect to source and target databases– Connection name, user ID, password, connection string, Connection name, user ID, password, connection string,

database typedatabase type

Register ServerRegister Server– TCP/IP host name / IP address, port # or dotted IPX/SPX TCP/IP host name / IP address, port # or dotted IPX/SPX

addressaddress

Log files pathLog files path– Server path to write session logServer path to write session log– Server path to write session errorsServer path to write session errors

You may also need to :You may also need to :– FTP Properties if Source/target files will be sent to server by FTP Properties if Source/target files will be sent to server by

FTP FTP – Create External Loader Connections like Sybase IQCreate External Loader Connections like Sybase IQ

Server Output

.perf.perf

.control.control

.dat.dat

.Email.Email

.Error Log.Error Log

.Event Log.Event Log

.log.log

.bad.bad

.idx.idx

ServerServerTargetTarget DatabaseDatabase

Session Configuration Wizard

The Session Manager provides a Session Wizard for The Session Manager provides a Session Wizard for easyeasy

Session Configuration.It has four primary pages :Session Configuration.It has four primary pages : GeneralGeneral

– Source SettingSource Setting– Target SettingTarget Setting– Pre and Post Session CommandPre and Post Session Command– Performance and Advance OptionsPerformance and Advance Options

TimeTime Log Files Log Files TransformationTransformation

Session Configuration Wizard Contd.…

Source SettingSource Setting


Target SettingTarget Setting


Working With FilesWorking With Files

– Source and Target FilesSource and Target Files

– Using Multiple Source FilesUsing Multiple Source Files

– FTP Source and Target FilesFTP Source and Target Files

– External LoaderExternal Loader


File Source File Source – Delimited File Delimited File – Fixed WidthFixed Width

Heterogeneous SourceHeterogeneous Source– Two or more tables existing in different DatabaseTwo or more tables existing in different Database– Two or more flat files existing in different SystemTwo or more flat files existing in different System– Two or more ODBC SystemsTwo or more ODBC Systems– Relational and flat file sources in one mappingRelational and flat file sources in one mapping

File Targets (Output Files)File Targets (Output Files)– Delimited FilesDelimited Files– Fixed widthFixed width

External LoaderExternal Loader


External Loading External Loading

– Increase Session Performance Increase Session Performance

– Informatica Provides two options Informatica Provides two options

• Configure through Session ManagerConfigure through Session Manager

– Oracle , SybaseOracle , Sybase

– Creates control file and starts process to call external loaderCreates control file and starts process to call external loader

• Create control file with Control File WizardCreate control file with Control File Wizard

– Oracle , Sybase, teradataOracle , Sybase, teradata

– Creates control file onlyCreates control file only


External Loader Process External Loader Process

NT FileNT File

Unix PipeUnix Pipe

.bad file.bad file

External Loader ProcessExternal Loader Process

DB Reject FileDB Reject File

Control FileControl File

Informatica ServerInformatica Server

Transformation - Session Wizard

AttributesAttributes– Override transformation attributes in mappingOverride transformation attributes in mapping

ExamplesExamples– Change source qualifier filter statementChange source qualifier filter statement

– Tune SQL statementsTune SQL statements

– Enable lookup cachingEnable lookup caching

Aggregate BehaviorAggregate Behavior– Specify Incremental aggregationSpecify Incremental aggregation

– Reinitialize CacheReinitialize Cache

Session Commands - Session Wizard

Pre - Session CommandsPre - Session Commands

– Specify operating system commands that need to Specify operating system commands that need to

be run on the server when the session startsbe run on the server when the session starts

– Pre-session shell commands/.bat file should return Pre-session shell commands/.bat file should return

0 for success and non-zero(any positive) for failure0 for success and non-zero(any positive) for failure

– Good for dropping/truncating tables, deleting .bad Good for dropping/truncating tables, deleting .bad

filesfiles

Session Commands - Session Wizard

Post - Session CommandsPost - Session Commands

– Specify any operating system commands that need Specify any operating system commands that need

to be run on the server when the session endsto be run on the server when the session ends

– Good for rebuilding indexes and dropping Good for rebuilding indexes and dropping

temporary tablestemporary tables

– Can specify different scripts for success and failure Can specify different scripts for success and failure

of session. of session.

E-mail Configuration

Used to send e-mail message after a session runUsed to send e-mail message after a session run– VariablesVariables

• %s : session name%s : session name• %l : total rows loaded%l : total rows loaded• %r : total rows rejected%r : total rows rejected• %e : session error status%e : session error status• %t : table details%t : table details• %b : session start time%b : session start time• %g : attaches the session log to the message%g : attaches the session log to the message

– format tagsformat tags• \t : tab\t : tab• \r : new line\r : new line

Dependent on success and failure tabsDependent on success and failure tabs Always check the session log for more messageAlways check the session log for more message

Process for NTProcess for NT– Must use Microsoft Exchange e-mail systemMust use Microsoft Exchange e-mail system– Exchange client software must be installed on the Exchange client software must be installed on the

serverserver– Exchange service must use network securityExchange service must use network security– User starting Power Mart service must be global User starting Power Mart service must be global

user. Determine the power mart startup account by user. Determine the power mart startup account by selectingselecting• Control panel |servicesControl panel |services• Power mart servicesPower mart services

– Create mail box for the user starting the power mart Create mail box for the user starting the power mart serviceservice

– Create profile for user starting the power mart Create profile for user starting the power mart serviceservice

E-mail Configuration Contd..

E-mail Configuration Contd...

The profile must contain only the following servicesThe profile must contain only the following services– MS Exchange ServerMS Exchange Server

– Personal address bookPersonal address book

Add post-session e-mail recipients to the personal Add post-session e-mail recipients to the personal

address book.address book.

In UnixIn Unix– Unix server uses mail to send post-session e-mailUnix server uses mail to send post-session e-mail

– User who start the pmserver process should have the User who start the pmserver process should have the

mail in their PATH.mail in their PATH.

Launching Sessions

Manual StartManual Start– Manually launch a session from within the Server Manually launch a session from within the Server

ManagerManager Session ScheduleSession Schedule

– Schedule a session using business cycle start,stop Schedule a session using business cycle start,stop & repeat internal & repeat internal

BatchingBatching– Use batches to run session concurrently or Use batches to run session concurrently or

SequentiallySequentially Event Based Event Based

– Configure a session to launch based upon the Configure a session to launch based upon the appearance of an indicator fileappearance of an indicator file

Command Line Command Line – Launch a session from the command line promptLaunch a session from the command line prompt

Launching Sessions

Privileges/Permissions needed to run sessions Privileges/Permissions needed to run sessions

– Create Sessions & Batches privilege with read Create Sessions & Batches privilege with read

and execute sessions and execute sessions

– Sessions Operator privilege with read permissionSessions Operator privilege with read permission

– Super user privilegeSuper user privilege

Time - Session Wizard

Informatica provides business

cycle-based start

date,stop and repeat intervals

for running sessions

Scheduling Batches

Nest sessions and batches to run concurrent and Nest sessions and batches to run concurrent and

sequential , or a combination of the twosequential , or a combination of the two

Batch scheduler has same scheduling options as Batch scheduler has same scheduling options as

session wizardsession wizard

Types of batches Types of batches – Concurrent Concurrent

– SequentialSequential

Scheduling Tips

Load order - Load dependent tables firstLoad order - Load dependent tables first

Run Sessions in Parallel Run Sessions in Parallel – Run concurrently in batchesRun concurrently in batches– Effective with SMP systems or parallel databases Effective with SMP systems or parallel databases – Caution not to overload target or hardware platform Caution not to overload target or hardware platform

Load one target in parallelLoad one target in parallel

Load multiple targets in parallelLoad multiple targets in parallel

Event - Based Scheduling

Configure event-based schedulingConfigure event-based scheduling– Create shell script to send indicator file to directory Create shell script to send indicator file to directory

local to the Informatica Serverlocal to the Informatica Server

– Enter path and file name in the advanced propertiesEnter path and file name in the advanced properties

– Schedule the sessionSchedule the session

Session Log ImpactSession Log Impact– The time the file appearsThe time the file appears

– Deletion of the fileDeletion of the file

Monitor Session Monitor Session – When the watcher is in wait, status = file waitWhen the watcher is in wait, status = file wait

PMCMD - Command Line Interface

Can be used with 3rd party scheduling software, Can be used with 3rd party scheduling software,

scripts or program to give more more versatility in scripts or program to give more more versatility in

running sessionsrunning sessions– E.g. CA7, TivoliE.g. CA7, Tivoli

Must have proper Informatica permissions and Must have proper Informatica permissions and

privileges to take below actionsprivileges to take below actions ActionsActions

– Ping ServerPing Server– Start and Stop Sessions and batchesStart and Stop Sessions and batches

Server returns success or failure messagesServer returns success or failure messages Type DEBUG after the command for more feedbackType DEBUG after the command for more feedback

PMCMD - Syntax

Pinging the Informatica ServerPinging the Informatica Server– pmcmd ping {user_name | %user_env_var} {password | pmcmd ping {user_name | %user_env_var} {password |

%password_env_var} {[TCP/IP:][hostname:]portno | %password_env_var} {[TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address}IPX/SPX:ipx/spx_address}

Starting and Stopping Sessions and BatchesStarting and Stopping Sessions and Batches– pmcmd {start | stop} {user_name | %user_env_var}{password pmcmd {start | stop} {user_name | %user_env_var}{password

| %password_env_var}{[TCP/IP:][hostname:]portno | | %password_env_var}{[TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address} [folder_name:]{session_name | IPX/SPX:ipx/spx_address} [folder_name:]{session_name | batch_name} session_flag [wait_flag]batch_name} session_flag [wait_flag]

Recover SessionsRecover Sessions– pmcmd startrecovery {user_name | %user_env_var}pmcmd startrecovery {user_name | %user_env_var}

{password | %password_env_var} [TCP/IP:][hostname:]portno {password | %password_env_var} [TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address}[folder_name:]session_name | IPX/SPX:ipx/spx_address}[folder_name:]session_name wait_flagwait_flag

After the Session Launches...

Poll | Refresh session StatusPoll | Refresh session Status

View Session DetailView Session Detail

View Performance monitorView Performance monitor

View LogsView Logs

Tune the sessionTune the session

Correct Session problemsCorrect Session problems

Monitor Session

Session DetailsSession Details– Select Server Requests | Session DetailsSelect Server Requests | Session Details

– Number of rows loaded/failedNumber of rows loaded/failed

– Read/Write thoroughputRead/Write thoroughput

– Most current Server messageMost current Server message

– Audit trail repository or session logAudit trail repository or session log

Monitor Session

Performance MonitorPerformance Monitor– Select Server Requests | Session Performance Details or Open Select Server Requests | Session Performance Details or Open

file <sessionname>.perffile <sessionname>.perf

– Need to configure session properties to save the performance Need to configure session properties to save the performance

detail countersdetail counters

– Help determine where session performance can be improvedHelp determine where session performance can be improved

Log Files - Session Wizard

Log file path and nameLog file path and nameReject file path and nameReject file path and nameSession log and archive optionsSession log and archive options

Monitor Session

Session LogSession Log– Select Server requests | Session Log or Select view Select Server requests | Session Log or Select view

log file on button bar or Select Open Log File from log file on button bar or Select Open Log File from

Session Details WindowSession Details Window

– Gives complete details on a specific session runGives complete details on a specific session run

Session Sequence of Events

Lock the session and read session properties Lock the session and read session properties Create session log fileCreate session log file Initialize the DTMInitialize the DTM Allocate session shared memory and creates buffersAllocate session shared memory and creates buffers Execute Pre-session scriptsExecute Pre-session scripts Create Reader and Writer process and connect to the Create Reader and Writer process and connect to the

file or database connection.file or database connection. Execute the Stored proceduresExecute the Stored procedures Form SQL insert, delete, update statementsForm SQL insert, delete, update statements Create Index cache and data cache(s)Create Index cache and data cache(s) Parse TransformationsParse Transformations Move DataMove Data

Session Sequence of Events

Execute post Stored Procedures Execute post Stored Procedures

Write historical incremental aggregation data to Write historical incremental aggregation data to

diskdisk

Close the session logClose the session log

Send post-session e-mailSend post-session e-mail

Return memory to systemReturn memory to system

Session Errors

Session might fail becauseSession might fail because– Server cannot allocate enough memoryServer cannot allocate enough memory– Session exceeds maximum number of sessions Session exceeds maximum number of sessions

configuredconfigured– Server cannot cannot obtain execute lockServer cannot cannot obtain execute lock– Server cannot execute post-session commandsServer cannot execute post-session commands– Server encounters database errorServer encounters database error

Circumstances effecting sessionCircumstances effecting session– Server encounters transformation row errors. Server encounters transformation row errors. – Server executes post-session command Server executes post-session command

unsuccessfullyunsuccessfully– Server is unable to send post-session emailServer is unable to send post-session email

Error Handling - Session Wizard

Error Handling Error Handling Perform RecoveryPerform RecoveryOverride tracingOverride tracingStop or Continue on ErrorsStop or Continue on Errors

Reject Files

Created by the Server to store the data rejected Created by the Server to store the data rejected by the writer and/or target databaseby the writer and/or target database

Conditions causing rejected data include Conditions causing rejected data include – Target database constraint violations,null values not Target database constraint violations,null values not

accepted,out of space accepted,out of space – Target table properties ‘Reject truncated/Overflowed Target table properties ‘Reject truncated/Overflowed

Rows Rows Rejected rows are saved in a file <tablename>.badRejected rows are saved in a file <tablename>.bad Rejected rows are prefaced with an indicator Rejected rows are prefaced with an indicator

columncolumn– e.g. 3,D,1,D, , , , , , , , e.g. 3,D,1,D, , , , , , , ,

Reject Indicators Reject Indicators – DData, ata, OOverflow, verflow, NNull or ull or TTruncatedruncated

Reject File Loader ( pmrejldr )

Rename the .Rename the .bad bad file by appending the suffix .file by appending the suffix . in in

to the filename. E.g., <tablename>.bad.into the filename. E.g., <tablename>.bad.in

From the directory where the reject files are From the directory where the reject files are

located, issue commandlocated, issue command– pmrejldr<config_file_name> [folder_name:] < pmrejldr<config_file_name> [folder_name:] <

session_name>session_name>

• Config_file_name> is for UNIX onlyConfig_file_name> is for UNIX only

• Session and older name are case-sensitive Session and older name are case-sensitive

Recovery

Perform Recovery when session stops before Perform Recovery when session stops before

completioncompletion OPB_SRVR_RECOVERY stores the number of OPB_SRVR_RECOVERY stores the number of

records written to the target tablerecords written to the target table Run session with Run session with Perform RecoveryPerform Recovery checked checked

under under Log FilesLog Files tab tab Uncheck Perform Recovery upon completionUncheck Perform Recovery upon completion Source should remain staticSource should remain static Cannot perform recovery if the target has bulk Cannot perform recovery if the target has bulk

inserts enabledinserts enabled

Performance Tuning and Trouble Shooting

You can tune the performance in four general You can tune the performance in four general

area :area :

– SystemSystem

– Source and Target DatabaseSource and Target Database

– SessionSession

– MappingsMappings

Optimizing the System

Accessing Data Over a NetworkAccessing Data Over a Network

– Flat filesFlat files

– Relational DatabaseRelational Database

– Staging AreasStaging Areas

Using Multiple ServersUsing Multiple Servers

Eliminating PagingEliminating Paging

Optimizing the Source and Target Database

Optimizing the Source DatabaseOptimizing the Source Database

– Optimizer HintsOptimizer Hints

– SQL Servers and Joining Large tablesSQL Servers and Joining Large tables

– Sybase and SQL Server Packet SizesSybase and SQL Server Packet Sizes

Optimizing the Target DatabaseOptimizing the Target Database

– Indexes and key ConstraintsIndexes and key Constraints

– Bulk LoadingBulk Loading

Optimizing the Session

Running Parallel SessionRunning Parallel Session

Using Incremental AggregationUsing Incremental Aggregation

Eliminating Transformation ErrorsEliminating Transformation Errors

Tuning Session ParametersTuning Session Parameters

Optimizing the Session Contd....

Tuning Session ParameterTuning Session Parameter

– Shared Memory allocationShared Memory allocation

– Buffer Block SizeBuffer Block Size

– Index Cache SizeIndex Cache Size

– Data Cache SizeData Cache Size

– Commit IntervalCommit Interval

– Decimal ArithmeticDecimal Arithmetic

– Error TracingError Tracing

Questions

Documents

Informatica Server Manager