2/7/13 SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing 1/4

Added by Vicky Bolster, last edited by Vicky Bolster on Mar 05, 2012

Welcome, GuestLoginRegister

Getting StartedNewslettersStore

SolutionsServices & SupportAbout SCNDownloadsIndustriesTraining & EducationPartnershipDeveloper CenterLines of BusinessUniversity AlliancesEvents & WebinarsInnovation

Selective Reading and Postprocessing

Sometimes the task is not that easy. The pattern might change, e.g. read all f iles matching the pattern "plan_[BOBJ:todays date].txt", or you w ant to rename the f iles after successful load. There is no

direct support for that but a w orkaround - you can use parameters as f ilename as w ell.

So the idea w ould be to

use a function to get the f irst f ilename

w hile a f ilename exists loop

w ithin the loop call the dataflow w ith that f ilename to load and afterw ards rename the f ile and f ind the next f ilename - if there is one

The f irst step is to create a Workflow to hold the entire logic. Inside that w orkflow a variable is created called $filename to hold the next f ilename to be processed.

In a f irst script w e use the DI 11.7 function w ait_for_file() to set the variable $filename to the f irst f ile matching our pattern or NULL if there is none. As the function does either use '\' or '/' a path

separator char w ith the latter being the default setting in the DSConfig.txt, w e use a replace_substr() function to change it back to backslash. The DI internal functions can deal w ith either character as

path separator, but w e w ill use exec() to run a msdos command later.

And now w e use a w hile loop to call a dataflow reading the f ile repetitive unless there are no more f iles ("$filename is not null").

2/7/13 SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing 2/4

In the dataflow - a "procedure" of its ow n - w e w ant to use the value of the local variable of the w orkflow as w ell. So w e need to define a parameter for the dataflow by clicking on the variable

w indow w hen view ing the dataflow and define it. For clarity I called it $df_filename.

Now the dataflow requires a parameter and everyw here you call that dataflow "procedure" you need to apply a value. Where do w e call it? In the w orkflow . So go back to the w orkflow , open the calls

tab of the variable w indow and provide a value - the value of $filename.

Then w e can use the source f ile name $df_filename in the dataflow 's objects, in our case as a f ilename. But w atchout, the variable is used in the f ile reader call, not in the f ile reader class. So open the

object inside the dataflow to set the variable, not form the object library -> f ile formats as there you define the object itself. Not the call properties. Obviously you could have used global variables to

avoid that passing along of variables but...

2/7/13 SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing 3/4

Last object is another script for the post processing. If a dataflow did load a f ile successfully, w e w ant to rename the f ile into something else. Today there are still no native functions in DI so w e have

to go via the exec function.

exec('cmd.exe', 'move [$filename] [$filename].old', 0);

And finally w e reset $filename to the next available f ilename inside the w hile loop.

The function w ait_for_file() has the syntax w ait_for_file(f ilename pattern, timeout, poll interval, how many f ile names should be returned, a variable holding the returned f ilename list, returned size of list,

list separator char).

In our case w e do not need the last tw o optional parameters. We w ant the function to return immediately (timeout=0), as w e w ill not poll multiple times the poll interval can be anything, w e w ant just one

file to be returned and into the variable $filename.

wait_for_file('c:\\temp\\plan_[year(sysdate())]*.txt', 0, 0, 1, $filename);

Note: This entire chapter is based on the DI 11.7 function w ait_for_file(). But you can get the same result by setting the variable using

$filename='c:\\temp\\' || word(exec('cmd.exe', 'dir /b c:\\temp\\plan*.txt', 0), 1);

Change the Row Delimiter

Errorhandling in f ile formats

Excel- Save to DI

How to create a f lat f ile format from a Query

Multirecord Files

Reading a large XML file

Reading multiple f iles at once

File Group Reader

Selective Reading and Postprocessing

Shared Directory access

Writing a large XML file

Follow SCN

Contact UsSAP Help Portal

PrivacyTerms of UseLegal DisclosureCopyright

2/7/13 SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing 4/4