Ab Initio Custom Component

Embed Size (px)

Citation preview

  • 8/20/2019 Ab Initio Custom Component

    1/33

    Vineet Agarwal  Page 1 4/18/2007 

    Presentation On

    Ab Initio Custom Component 

    ByVineet Agrawal

    [email protected]

  • 8/20/2019 Ab Initio Custom Component

    2/33

    Vineet Agarwal  Page 2 4/18/2007 

    Contents

    • Overview 

    • Syntax of MPC file 

    • Writing A Program Specification File 

    • Steps To Create New Component 

    • Detailed Descriptions of Attributes & Variables 

  • 8/20/2019 Ab Initio Custom Component

    3/33

    Vineet Agarwal  Page 3 4/18/2007 

    Overview

    • Program specification files provide the Co>Operating Systemwith the information it needs to run program or shell script.

    • Program specification files should be with .mpc extensions.

    All program specification files must start with

    • The line is followed by a series of attribute:value lines that describe the attributes of program.

  • 8/20/2019 Ab Initio Custom Component

    4/33

    Vineet Agarwal  Page 4 4/18/2007 

    Syntax

    Program specification file uses the following syntax

    [ label: label ]

    [ author: author-name ]

    [ version: version-number ]

    [ comment: comment ]

    image: path

    [ exit: code ]

    [ port: type direction name [location] [ordering]

    [fan-preference] [min-flows] [max-flows] [record-format] ]

    [ environment : env-variable value ]

    [ argument: literal value1 [value2 ...] ]

    [ argument: flow portname ]

    [ argument: partition ]

    [ argument: depth ]

  • 8/20/2019 Ab Initio Custom Component

    5/33

    Vineet Agarwal  Page 5 4/18/2007 

    [ argument: file filename ]

    [ argument: expression file filename ]

    [ argument: expression string ]

    [ argument: transform file filename ]

    [ argument: transform string ]

    [ parameter: placement name type default-value Þ

    description [restrictions] ]

    [ metadata type: dest-portname = {source-portname | value} ]

  • 8/20/2019 Ab Initio Custom Component

    6/33

    Vineet Agarwal  Page 6 4/18/2007 

    Writing a program specification file

    • Must include the line and the image line, all theother lines are optional.

    • Can use as many port, argument, parameter, and metadatatype lines as you need to describe your particular program.

    • Comments in a program specification file are denoted by the

    following:

    Lines that start with #

    From // to the end of the line

    From /* to */

    To write a program specification file in the ComponentOrganiser:

    1. In the GDE, open the Component Organiser.

    2. Right-click My Components.

    3. From the shortcut menu, choose New > Program.

    A New Component icon appears under My Components.

  • 8/20/2019 Ab Initio Custom Component

    7/33

    Vineet Agarwal  Page 7 4/18/2007 

    4. Do one of the following:

    Enter a name in the New Component icon.

    Press Enter to accept New Component as the name.

    5. Right click the New Component icon.

    A pop-up menu appears.

    6. From the pop-up menu, choose Edit As text.

    The Edit Program Component window opens.

    7. Write your own program specification file by editing thetemplate in the window.

  • 8/20/2019 Ab Initio Custom Component

    8/33

    Vineet Agarwal  Page 8 4/18/2007 

    Steps to Create New Component

    Step 1

  • 8/20/2019 Ab Initio Custom Component

    9/33

    Vineet Agarwal  Page 9 4/18/2007 

    Step 2

  • 8/20/2019 Ab Initio Custom Component

    10/33

    Vineet Agarwal  Page 10 4/18/2007 

    Step 3

  • 8/20/2019 Ab Initio Custom Component

    11/33

    Vineet Agarwal  Page 11 4/18/2007 

    Detailed Descriptions ofAttribute & Variables

    LabelSyntax: label: label 

    This line specifies the name you want to appear for thecomponent in the Component Organiser and on the componenticon in the GDE.

    Example:

    Lebel: “This Is My Own Component” 

    Mpname

    mpname: "mp component name"

    AuthorSyntax: author: "Your Name" 

    This line specifies the author's name that appears on theProperties Description tab of the Properties dialog box in theGDE.

    Example: author: “Vineet Agrawal” 

  • 8/20/2019 Ab Initio Custom Component

    12/33

    Vineet Agarwal  Page 12 4/18/2007 

    Version

    Syntax: version: version-number 

    This line specifies the version number that appears on theProperties Description tab of the Properties dialog box in theGDE.

    For built-in components, use “Built-in :”

    Examples: version: "1.0" 

    Comment

    Syntax: comment: comment 

    This line specifies the comment that appears in the GDE in two

     places:

    • On the Properties Description tab of the Properties dialog box

    • In the Description Panel of the Component Organizer whenyou select the component

    Examples: comment: "Your Description Here" 

  • 8/20/2019 Ab Initio Custom Component

    13/33

    Vineet Agarwal  Page 13 4/18/2007 

    Image

    Syntax: image: "path"

    This line specifies the path for the executable program or scriptfor this component; path represents this path.

    Specify an absolute or relative path. A relative path is relative tothe current search path specified in the PATH configurationvariable.

    Supply one image line in the program specification file. This isthe only line other than the line that you must have inthe program specification file.

    When the Co>Operating System runs the program, it uses thesame path on all processing nodes. If the Co>Operating Systemdoes not find the program on a processing node by followingthat path, it copies the program from the control node to a cache

    directory on the processing node, runs it, and saves it for futureuse.

    Examples: image: "name-of-executable" NOTE: Enclose path in double quotes. 

  • 8/20/2019 Ab Initio Custom Component

    14/33

    Vineet Agarwal  Page 14 4/18/2007 

    Exit

    Syntax: exit: code 

    This line specifies the integer exit code that indicates asuccessful termination. Use a specific integer or one of thefollowing:

    Any negative positive even non_negative oddnon_positive

    The default is 0.Examples: exit: negative

    Port

    Port lines define the ports that the component supports.

    Syntax: port : type direction name [ordering ] [location ]

    [fan-preference ] [min-flows ] [max-flows ] [record-format] 

    Ports in Ab Initio components represent the input/output behaviour of program.

  • 8/20/2019 Ab Initio Custom Component

    15/33

    Vineet Agarwal  Page 15 4/18/2007 

    1. type

    The type variable specifies one of four different types of ports,as follows:

    • std - Invokes program with the standard input, standardoutput, or standard error already connected.

    In order to use a std port, program must read sequentially fromits standard input flow or write sequentially to the standardoutput or standard error flow, without seeking.

    The .mpc file needs no additional lines for a std port. 

    • file - Passes a path to program.

    In order to use a file port, program must accept a path argumenton the command line, and either read a file without modifying it(for an in port) or write a file with that path (for an out port).

    For an in port, your program may open the file any number oftimes, and it may seek in the file.

    For an out port, your program must open the file at least once, ormust create it in some other way.

    The .mpc file should contain a line of the following form foreach file port:

    argument: flow portname 

  • 8/20/2019 Ab Initio Custom Component

    16/33

    Vineet Agarwal  Page 16 4/18/2007 

    • npipe- Creates a named pipe or file and passes its path toyour program.

    In order to use an npipe port, your program must accept a pathargument on the command line and perform sequential I/O on it.

    For an in port, your program must open the file for readingexactly once and then read sequentially to the end withoutseeking or writing in the file.

    For an out port, your program must open the file for writingexactly once, and then write sequentially without seeking orreading in the file.

    Ports of the npipe type allow the graph to run faster, due to the pipeline parallelism they support.

    The .mpc file should contain a line of the following form for

    each npipe port:

    argument: flow portname

    • soc - The SOC interface is used by Ab Initio built-incomponents 

  • 8/20/2019 Ab Initio Custom Component

    17/33

    Vineet Agarwal  Page 17 4/18/2007 

    2. direction

    Specifies the direction of the flow for a port.

    direction Port

    in input port or stdin

    out output port or stdouterr Stderr

    3. name

    Specifies the name of the port. For example, the input port isusually named in.

    4. location

    Specifies on which side of the component icon you want the

     port to appear in the GDE.

    Choose one of the following:top bottom left right

    5. ordering

    Controls whether or not the GDE may reorder multiple flowsconnected to a port:

    • If you specify ordered, the GDE does not reorder multipleflows connected to the port.

    • If you do not specify ordered, the GDE reorders portconnections for readability.

    • Ordered indicates ordering of attached flows is important(e.g. input to CONCAT) (default NO)

  • 8/20/2019 Ab Initio Custom Component

    18/33

    Vineet Agarwal  Page 18 4/18/2007 

    6. fan-preference

    Specifies the type of flow the GDE creates when you connect aflow to the port.

    The fan-preference option works as follows:

    If you specify multiple, the GDE creates fan-in flows, fan-outflows, or all-to-all flows where appropriate.

    If you do not specify multiple, the GDE creates straight flows. 

    7. min-flows and max-flows 

    Specifies the minimum and maximum number of flow partitionsyou can connect to a port.

    - use 0 for mintaps to indicate the port need not beconnected.

    - use a big number like 99999 for maxtaps to indicate nolimit.

    - defaults are 1 and infinity if not given

    This is used to guess flow patterns. 

  • 8/20/2019 Ab Initio Custom Component

    19/33

    Vineet Agarwal  Page 19 4/18/2007 

    8. record-format

    The default is for ports to propagate their metadata from the

     ports to which they connect.

    metadata is quoted metadata default, either "=string" or"&remotefilepath" or "Llocalfilepath" 

    Examples:

     port: soc in in ordered 1 99999

     port: soc out out 1 1 port: soc out log 0 1 port: npipe in indat ordered left multiple 1 1 "=string('\n')" 

    Environment

    Syntax: environment: "env-variable value"

     NOTE: Enclose env-variable value in double quotes.

    This line lets you direct the Co>Operating System to set theenvironment variable named by env-variable to value in the run-time environment of each partition of the component. 

    Examples: environment: “BB_DQ_COUNT 10” 

  • 8/20/2019 Ab Initio Custom Component

    20/33

    Vineet Agarwal  Page 20 4/18/2007 

    Parameter

    Used by the GDE to construct "mp custom" commands in the

     job script.

    The "mp custom" line will contain the parameters in thespecified order.

    Syntax: parameter: ""

    "" [reqspec] [visibility] [""] [codegen]

    form is how parameter looks on mp line in script, either positional or keyword

    • name is parameter name (as seen on parameters property page)

    • type is one of: expression transform integer float stringexpression file dataset metadata transform layout date mode

     protection bool special choice literal infile outfile.

    • default is quoted default value (for choice parameters, a listof the choices, first is default)

    • doc is quoted documentation string (keep it short)

  • 8/20/2019 Ab Initio Custom Component

    21/33

    Vineet Agarwal  Page 21 4/18/2007 

    • reqspec is optional or required (default is required for positional, optional for keyword)

    • visibility is visible, cond_visible, or hidden (default iscond_visible)

    • condition specifies when to display the parameter in theParameters Properties page (when visibility is cond_visible),and when to generate code on the mp line for the parameter(when codegen is cond_codegen).

    • codegen is codegen, cond_codegen, or nocodegen 

    default combination is based on visibility as :

    visible->codegencond_visible->cond_codegenhidden->nocodegen

  • 8/20/2019 Ab Initio Custom Component

    22/33

    Vineet Agarwal  Page 22 4/18/2007 

    EXAMPLES:

     parameter: positional Key collator "" "Key specifier on which toaggregate"

     parameter: positional EncodingMode choice "-ascii -binary""add -ascii (default) or -binary to mp cmd" required

     parameter: positional PickOne choice "\"\" c1 c2" "add c2 or c2

    or neither (default)" optional

     parameter: keyword Force bool "True" "Adds -force to mp lineif set TRUE"

  • 8/20/2019 Ab Initio Custom Component

    23/33

    Vineet Agarwal  Page 23 4/18/2007 

    Argument

    Argument lines indicate the ordered arguments to the

    executable.

    The argument lines specify the arguments that the Co>OperatingSystem passes to program. The Co>Operating System passes thearguments to the program in the same order the argument linesoccur in the .mpc file.

    Syntax: 

    You can write nine different types of argument lines:

    argument: literal

    argument: flow

    argument: partition

    argument: depth

    argument: file

    argument: expression file

    argument: expression

    argument: transform file

    argument: transform

  • 8/20/2019 Ab Initio Custom Component

    24/33

    Vineet Agarwal  Page 24 4/18/2007 

    • argument: literal

    Syntax: argument: literal "value1" ["value2" ...] 

     passes value to your program as is.

    • argument: flow

    Syntax: argument: flow portname 

     passes the path of the named pipe or file attached to the portnamed by portname.

    If you attach multiple flows to a port and then use the name ofthat port for portname, a flow argument will pass the paths ofthe named pipes or files represented by those flows to your

     program as a series of arguments, one path per argument.

    • argument: partition

    Syntax: argument: partition

     passes the number of the partition on which its copy of thecustom component is running. It passes this partition number onthe command line, as a number between 0 and the number of

     partitions minus 1.

  • 8/20/2019 Ab Initio Custom Component

    25/33

    Vineet Agarwal  Page 25 4/18/2007 

    • argument: depth

    Syntax: argument: depth

     passes the number of partitions on which the custom componentis running to program.

    • argument: file 

    Syntax: argument: file filename

     passes filename to program.

    If the Co>Operating System does not find the file named byfilename on the processing node where a particular partition ofthe component is running, the Co>Operating System copies it toa cache directory on that node, passes the name of the cachedcopy to the program, and saves the copy for future use.

    argument: expression file 

    Syntax: argument: expression file filename

     passes filename to program in the same way that a file argumentdoes; use the expression file argument for files that containDML expressions.

  • 8/20/2019 Ab Initio Custom Component

    26/33

    Vineet Agarwal  Page 26 4/18/2007 

    • argument: expression 

    Syntax: argument: expression string

     passes string to your program as is; use the expression argumentfor a string that denotes a DML expression.

    • argument: transform file 

    Syntax: argument: transform file filename

    A transform file argument passes filename to your program inthe same way a file argument does; use the transform fileargument for files that contain a DML transform function.

    • argument: transform 

    Syntax: argument: transform string

    A transform argument passes string to your program as is; stringrepresents a DML transform function.

  • 8/20/2019 Ab Initio Custom Component

    27/33

    Vineet Agarwal  Page 27 4/18/2007 

    Metadata TypeSyntax: metadata type: =  

    metadata type: dest-portname = {source-portname |value}

    These lines tell the GDE how to set the record format for thecomponent's ports.

    Choose one of the following:

    • dest-portname = source-portname - The GDE propagates(copies) the record format from the source port to thedestination port

    • dest-portname = value - the GDE sets the record format to

    value

    The value above represents a record format in double quotes,with any embedded double quotes backslashed. The following isan example:

    metadata type: log = "record string(\"|\") node, timestamp,component, subcomponent, event_type; string(\"|\\n\")

    event_text; end" 

    EXAMPLES:

    metadata type: out = inmetadata type: in = out 

  • 8/20/2019 Ab Initio Custom Component

    28/33

    Vineet Agarwal  Page 28 4/18/2007 

    Sample mpc file of Sort Component

  • 8/20/2019 Ab Initio Custom Component

    29/33

    Vineet Agarwal  Page 29 4/18/2007 

    Filter By Expression

    Description Panel 

  • 8/20/2019 Ab Initio Custom Component

    30/33

    Vineet Agarwal  Page 30 4/18/2007 

    Properties Description Tab

  • 8/20/2019 Ab Initio Custom Component

    31/33

    Vineet Agarwal  Page 31 4/18/2007 

    Sample mpc file of Component ‘Filter By Expression’

    label: "Filter by Expression"

    mpname: "select-transform"image: "unitool"author: "Ab Initio Software"version: "Built-in"comment: "Filters data records according to a specified DMLexpression."

     port: soc in in port: soc out out 1 1

     port: soc out deselect right 0 1 port: soc out reject bottom 0 1 port: soc out error bottom 0 1 port: soc out log bottom 0 1

     parameter: positional select_expr valid_expression "" "Filter expression"

     parameter: implicit "reject-threshold" choice "0Abort\177on\177first\177reject Never\177abort Use\177limit/ramp"

    "When to abort if input records are rejected" optional visible "" parameter: implicit limit integer "0" "Maximum rejected records beforefailure" optional visible "param reject-threshold Use*" parameter: implicit ramp float "0.0" "Rate of rejected records" optionalvisible "param reject-threshold Use*" parameter: local keyword limit_keyword integer "0" "Maximum rejectedrecords before failure" optional visible "value reject-threshold Never* 0value reject-threshold Abort* 0 sameas limit" parameter: local keyword ramp_keyword float "0.0" "Rate of rejectedrecords" optional visible "value reject-threshold Never* 99.0 value reject-threshold Abort* 0.0 sameas ramp" parameter: local implicit "keyword-map" string "limit_keyword limitramp_keyword ramp" ""

  • 8/20/2019 Ab Initio Custom Component

    32/33

    Vineet Agarwal  Page 32 4/18/2007 

     parameter: implicit logging bool "False" "Log internal events" parameter: local keyword log string "" "Special log parameter" "paramlogging True log_concat" parameter: implicit log_input choice "0 \177 1 10 100 1000 10000

    100000" "Frequency of input records to log" optional visible "paramlogging True" parameter: implicit log_output choice "0 \177 1 10 100 1000 10000

    100000" "Frequency of output records to log" optional visible "paramlogging True" parameter: implicit log_reject choice "0 \177 1 10 100 1000 10000100000" "Frequency of reject records to log" optional visible "paramlogging True"

    argument: literal "select"argument: literal $1 /* name */argument: literal $2 /* select_expr */argument: literal $3 /* limit */argument: literal $4 /* ramp */argument: literal $5 /* log */metadata type: out = inmetadata type: in = outmetadata type: deselect = inmetadata type: in = deselect

    metadata type: deselect = outmetadata type: out = deselectmetadata type: reject = inmetadata type: reject = outmetadata type: reject = deselectmetadata type: error = "string('\n')"metadata type: log = "record string(\"|\") node, timestamp, component,subcomponent, event_type; string(\"|\\n\") event_text; end" parameter: implicit "condition" string "" "" optional ""

     parameter: implicit "conditionInputPort" string "in" "" optional "" parameter: implicit "conditionOutputPort" string "out" "" optional "" parameter: implicit "condition-interpretation" choice "0Replace\177with\177flow Remove\177completely" "" optional ""

  • 8/20/2019 Ab Initio Custom Component

    33/33

    Thank You