STREAMANALTIX 2.1.6 Register entitiesdocs.streamanalytix.com/2.1.6/pdf/RegisterEntities.pdf · Register Entities allows you to register custom components i.e. custom parsers, channels

StreamAnalytix 2.1.6 Register Entities

pg. 1

STREAMANALTIX 2.1.6

REGISTER ENTITIES

Register Entities like Functions, variables and more.


pg. 2

Introduction Welcome to StreamAnalytix! StreamAnalytix platform enables enterprises to analyze and respond to events in real-time at Big Data scale. With its unique multi-engine architecture, StreamAnalytix provides an abstraction that offers a flexibility to execute data pipelines using a stream processing engine of choice depending upon the application use-case, taking into account the advantages of Storm or Spark Streaming based upon processing methodology (CEP, ESP) and latency.

About This Guide This guide will help you to extend Streamanalytix usability by connecting the pipelines with custom components. The guide provides details on what support Streamanalytix provides for some of the widely used entities and how to customize them. More Information Please visit www.streamanalytix.com To give us your feedback on your experience with the application and report bugs or problems, mail us at [email protected] To receive updated documentation in the future please register yourself at www.streamanalytix.com We welcome your feedback.

http://www.streamanalytix.com/

mailto:[email protected]

http://www.streamanalytix.com/


pg. 3

Terms & Conditions This manual, the accompanying software and other documentation, is protected by U.S. and international copyright laws, and may be used only in accordance with the accompanying license agreement. Features of the software, and of other products and services of Impetus Technologies, may be covered by one or more patents. All rights reserved.

All other company, brand and product names are registered trademarks or trademarks of their respective holders. Impetus Technologies disclaims any responsibility for specifying which companies own which marks or which organizations.

USA Los Gatos Impetus Technologies, Inc. 720 University Avenue, Suite 130 Los Gatos, CA 95032, USA Ph.: 408.252.7111, 408.213.3310 Fax: 408.252.7114 © 2017 Impetus Technologies, Inc., All rights reserved.

If you have any comments or suggestions regarding this document, please send them via e-mail to [email protected]



pg. 4

Table of Contents Introduction ............................................................................................................................................... 2

About This Guide.................................................................................................................................... 2

More Information .................................................................................................................................. 2

Terms & Conditions ................................................................................................................................... 3

REGISTER ENTITIES..................................................................................................................................... 5

Register Components ............................................................................................................................. 5

Functions .............................................................................................................................................. 10

System Defined Functions ................................................................................................................ 10

User Defined Functions .................................................................................................................... 12

Variables............................................................................................................................................... 13

Transformation .................................................................................................................................... 14

Tables …………………………………………………………………………………………………………………………………………….16

Models.................................................................................................................................................. 16


pg. 5

REGISTER ENTITIES Register Entities allows you to register custom components i.e. custom parsers, channels and processors to be used in the pipelines. There are six types of entities:

Register Components Register custom code for using more out of components. Functions A rich library of pre-defined functions, and user defined functions. Variables Use variables in your pipelines at runtime as per the scope. Transformations A transformation rule explains how the value of field in a message should

be transformed to be used in predictive model. Tables Register Tables functionality enables you to store streaming data coming

from different sources (AWS S3, HDFS, JDBC connection) in Tables, which can be queried using SparkSQL processor.

Models Register Models functionality allows you to register the predictive models that are created using Spark Machine Learning API and are stored on HDFS or a Local file system.

Register Components Custom Code Implementation StreamAnalytix allows you implement your custom code in the platform to extend functionalities for:

• Channel: To read from any source.

• Processor: To perform any operation on data-in-motion.

• Parser: To parse the incoming data.

• Function: Custom function to be executed on data-in-motion via the Enricher operator.

• CEP (Storm only): Custom logic for executing on data over a CEP window.

StreamAnalytix makes custom code implementation very easy by providing: a simple interface to import and use custom components, and features like versioning that allows you to version your custom code.

You can download a Maven based project that contains all the necessary StreamAnalytix dependencies for writing custom code and sample code for reference.

Pre-requisites for custom code development

1. JDK 1.7 or higher

2. Apache Maven 3.x

3. Eclipse or any other IDE


pg. 6

Steps for Custom Code Implementation

Custom Code Interface Definitions (Spark)

Download a Sample Maven based project from ‘Data pipeline’ page

Import the project in Eclipse or any IDE of your choice

Write custom code against one or more of the interfaces

Build the maven project

Upload / Register component


pg. 7

Example of a Custom Processor Implementation - BlueCoat Processor

BlueCoatProcessor implements JsonProcessor – enables processing JSON data.

• Implement life cycle methods

• Init – initialization code (example establish connection with database or any other system)

• Process – Business logic goes here

Example:

o Change blank/empty CS_URL_PORT field to ‘123456789’

o Change All Other blank/empty fields by ‘-‘

• Cleanup – clean up code (example close connection)


pg. 8

Build Custom Code

• Provide all the dependencies required for the same in pom.xml available in the project.

• Build project using mvn clean install.

• Use jar-with-dependencies.jar for component registration. Register Custom Code

To register a custom component, go to Register Entities page.


pg. 9

Select component(s) that you want to register by providing a Component Name and a Scope (Local or Global).

Provide the class name of your custom code implemented in the uploaded jar file.


pg. 10

How use a registered component in your pipeline To use a Custom Parser, while configuring a message, on the Message Configuration tab, select Custom for the Message Parser Type. Custom Channel and Custom Processor are available on the pipeline canvas right panel under Channels and Processors.

Version Support (Versioning) in component registration You can register multiple versions of a registered component can use any version in your pipeline. NOTE: If you have used any registered component in the pipeline, make sure that all of the used registered components (that is registered with single jar) should be of the same version. If you have registered component with any fully qualified name, then component with that fully qualified name cannot be registered with another jar in that workspace. Same FQN and same jar will create new version of that component(s).

Functions Functions enables you to enrich an incoming message with additional data not originally provided by the source.

System Defined Functions StreamAnalytix provides a rich library of system-defined function as explained below:

Date

Function Name Description

addDT Adds the specified amount of time to the given day field, based on the calendar's rules.

afterDT Tests if this date is after the specified date beforeDT Tests if this date is before the specified date. currentDT Returns a Date object representing this Calendar's time value.

DiffInDTs Specifies the difference between two dates in terms of number of days

formatDT Formats a Date into a date/time string. isvalidDT Tests if the date is valid calendar date.


pg. 11

subDT Subtract the specified amount of time to the given day field, based on the calendar's rules.

withinDT Test if this date falls in between the specified dates.

String

Function Name Description isStrEmpty Returns true if, and only if, length () is 0. strConcat Concatenates the specified string to the end of string.

strContains Returns true if and only if string contains the specified sequence of char values.

strEquals Compares string to the specified object. The result is true if and only if the argument is not null and is a String object that represents the same sequence of characters as object.

strEqualsIgnore

Compares String to another String, ignoring case considerations. Two strings are considered equal ignoring case if they are of the same length and corresponding characters in the two strings are equal ignoring case. Two characters c1 and c2 are considered the same ignoring case if at least one of the following is true The two characters are the same (as compared by the == operator) Applying the method Character.toUpperCase(char) to each character produces the same result Applying the method Character.toLowerCase(char) to each character produces the same result.

strFormat Returns a formatted string using the specified format string and arguments.

strIndexOf

Returns the index within string of the first occurrence of the specified character. If a character with value ch occurs in the character sequence represented by String object, then the index (in Unicode code units) of the first such occurrence is returned. For values of ch in the range from 0 to 0xFFFF (inclusive), is the smallest value k such that: this.charAt(k) == ch is true. For other values of ch, it is the smallest value k such that: this.codePointAt(k) == ch is true. In either case, if no such character occurs in string, then -1 is returned.

strIsNull Checks for string to be null and return true if it is null.

strLength Returns the length of string. The length is equal to the number of Unicode code units in the string.

strMatches Tells whether string matches the given regular expression. Lookup

Function Name Description


pg. 12

lookupCassandra Returns the result, with respect to key from cassandra.

lookupES Returns the result, with respect to query, from Elasticsearch Index.

lookupHbase Returns the result, with respect to row and column-family passed as argument to the function, from hbase table.

lookupRDBMS Returns the result of the SQL query passed by user as an argument. Currently application supports database lookup for postgres, mysql and oracle.

lookupWS Returns the response of the web service. Currently supports the rest based web service.

Object

Function Name Description objDecrypt Decrypts an object objEncrypt Encrypts an object.

User Defined Functions StreamAnalytix provides a rich library of pre-defined function, and you can also define your own functions as user defined functions.

Field Description Function Name Name of the function

Arguments Input arguments for the function Parameters Function configuration parameter Upload Jar

Register functions by uploading a jar file containing custom code


pg. 13

Variables Allows you to use variables in your pipelines at runtime as per the scope. To add a variable, click on Add New Variable and provide details as explained below.

Field Description Name Provide a name to the variable Value Value of variable (it can be an expression)

Data Type

Select the Data Type of the variable. For example: Number, Decimal, or String.

Scope Select the Scope of the variable. Following are the types of scope:

• Global: The scope of this variable is global throughout the application.

• Workspace: The scope of the variable will be within all the topologies of the workspace.

• Pipeline: The scope of the variable is within the pipeline. If you select the option as pipeline, a dropdown is listed with all the pipelines, of the same workspace.

For example, if you create the following variables: Name, Salary and Average. Then by calling the following code you will get all the variables in the varMap in its implementation class. Map<String, ScopeVariable> varMap = (Map<String, ScopeVariable>) configMap.get(svMap); If you want to use the Name variable that you have created by calling the following code you will get all the details of the scope variables.


pg. 14

The variable object has all the details of the variable Name, Value, Datatype and Scope. ScopeVariable variable = varMap.get(Name); String value = variable.getValue();

Transformation A transformation rule explains how the value of field in a message should be transformed to be used in predictive model. Transformation variable is associated with a message and can be defined on any field in the message. Once it is defined, it can be used as any other message field, while defining the model. Create a Transformation Select a Message and click on Create Transformation.

There are four types of transformations: Normalization: Normalization field is dependent on continuous field. This transformation takes a continuous field in input and transforms it into another continuous value by normalizing the input on a linear scale. For example, If a message M has a continuous field F then a Transformation variable T of type continuous can be created using F of M. The linear scale for transformation can be defined on Transformation screen.


pg. 15

Map Value: This transformation is of categorical type and depends upon a categorical type of field of a message. For example, If a message M has a categorical field F with categories setosa, virginica, versicolor then a transformation variable T of type categorical with categories type1, type2, type3 can be defined on transformation screen such that, setosa will map to type1, virginica will map to type2 and versicolor will map to type3.

Continuous to Categorical: This transformation is of categorical type and depends upon a continuous type of field of a message. For example, If a message M has a continuous field F, whose value can be from -infinity to +infinity, then a transformation variable T of type categorical with categories low and high can be defined such that, values of F between -infinity to +1 (including +1) will map to low category of T and values of F between +1(excluding +1) to +infinity will map to high category of T.

Categorical to Continuous: This transformation is of continuous type and depends upon a categorical type of field of a message. For example, If a message M has a categorical field F with categories setosa, virginica, versicolor then 3 transformation variables T1, T2, T3 of type continuous will be created that will depend upon F with categories setosa, virginica, versicolor.

Next, select the Dependent Variable and fill Missing Value with a value as shown below:

Add Fields allows you to enter Input Value & Normalized Table. It defines the value of dependent variable, so that any value between the defined input values will be normalized between the normalized values defined by you.


pg. 16

Tables Register Tables functionality enables you to store streaming data coming from different sources (AWS S3, HDFS, JDBC connection) in Tables, which can be queried using SparkSQL processor. To register a table, go to Register Entities > Register Tables tab and click on Register Table link.

Field Description Name Name of the table to be registered. Data Sink AWS S3, HDFS or JDBC as data source to fetch the data from

Connection Name Select a connection

Database Table Name

Name of the database table.

Models Register Models functionality allows you to register the predictive models that are created using Spark Machine Learning API and are stored on HDFS or a Local file system. Follow below steps to register the model with StreamAnalytix: Step 1: Train the model on certain dataset and save it at any location on HDFS or local file system. Step 2: Register a model.


pg. 17

Field Description Name Name of the model Model API Spark API on which the model is built Model Type Types of models supported for the chosen Spark API

Upload Select this option if model is saved on a local file system

HDFS Select this option if model is saved on HDFS

Validate

Validates the model located at the given location

To give us your feedback on your experience with the application and report bugs or problems, mail us at [email protected]


Documents

STREAMANALTIX 2.1.6 Register entitiesdocs.streamanalytix.com/2.1.6/pdf/RegisterEntities.pdf · Register Entities allows you to register custom components i.e. custom parsers, channels