• Home

  • Documents

  • Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester...
33
` Multidomain MDM 10.4 Smart Search Prashant Gupta

Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

`

Multidomain MDM 104Smart Search

Prashant Gupta

2 copy Informatica Proprietary and Confidential

Housekeeping Tips

Todayrsquos Webinar is scheduled for 1 hour

The session will include a webcast and then your questions will be answered live at the end of the presentation

All dial-in participants will be muted to enable the speakers to present without interruption

Questions can be submitted to ldquoAll Panelists via the QampA option and we will respond at the end of the presentation

The webinar is being recorded and will be available to view on our INFASupport YouTube channel and Success Portal The link will be emailed as well

Please take time to complete the post-webinar survey and provide your feedback and suggestions for upcoming topics

Success Portal httpssuccessinformaticacom

Learn Adopt Succeed

copy Informatica Proprietary and Confidential

FREE Product Learning Paths

and weekly Expert sessions

Bootstrap product trial experience

InformaticaConcierge with

Chatbot integrations

Enriched Onboarding experience

Tailored training and content

recommendations

4 copy Informatica Proprietary and Confidential

Safe Harbor

The information being provided today is for informational purposes only The

development release and timing of any Informatica product or functionality

described today remain at the sole discretion of Informatica and should not be

relied upon in making a purchasing decision

Statements made today are based on currently available information which is

subject to change Such statements should not be relied upon as a

representation warranty or commitment to deliver specific products or

functionality in the future

5 copy Informatica Proprietary and Confidential

Agenda Overview

SS Architecture

Configuration of Smart Search

Indexing Job Vs Load Job

BE Search APIrsquos

Troubleshooting SS outside MDM via REST Calls

Elastic Search Methodology

MDM 104 Features

Performance amp Best Practices

Known Limitations

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 2: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

2 copy Informatica Proprietary and Confidential

Housekeeping Tips

Todayrsquos Webinar is scheduled for 1 hour

The session will include a webcast and then your questions will be answered live at the end of the presentation

All dial-in participants will be muted to enable the speakers to present without interruption

Questions can be submitted to ldquoAll Panelists via the QampA option and we will respond at the end of the presentation

The webinar is being recorded and will be available to view on our INFASupport YouTube channel and Success Portal The link will be emailed as well

Please take time to complete the post-webinar survey and provide your feedback and suggestions for upcoming topics

Success Portal httpssuccessinformaticacom

Learn Adopt Succeed

copy Informatica Proprietary and Confidential

FREE Product Learning Paths

and weekly Expert sessions

Bootstrap product trial experience

InformaticaConcierge with

Chatbot integrations

Enriched Onboarding experience

Tailored training and content

recommendations

4 copy Informatica Proprietary and Confidential

Safe Harbor

The information being provided today is for informational purposes only The

development release and timing of any Informatica product or functionality

described today remain at the sole discretion of Informatica and should not be

relied upon in making a purchasing decision

Statements made today are based on currently available information which is

subject to change Such statements should not be relied upon as a

representation warranty or commitment to deliver specific products or

functionality in the future

5 copy Informatica Proprietary and Confidential

Agenda Overview

SS Architecture

Configuration of Smart Search

Indexing Job Vs Load Job

BE Search APIrsquos

Troubleshooting SS outside MDM via REST Calls

Elastic Search Methodology

MDM 104 Features

Performance amp Best Practices

Known Limitations

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 3: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

Success Portal httpssuccessinformaticacom

Learn Adopt Succeed

copy Informatica Proprietary and Confidential

FREE Product Learning Paths

and weekly Expert sessions

Bootstrap product trial experience

InformaticaConcierge with

Chatbot integrations

Enriched Onboarding experience

Tailored training and content

recommendations

4 copy Informatica Proprietary and Confidential

Safe Harbor

The information being provided today is for informational purposes only The

development release and timing of any Informatica product or functionality

described today remain at the sole discretion of Informatica and should not be

relied upon in making a purchasing decision

Statements made today are based on currently available information which is

subject to change Such statements should not be relied upon as a

representation warranty or commitment to deliver specific products or

functionality in the future

5 copy Informatica Proprietary and Confidential

Agenda Overview

SS Architecture

Configuration of Smart Search

Indexing Job Vs Load Job

BE Search APIrsquos

Troubleshooting SS outside MDM via REST Calls

Elastic Search Methodology

MDM 104 Features

Performance amp Best Practices

Known Limitations

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 4: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

4 copy Informatica Proprietary and Confidential

Safe Harbor

The information being provided today is for informational purposes only The

development release and timing of any Informatica product or functionality

described today remain at the sole discretion of Informatica and should not be

relied upon in making a purchasing decision

Statements made today are based on currently available information which is

subject to change Such statements should not be relied upon as a

representation warranty or commitment to deliver specific products or

functionality in the future

5 copy Informatica Proprietary and Confidential

Agenda Overview

SS Architecture

Configuration of Smart Search

Indexing Job Vs Load Job

BE Search APIrsquos

Troubleshooting SS outside MDM via REST Calls

Elastic Search Methodology

MDM 104 Features

Performance amp Best Practices

Known Limitations

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 5: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

5 copy Informatica Proprietary and Confidential

Agenda Overview

SS Architecture

Configuration of Smart Search

Indexing Job Vs Load Job

BE Search APIrsquos

Troubleshooting SS outside MDM via REST Calls

Elastic Search Methodology

MDM 104 Features

Performance amp Best Practices

Known Limitations

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 6: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

6 copy Informatica Proprietary and Confidential

Overview Smart Search is the search done in Entity 360 UI based on Business Entity

It was introduced starting MDM 10x release

Business Entity service provide three different ways to search for entity

records

Match ndash SSA based search

Query ndash DB based search

Search ndash Google like text search

Provides a Google-like text search engine for text data

It searches within a specific searchable BE

SS request uses the indexes to search for the matching records

Filters can be applied to identify more relevant results

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 7: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

7 copy Informatica Proprietary and Confidential

Smart Search in E360 UI

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 8: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

8 copy Informatica Proprietary and Confidential

Migration Strategy for MDM 104 Upgrade

From SOLR

Need to perform fresh installation amp Configuration of

Elastic Search Indices will generate again

From Elastic Search

Upgrade the MDM Hub to 104 will not lose their ES

setup from the previous release No change for them

config--analysis(folder)

--stopwordstxt When you perform a search MDM can ignore common words such as and an and is

--synonymstxt when you search for William the search result can include the synonyms Will and Willy

--lang (folder)--mapping-FoldToASCIItxt To use the default Elasticsearch analyzers for languages such as Chinese Japanese and Korean

--stopwords_jatxt

Fresh Installation

Download the ES package from web amp extract

Need to explicitly create the analysis and lang

folder mentioned below

Reach out to Informatica GCS to get the below

highlighted files

Once all file in place then we are good to

Configure Smart Search

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 9: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

9 copy Informatica Proprietary and Confidential

Elastic Search - High Level Architecture

siperian-mrmear

Elastic Client

Elasticsearch

Nodeor

Cluster

Can be on same machine as MDM or different machine(s)

Process ServerHTTPS

MDM

siperian-mrm-cleanseear

Elastic Client

Hub Store

Done from provisioningSearchable configurationElastic Search NodeCluster

Done from Hub ConsoleProcess server that are search enabled

Configuring security and enabling communication with MDM hubelasticsearchyml

Enable Search processing in properties file

Execute MDM batch jobs

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 10: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

10 copy Informatica Proprietary and Confidential

Install amp Configure Smart Search

Configure the elasticsearchyml file ( [MDMHubServer]thirdpartyelasticsearch-623config)

Configure SS in cmxserver and cmxcleanse properties

Enable Search Processing on Process Servers in MDM Hub Console

Register the ES NodeCluster in Provisioning Tools

Restart the ES server followed by MDM server

Execute Indexing Job

For more information detailed steps are present in KB 564677

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 11: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

11 copy Informatica Proprietary and Confidential

cmxssenabled Indicates whether to enable search In a new installation the default is true When upgrading if this property is set the value remains set to the pre-upgrade value cmxssengine Required if you want to use the Elasticsearch engine for search Manually add the property and set to es sslkeyStore sslkeyStorepassword ssltrustStore ssltrustStorepassword Required if you use the HTTPS port of the

application server to configure the Hub Server Manually add the property Absolute path and file name of the KeyStore file exmaxconnperhost This property is the maximum number of connections for a given host in the pool Default is 100 MDM

maintains a pool of http connections specific to Elastic Search communication exmaxthreads Sets the maximum number of threads that you want the Apache asynchronous non-blocking receiver to use for

each node in the Elasticsearch cluster Default is 1

Cmxserver amp Cmxcleanse

esindexrefreshinterval Sets the interval in seconds for Elasticsearch to commit the changes to the data after an Initially Index Smart Search Data batch job is run The data is available for search after this time interval Default is 30 seconds cmxserverbatchsmartsearchinitialblock_size Indicates the block size during indexing job cmxssdirtyIndexdisable=true This property allows the Batch Viewer tool to start and prevents the warning icon that indicates

an outdated index from being displayed in the Batch Viewer tool

Only Cmxserver

-Dmdmesindexclientttl If there are any connection issues between the AppServer and Elasticsearch processes this value will set the timeout as to how long the AppServer process should wait before getting a response This property is in seconds andshould be set in the AppServer startup properties to 86400 which translates to 24 hours

JVM Parameter

SS Properties

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 12: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

12 copy Informatica Proprietary and Confidential

Configuring Process Servers

Process Servers act as clients for Search Requests

No indexing is done on Process Servers

You will need to register at least one Process Server as search enabled

You can register more than one process server

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 13: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

13 copy Informatica Proprietary and Confidential

Configuring Node(s) in Provisioning After installation start the elastic search process

Once the process is available register each ES Node under infrastructure settings

Ensure that ES Node name and the HTTPS URL is reachable

If you have more than one ES node register each one individually

Save and publish changes Validate ORS

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 14: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

14 copy Informatica Proprietary and Confidential

Configuring Searchable BE in Provisioning

Searchable ndash Enables search for that field

Suggester ndash Type-ahead search for Searchable field

Do not enable suggester for sensitive data

Filterable ndash Enables Filtering on a Field

Search Analyzer ndash Specifies the custom search

analyzer for the field

Facets ndash Grouping of similar data

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 15: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

15 copy Informatica Proprietary and Confidential

Indexing Job or Load Job

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Indexed Internally by MDM

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

Run Initial

Indexing

Existing Data

NewUpdated Configuration

Indexes Deleted

Incremental Loads

API Calls amp E360 Inserts

Updates amp Merges

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 16: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

16 copy Informatica Proprietary and Confidential

Indexing job is being executed from Batch Viewer in MDM Hub Console Need to configure at least one process server marked as lsquoEnable Search Processingrsquo

Execute Smart Search Indexing Job

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 17: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

17 copy Informatica Proprietary and Confidential

Stopwords Common list of words that

search engines filter out after processing

It improves indexing performance ie doesnrsquot index the common words present in stopwordstxt

Can use the default list of common words to ignore in a search or even can customize the list

Configure Stop words a Use a text editor to open the stopwordstxt file in the following location

ltelasticsearch installation directorygtconfiganalysisb Edit and save the stopwordstxt filec If data is indexed before you edit the stopwordstxt file manually delete

the indexes restart Elasticsearch restart MDM and then reindex the data

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 18: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

18 copy Informatica Proprietary and Confidential

Smart Search Configuration Table Searchable columns configurations are stored as SEARCHABLE_CO xml in C_REPOS_CO_CS_CONFIG table

ES node URL configurations are stored as INFRASTRUCTURE xml in C_REPOS_CO_CS_CONFIG table

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 19: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

19 copy Informatica Proprietary and Confidential

BE Search APIrsquos Search call to a Smart Search Enabled Entity can be made using a Generic Query or a Fielded Query

-gt In Simple Search Query the search term will be looked up or searched for in all the Business Entity fields while In fielded query the search is only carried out on that particular business entity field

Generic Search or a Simple Search can be made using qrsquo

-gt httphostportcmxcsltorsIdgtPersonq=John

Fielded Search can be made using the fqrsquo

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=lsquoJOHNrsquo

Exact Queries Search on BE field Parameters

-gt httphostportcmxcsltorsIdgtPersonaction=queryampfilter=firstName=JIM

Filtered Query Search Adds a filter to the search results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampfilters=AddressesAddresscityName=Washington

Facets provide an option of group-based filtering

-gt httphostportcmxcsltorsIdgtPersonfq=firstName=Johnampfacets=displayName

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 20: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

20 copy Informatica Proprietary and Confidential

Contd To get the data from child entities use the depth option Note the search is always done on all searchable

fields (including child) The depth option is to include the data from the child in the response

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5

To sort the data on a certain field

-gt httphostportcmxcsltorsIdgtPersonq=Johnamporder=lastName

To fetch the higherlower number of records Or restricts the number of records (parent)

-gt httphostportcmxcsorsIdPersonq=JohnamprecordsToReturn=3

Range Filter

-gt httphostportcmxcsltorsIdgtPersonfq=displayName=STEVEampfilters=age=[3545]

To suppress the links in results

-gt httphostportcmxcsltorsIdgtPersonq=Johnampdepth=5ampsuppressLinks=true

defaultPageSize Changes the number of records for each returned page-gt httphostportcmxcslt orsId gtPerson1415depth=2ampdefaultPageSize=20

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 21: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

21 copy Informatica Proprietary and Confidential

E360 Search Call ndash BE View

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 22: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

22 copy Informatica Proprietary and Confidential

To configure ES rest call in web browser refer KB 566891

List all the indices

httpshost9200_catindicesv=

This will list down all the BE indices along with their health status amp docs count

To check for all the columns associated into Specific Indiceshttpslthostnamegt92004d5354312e375054-

organization_mapping_docpretty=

Check the BE data record

httpslthostnamegt92004d5354312e375054-organization_docltBO RowIDgtpretty=

Troubleshoot SS outside MDM ndash via REST calls

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 23: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

23 copy Informatica Proprietary and Confidential

Can use the Provisioning tool to configure search filters and the order in which these appear in the Data Director Search page

Configuration gets stored in C_REPOS_COMPONENT_INSTANCE table

MDM 104 Features Re Ordering of Filters column in Smart Search

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 24: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

24 copy Informatica Proprietary and Confidential

Can create custom Elasticsearch index settings from the Provisioning tool You can configure custom analyzers tokenizers token filters and character filters for the Elasticsearch index

Externalize the ElasticSearch Analyzer Chain

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 25: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

25 copy Informatica Proprietary and Confidential

ES ndash Different Architecture Set Up

siperian-mrmear

Elastic Client

Hub Server

siperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Smaller Implementations ndash Fewer BEs and lower volumes Good for smaller incremental loads and when E360 is used for searches

Option 1 ndash Co-located with HubProcess Servers

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 26: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

26 copy Informatica Proprietary and Confidential

Option 2 ndash Dedicated ES Node

siperian-mrmear

Elastic Client

Hub Serversiperian-cleanseear

Elastic Client

Process Server

Machine A

ElasticsearchNode

Database

Machine B

Fewer BEs with high volume OR More BEs with lower volume Daily load with some API activity for CreateUpdateSearch

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 27: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

27 copy Informatica Proprietary and Confidential

Option 3 ndash Co-located Cluster in a multi-node MDM

Hub

Elastic Search

Machine A

ElasticsearchNode

Database

Process

HubMachine B

Process

ElasticsearchNode

Elastic Cluster

Multiple BEs with medium volume data Good amount of daily loads E360 search users API activity for createsupdatessearches

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 28: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

28 copy Informatica Proprietary and Confidential

Option 4 ndash Clustered ES in multi-node MDM

Hub

Elastic Search

Machine A

Database

Process

Elastic Cluster

ElasticsearchNode N

Machine N

ElasticsearchNode 1

Machine 1

ElasticsearchNode 2

Machine 2

HubMachine B

Process

HubMachine Z

Process

High Volume Systems with Frequent Loads Heavy API usage and E360 Searches N should always be an Odd number

MDM Cluster

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 29: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

29 copy Informatica Proprietary and Confidential

Performance amp Best Practices Run indexing on the Look Up tables first

Need to execute indexing job on all respective BO which are participating in the Searchable BE

CleanTable API does not delete the indexes we need to manually delete indices amp regenerate them Delete ES REST API

Simulate a change to searchable column through provisioning

Avoid running Indexing jobs in parallel

Avoid running 2 load jobs in parallel if they have searchable fields

Do not use too many search fields ie limit searchable fields on BE (parentchild) There are limitations on the number of columns in a document in ES (Limit of nested fields (default = 50) has been reached)

There are limitations on the amount of data that is returned by ES REST calls The more the number of indexed columns the more the chances of hitting this limit (104857600 bytes of data as a response to a REST call)

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 30: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

30 copy Informatica Proprietary and Confidential

Contd Remove the searchable checkbox on non usedextra BErsquos (eg Customer360 Accelerator) This slows down the performance of jobs that need to delete the indexes

Limit the use of facets

Facet field should be chosen on a field with low entropy that is fields with a low set of unique values

Retrieve records till a usable depth

Increase auto commit property (esindexrefreshinterval)

There should be a monitoring tool which keep an eye on the disk space of ES server ES will mark all indexes read only if there is less than 15 free disk space

indexblocksread_only_allow_delete falsersquo

To increase the mmapfs count which is used by Elasticsearch to store its indices (KB 589503)

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 31: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

31 copy Informatica Proprietary and Confidential

Known Limitations Case sensitive mismatch in UI amp DB values when configured Facets on Searchable columns

Search results doesnrsquot get displayed when lsquorsquo is used in Search

No way to perform search on multiple BErsquos in a single call

Provision to enable role based restriction to access specific BEs sharing same root BO for Search Queries and BES

For Auto merge of child records it performs GET (instead of MultiGet) for each record ndash which leads to job failure

Auto merge job doesnt obsolete redundant ES indices resulting in duplicate search results

Fixed Limitation

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 32: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

32 copy Informatica Proprietary and Confidential

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33
Page 33: Multidomain MDM 10...Suggester – Type-ahead search for Searchable field. Do not enable suggester for sensitive data. Filterable – Enables Filtering on a Field. Search Analyzer

`

Thank You

  • Slide Number 1
  • Housekeeping Tips
  • Slide Number 3
  • Safe Harbor
  • Agenda
  • Overview
  • Smart Search in E360 UI
  • Migration Strategy for MDM 104
  • Elastic Search - High Level Architecture
  • Install amp Configure Smart Search
  • SS Properties
  • Configuring Process Servers
  • Configuring Node(s) in Provisioning
  • Configuring Searchable BE in Provisioning
  • Indexing Job or Load Job
  • Execute Smart Search Indexing Job
  • Stopwords
  • Smart Search Configuration Table
  • BE Search APIrsquos
  • Contd
  • E360 Search Call ndash BE View
  • Troubleshoot SS outside MDM ndash via REST calls
  • Slide Number 23
  • Slide Number 24
  • ES ndash Different Architecture Set Up
  • Option 2 ndash Dedicated ES Node
  • Option 3 ndash Co-located Cluster in a multi-node MDM
  • Option 4 ndash Clustered ES in multi-node MDM
  • Performance amp Best Practices
  • Contd
  • Known Limitations
  • Slide Number 32
  • Slide Number 33

Chateau - ldartools.com · Included Chateau Innovations ⚫Enhanced Picklist Controls ⚫Customizable Audits, Alerts and Porches ⚫Comprehensive Dashboard ⚫Track Processor, Suggester,

Chateau - ldartools.com · Included Chateau Innovations ⚫Enhanced Picklist Controls ⚫Customizable Audits, Alerts and Porches ⚫Comprehensive Dashboard ⚫Track Processor, Suggester,

Documents
Feature suggester

Feature suggester

Documents
Seasonal Variation of the Physico-chemical Parameters and ... C 9 16.pdf · accordance with the national standard STAS 6953-81. Filterable residue determination was carried out in

Seasonal Variation of the Physico-chemical Parameters and ... C 9 16.pdf · accordance with the national standard STAS 6953-81. Filterable residue determination was carried out in

Documents
Smart measurements and analysis for software quality ...Smart measurements and analysis for software quality enhancement 3 - the development of techniques and tools, Metrics Suggester

Smart measurements and analysis for software quality ...Smart measurements and analysis for software quality enhancement 3 - the development of techniques and tools, Metrics Suggester

Documents
1 Virology - Chapter 13 Not responsible for details of Protein & genome synthesis pp 399-401 A little history… “Filterable viruses” Bacteriophages Wendell

1 Virology - Chapter 13 Not responsible for details of Protein & genome synthesis pp 399-401 A little history… “Filterable viruses” Bacteriophages Wendell

Documents
POET Biorefining - PermitA. POET Biorefining - Laddonia, LLC shall emit less than 50.0 tons of total PM10 (condensable and filterable) in any consecutive 12-month period from the entire

POET Biorefining - PermitA. POET Biorefining - Laddonia, LLC shall emit less than 50.0 tons of total PM10 (condensable and filterable) in any consecutive 12-month period from the entire

Documents
Columbia Municipal Power Plant - DNR. Columbia Municipal Power Plant shall control filterable particulate matter emissions from Boilers 6 and 7 using fabric filters. B

Columbia Municipal Power Plant - DNR. Columbia Municipal Power Plant shall control filterable particulate matter emissions from Boilers 6 and 7 using fabric filters. B

Documents
Search search search

Search search search

Technology
1 /Reference · Project: 001266 . 2.0 SUMMARY OF RESULTS OMYA. Inc. Florence, VT Particulate matter was measured as a non-filterable sample fraction collected by a filter media and

1 /Reference · Project: 001266 . 2.0 SUMMARY OF RESULTS OMYA. Inc. Florence, VT Particulate matter was measured as a non-filterable sample fraction collected by a filter media and

Documents
DRAFT SOUTH AFRICAN STANDARD (DSS): PUBLIC ......ASTM D5907, Standard test methods for filterable matter (total dissolved solids) and nonfilterable matter (total suspended solids)

DRAFT SOUTH AFRICAN STANDARD (DSS): PUBLIC ......ASTM D5907, Standard test methods for filterable matter (total dissolved solids) and nonfilterable matter (total suspended solids)

Documents
DynaSand™ Continuous Filters - Circuit€¦ · DynaSand Sand filter Mechanical filtration Mechanical filters retain suspended filterable matter without precipitants and are used

DynaSand™ Continuous Filters - Circuit€¦ · DynaSand Sand filter Mechanical filtration Mechanical filters retain suspended filterable matter without precipitants and are used

Documents
Introduction to Virology Class I VirusesEarly History • 1889 - Beijerinck, Tobacco Mosaic Virus, “filterable” • 1911 - Rous, first cancer virus (Rous Sarcoma) • 1915 - Twort,

Introduction to Virology Class I VirusesEarly History • 1889 - Beijerinck, Tobacco Mosaic Virus, “filterable” • 1911 - Rous, first cancer virus (Rous Sarcoma) • 1915 - Twort,

Documents
MATS Solutions for Oil-Fired Power Plants 2013.08cdn.wheelabrator.us/Publications/particulate... · Electrostatic Precipitators (ESPs) for filterable particulate collection and opacity

MATS Solutions for Oil-Fired Power Plants 2013.08cdn.wheelabrator.us/Publications/particulate... · Electrostatic Precipitators (ESPs) for filterable particulate collection and opacity

Documents
elasticsearch suggesters...CONTEXT SUGGESTER •Used with completion suggester •Two types of context: • Category • Geo •Requires special mapping • Increased index size of

elasticsearch suggesters...CONTEXT SUGGESTER •Used with completion suggester •Two types of context: • Category • Geo •Requires special mapping • Increased index size of

Documents
Introduction to Virology Class I Viruses - WOU Homepageboomers/FullProf06/teachingstuff/331stuff/05vir1PPT.pdf · Early History • 1889 - Beijerinck, Tobacco Mosaic Virus, “filterable”

Introduction to Virology Class I Viruses - WOU Homepageboomers/FullProf06/teachingstuff/331stuff/05vir1PPT.pdf · Early History • 1889 - Beijerinck, Tobacco Mosaic Virus, “filterable”

Documents
About the Assessment Guide - AWG · About the Assessment Guide 4 Assessment Catalog The Assessment Catalog is a searchable, filterable database of student SEL competency assessments

About the Assessment Guide - AWG · About the Assessment Guide 4 Assessment Catalog The Assessment Catalog is a searchable, filterable database of student SEL competency assessments

Documents
CS 229: Machine Learning Final Report Identifying Driving ...cs229.stanford.edu/proj2013/Karol-Identifying... · Robert F. Karol Project Suggester: Danny Goodman from MetroMile December

CS 229: Machine Learning Final Report Identifying Driving ...cs229.stanford.edu/proj2013/Karol-Identifying... · Robert F. Karol Project Suggester: Danny Goodman from MetroMile December

Documents
Bontiac's 1973 Grand Prix. · Representing Today's Suggester James Parmakian 26 Deliver Them from Obsolescence Charles Albano 28 Idea Programs and Management Attitudes AmbyBair32

Bontiac's 1973 Grand Prix. · Representing Today's Suggester James Parmakian 26 Deliver Them from Obsolescence Charles Albano 28 Idea Programs and Management Attitudes AmbyBair32

Documents
DEPARTMENT OF THE NAVY CHIEF 2000 the suggestion against normal job expectancy: (a) Does the suggester have the authority to put the idea into effect (if yes, is the suggestion within

DEPARTMENT OF THE NAVY CHIEF 2000 the suggestion against normal job expectancy: (a) Does the suggester have the authority to put the idea into effect (if yes, is the suggestion within

Documents
Succint Filterable Decryption

Succint Filterable Decryption

Documents
Technical Report WRD92038 - Territory Stories: Home · Technical Report WRD92038 ... SUSPENDED SOLID AND METAL ... kjeldahl nitrogen, total suspended solids, and residual and filterable

Technical Report WRD92038 - Territory Stories: Home · Technical Report WRD92038 ... SUSPENDED SOLID AND METAL ... kjeldahl nitrogen, total suspended solids, and residual and filterable

Documents
GemAccountant Specs · Gemstones, Jewelry, Manufactured Goods, WIP) • Inventory Transaction History shows each inventory transaction filterable by dates and transactions types and

GemAccountant Specs · Gemstones, Jewelry, Manufactured Goods, WIP) • Inventory Transaction History shows each inventory transaction filterable by dates and transactions types and

Documents
Overview - TFEIP Secretariat Web viewThis reflects the difference in emissions between manual feed and ... A range of filterable PM measurement methods are applied around the world

Overview - TFEIP Secretariat Web viewThis reflects the difference in emissions between manual feed and ... A range of filterable PM measurement methods are applied around the world

Documents
Definition and Properties of a Virus Viruses are filterable agents. Viruses are obligate intracellular parasites. Viruses cannot make energy or proteins

Definition and Properties of a Virus Viruses are filterable agents. Viruses are obligate intracellular parasites. Viruses cannot make energy or proteins

Documents
Is Quantitative Bioanalysis ready for HRMS? · At the core of UNIFI is Oracle database – DATA IS ORGANIZED – DATA IS MINEABLE, FILTERABLE, Linked from beginning to end ©2012

Is Quantitative Bioanalysis ready for HRMS? · At the core of UNIFI is Oracle database – DATA IS ORGANIZED – DATA IS MINEABLE, FILTERABLE, Linked from beginning to end ©2012

Documents
Mobile Application Development · For example the ArrayAdapter class implements the Filterable interface and contains a default filter implementation called ArrayFilter as an inner

Mobile Application Development · For example the ArrayAdapter class implements the Filterable interface and contains a default filter implementation called ArrayFilter as an inner

Documents
I 1Reference - US EPA · Project: 001266 OMYA, Inc. Florence, VT 2.0 SUMMARY OF RESULTS Particulate matter was measured as a non-filterable sample fraction collected by.. a filter

I 1Reference - US EPA · Project: 001266 OMYA, Inc. Florence, VT 2.0 SUMMARY OF RESULTS Particulate matter was measured as a non-filterable sample fraction collected by.. a filter

Documents
High growth potential and activity of 0.2 P m filterable ...1 High growth potential and activity of 0.2 P m filterable bacteria habitually present in coastal seawater Yumiko Obayashi

High growth potential and activity of 0.2 P m filterable ...1 High growth potential and activity of 0.2 P m filterable bacteria habitually present in coastal seawater Yumiko Obayashi

Documents
GUM ARABIC SOLUTION, LEVOROTATORY AND FILTERABLE ... · ICON® GUM GUM ARABIC SOLUTION, LEVOROTATORY AND FILTERABLE COMPOSITION Levorotatory gum arabic (E 414) 20%, sulphur dioxide

GUM ARABIC SOLUTION, LEVOROTATORY AND FILTERABLE ... · ICON® GUM GUM ARABIC SOLUTION, LEVOROTATORY AND FILTERABLE COMPOSITION Levorotatory gum arabic (E 414) 20%, sulphur dioxide

Documents
Introduction to Virology Scott M. Hammer, M.D.. Landmarks in Virology Introduction of concept of ‘filterable agents’ for plant pathogens (Mayer, Ivanofsky,

Introduction to Virology Scott M. Hammer, M.D.. Landmarks in Virology Introduction of concept of ‘filterable agents’ for plant pathogens (Mayer, Ivanofsky,

Documents