HCI 1.1.0 Release Notes · What'snewin1.1.0 Actions HCI dataconnectionscannowperformactionsondatasources.Previously, dataconnectionscouldonlyreaddatafromdatasources,butnowtheycan:

Hitachi Content IntelligenceHCI 1.1.0 Release NotesHCI software build 1.1.0.62

RN-HCI004-001/20/2017

© 2017 Hitachi Data Systems Corporation. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying and recording, or stored in a database or retrieval system forcommercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data SystemsCorporation (collectively, “Hitachi”). Licensee may make copies of the Materials provided that any suchcopy is: (i) created as an essential step in utilization of the Software as licensed and is used in no othermanner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials."Materials" mean text, data, photographs, graphics, audio, video and documents.

Hitachi reserves the right to make changes to this Material at any time without notice and assumes noresponsibility for its use. The Materials contain the most current information available at the time ofpublication.

Some of the features described in the Materials might not be currently available. Refer to the most recentproduct announcement for information about feature and product availability, or contact Hitachi DataSystems Corporation at https://support.hds.com/en_us/contact-us.html.

Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicableHitachi agreements. The use of Hitachi products is governed by the terms of your agreements with HitachiData Systems Corporation.

By using this software, you agree that you are responsible for:

1) Acquiring the relevant consents as may be required under local privacy laws or otherwise fromauthorized employees and other individuals to access relevant data; and

2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance withrelevant laws.

Notice on Export Controls. The technical data and technology inherent in this Document may be subject toU.S. export control laws, including the U.S. Export Administration Act and its associated regulations, andmay be subject to export or import regulations in other countries. Reader agrees to comply strictly with allsuch regulations and acknowledges that Reader has the responsibility to obtain licenses to export, re-export, or import the Document and any Compliant Products.

Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries.

AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eServer, FICON, FlashCopy,IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/OS, z9, z10, z13,z/VM, and z/VSE are registered trademarks or trademarks of International Business Machines Corporation.

Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft,the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQLServer, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo, Windows Azure, WindowsPowerShell, Windows Server, the Windows start button, andWindows Vista are registered trademarks ortrademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission fromMicrosoft Corporation.

All other trademarks, service marks, and company names in this document or web site are properties oftheir respective owners.

https://support.hds.com/en_us/contact-us.html

What's new in 1.1.0

Actions

HCI data connections can now perform actions on data sources. Previously,data connections could only read data from data sources, but now they can:

• Create new files in data sources

• Delete files from data sources

• Change the metadata on files in data sources

To perform an action, you can:

• Add an Action stage to a pipeline

• Add a data connection as an output for a workflow

New built-in data connections

• HCP Anywhere data connection

• Amazon® S3 data connection

• S3 Compatible data connection

New built-in stages

• Field Parser — Takes a specified field, examines that field's values, andsplits those values into smaller segments named tokens. You can thenselect the tokens that you want and add them as values to other fields.

3

HCI 1.1.0 Release Notes

• Geocoding— Adds new fields to documents that contain latitude andlongitude data. For each set of coordinates, the stage adds new fieldsthat contain the corresponding city, state or providence, country, andtime zone

• Read Lines— Creates a new document for each line in a content stream.

• Reject Documents— Reports a failure for each document that itprocesses. You can use conditional statements to filter problemdocuments into a Reject Documents stage, preventing those documentsfrom progressing further in your pipeline.

• XML Formatter — Takes a list of document fields, formats those fields asXML, and then outputs them to a stream.

Packages

Previously, you could import and export only workflow components. Inversion 1.1.0, the packages feature now lets you import and export systemconfiguration settings and your custom plugins, in addition toworkflow components.

New and improved service management options

With release 1.1.0, you can now run the Repair operation on HCI services.This operation shuts down and restarts an unresponsive service.

Also, changing the configuration settings for a service is now easier:

• A new Configure operation has been added to the Administration App.

• The applicable REST API calls have been simplified.

New service - Workflow-Agent

Release 1.1.0 adds the Workflow-Agent service. This service lets you specifywhich instances in your system can be used to run your workflows.

Tip: For best workflow performance, run this service on as many instancesas possible.

What's new in 1.1.0

4


Log download tool

Each HCI instance now includes the hci_log_download script, which lets youeasily collect system logs and diagnostic information from all instances inthe system.

Plugin SDK test harness

The HCI plugin SDK now includes a test harness that you can use tovalidate your custom HCI plugins before uploading them to the system. Youdo not need a running HCI system to use the harness for testing yourplugins.

Licensing changes

In release 1.0, HCI usage was licensed based on the amount of data readfrom your data sources. In release 1.1.0, this licensing model has beenreplaced with one based on units called HCI cores.

Updating HCI

Release 1.1.0 adds a new Updates page in the Administration App that youcan use for updating your HCI system to later HCI releases. You can alsouse the administrative REST API and CLI to initiate and monitor HCIupdates.

REST API changes

The HCI administrative and search REST APIs have been updated to version1.1.0.

This section lists changes in the administrative REST API from release 1.0.There have been no changes to the search REST API.

• The REST API endpoints for importing and exporting workflowcomponent bundles have been moved from /services to /workflows.

• All instances of /directoryServers have been changed to/identityProviders.

• POST /services/configure now requires that you specify an action as partof the request body.

What's new in 1.1.0

5


• The id property for each HCI service has changed from an integer to amore descriptive string value.

For example, the id for the Index service has changed from 7 tocom.hds.ensemble.service.SolrService.

This change affects both the request and response bodies forall REST API endpoints that include /services.

• To make service configuration changes easier:

o The ServiceInfoModel schema now returns lists of both the instanceswhere a service is configured to run and the instances on which aservice is not configured to run.

o The ServiceConfigModel schema now requires serviceId andserviceName fields instead of a ServiceInfoModel object.

These changes affect both the request and response bodies forall REST API endpoints that include /services.

For information on specific REST API methods, in the Administration App,click on the help icon. Then:

• To view the administrative REST API methods, click on Admin API.

• To view the API methods used for performing searches, click onSearch API.

Changes to existing built-in stages

• Text and Metadata Extraction — The input and output stream names forthis stage are now configurable.

• Appender — Can now append field values to streams or other fieldvalues.

What's new in 1.1.0

6


• Date Conversion — This stage includes a new RegEx pattern formatching the file_modified_date field format produced by the Textand Metadata Extraction. In release 1.1.0, newly created instances ofthis stage include the new RegEx pattern by default.

Note: When you import a 1.0 workflow component bundle that includesthe Date Conversion stage, the imported Date Conversion stage doesnot automatically receive this new RegEx pattern. You need to manuallyadd it to the stage.

• Content Class Extraction — New Stream or field content size limitsetting.

• Archive expansion stages — New Expanded file path setting. Forinformation, see:

o ZIP Expansion

o TAR Expansion

o PST Expansion

o Mbox Expansion

o Email Expansion

• Conditional statements — Now include the equals_ignore_case andcontains_ignore_case comparators.

New Docker requirements

HCI now requires each instance to run Docker version 1.10.3 or later.

System requirements and sizing

This section lists the hardware, operating system, and networkrequirements for running an HCI system with one or more instances.

Operating system requirements

To be an HCI instance, each server or virtual machine you provide must berunning a 64-bit Linux distribution that is capable of running Docker 1.10.3and later.


7


HCI has been qualified using these Linux distributions:

• Fedora 24

• CentOS 7.2

• Ubuntu 16.04 LTS

Docker requirements

HCI requires Docker containerization software. Each server or virtualmachine you provide must be capable of running Docker 1.10.3 or later.

Important: For a production HCI system, do not run the Dockerdevicemapper storage driver in loop-lvmmode. This can cause slow HCIperformance or, on certain Linux distributions, HCI may not have enoughspace to run.

Virtual machine host requirements

To deploy the HCI example OVF, your virtual machine host must runVMware ESXi version 6.0 or later.

The HCI example OVF has been qualified on these virtual machine hostplatforms:

• VMware ESXi 6.0

• Hitachi Unified Compute Platform (UCP) 4.1.0


8


Hardware resources

This table lists:

• Minimum —The minimum amounts of RAM, CPU, and available diskspace required to run an instance of HCI.

Every server or virtual machine you provide as an HCI instance mustmeet these minimums.

• Recommended— The amounts of RAM, CPU, and available disk spacethat HDS recommends for HCI instances.

A system in which all instances meet or exceed these recommendationscan index more documents and can process documents faster than asystem whose instances meet only the minimum requirements.

Resource Minimum Recommended

RAM 16 GB 32 GB

CPU 4-core 8-core

Available disk space 50 GB 500 GB

Important:

• Each HCI instance uses all available RAM and CPU resources on theserver or virtual machine on which it's installed.

• A large number of factors determine how many documents HCI canindex and how fast it can process them, including: the number ofdocuments to be indexed; the contents of those documents; whatsearch features (such as sorting) the index supports; the number offields in the index; the number of users querying the system; and so on.

Depending on how you use HCI, you may require additional hardwareresources to index all the documents you want and at the rate yourequire.

For more information on sizing your HCI system, see the AdministartorHelp, which is available from the Administration App.


9


Network connection requirements

IP address requirementsEach HCI instance must have a static IP address.

Important: If the IP address of any instance changes, see "HandlingIP address changes" in the administrator help, which is accessible from theAdministration App.

Allowing access to external resourcesYou need to configure your network to ensure that each HCI instance canaccess the external resources you want to use, such as:

• The data sources where your data is stored

• Identity providers for user authentication

• Email servers that you want HCI to use for sending email notifications

• Any external search indexes (for example, HDDS indexes) that you wantmake accessible through HCI

Port usageThe following table contains information about the ports used by each HCIinstance.

Port Purpose Required accessibility

8000 Access to HCIadministrativeinterfaces:

• AdministrationApp

• AdministrativeREST API

• AdministrativeCLI

Must be accessible from both ofthese:

• Any network that requiresadministrative access to HCI

• Every other HCI instance inthe system


10


Port Purpose Required accessibility

8888 Access to HCIsearch interfaces:

• Search App

• Search RESTAPI

• Search CLI

Must be accessible from both ofthese:

• Any network that requiressearch access to HCI

• Every other HCI instance inthe system

2181: Zookeeper client port2888: Zookeeper server-server communication3888: Zookeeper leaderelection5050: Mesos-master5051: Mesos-slave5555: Watchdog JMX5601: Kibana7199: Cassandra JMX7203: Kafka8022: Watchdog SSH8080: Marathon8081: Chronos8889: Sentinel app8983: Solr9042: Cassandra9092: Kafka9200: Elastic9201: Elastic9300: Elastic9301: Elastic15050: Mesos-masterinternal18000: Admin app internal18080: Marathon internal31000-34000: High portsused by both mesos anddocker

Internal systemcommunication

Must be accessible from everyother instance in the HCIsystem.

Must not be accessible fromoutside the HCI system.


11


Supported browsers

The HCI Administration App and Search App support these web browsers:

• Google Chrome latest

• Mozilla Firefox latest

• Microsoft Edge latest

• Microsoft Internet Explorer 11

HCI documentation

The following documents contain additional information about HCI:

• Administrator Help— Contains information on using HCI to process yourdata and make it searchable. Also includes information on administeringthe HCI system.

This document is accessible from the HCI Administration App.

• Search Help— Contains information on using HCI to search for files.

This document is accessible from the HCI Search App.

• Installing HCI — Contains information on installing HCI, either onphysical servers or virtual machines that you provide.

This document is contained within the HCI-<version-number>.tgzinstallation package and is also available from within the AdministratorHelp.

• Deploying the HCI Example OVF — Contains information on deploying anHCI system for testing and evaluation using the HCI-OVF-EXAMPLE-<version-number>.tgz installation package, provided by HDS.

This document is contained within the HCI-<version-number>.tgzinstallation package.

Upgrade notes

Upgrades are not supported from version 1.0 to any later version.

HCI documentation

12


Note about using the log download tool

When using the hci_log_download script, if you specify the --output option,do not specify an output path that contains colons, spaces, or symboliclinks. If you omit the --output option, you cannot run the script from withina directory path that contains colons, spaces, or symbolic links.

Known issues

This table summarizes the known issues with HCI version 1.1.0.

Referencenumber Area affected Description

ENS-2099 Textand MetadataExtractionstage

Text and Metadata Extraction stage extracts onlythe first authorWhen a Microsoft Office document passes through theText and Metadata Extraction stage, the stage adds afield called Author to it. If a document has multipleauthors, the stage adds only the first author as a valuefor the Author field.

Note about using the log download tool

13



ENS-3692 Workflow tasks Task halted due to missing scheduler jobOccasionally, workflow tasks that have been running forlong periods of time may halt with this error message:

Task halted due to missing scheduler job

This occurs on systems where each instance:

• Runs Fedora 24

• Does not have a swap partition

Workarounds• Add a swap partition of any size to each instance.

• Use different data connections:

1. Create new data connections that access smallersubsections of your data than your original dataconnections. For example, instead of having onedata connection for an entire HCP namespace,create several data connections that accessdirectories within that namespace.

2. Replace the original data connections in the haltedworkflow with the newly created ones.

3. Run the workflow task again.

Known issues

14



ENS-3936 S3 Compatibledataconnection

When reading from HCP, S3 Compatible dataconnection skips an entire directory if one objectname contains illegal URL-encoding hexidecimalcharactersWhen you use the S3 Compatible data connection to readobjects from HCP, if an HCP object has a filename thatcontains illegal URL-encoding hexidecimal characters (forexample, file%name.txt instead of file%25name.txt),HCI fails to read any objects from the directorycontaining the object.

WorkaroundUse the either the HCP data connection or HCP MQE dataconnection to read objects from the affected directory.

Known issues

15



ENS-4051 Services,systemdeployment,update

Service fails to start during system deployment andupdateRarely, a system deployment, service managementoperation, or system update may fail because a servicefails to start. When this happens, the Administration Appis inaccessible from the instance where the failureoccurred.

The logs in the watchdog-service log directory containthis error:

Error response from daemon: Conflict. The name"<service-name>" is already in use by container<Docker-container-id>. You have to remove (orrename) that container to be able to reuse thatname.

WorkaroundRestart the Docker service on the instance where theservice failed to start. For example, if you are usingsystemd to run Docker, run:

systemctl restart docker

After restarting Docker, retry the system deployment,service management operation, or system update.

ENS-4304 Log downloadscript

Log download script fails for instances notphysically connected to the networkThe hci_log_download script to collect logs from an HCIinstance when:

• The instance has been physically disconnected fromthe network, and

• The --offline option is specified. For example:

./hci_log_download --offline

Known issues

16



ENS-4344 Databaseservice

The Database service cannot be run on additionalinstances if any instance of that service is downIn these situations, you cannot configure the Databaseservice to run on additional instances:

• The Database service has a status of Failed on one ofthe instances in the system.

• One of the instances is down and the Databaseservice is configured to run on that instance.

WorkaroundTo be able to run the Database service on additionalinstances, you need to first remove the failed Databaseservice, either by:

• Removing the failed Database service from theinstance it's configured to run on.

• Removing the failed instance from the system. Inthis case, all services running on the removedinstance are automatically relocated to otherinstances.

In either case, wait for the Database service to finishbeing removed from the applicable instance beforeattempting to run the service on additional instances.

ENS-4409 Workflowtasks, Textand MetadataExtractionstage

Workflow task halts due to error with Textand Metadata Extraction stageRarely, if you use the Text and Metadata Extraction stagein your pipeline, your workflow tasks may halt.

WorkaroundCheck the error message for the task failure. IfExtractStagePlugin appears in the message, restart theworkflow task.

Known issues

17



ENS-4422 Installing HCIdocumentation

Instructions for unpacking installation package areincorrectOn page 29 of the document Installing HCI for release1.1.0, the command:

tar -zxf HCI-<version-number>.tgz

Should instead be:

sudo -zxf HCI-<version-number>.tgz

Known issues

18


Resolved issues

This table summarizes issues resolved in the current release.


ENS-2131 AdministrationApp, workflowand systemtasks

Administration App and tasks are unresponsiveIn rare cases, the HCI system may become unresponsiveand display these behaviors:

• Workflow or system tasks may appear to be runningfor a significant amount of time without makingprogress.

• Service movement or scaling tasks may appear to bestuck on one step for a significant amount of time.

Fix: HCI no longer experiences these behaviors.

ENS-2856 Workflow tasks Workflow task halts with no errorRarely, a workflow task may halt with no error messagedue to an internal connection failure.

Fix: Workflow tasks no longer halt for this reason.

ENS-2868 HCI instances Docker version restrictionHCI did not run correctly with Docker versions 1.10 andlater.

Fix: HCI now runs correctly on Docker 1.10 and later.

ENS-3197 AdministratorHelp

Incorrect instructions for adding new instancesThe 1.0 version of the HCI Administrator Help stated toomit the -m option from the hci_setup command whenadding new instances to the system. This is incorrect;you need to specify the -m option with the IP address of atleast one existing master instance in the system.

Fix: The Administrator Help has been updated with thisinformation.

Resolved issues

19



ENS-3023 AdministratorHelp

Incorrect description for data connection settingThe 1.0 version of the HCI Administrator Help statedthat when the Include delete operations setting isenabled for an HCP MQE data connection, deleteddocuments were removed from HCI indexes. However,these documents were not actually deleted from HCIindexes when this setting was enabled. Instead, thesedocuments are marked with this field/value pair:

HCI_deleted:true

Fix: The Administrator Help topic has been updated withthis information.

ENS-3151 SearchREST APIdocumentationpage

Search REST API documentation correctionsThe search REST API documentation page now correctlystates that the following methods can return HTTP 404response codes when a user attempts to search an indexthat cannot be found:

• POST /query

• POST /federated

• POST /suggest

For information on the REST API documentation page,see the Administrator Help.

ENS-3151

ENS-3814

AdministrativeREST APIdocumentationpage

Administrative REST API documentation correctionsThese corrections have been made to the administrativeREST API documentation page:

• DELETE /pipelines/{uuid} — Attempting to deletethe default pipeline returns an HTTP 400 responsecode, not a 403.

• GET /certificates/system/modified — The pagepreviously claimed that this method can return anHTTP 400 response, which it cannot.

For information on the REST API documentation page,see the Administrator Help.

Resolved issues

20



Hitachi Data Systems

Corporate Headquarters2845 Lafayette StreetSanta Clara, California 95050-2627U.S.A.www.hds.com

Regional Contact Information

Americas+1 408 970 [email protected]

Europe, Middle East, and Africa+44 (0) 1753 [email protected]

Asia Pacific+852 3189 [email protected]

RN-HCI004-00

http://www.hds.com/

http://[email protected]/

