Federator.ai Data Collection Methods and Functionalities · Virtual SAN (vSAN) is a hyper-converged software-defined storage infrastructure developed by VMware. VMware vSAN abstracts

© 2017 ProphetStor Data Services, Inc. 0

Federator.ai® Data Collection Methods and Functionalities

1 © 2018 ProphetStor Data Services, Inc. All Rights Reserved



Contents

Introduction ...................................................................................................................................... 2

Federator.ai® overview ..................................................................................................................... 2

Different types of data collection ....................................................................................................... 2

Metadata ................................................................................................................................................................ 2

Performance metrics .............................................................................................................................................. 2

Events ..................................................................................................................................................................... 2

Logs ........................................................................................................................................................................ 2

S.M.A.R.T. ............................................................................................................................................................... 2

Collecting data from multiple layers .................................................................................................. 2

Application layer .................................................................................................................................................... 2

Virtualization layer ................................................................................................................................................. 3

Infrastructure layer ................................................................................................................................................ 3

Incomplete/missing data handling ......................................................................................................................... 5

Analytics of data collected ................................................................................................................. 5

Cross-layer correlation ........................................................................................................................................... 5

Performance anomaly detection ........................................................................................................................... 5

Performance prediction ......................................................................................................................................... 5

Disk health trending prediction .............................................................................................................................. 6

Memory error detection/prediction ...................................................................................................................... 6

Log/event noise reduction ..................................................................................................................................... 6



Introduction

In the big data era, IT operations are the de facto

keys to augment business growth, and machine data

plays a crucial role in IT operation efficiency. This

document describes what and how data is collected

and represented for IT operations using

Federator.ai®.

Federator.ai® overview

ProphetStor's Federator.ai® is an Artificial

Intelligence for IT Operations (AIOps) platform

providing insights and foresights by collecting

machine data from application, virtualization, and

infrastructure layers. It uses state-of-the-art artificial

intelligence (AI) technology to bring insights into

data centers and lets enterprises make better

decisions from the foresights it provides.

Different types of data collection

Federator.ai® uses the agents installed in different

environments and platforms to collect data. The

different types of data collection include the

following:

Metadata

Metadata are data that provide descriptions of and

information about data sources and may indicate the

relationships among different IT devices

Performance metrics

Performance metrics describe the loading and

utilization of system resources, including availability,

response time, latency, throughput, service time,

completion time, power consumption, processing

speed and so on.

Events

An "event" is the description of a user's actions or

system occurrences detected by software. Events are

triggered when an operation runs, or the system

processes meet its' criteria. They usually identify

operational anomalies.

Logs

Logs record system event occurrences, application

processes, service transitions, and the

communication information between different

software and systems.

S.M.A.R.T.

S.M.A.R.T. (or SMART) stands for Self-Monitoring,

Analysis, and Reporting Technology and is a

monitoring mechanism included in hard drives, SSDs,

and NVMes. SMART data indicates the various

attributes representing the health state of a given

hard drive and is used by Federator.ai® as the key

parameter set for predicting imminent disk failure.

Collecting data from multiple layers

Federator.ai® supports cross-layer data collection

from application, virtualization, and infrastructure

layers. Federator.ai® uses agents installed on the

monitored platforms from different layers to collect

data and sends the data to the Federator.ai® server

for data analysis, monitoring and prediction. The

following describes the details of supported systems

from different layers and how the data is collected.

Application layer

APM

Federator.ai® seamlessly integrates other third-party

Application Performance Monitoring & Management

(APM) tools including AppDynamics, New Relic, and

BMC. The following shows the collected data types

and collection methods.

Collected data types: Metadata, events, and

logs.

Data collection: Install a Federator.ai® agent to

communicate with third-party APM's APIs to get

metadata and events.

Log data: An agent installation is required on

each monitored node to collect log data.



Virtualization layer

Containers

Federator.ai® supports container-base virtualization

including Kubernetes and OpenShift. The following

describes the collected data types and collection

methods.

Collected data types: Metadata, performance

metrics, events, and logs.


communicate with Kubernetes and OpenShift

APIs to get metadata, performance metrics, and

events.

Log data: An agent installation is required on

each monitored node to collect log data.

Hypervisor

Federator.ai® supports VMware hypervisor, and

utilizes VMare’s vCenter, which is a unified platform

that manages the vSphere environment. The

following describes the collected data types and

collection methods.


metrics, events, logs, and S.M.A.R.T.


communicate with a vCenter via its API to get

the metadata, performance metrics, events, and

logs.

S.M.A.R.T. data: Each ESXi host needs to be

installed with smartmontools (SMART

Monitoring Tools), which collects SMART data

from disks. Federator.ai® agent communicates

with smartmontools via ESXi’s SSH. Thus, the

SSH access needs to be enabled on each ESXi

host.

HCI

Hyper-converged infrastructure (HCI) is the

convergence of a hypervisor, server, and storage into

a single system. Nutanix is software-based HCI that

runs cross-virtual hypervisors on one platform. The


collection methods.


metrics, events, and S.M.A.R.T.


communicate with Nutanix CVM API and get the

metadata, performance metrics, and events.

S.M.A.R.T. data: Except for Federator.ai® agent,

smartmontools needs to be installed on each

host.

Infrastructure layer

The infrastructure layer includes bare metal servers

and storage systems.

Linux/Windows hosts

Federator.ai® supports OS-based infrastructures

including Linux and Windows. The following

describes the collected data types and collection

methods.


metrics, logs, and S.M.A.R.T.

Data collection: Install one agent on each

monitored host to collect metadata,

performance metrics, and logs.

S.M.A.R.T. data: Except for Federator.ai® agent,

smartmontools needs to be installed on each

host.

vSAN

Virtual SAN (vSAN) is a hyper-converged software-

defined storage infrastructure developed by

VMware. VMware vSAN abstracts physical storage

into virtual pools and unifies resources under its

policy-based management. The following describes

the collected data types and collection methods.


metrics, events, and logs.


communicate with vCenter’s API to get

metadata, performance metrics, events, and logs.



Ceph

Ceph is a distributed storage system designed to run

on commodity hardware and provides highly

scalable object-, block- and file-level storage. The


collection methods.


metrics, and S.M.A.R.T.

Data collection: Install Ceph built-in

diskprediction plugin or Federator.ai® agent to

collect data from Ceph clusters.

The following table shows the overview of data collection from different layers and platforms.

Layer Category Platform/System Collected data types How data is collected

Applications APM AppDynamics

New Relic

BMC

Metadata

Events

An agent is installed to connect to third- party APM API.

Logs An agent needs to be installed on each monitored node to collect logs.

Virtualization Containers Kubernetes

OpenShift

Metadata

Performance Metrics

Events

An agent is installed to connect to the container API.

Logs An agent needs to be installed on each monitored node to collect logs.

Hypervisor VMware vSphere Metadata

Performance Metrics

Events

Logs

An agent is installed to connect to a vCenter via API.

S.M.A.R.T. Each ESXi host needs to be

installed with

smartmontools.

SSH needs to be enabled on

each ESXi host.

An agent is installed to

connect to smartmontools

via SSH.

HCI Nutanix Metadata

Performance Metrics

Events

An agent is installed to connect to Nutanix CVM API.

S.M.A.R.T. Each host needs to be

installed with

smartmontools.

Each host needs to be

installed with an agent.



Layer Category Platform/System Collected data types How data is collected

Infrastructure OS Linux/Windows Metadata

Performance Metrics

Logs



S.M.A.R.T. Each host needs to be

installed with

smartmontools.



Storage vSAN Metadata

Performance Metrics

Events

Logs

An agent is installed to connect

to a vCenter via API.

Ceph Metadata

Performance Metrics

S.M.A.R.T.

Enable the Ceph built-in

diskprediction plugin or install

Federator.ai® agent to collect

data.

Incomplete/missing data handling

Incomplete/missing data refers to the situation when

the data sent from an agent has insufficient data or

some of the fields in the data do not exist.

Federator.ai® can still generate a prediction when

this situation occurs. However, the predicted results

cannot provide the same degree of accuracy as would

sufficient data. Federator.ai® uses a confidence value

(a five-star rating system) to indicate the quality of

the prediction results.

For example, when some of the S.M.A.R.T. data from a

disk do not exist, the corresponding confidence

values will have fewer stars.

Analytics of data collected

The following paragraphs describe how collected

data is used and how Federator.ai® brings out the

insightful information from this data to enhance IT

operation efficiency.

Cross-layer correlation

Federator.ai® analyzes metrics and resource

utilization across the application, virtualization, and

infrastructure layers. It also provides visualization of

the relationships between entities within each layer.

This cross-layer correlation helps IT administrators

proactively act on an entity that may be adversely

affected by another entity. The benefits of this feature

are to reduce MTTR (Mean Time To Repair) and

increase MTBF (Mean Time Between Failures).

Performance anomaly detection

Federator.ai® uses performance metrics to analyze

the performance use and resource utilization status

and then detect the anomalies. Performance anomaly

detection provides IT administrators with warnings on

unusual behavior that may adversely affect system

operations, allowing for a quick diagnosis of and

resolution for the problem.

Performance prediction

Using the state-of-the-art algorithms, Federator.ai®

predicts future performance trends and enables users

to optimize resource utilization.



Disk health trending prediction

With the patented AI technology, Federator.ai® uses

S.M.A.R.T. and disk metadata for disk health trending

prediction. It also uses performance metrics of disks

and hosts as supplementary data to reinforce

prediction accuracy.

Memory error detection/prediction

Federator.ai® also applies AI to detect and predict

potential memory errors from events and logs of

physical hosts.

Log/event noise reduction

IT administrators typically have problems in

understanding system logs and events as many of

them are redundant messages. In addition, too much

redundant information may cause operation

inefficiency and waste resources. Federator.ai® can

filter out the log/event noise and extract the

significant information from logs and events to

benefit IT operations.

The table below shows how data is used in Federator.ai®.

Data sources

AI features Metadata

Performance metrics

Disk S.M.A.R.T. Log/event

Cross-layer correlation

Performance anomaly detection from baseline

Performance prediction

Disk health trending prediction

Memory error detection/prediction

Log/event noise reduction

Documents

Federator.ai Data Collection Methods and Functionalities · Virtual SAN (vSAN) is a hyper-converged software-defined storage infrastructure developed by VMware. VMware vSAN abstracts