1EMC CONFIDENTIAL—INTERNAL USE ONLY Integration of storage device drivers to CoprHD Evgeny Roytman

1EMC CONFIDENTIAL—INTERNAL USE ONLYEMC CONFIDENTIAL—INTERNAL USE ONLY

Integration of storage device drivers to CoprHD

Evgeny Roytman


Current API for Device Access layer in CoprHD

On the device access level the API has three parts: Discovery and Metering, Monitoring, Provisioning.

Discovery API: • Scan provider to get list of storage systems• Discover storage system properties (serial number, ip address,

firmware version, etc. ), discover storage pools, discover storage ports, discover brownfield resources

• Populate all data in Cassandra/ZK which CoprHD needs for provisioning operations

• Update CoprHD task with operation status

Metering API:• Collect statistic information about storage resources:

allocated/provisioned capacity of volumes and file systems, bandwithIn, bandwithOut for volumes and file systems, snapshot capacity, etc.

• Populate Stat timeseries data in Cassandra• Update CoprHD metering task with operation status


Current API for Device Access Layer in CoprHDProvisioning API: • Provision storage resources on physical storage system • Populate physical properties for provisioned resources in Cassandra• Update CoprHD task with operation status

Monitoring API:• Collect events and alerts for storage arrays• Populate Cassandra timeseries Event table • Update ZK monitoring queue and update CoprHD task with operation

status


Current CoprHD Discovery and Metering API for devices

All CoprHD discovery drivers extend abstract base class ExtendedCommunicationInterfaceImpl which provides base implementation of ExtendedCommunicationInterface methods.

CommunicationInterface is a base interface of ExtendedCommunicationInterface. This base interface defines discovery and metering API for discovery drivers:

public void scan(AccessProfile accessProfile) throws BaseCollectionException; public void discover(AccessProfile accessProfile) throws BaseCollectionException; public void collectStatisticsInformation(AccessProfile accessProfile) throws BaseCollectionException; public void cleanup();

Each discovery driver has to provide its own implementation of CommunicationInterface methods.

ExtendedCommunicationInterface adds several “inject” methods for controller to set CoprHD controller services into device driver at discovery time. Controller injects the following services: database client, coordinator client, controller locking service, network device controller, task completer.


Current CoprHD Discovery and Metering API

Each device type should specify device driver to use for discovery and metering in device type specific context file.Example of content of discovery-hds-context.xml:<?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans.xsd">

<bean id="hds"class="com.emc.storageos.volumecontroller.impl.plugins.HDSCommunicationInterface">

<property name="hdsApiFactory" ref="hdsApiFactory" /> <property name="volumeDiscoverer" ref="volumeDiscoverer" /> <property name="partitionManager" ref="partitionManager"/>

</bean> 

<bean id="httpClient" class="org.apache.http.impl.client.DefaultHttpClient" />

<bean id="volumeDiscoverer"class="com.emc.storageos.volumecontroller.impl.hds.discovery.HDSVolumeDiscoverer"><property name="hdsApiFactory" ref="hdsApiFactory" />

</bean>

<bean id="hdsApiFactory" class="com.emc.storageos.hds.api.HDSApiFactory" init-method="init"/>

<bean id="partitionManager" class="com.emc.storageos.plugins.common.PartitionManager" />

</beans>


Current CoprHD Provisioning API• BlockStorageDevice – for block provisioning• FileStorageDevice – for file provisioning• Several operation specific interfaces which are used

internally by storage device implementations in CoprHD: CloneOperations, SnapshotOperations, MetaVolumeOperations, MirrorOperations, ExportMaskOperations, etc.

Platform specific device drivers implement BlockStorageDevice/FileStorageDevice interfaces and use platform specific implementations of operation interfaces.Method declarations in storage device interfaces follow similar pattern: void return type, arguments tied to CoprHD persistent model, reference to CoprHD task completer and CoprHD workflow step.

Implementation is self contained, directly communicates with CoprHD infrastructure (database/zk access, task completion), tightly coupled with CoprHD infrastructure services.


Current CoprHD Monitoring API

• Device specific monitoring drivers implement IMonitoringStorageSystem interface

• Drivers are invoked by CoprHD infrastructure (MonitoringJobConsumer) periodically to collect events and alerts from storage arrays

• Drivers collect events and alerts, process them and populate CoprHD database

• When driver detects that monitored device was removed from CoprHD, the driver updates ZK queue to remove monitoring job for the device


Limitations of Current CoprHD Southbound API

Tightly coupled with CoprHD infrastructure – requires driver to know how to work with CoprHD database, zookeeper, task completers, workflows.Uses persistent data model with many CoprHD housekeeping and vendor specific details.

Does not have device independent capability model.

This approach works for internal device access layer, but this model is hardly suitable for third party device drivers.


Vision for a New CoprHD Southbound API

• Should use generic object model, not bound to CoprHD persistent data model classes

• Should not have any dependency on CoprHD infrastructure services in its implementation

• Should not have direct access to Cassandra, Zookeper, task completers from API methods

• Provide similar scope of storage operations as current CoprHD device access layer.

• Support for 3rd party drivers to advertise capability metadata and capability instances of their storage resources (supported drive types, raid types, performance levels, local/remote replication types and vendor specific capabilities)

• Provide capability to integrate device specific orchestrations into CoprHD

Ultimate goal easy plugbility of device drivers into CoprHD (similar to VASA2 providers integration with vCenter).


Options for Southbound API Implementation

• Implement new native Southbound API for all drivers, new drivers and drivers for already supported devices

This will require to change API, which controller layer uses to talk to “storage device” layer in CoprHD controller. Large rework task with wide impact to controller code – major classes for orchestration and device controllers have to be modified. This presents high risk for regression and will distract developers from working on new features.

• Implement new native Southbound API for new drivers and continue to use existing device access API for already supported devices.

Continue to use old API for controller layer to talk to “storage device” layer. Isolate access to new Southbound API inside external storage device implementation. Minimum disruption to existing controller and existing device driver code.

• Use Cinder solution to integrate 3rd party device driversSee next slide for problems with this approach


Cinder vs. native Southbound API

• Cinder controller as Southbound integration point

With this option CoprHD will lose many enterprise grade features which differentiate it from other offerings:• Discovery/re-discovery• Support for brownfield environment• Advanced class of service support• Enterprise level performance• High availability• Wide range of Volume protection, DR and availability capabilities• Ease of deployment

• Direct access to Cinder drivers for Southbound integrationThis option was evaluated by India Cinder team and the decision was not to use it. They decided not to do this for the following reasons:

• All the cinder drivers import different python libraries, and it became difficult to manage these dependencies.

• There was a lot of concern about running third party code written in python which could potentially touch files on the CoprHD controller nodes.

• Does not resolve discovery related shortcomings in the Cinder driver.

• Cinder release cycle is not aligned with CoprHD release cycle.


Proposal for a New Southbound API

New native Southbound API will be enabled for CoprHD third party driver developers. No changes to the existing device access interfaces in the controller. It will run parallel to current Southbound Cinder solution. The API will expose the same capabilities as we currently have in the native CoprHD device drivers.

The API will be rolled out in three phases:• First phase: API for discovery (including capabilities) and basic block

operations, such as create, update, delete, export, snapshot, mirror and clone operations. API for file operations: create, update, delete, export/share, snapshot and clone operations. Object model for the API methods.

• Second phase: enable third party arrays to be supported in EMC protection and high availability solutions, RP and VPLEX. Provide API for metering and monitoring.

• Third phase: API to enable addition of third party data protection/recovery and high-availability orchestrations. CoprHD will provide backward compatibility for drivers based on the two first API versions.


Southbound API Road Map

Api for block/file discovery and

basic block and file provisioning

operations

Enable RP and VPLEX

orchestration for 3rd party arrays, metering and

monitoring APIs

Api to enable 3rd party specific

orchestrations for data

protection/high availability

Phase 1

Phase 2

Phase 3


Integration of 3rd party device drivers with CoprHD

API Service

Controller Service

Native Drivers

VMAX VNX ExternalStorageDevice 3rd party support module (ExternalCommunicationInterface, ExternalBlockStorageDevice, ExternalFileStorageDevice)

HDS

CloudArray driver

Infinity array driver

Other 3rd party driver

3rd party device drivers will be registered with ExternalCommunicationInterface, ExternalBlockStorageDevice and ExternalFileStorageDevice. These device access classes will interact with controller on the north side and will interact with 3rd party drivers on the south side. They will provide infrastructure services and adaptation for 3rd party drivers.

XIV ScaleIO


ExternalCommunictionInterface Configuration

CommunicationInterface API (controller facing API)

dbClient coordinatorClient

BlockStorageDriver APIDiscoveryDriver API

3rd party device driver

3rd party array

The below diagram shows ExternalCommunicationInterface instance. This instance is a single point for discovery requests for all 3rd party drivers. The ExternalCommunicationInterface delegates requests to 3rd party discovery drivers. Processes discovery data returned by drivers and manages discovery task completion. It essentially serves as an adapter between the controller and 3rd party drivers.

3rd party array

3rd party support module for discovery operations (ExternalCommunicationInterface)

FileStorageDriver APIDiscoveryDriver API


3rd party array

task completer


New Southbound Discovery API for third party arraysDiscovery:• Discovery API for 3rd party discovery drivers – DiscoveryDriver.

• 3rd party discovery drivers won’t have access to CoprHD infrastructure including database access and will use new transient object model independent from persistent data model. We will provide limited API to the drivers to store and access registry type data in CoprHD database. We also may provide streaming API to stream data to CoprHD.

• ExternalCommunicationInterface will serve as an adapter for discovery requests from controller to 3rd party discovery drivers. It will expose the same interface to controller as other communication interface classes.

• 3rd party discovery drivers will be driven by ExternalCommunicationInterface. Developers will register discovery drivers with CoprHD.

• ExternalCommunicationInterface will route discovery requests from controller to 3rd party drivers based on the storage system type in the request.

• ExternalCommunicationInterface will process discovery data from 3rd party drivers and it will handle all communication with infrastructure required to execute the requests(Cassandra updates, interaction with ZK and task completion among these tasks).


ExternalBlockStorageDevice Configuration

dbClient

BlockStorageDecivce API (controller facing API)

coordinatorClient



3rd party array

The below diagram shows ExternalBlockStorageDevice instance. This instance is a single point for block provisioning requests for all 3rd party drivers. The ExternalBlockStorageDevice delegates requests to 3rd party storage drivers. Processes results returned by drivers and manages provisioning task completion. It essentially serves as an adapter between block device controller and 3rd party drivers.

3rd party array

3rd party support module for block provisioning operations (ExternalBlockStorageDevice)



3rd party array 3rd party array

task completer


New Southbound Block Provisioning API for third party arrays

Block Provisioning:• Provisioning API for 3rd party storage drivers – BlockStorageDriver.

• 3rd party block operation drivers won’t have access to CoprHD infrastructure including database access and will use new transient object model independent from persistent data model.

• ExternalBlockStorageDevice will serve as an adapter for provisioning requests from controller to third party block storage drivers.

• 3rd party block storage drivers will be driven by ExternalBlockStorageDevice.

• ExternalBlockStorageDevice will route provisioning requests from controller to 3rd party drivers based on the storage system type in the request. It will expose the same interface to the controller as other implementations of BlockStorageDevice.

• ExternalBlockStorageDevice will process data returned from 3rd party drivers and it will handle all communication with infrastructure required to execute the requests (Cassandra updates, interaction with ZK and task completion among these tasks).


ExternalFileStorageDevice ConfigurationThe below diagram shows ExternalFileStorageDevice instance. This instance is a single point for file provisioning requests for all 3rd party file drivers. The ExternalFileStorageDevice delegates requests to 3rd party storage drivers. Processes results returned by drivers and manages provisioning task completion. It essentially serves as an adapter between file device controller and 3rd party drivers.

dbClient

FileStorageDecivce API (controller facing API)

coordinatorClient




3rd party support module for file provisioning operations (ExternalFileStorageDevice)




task completer


New Southbound File Provisioning API for third party arrays

File Provisioning:• Provisioning API for 3rd party storage drivers – FileStorageDriver.

• 3rd party file operation drivers won’t have access to CoprHD infrastructure including database access and will use new transient object model independent from persistent data model.

• ExternalFileStorageDevice will serve as an adapter for provisioning requests from controller to third party file storage drivers.

• 3rd party file storage drivers will be driven by ExternalFileStorageDevice.

• ExternalFileStorageDevice will route provisioning requests from controller to 3rd party drivers based on the storage system type in the request.

• ExternalFileStorageDevice will process data returned from 3rd party drivers and it will handle all communication with infrastructure required to execute the requests (Cassandra updates, interaction with ZK and task completion among these tasks).

• The same device driver can provide both block and file provisioning API implementations when the device supports both file and block storage


Deployment of 3rd Party Drivers

• Java library (jar) running as part of controller service processPros: follows typical driver library deployment model (similar to existing native drivers, sblim cim client, curator client, etc), part of CoprHD distribution, covered by CoprHD high availability support, ease of deploymentCons: 3rd party drivers running inside controller process may introduce risk for controller performance, impact to controller memory consumption

• Java library (jar) running as part of separate JVM dedicate to 3rd party drivers. Controller (ExternalStorageDevice) talks to the drivers over RMI.

Pros: Part of CoprHD distribution, isolated from CoprHD controller process. Cons: Ads network latency (RMI registry lookup, RMI call) in communication between CoprHD controller and device drivers.

• Separate application with REST interfacePros: isolated from CoprHD controller processCons: not typical driver library deployment, requires its own HA solution, complicated deployment


Certification Process for third party drivers

Goals:

• Ensure that drivers are compliant with API specification: all mandatory API methods are implemented, all mandatory object model properties are set.

• Ensure that drivers are safe to run as part of controller service process (no memory leaks, no unauthorized access to CoprHD infrastructure classes (this will be automatically ensured by using separate class loaders for drivers),…)

The certification process will be conducted through set of API tests and code reviews.


CoprHD Southbound API and VMWare Vvols API

CoprHD Southbound API and VMWare Vvols API serve different purposes:

• CoprHD Southbound API is for policy based provisioning of raw storage, block LUNs and file systems on storage arrays, and export them to hosts. Block LUNs and file systems are created in storage pools configured on storage arrays.

• Vmware Vvols API is for policy based provisioning of virtual machines (Vvols) in already configured Vvols datastores (mapped to storage containers on array) on ESXi hosts. vSphere admin should create Vvol datastores in vSphere and map them to storage containers with preconfigured storage capabilities (replication, high-availability, performance) before Vvols can be created. CoprHD can be used by storage admin to create storage containers on arrays with required capability profiles. These storage containers are then used for Vvols datastores in vSphere.

25© Copyright 2015 EMC Corporation. All rights reserved.

Documents

1EMC CONFIDENTIAL—INTERNAL USE ONLY Integration of storage device drivers to CoprHD Evgeny Roytman