© 2015 IBM Corporation

Bringing Compute to Openstack Swift

Eran Rom, System and Storage Researcher, IBM Research - Haifa

© 2015 IBM Corporation2

The Storlet engine extends Openstack Swift by enabling the push down of filtering, transforming and analysis

tasks to Swift instead of bringing the data to the computation.

© 2015 IBM Corporation3

Proxy Tier

Storage Tier

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet Stuff

Containers

Powered Swift

Cluster

© 2015 IBM Corporation4

Containers Powered Swift Cluster

Proxy Tier

Storage Tier

© 2015 IBM Corporation5

Image Registry

Proxy Tier

Storage Tier

Containers Powered Swift Cluster

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet StuffUbuntu 14.04

Storlet Stuff

© 2015 IBM Corporation6

Containers Powered Swift Cluster

Image Registry

Proxy Tier

Storage Tier

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet Stuff

Ubuntu 14.04

Storlet StuffUbuntu 14.04

Storlet Stuff

© 2015 IBM Corporation7

Proxy Tier

Storage Tier

Developer

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Writing and Deploying a Storlet

@Override public void invoke(ArrayList<StorletInputStream> inStreams, ArrayList<StorletOutputStream> outStreams, Map<String, String> parameters, StorletLogger logger) throws StorletException { … }

© 2015 IBM Corporation8

Proxy Tier

Storage Tier

Developer

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Writing and Deploying a Storlet

PUT http://my.swift.co.il/v1/AUTH_111/Storlet/watermarkstorlet-10.jar

© 2015 IBM Corporation9

Proxy Tier

Storage Tier

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Invoking a Storlet on GET

GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : watermarkstorlet-10.jar

© 2015 IBM Corporation10

Proxy Tier

Storage Tier

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Invoking a Storlet on GET

Forward Request to an Object Server holding a Replica

GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : watermarkstorlet-10.jar

© 2015 IBM Corporation11

Invoking a Storlet on GET

Ubuntu 14.04

Storlet Stuff

ffmpegIntercept Request

Invoke Deamon, and Establish Connection with the Account’s Docker Container

Pass Input and Output FDs

Send back the Object’s Metadata and the Object’s Data Stream

Continue the GET Flow, where the Storelt’s response is sent back to the Proxy

Object Server

© 2015 IBM Corporation12

Proxy Tier

Storage Tier

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Ubuntu 14.04

Storlet Stuff

ffmpeg

Invoking a Storlet on PUT

PUT http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : MetadataExtractorStorlet-10.jar

© 2015 IBM Corporation13

Invoking a Storlet on PUT

Ubuntu 14.04

Storlet Stuff

ffmpegIntercept Request

Invoke Deamon, and Establish Connection with the Account’s Docker Container

Pass Input and Output FDs

Send back the Object’s Metadata and the Object’s Data Stream

Continue the Proxy PUT Flow with the returned Metadata and data stream

Proxy Server

© 2015 IBM Corporation14

Demo

http://159.8.49.36:8080/raimediabridge/login.html 

© 2015 IBM Corporation15

The Storlet Engine is in the Open

https://github.com/Open-I-Beam/swift-storlets Comprehensive Documentation

• Installation guide• How to write, deploy and invoke storlets guide• How to tailor your own image for running storlets

S2AIO• Install swift and storlets in a single VM for experimentation

Public users will be able to experiment with the Storlet engine thanks to FiWare – An EU research project delivering a novel service infrastructure for the future internet (http://www.fiware.org/)

© 2015 IBM Corporation16

Media Workflow at NAB (with RAI)Storlets run as material is uploaded and downloaded

Metadata Enrichment

Storlet

Swift Object Store

UploadEnrichedMetadata

Data

myvideo.mxf

Metadata

Data

myvideo.mxf

Data

Loudness Calculation

Storlet<Loudness,-23>

myvideo.mxf

Fix Loudness

Storlet

© 2015 IBM Corporation17

Acknowledgements

The Storlets project is partially funded by the following EU research projects

Digital Preservation Inspired by Human Memory

IoT for Smart Cities