ManagedFileTransferanno2016,oldfashionedstuff

JochenTrauncker&RubenvanderZwanYenlo

ManagedFileTransfer

§  YenloLabsini<a<ve

§  YenloLabsisR&DofYenlo

§  Yenloisaninterna<onalopera<ngintegra<onspecialistandPremierPartnerofWSO2

§  Goal:simplifyintegra<onoffiles

Itis2016.Dowes<llneedtotalkaboutfiletransfer?

•  Thereare45508companieslistedinthestockexchanges

§  Thereareintotal115Mcompaniesregisteredworldwide

•  Mostofthems<llusefiles

Companieslovefiles

§  Bigfiles(Gigabytes)§  Smallfiles(justafewKilobytes)§  Onlyafewfileseveryonceinawhile§  Manyfilesinonefolder§  Manyfolderswithmanyfiles§  Andsoon

Filescanbemeanfellows

§  Encoding(>100KQ&AonStackoverflow)

§  JavaandFile(>200KQ&AonStackoverflow)

§  Businessseman<csinfilenames

§  Filesystemspecificidiosyncrasies

§  Permissions

§  Locking§  Concurrency

Example:whencanafileget„processed“?

§  Visibilityinthefilesystemisabadidea§  Atomicopera<on

§  “mv"fromsome“hidden”loca<on§  Filenameconven<on

§  “abc.xml.s<llwri<ng”§  mv”abc.xml.s<llwri<ngàabc.xml”

§  Lock-File§  ”abc.xml”&“abc.lck”

§  Filesizedelta§  Welldefinedpointin<me

Example:whencanafileget„processed“?

§  Ul<matefileprovider&consumers§  Mostlylegacy§  Nowaytochangethem

§  MFThastotakecareof§  Filelocking§  Filesizemonitoring§  Atomicfilemovement§  Andsoon

Andfinallythetransportoffilesshouldbe...

§  100%guaranteed,performantandreliable§  Encrypted,archived,monitored§  Ordered,reordered,controlledandauditable

ManagedFileTransferistakingcareof...

§  Transport

§  Security

§  Compliancy

§  Control

§  Workflowintegra<on

Transport:local

§  copy/movebetweendirectories§  onehostwithseveralmountpoints

Transport:host-to-host

§  transfercontentvia§  HTTP(S)§  (S)FTP§  ….

§  managesource/targetloca<on§  viatransferredmeta-data§  up-frontconfigura<on(conven<on,content,…)

§  directcouplingofsourceandtarget

Transport:host-queue-host

§  transfercontentviaqueueasintermediary

§  decouplessourcefromtarget

Transport:host-intermediary-host

§  intermediaryprovides§  intermediarystorage§  managementofthetransfer

§  helpfulfor§  (re)orderingofdelivery§  <meddelivery§  mergingoffiles

Security

§  Deliverfilestoen<tledaudience§  Maliciouscontent§  DoSamacks

§  Toomanyfiles§  Toobigfiles

§  Blacklis<ng§  Whitelis<ng

Compliancy

§  Guaranteeddelivery§  Timelydelivery

§  Audi<ngandLogging§  Encryp<on

§  Checksumming

Control

§  Processingorder§  Deliveryorder§  Timing§  Rou<ng§  Parallelizing§  Synchronizing

Workflowintegra<on

§  Long-runningprocesses

§  Triggeringseveralsubsystemsasynchronous

§  Humantaskintegra<on

Sothatiswhatwedid….

§  WestartedbuildinganMFTsolu<onontopofWSO2

§  Basedon:§  WSO2ESB§  WSO2MessageBroker§  WSO2DataAnaly<csServer§  WSO2BusinessProcessServer

Keyfeatures

§  Rou<ngbasedonnameorcontent§  PushandPullmethodsfor(S)FTP,SMB,andmore…§  Guaranteeddelivery(+DeliveryRetrymechanism)§  Automa<carchiving§  Detec<onofdoublefileinput§  Singlesource,mul<pleTargets§  DeterminatesendingorderoffilestoTarget(s)§  MonitoringandAler<ngop<ons

Wheredowestand?

§  Therun<meis(almost)ready§  Theuser-interface(web-app)isworkinprogress….§  Extensivetes<ngonvariousfilesystems

§  Aimtolaunchfullsolu<onendof2016

Someremainingchallenges

§  Chunkingof(large)files§  Streamingof(large)filesthroughseveralintermediaries§  Individualfilesystemimplementa<onsandtheirproper<es§  Deduc<onoffileencoding

Finalnote

“Neverunderes=matethebandwidthofasta=onwagonfulloftapeshurtlingdownthehighway.”

AndrewS.Tanenbaum

ThankYou!

#WSO2ConEU

Shareyourfeedbackforthissessionwso2con.com/app