Accelerating File Transfers

8/9/2019 Accelerating File Transfers

1/13


2/13

ContentsIntroduction

..............................................................................................................................................

2

CommonFileDeliveryMethods............................................................................................................... 2

UnderstandingFTP................................................................................................................................... 3

LatencyanditseffectonFTP............................................................................................................. 3

PacketlossanditseffectonFTP........................................................................................................ 4

ExistingFileTransferProcesses................................................................................................................ 5

FTPServerSolution............................................................................................................................ 5

Email

File

Attachments

......................................................................................................................

5

InstantMessaging.............................................................................................................................. 6

PhysicalMedia................................................................................................................................... 6

OvercomingFileTransferIssueswithAcceleration.................................................................................. 7

OptimizingthroughputwithUDP....................................................................................................... 7

OptimizingTCPwithMultipleStreams.............................................................................................. 7

ReducingDatawithCompression...................................................................................................... 8

MinimizingdatawithDeltaTransfers................................................................................................ 8

TheCostofSlowFileTransfers................................................................................................................. 9

Thenumbercrunching....................................................................................................................... 9

FactorstoConsiderforyourFileTransferSolution................................................................................ 11

AboutFileCatalyst................................................................................................................................... 12

2009UnlimiTechSoftware,Inc. [1] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009


3/13

IntroductionAccording to a Hilty Moore & Associates study, companies have increased

their reliance on digital file transfer via File Transfer Protocol (FTP) by 66%

from2006to2007.

FTPresultedin

incompletedeliveryas

muchas1in5times.

These companies were in not only hightech, but also health, financial,

manufacturing and service industries. Most respondents experienced

deliverystoppagesorincompletetransmissionswithFTPasmuchas1outof

5times.Amorereliablealternativeisneeded.

Thispaper

addresses

technical

issues

faced

by

enterprises

when

transferring

largefiles.ItwillalsopresentabusinesscasebehindswitchingtoanonFTP

digitaldeliverysystem,andguidestheenterprisetofindtheoptimalsolution

foritsrequirements.

Common File Delivery MethodsCompanieswithlargedatasetscontinuetousephysicalmethods,recording

toDVD,Flash,orHDDandusingadeliveryservice.Thisworksonanadhoc

basisbut

can

lead

to

high

costs.

Standarddigitalalternativesincludeemailorinstantmessagingservices,but

thesehaveissuesincludingfilesizelimitationsorrelianceonthesameTCP/IP

backbonecausingFTPtofail.

Alternativesshould

overcomelimitations

ofFTP,maximize

existinginfrastructure,

andbesimple.

Digitaldeliveryisstilltheultimatesolutionformostenterprises.However,a

solutionshouldovercomethestoppagesandslowdownscommontoFTP. It

should allow organizations to maximize their existing infrastructure, saving

moneythatwouldotherwisehavebeenwastedacquiringhigherbandwidths.

Finally,itshouldbeeasytoimplementandmoreimportantlyeasytouse.




4/13

Understanding FTPFile

Transfer

Protocol

(FTP)

is

one

of

the

oldest

protocols

on

the

Internet,

but

isubiquitous for filetransfer. FTPresulted fromanRFC firstdrafted in the

1970s,andhassincegonethroughrelativelyfewchanges.FTPwasdesigned

tobecompletelyagnosticindependentfromoperatingsystems,filestorage

systemsandfiletypes.

FTPrunsonTCP/IPprotocol,usingtwoTCPconnections.Thefirstisacontrol

channel used to exchange instructions, and is always open. The other is a

data channel, used for file transfer and directory listings. The channel is

openedorclosedasneeded.

datachannelopened asneeded

2waycontrol channelalways open

ServerClient

FTPhasexistedsince

the1970s,andis

ubiquitous

TCP/IP is themostwidelydeployedprotocolon the internet;consequently,

FTP is the de facto standard for file transfer. However, it also inherits the

drawbacksofTCP.

Latency and its effect on FTP

Inordertoprovidereliabledatatransmission,TCP/IPrequiresthereceiverto

acknowledge each packet being sent, in sequential order. Each such

communicationismeasuredasroundtriptime(RTT),orthetimeittakesfor

apackettobesentandacknowledged.

Latency(RTT)causesFTPtoslowdown

considerably

TCP responds to latency by adjusting the amount of unacknowledged data

that can be on the link before waiting for a reply. The optimal amount of

unacknowledged data en route should equal the endtoend bandwidth

multiplied by the RTT, also called the bandwidthdelay product. TCP

continuallyestimatesthisvalue,settingaTCPwindowtocontrolhowmuch

datashould

be

sent.

TCP

has

limits

on

the

size

of

this

value,

so

when

the

bandwidthdelay product exceeds a certain threshold, the result is a lot of

waitingordeadair. Satelliteconnectionscanbeintothehundredsoreven

thousandsofmillisecondsofRTT.




5/13

Packet loss and its effect on FTP

Network congestion typically causes buffer overflows of intermediate

routers,ausingpacketloss.Sincepacketsaresentsequentially,thiscancause

aholdup inthecycle.UnacknowledgedpacketsalsocausetheTCPwindow

toshrinkorevenclosecompletelyforperiodsoftime.Wirelessandsatellite

transfers can have even higher packet loss due to sources of interference

suchascloudsorphysicalstructures.Packetlosscombinedwithhighlatency

createseven

worse

performance

for

FTP

transfers.

Throughputis

reduced

dramaticallyat

only80msRTTto

justover5,000kbps

After0.03%the

throughputdrops

exponentially.

Publicinternet

averages0.5%

packetloss.




6/13


7/13

Instant Messaging

According

to

PCWorld

,Instant

Messaging

has

become

the

most

common

formofelectroniccommunications,surpassingemail.Therearesomeissues

tobeawareof:

Interoperability:AlthoughdifferentIMclientsareoftenabletosharetextmessages with one another, file transfer is not usually possible between

differentclients.Interoperability,TCP/IPdrawbacksand

securityrisksmakeIM

lessthanidealfor

enterprises.

Security:PopularIMclientscontinuetobetargetsfortrojansandexploits.Transmissionsarenotusuallyoversecurechannels.

No tracking: Transferring files over IM makes it difficult or impossible totrack

files

moving

between

different

endpoints.

Poor Network utilization: TCP/IP is typically used as the transport layer.ThesamenetworkissuesfacingFTPalsofaceIMfiletransfer.

Physical Media

Organizations may record data to a DVD, hard drive, or flash storage, then

physicallyshipittotherecipient.Thisprocesscanbeusefulwhenthereare

hugedatasetsorwhenthere ispoorconnectivitybetweenthetwoparties,

butithassomeobviousdrawbacks:

Post-production

housesexchange

terabytesofdataby

shippingphysical

media.Thisislabor-intensiveand

ex ensive.

PreparationTime:Thisprocessrequireshumaninteraction,fromrecordingthedatatoprintingoutmailinglabelsandshippingthemedia.

Cost: Depending on the urgency of the files, costs for using a courierservicecaneasilyaddup.

DeliveryTime:Sincephysicaldeliveryisinvolved,theprocesscantake1to5 business days. This may be an unwelcome delay and makes time

sensitivedeliveryanunreliableproposition.




8/13

Overcoming File Transfer Issues with Acceleration

OvercomingTCP/IP

bottlenecksallows

digitaldeliveryto

becomeviable

Most

companies

realize

that

transferring

files

online

is

more

costeffective

than shipping physical media. By overcoming TCP/IP bottlenecks, digital

delivery becomes a realistic option. File transfer acceleration is possible

through two complementary but separate methods. Generally speaking,

theseareoptimizingthroughputandreducingdatabeingsent.

Optimizing throughput with UDP

UserDatagramProtocol(UDP)squeezesperformancefromanIPnetworkby

notimplementing

all

features

of

TCP.

UDP

is

a"connectionless"

protocol

that

does not depend on sequenced acknowledgements and retransmissions.

Althoughthis impliespotentialfor incompletetransfers,vendorsarefreeto

buildtheirownreliabilityintotheirapplications.

UDPtakes

latency

out

oftheequationbecause

itdoesnotrequire

acknowledgment

SinceUDPallowspacketstobereceived inadifferentorderthantheywere

sent itdoesnotstall ifpacketsaredropped.Lostpacketscanberequested

with a command channel: data flows over UDP while errorcorrecting

commands are periodically exchanged over a TCP/IP stream. Without the

sameacknowledgmentrequirementsasTCP,latencyiseffectivelyeliminated.

Optimizing TCP with Multiple Streams

WithstandardFTP,onlyonedatachannelisused.Thischannelbecomesidle

duringacknowledgementandotherhousekeeping.Byincreasingthenumber

ofdatastreams,there is lessdeadair,asoneormorestreams is likelyto

ensivedue

to

the

mustberunning.

Multiplestreams

meanslessdeadair

duringatransfer,butis

resourceintensive

besendingdataatanygiventime.

Thismethodoftransferringfilesiseffectivewhennetworkdegradationisnot

extreme.However,

the

process

is

extremely

processor

int

largenumberofconcurrentthreadsthat




9/13

Reducing Data with Compression

Datacompression

is

ageneral

term

for

agroup

of

technologies

that

encode

large files in order to shrink them down in size. Fewer bytes to send will

naturallyresult ina fastertransfer,whetherthenetwork isthe internet,an

intranet,oralocalareanetwork(LAN).

Smallerfilesizes

requirelesstimeto

transfer

Data compression involves applying an algorithm to data that stores

repetitive bits as a shorthand. The receiving end uses an appropriate

decoder to restore these short forms to their original state, resulting in an

identicalcopyoftheoriginalfile.

Some files can be compressed more than others. Textbased files, such as

documents, reports, database files are highly compressible. On the other

hand, many media files such as MP3 or JPG are already compressed and

cannotbecompressedfurther.

Minimizing data with Delta Transfers

Afilemayalreadyexistonbothsidesofthetransfer,buthasbeenmodified

atthesource.Whenadifference isdetected,analgorithmcancalculatethe

differences and store them in discrete files. These small delta files are

transferred

instead

of

resending

the

entire

file,

then

applied

as

a

patch,

updatingthedestinationfileinordertomakean identicalcopy.Thebenefit

is clear: imagine being able to send a 4MB delta instead of a 2TB, for

example.OnepopularUnixapplicationthatsupportsdeltatransfersisrsync.

Transferringafile

deltacanreducedata

sent

by

a

staggering

amount




10/13

The Cost of Slow File Transfers

Slowtransfersincur

significanthardand

softcosts

Most

enterprises

do

not

realize

the

cost

of

slow

file

transfers.

The

most

significanthardcosts isbandwidth;softcosts include losthours intermsof

employee productivity and project completion. Fixing the problem of slow

transfersyieldssignificantreturnsbothonhardandsoftcosts.

The number crunching

Accelerationtechnologiesprovidereturnoninvestment(ROI)bymaximizing

existing infrastructures. Spending money on more bandwidth makes little

sense if current bandwidth is underutilized. Based on the following chart,

considersomescenarios inwhichaccelerationprovidescostorproductivity

benefits:

MonthlyInternetCosts

Bandwidth(mbps) 1.5 3 5 10 50 100

Cost(month) $75 $250 $500 $750 $1000 $1300

PricingbasedonGoWebManandRogersCable

Scenario 1reclaim lost time: A postproduction house in New Zealand

transfers files to and from itsLA office.Their 50mbps link costs $1000/mo,

and is theoretically very fast. However, the average latency of panPacific

transfers is 200ms, with an average packet loss of 1%. Under these

conditions,FTPwillonlytransferat490kbps,amere1%ofthelargepipe.Improvingfiletransferreturnssignificanttimesavingstothe

organizationWith UDPbased acceleration, transfer speed will reachnear linkspeed,or

100X fasterthanFTP.Artthat formerlytookahundredminutesnow takes

one.Withoutspendinganymoremoney,thecompanyhasregainedninety

nine valuable minutes that can be applied to productivity and project

completion.

Scenario

2save

IT

expense:

A

company

transfers

files

between

North

AmericaandEuropeovera10mbpsconnection.Dueto latencyandpacket

loss, the effective throughput is 613kbps or 6% of bandwidth. The internet

serviceprovidersuggeststhecompanypurchaseafasterlink.




11/13

Maximizing

bandwidthislessexpensivethanbuying

afasterlink

Insteadofaddingtotheirmonthlyoperatingexpense,thecompanydeploys

accelerationtechnology.

By

using

their

bandwidth

to

its

fullest,

the

company

experiencesfastertransfersandhasavirtualsavingsof$670/mocompared

totheupgradeoption.

SummaryofScenarios

Scenario Line

Speed

Latency/

Loss

Previous

speed

Acceleration

Advantage

1reclaim

losttime

50

Mbps

290msRTT

1%packet

loss

490Kbps

(1%)

Increasetransferspeed

by100X,reclaiming

manhoursand

improvingturnaround

2

save

IT

expense10

Mbps160ms

RTT

0%packet

loss

613Kbps

(6%)Increase

transfer

speed

by15X,givingcompany

higherperformance

andavoid$670/mo

expense.

Optimizing bandwidth generates both virtual and actual returns on

investmentmadeinimplementingafileaccelerationsolution.




12/13

Factors to Consider for your File Transfer SolutionThe

optimum

file

transfer

solution

should

be

able

to

connect

the

various

file

storage systems of the enterprise regardless of network topology, physical

location or operating system. Many commercial vendors provide a file

transfer solutionsome offer a comprehensive platform, while others

provideasubscriptionservicesolution.Thereareuniversalconsiderationsto

bemadeforeachoption:

TheIdealSolutionScalable

Thesolutionshould

scaleinacost-effective

way.Thesolution

shouldbeeasily

expandable

Reliable

Onceafiletransferis

initiated,usersshouldexpectdelivery.The

systemshouldrecover

fromerrors.

Simple

Thesystemmusthave

agentlelearningcurve

forend-users

Secure

Thesolution

should

be

abletomeetexisting

orpotentialsecurity

mandates

Acceleration: How important are accelerated file transfers to yourenterprise? Doanyofyourprocessesinvolvelargefiletransfersthatmay

bereduced?

Automation and Scheduling: Will you benefit from automating routinefiletransfers?Wouldautomaticschedulinghelpyourbusinesscomplete

projectsandimproveitsbottomline?

WebBasedInterface:Shouldendusershaveaccessviawebbrowser,orisaclientsideapplicationpreferred?

Easeofdeployment:Howmanyserversandclientsdoyouneedtoinstallandwhere? Isinstallationfeasibleforremotelocations?

SecurityandCompliance: What are the major security issues that yourorganizationfaces?Arethereanysecuritymandatesyoumustmeet,and

doestheproposedsolutionmeetthem?

TrackingandReporting: How much tracking functionally do you need?Whattypeofreportswouldyou liketogenerate,anddoesthesolution

supportthis?

Aftersales support and service: Does the vendor provide reasonablesupportandservicefortheproduct?




13/13



About FileCatalystFileCatalyst

software

solutions

are

developed

by

Unlimi

Tech

Software,

Inc.

Foundedin2000,UnlimiTechSoftwareisaprivatelyheldcorporationbased

in Ottawa, Canada, operating with a global reseller network. UnlimiTech

products are focused on solving file transfer challenges in diverse

environments, from end user desktops to sophisticated WAN and satellite

basedmulticastsystems.

LearnmoreaboutUnlimiTechSoftwareandFileCatalyst:

www.filecatalyst.com

[email protected]

tf +1.877.327.9387

t +1.613.667.2439

f +1.613.667.2439

Learnmoreaboutour

technologyat

www.filecatalyst.com
http://www.filecatalyst.com/mailto:[email protected]?subject=Whitepaper%20Questionmailto:[email protected]?subject=Whitepaper%20Questionhttp://www.filecatalyst.com/http://www.filecatalyst.com/http://www.filecatalyst.com/mailto:[email protected]?subject=Whitepaper%20Questionhttp://www.filecatalyst.com/

Documents

Accelerating File Transfers