SYNCHRONIZATION OF MULTIMEDIA OB.IECTS

D. Verdejo. A. Kannoüch, N. D. Georganâs

Telecommunicâiions Rcsearch Insailutc ol OnlanoUniversitÿ of oltâwâ Medicâl Communicâlions Rcsearch Centcr

16l Louis Pas@ur, Onawâ Ontario, Cânâdâ KIN 6N5e.maiI: \GRDEJO@UOTTAWA.BITNET

FAX: + I'613'5&l-7681

Absùacl

Uso of muldmediâ documenB involves presenhlion o[ datalrom dilfcrcnl succm\. rn lhe mosr gcneral ürse. (ùmrngfrom distributed dalâbâses. A mcdium rciers to â particularkind of s[eâm such as vidco, audio, and datâ streams likcgrâphics, text ând displây mânâgemenl.In this issue we deal wilh â cenralizcd synchronlzationschemâ for media objects (MO) considered âs separatecniities which âre intendcd to cooperatc in a commoncnvironment 1o constilutc thc MulliMedia Document(MMD).Thc synchrcnizaiion Droblem is râised a! two dislincl lc\'cls.Firsl il appears at lhe document specificalion level when thcMMD is lo be dcfined, lhen al ùe plâybâck ol dre MMDwhen the data involved in its consritution arc !o be gâûercdàgain. We use here a relâtionâl dâtâbâse model scenârio às

tho basic represeû1âdon of thc synchronizarion needs, Wcâlso present an implementation of the Masler ControilcrSynchronizer (MCS). the centralizcd cntity managing mcdiâ

l.lnEoduction

the cvolution ol mullimedia lcchnologies (vidco, opticalsrorâge)[1] ând broadbând rclccommunicadon ncr\ÿorks hils

significândy coûribuæd to thc cmergencc ol nc\ÿmullimediâ âpplicâlions. In mLrltiplc liclds. like compurcrl.r\eü tmining. rclc.hoppin! )rd bu'rnc.r nrcscnrrriônAudiovisual Inleracdve appticâtions (AVI) lurn out to bcpo\rertrll communicaiions toois 21. SûDultaneously. Lh.needs for massive inlcrconnccûon ol severâl usùrs lbrLeieconlèrenciûg grows in application ilelds like Cornputcrsupporrcd Collrborauve vÿork ànù datrnt . :.ininll rc,turrinrreâl time video and sound communication In lhls ncw area.

ùere is a great need for homogeneily. |o increase thcJe\clopmcnr ol qVl Jpphcrlionc Jnü Ihelr (ornmunrcJln,n..However, in our âpplicalion. the documens manipulalcdmây contâin vârious and large amouo6 of dâla, Thcrclbrc âdislributed ltrchitecturc wherc dillèrcnt portions ol lhcdocumentare dispÂrched on differcnl servcrs is required. Oncof ûe mos! importânt problems ùal wc must dcal witlr insuch an archilecturc is ürc synchronization of dilfcrcnlcomponen§ of the documen! during thc prcsenlation ol thcmultimediâ documenl on ùe displây workÿation (DWS).Indeed we have to retrieve the ditlcrcnt rcsoùrces associâtcdwllh fte mediâ before displaying thcm. These dâtâ. undcr lhcform of files havc to be ûânsmilted over ûc nc!$'ork, lromthe seûers to ûe DWS. Basically, those synchronizationconstrâinB come from the archiicclure of ûc MlilD. As wcujc lools \uch i\ mullmclir cJiro'. rnrl :ltrlr, rrn! ') l(rn'

f4l ro crcarc rhe MMD, wc spccify timing consraints for theJi\plâ) of cvcry mcdium Jnd lhese conslrxints move tosynchronizadon problèrhs for the phyback oflhe documert.wc disringuish the se)f synchronialion which considcrs theyârious (communicalion) pârameters introduced by ûenersork lo evirlu]lc Lhc dclJy\ nccessary ior Scning  givenrcsource ând üe intcrmcdia synchronization thal describes themcrhods to plâyback the diffcrenl mcdia logether on theDWS when ùeir resou.ces ârcâvailâblc.Among rhe diflerenl approaches tor ùe crcaiion ol a MMD\ÿe can cile rhc slorc and forward mcahod whcrc no æmporaldcpcndency urisls bctwcen thc time of crcation of thedocumenr ând thc momenl it is plîyed back, ând the realtimc method whcre both authors ând rcadcrs ol lhc documentârc intcrconnectcd viâ a nelwork whiie the MMD is beingcrcâlcd. lvlosr rccenr work ât lhis lcvcl indicatcs thât thc\ynchronizâ1ion requircmcnls could be fuliillcd ar thc OSIrransport lclcl l3l.In this prpcr wc chose ro locus on the store ând forwardcnvironmcnr as previous reseârch hàs beefl done in this fieldin our lâborâtory [:1] and wc studied the synchronizâtionconccpr âpplicd !o MMDs. Wc must there distinguish twodirectrons in thc MMD .rpp.oâch. One is focuscd on lhedocument tcchnology. thc d.rumenl architecùre, representedhy rhe ODA/ODIF s!ând.rrd which deals wiû thc structure ofthc documcnt ând lhc rrclhods neccssery lbr iLs layout. Thismodcl is bcing cxtcndcd to cope with rhc synchronizatioûconsûtlinrs ol lvllt4Ds.The olher diroction is uken by the Frcnch CCETT wiû itsproposàl oi standud RAVI (RcpresenÉtion of AVI) whichdcscribcs rll thc Ircets ol rhc proccssing ol ân AVI.rpplicarion 8ll9l providing r hngurge to describe lheapplications. This approâch is more onented towards thcrnulrjmcdiâ rppiic.rtion rn Ènns ol computer program thanrhc archirecrurc ol Lhc N,llvlD mânipulâtod. 1n RAVI a MMDis â scqucnce of compuler progrâm struclured âs â trcc oloperàors wùh their parâmetcrs, including ÿrucrural ândpresenrâlion opcrâlors, Thc p.ogram wrilEn wilh ûeseopentors conslilutc the sccnârio of ùe MMD which, wheninrerprclcd consûtutes rhe Mullimcdia Applicâtion. Hcre, the§cenario rcplâccs and exrends the nolion ol documentarchirecture, allowing references to any kind ofresource ândpcnnirling to hândlc intcracliviry with thc uscr ùrough a setol dcdicaled op(rârcrs.Whichcver di.ccrion is Eken lo dcal with MMD. scenârio ordocumenr architeclurc. tho playback ol the MMD râisesproblenrs ol si7e and loürxon ol ùc components: the display

frn.,c..;cncrJll\ crnnor rcr rhc wloic MMD at oncc in ir\local mcmory and needs rime to retrievc thc.cmorcinlbrmalion. This âppears c1cârly in thc râdiologicâlJttrrtrLr i (l(ÿ. "inc,l in rh. \'lLlrIncJiJ C.,rnrnunicflionL.rbora()ry lbr jnÿancc, whcrc prcrurcs consliLuling thc

2

râdiological examinâ1ion âre 1.2 Mcgabyles in sizc (10Mbirs) and are localed on dedicâled servcrs. Altemauvely. üeC'D ROM tcchnology l5l solves thcse problems bycompiling on ûe sâme physical rcsourc'e (opdcal discs) allûe MMD. The plâyback is then simply the sequenliâlreading of the disc wiù a ùroughput thal allows all ûemedra to be drspâtched ând plâycd on tJleir own deviccs.Io our system. the media ârc kep! separâte in disributed tilcsystems unlil thc documenl has to be playcd bâck on âDisplây Vr'orksÉ1ion (DWS). We üen nccd âspeciâl process

ro ordcr the rcuieval ol the difierenl rcsources from theremole servcrs. 11 is the Master Controllcr Synchronizer(MCS) which accesses the remporâl and sructurâlinformation âbout thc MMD and builds â rcsource planningin order 1o provide dala for üe ou1pu1 dcviccs associaied tofte media, âccording to the memory spâce avaiiable ând thcdâla lrânsmission rales requircd. Thus thc MCS acls às a

plaoner for the communications bctwccn ùe network âIrd thcmedia obje.r.s (MO).This paper dcscribes lhc rcsearch in the mulLimediacommunicadons laboralory of ùe unilersily of Ollawâ. Wcdescribe the conlexl of this rescarch and rhe prololypedeveiopcd to implcmcnl o0r ideâs ând deiine our approach torhe probiem ol synchronizarion in this pârticulâr conrexr andûe meùods we usc to solvc it.

2. Ivlul mediâ Slorc ând Foôÿâd Environnrcnt

The Uni!crsit) of Otl.rwJ \lcil.Jl CommLrnicr'.ôn.Research Ccnrc has developed a Multimcdia RadiologyInformâlion ând Communicâtions system i4l. Thc systcrrrprovidcs radiologists and physicians with a sel ol'rvorkslalions liom lvhich râdidogy iniormalion cân bcelccEonically caprured. rransmitled, updatcd ând coosulte(1.Our âpproach is bâsed on thc usc ol thc Rclâlional DBùIS(Dala Baso Managemcnl Sysrcm) to dcscribcs mullimcdirobiects and dreir limc and lunctionnal dcpcndcncics, ('ilhsome external cxtensions to suppo( unlormaLlcd dâtn(imâges ând voice). Our prolot)pc has bcen implcmcntcdusing ORACLE on SUN] as a Drtabase Scncr. Thc Scnùis connecrcd viâ a 10 \'Ibib/s ELhùrnct L,\N kr scÿùrll i1116

based worksLrtions- using TCP/lP end approprrrt.Application prorcool§. Two iill6 workstâtions rrc uscd x\serÿers lor 1cxr, ÿoicc and imâse.In this environmenr wc dcline lwo levels oi synchronizati{rnùâl crn be âssociâtcd 1o dre phases oi crcalion and ph) brckoi the MMD:Firsr is the coarse Srain synchronizatron [6] delincd a! thcuscr lcvcl in order to specif-v the links berween thc mcdiaJuring dr,phy: hcrc lh( qucilùn r. s hJr rrLJ.r :dc .:uir': 1,,

be tied in Lhe documcnl and \!hen Ifig-11.For cxamplc wc câo cile lip-synching on a vidco documentwhcrc voicc ând film are on sepuatc resourccs, Wc thcrcb)'fâce dreproblem ofspecilying the way wc nced ûc mcdia tobc mixcd at thc moment they will be displâycd. This is thepurpose of rhe crcâ1iofl ol rhc sccnlrio (scrip, ol thcdocumenr. Duflng Ine plly birck ul lhc,locum.nl. ll I., ncof the duties oi lhc Synchronizer lo manâlc suchsynchronizâtion whjch \e !vill cùll intcr mcdjxsynchronizâ1ion,

Frg.l Snccrli(rtiunùl.rnrullimu(li;! s(enùio

CIock

Fig.: Drt:l RcquircmenL! Chârt lbrâ Multimcdia

On ùc olher hrnd, the lvlMD s playback lcads to ân orderedlist ol rctrieÿ.rls ol poftir)ns ol documenCs conæn! so as torcspcct thc tirning requirements during lhc display.Weâssume rhâr eâch mcdium must bc proccssed scparaleiy tomcct its own drt, ând dmc requiremcnts, and ûat the role olMCS is to takc into accounr lhc schcdulc consûain§ o1'

clerÿ media as dclined in the scenârio. Consc4ucnlly we willcâll lhis kind ol synchronization sclËsynchronizarion-. Thesynchronizrlion ol two mcdià is though solcly symboiizedhy rhc raLr lhrl lhcy aivc dxrJ rr.luircmcnt. cr r sarmc lime.ft seems conlcnicnt lo represcnt drc sct ofconsuâinls by lhcchândcpictcd in Jlig.2l whcrc ùc dashcd surfaces indica@ùctolal ûmounts ol'mcnrcry requircd by eâch medium duringrl'c lr..cnrirlr"n ul rhc N,l\4D Thc .)r)chrollr/Jrron nroce§scân rhcn bc dcpictcd u\ a lwo stcps proccss, first is seli-

picture i Tcirt T.xr2

GraPhic, IEnd ol

3

synchronization ùat resolves lbe extemal constraints thcn isûe inter-mediâ synchronizadon àl the local level.Synchronizâtion is a sub-problem of lhe presenlation oi û

MMD. By presentation we mean herc'execuaion of ücapplicâtion ahal will play bâck ùe scenario describing lhcdocument Maûy efforts havc been mâde to formalizc Lhe

represenuiion of multimedia audio visuâl inleracti\capplicarions (AVI)[71. The French CCITT s proposal olslandard, RAVI, introduces a sel of operators to design drc

slrucure of âny AVI like â compuler program. Some ol dre

operâtors arc used 10 reference and process ûc mediâ ând

more generâily afly resource. othcrs are solely sutemen§ lbrslrucluring the application. Among üc opcrators are lhcsyûchronizalion operâtors. which âllow the deiinitiûr ol'relâuons of sequence ând pârallelism betwcen the mcdia

As a cenlrâlized cndly, lhe syûchronizer should drcrclorrJicrrbule lhe ordcrs ol prelJrrllun. srrl inri Ic'lilJli,,lr .'thc differen! MOs and dccide on Ûeir lilèlime according 1o

thc scenà o [fig.4].We designed a spccilicâ1ion lânguagc olihc synchronization rcquirements in our mul!ime(i1rdocuments, which permils rhe crcâiion ol simplc sccnânos

In rhis lânguage, a first sel of commands wiil albw thc

dcfinilion of the dillcrent objeds involvcd in the multimcdi,applicatron. defining the tlpc of the medium ând thc nr c olthe file'rhich contains its resource. Accordin! lo ihe t)pc.ùe resource will thcn be reaievcd liom a pârliculâr scrvcr. lna second sct arc the commânds lbr spcciiying thcsynchronizalioo condrtions ând lhe rcLions synchronizc(I.mostly thc playbacks of ûe MOs-

.1. Funoliontl Principlcs

As mentioncd in the preceding scctions, lhe nolion olp.cparalion of lhe resources is cssenrial to insurc thât thc

mcdia will be ready ior lurlher synchroniTâlion.Prcparation includes rcscrvalion of memory spacc. âllocationof lhe playback chânncls, downloâding ol resourccs and

burering.Thc prcpÏcuon mu.t be ôrdcr..r I \ rhc r) n r1n 'rrr/cr u r. { r'

lhc execution ol any mediutr. thc I'ICS rnâtâgcs urÿ,rmhcr ol mrdiâ in\olved rn r docurucnt. Thù inLcrllr.

prccess cân then bc seen as â mcdium consuntcr \'hllc th.producers would bc the diu sêncrs.Therctorc managrng a mcdiu consisl ol scülng up ll

dialogùe wirh bolh lhc scrver rel:rted lo this lrrcdiurlr a d lhcconsumer âppiicadon. The dialogue !vith thc âpplicâtion isdirecÉd by the limes vr'hen dâtâ should bc âyàilàblc {ApD§)and is achieved locâlly on ùe DwS, whilc ùc diâloguc §'ithûe scrver is dlected by thc times when lhc loading requests

for dâtâ should be made (Lpips) ând is âchiclcd ûrough â

communicâlion channcl (CCH). One ol thc pu.poses ol thc

MCS is thc compulation of thc serial ol Lpips in iunclionof thc scrjâl ol Apips draL is prcvided by thc interpr.lâliorl o!'

thc scenârio. Such mânagement should lcad to lhcestablishmenl of links belween servers ând clients \rith an

averâge throùghpur corresponding to the media needs andrimês dcfined in thc scenârio. Our aim is to cnabie the IICS|o managc the trânslers of dal.â belwccn scrvers ând mcdlilobjects wilh ùe finest granularily possible. For ûât purposc

wc dcfine a protocol which allows ùc MCS to gelinlbrmation about all the mcdià objccts înd scqucnce tlcir

downloading lrccly, compulingfequesEd on eâch scrvcr-

Fig.,l: Mânagemcnt ol mcdia Obtcct! by -Ihc Synchronizer

One could rrguc ûa! tolal do\ÿnbâding on ûe DwS of ùeresourccs necessâry for MMD plâybâok would bc sulficientro ensurc good rclricval delays. uûfortunâlely we musl.onsidcr rhrt- il thc hcâl hârd disk ol thc DVr'S has the

capâciry «) storc thc cnlire documcn( il will ccrtâinly notprovidc cnough ûroughput lo play back all ùc medialogelhcr. Thc rhroughput ol thc disk musl be split in as

mânÿ pûrls âs ùe nunrber o[ mcdia. considcriûg thâl ilsclliciency dccrcrscs with this ûumbcr. Wc made a seriâl oilù§rs on our SUN3 Workst:r!rcn. involving lwo mediâ\imul:llcd rcquircmlnts- ÿ(lco ând lorcc wiû respccuve bilrurcs ol l50 and Il Kbytcs/s.This sLudy has shown hrt a\inglc hàd disk is unâblc to providc suiiicicnl ûrorghpulror mulriplc prcccsscs rn this contcxL. so unlcss 1l mul1i

disk.rrchircclurc rs aÿai1âblc, wc must usc thc throughput

nnr r.)J\l h) rhcnct$.rÀ Elhcrncl ,roürùr.J.§hich.llow.Jhour ôr' Kh) rcv' Iur.J! h lnnn(JlcJ soclcl I l0l.As â ùalrcr oi làcl. we deconrposcd thc problcm ofsynchronizalion in lwo sub problenrs: ânlicipalion and

communicàlion. Since lhc ânlicipalion hrs to do with the

rirning consuaints and thc sccnârio drc MCS dcâls with it-

On lhc oùcr lrand thc communications responsibility isgilcn t) thc COI'IM (Commonicîtion managco to frec thcI\'ICS liom lhcsc lunctions. Thc MCS slâys ûc mânagcr ofrhc whole process by imposing â synchronizâlion cyclingrime i.c. by slicing the timc in slots oi lixed sizc whoro allcvcnr occurring should lakc plûcc. In a given cyclc the MCSûchieves ùrce tâsks ol cquâl importancei it ânlicipâtes byordering re COMM to conneci objccls !o CCH Lo fill lhcirresourccs, it displâys ready obiects whcn their display limehas comc, and cvcntuâlly i! ordcrs the COMM to poll the

tho amounts of memory

4

differcnt open CCH âllributed to connected obie.§ 10 retrieve

dam on them. The COMM mâin duly is ùen to âchieve thispolhng ano di.conncct tilc ohrects when ùcy hrve comPlclcÙrh.ir frllins

ii. Conclù§ion

Wo treâÉd thc subiect of synchronizâtion of MuldmediaObjecrs in a particulâr environmenl ûat is lhe store ând

forward docùmcnls, on ât Etheme! locâl âreâ network. Such

link is unforlunatety asynchronous and thc prolocoi does nor

gu.rans any quaLly ùl §cnicc. For our purlose. whrch i\ ru

Lnk rogeLher a fc* .enerj ând a drsplry $ork'!âlron rn orJer10 play bâck â multimedia document. oûer kinds ofphysicalarchireclurcs ând dcdicatcd prorccols likc ISDN oTFDDI Jl l l

would bc more suiÈble. Rcalistic rcnderings are mrdcpo'!rble by inlcrlrcrnt our moriel ol mJllrnrcdir dôcur.,.nlprc.enLrlion ùrlh 3n X Window icr\cr ll Jnd clft(rc].mock-ups are being rcalized in the Mullimc(liaCommunication Lrb. of thc Univcrsity ol Ottawr.Althoùgh, the mclhods wc proposcd ibr achicving !h.synchronizalion arc closeiy dcpcndânt on the tcchnologycrrenlly available the functional model oi the synchronizcrând thc dilfcrenl enlitics dclincd shoLrld âcluâlly rcmàinrÿhatcver ùc meâns cen bc,

6. Acknowledsment

This rcsearch wâs supported in pùt by ùe Cânâdiân InstituLc

lor Tclccommunicâtions Rcsearch (CITR) and iû parl bl'lhcTcle.ommunications Rcscârch Inÿitute ol On1âr jo ('fRIO).

I1l

tll

tll

i6l

T.Relerenccs

Karen A. Frcnkcl, "The ncx! gcneration ol inteructilctechnologics', Communicalioos oi Thc ACM, luly1989, Volumc 32 Nùmber 7, pp. 872-8U1.

Edsard A. FOX. 'Thc -omrnÊ rc",,lurion rr

inleractive digi!âl video . Communicalions Oi TheACM. July 1989. Volumc 3: -wumber 7. pp. 791-

u01.

lvl. Sâlmony ilnd D. Sh.phcfu. E\lcndrng OSI 1()

rupporr synchroorrâ!roIr rcquircd h)' ntultlmodrrapplicâtions . IBM ENC Tcch. Rcp no 1i.390'1.Apr, 1989.

A. Kârmouch. L. Orozco Bâ-rhosa. N.D. Gùorgânr\'M. Goldberg, "A Mullimedia Medic:rlCommunicalion Syÿem , IEEE JournÀl on Sclectc(lAreâs in Communicalions. Volumc il Numbor 3.

April 1990, pp. 325-339.

compr(t Di(k_lnlerrclr\c. J ilcjiSncr \ "vcrv

c§ediled by Philips, Kluwcr Technicâl Books, Dovcntcr.Anlwerpcn, 1987.243 pages.

Rrll Srcinmcl/. Sln.hronru rrrr,n Ptnfcrlr(' 1

Multimcdiâ Syslems", 1EEE Jou.nrl on Sele.:rùlAreas in Communications. Volumc 8 Numbor.l,April 1990, pp. 401-112.

Final Report of the AFNOR Z60R - Commitlcô onlhc srândàrdizalion ol lhe Representatiorl and

frùl^ùol. lor Intcrchrngc ol AudlovisuJl InlerâctiveApplicalions (AVIs), " ISO JTCl N424, March 1989

89 pîges.

F. Oguet, C. Schwârtz, F. Krciz, M. Quere, "RAVI,A Proposcd Sundârd ior the Inlerchânge ofAudio/Visuâl Inieracdve Applicâdons". IEEE Journalon Sciecred Arcas in Communicâtions. Volume 8Number 1. AnIl I9o0. pp.428 416.

RAVI, spccification version V1 30/4/1987, revisedvcrsion Vl.2 30/9/88. Brochures 1.2.3,4. CCETT,Dpt PAA/MGA/DCT, 38-:10 rue du General Lcclerc,92I3I ISSY Lcs MoLrlineaux, France.

I.lrcmcru,,rking $ lr TaP-lP. D Cnmcr. Prcnuce-Hrll Pub.. 1989.

t7l

l8t

I9l

I10l

L11l F.E. Ross, An O!crÿicw ot FDDI: The FiberDi\rrihurc(l Dâtâ Intorlàcc'. IEEE Journalon SclectcdAreas in Communicalions, volùmc 7 Number 7,

Scptcmbcr l9li9. pp. 1043- 1051.

ll2l Win(io§, S)§tcms ProgrÀm,ning and Appiicationswi!h Xr. Douglas A. Young. Prcnlicc Hall Ed.. 1989.

I2l

i5l