6

Click here to load reader

Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

  • Upload
    voquynh

  • View
    213

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

Collaborative Multimedia Authoring: Scenarios andConsistency Maintenance

Bo XiaoFraunhofer IPSI

Dolivostr. 15, 64293 Darmstadt, [email protected]

ABSTRACTMultimedia authoring involves specifying the temporalcharacteristics of the media. In this paper collaborativemultimedia authoring (CMA) is advocated and its usagescenarios are described. A system model is presented toaddress CMA from various levels. The major issuesregarding consistency maintenance and temporalcharacteristics of CMA are identified. An approach usingoperational transformation to achieving consistency isgiven and investigated.

KeywordsCSCW, Collaborative multimedia authoring, Consistency.

INTRODUCTIONMultimedia authoring is a creative process involvingorganizing media objects in the spatiotemporal spaceaccording to the author’s plan. In this paper, the termmultimedia refers to a media composite with several mediatypes including time-dependent media and time-independent media. Because of its rich expressive power,multimedia authoring is used increasingly in a variety ofareas for entertainment, educational or business purposes.However, existing solutions for multimedia authoring areprimarily designed for single users, i.e., they do not addressthe requirements of supporting multiple users to performcommon authoring tasks at the same time. While CSCW isa research discipline on the communication, coordinationand cooperation among a group of people, its softwareoutput, the groupware, does not pay enough attention tointegrating multimedia as the involved content. In today’sresearch and industry there is a lack of support forcollaborative multimedia.For real-time (or synchronous) groupware, issues as for thearchitecture, the concurrency control policy and the userinterface and interaction mode are interesting topics toresearchers and system developers. In a networkenvironment where noticeable delays are inevitable (liketoday’s Internet), a suitable group editing system is the one

that avoids a single point of failure, enables quick response,provides reasonable user perspective (regarding e.g., theorder of operations, the granularity of awareness) andmaintains the consistency of the state and event orderscorrectly. Towards this aim, the following choices arepreferred [1,2]: a replicated architecture, optimisticconcurrency control schemes and methods to consistencymaintenance in such a context.In this work-in-progress paper, the temporal aspect ofCMA is the main topic. We present our work on usagescenarios, the features of collaborative multimediaauthoring, consistency maintenance issues, and theexploration of applying operational transformation [2,3,4],an advanced method developed for consistencymaintenance in collaborative editing systems in the lastdecade, to this area.The rest of this paper is organized as follows. Section 2provides an overview of the related work on multimediaauthoring and consistency maintenance for collaborativeediting. In Section 3, the usage scenarios are described anda classification of manipulation modes is made regardingthe temporal specification. In section 4, a system modelstructuring the levels of CMA is set up to facilitate theanalysis and designing. In section 5, an application modelis specified and an approach based on operationaltransformation to achieving consistency is given andinvestigated. In the last section a summary and our futurework are given.

RELATED WORKConsistency maintenance of shared documents under theconstraints of short response time and support for free andconcurrent editing in distributed environment is one of thefundamental and challenging issues in real-time groupeditors [3].Logical clocks or timestamps are basic tools forconsistency maintenance. In order to explore the eventordering issues in distributed systems, Lamport defined the“happened before” relation and based on it introducedlogical clocks into distributed systems [5]. The operationaltransformation technique was pioneered by the GROVEsystem [2]. Since then, the operational transformationmethod was extended by several research groupsindependently. The GROVE approach and the REDUCE

Page 2: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

approach [6] are based on the one-dimensional model ofconcurrent interaction. The adOPTed approach [4] is basedon a multi-dimensional model. The consistency correctnesscriteria are identified specifically.The techniques in collaborative editing are applied to avariety of collaborative editors for different applicationdomains: character-based text group editors (e.g., theREDUCE System [6]), table-based spreadsheet groupeditors [7], graphical object-based graphics group editors[8], and tree-based XML group editors [9].Collaborative systems integrating multimedia embrace thetasks on communication, presentation and authoring. JETSand jStreaming [11] demonstrates video communication incollaborative systems. The work in [12] focuses oninteractively presenting distributed media streams in thediscrete domain, primarily in simulation and cooperativeinteraction with 3-D models.JavuNetwork [10] provides a distributed architecture with acentral video editing application server and multiple java-based thin clients. In collaborative systems, its applicationdomain is the closest to ours. The difference is that it is notbased on a replicated architecture. Moreover, there is noresearch reported on the real-time concurrent interaction inediting.There is research work on multimedia authoring systems([13], [14]), yet it is not sufficient enough in thecollaborative context we are focusing on, e.g., it does notexplicitly consider the participation of multiple users in adistributed setting and working jointly on a single task.In multimedia authoring the capabilities of specifyingtemporal relationships are important. A number of temporalrelationships are presented by Allen [18] to representtemporal knowledge. Regarding the temporal aspect ofmultimedia, some layered models [15] have been developedto describe the characteristics, analyze the requirements andspecify the relationships concerning timing issues at eachlayer. The models classify the temporal concerns intodifferent levels and thus help the discussion in research anddevelopment at different levels of abstraction.

SCENARIOMultimedia content is being increasingly used forpresentations. Multimedia presentations contain severalmedia streams, for example, video, audio, slides, and text([16]). Authoring a multimedia presentation involvesspatial and temporal relationships governing the objectscomposing a document ([17]). In this paper only thetemporal aspect of CMA is discussed.Here we describe a scenario of using CMA. A group isgoing to make a video report on birds. After the camerawork of the team, the raw films are translated into thedigital form. The task of the postproduction team is editingthe digital media clips, creating special effects, insertingcaption and so on. The authoring sites are located indifferent cities. After exchanging raw materials, the staffstarts to edit the materials collaboratively. In the working

environment, they see a shared graphical view allowing formanipulation over tracks, e.g., a video track and an audiotrack as shown in Figure 1. The tracks are aligned by atimeline. As is in most of the multimedia authoring tools,the staff uses mouse operations to directly manipulate onthe graphical view and express their intended editingoperations. There are two modes of specifying the timepoint or time ranges which are parameters of editingoperations.

The first mode, which is time-based, uses absolute time. Inthis mode, time values in the timeline are used to specifythe operation on media clips at certain time points or duringcertain time ranges. Examples of editing operation are“exchanging the position of the video clip Flying Birds andthe video clip Landscape while keeping the backgroundmusic”, “removing the video clip between 3:20 and 3:30”and so on. Figure 2 shows the effect of the first example.

The second mode, which is relationship-based, uses pre-defined temporal relationships among the media clips. Thestaff may specify some relationships between mediaentities and use this specification in later editing operations,when the defined temporal relationships will be kept. Theycan specify, for instance, to have the audio clip BirdSinging start immediately after the starting of the video clipA Bird. Afterwards, when they perform an operation of“moving the video clip A Bird after the video clip FlyingBirds”, not only the video clip A Bird but also the audioclip Bird Singing will be moved accordingly so that thetemporal specification set previously is kept. Figure 3shows the effect of this example.From the above examples, we see that both absolute timecharacteristics and temporal relationship characteristics can

A bird Flying birdsLandscape

Bird singing Background music

Video

Audio

03:00 03:30 03:50 04:20

Fig.2. Effect of an exchange operation in a time-based mode

Video

Audio

A bird Flying birds Landscape

Bird singing Background music

03:00 03:30 03:55 04:20

Fig.1. User interface of a multimedia authoring system

Page 3: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

be involved in multimedia authoring systems. The CMAscenarios interesting us are those supporting the timeline-based editing interface, including multiple tracks, andallowing for manipulation within a track, across tracks(e.g., for multiple video windows) and on the relationshipsbetween media entities in different the tracks.Our following discussion on CMA in this paper is based ona replicated architecture with optimistic concurrencycontrol. The operational transformation method will beused to maintain the consistency within the sharedworkspace.

SYSTEM MODELWe propose a three-layered system model to describe howa collaborative multimedia authoring system is viewed inabstraction. The model consists of the media data layer, themedia interaction layer and the authoring UI layer, asshown in Figure 4.

The model provides different access levels in a CMAsession. The collaborative media data layer allows forshared access to the data of time-dependent media withtiming parameters concerning the QoS for intrastream andinterstream synchronization. The collaborative mediainteraction layer provides shared access to the logicalmodel of the media and temporal relationships amongthem. The collaborative authoring layer is for the multi-user interface of a CMA application.At the collaborative authoring layer, the objects of thegraphical view are targets of shared manipulation. Thesystem at this level seems to be similar to a group graphicobject editor, with the main difference that the editingoperations and effects are constrained to tracks. At thecollaborative media interaction layer, the model of media

content and temporal characteristics is manipulated. Since amedia track is basically modeled as a sequence of units,the system at this level seems to be similar to a group texteditor, with the difference that the data units contains muchmore content with temporal characteristics. At thecollaborative media layer, the physical data of mediastreams are manipulated. The media chunk data are oftenorganized according to specific formats. There are fewanalogies in existing group editors.The model allows for an analysis of the CMA, expressionof requirements at different levels and a way towardsfinding analogies in existing group editors. The remainingpart of this paper discusses issues of CMA in thecollaborative authoring layer and the collaborative mediainteraction layer.

CONSISTENCY MAINTENANCEModel of data and operationsWe define an authored multimedia document MD as a set oftracks ST = {Ti | i = 1, 2, …} and a set of media entities SE = {Ei |i = 1, 2, …}. A track T is a one-dimensional structure andconsists of media entities and empty segments (not coveredby any media entities). The boolean function IsEmpty(T, P1,P2) is used to indicate if the time range at [P1, P2] in thetrack T is empty. A media entity E is defined as a tuple (T,P, L), with T being the track containing E, P being thestarting time point, and L being the time length of E. Weuse P(E) and L(E) to represent the P and L components of Erespectively.In this data model of CMA, a multimedia document ismodeled as a set of tracks. Each track consists of mediaentities that can be taken as segments of the track. A trackis one-dimensional, with a time axis associated with it, andcontains a sequence of segments. A segment is associatedwith a time interval and thus has a start time point and anend time point which are defined by the time axis of thecontaining track. In a track no segments are allowed tooverlap in terms of the time. The document consists oftracks and owns a main time axis with which all the tracksare associated.Next we give two primitive editing operations in the datamodel: insert and delete. We use SE′ to denote the new setof entities, and P′(Ei) to denote the new starting point of anentity Ei after an operation.The operation Insert(E, T, P) is to insert the media entity Einto track T at the time point P. The effect of this operationis defined as follows:

(1) SE′ = SE ∪ {E}.(2) if IsEmpty(T, P, P + L(E)) is true, P′(E) = P.Otherwise,(3) if ∃ Ek ∈ SE, P(Ek) < P < P(Ek) + L(Ek), then P′(E) =P(Ek) + L(Ek).(4) if ∃ Ek ∈ SE, P < P(EK) < P + L(E), then let E0 =INF({Ei | Ei ∈ SE, P < P(Ei) < P + L(E)}), where INF(S)is the greatest lower bound of the set S. The effect

A birdFlying birds Landscape

Bird singingBackground music

Video

Audio

03:30 03:55 04:20 04:50

Fig.3. Effect of an exchange operation in a relationship-based mode

Collaborative Authoring

Collaborative Media

Collaborative Media Data Layer

Fig.4. Three-layer model for CMA

level of

abstraction

Page 4: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

of the operation is: P′(E) = P; ∀ Em satisfying P′ (Em) ≥P(E0), P′ (Em) = P(Em) + P′ (E) + L(E) – P(E0).

If the track area corresponding to the time range of E isempty before S is inserted, S is put into track T, withoutany changes to the rest part of T. If the track areacorresponding to the time range of E is not empty, thingswill be more complicated. If the head of E overlaps anexisting media entity E1, E should be moved forward(right) along the time axis until the head of E is alignedwith the tail of the E1. If E still overlaps some existingmedia entities, E should remain at P and all the entitiesoverlapped by E should be moved forward (right) adistance along the time axis until E overlaps none of them. The operation Delete(T, E) is to delete the media entity Efrom the track T. The effect of this operation is defined asfollows:

SE′ = SE – {E}.As defined above, a delete operation does not lead to anychanges to the media entities except for the deleted one.Some useful editing operations such as move, cut, copy,paste, and so on can be represented as a composition of theabove two primitive operations.

Appoach and issuesIn collaborative editing systems, the type of the involvedcontent being operated has great impact on consistencymaintenance algorithms based on operationaltransformation. Different types or natures are supported bydifferent algorithm components such as transformationfunctions. While there can be a generic operationaltransformation algorithm indicating when to do thetransformation, transformation functions defining how todo the transformation are dependent upon the specificcontent type that the algorithm is working for. Both thewhen and how-to factors are essential to achieve aconsistency maintenance scheme which should meet theusers’ expectation as well as possible.We find that the application independent part of theexisting solutions on consistency maintenance can be usedto CMA. With regard to the concrete application type, theapplication dependent result of the existing solutionscannot be used directly due to the difference between themodels. Now we give what we think is worth noticing.In the time-based mode, from the point of view of theoperational features, CMA is comparable to neithercollaborative character-based editing nor collaborativegraphic object editing. The design of transformationfunctions for CMA will be discussed in the next sub-section.In the relationship-based mode, besides the above issues,there are issues due to the interaction among the mediaentities. Because there are temporal relationships and amulti-track structure in CMA, there is a possibility thatcertain correlation exists between some segments in sometracks. Hence an operation on one segment of a track may

affect corresponding segments of other tracks. In this case,the effects have impact on both the collaborative authoringlayer and the collaborative media interaction layer.

Transformation functionsHere we give an example of concurrent editing in the sametrack that leads to inconsistency. In this example there aretwo users working at two sites. The current states of thesetwo sites are assumed to be the same. An operation O1 isissued at site1 and another operation O2 is issued at site2independently. To avoid inconsistency, we need totransform O1 into O1′ at site2 so that O1′ takes into accountthe existence of O2 and thus user1’s intention is maintained.As the context of O1 and the context of O2 are the sameaccording to the assumption, the computing of O1′ isactually to be the inclusion transformation function IT(O1,O2). It is the same for O2. We leave the discussion ontransformation functions to section 5.

As shown in figure 5, at two sites with the same contexttwo operations O1 and O2 are issued concurrently:

O1 = Insert(E1, T, P1)

O2 = Delete(E2, T)

If the operations are executed at remote sites withouttransformation, there can be problem of inconsistency. Asillustrated in figure 6, (a) is the initial state that is commonto both sites. (b) shows the intended insertion of E1. (c)presents the resulted state at site1 after a concatenation of O1

and O2. (d) presents the resulted state at site2 after aconcatenation of O2 and O1. Obviously, the final states at

Fig.5. Independent editing operations

Context(O1) = Context(O2)

O1

O1′

O2′

O2

t

initial state(a)

Fig.6. States before and after operations

(b)

(c)

E2 E3

E1 to be insertedE2 E3

state after O2 + O1

state after O1 + O2E3

E1

E1

(d) E1 E3

Page 5: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

the two sites are divergent. Operational transformations arenecessary.In example, we define that the final result of the concurrentediting operations is the state demonstrated by (c) in figure6.The result of the transformed operation of O1 at site2, O1’ =IT(O1, O2), depends on the relationship between E1 and E2.Therefore, six temporal relationships between them, whichare important for the inclusion transformation in theexample, are listed in figure 7.

The result according to the particular relationship is asfollows. O1’ = IT(O1, O2) =

For (a), O1’ = O1

For (b), (the case in the above example)O1’ = O1, and the media entities that werefollowing E2 should be left-shifted s, where sis the distance that the media entitiesfollowing E2 were right-shifted due to O2.

For (c), same as the case of (b)For (d), O1’ = Insert(E1, T, P(E2) + L(E2), and the media

entities that are following E1 should be right-shifted to the position where they would havebeen is E2 were not deleted..

For (e), same as the case of (d)For (f), O1’ = O1

The procedure of getting the results is omitted here.From the example we see that in the editing scenario ofCMA, the result of operational transformation should

involve not only the transformed operation itself but alsoother parts of the state.

CONCLUSIONThis paper advocates the usage of authoring multimediajointly to expand the document types in collaborativeediting. The usage scenarios described here includes twooperational modes, namely the time-based mode and therelationship-based mode. A three-layer system model ispresented to address CMA from various levels ofabstraction. Consisting of the collaborative authoring layer,the collaborative media interaction layer and thecollaborative media data layer, the model intends tofacilitate the analysis and designing work and discussion inthe research of CMA. Based on the data model andoperation model with respect to applications, the nature ofCMA is extracted and the impact on consistencymaintenance is analyzed. An approach based on operationaltransformation is presented and discussed. An exampleproblem is given and and some transformation functionsare designed for it.This paper does not provide a complete solution to theconsistency maintenance in CMA. That belongs to thegoals of our future work in this area. In our three-layeredmodel, low-level issues are not discussed here. Some topicsinteresting to us but not included in this paper are thespacious aspect of the spaciotemporal specification ofmedia relationships, multi-version tracking, and groupundo/redo.A prototype of CMA is under development. We are goingto evaluate our schemes in the prototype and work towardsa natural user perception. The concepts and prototype willbe integrated into our future cooperation platform.

ACKNOWLEDGMENTSWe would like to thank Torsten Holmer, Stefan Muenzer,Martin Wessner, Peter Dawabi, Friederike Joedick andJessica Rubart and for the discussion on this research topic.Martin Wessner reviewed the paper. Their comments thathelped to improve this paper are gratefully appreciated.

REFERENCES1. S. Greenberg and D. Marwood. Real Time Groupware as a

Distributed System: Concurrency Control and it’s Effect onthe Interface. In Proc. 1994 ACM Conference on CSCW,Chapel Hill, USA, October 1994, pp.207-217.

2. C.A. Ellis and S.J. Gibbs. Concurrency control in groupwaresystems. In Proc. ACM SIGMOD Conference. onManagement of Data, Portland, USA, June 1989, pp. 399-407.

3. C. Sun and C.A. Ellis. Operational Transformation in Real-Time Group Editors: Issues, Algorithms, and Achievements.In Proc. 1998 ACM Conference on CSCW, Seattle, USA,November 1998, pp. 59-68.

4. M. Ressel, D. Nitsche-Ruhland, and R. Gunzenhaeuser. AnIntegrating, Transformation-Oriented Approach toConcurrency Control and Undo in Group Editors. In Proc.1996 ACM Conference on CSCW, Cambridge, USA,November 1996, pp. 288-297.

E1

Fig.7. Temporal relationships between two media entities

E2

E1 E2

E2

E1

(a) E1 before E2

(b) E1 before E2 with intersection

(d) E1 contained by E2

(c) E1 containing E2E1

E2

(e) E1 after E2 with intersection E2 E1

(f) E1 after E2 E2 E1

t

Page 6: Collaborative Multimedia Authoring: Scenarios and ... fileCollaborative Multimedia Authoring: Scenarios and Consistency Maintenance Bo Xiao Fraunhofer IPSI Dolivostr. 15, 64293 Darmstadt,

5. L. Lamport. Time, Clocks, and the Ordering of Events in aDistributed System. Communications of the ACM, 21(7),July 1978, pp. 558-565.

6. C. Sun, X. Jia, Y. Zhang, and et. al. Achieving Convergence,Causality-preservation, and Intension-preservation in Real-time Cooperative Editing Systems. ACM Transactions onComputer-Human Interaction, 5(1), March 1998, pp. 63-108.

7. C. R. Palmer and G.V. Cormack. Operation Transforms for aDistributed Shared Spreadsheet. In Pro. 1998 ACMConference on CSCW, Seattle, USA, November 1998, pp.69-78.

8. D. Chen and C. Sun. A distributed Algorithm for GraphicObjects Replication in Real-time Group Editors. In Proc.1999 ACM Conference on Supporting Group Work, Phoenix,USA, November 1999, pp. 121-130.

9. A.H. Davis, C. Sun and J. Lu. Collaborative Editing forXML Documents – An Operational TransformationApproach. Third Annual Collaborative Editing Workshop,ACM Group 2001, Boulder, USA, September 2001

10. S. Kamara. JavuNetwork: Remote Video Production andStorage. IEEE Multimedia, 6(3), July-September 1999, pp.78-80.

11. J. C. de Oliveira, F. Malric, D. Yang, S. Nourian and N.D.Georganas. Java Multimedia Telecooperation. In Proc. 2001ACM Conference on Multimedia, Ottawa, Canada, 2001, pp.607-708.

12. M. Mauve. Consistency in Replicated Continuous InteractiveMedia. In Proc. 2000 ACM Conference on CSCW,Philadelphia, USA, December 2000, pp. 181-190.

13. R. Boutaba, N. Ren, Y. Rasheed and A. Leon-Garcia.Distributed Video Production: Tasks, Architecture and QoSProvisioning. Multimedia Tools and Applications, 16, 2002,pp. 99-136.

14. M. Vazirgiannis. Interactive Multimedia Documents. LectureNotes in Computer Science #1564, Springer, 1999.

15. R. Steinmetz and K. Nahrstedt. Multimedia: Computing,Communications and Applications; 1st edition. Prentice HallPTR, July 1995.

16. B. Prabhakaran. Multimedia authoring and presentationtechniques. Multimedia Systems, 8, 2000, pp. 157.

17. C. Huang and C. Wang. Synchronization for InteractiveMultimedia Presentations. IEEE Multimedia, 5(4), October-December 1998, pp. 44-62.

18. J.F. Allen. Maintaining Knowledge about TemporalIntervals. Communications of the ACM, 26(11), November1983, pp. 832-843.