9. Multimedia Authoring & Synchronization - Copyright © Denis Hamelin - Ryerson University Lesson #9 Multimedia Authoring & Synchronization

9. Multimedia Authoring & Synchronization - Copyright © Denis Hamelin - Ryerson University

Lesson #9

Multimedia Authoring &

Synchronization


Synchronization

• Limited forms of multimedia have always been on the web:• Play sound in background on HTML page. • Download AVI/MOV files and play locally. • Stream video/audio files.• Shockwave or Flash animations via Plug-Ins.• Animated GIFs • Various Dynamic HTML to animate text & images


Synchronization Constructs

• Spatial location. • Timing: Parallel and Sequential. • Associate hyper links. • Load Balancing: Different media on different servers. • Language Selection: Audio track is available in several

languages. • Bandwidth Selection: Audio track encoded for different

transmission bandwidths. • Content media.


Synchronization Systems• IconAuthor (Asymetrix): iconic/flow control paradigm.• MetaCard: card/scripting authoring system.• Authorware (Macromedia): iconic/flow control

paradigm.• Dazzler: hierarchical object paradigm.• Director (Macromedia): cast/score/scripting paradigm. • Flash (Macromedia): cast/score/scripting paradigm.• Formula Graphics: hypermedia linkage authoring

system.• And many more...


Synchronization Paradigms

•Scripting Paradigm: The Scripting paradigm is the authoring method closest in form to traditional programming. The paradigm is that of a programming language, which specifies (by file name) multimedia elements, sequencing, hotspots, synchronization, etc. A powerful, object-oriented scripting language is usually the centrepiece of such a system; in-program editing of elements (still graphics, video, audio, etc.) tends to be minimal or non-existent. The scripting paradigm tends to be longer in development time . •Ex: HyperTalk, OpenScript, Lingo.



•Iconic/Flow Control Paradigm: This tends to be the speediest (in development time) authoring style; it is best suited for rapid prototyping and short-development time projects. The core of the paradigm is the Icon Palette, containing the possible functions/interactions of a program, and the Flow Line, which shows the actual links between the icons. •Ex: Authorware, IconAuthor.



• Frame Paradigm: The Frame paradigm is similar to the Iconic/Flow Control paradigm in that it usually incorporates an icon palette; however, the links drawn between icons are conceptual and do not always represent the actual flow of the program. This is a very fast development system, but requires a good auto-debugging function, as it is visually un-debuggable. The best of these have bundled compiled-language scripting, such as Quest (whose scripting language is C) or Apple Media Kit.



• Card/Scripting Paradigm: The Card/Scripting paradigm provides a great deal of power (via the incorporated scripting language) but suffers from the index-card structure. It is excellently suited for Hypertext applications, and supremely suited for navigation intensive (a la Cyan's "MYST" game) applications. They are widely used for shareware applications. The best applications allow all objects (including individual graphic elements) to be scripted; many entertainment applications are prototyped in a card/scripting system prior to compiled-language coding.



•Cast/Score/Scripting Paradigm: This paradigm uses a music score as its primary authoring metaphor; the synchronous elements are shown in various horizontal tracks with simultaneity shown via the vertical columns. The true power of this metaphor lies in the ability to script the behavior of each of the cast members. The most popular member of this paradigm is Director, which is used in the creation of many commercial applications. These programs are best suited for animation-intensive or synchronized media applications.•Ex: Macromedia Director and Flash.



• Hierarchical Object Paradigm: The Hierarchical Object paradigm uses a object metaphor (like OOP) which is visually represented by embedded objects and iconic properties. Although the learning curve is non-trivial, the visual representation of objects can make very complicated constructions possible.



• Hypermedia Linkage Paradigm: The Hypermedia Linkage paradigm is similar to the Frame paradigm in that it shows conceptual links between elements; however, it lacks the Frame paradigm's visual linkage metaphor.



• Tagging Paradigm: The Tagging paradigm uses tags in text files (for instance, SGML/HTML, SMIL (Synchronised Media Integration Language), VRML, 3DML and WinHelp) to link pages, provide interactivity and integrate multimedia elements.


SMILSynchronized Multimedia Integration Language

• SMIL (pronounced "smile") is defined as a set of XML modules which are used to describe the temporal, positional, and interactive behavior of a multimedia presentation. Informally, SMIL is a text-based markup language (similar to HTML) which can be used to add timing, layout, animation, and events to what would otherwise be just another static web page.

• A simple text editor is the only tool necessary for creating and editing SMIL presentations.


Advantages of SMIL

SMIL is free and open. Anyone who wants to create a SMIL presentation can do so free of charge. Not only that, but software writers can also create software to edit and render SMIL presentations without having to pay any fees or purchase any special development tools. SMIL is easy to learn and use. Anyone familiar with HTML will quickly adjust to writing SMIL documents. And, since SMIL was written as a set of modules, you need to learn only the modules which you want to use.


Advantages of SMIL

SMIL provides capabilities beyond those of any multimedia format yet seen on the web. While many programs and plug-ins exist for animations, layout, etc, none of them combine all of these features seamlessly while providing unique content-control, synchronization, and meta information facilities. However, SMIL was not designed to replace any of these other formats, but instead to supersede them. A SMIL file can encapsulate your favorite audio, video, and animation files and provide you with greater control over their presentation.


Writing SMIL

SMIL documents are actually XML documents which conform to the SMIL XML DTD (XML stands for eXtensible Markup Language and DTD stands for Document Type Definition).

Basic declarations:

<?xml version="1.0"?><!DOCTYPE smil PUBLIC "-//W3C//DTD SMIL 2.0//EN" "http://www.w3.org/2001/SMIL20/SMIL20.dtd">


SMIL Syntax

•A document is made up of elements. •Each element has either a start-tag and end-tag or an empty-element tag. •Tags are identifiers enclosed within less-than (<) and greater-than (>) characters. •Elements are ended with a forward-slash (/) character, either at the beginning of an end-tag or at the end of an empty-element tag. •Elements may have one or more attributes. Each attribute is a name/value pair in the form name="value" (The value must be enclosed in quotes).


SMIL Syntax

• A set of start and end tags surround the text (or other elements) which they're marking up. (Empty-element tags contain all of their information as attributes.)

• Elements may contain other elements (this is called nesting).

• XML tags are case sensitive (like XML, unlike HTML).• Empty-element tags require a forward-slash (/) at the

end of the tag (before the greater-than (>) character). • Elements with a start-tag must have a corresponding

end-tag.


The <smil> Root Element

Like the <html> element in html, it surrounds the entire document.

<smil xmlns="http://www.w3.org/2001/SMIL20/Language" xml:lang="en" title="SMIL template">

smil code here...

</smil>


The <head> and <body> elements

• Just like an HTML document, a SMIL document is divided into its <head> and <body>. The <head> of the document is used to describe the content and the <body> is used to present it. Most of a SMIL presentation is located inside of the <body>.

<?xml version="1.0"?><!DOCTYPE smil PUBLIC "-//W3C//DTD SMIL 2.0//EN" "http://www.w3.org/2001/SMIL20/SMIL20.dtd"><smil xmlns="http://www.w3.org/2001/SMIL20/Language" xml:lang="en" title="SMIL template"><head></head><body></body></smil>


SMIL Media Types

• img: a still image, just like one you would use in an HTML page (jpeg, gif, png, etc).

• video: a video, possibly also containing audio (mpeg, avi, mov, etc).

• audio: stand-alone audio, either compressed or raw (mp3, wav, au, etc).

• animation: an animation, usually using vector graphics (svg, swf, vml, etc).


SMIL Media Types

•text: plain old text, promoted to the level of media object (txt, ascii, etc). •textstream: stream of plain old text along with timing (and possibly other) information (sub, rt, sami, etc). (Think of a news ticker or subtitles.).•SMIL distinguishes between discrete (images, text) and continuous (video, audio, animation, textstream) media.

<img src="picture.jpg"/><video src="movie.mpg"/>will show the image then play the video.


SMIL Accessibility

•alt: This is a brief textual description of the file, usually not even a complete sentence. •longdesc: With longdesc you can provide a link (uri) to a more detailed description, from a simple sentence to a book-length document. There is no set category for the type of description.<video src="movie.mpg" alt="clip from the Terminator 2 movie" longdesc="moviedescription.txt"/>•If someone is unable to view the picture or movie they'll still be able to have some idea about the content of the files.


SMIL : Media Clipping

•Clipping can only be applied to continuous media such as audio, video, and text streams. •No extra software is required!•Specify when the clip should begin and end using the clipBegin and/or clipEnd attributes. Both attributes take time values in the smpte or npt formats.

– Starting 20 seconds into the video (30 min. movie): <video src="movie.mpg" clipBegin="20s"/>

– Cutting out the last 3:30 minutes: <video src="movie.mpg" clipEnd="26:30"/>

– Showing an interesting frame: <video src="movie.mpg" clipBegin="14:55.7" clipEnd="14:55.7"/>


SMIL : Timing and Synchronizing

• Perhaps smil's greatest strength is its very robust and flexible method of sequencing multimedia. You can be as precise or vague as you like, specifying timing down to the sub-frame level or leaving the calculations up to the smil player itself.

• Presenting Media Serially: The element you would use in this case would be <seq>, which stands for sequence. The semantics for <seq> are very simple: it plays each child media element in order, one-at-a-time.



<seq>

<video src="video1.mpg" alt="video#1"/>



</seq>

• The three videos will play one after the other.



• The <par> tag indicates that the elements will play in parallel (like picture-in-picture).

<par>

<img src="arnold.jpg"/>

<video src="bike.mpeg"/>

</par>

• The two videos will play simultaneously.


SMIL : Timing

•SMIL gives you a number of ways to specify the start and end times of media objects. This is done by adding extra attributes to a media object. However, an element's timing only makes sense in the context of its parent synchronization element.

•Here are some examples:– "16" : 16 seconds (the same as "16s"). – "02:45:14.273" : 2 hours, 45 minutes, 14 seconds, and

273 milliseconds. – "07:00" : 7 minutes.


SMIL : Timing

– "30m" : 30 minutes.

– "1h" :1 hour.

– "1.48" :1 second and 480 milliseconds.

– "1480ms" : 1480 milliseconds (the same as above).

– Values after a decimal point indicate fractions of a second when dealing with timing and frame numbers when dealing with clipping.


SMIL : Timing Sequences

•Note this sequence:

<seq>

<video src="video1.mpg"/>

<video src="video2.mpg" begin="00:02"/>


</seq>

•A 2-second delay is inserted between each video.

•Clock values are specified as offsets. They're timed from the point at which the element would otherwise normally begin.


SMIL : Timing in parallel

•An element without a begin value is assumed to have one set to "0". For parallels, the timing involves the beginning of the parallel segment.<par>

<video src="video1.mpg"/>



</par>

•The second and third videos will start 2 seconds after the first one has started.



• You can end a group of clips at the same time by using the endsync attribute with the <par> tag.

• Use the attribute endsync="first" to stop the <par> group when the clip with the shortest timeline finishes playing. All other clips in the group stop playing at that point, regardless of their playback status or any timing parameters specified for them.

• The attribute endsync="last" causes the <par> group to conclude when all clips have finished playing. Because this is the default value, you can omit the endsync attribute from the <par> tag to achieve this effect.



• The attribute endsync="id(clip ID)" causes the <par> group to conclude when a designated clip reaches its end. All other clips in the group stop playing at that point, regardless of their playback status or any timing parameters used with them. The designated clip must have a corresponding id value in its source tag.


SMIL: Synchronizing Off of Another Element

• A <seq> automatically synchronizes elements based off of the end time of the previous element. It turns out that we can specify this type of behavior inside of a <par>. For parallels, the timing involves the beginning of the parallel segment.

<par><video id="v1" src="video1.mpg"/><video id="v2" src="video2.mpg" begin="v1.end+00:05"/></par>

• The second video will start 5 seconds after the first one has ended.


SMIL: Nested Elements

• Elements can be nested: <par> <audio src="ad_music.mp3"/> <seq> <video src="ad1.mpg"/> <video src="ad2.mpg" begin="00:02"/> <video src="ad3.mpg" begin="00:02"/> </seq> </par>• The audio part will play in parallel with the three videos

playing in sequence.


SMIL: Looping Media

•You can specify iterations: <audio src="ring.wav" alt="telephone ring" repeatCount="4"/>

•You can specify a fixed duration: <audio src="ring.wav" alt="telephone ring" repeatDur="16s"/>

•You can synchronize with another element: <par> <audio id="ring" src="ring.wav" alt="telephone ring" repeatCount="4"/>

<img src="jumping_phone.gif" alt="a violently ringing telephone" begin="ring.repeat"/>

</par>


SMIL: Looping Media

• In the previous example, the image appears only at the second ring, not the first. It is because only the second ring is considered a repetition. To solve this we can add multiple timing. We separate each timing by a ;.

<par> <audio id="ring" src="ring.wav" alt="telephone ring" repeatCount="4"/>

<img src="jumping_phone.gif" alt="a violently ringing telephone" begin="0; ring.repeat"/>

</par>

• We can use ring.begin instead of 0.


SMIL: Looping Media

• There is also another legal value for both repeatCount and repeatDur: "indefinite". This will loop a media object continuously until something causes it to stop.

<seq repeatCount="indefinite"> <audio src="ring.wav" alt="telephone ring" repeatCount="9"/>

<audio src="operator.wav" alt="operator saying that the party is not answering"/>

</seq>

• As you can see, the loop option can be applied to the seq or par tag also.


SMIL: Dividing Space into Regions

•SMIL regions are always rectangular.

•Arranging media in SMIL is done (usually) as a two-step process. First, a region is created, and then a media object is tied to that region. For creating regions, SMIL has the element <region>. It has a number of attributes (11 to be exact).

– width and height: width and height of the region. – left, right, top, and bottom: values for the extremities of the

region. – backgroundColor and showBackground: color of a region (or

at least the parts of the region not obscured by media objects).



– fit: the fit of a region controls what happens to a media object when its intrinsic width and height do not match that of the region it's tied to.

– regionName: region name is similar to an id attribute. However, the regionName need not be unique.

– z-index: the z-index gives a weighted value of the region's position on the z-axis. (The z-axis is the third axis of three-

dimensional space, the first two being the x and y axes.) .•Attributes which are screen measurements (width, height, left, right, top, and bottom) can take relative values (px), absolute values, or percentages.



Examples:<head><layout><region id="CIF-NTSC" width="352px" height="240px"/><region id="US-photo" width="200px" height="100px"/><region id="half-center" left="25%" top="25%" width="50%" height="50%"/></layout></head>

• See that the regions are declared in the layout tag in the head section.


SMIL: topLayout• Indicates different windows. The <topLayout> element

has five attributes, three of which we've already seen in the <region> element. They are width, height, backgroundColor, open, and close.

• Both the open and close attributes control when a window specified by a <topLayout> is displayed. open can take the value of "onStart" and "whenActive". close, in contrast, can be either "onRequest" or "whenNotActive".

<layout>

<topLayout width="640px" height="480px" backgroundColor="#FFFFFF" open="whenActive" close="whenNotActive"/>

</layout>


SMIL: <root-layout>

• In the SMIL file header, between <layout> and </layout> tags, the <root-layout/> tag indicates the width and height of the overall playback area in pixels. The a root-layout region that sets the overall size of the media player main window and additional regions that define the sizes and locations where clips will be played. The following example creates a root-layout region 450 pixels wide by 450 pixels high.

<root-layout width="450" height="450"/>


SMIL: Sound layout

• The concept of audio layout in smil is actually very simple. In fact, the AudioLayout module adds only one thing to smil: the soundLevel attribute for a <region>. Basically you can think of soundLevel as a volume setting.

• soundLevel is specified as a percentage greater than or equal to 0. As with other <region> attributes, soundLevel is relative.


SMIL: Layers

• Using the simple coordinate system, you may choose to define regions that overlap. If so, you also need to indicate the layer order of all of overlapping regions. If you don't specify a layer order, the media player default is to display the list of clips from bottom to top (that is, the one listed first will be on the bottom and the one listed last will be on top).

• The z-index parameter allows you to assign a numerical value to each region, where 0 is the layer displayed on the "bottom" and an infinite number of regions can be created on top by assigning playback regions values such as z-index="1" or z-index="6".


SMIL: Background Color

• By default, the root-layout region is black. All other regions use transparency as their default. In the SMIL layout, you can specify a different background color for any region.

<layout> <root-layout width="450" height="450" background-color="#ffcc00" />

<region id="image" top="5" left="5" width="150" height="75" z-index="2" background-color="navy" />

<region id="text" top="50" left="50" width="100" height="100" z-index="3" background-color="red" />

</layout>


SMIL: Tying Media Objects to Regions

• Once you've declared a region in the <layout> section of your document, it's a simple matter to have media objects rendered to it. Just add the id of the region to a region attribute in a media object.

<head> <layout> <region id="half-center" left="25%" top="25%" width="50%" height="50%"/>

</layout> </head> <body> <img src="photo.jpg" alt="picture of a snowman in a blizzard" region="half-center"/>

</body>


SMIL: Stretching and Squeezing

• What would happen to the above <img> if its intrinsic width and height had been different from that of the <region>? That depends on the value of the fit attribute of the region.

• By default, a region has a fit attribute of "hidden". What this means is that if a media object is small enough to fit inside of the region, it will be rendered starting from the top-left corner of the region and any remaining space will be filled by the region's backgroundColor. If the object doesn't fit within the region it will be hidden (not rendered). There are also other fit values you can use.


fit = "hidden"



• "slice": functions the same as "hidden" except that when the media object is too large for the region it is clipped to fit. (This is spacial clipping, not to be confused with temporal clipping which you encountered earlier.)



• "scroll": functions the same as "hidden" except that when the media object is too large for the region, scrollbars are added.



• "fill": automatically fills or scales (stretches or squeezes) media objects to fit exactly within the region.



• "meet": functions the same as "scale" except that it preserves the aspect ratio. This means that both the width and height are scaled at the same rate until either of them reach the full extent of the containing region.


SMIL: Fills

• If you don't want the clip to loop, you can use the fill attribute to either remove or freeze it.

• The default value fill="remove" removes the clip. When this attribute is used with a still image, the image disappears as soon as the end time has been reached.

• Use fill="freeze" to freeze a clip on its last frame. When you use fill="freeze" with a video, the video's last frame stays on the screen. If used with an image, the image remains in its playback area. For example:

<img region ="five" src="arnold.jpg" fill="freeze" />


SMIL: Nesting Regions

• This allows you to make much more flexible and complex layouts. Syntactically, nesting regions is very simple. Instead of using an empty-element <region>, use one with start and end tags.

<region id="two-column" width="100%" height="100%"> <region id="column1" left="0%" top="0%" width="50%" height="100%"/>

<region id="column2" left="50%" top="0%" width="50%" height="100%"/>

</region>


SMIL: Picture-in-Picture

• Our goal here is to design a smil presentation which acts like a picture-in-picture television screen. To do so we'll need two video sources, one which will fill the screen and another which will occupy the 1/9th of the screen area in the bottom-right corner. This means that the second video should be 1/3rd the width and 1/3rd the height of the first.

• First, we create the desired regions, including a <topLayout> which matches the size of our largest video.


SMIL: Picture-in-Picture

<layout> <topLayout width="320px" height="240px"> <region id="main-video" left="0%" top="0%" width="100%" height="100%"> <region id="corner-video" left="67%" top="67%" width="33%" height="33%" fit="scale" soundLevel="0%" /> </region> </topLayout></layout>

<par> <video id="chameleon-video" src="chameleon.mpg" alt="an animated chameleon" region="main-video"/> <video id="earthquake-video" src="earthquake.mpg" alt="San Fransisco earthquake aftermath" region="corner-video" end="chameleon-video.end"/></par>


SMIL: Interaction

• The <a> element has the same function as in HTML. Its main attribute is href but there are others in SMIL.

• show: this allows you to control whether or not activating the link replaces the currently playing presentation ("replace") or creates a new one to run in parallel ("new").

• destinationPlaystate: you can set this to "play" (default) to have the presentation in the link immediately start playback or "pause" to wait for a command from the user instead.


SMIL: Interaction

•SourcePlaystate: whether or not you have a link replace the current presentation, you can control how the currently running presentation behaves.

• "play": continue playing the presentation. Note that this really only makes sense if you did not set show to "replace".

• "stop": pause the presentation. The user will be able to restart it at the point when it stopped before the link was followed.

• "pause": this is the same as "stop" except that the presentation resumes where it left off.


SMIL: Interaction

• target: specifying a target allows you to load a link in a specific <region>. This can also be used for html frames and other targets which I won't cover in this tutorial.

• accesskey: this functions the same as the html accesskey attribute. Simply provide a character and the user will be allowed to type that key instead of or in addition to clicking in order to follow the link.

• sourceLevel and destinationLevel: both of these attributes function like soundLevel except that they control the volume of the entire source and/or destination presentation once the link is activated.


SMIL: Interaction

• actuate: this specifies whether the link must be activated (clicked) by the user ("onRequest") or if it is activated automatically ("onLoad").

• To use <a> to create a link, simply enclose a media object with it.

<a href="second.smil" alt="the next presentation in the series" show="replace">

<animation src="credits.svg" alt="the names of the makers of this presentation"/>

</a> <a href="pause.smil" alt="simple animated pause loop" show="new" sourcePlaystate="pause" accesskey="p">

<img src="pause.png" alt="two vertical bars"/> </a>


SMIL: Interaction

• You can link to different destinations in the same show either spatially or temporally by using the <area> element.

<video src="advertisements.mpg" alt="a series of advertisements" dur="01:30">

<area href="http://advertiser1.com/" begin="00:00" end="00:30"/>



</video>

• For temporal image maps: http://www.abiglime.com/webmaster/reference/html/tags/area.htm


SMIL: Animation Elements

• <animate>:This is used for creating simple animations, such as linear changes to an object's attributes.

• <animateMotion>:This is the type of animation we usually think of when hearing the term animation. As you can guess, this is used for creating movement.

• <animateColor>:Yes, color can be animated too! Imagine psychedelic and fade-in/out effects and you'll have the idea behind this element.

• <set>:This is perhaps the most boring animation element. If the other animation elements are triangle and sine waves, this one is the square wave.


SMIL: Animation Elements Attributes

• All the timing attributes plus a few more:

• attributeName: Specifies which attribute of the parent element will be animated. This is the name on the left side of the equal sign (=): for example, the width in width="100%".

• targetElement: This allows you to animate an element without directly embedding the animation element inside of it. This would be useful for animating a region, for example.



• from, to, and by:These specify the target range of animation. You can think of these as the animation equivalents of begin, end, and dur respectively.

• values:This allows you to use a list of values rather than a range. The syntax for this list of values is the same as that for a list of begin values: each value is separated by a semi-colon (;).



• calcMode: can be either "discrete", "linear", or "paced". A linear animation can have constant acceleration while a paced animation always has constant velocity, and a discrete animation has constant position (I use that last term loosely). Anyone familiar with the calculus should see a relationship between these.

• A cumulative animation uses a repeatCount or repeatDur attribute to repeat the animation. It also uses accumulate="sum" to increase or decrease the animated value with each iteration. You can use cumulative animations for sizes and placements.


More SMIL

• See the Bookcase for chapters on• Animation• Transition Effects• RealText Markup• SMIL in Internet Explorer• SMIL and SVG


End of lesson

Documents

9. Multimedia Authoring & Synchronization - Copyright © Denis Hamelin - Ryerson University Lesson #9 Multimedia Authoring & Synchronization