COM 597Streaming Media

Class 2

June 28, 2007

Today’s topics:

• The nuts and bolts of Audio• The encoding and delivery process (sampling

& resampling)• Aliasing• Audio Workshop• Codecs and Players• Compression demonstration• Compression workshop

The process of compressing a file from 40,000 feet up (the big picture)

What is Video?

Video is basically a three-dimensional array of color pixels. Two dimensions serve as spatial (horizontal and vertical) directions of the moving pictures, and one dimension represents the time domain.

A frame is a set of all pixels that (approximately) correspond to a single point in time.

Digitizing• This is the process of taking an analog signal and turning

it into an approximate digital representation of the original image and sound.

• Examples of digitizing: scanning a photograph, capturing video into a computer, capturing a recording

• Ripping a CD is not technically “digitizing” because it is already data. It is usually referred to as “capturing”

Compression in simple terms is reducing the data used to display an image, play an audio file or present video. It is used throughout the industry.

• Cable TV• Editing platforms• DVD• TiVo / PVD• Video Acquisition

Data can not be put back once it has been removed.

Spatial Compression• Spatial encoding is performed by taking

advantage of the fact that the human eye is unable to distinguish small differences in color as easily as it can changes in brightness and so very similar areas of color can be "averaged out"

• Common image file examples are .jpg, .tiff, .gif and .png

Spatial Compression

Points are usually described with Cartesian values (X-Y)

• X side to side• Y up and down• Z close and far away

Temporal Compression

With temporal compression only the changes from one frame to the next are encoded as often a large number of the pixels will be the same on a series of frames

Low motion example 276 KbpsMedium motion example 344 KbpsFast motion example 354 Kpms

When not to compress

• Acquisition and origination• Archiving• Mastering

You (your client or boss) have to decide what is an acceptable level of compression based upon the variables we will be discussing in class.

Other nifty terms we will be using:• Metafile: A small file on a web server that includes

information (metadata) that informs a player where to locate a file on a media server. A roadmap, if you will

• Encoder: a software or hardware application that transforms a source media file into a file that can be streamed efficiently.

• FTP Client: software on a client computer or server that uses File Transfer Protocol to upload or download files from another computer at a remote location.

Additional websites for reference:

http://www.streamingmedia.com/magazine/http://www.digitalproducer.com/http://www.streamingmediabible.com/http://www.w3.org/AudioVideo/

Audio

• How do you deal with legacy material?

Audio sources:RecordsOpen reel tapeCDDVDOff-air recordingDownloaded filesOptical sound from a filmSurround soundEmbedded audio

Analog v. digital audio

What is the difference?Analog material must be played in real timeYou will need to use a software tool or hardware device to convert these different physical representations of sound (grooves in a record for example) into a digital representation of the sound. This process can happen in a number of different ways during your production path.A-to-D converters (scanners)

Analog v. digital audio

What is the difference?

Sample rate will determine the fidelity of your final product

Remember you cannot add detail to a sampled media file so it is best to start with a file that is as large as is reasonable. (RAW v. JPEG for example)

Raw Image .jpg Image

Resolution Comparison Source v. Delivery

Raw Image .jpg Image

Resolution Comparison Source v. Delivery

Compression and Bitrate

• To easily understand how compression can impact the fidelity (quality) of a file listen to this demo.

Click me… please

At a certain threshold there is not a perceptible difference in the sound

• Sample rate is not bits-per-second, although they are related

• Sample rate is the number of “slices” of sound per second where bps is about how much data is begin delivered in the signal. Bps can be 22KHz or 96KHz or even 192KHz. For this class most of our work will be either in 44.1KHz or 48KHz.

• When you sample a sound you need to record both the wave peaks and troughs

• Human ear is limited to detecting frequencies

no higher than 22KHz. • It was believed that simply doubling this

number would give a sample rate that would reproduce all the sounds we can hear. Therefore, CDs run at 44.1KHz

Resampling a fileAnd the problems inherent in the

process

The difficulty with 44.1 as a standard is that it does not integrate easily into a

standard production path.• Most audio files are converted using integer-

based averaging. • This becomes apparent when you see that the

common rates used for mastering are 48, 96 and 192 Khz. Moving down in quality involves simple averaging and not a complex resampling of the file.

Aliasing Artifacts

Aliasing Artifacts

Aliasing Artifacts

Aliasing Artifacts

Aliasing Artifacts

Aliasing Artifacts

• These quantization errors become evident when you convert between CD rates and film/television production rates. It is important to know your delivery before you start the process of sampling your sound.

• This is also why it is important to make certain all the tools in your production path are in agreement. (set to the same sample rate)

Audio formats you are likely to encounter

• Mono one track of audio• Dual Mono (this is not stereo) identical audio on

both tracks• Stereo multi-channel audio on two tracks• Mix Minus production audio with two mono tracks

containing different sources (dialogue on one and effects on the other is one

common example)• 5.1 surround Home theater is the most common

application with three front speakers (left, right and center), two rear speakers and once low-frequency speaker (sub-woofer)

• AC3 An alternative to Dolby Labs 5.1 surround format.

Codecs

• Which is best? There really is no one-stop solution.

• All of the major standards do a very good job of compressing most media

• The differences are subtle

There are real advantages to choosing one standard as your default distribution method

• Users will only need to rely on one playback solution

• Hardware will be optimized to that solution (servers and production tools)

• Licensing of technology is easier• Staff is trained with emphasis on only one

platform• Support is easier

There are real disadvantages to choosing one standard as your default distribution method

• Not a large user base for the technology• Format of files is not competitive with

changes in the marketplace• High licensing fees• Flaky software• Challenges with using the files on all types of

hardware (WM on a Mac)• You are tied to the fortunes of the company

that created your codec

Format compatibilities and incompatibilities

• QuickTime Mac and PC are OK, no Linux support • Windows Media PC is great, Mac support is

limited to older versions, no Linux• Real Networks Supports most every operating

system• Flash Supports most every operating system

because it runs in a browser• MPEG will play in many of the proprietary

systems H.264 shows promise as a common standard.

How might you examine a codec to see if it will work for your needs?

You might choose one compression solution for your production and another for delivery to your users.

Property Production Codec Delivery Codec

Compression Ratio Low usually between 1:1 and 4:1

High typically 50:1 or more

Quality Lossless (perfect) or close Good to acceptable, will denigrate to sub-VHS quality

Lossy Zero for some codecs, for others very minimal

Very lossy

Bit rate 25-50 and up to 150 Mbps As low as 300 Kbps for video and 24Kbps for audio

File Size :20 seconds per GB (180 GB per hour)

As low as .4 GB per hour

Editable Yes Not really, although many editors will accept a compressed file as a source. The quality will be dubious at best

Compositable Yes Only if file has alpha channel

Conversion to other formats Easy Additional degradation of quality is likely

General rule is that as compression schemas improve the files get smaller and the images either improve or stay the same.

Let’s look at how we compress a file to get an understanding of all the variables we can manage

Prior to compression you need to get your media in the best shape possible. Often it is easier in production to prep the media than it is to heal bad video during compression.

Video Preparation

• Deinterlacing the footage

• Frame rate agreement (2-3 cadence, 15fps, PAL v. NTSC, HD)

• Noise filters (dubious benefits at best)

• Grain removal

Video Preparation

• Color correction (gamma adjustments)

• Scaling (size your canvas can speed up compression significantly)

• Aspect ratio adjustment including anamorphic adjustments

Compression Example

• What the heck do those letters mean? And why do I care?

• I-frame Intra frame or key frame• P-frame predicted difference frame• B-frame bi-directional difference frame

• Compression Demo