Embed Size (px)
DESCRIPTION
Citation preview
Grid technologyfor next-gen media processing
Jens Buysse - Stijn De Smet - Koen Segers-Bruno Volckaert
2
Overview
MediaGrid concept Distributed video transcoding Enabling technologies Setup overview Test results Simulation results Conclusions
MEDIAGRID CONCEPT
3
Originating problems
Tape-based media to file-based media
Multitude of file-based media transfers and processing Storage / retrieval / transfer of media Conforming Transcoding Upscaling Editing
Geographically disperse facilities / resources / media storage
4
Grid technology as solution?
Grid technology
5
a Grid is a distributed processing architecture where heterogeneousresources are shared between different participating organizations, across
an interconnecting network
Resources Storage (media archive, temporary storage, etc.) Computational (rendering farm, work stations, etc.) Specialized (broadcasting, ingesting, etc.)
High speed interconnecting network (1-10 Gbit/s)
MediaGrid
6
MMMhires
MhiresM
hires
MediaGrid
7
MMMhires
MhiresM
hires
EDL
Grid Middleware
MediaGrid: enabling virtual organisations
8
VO 1
VO 2
DISTRIBUTED VIDEOTRANSCODING
9
Grid technology proof-of-concept
Investigated the viability of Grid technology for processing tasksin media production / distribution companies Transcoding of media Upscaling of media
Video transcoding is a resource intense process I/O Processing needs
10
Video transcoding deals with converting a video signal intoanother one with different format, such as different bit rate,
frame rate, frame size, or even compression standard
Need for transcoded / rescaled video
VRT online mediahttp://www.deredactie.be
YouTubehttp://www.youtube.com
11
Distributed video transcoding
How can we accelerate this process?
12
00:00:00 00:51:53
00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:51:5300:00:00 00:13:15
Server 1 Server 2 Server 3 Server 4
ENABLING TECHNOLOGIES
13
Enabling technologies
OS SuSe enterprise
Transcoding software Transcode library
Grid Middleware TORQUE (openPBS) Maui scheduler Grid distributed transcoder: custom Java application
Data retrieval / storage technology GPFS
14
Enabling technologies: TORQUE
TORQUE : open PBS
15
Queue 1
MauiSheduler
Policy
pbs_mom
pbs_mom
TORQUE Server
User
Queue 2
Job
Enabling technologies
Job / batch / workflow submitter Consider job dependencies
16
1
2
Stock 1 3 4
5 6
Stock 27
8
Stock 3
Enabling technologies
Grid distributed transcoding application
17
Enabling technologies
Grid distributed transcoding application
18
SETUP OVERVIEW
19
Setup overview
… TORQUE
… with GPFS cluster as media storage
… Java distributed transcoding front-end
… on each computational resource Transcode libraries
… the will to transcode in a distributed fashion
20
00:13:15 00:26:3000:26:30 00:39:4500:39:45
00:51:5300:00:00 00:13:15
First distributed transcoding workflow
2121
User
TORQUE1. Split phase2. Transcoding3. Merge phase
Node 2
Node 4
Node 3
GPFS
Node 1
00:00:00 00:51:53
00:13:15 00:26:30 00:00:00 00:13:15
00:39:4500:51:53
00:26:30 00:39:45
00:00:00 00:51:53
00:00:00 00:51:53
Current distributed transcoding workflow
22
User
TORQUE1. Preprocess phase2. Demux phase3. Transcoding4. Merge / multiplex
00:13:15 00:26:30
00:26:30 00:39:45
00:39:45 00:51:53
Nav.log
Audio.mp3
00:00:00 00:13:15
Node 2
Node 4
Node 3
GPFS
Node 1
00:00:00 00:51:53
23
Future distributed transcoding workflow
23
User
TORQUE
1. Prefetch2. Preprocess3. Demux4. Transcode5. Merge /
multiplex
Node 2
GPFS Remote
Node 1
GPFS local
WAN
00:00:00 00:51:53
Node 3
00:00:00 00:51:53
Discussion
Old version Video files were physically split Split / merge step could introduce artifacts
Current version File is inspected and navigation file created allowing for easy frame-
addressing Audio ripped and transcoded in separate step No artifacts Less media-transfers than in previous versions
Future version Pre-fetching / replication of media to remote sites
24
TEST RESULTS
25
Test topology
26GPFS node
Traffic Shaping
Torque Server
Test results
Input media Vob file MPEG-2 video encoding AC3 audio encoding Size: 1,64 GB
Output media Avi file Xvid video encoding MP3 audio encoding Size: 700 MB
Currently no HD video input modules! Not the most optimized video transcoders
Focus on measuring benefits of distributing
27
Results – Grid overhead
Grid Overhead
28
Results – Preprocess phase
Preprocess
29
Results – Audio ripping phase
Rip audio
30
Results – Merging phase
Merging phase
31
Results – 1Gbit/s WAN
32
Results – Parameterised WAN interconnection
33
Video (up)scaling
34
Video scaling is converting video signals from one size or resolution to another: usually"upscaling" or "upconverting" a video signal from a low resolution (e.g. standard definition)
to one of higher resolution (e.g. high definition television).
00:00:00 00:51:53
00:00:00 00:51:53
720X576 984x752
Video (up)scaling results – 52Mbit/s WAN
35
Video (up)scaling results – 52Mbit/s WAN
36
SIMULATION RESULTS
37
Simulation results
We introduced a WAN connection to a remotecomputational resource provider
38
User
Node 4GPFS
Node 1
Node 3
Node 2
TORQUE
• 1 Gbit/s• 100 Mbit/s• 52 Mbit/s• 35 Mbit/s
Simulation results – total job turnaround time
39
Comparison with measured results
40
Comparison with measured results
41
Simulation results
Simulations provide very accurate total job turnaround times Real-life transcoding behaves erroneously when
interconnecting GPFS with computational resource provider bymeans of WAN link lower than 35Mbit/s
Simulation results show what would happen to job turnaroundtime for lower WAN interconnections
42
Click Router
Control Traffic
Data
Control Traffic
Data
GPFS
Simulation results – low-speed WANinterconnection
43
Simulation results – 10 chunks
44
CONCLUSIONS
45
Conclusions
Grid technology is a viable technology for dealing with mediaproduction / distribution tasks Inherent support for parallelism can seriously decrease the total
processing time Need for adaptation of media tasks Grid overhead is no issue
Outsourcing task processing to remote resource providers Viable when interconnection is sufficient Technical limitations (e.g. GPFS time-outs)
MediaGrid simulator can provide accurate performancepredictions
46
Questions ?Feel free to e-mail: [email protected]
