Michał Kwiatek IT-IS Working meeting 10 October 2006 1

Media Archive

File storage and conversion process

Michał Kwiatek IT-IS Working meeting 10 October 2006 2

I. Project foundations

Michał Kwiatek IT-IS Working meeting 10 October 2006 3

Project foundations• Media master files are stored on DFS in

\\cern.ch\dfs\Services\MediaArchive• Master formats are:

– DV/AVI or MPEG2 for video– PCM WAVE for audio– JPEG for photo

• Metadata for the masters is taken from CDS and for video includes: width, height, fps and frame ratio

• Files dropped in the masters directory are automatically encoded according to encoding profiles. Slave formats are:– .wmv + .jpg thumbnails for video– .mp3, .ogg, .wma for audio– .jpg for photo

Michał Kwiatek IT-IS Working meeting 10 October 2006 4

II. What we have

Michał Kwiatek IT-IS Working meeting 10 October 2006 5

Automatic conversion software• The software that discovers and propagates

to slaves changes in masters and profiles (new, updated or deleted) is ready

• Conversion to all required slave formats has been implemented

• Depending on configuration, process starts automatically when a new file appears or after a web-service call. This is an offline process.

• Process may go on without metadata in exceptional cases

• After generation of each slave, status xml is generated.

Michał Kwiatek IT-IS Working meeting 10 October 2006 6

Web server / streaming service

• All slave files are available trough the web (currently test configuration)– HTTP:

http://webdfs02.cern.ch/mediaarchive/– Streaming:

mms://webdfs02.cern.ch/mediaarchive/Video/Public/Movies/1950/CERN-MOVIE-1950-001/CERN-MOVIE-1950-001-043-Multirate120-1000kbps-audio20-160k-384x288x25fps-.wmv

Michał Kwiatek IT-IS Working meeting 10 October 2006 7

File storage

• We have:– New DFS file server with two partitions:

• 1,8 TB, currently stores photo and audio data• 2,7 TB, currently stores video data

– Master data organised by type, then by year, which allows flexible space management

– Group-driven access rights to be mamaged by CDS responsible via web interface at https://websvc02.cern.ch/WinServices/Services/GroupManager/GroupManager.aspx

– 3 DFS shares defined on the top level (audio, photo, video).

Michał Kwiatek IT-IS Working meeting 10 October 2006 8

File storage (cont’d)

2006, 2005, 2004, ...

2006, 2005, 2004, ...

2006, 2005, 2004, ...

2006, 2005, 2004, ...

2006, 2005, 2004, ...

Michał Kwiatek IT-IS Working meeting 10 October 2006 9

III. What we need

Michał Kwiatek IT-IS Working meeting 10 October 2006 10

What we need to perform final conversion of the entire

archive• Metadata • Digitalised video archives • Video masters need to be organised

• Profiles should be reviewed (who?)

0.5 TB

> 2 TB

Michał Kwiatek IT-IS Working meeting 10 October 2006 11

IV. Further discussions

Michał Kwiatek IT-IS Working meeting 10 October 2006 12

Points to discuss• High definition videos:

– Out of project scope– No real standard– Huge files

– Eat up all available diskspace

Michał Kwiatek IT-IS Working meeting 10 October 2006 13

Points to discuss (cont’d)

• Retransmission• Webcast

– Thomas’ presentation– Multicast

Michał Kwiatek IT-IS Working meeting 10 October 2006 14

Michał Kwiatek IT-IS Working meeting 10 October 2006 15

Space requirements

  2006 2007photo 600 200

video 34004000+3000

*)audio 300 ?

*) 4TB for the archive + 1TB/year for conference recs + 2TB/year for HD recs