Upload
sharon-hensley
View
218
Download
2
Tags:
Embed Size (px)
Citation preview
MP design and implementation
CS414: Multimedia SystemInstructor: Klara Nahrstedt
April 13, 2012
Covered Aspects of Multimedia in CS414 MP
Image/VideoCapture
MediaServerStorage
Transmission
CompressionProcessing
Audio/VideoPresentationPlaybackAudio/Video
Perception/ Playback
Audio InformationRepresentation
Transmission
AudioCapture
A/V Playback
Image/Video InformationRepresentation
MP1: AV Capture and Playback
Image/VideoCapture
Compression/ Processing
Audio InformationRepresentation
AudioCapture
Audio/Video Perception/ Playback
MP1: Video Capture and Store in a File
Src
Sink Src Sink Src
Video Source Video Filter Media Encoder
Sink Src
Video Muxer
Sink
Video Sink
final Element videosrc = ElementFactory.make("v4l2src", "source");
final Element videofilter = ElementFactory.make("capsfilter", "flt");videofilter.setCaps(Caps.fromString("video/x-raw-yuv, width=640, height=480, framerate=10/1")); final Element formatConverter = ElementFactory.make("ffmpegcolorspace", "formatConverter"); final Element encoder = ElementFactory.make("ffenc_mpeg4", "encoder");final Element muxer = ElementFactory.make("fmux_mpeg", "muxer");final FileSink filesink = (FileSink) ElementFactory.make("filesink", "filesink");filesink.setLocation("capture01.avi");
Snippet
pipe.addMany(videosrc,formatConverter, videofilter, encoder, filesink);Element.linkMany(videosrc, formatConverter, videofilter, encoder, filesink);pipe.setState(State.PLAYING);Gst.main();pipe.setState(State.NULL);
START
Video Capture and Store in a Queue
Video Source Video Filter
final Element videosrc = ElementFactory.make("v4l2src", "source");
final Element videofilter = ElementFactory.make("capsfilter", "flt");videofilter.setCaps(Caps.fromString(Caps.fromString(String.format("video/x-raw-yuv, width=%s, height=%s" + ", bpp=24, depth=16, framerate=%s/1",WIDTH+"", HEIGTH+"", CAPTURERATE+""))); videorate = ElementFactory.make("videorate", "videorate");ratefilter = ElementFactory.make("capsfilter", “rateFilter");ratefilter.setCaps(Caps.fromString(String.format("video/x-raw-yuv, framerate=%s/1”,FRAMERATE+""))); final Element encoder = ElementFactory.make(” theoraenc", ”H264");final Element muxer = ElementFactory.make(”oggmux", "muxer");final AppSink appsink = (FileSink) ElementFactory.make(”appsink", ”appsink");
Snippet
Src
Sink Src
Media Encoder Video Muxer
Sink Src Sink Src Sink
App SinkVideo Rate Rate Filter
Sink Src Sink Src
Gstreamer-java AppSink
Sink
App Sink Element
A callback function is used to get the frames from the AppSink Element
appsink.set("emit-signals", true);appsink.setSync(false);appsink.connec(new AppSink.NEW_BUFFER(){
public void newBuffer(AppSink as){Buffer buffer = as.getLastBuffer();ByteBuffer buf = buffer.getByteBuffer();byte[] b = new byte[buf.remaining()];buf.get(b);int timestamp =
(int)buffer.getTimestamp().toMillis();ApplicationPacket packet = new
ApplicationPacket(b, timestamp)Queue.enQueue(packet);
}});
QueuePacket
ArrayBlockingQueue
MP1: Video Playback from a File
Pipeline pipe = new Pipeline(”Decode pipeline");Element videosrc = ElementFactory.make("filesrc", ”filesrc");src.set("location", "Avatar.avi");DecodeBin2 decodeBin = new DecodeBin2("Decode Bin");Element decodeQueue = ElementFactory.make("queue", "Decode Queue");final VideoComponent videoComponent = new VideoComponent();
pipe.addMany(videosrc, decodeQueue, decodeBin);Element.linkMany(videosrc, decodeQueue, decodeBin);
decodeBin.connect(new DecodeBin2.NEW_DECODE_PAD()){public void newDecodePad (DecodeBin2 elem, Pad pad,
boolean last){Caps cap = pad.getCaps();Structure struct = caps.getStructure(0);if(struct.getName().startsWith(“video/”){
pad.link(videoComponent.getElement.getStaticPad(“Sink”));} } }
Src
Sink Src
Media Source Media Decoder
Sink Src
Media DeMuxer
Sink
Media Sink
Sink Src
DecodeBin
Video Playback from a Queue
Src
Sink Src Sink Src
App Source Video Filter Media Encoder
Sink Src
Video Muxer
Sink
Video Sink
final AppSrc appsrc = (AppSrc) ElementFactory.make("appsrc", ”appsrc");appsrc.setCaps(Caps.fromString(Caps.fromString(String.format("video/x-raw-yuv, width=%s, height=%s" + ", bpp=24, depth=16, framerate=%s/1",WIDTH+"", HEIGTH+"", CAPTURERATE+"")));
final Element decodeBin = ElementFactory.make("decodebin2","decode");final VideoComponent videoComponent = new VideoComponent();
pipe.addMany(appsrc, decodeBin);Element.linkMany(appsrc, decodeBin);
decodeBin.connect(new DecodeBin2.NEW_DECODE_PAD()){public void newDecodePad (DecodeBin2 elem, Pad pad, boolean last){
Caps cap = pad.getCaps();Structure struct = caps.getStructure(0);if(struct.getName().startsWith(“video/”){
pad.link(videoComponent.getElement.getStaticPad(“Sink”));}
}}
Snippet
Sink Src
Decode Bin
Gstreamer-java AppSrc
Source
App Source Element
A callback function is used to put the frames into the AppSrc Element
appsrc.set("emit-signals", true);appsrc.setSync(false);appsrc.connect(new AppSrc.NEED_DATA() { public void needData(AppSrc as, int size) {
ApplicationPacket packet = Queue.dequeue();
while (packet == null) {System.out.println("Null
data. trying again");packet = Queue.dequeue();
}byte[] data = packet.getData();Buffer buffer = new
Buffer(data.length);buffer.getByteBuffer().put(data);as.pushBuffer(buffer);
}{);
Queue ArrayBlockingQueuePacket
MP2 Design Space
Image/VideoCapture
MediaServerStorage
Transmission
CompressionProcessing
Audio/VideoPresentationPlaybackAudio/Video
Perception/ Playback
Audio InformationRepresentation
Transmission
AudioCapture
A/V Playback
Image/Video InformationRepresentation
MP2: Audio Video Streaming
Server Client
• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz
• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and
Passive Mode
Control Channel
Data Channel
• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480
• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240
Video Data Flow Architecture: Server
RTP
Pack
et
Rate Control
Network
Video Source Video Filter Media Encoder Video Muxer App Sink
Payload
Media Type Timestamp RTP headerID
UDP Packet Payload
Segment
UDP Communication: Server
Payload
Media Type Timestamp
RTP Packet
ID
UDP Packet Payloads
PID SEQ SIZE PAYLOAD SIZE
UDP Packet Payload
System.arraycopy(pid_byte, 0, senddata, 0, pid_byte.length);System.arraycopy(seq_byte, 0, senddata, 4, seq_byte.length);System.arraycopy(size_byte, 0, senddata, 8, size_byte.length);
While(true){int remaining = (databyte.length - from);if(remaining < 0 ) break;else if(remaining > UDP_PAYLOAD_SIZE){
byte[] payload_size_byte = toBytes(UDP_PAYLOAD_SIZE); System.arraycopy(payload_size_byte, 0, senddata, 12, payload_size_byte.length); System.arraycopy(databyte, from, senddata, 16, UDP_PAYLOAD_SIZE);
}else{ byte[] payload_size_byte = toBytes(remaining); System.arraycopy(payload_size_byte, 0, senddata, 12, payload_size_byte.length); System.arraycopy(databyte, from, senddata, 16, remaining); }}
sendData(7777, ”IP_Address", senddata); seq++;from += UDP_PACKET_SIZE;
DatagramSocketDatagramPacket
UDP Packet Header
Video Data Flow Architecture: Client
Network
RTP
Pack
et
App SourceDecode Bin2Video Sink
Synchronization
Payload
Media Type Timestamp RTP headerID
UDP Packet Payload
Merge
UDP Communication: Client
Payload
Media Type Timestamp
RTP Packet
ID
PID SEQ SIZE PAYLOAD SIZE
UDP Packet Payload
System.arraycopy(recv_data, 0, pid_byte, 0, 4);System.arraycopy(recv_data, 4, seq_byte, 0, 4); System.arraycopy(recv_data, 8, size_byte, 0, 4);System.arraycopy(recv_data, 12, payload_size_byte, 0, 4);
Hash
Table
index w
ith
PID
SIZE
if(!received_frame.containsKey(pid)){ ApplicationPacket app_frame = new ApplicationPacket(); app_frame.frame = new byte[size]; app_frame.array_size = size; System.arraycopy(recv_data, 16, app_frame.frame, seq*UDP_PACKET_SIZE, payload_size); received_frame.put(pid, app_frame); app_frame.array_size -= payload_size;}else{ ApplicationFrame app_frame = (ApplicationFrame) received_frame.get(pid); System.arraycopy(recv_data, 16, app_frame.frame, seq*UDP_PACKET_SIZE, payload_size); app_frame.array_size -= payload_size; received_frame.put(pid, app_frame);}if(app_frame.array_size == 0)finish_frame(pid);
received_frame
Video Data Flow Architecture
RTP
Pack
et
Rate Control
Network
Video Source Video Filter Media Encoder Video Muxer App Sink
RTP
Pack
et
App SourceDecode Bin2Video Sink
Synchronization
Payload
Media Type Timestamp RTP headerID
UDP Packet Payload
Segment
Merge
Rate Control: QoS Enforcement
• You can build leaky bucket• You can implement token bucket• You can use gstreamer rate control• You can implement your own
method
Sleep (100) => 10 fps
You can simply read using a loopWhile(true){ SendPacket(); Thread.sleep(100);}
Resource Admission: Client
• Client– Available Application Bandwidth ABN
– Application Frame Size, MN = ?
– Application Frame Rate, RN = ?
– Audio Bandwidth = 8000 * 16– Request Bandwidth
• Active Mode: BN = (MN * RN) + Audio Bandwidth
• Passive Mode: BN = (MN * RN)
resource.txt
Optimistic Allocation
8000Hz Audio Signal
How to define RN for active mode?
Show your computation on the Screen
fps: 10
Resource Admission: Client
• Client– Available Application Bandwidth ABN
– Application Frame Size, MN = ?
– Application Frame Rate, RN = ?
– Request Bandwidth• Active Mode: BN = (MN * RN) + Audio
Bandwidth
resource.txt
Optimistic Allocation
Show your computation on the Screen
ABN = (MN*RN) + Audio Bandwidthif RN > 25, RN = 25if RN < 15, REJECTelse RN
Resource Admission: Client
• Server– Available Application Bandwidth ABN
– Used Bandwidth for Server 1 B1
– Available Bandwidth ABN = ABN –B1
– Admission is successful for B2 if B2 <= ABN
resource.txt
Server 1 60 Kbps
Server 2 35 KBps
Resource Table
Using RtpBin @ ServerVideo Source Video Filter
Tee Video Scale Scale Filter
Video Rate Rate Filter Video Encoder RTP Payload
RTP Bin
UDP Sink
Queue
UDP SinkUDP Source
RTCP
RTCP
RtpBin combines • RtpSession• RtpSsrcDemux• RtpPtDesmux• RtpJitterBuffer
Using RtpBin @ Client
Video SinkVideo DecoderRTP DePayload
RTP Bin
UDP Source
UDP SourceUDP Sink
RTCP
RTCP
rtpBin.connect(new Element.PAD_ADDED() {public void padAdded(Element element, Pad pad) {
if (pad.getCaps().toString().contains("video")) {
//LINK TO VIDEO SINK
} else if (pad.getCaps().toString().contains("audio")) {
// LINK TO AUDIO SINK }
}});
Internals of RtpBin
http://gstreamer.freedesktop.org/documentation/rtp.html
MP3: Multi-view Surveillance System
Window for server 1
Window for server 2
• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz
• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and
Passive Mode
• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480
• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240
Additional Features for MP3?
• It is highly recommended that you build an additional feature in MP3 to SURPRISE US!
• Bonus Point: 20• Bonus point will be awarded based on the
amount of surprise!