Embed Size (px)
Citation preview
Media Foundation: Supporting Hardware Codecs and CamerasStan PenningtonProgram ManagerMicrosoft Corporation
CL34
Agenda
> Media Foundation (MF) Hardware Codecs, and Asynchronous Media Foundation Transforms (MFTs) in general
> MF video capture support, with focus on UVC 1.1 HD webcams
> But first, let’s all get up to speed on MF…
MFT
MF
MF Background: Media Infrastructures in WindowsPerspective> Team focus on unmanaged APIs
> Performance concerns for managed data flow components> Non-deterministic finalization of GC
> Can expose configuration, control, and status through managed. Legacy and roadmap issues with wrappers
> Four infrastructures> Video for Windows (VfW)> DirectShow (DShow)> Windows Media Format SDK (WM FSDK)> MF
MF Background: Media Infrastructures in WindowsPerspective (cont’d)> One or both of two development models for
ISVs> Applications built on APIs> New data flow components using extensibility points –
often driven by media format or DSP needs
MF Background: Media Infrastructures in WindowsEvolution
1990 1995 2000 2005 2010
VfW
WM FSDK
MF
ACM Codecs
DVD, TV (Filter, DMO, ACM, VCM Codecs)
WM True Streaming (DMO, ACM Codecs)Key Mainstream Scenarios & New Features (MFT Codecs)
DVD TV
Current Usage
DShow
Windows 7
MF Background: Media Infrastructures in WindowsTypical Microsoft usage in Windows 7
> MF> Playback of non-streaming ASF, MP3, MPEG-4,
and selected AVI permutations for digital still cameras
> New metadata handling in Shell Property Handlers
> Transcoding and new media sharing> DShow
> DVD, TV, MPEG-2, MPEG-1, custom I/O, video editing
> FSDK> Best compatibility for ASF true streaming> Historic integration of ASF read, write
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences
> Information provided to set expectations, subject to change in servicing or major releases> Windows Media Player (WMP) and WMP OCX
> Opportunistically use MF, FSDK, or DShow as necessary for best end-user playback experience
> MF (+ DShow source hybrid) for Windows 7 transcoding> Windows Media Center (WMC)
> WMP OCX with custom presenter - for playback other than TV, DVD
> Shell drag-and-drop transcoding for devices (Windows 7 only)> MF (+ DShow source hybrid)
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences (cont’d)
> Change: Protected Media Path (PMP) in WMP and OCX> Windows Vista: PMP used for MF even with
unprotected content, providing seamless cross-fade in mixed DRM/non-DRM playlists
> Windows 7: PMP only used for DRM-protected content in MF
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences: Summary
> Cannot assume WMP or WMC extensibility from media infrastructure documentation> Only WMP SDK plug-ins are documented as
extensibility points, per http://msdn.microsoft.com/en-us/library/dd564682(VS.85).aspx
> WMC extensibility is per http://msdn.microsoft.com/en-us/library/bb895967.aspx
> Ask Microsoft for technical clarifications or documentation enhancements
MF Background: Why does Microsoft use MF?
> Stream objects (IStream) for media input, output
> Most connectivity and data flow divorced from streaming components, moved to higher layer> No push-pull component issues> Better potential for dynamic changes
> Better control of threading> More constrained component roles and
graphs topologies
MF Background: MF Features
> All COM, C++> Windows Vista
> High-level API: Media Session> Formats: ASF w/ WMV and WMA, MP3
> Windows 7> Backward compatibility> New high-level APIs: Source Reader, Sink Writer,
Transcode API (part of “Fast Transcode” feature)> Formats: + AVI w/ DV and MJPEG, + MPEG-4 container w/
MPEG-4 ASP, H.264/AVC, AAC LC, HE v1, HE v2, key ACM wrapping
> Asynchronous MFT and Hardware Codecs> Capture support: Device Proxy
TopoEdit = GraphEdt for MF
demo
Async MFT Model
> Sync MFT refresher> DMO enhanced for MF media types and samples> Simplest use is to call ProcessInput,
ProcessOutput, repeat until output ready status> Need for adding async model
> Inefficient for high level to poll for hardware completion
> Async changes> Locked as sync for backward compatibility> Limited interface use until unlocked> Events fired on input needed, output ready,
drain complete, marker processed
Async MFT Hardware Acceleration
> Candidates> Video and audio codecs, processors
> Why codecs as a new model?> No DXVA support for encode> No efficient hybrid of DXVA decode and non-GPU
encode for transcoding
Async MFT Hardware Acceleration Usage
> Generically, any ISV can write, any ISV can use> Subject to any licensing rules, of course> Otherwise, use local registration
(MFTRegisterLocal)
Async MFT Hardware Acceleration Usage (cont’d)
> Picked up by high-level MF APIs, no need to code directly to Async MFTs, if:> Overall enable/disable registry keys enabled> MF API opt-in when not using WMP or OCX. Set
MF_TOPOLOGY_HARDWARE_MODE attribute on topology
> Codec has Trusted Merit if overlapping in-box functionality, and preferred media subtypes match, or fallback subtypes not satisfied by in-box component
> Can specify directly in own MF topology (MFTEnumEx)
Async MFT Hardware Acceleration Usage (cont’d)
> For Windows Media Experiences and automatic Media Session selection, must constrain preference> Codec Trusted Merit cryptographic
challenge/response> Keys and certificates are managed through
Microsoft Hardware Codecs license program
Hardware Codecs License Program
> Only for video codecs, video processors> Preferred over in-box MFT codecs for
Windows Media Experiences if properly licensed through [email protected]
> Only for developer of actual codec core> Two hardware models supported
> GPU – vendor writes complete MFT> AVStream – vendor uses Microsoft-provided
proxy MFT, or writes complete MFT> Test suite protects preference
> Shares test content with Media Performance Test Kit
Under the Hood
Hardware Codecs License Program (cont’d)
> Two codec types> Transcoding only> General – for transcoding and playback
> Proper MF attributes must be set (see additional resources links)> Directly on vendor MFTs> Through KS property sets for AVStream
> AVStream MFTs can use hardware data flow via Hardware Mediums
Under the Hood
Hardware Codecs Windows Usage
> Windows Media Experiences> WMP and Shell leverage the Transcode API for
transcoding to device> Note: No sink extensibility
> Several events documented for Hardware Codecs in general are optional in WMP usage. 3rd-party applications may be able to make hardware state machines more efficient by using optional events
Under the Hood
Hardware Codecs Merit Rules
> Video media subtypes in scope for Trusted Merit *
Under the Hood
Media subtype Transcoding decoder
Transcoding encoder
Playback decoder
H.264 Yes Yes Yes *MPEG-2 Yes No Yes *MPEG-4 Part 2 Advanced Simple Profile (ASP, compatible with DivX/Xvid/3ivx)
Yes No Yes
MPEG-4 Part 2 Simple Profile (SP)
Yes No Yes
VC-1 and compatible WMV9
Yes Yes No *
WMV not compatible with VC-1
No No No
MJPEG No No No* Test suite requirements are subject to change per the licensing program** Existing technology may already provide hardware acceleration
Hardware Codecs Fast Transcoding
> Focused on conversion for portable devices> Based on MF Media Session, implemented
in WMP and Shell layers> Non-public DShow wrapper Media Source
used for broad source format support, can output compressed or uncompressed
Under the Hood
Hardware Codecs Media Session-Based Transcoding
Byte Stream
Media Session
Source Resolver
Media Source
Scheme Handler
Byte Stream
Byte Stream Handler
MFT Decode
r
Media Sink
Hardware Acceleration
Topology & Loader
File,Network
Custom Input
Webcam
Custom Output
Pipeline Topology
MFT Encode
rProc
Encoding Profile
Management
File,Network
or
or
or
Video Scaling/Conversion, Other Processing
Under the Hood
GPU-Based Hardware Codec Model Shown
Provided by:
Microsoft
Microsoft or ISV
IHV
Hardware Codecs: New Quick Transcoding Sample
demo
MF Capture Overview
> Device Proxy (DevProxy) – internal object for lighter-weight use of Kernel Streaming (KS) than typical DShow
> AVStream driver model supported> USB Video Class (UVC) 1.1 Webcams> Audio capture through WASAPI> Other, but not full tuner capabilities
MF Capture Overview (cont’d)
> Picked up by MF high-level APIs through MF Media Source, no need to code directly to DevProxy> Create Media Source and pass in to Source Reader using
MFCreateSourceReaderFromMediaSource> Hardware codecs supported by setting
MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS attribute
> Focus here on UVC capture> 1.1 includes value-add plug-in model for camera vendors> Diagram:
http://msdn.microsoft.com/en-us/library/ee349147.aspx
MF Capture Use with Source Reader
> Source Reader high-level API benefits> Easy way to obtain uncompressed or
compressed samples from media or device> Has constrained extensibility> Async and sync APIs. Async is best> Good model for feeding a composition engine or
adding to existing media app design
MF Capture Dev Proxy + Source Reader
Source Reader
Media Source
MFT Decoder
*
Hardware Acceleration
Specific Conversions
*
File,Network
Custom Input
Webcam
Specify URL, Byte Stream, or Media Source
Async or Sync Control and Status
Direct access to consume and modify media samples
* Optional
Provided by:
Microsoft
IHV
AVStream Hardware Codec Model Shown
MF Capture: SDK Sample
demo
>>FUTURE
MF Development Tools
> Tracing> Challenges
> Async programming> Worker queues for Media Sources> Constrained ETW events> Some Windows Vista WPP events removed for performance
optimizations. Windows TMFs not provided> Solution
> New MFTrace tool, planned for release in 2010> Detours hooking of public MF interfaces, adding events> Consolidation of existing events
> Usage> Live console output> Redirect to file, process with tool such as TextAnalysisNET> Topology and timestamp graphing cmds
MF Development Tools (cont’d)
> General> TopoEdit – now aware of hardware-accelerated
MFTs
Summary
> MF high-level APIs in Windows 7 provide integration for:> Hardware-accelerated transforms,
including Hardware Codecs> Resultant fast transcoding> Simple AVStream-based audio/video
capture, including HD webcams compliant with UVC 1.1
> Development and tracing tools are being improved
Call to Action
> Evaluate MF for new designs> Consider MF in opportunities for new
functionality or improvements in existing designs
Additional Resources
> MSDN MF documentation in Windows SDK > Top level:
http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx
> Async MFTs: http://msdn.microsoft.com/en-us/library/dd317909(VS.85).aspx
> Writing a custom MFT: http://msdn.microsoft.com/en-us/library/dd940441(VS.85).aspx
> Source Reader: http://msdn.microsoft.com/en-us/library/dd940436(VS.85).aspx
> Video capture: http://msdn.microsoft.com/en-us/library/dd940439(VS.85).aspx
> TopoEdit help: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx
Additional Resources (cont’d)
> MSDN AVStream documentation in Windows DDK> Hardware Codec support in AVStream:
http://msdn.microsoft.com/en-us/library/dd568169.aspx
> MSDN MF Forum: http://social.msdn.microsoft.com/forums/en-US/mediafoundationdevelopment
> Blog: http://blogs.msdn.com/mf
YOUR FEEDBACK IS IMPORTANT TO US!
Please fill out session evaluation
forms online atMicrosoftPDC.com
Learn More On Channel 9
> Expand your PDC experience through Channel 9
> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses
channel9.msdn.com/learnBuilt by Developers for Developers….
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
