Web Page Rendering and Accelerated Compositing

Web Page Rendering and Accelerated Compositing

[email protected]@gmail.comwww.twitter.com/roger2yi+易旭昕 in Google+

mailto:[email protected]

mailto:[email protected]

http://www.twitter.com/roger2yi

2012-4-24 Roger, UC

Page Rendering Basic

2012-4-24 Roger, UC

RenderObjects

• Each node in the DOM tree that produces visual output has a corresponding RenderObject.

• RenderObject's are stored in a parallel tree structure, called the Render Tree.

2012-4-24 Roger, UC

Cont.

• A RenderObject knows how to present (paint) the contents of the Node on a display surface.

• It does so by issuing the necessary draw calls to the GraphicsContext associated with the page renderer.

• The GraphicsContext is ultimately responsible for writing the pixels on the bitmap that gets displayed to the screen.

2012-4-24 Roger, UC

Render Tree

2012-4-24 Roger, UC

Cont.

2012-4-24 Roger, UC

Cont.

2012-4-24 Roger, UC

RenderLayers

• Each RenderObject is associated with a RenderLayer either directly or indirectly via an ancestor RenderObject.

• RenderObjects that share the same coordinate space (e.g. are affected by the same CSS transform) typically belong to the same RenderLayer.

2012-4-24 Roger, UC

Cont.

• RenderLayers exist so that the elements of the page are composited in the correct order to properly display overlapping content, semi-transparent elements, etc.

• There's a number of conditions that will trigger the creation of a new RenderLayer for a particular RenderObject, as defined inRenderBoxModelObject::requiresLayer() and overwritten for some derived classes.

2012-4-24 Roger, UC

Cont.

• In general a RenderObject warrants the creation of a RenderLayer if– It's the root object for the page

– It has explicit CSS position properties (relative, absolute or a transform)

– It is transparent

– Has overflow, an alpha mask or reflection

– Corresponds to <canvas> element that has a 3D (WebGL) context

– Corresponds to a <video> element

2012-4-24 Roger, UC

Cont.

2012-4-24 Roger, UC

Render

• The RenderLayer hierarchy is traversed recursively starting from the root and the bulk of the work is done in RenderLayer::paintLayer() which performs the following basic steps (the list of steps is simplified here for clarity):

2012-4-24 Roger, UC

Cont.

– Determines whether the layer intersects the damage rect for an early out.

– Recursively paints the layers below this one by calling paintLayer() for the layers in the negZOrderList.

2012-4-24 Roger, UC

Cont.

– Asks RenderObjects associated with this RenderLayer to paint themselves. This is done by recursing down the RenderTree starting with the RenderObject which created the layer. Traversal stops whenever a RenderObject associated with a different RenderLayer is found.

– Recursively paints the layers above this one by calling paintLayer() for the layers in the posZOrderList.

2012-4-24 Roger, UC

Cont.

2012-4-24 Roger, UC

Render by WebView• 一个页面被封装在一个UI组件中呈现，这个UI组件一般被称为

WebView

• 最简单的页面渲染方式就是在WebView绘制自身的时候走一遍页面渲染流程

• 不过一般来说，页面内容更新引起的WebView重绘通常会分为相对独立的两个步骤– 页面按实际状况调整Layers结构，然后渲染（或者记录）更新后的内容– WebView在绘制自身时把更新后的页面绘制到窗口中

• 从而演变出各种不同的渲染架构

– 页面内部采用位图或者矢量图（Backing Store）记录页面内容

– 多线程，多进程

2012-4-24 Roger, UC

Accelerated Compositing

2012-4-24 Roger, UC

Accelerated Compositing

• The Accelerated Compositing concept comes to optimize for cases where an element would be painted to the screen multiple times without its content changing. For example, a menu sliding into the screen, or a static toolbar on top of a video.

• It does so by creating a scene graph, a tree of objects (graphics layers), which have properties attached to them - transformation matrix, opacity, position, effects etc., and also a notification when the layer's content needs to be re-rendered.

2012-4-24 Roger, UC

Cont.

• When accelerated compositing is enabled, some (but not all) of the RenderLayer's get their own backing surface (compositing layer) into which they paint instead of drawing directly into the common bitmap for the page.

• A subsequent compositing pass composites all the backing surfaces onto the destination bitmap.

2012-4-24 Roger, UC

Cont.

• The compositor is responsible for applying the necessary transformations (as specified by the layer's CSS transform properties) to each layer before compositing it.

• Since painting of the layers is decoupled from compositing, invalidating one of these layers only results in repainting the contents of that layer alone and recompositing.

2012-4-24 Roger, UC

Cont.

• While in theory every single RenderLayer could paint itself into a separate backing surface to avoid unnecessary repaints, in practice this could be quite wasteful in terms of memory (vram especially).

• In the current WebKit implementation, one of the following conditions must to be met for a RenderLayer to get its own compositing layer (see RenderLayerCompositor::requiresCompositingLayer()):

2012-4-24 Roger, UC

Cont.– Layer has 3D or perspective transform CSS properties

– Layer is used by <video> element using accelerated video decoding

– Layer is used by a <canvas> element with a 3D context

– Layer uses a CSS animation for its opacity or uses an animated webkit transform

– Layer has a descendant that has a compositing layer

– Layer has a sibling with a lower z-index which has a compositing layer (in other words the layer is rendered on top of a composited layer)

2012-4-24 Roger, UC

2012-4-24 Roger, UC

2012-4-24 Roger, UC

2012-4-24 Roger, UC

2012-4-24 Roger, UC

2012-4-24 Roger, UC

Cont.

• 页面渲染流程可以被分为– Layers Sync– Layers Compositing

• WebCore主要负责Sync，Sync包括调整Layers的结构和渲染（或者记录）Layer本身的内容（实际的渲染也有可能延迟到第一次Compositing的时候）

• Compositing通常会交由WebKit Port自己来负责，除了Root Layer外，其它带Backing的RenderLayers不再在正常RenderLayers渲染流程中进行绘制

2012-4-24 Roger, UC

Cont.

2012-4-24 Roger, UC

Cont.

• Compositing通常会发生在WebView绘制页面到窗口的时候

• 由页面内容更新引起的，WebView重新绘制通常可以分为相对独立的三个步骤

– Layers Sync

– Layers Draw to Backing Store

– Compositing Backing Store to Window by WebView

2012-4-24 Roger, UC

Common Structure

RenderLayerCompositor

GraphicsLayer

RenderLayer

GraphicsLayerClient

RenderLayerBacking

RenderBlockRenderView

RenderObjectRenderBoxMobelObject

+m_backing+m_compositor

+m_layer

+m_graphicsLayer

+m_client

rootPlatformLayer()

2012-4-24 Roger, UC

Cont.

• GraphicsLayer是一个抽象类，需要由平台提供自己的实现类（GraphicsLayerCA，GraphicsLayerQt，GraphicsLayerAndroid，etc...）

• 平台提供的GraphicsLayer实现类会包含一个PlatformLayer（或者自身就是PlatformLayer，PlatformLayer是一个平台相关类型的typedef）

• GraphicsLayers会构成一个跟RenderLayers结构并行的树结构，一般而言PlatformLayers也会构成一个跟GraphicsLayers并行的树结构，真正进行Compositing的一般是PlatformLayers

2012-4-24 Roger, UC

Cont.

• GraphicsLayerClient是一个提供给GraphicsLayer的回调接口，GraphicsLayer通过这个回调接口– notifyAnimationStarted - 通知页面该Layer启动一个

CSS动画，用于实现硬件加速的CSS动画

– notifySyncRequired - 通知页面该Layer的样式属性被改变，需要重新渲染和混合

– paintContents - 把对应的RenderLayer上的内容绘制到自身内部的Backing Surface上

2012-4-24 Roger, UC

Cont.

• GraphicsLayer的各个平台实现中，内部用来作为Layer Backing Store的对象可以分为三种不同的类型– Vector : Android-SW/HW（SkPicture）

– Bitmap : Qt GraphicsView（QPixmap），Qt TextureMapper-SW（QPixmap）

– Texture : Qt TextureMapper-HW（FBO），Mac/iOS（CALayer），Android-HW

2012-4-24 Roger, UC

Qt Implementation

2012-4-24 Roger, UC

Qt GraphicsView 框架

• QGraphicsView是一个UI组件（QWidget），为场景绘制提供了一个窗口，定义了ViewPort

• QGraphicsScene定义了一个绘图场景，负责管理场景里面的QGraphicsItem

• QGraphicsItem定义了绘图场景里面的一个基本绘图对象，QGraphicsObject是QGraphicsItem的一个重要派生类，同时派生自QObject，支持Qt的元对象模型，支持信号与槽，QGraphicsWidget派生自QGraphicsObject，提供了跟QWidget相似的API

2012-4-24 Roger, UC

Qt 基于 GraphicsView 的实现

RenderObjectRenderBoxModelObject

RenderLayer


GraphicsLayer


GraphicsLayerClient

RenderLayerBacking GraphicsLayerQt

QGraphicsItem

«PlatformLayer»QGraphicsObject

GraphicsLayerQtImpl QGraphicsWidget

QGraphicsWebView

+m_composi tor

+m_owningLayer

+m_backing

+m_layer

+m_impl+m_graphicsLayer

2012-4-24 Roger, UC

Cont.

• PlatformaLayer类型定义为QGraphicsObject，实际上是QGraphicsObject的一个派生类GraphicsLayerQtImpl

• GraphicsLayer的实现类GraphicsLayerQt包含一个GraphicsLayerQtImpl（PlatformLayer）

• QGraphicsWebView派生自QGraphicsWidget，定义了一个WebView的封装类，同时作为GraphicsLayerQtImpls构成的树结构的根节点

• QGraphicsWebView自身的绘制（QGraphicsWebView::paint）开始了正常的RenderLayers绘制流程，但是在AC开关打开的情况下，只绘制了Root Layer，其它带Backing的RenderLayers都不走正常的绘制流程

2012-4-24 Roger, UC

Cont.

• 带Backing的RenderLayers，会根据实际状况把它们的内容预先绘制在GraphicsLayerQtImpl内部的一个缓存位图中

• GraphicsLayerQtImpl被GraphicsView框架绘制时（被绘制在QGraphicsWebView之上），如果已经存在内部缓存位图，则直接绘制该内部缓存位图

• 是否使用硬件加速取决于QGraphicsWebView是否处于一个OpenGL上下文，全局的QPainter是否走OpenGL的绘制路径（Qt在程序初始化时可以选择不同的渲染引擎），这个实现本身并没有对硬件加速的状况进行专门优化

2012-4-24 Roger, UC

Cont.

• Pros– 自身的结构比较简单

• Cons– 效率不高，不能充分利用硬件加速能力

– 使用Qt专有的GraphicsView API，不是一种通用的机制

– 依赖于GraphicsView复杂的绘图框架，有些效果无法正确实现，未来进一步扩充支持更多CSS特效比较困难

2012-4-24 Roger, UC

Qt 基于 TextureMapper 的实现

RenderObjectRenderBoxModelObject

RenderLayer


GraphicsLayer


GraphicsLayerClient

RenderLayerBacking GraphicsLayerTextureMapper

TextureMapperNode

TextureMapperSurfaceManagerTextureMapper BitmapTexture

BitmapTextureQt BitmapTextureGL

TextureMapperQt TextureMapperGL

+m_node

+m_graphicsLayer

+m_surfaceManager

+m_layer

+m_owningLayer

+m_backing+m_compositor

+m_textureMapper +m_tiles *

+surfaces

*

2012-4-24 Roger, UC

Cont.

• BitmapTexture可以作为绘制的目标对象，也可以被绘制到另外一个BitmapTexuture上面，软件的实现是BitmapTextureQt，硬件的实现是BitmapTextureGL

• TextureMapper可以绑定一个BitmapTexture作为绘制目标，从而将另外一个BitmapTexture绘制到绑定的BitmapTexture上面，也可以直接设定一个GraphicsContext作为绘制目标，软件的实现是TextureMapperQt，硬件的实现是TextureMapperGL

2012-4-24 Roger, UC

Cont.

• GraphicsLayerTextureMapper是GraphicsLayer的实现类，它包含一个TextureMapperNode作为内部实现（类似GraphicsLayerQtImpl）

• TextureMapperNode包含一个或者多个BitmapTexture作为内部缓冲

• TextureMapperNodes构成一个跟GraphicsLayerTextureMapper/RenderLayer并行的树结构

• TextureMapperSurfaceManager是一个辅助类，主要用于分配和回收临时使用的BitmapTexture

2012-4-24 Roger, UC

Cont.

• TextureMapperNode在syncCompositingState的时候把对应的RenderLayer上的内容先渲染到内部的BitmapTexture上（TextureMapperNode::renderContent）

• QWebView在重绘时，先走正常的RenderLayers渲染流程，渲染Root Layer，然后调用TextureMapperNodes的根节点的paint方法进行Composting

• 在Composting的时候，TextureMapperNode将内部的BitmapTexture渲染到一个外部传进来的TextureMapper（该TextureMapper由QWebView提供，它包含一个绘制到QWebView上的GraphicsContext）

2012-4-24 Roger, UC

Cont.

• Pros– 跨平台的架构，不依赖于平台私有的API，容易移植到其它平台（GTK+，Android）

– 同时支持硬件加速和非硬件加速两种方式，硬件加速部分使用OpenGL/ES API，平台兼容性好，比较充分利用GPU

• Cons– 自身的结构比较复杂（相对于基于

GraphicsView的实现）

2012-4-24 Roger, UC

Android Implementation

2012-4-24 Roger, UC

Android 4.0 Stock Browser

• Android 4.0 内置浏览器支持Accelerated Compositing和Hardware Acceleration

• 它同时支持硬件加速和软件两种不同的渲染方式，在运行时由WebView是否处于一个支持硬件加速渲染的窗口来决定，也就是传给WebView的Canvas是否是硬件加速的Canvas

• 硬件和软件渲染路径都支持Accelerated Compositing，Layer Backing Store使用Vector和Texture两种形式

2012-4-24 Roger, UC

Cont.

• 软件渲染– Page Backing Store只存在Vector（PictureSet）形式

– Layer Backing Store只存在Vector（SkPicture）形式

– Sync的时候，Root Layer的内容先记录到Page的Vector Backing Store，其它带Backing的Layer，内容先记录到自己内部的Vector Backing Store（WebCore线程）

– WebView被重绘时，进行Compositing，先绘制Page Vector Backing Store，然后依次绘制Layers Vector Backing Store（UI线程）

2012-4-24 Roger, UC

Cont.

• 硬件加速渲染– Page Backing Store同时存在Vector和Texture 两种形式

– Layer Backing Store同时存在Vector和Texture 两种形式

– Sync的时候，Root Layer的内容先记录到Page的Vector Backing Store，其它带Backing的Layer，内容先记录到自己内部的Vector Backing Store（WebCore线程）

2012-4-24 Roger, UC

Cont.

• WebView被重绘时，进行Compositing（UI线程）

– 先绘制对应当前Page ViewPort的Root Layer的Tiled Texture

– 再依次绘制其它带Backing的Layers的Tiled Texture

• 如果Root Layer的Tiled Texture和其它带Backing的Layers的Tiled Texture还没有生成，它们需要在TextureGenerator线程生成

• 目前Tiled Texture生成的Code Path同时包括使用CPU和GPU两种方式，但是在Android4.0浏览器默认还是使用CPU的方式，估计GPU的方式还没有优化好（速度更慢并且存在渲染错误）

2012-4-24 Roger, UC

Cont.

• 使用CPU的方式生成Tile Texture包括以下步骤

– 使用一个全局的SkBitmap，每次先清空（大小为一个Tile的大小）

– 将SkPicture绘制在全局的SkBitmap上

– 将全局的SkBitmap直接通过memcpy拷贝到为这个Tile分配的Graphics Buffer上

2012-4-24 Roger, UC

Cont.

• WebCore线程所完成的工作在硬件加速和软件两种不同方式下基本一样，都是负责记录页面内容到Vector里面

• 软件渲染的方式，没有TextureGenerator线程，也没有Tiled Backing Store，也没有Layer Texture

• 硬件加速的情况下，所有真正的渲染都发生在TextureGenerator线程，包括Page的Vector Backing Store生成Texure，Layer生成Texture

• Compositing发生在UI线程，软件渲染的方式下，只是依次绘制Root Layer的Vector和其它Layer的Vector，硬件加速的方式下，绘制它们预先生成的Texture

2012-4-24 Roger, UC

GraphicsLayerAndroidGLWebView State BaseLayerAndroid LayerAndroid

LayerSkPicturePictureSet

BaseTile

TiledPage

TiledTexture

TilesManagerBaseTileTexture

DoubleBufferedTexture

TilePainter

BaseRenderer

GaneshRendererRasterRenderer

GaneshContext


ScrollableLayerAndroid

GraphicsLayer

RenderLayerGraphicsLayerClient

RenderLayerBacking

RenderView RenderBlock RenderBox

RenderBoxMobelObject

RenderObject

ImageTexture PaintedSurface

SurfacePainter

DualTiledTexture

SharedTexture

+m_recordingPicture

-m_tiledPageA

-m_baseTiles

-m_renderer

-m_backTexture

-m_frontTexture

-m_page

-m_painter

+mPictures

-m_currentBaseLayer

-m_paintingBaseLayer

-m_tiledPageB

-m_glWebViewState

-m_foregroundClipLayer

+m_layer

+m_compositor

+m_graphicsLayer-m_glWebViewState

+m_backing

-m_contentLayer

-m_foregroundLayer

+m_content

+child

+m_client

rootPlatformLayer()

+m_imageTexture+m_texture

-m_drawingLayer

-m_surface

+m_texture

+m_tiles

+m_paintingPicture

-m_textureB-m_textureA-m_backTexture-m_frontTexture

-m_tiledTexture

-m_writeableTexture#m_textureA-m_lockedConsumerTexture#m_textureB

2012-4-24 Roger, UC

Cont.

• GraphicsLayerAndroid是GraphicsLayer的实现类，它包含一个LayerAndroid（PlatformLayer）

• LayerAndroid包含一个SkPicture作为Layer的Backing Store，在Sync的时候，GraphicsLayerAndroid将对应的RenderLayer的内容记录到LayerAndroid的SkPicture里面（GraphicsLayerAndroid::syncCompositingState）

• LayerAndroids将内部的SkPicture预先绘制在Texture上，然后再绘制在WebView上（HW），或者直接绘制在WebView上（SW）

2012-4-24 Roger, UC

Cont.

• Pros– 充分利用硬件加速的能力

– 多线程架构，充分利用多核CPU的能力，各个线程之间基本不需要同步，几乎可以并发运行

– UI线程的负荷很小，只需要执行Texture的Composting（GPU），可以快速响应用户的操作

• Cons– 渲染架构十分复杂，特别是硬件加速的方式

– 依赖于Android 4.0专有的私有API来达成Texture Sharing

– 在软件渲染方式下效率并不高，UI线程的负荷仍然很重

2012-4-24 Roger, UC

Reference• GPU Accelerated Compositing in Chrome

• Hardware Acceleration on Mobile

• Texture Mapper: Accelerated compositing evolved

• QtWebKit Accelerated Compositing Report

• Enable Accelerated Compositing on the WebKit Android NDK port

• fluid animation with accelerated composition

• Understanding Hardware Acceleration on Mobile Browsers

The End

Thank you for your listeningYours Sincerely, Roger

Technology

Web Page Rendering and Accelerated Compositing