IMAGECON 2017The Physics of Fast GraphicsTobin Titus @tobint

IMAGECON 2017

The Impact of Images 100 resource requests, 18 Domains, 33 TCP Connections

Source:

54 7 23 9 7

Images CSS JavaScript HTML Other

1627 81 408 48 101

Images CSS JavaScript HTML Other

2.467 MB Total Size, 51% Cacheable

IMAGECON 2017

Seemingly simplistic …

Download Decode Display

IMAGECON 2017

.. but surprisingly complexNetworkAcquire

Disk Cache / Storage

CPUDecode the Image

MemoryHold the Decoded Image

Memory BusTransmit to the GPU

GPUTransform and Update Display

DisplayDraw the GPU Output

BatteryPower all the Things!GB

IMAGECON 2017

Responsiveness makes it more complex!

IMAGECON 2017

DownloadGetting data to your device

© 2012 mass:werk

IMAGECON 2017

Bandwidth vs ThroughputBandwidth Throughput

IMAGECON 2017

Global Mobile Network Speeds

0

2

4

6

8

10

12

14

16

18

20

0.1

3.3

10.8

17.4

2G 3G WiFI 4G LTE

Spe

eds

(Mbp

s)

Source:

State of LTE – Nov 2016

IMAGECON 2017

Overhead Plays Significant Role

NYC to: LatencyKM Miles Vacuum Fiber RTT 1 Packet

Seattle 3,870 2,405 12.9 19.7 39.3 157.6London 5,576 3,465 18.6 28.3 56.6 226.4Beijing 11,001 6,835 36.7 55.9 111.7 447.2

TCP

TLS

Data

IMAGECON 2017

Last Mile LatencySource:

Measuring Broadband America - 2016

IMAGECON 2017

DecodeUnpacking the Downloaded Image File

IMAGECON 2017

Download Size vs Memory SizeDownload sizes may seem small, but image memory adds up very quickly

Dimensions: 683 x 1024Color Space: RGBADisk Size: 557 KBMemory: width * height * 4

2.67 MB Decoded

IMAGECON 2017

Memory UtilizationResponsible memory management is crucial when targeting low-cost devices with your sites and apps

-

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

160,000,000

180,000,000

200,000,000

5000 sites’ image memory utilization

Images: Downloading

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Decoding

Images: Continue Downloading

Req

uest

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Par

tial

Res

pons

e R

ecei

ved

Decoding

Par

tial

Res

pons

e R

ecei

ved

Images : Decoding

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Decoding

Processing

Par

tial

Res

pons

e R

ecei

ved

Images: Processing

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Decoding

Images: Copy to the GPU

Req

uest

Cop

y to

GP

U

Par

tial

Res

pons

e R

ecei

ved

Par

tial

Res

pons

e R

ecei

ved

ProcessingDecoding

Cop

y to

GP

U

Images: Download to Display

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Par

tial

Res

pons

e R

ecei

ved

Dis

play

ProcessingDecoding

Images: Download to Display

Decode

UI Thread

Stylesheet

HTML Doc

Image

GPU Usage

Dow

nloa

dsC

PU

U

sage

GP

U

Usa

ge

GPU Copy

Cop

y to

GP

U

Req

uest

Par

tial

Res

pons

e R

ecei

ved

Par

tial

Res

pons

e R

ecei

ved

Dis

play

ProcessingDecoding

IMAGECON 2017

DisplayShowing Your Images to The User

IMAGECON 2017

Image Formats MatterImage formats use different means to depict colors Y

CbC

r (Y

UV

)

RGB to YCbCr Color Conversion

Chroma Subsampling

DCT / Quantization

Huffman Encoding

Enc

ode

& S

ave

RG

B

Chroma Subsampling

+

Original Luma (Y) Chroma (Cb)

Chroma (Cr)

= + +4:4:4

Original Luma (Y) Chroma (Cb)

Chroma (Cr)

= +4:2:0

Original Luma (Y) Chroma (Cb)

Chroma (Cr)

= + +4:2:2

62.5% less memory than RGB

(no subsampling)

JPEG images were previously decoded, upsampled and color-converted on the CPU before copying them to the GPU

JPEG – From Download to DisplayIn newer browsers, JPEG is hardware-accelerated. Now JPEG has all the benefit of small file size as well as decreased CPU decoding, smaller graphics memory footprint, and faster GPU copy operations Huffman Decoding Dequantization /

iDCTChroma

UpsamplingYCbCr to RGB

Color Conversion

Dec

ode

Cop

y to

G

PU

Req

uest

Par

tial

Dat

a

Par

tial

Dat

a

Dis

play

ProcessingDecoding

Pas

t

Cop

y to

G

PU

Req

uest

Par

tial

Dat

a

Par

tial

Dat

a

Dis

play

Decoding

Pro

cess

ing

Pre

sent

IMAGECON 2017

Drain!Powering your Device without Overheating

Power ConsumptionLet it rest! Power efficiency can drain your users’ battery and decrease satisfaction with your application

CP

UU

tiliz

atio

nG

PU

Util

izat

ion

VS

ync

Power ConsumptionLet it rest! Power efficiency can drain your users’ battery and decrease satisfaction with your application

CP

UU

tiliz

atio

nG

PU

Util

izat

ion

VS

ync

Power ConsumptionLet it rest! Power efficiency can drain your users’ battery and decrease satisfaction with your application

CP

UU

tiliz

atio

nG

PU

Util

izat

ion

VS

ync

IMAGECON 2017

Memory And Power ConsumptionMaintaining memory consumes power and generates heat. The sites to the left of the graph use far less power than the sites on the right.

-

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

160,000,000

180,000,000

200,000,000

5000 sites’ image memory utilization

IMAGECON 2017

Navigation App or SiteNetwork / GPS1 watt per antenna

CPU1 watt per core

GPU2 watts

MemoryIt depends

GB

IMAGECON 2017

ResourcesOverview ConceptsHigh Performance WebsitesSteve Souders, September 2007

Event Faster Websites: Best PracticesSteve Souders, June 2009

High Performance Browser NetworkingIlya Grigorik, September 2013

A Picture Costs a Thousand WordsGuy Podjamy, November 2013

High Performance ImagesColin Bendell, Tim Kadlec, Yoav Weiss,Guy Podjarny, Kick Doyle & Mike McCallNovember 2016

Microsoft GuidanceHigh-Performance Web PlatformTobin Titus, Build 2014

Web Runtime PerformanceTobin Titus, Build 2013

Performance Tricks to Make Apps & Sites FasterJatinder Mann, Build 2012

50 Performance Tricks for Windows Store AppsJason Weber, Build 2011

W3C Web PerformanceWeb Performance Working Group Homepage

Blog Posts1) JPEGXR updates (comparisons to WebP)2) Debugging/Tuning Web Sites w/F12 Developer To

ols3) Measuring Performance with ETW/XPerf4) Measuring Performance in Lab Environments5) Browser Subsystems Overview6) What Common Benchmarks Measure

ToolsWindows Performance ToolkitFiddler Web Debugger

Contact

Tobin TitusTwitter: @tobintEmail: tobint@microsoft.com

IMAGECON 2017

© 2017 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.