HVS-DBS:Human Visual System-aware Dynamic Luminance Backlight Scaling for Video

Streaming Applications

Andrea Bartolini, University of Bologna, DEIS

Italya.bartolini@unibo.it

2

Outline

• Introduction

• Dynamic Backlight Scaling (DBS)

• Related work

• HVS-DBS - Idea

• HVS-DBS - Framework Implementation

• Conclusions

3

IntroductionEvery modern portable handheld device is equipped with a color LCD display:

• Increasing panel size

• Power consumption proportional to the panel area

• LCD power consumption is one of the main limiters to battery lifetime

Proc

LCD

RF

Proc

LCD

RF

RF

CPU

LCD

(Voice only dev) (Smartphone) (Gaming dev)

Proc

LCD

RF

Mobile device power breakdown from a legacy mobile device to a Smartphone/multimedia mobile device

to a gaming targeted mobile device. [1]

[1] “POWER MANAGEMENT IN MOBILE DEVICES”, Findlay Shearer, ELSEVIER

Video System Power

Breakdown [1]

4

Dynamic Backlight ScalingLCD emitted light is function

of 2 parameters:

• Backlight intensity

↓ power saving ↓ darker image

• LCD pixels trasmittance

↑ lighter image

LCD

Power Backlight

Pixel

trasmittance

final user

Power

Saving

Same

image

Ideally we can achieve the sameperception at human eyes with lowerbacklight intensity and higher LCD tramittance, while saving power.

In practice due to display system non-idealities image degradation.

Distortion

!!!

How to control & keep it constant for all the rendered images ??

Overall quality loss depends on :• pixel trasformation• pixel value• final backlight intensity level• LCD display electrical and optical properties• human visual system (HVS) features

5

How to control QoS degradation

Two main approaches in state-of-the-art DBS tecniques:• Frame Dependent: [1] [2] [3] [4]

• It computes on-line the amount of backlight scaling on every frame using a simpleimage distortion metric as constraint.

• The common approach is to on-line create the frame luminance histogram to keep constant the number of pixel that will saturate inside the increased luminance frame

• The simple performance metric adopted doesnot account for both HVS peculiarities and display system non-idealities weak relationship between the adopted metric and real perceived degradation

[1] “DLS: dynamic backlight luminance scaling of liquid crystal display.”, Chang N. Et al, IEEE Trans. VLSI Systems 2004

[2] “A Backlight Power Management Framework for Battery-Operated Multimedia Systems”, Shim H. et al, IEEE Des. Test 2004[3] “Quality Adapted Backlight Scaling (QABS) for Video Streaming to Mobile Handheld Devices, Cheng L. et al

[4] “Perception-guided power minimization for color sequential displays”, Cheng W et al, ACM Symposium on VLSI. 2006

Goal

Distortion

Input ImageDistortion

LevelProcessing

(not HVS based)

LuminaceCompensation

BacklightScaling

LCD

Backlight

# of saturated pixels

6

How to control QoS degradation

Two main approaches in state-of-the-art DBS tecniques:• Frame Independent: [1] [2] [3] [4]

• Very complex HVS aware frame processing algorithm it can not be evaluated on-line.

• The relationship between image distortion and BL scaling is statistically analysed off-line.• Uses a set of benchmark images to calculate an image-independent empirical function

which relates the image degradation levels to the transformation parameter applied to the image itself.

• Image degradation dependent only on the applied BL scaling.

• This approach does not consider the perceived image distortion as a function of the image itself.

[1] “Dynamic tone mapping for backlight scaling.”, Iranli A. and Pedram M., DAC 2005

[2] “HEBS: Histogram Equalization for Backlight Scaling”, Iranli A. et al, DATE 2005[3] “Backlight dimming in power aware mobile displays, Iranli A. et al, DAC 2006

[4] “HVS-Aware Dynamic Backlight Scaling in TFT-LCDs”, Iranli A. et al, IEEE Trans. VLSI Systems 2006

7

Our approach & main contributions

We are focused on:• Video streaming applications

• QoS-Power saving tread-off

• Frame dependent processing algorithm

• Overcome the QoS limitations of today DBS techniques

• Low power overhead

• Real implementation

• Freescale i.MX31 eval. board + 3.5” SHARP TFT LCD Display

Our technique is:

• Human Visual System – and Display non-ideality aware

• Capable of keeping final QoS under control

• No impact on video performance and low power overhead

• We exploit in a smart and efficient way existing hardware facilities

• no extra HW or display modification are required

We introduced a new way to predict on-line the final distortion for backlight scaling techniques

8

HVS-DBS – ideaOur approach is • Frame dependent

• Uses a HVS-aware on-line distortion metric

• Based on a try-and-test approach to find the minimum backlight level that guaranties the goal QoS performance

Target

QoS

check

Input

frame

DBS

trasformation

α, backlight

No !Update

α, backlight

HVS-aware

QoS evaluation

Yes !

Apply to LCD

α, backlight

Block Diagram1. The incoming frame

pixels luminance is increased with a starting DBS parameter set

2. HVS-aware QoS evaluation: distortion between the original frame and the DBS transformed one

3. Is the target QoS we want ?

a. No !!

• Update α, backlight

• Re-iterate 1,2,3

a. No !!

b. Yes !!

• Apply α, backlight to the LCD

Due to real time constraints all these steps must be executed before

next frame has been decoded !!

This bound the number of iterations and (α, backlight ) possible

value !!

Iteractive section

9

• The i.MX31 represents the next step in high performance application processors

• Based on an ARM11 core

• Multi-Media and Floating-Point HW Acceleration supporting:• MPEG4 real-time encode of up to VGA at 30 fps

• 3D Graphics and other applications acceleration

• Image Processing Unit (IPU)

• IPU is designed to:

• interface to video/still image sensors and displays

• IPU main functions:

• Capturing image

• Preprocessing & Postprocessing:

– Color space format conversion

– Combining video and graphics planes

• Post-filtering

• Displaying video and graphics on:

– Synchronous Displays

– Asynchronous Displays

– TV

• The IPU main functionality is present in every new multimedia processor:

• the HVS-DBS framework can be easily ported on other different platforms.

Freescale i.MX31

10

Framework implementation

Video decoder

Input

Video

Stream

MemBuffer

Post-processing

Color Space Conversionand image resizing

LCD Display

Video chain original

1. Input Frame coming from

video decoder is memorized

in a memory buffer

2. The Video driver send it to

the IPU post-processing to

adapt its color format and

size to the LCD Display

3. Then the frame is displayed

on the LCD pannel

11

Video chain HVS-DBS

Framework implementation

Video decoder

Input

Video

Stream

Proc. rate

MemBuffer

Down-sampling

QoS Embedded

Framework

search engine

Target QoSreached?

Target QoS

Post-processing

Color Compensation on Original Frame

and

LCD Backlight Scaling

Pre-processing

DBS Trasnformationon DownSampled

Frame

α0

Backlight 0

New values

Backlighti+1 & αi+1

NO!

Backlight i+1

αi+1

LCD Display

Apply

BacklightBEST & αBEST

YES!

IDS

IPREP

QoSi

Iteractive

section

1. YES! Select the BEST α and Backlight pair that

match target QoS and send it to Post-process stage

2. The original frame (Full Dimension) pixels are

luminance increased of α BEST factor with IPU HW

3. Send the Enhanced frame to the LCD display and

Backlight BEST value

Software routine

executed by the CPU !!

Executed in HW

by the IPU !!1. Each N frames the frame is

downsampled to reducing it dimension

2. The downsampled frame is luminance

enhanced using the IPU HW CSC block

3. The QoS Embedded Framework

computes the QoS score between the

Downsampled and the pre-processed

frames

4. Is this QoS score the one we want?

5. NO ! Try with another α and Backlight

pair

12

QoSi

IDS

IPREP

Framework implementation

Video decoder

Input

Video

Stream

Proc. rate

MemBuffer

Down-sampling

QoS Embedded

Framework

search engine

Target QoSreached?

Target QoS

Post-processing

Color Compensation on Original Frame

and

LCD Backlight Scaling

Pre-processing

DBS Trasnformationon DownSampled

Frame

α0

Backlight 0

New values

Backlighti+1 & αi+1

NO!

Backlight i+1

αi+1

LCD Display

Apply

BacklightBEST & αBEST

YES!

Post-process and pre-process routine

• Increase the input image pixels luminosity

• Require to multiply all the image pixels for a matrix

)(

00

00

00

)( RGBpixelRGBpixel

B

G

R

ENHANCED

too expensive in

CPU !!!

ENERGY EFFICIENCY

REAL-TIME PERFORMANCE

REQUIREMENT

Executed by Image Processing Unit HW

Color Space Conversion module

Low performance and

power overhead !!!

13

IDS

IPREP

QoSi

Framework implementation

Video decoder

Input

Video

Stream

Proc. rate

MemBuffer

Down-sampling

QoS Embedded

Framework

search engine

Target QoSreached?

Target QoS

Post-processing

Color Compensation on Original Frame

and

LCD Backlight Scaling

Pre-processing

DBS Trasnformationon DownSampled

Frame

α0

Backlight 0

New values

Backlighti+1 & αi+1

NO!

Backlight i+1

αi+1

LCD Display

Apply

BacklightBEST & αBEST

YES!

14

QoS embedded framework

QoS

Embedded Framework

LCD Model

RGB2IPT

LCD Model

RGB2IPT

SSIM index

It implements an embedded version of the

Visual QoS framework presented in [1].

2. RGB2IPT

• It transforms LCD Model output images from RGB to IPT colour space

• IPT accounts for human eyes different sensibilities for different colours

3. SSIM index routine

• models the HVS peculiarities and evaluates the differences between the two images from the LCD model block in IPT color space

• based on the assumption that the human visual system is efficient in extracting structural information from the viewing field

• It has been demonstrated to be able to quantify the HVS perceived differences between two images.

Basic Blocks:1.LCD model

• models the behaviour of a real embedded LCD panel and its non-ideality

• Input: (RGB image & backlight level)

• Output: image that evaluate the final rendering on the LCD display

[1] “Visual quality analysis for dynamic backlight scaling in LCD systems”, Bartolini A. Ruggiero M. and Benini L., DATE 2009

IDS IPREP

QoSi

15

LCD Model and RGB2IPT

QoS

Embedded Framework

SSIM index

What inside the LCD Model and RGB2IPT conversion ?

• LCD gamma & pixels saturation model

• Display image rendering estimation

• LMS cones responce color space rapresentation,

describe human eyes cone sensitivity suitable for detecting color adaptation artifact

• IPT color space rappresentation

• Enhanced version of CIELAB

• Directly related to percieved lightness, hue

and saturation

BACKLIGHT LEVEL

43.0|| R

G

B

43.0||

43.0||

I

P

T

Not linear !! Linear

X

Y

Z

L

M

S

LCD Model

RGB2IPT

LCD Model

RGB2IPT

too expensive with

only CPU ALU

operations !!!

Input frame format:

• RGB565

•5 bit red & blue

•6 bit green

Set of Look-up-Table (LuT)

• Accessed with the RGB565 pixel value

• Contains the I,P,T value (8bit each)

• One for each backlight level allowed 9

levels

16bit

We can

compute

SSIM only

over I

LuT need to

memorize

only I

Saving

memory

Improove

performance

IDS IPREP

QoSi

16

SSIM index

QoS

Embedded Framework

What inside the SSIM index ?

• Too high computational cost !!

• Cost dominated by the 2D filter

routine input size

• Input image down-sampled

• Gaussian window must be

scaled down too to preserve the

same locality information

• Fixed point aritmetic

• Error map must be averaged

• Downsample it before

reducing the average and

filters cost

LCD Model

RGB2IPT

LCD Model

RGB2IPT

SSIM index

2D Filter

2D Filter

2D Filter

2D Filter

2D Filter

-+-+ -+

ImO ImDLS

ImO2 ImDLS

2

µDLS

µDLS2µOµDLS

µO

µO2

σO2 σDLS

2σOσDLS

22

2

DLSO

DLSO

22

2

DLSO

DLSO

avg SSIM

Error Map

IDS IPREP

QoSi

17

IDS

IPREP

QoSi

Framework implementation

Video decoder

Input

Video

Stream

Proc. rate

MemBuffer

Down-sampling

QoS Embedded

Framework

search engine

Target QoSreached?

Target QoS

Post-processing

Color Compensation on Original Frame

and

LCD Backlight Scaling

Pre-processing

DBS Trasnformationon DownSampled

Frame

α0

Backlight 0

New values

Backlighti+1 & αi+1

NO!

Backlight i+1

αi+1

LCD Display

Apply

BacklightBEST & αBEST

YES!

Search engine

• Input : current SSIM index score

• It checks the conditions : SSIMi > SSIMTARGET

• SSIM index monotonically decreases on α

• It searches on a binary tree the future (αi, Backlighti )

• Binary search tree dept limited to real time constraints and performance

trade-off => 3 iteration => 9 Backlight levels

18

Experimental results

Framework implementation performance :

• Input test videoproperties:

• Different frame contents :

• Indoor scene darker pixels

• Outdoor scene lighter pixels

• HVS-DBS Power overhead

< Frame time budget !!

CPU 11%

Peripheral 1%

Main memory 13%

Δ T Frame-processing 17,3 ms

19

Histogram based DBS performance

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20% 8% 4%

% o

f v

ide

o f

ram

es

fo

r a

Qo

S

reg

ion

% of saturated pixels

Very low Low Medium High

QoS regions

Terminator3 frames for QoS regions Frame dependent (not HVS)

Luminance histogram technique

keeps the saturated pixels on

each frames below a specific

threshold

Bad frame even with

conservative settings !!!

There is always a noticeable

image distortion !!!

0%

5%

10%

15%

20%

25%

30%

20% 8% 4%

Po

wer

savin

g

% of saturatedpixels

Terminator 3 system power

Total video Indoor section Outdoor section

Entire system power

saving

• CPU

• Memory

• Peripheral• LCD

Power saving depends

on the frame content !!

Saturated pixels

histogram smalloverhead up 25%

power saving

20

Entire system power

saving

• CPU

• Memory

• Peripheral• LCD

HVS-DBS performance

0%

5%

10%

15%

20%

25%

Low Medium High Best

Po

wer

savin

g

Target SSIM

Terminator 3 system power

Total video Indoor section Outdoor section

QoS regions

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Low Medium High Best% o

f v

ide

o f

ram

es

fo

r a

Qo

Sre

gio

n

Target SSIM

Terminator3 frames for QoS regions

Very low Low Medium High

Absence of bad frames in

Medium – High – Best

settings

Fine grain tuning of the

final QoS !!!

HVS-DBS

Up 5% of system power saving

with no Image distortion

Up to 20% of system powersavings with low quality mode

allowed

21

Conclusion

• We proposed a novel dynamic backlight scaling approach to overcome state-of-the-art DBS techniques limitations

• Our solution is based on a on-line HVS-aware metric to findthe optimal backlight level for a specified QoS

• We provide a real implementation with reduced complexity in a multimedia embedded processor

• No video performance impact

• Significant system power savings - 5% to 20%

• Our HVS-DBS is robust and finds the optimal trade-offbetween QoS and power savings

22

Pubblications

• A. Marongiu, L. Benini, A. Acquaviva, A. Bartolini. Analysis of Power Management Strategies for a Large-Scale SoC Platform in 65nm Technology. In DSD 2008

• M. Ruggiero, A. Bartolini, L. Benini. DBS4video: Dynamic Luminance Backlight Scaling based on Multi-Histogram Frame Characterization for Video Streaming Application. In EMSOFT 2008

• A. Bartolini, M. Ruggiero, L. Benini. Visual quality analysis for dynamic backlight scaling in LCD systems. In DATE 2009

• A. Bartolini, M. Ruggiero, L. Benini. HVS-DBS: Human Visual System-aware Dynamic Luminance Backlight Scaling for Video Streaming Applications. In EMSOFT 2009