Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

QualityofExperienceinMultimediaSystemsandServices:

AJourneyTowardstheQualityofLife

ChristianTimmerer (AAUKlagenfurt,Austria)FernandoPereira(IST-IT,Portugal)

Touradj Ebrahimi (EPFL,Switzerland)

IEEEInternational Conference on Multimedia&Expo(ICME)

11thJuly 2016,Seattle,WA,USA

Outline

1. QualityofExperienceforMultimediaSystemsandServices

FernandoPereira

2. ApplicationsofQoE:AdaptiveVideoStreamingandSensoryExperience

ChristianTimmerer

3. TowardstheConceptofQualityofLifeTouradj Ebrahimi

FernandoPereira:AboutMe…• AssociateProfessoratUniversityofLisbon,Portugal• SeniorResearcheratInstituto deTelecomunicações,Lisbon,Portugal• Morethan250publicationsininternationaljournalsandconferences• OneofthedesignersoftheMPEG-4andMPEG-7standards• ICIP,PCS,VCIP,WIAMIS,QoMEX GeneralorTechnicalProgramChair• AssociateEditorofseveraljournals• Editor-in-ChiefoftheIEEEJournalofSelectedTopicsinSignalProcessing(2013-2015)• ISO/IECAwardforcontributionstotheMPEG-4VisualStandard• SPSDistinguishedLecturer• IEEEFellowin2008for“contributionstoobject-baseddigitalvideorepresentation

technologiesandstandards”• EURASIPFellowin2013for“contributionstodigitalvideorepresentationtechnologies

andstandards”• IETFellowin2015• IEEESPSBoardofGovernorsandEURASIPBoardofDirectors• SeveralExcellenceTeachingAwards• JPEG(currently)andMPEG(past)RequirementsChair


A. WhatisQuality

B. QualityofService(QoS)

C. QualityofExperience(QoE)

D. TrendsinQoE

A.WhatisQuality?

Quality:aSimpleyetDifficultConcept

• Likemanyhumansensations,qualityiseasytounderstandbutdifficulttodefine.

• AccordingtoWikipedia:– Aquality(fromLatin- qualitas)isanattribute oraproperty.

– Somephilosophers assertthataqualitycannotbedefined.

– Incontemporaryphilosophy,theideaofqualitiesand

especiallyhowtodistinguishcertainkindsofqualitiesfromoneanotherremainscontroversial.

AnOld,LargelyUnder-InvestigatedConcept

Aristotleclassifiedeveryobjectofhumanapprehensioninto10Categories– Substance– Quantity– Quality– Relation– Place– Time– Position– State– Action– Affection

Aristotle,384BC– 322BC,Greece

Quality:SomeDefinitionsfromtheDictionary(1)

• Definition1– General :Measureofexcellenceorstate ofbeingfreefromdefects,deficiencies,andsignificantvariations.

– ISO8402-1986standarddefinesqualityas“thetotalityoffeatures andcharacteristics ofaproductorservicethatbearsitsabilitytosatisfy statedorimpliedneeds”.

• Definition2– Manufacturing :Strictandconsistentadherencetomeasurable andverifiablestandardstoachieveuniformityofoutputthatsatisfiesspecificcustomer oruserrequirements.

Quality:SomeDefinitionsfromtheDictionary(2)

• Definition3– Objective :Measurableandverifiableaspectofathingorphenomenon,expressedinnumbersorquantities,suchaslightnessorheaviness,thicknessorthinness,softnessorhardness.

• Definition4– Subjective:Attribute,characteristic,orpropertyofathingorphenomenonthatcanbeobservedandinterpreted,andmaybeapproximated(quantified)butcannotbemeasured,suchasbeauty,feel,flavor,taste.

QualityAccordingtoISO9000

• ISO9000:afamilyofstandardsforqualitymanagementsystems.

• Qualityofsomethingcanbedeterminedbycomparingasetofinherentcharacteristicswithasetofrequirements– Highquality:ifcharacteristicsmeetrequirements– Lowquality:ifcharacteristicsdonotmeetallrequirements

• Qualityisarelativeconcept– Degree ofquality

QualityislikeanElephant…

Theblindmenandtheelephant,poembyJohnGodfreySaxe

QualityinQUALINET

• Quality:Istheoutcomeofanindividual’scomparisonandjudgmentprocess.Itincludesperception,reflectionabouttheperception,andthedescriptionoftheoutcome.

• Incontrasttodefinitionswhichseequalityas“qualitas”,i.e.asetof

inherentcharacteristics,QUALINETconsidersqualityintermsofthe

evaluatedexcellenceorgoodness,ofthedegreeofneedfulfillment,andin

termsofa“qualityevent”(seeMartens&Martens,2001,andJekosch,

2005).

• Event:Anobservableoccurrence.Aneventisdeterminedinspace(i.e.

whereitoccurs),time(i.e.whenitoccurs),andcharacter(i.e.whatcanbe

observed).

from“QualinetWhitePaperonDefinitionsofQualityofExperience”,March2013

Whatis QUALINET?

• Groupofinstitutionsandcompaniesinterestinmultimediaquality.• CoordinationofmultidisciplinaryQoE researchinEuropeandbeyond.• Strengtheningdisseminationeffortsthroughalreadyestablished,andnew

initiatives,e.g.QoMEX,specialevents,books,journals.• Strengtheninginteractionbetweenacademiaandindustry(industrial

forum,STSM,…).• StrengtheningeducationaleffortsinQoE,e.g.summerschools,PhD

events,exchangeofyoungresearchers.• Coordinatedcontributiontointernationalstandardizationbodies,e.g.

ISO/IEC,ITU-T,VQEG,MPEG,JPEG.

Insummary,groupofresearchersinterestedin(multimedia)QoEissues,boththeoreticalandpractical…

Opentonewresearchers…http://www.qualinet.eu/


A. WhatisQuality



D. TrendsinQoE

B.QualityofService

QualityofService(QoS):inTheory

“[The]Totalityofcharacteristicsofatelecommunicationsservicethatbearonitsabilitytosatisfystatedandimpliedneedsof

theuser oftheservice.”

ITU-TRec.E.800,2008

• QoS isfocusedontelecommunicationsservices.

• ThecontextofusageandtheusercharacteristicsarenotcomprehensiblyaddressedbyQoS asdefinedbytheITU.


QualityofService(QoS):De-Facto

• TheQoS de-factodefinitiondealsmostlywithphysical,measurable

performancefactorsofnetworksanddeliveryplatformsingeneral.

• Sometimes,alsoapplication-levelfactors,suchasencodings,and

theireffectontheunderlyingnetwork'sperformanceareaddressed,

e.g.bytakingmoreoftheavailablebandwidth.

QualityofService(QoS)referstoacollectionofnetworkingtechnologiesandmeasurementtoolsthatallowforthenetworktoguarantee deliveringpredictableresults.

partlyfrom“Qualinet WhitePaperonDefinitionsofQualityofExperience”,March2013

QualityinQoS Framework:SeveralDimensions

NetworkQualityCapacityCoverageHandoff

LinkQualityBitrate

Frame/Bit/PacketlossDelay

UserQualitySpeechfidelityAudiofidelityImagefidelityVideofidelity

ThemultimediasignalprocessingcommunityisalreadyoftenusingconceptssuchastheMeanOpinionScore

(MOS)whichdirectlyinvolvesusers…

QoS inComputerNetworksandCommunications

• QualityofService(QoS)– Resourcereservationcontrolmechanisms– Abilitytoprovidedifferentprioritytodifferentapplications,users,ordataflows

– Guarantee acertainlevelofperformance (quality)toadataflow

• (Service)Provider-centricconcept

QoS Boundaries

• Scope:QoS typicallyfocusesontelecommunicationsservices.

• Focus:QoS dealswithperformanceaspectsof physicalsystems.

• Methods:QoS hasaverytechnology-orientedapproach,anditreliesonanalyticapproachesandempiricalorsimulativemeasurements.


UserQuality:MostlySignalFidelity

• SubjectiveEvaluation

• ObjectiveEvaluation

SubjectiveEvaluation

• SubjectivetestsaimatproducingUserOpinionScoresasadelicatemixtureofingredientsandchoices:– Test&labenvironment– Testmaterial– Testmethodology– Testsubjects– Analysisofthedata

WhatisMeanOpinionScore(MOS)?

• Widelyusedinmanyfields:– Politics/Elections– Marketing/Advertisement– Foodindustry– Multimedia– …

• Thelikelylevelofsatisfactionofaspecificservice/productdimension,e.g.visualquality,asappreciatedbyanaverageuser(fromaproviderpointofview).

• Shouldbeperformedsuchthatitgeneratesreliable andreproducibleresults– Subjectiveevaluationmethodology– Morecomplexanddifficultthatitaprioriseems– Muchusedfor(andlimitedto)videoandaudiosubjectivequalities

ObjectiveEvaluation

• Subjectivetestsaretimeconsuming,expensive,anddifficulttodesign…

• Objectivealgorithms,i.e.metrics,estimatingsubjectiveMOSwithhighlevelofcorrelationaredesired– Fullreferencemetrics– No-referencemetrics– Reducedreferencemetrics

FR,RRandNRScenarios• FullReferenceapproach:

• Reduced Reference approach:

• No-Referenceapproach:

Input/Referencesignal

Output/Processedsignal

Signalprocessing



Signalprocessing

FRMETRIC

NRMETRIC



Signalprocessing

Featuresextraction RRMETRIC

AutomaticMOSPredictorsareEssential…

FullReferencescenario• MostautomaticMOSpredictorsarebasedonfidelitymeasures• Metricslookatthefidelity ofthesignalwhencomparedtoan

explicit́ perfect’reference:

processedsignal=perfectqualityreferencesignal+errorsignal• Examples:

– MeanSquareError(MSE)– PeakSignaltoNoiseRatio(PSNR)– WeightedPSNR– MaskedPSNR– StructuralSIMilarity (SSIM)– Multiscale StructuralSIMilarity (MSSIM)– VisualInformationFidelity(VIF)


A. WhatisQuality



D. TrendsinQoE

C.QualityofExperience

ChangingLandscape

UHD,4KHDRHFR3DLightfieldsPointclouds…

31

ManyEvents...BuildingExperiences...

• Event:Anobservableoccurrence.Aneventisdetermined in

space(i.e.where itoccurs),time(i.e.whenitoccurs),and

character(i.e.whatcanbeobserved).

– Sensation referstotheresponsesofsensoryreceptorsandsenseorganstoenvironmental stimuli.

– Perceptionisaprocesswhichinvolvestherecognitionandinterpretationofstimuliwhichregisteroursenses.

– Emotion isanyrelativelybrief consciousexperiencecharacterizedbyintensementalactivityandahighdegreeofpleasureordispleasure.

• Experience:Anexperienceisanindividual’sstreamofperceptionandinterpretationofoneormultipleevents.

partlyfrom“Qualinet WhitePaperonDefinitionsofQualityofExperience”,March2013

So,UsersareMorethanPerceptionEngines…

ManyServicesSellEmotions...

Multimedia Nowadays …

• Multimedia is aboutsharingexperiences(realorimaginary)with others.

• Inaway,it allstarted with storytelling andwall drawing around thefire inthecavesofearly men.

• Modernmultimedia systemsareevolvedversionsofthegoodold storytellingandwall drawing,which hopefullyofferincreasingly richer experiences.

• Thedegree ofrichness oftheexperiencemay be measured byQuality ofExperience(QoE).

Evolving Quality Paradigms

WhatdoPeopleTalkaboutwhentheyTalkaboutQoE ?

• “Thedegreeoffulfillmentofanintendedexperienceonagivenuser.”

byTouradj Ebrahimi,2001

• “perceiveduserexperience ispsychologicalinnatureandchangesindifferentenvironmentalconditionsandwithdifferentmultimedia

devices.”fromQoMEX 2009CallforPapers

• “Theoverallacceptabilityofanapplicationorservice,asperceived

subjectivelybytheenduser.”asdefinedbytheITU

Theterm‘experience’isappealingbecauseitimplicitlypromisesindividualengagement…Lookgood,soundgood,

andfeel good!

QUALINETQoE Definition• QualityofExperience(QoE)isthedegree ofdelightor

annoyance oftheuser(persona)ofanapplicationorservice.It

resultsfromthefulfillmentofhisorherexpectationswithrespecttotheutilityand/orenjoymentoftheapplicationorserviceinthelightoftheuser’spersonalityandcurrentstate(context).

• Experience:Anexperienceisanindividual’sstreamofperceptionandinterpretation ofoneormultipleevents.

• QoE feature:Aperceivable,recognizedandnamablecharacteristicoftheindividual’sexperienceofaservicewhichcontributestoitsquality.

Inthecontextofcommunicationservices,QoE canbeinfluencedbyfactors

suchasservice,content,network,device,application,andcontextofuse.


MovingtoQualityofExperience

• Quality of Service: Value of the average user’s servicerichness estimated by a service/product/contentprovider

• Quality of Experience: Value (estimated or actually measured)of a specific user’s experience richness

QualityofExperienceisthedual(andextended)viewofQualityofService!

QoS=provider-centricQoE=user-centric

FactorsImpactingQualityofExperience

Context

• System/TechnicalInfluenceFactors

• Human/UserInfluenceFactors

• ContextInfluenceFactors

• ContentInfluenceFactors

• SocialandPsychologicalInfluenceFactors

ExperiencesareIndividual!

• ApplicationsandServicesmayhavetobedesignedtoprovideindividualexperiences...

• Thisinvolvescapabilitiesallowingtheusertogaincontrol,e.g.interaction,personalization,recommendation,etc.

• Infact,theusercontributestobuildisownexperience...Ifthesystem/serviceallows...

HowShallaMultimediaUserExperienceBe?

Dependingonthespecificapplication,itmayhavetobe

• Faithful - accuracy• Truthful – realisticifrelevant,synchronization• Immersive – natural,multimodalconsistency

• Contextual - adaptive• Engaging – fun,intense,emotional

• Effective– fast,recognition

• Useful – taskperforming

• Interactive – natural,shortdelay

• Intuitive,Easy– interfaces

• …

QoE Modeling

QoE modelingmay considermoreor lessinfluence factors dependingon the

service/application,eachwith adifferentweight on the overall assessment.

QoE is multi-dimensional,multi-modal andmulti-sensorial.

Usercenteredinfluencefactorsareexpectedtobedominating.

• System factors– technical properties (as in

QoS)

• Human/User factors – individual properties– sensorial properties– perceptual properties– emotional properties

• Context factors– environmental/physical

properties– temporal properties– service properties– economic properties– social properties

• Content factors

• …

APracticalQoEModelExample:IPTV• Video quality• Audio quality• Audiovisualsyncronization• Stalloccurence• Errorresilience• Randomaccess• Channel hoppingdelay• Interfaceusability• Navigation capabilities• Personalization capabilities• Metadata quality• Immersioneffectiveness• …

QoE:NotanEasyTarget…WhyShoulditbe?

QoS

QoP

QoE

QoS/P/E:Quality of Service/Perception/Experience

Experiencesaremultisensorial...

BuildingMultisensorialImmersion...

• ToinsertMarianna’smovie...

Feel-around,fromKentuckyFriedMovie

QoE Assessment:AgainSubjectiveandObjective

QoS versusQoE

• Scope:QoS typicallyfocusesontelecommunicationsservices,whereasQoE coversamuchbroaderdomain,whichsometimesdoesnoteveninvolvetelecommunications.

• Focus:QoS dealswithperformanceaspectsofphysicalsystems,whereasQoE dealswiththeusers'assessmentofsystemperformance,ascoloredbycontext,culture,theusers'expectationswithrespecttothesystemorserviceandtheirfulfillment,socio-economicissues,andpsychologicalprofiles,amongotherfactors.

• Methods:QoS hasaverytechnology-orientedapproach,whereasQoErequiresamulti-disciplinaryandmulti-methodologicalapproachforitsunderstanding.

• ButitisalsoimportanttorememberthatQoE is,inalargepartof

instances,highlydependentonQoS.


QoE islikea(Bigger)Elephant…

Theblindmenandtheelephant,PoembyJohnGodfreySaxe

QoE inNetworkedMultimedia

QoE RelatedStandardizationEfforts

• Standardizationeffortsinqualityassessmentandmetrics– ITU-TSG12(Performance,QoS andQoE)

– MPEG/ITU-T(HighEfficiencyVideoCoding,HEVC)

– MPEG(3Dvideocoding,FTV,HDR)

– VideoQualityExpertsGroup(VQEG)

– JPEG(AdvancedImageCoding,AIC)

– …

QUALINETestablishedlinksanddeepcollaborationswithallofthem!


A. WhatisQuality



D. TrendsinQoE

D.TrendsinQoE

QoE isBecomingInevitable…

• Digitalworldhas(re-)discoveredthenotionofquality– Lowerqualitycontentislessandlesstoleratedbyend-usersin

someenvironments– However,otherenvironmentsseemtoacceptmuchlower

qualityandstillbesuccessful

• IncreasinginterestinQoE– Extendingfromdevice-centricandsystem-centricquality

optimizationtoend-to-endandespeciallyuser-centricoptimization

QoEHolisticApproach

• Marketing• Businessmodel,e.g.prices,fidelization• Systemfactors• Contextfactors• Humanfactors• Personalization• Content(andmetadata)• Interface• Clientsupport• ...

NOSUMA:anExample

• UltraHD4K

• Portabilityaccrossterminals,i.e.followsyou

• Voicecontrol(voicerecognition?),i.e.recognizesyou

• Userprofileswithinsamefamily,i.e.individualizesyou

• Recommendationsbasedonusercharacterization,i.e.targetsyou

• Complementarycontentforthefavouriteseries,i.e.thinksonyou

• Timewarping,i.e.helpsyou

• ...ATVthat

knowsyou!

QoE inIndustry

• QoE isbecomingmainstream.

• ManycompaniesnowspeakaboutQoE.

• Personalization,interactionandrecommendationcapabilitiesempowertheusertocreatemoreindividualexperiences!

• However,QoE hasabudgetimpactintermsofnetworkandsystemdesign,dimensioning,operation,maintenance,etc.

• ButQoE isbecomingmoreaffordableinmanyapplicationdomains…

• Embracing QoEprinciplesmaybringrevenue,e.g.byincreasingviewingtimesandreducingchurn.

ChallengesAhead• Content-dependentquality assessmentmethods andmetrics• Context-dependentquality assessmentmethods andmetrics• Quality assessmentmethods andmetrics beyondAV(haptics,smell,

…)• Multi-modalqualityassessmentmethods andmetrics(AV,…)

• 3D quality assessmentmethods andmetrics (3Dsound,3Dvideo,…)

• Newmodalities contentquality assessmentmethods andmetrics

• Interactionqualitymetrics (closely related tousability)

• Presence/immersionqualitymetrics• Roleofemotions• Virtualreality immersiveexperiences• …

NewSensors...

ALightFieldImage …

Behindeachmicrolens,amicro-image(MI)isformed…

LightFieldPhotography:ArrayofCameras

NewDisplays

MicrosoftHololensOculusRift

Microsoftholographic display

InnoVision DiamondSeriesholographicprojector

USClightfielddisplay

HolografikaHoloViziolightfielddisplay

QoEforVirtualReality• Compellingimmersiveandrealisticvisualexperiences!

• Providesvisualdepthcues,suchasstereopsis,binocularocclusions,vergence,fullmotionparallaxandnaturalview-dependentlighting.

• Highresolutionandhighframerate.

• Lowlatencyspatialrandomaccess.

• Lowmotion-to-photonlatency.

OncurrentHMDs, theclosestdepth foranobjectofinterestisrecommended tobeat0.75mwithoutcausingexcessiveeyestrain.

Holoportation:Virtual3DTeleportation

CourtesyofP.Chou,Microsoft

WhatDoesthisallMean?

• Eraofuser-centricmultimediahasalreadystarted…UserisKing/Queen!

• Itisnotanymoresufficienttomerelyaddnewfeaturesandfunctionalitiestomultimediasystems.

• Trueaddedvalueintermsofimpactonuser’sexperienceofsuchfeaturesandfunctionsshouldbeevaluatedanddemonstrated.

• QualityofExperienceplaysacentralroleinthisnewgame!Alreadytargetingrevenue…

AssessingQualityofExperience…ABitLikeMeasuring‘Happiness’…

Take-HomeMessages

• QoEisuser-centric!• QoEisindividual,multidimensionalandmultisensorial.• ServicesandsystemsareincreasinglydesignedtoallowtheuserstomaximizeitsQoE.

• IndustryisincreasinglyembracingQoEprinciplesbecausetheymaybringrevenue.

• QoE assessmentiscostlyandriskybutworthdoingit.• Int’lConferenceonQualityofMultimediaExperience(QoMEX):http://qomex.org/

2. ApplicationsofQoE:AdaptiveVideoStreamingandSensoryExperience

A. AdaptiveVideoStreamingPrinciplesandQoE

B. QualityofSensoryExperience(QuaSE)

ChristianTimmerer:AboutMe…• AssociateProfessoratAlpen-Adria-Universität Klagenfurt,Austria

(blog.timmerer.com,dash.itec.aau.at)• ChiefInnovationOfficer|HeadofStandardizationandResearchat

Bitmovin Inc.,bitmovin.com• Geschäftsführer Förderverein Technische Fakultät,ftf.or.at• Lecturer|CarinthiaUniversityofAppliedSciences,www.fh-kaernten.at• Researchinterest:immersivemultimediacommunication,streaming,adaptation,Quality

ofExperience, andsensoryexperience• Morethan170publicationsininternationaljournalsandconferences• Generalchair:WIAMIS’08,QoMEX’13,QCMan’14,MMSys’16• Associateeditor/editorialboard:IEEEComputer,IEEETrans.onMultimedia,Signal

Processing:ImageCommunication,MTAP,IEEEComputingNow,ACMSIGMMRecords,ACMSIGMMOSSC

• VicechairofIEEEComSoC MMTC,WGleaderinQUALINET• Researchprojects:FP6-IST-DANAE(2004-2006),FP6-IST-ENTHRONE(2006- 2008),FP7-

ICT-P2P-Next(2008-2012),FP7-ICT-ALICANTE(2010-2013),FP7-ICT-SocialSensor(2010-2014),COST-IC1003-Qualinet(2010-2014),FFG-AdvUHD-DASH(2014-2016),andFP7-ICT-ICoSOLE(2013-2016)

• MPEG:MPEG-21,MPEG-M,MPEG-V,MPEG-DASH• IEEESeniormember;ACMmember

ApplicationsofQoE:AdaptiveVideoStreamingandSensoryExperience

Priv.-Doz.Dr.ChristianTimmerer[Ack:AliC.Begen,MediaMelon,Inc.,OzyeginUniversity]

Alpen-Adria-UniversitätKlagenfurt(AAU)w FacultyofTechnicalSciences (TEWI) w DepartmentofInformationTechnology(ITEC)wMultimediaCommunication(MMC)w SensoryExperience Lab(SELab)

http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w [email protected](CIO)atbitmovinGmbH

http://www.bitmovin.comw [email protected]

Tutorial@ICME2016,July2016

http://www.slideshare.net/christian.timmerer

ImportanceofMultimediaDelivery• Multimediaispredominanton

theInternet• Real-timeentertainment

– Streamingvideoandaudio– Morethan70%ofInternettraffic

atpeakperiods• Popularservices

– YouTube(17.85%),Netflix(37.05%),AmazonVideo(3.11%),Hulu (2.58%)

– Alldeliveredover-the-top (OTT)

July2016 ICME2016Tutorial,C.Timmerer 2GlobalInternetPhenomenaReport:Dec2015

OpenDigitalMediaValueChain

July2016 ICME2016Tutorial,C.Timmerer 3

CreateContent

AggregateMonetize

DistributeContent

ConsumeContent

AnyContent AnyStorefront AnyNetwork AnyDevice

CDNsMediaProtocols

InternetTransport

DRMEncoding

EncapsulationDynamicAds

Clients

HappyUser

CommonAnnoyancesinStreaming• Wrongformat• Wrongprotocol• Pluginrequirements• DRMissues• Longstart-updelay• Poorquality• Frequentstalls• Qualityoscillations• Noseekingfeatures


Over-The-Top– AdaptiveMediaStreaming• Inanutshell…


Adaptation logic is within the client, not normatively

specified by the standard, subject to research and

development

Multi-BitrateEncodingandRepresentationSwitching


Contents ontheWebServerMovieA– 200Kbps

MovieA– 400Kbps

MovieA– 1.2Mbps

MovieA– 2.2Mbps

...

...

MovieK– 200Kbps

MovieK– 500Kbps

MovieK– 1.1Mbps

MovieK– 1.8Mbps

.. .

.. .

Time(s)

Startquickly

Keep requestingImprove quality

Loss/congestiondetection

Revampquality

...

...Segments

AdaptiveStreamingoverHTTP


…………

HTTP GETs

ClientBuffer

MediaPlayer

HTTPServer

ScopeofDASH:whatisspecified?


Media Presentation on HTTP Server

DASH-enabled ClientMedia Presentation Description

.

.

.

Segment

…

.

.

.Segment

…

.

.

.

Segment

…

.

.

.Segment

…

…

Segments located by HTTP-URLs

DASH Control Engine

HTTP/1.1 HTTP Client

MPD Parser

Media Engine

On-time HTTP requests to segments

ScopeofDASH:whatisspecified?


Media Presentation on HTTP Server

DASH-enabled ClientMedia Presentation Description

.

.

.

Segment

…

.

.

.Segment

…

.

.

.

Segment

…

.

.

.Segment

…

…

Segments located by HTTP-URLs

DASH Control Engine

HTTP/1.1 HTTP Client

MPD Parser

Media Engine

On-time HTTP requests to segments

DASHDataModel


MPD

Period id=1start=0s

Period id=3start=300s



AdaptationSet0subtitleturkish

AdaptationSet2audioenglish

Adaptation Set1BaseURL=http://abr.rocks.com/

Representation2Rate=1Mbps


Representation1Rate=500Kbps

Representation 3Rate=2Mbps

Resolution =720p

SegmentInfoDuration=10s

Template:3/$Number$.mp4

Segment Access

InitializationSegmenthttp://abr.rocks.com/3/0.mp4

MediaSegment1start=0s

http://abr.rocks.com/3/1.mp4

MediaSegment2start=10s

http://abr.rocks.com/3/2.mp4

AdaptationSet3audiogerman

AdaptationSet1video



Selectionofcomponents/tracks

Well-definedmediaformat

Selectionofrepresentations

Splicingofarbitrarycontentlikeads

Chunkswithaddressesandtiming


type=static typically,for on demand content

Base URL of the segments

Subtitles

Audio adaptation set with different representations (bw)

Video adaptation set with different representations (bw)

Different codecs (profiles)

Segment URL constructed with template and base URL

http://www.dash-player.com/demo/


AdaptiveStreamingContentWorkflow


Source Transcoding Encapsulation Encryption OriginServer

HelperDistribution

Client

Linear:MulticastVoD:FTP,RTMP,HTTP,etc.

UnicastHTTP(PUSH),FTP,etc.

HTTPGETsmallobjects

Singlehighest-bitratestream Multiplestreamsat

targetbitrates

Multiplestreamsattargetencapsulationformats

Largevideo/virtualfilesandmanifests

AdaptiveStreamingContentWorkflowSimplified


StandardDeliveryInfrastructure(CDN)Source Transcoding Encapsulation Encryption

Multiplestreams:video[bitrate(32000-20000000),profile(baseline,main,high),preset(standard,professional,premium),height(128-7680),width(96-4320),framerate(1-120),codec(h264,hevc)],audio:[bitrate(8000-256000),samplerate(0,8000,11025,12000,16000,22050,24000,32000,44100,48000,64000,88200,96000)]

Singlehighest-bitratestream:HTTP,FTP,RTMP;mp4,ts;AVC,AAC,Subtitles

Multiplestreamsattargetencapsulationformats:DASH(MPD+mp4),HLS(m3u8,ts)

MultiplestreamswithmultipleDRMformats:MPEG-CENC,Widewine,PlayReady,PrimeTime,Fairplay

Player

HeterogeneousClientse.g.BitmovinHTML5AdaptivePlayerDASH,HLS,HTML5,MSE,EME

QoEforDASHServices• DifferentapplicationdomainshavedifferentQoE

requirements– NeedtoprovidespecializationsofthegeneralQoE

definition– Takeintoaccountrequirements formulatedbymeansof

influence factorsandfeaturesofQoE• QoEinfluencefactorsforDASH

– Initial/start-updelay(low)– Bufferunderruns, stalls,freezes(zero)– Qualityswitches(low)– Mediathroughput (high)– …


!

h t t p s : / / b i t m o v i n . c om /

QoEEvaluationforDASH-basedServices• Testsequence

– Manydatasetsavailable– AdoptedBigBuckBunny&DASHed it

• Players– bitdash– Proprietarysolutions(smooth,HLS,HDS)– YouTube,dash.js,DASH-JS– …andcompareitwithtendifferentadaptationalgorithms

• Objectiveevaluation– Commontestsetupusingnetworkemulation&bandwidthshaping– Predefinedbandwidthtrajectory(orrealnetworktraces)

• Subjectiveevaluation– Lab[ITU-TB.500/P.910]vs.

crowdsourcingwithspecialplatformsorsocialnetworks


CrowdsourcedQoEEvaluation• QualityofExperience …

– MeanOpinionScore[0..100]– [otherobjectivemetrics:

start-uptime,throughput,stalls]• …Web-basedAdaptiveHTTPStreamingClients…

– HTML5+MSE: DASH-JS (dash.itec.aau.at), dash.js (DASH-IF,v1.1.2),YouTube• …Real-WorldEnvironments…

– DASH-JS,dash.js hostedatITEC/AAU (~10Gbit/s)– YouTubehostedatGoogledatacenters– Content:TearsofSteel @144p(250kbit/s),240p(380kbit/s),360p(740kbit/s),480p(1308kbit/s),and

720p(2300kbit/s);segment size:2s– Usersaccess contentovertheopenInternet (i.e.,real-worldenvironment)

• …Crowdsourcing– CampaignatMicroworker platform(othersalsopossible:Mechanical Turk,socialnetworks) limitedto

Europe,USA/Canada,India– ScreeningTechniques: Browserfingerprinting,stimuluspresentationtime, QoEratingsandpre-

questionnaire

July2016 ICME2016Tutorial,C.Timmerer 17B. Rainer, C. Timmerer, “Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing”, Proceedings of International Workshop on VideoNext: Design, Quality and Deployment of Adaptive Video Streaming, Sydney, Australia, Dec. 2014.

MOS andAverageBitrate

• 288microworkers,33screened(Fingerprinting: 20,presentationtime:6,QoEratingsandpre- questionnaire: 7),175maleand80female,majority(80%)isagedbetween18and37


StartupTimeandNumberofStalls


ResultsSummary• DASH-JS

– Highstart-uptime– Lownumberofstalls– Goodthroughput,QoE

• dash.js– Lowstart-uptime– High#stalls– Lowthroughput– LowQoE

• YouTube– Lowstart-uptime– Lownumberofstalls– Bestthroughput,QoE


Now,10differentadaptationlogics…


Adaptationlogicswell-known inresearchliterature

Predefinedbandwidthtrajectoryandtestsetup

Differentsegmentsizes,RTTs,HTTP/2,etc.

C. Timmerer, M. Maeiro, B. Rainer, “Which Adaption Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media Streaming Systems”, arXiv cs.MM, June 2016, http://arxiv.org/abs/1606.00341.



DASH-JSvs.bitdash


C. Timmerer, D. Weinberger, C. Mueller, and S. Lederer, “Ultra-High-Definition-Quality of Experience with MPEG-DASH”, Proceedings of the Broadcast Engineering Conference (BEC), NAB2015, Las Vegas, NV, USA, April 2015.

ObjectiveEvaluations


Stalls(lowerisbetter)AverageBitrate(higherisbetter)

Stallsarereallybad…


Conviva:ViewerExperienceReport.2014

Conclusions(1)• MPEG-DASHdefinesformatsonly

– MediaPresentationDescription (MPD)– Segment format:mp4, ts

• MPEG-DASHisnot– System,protocol,presentation,codec,interactivity,DRM,clientspecification– Otherstandardsrequired foracompleteecosystem:e.g.,DASH-IF,WAVE,

HMTL5,MSE,EME• DoweneedMPEG-DASH?(foradaptivemediastreaming)

– Notnecessarily:e.g.,WebM +VPx +manifest&controlend-to-end– Requiredtoaddressheterogeneous environments

• Roleofstandardssometimesoverrated butoftenunderestimated


Conclusions(2)• QoEforDASH-basedservices(aruleofthumb)

– Startupdelay(low[butlivevs.on-demand&shortvs.long-tailcontent])

– Bufferunderrun /stalls(zero)– Qualityswitches(low)andmediathroughput(high)– Energy- andcost-awareness(dataplan)

• NogeneralapplicableQoEmodelforDASH– (Too)manyfactorsinfluencing/featuresofQoEforDASH-based

services– Methodologyforreproducibleresearchisinplaceandwellestablished– Ampleresearchopportunities


MainQoEfactorsforDASH

� ComeupwithourownQoEfactoranddesign, conduct,analyzeasmall-scaleexperiment

QualityofSensoryExperience• Consumptionofmultimediacontentmaystimulatealsoothersenses

– Visionorhearing– Olfaction,mechanoreception, thermoception, …

• Annotationwithmetadataprovidingso-calledsensoryeffectsthatsteerappropriatedevices capableofrenderingtheseeffects


…givingher/himthesensationofbeingpartoftheparticularmulsemedia�worthwhile,informativeuserexperience

GeneralPrinciple– Outline• Generalprinciple:thereisaneedforascientificframeworktocapture,measure,quantify,judge,andexplain thequalityof(sensory)experience

• Outline– [Howtocreate,delivery,consume?]– Howtocapture andmeasure?– Howtoquantify?– Howtojudgeandexplain?


Howtocreate,delivery,consume?• SensoryEffectDescriptionLanguage(SEDL)

– Basicbuilding blockstodescribe,e.g.,light,wind, fog,vibration, scent– MPEG-VPart3,Sensory Information:Effects,GroupOfEffects– AdoptedMPEG-21DIAtoolsforadding timeinformation (synchronization)

• DescriptionconformingtoSEDL:==SensoryEffectMetadata(SEM)– Canbeassociatedtoanykindofmultimediacontent(e.g.,movies,music,Web

sites,games)– Support tobeincluded infile(MP4)andtransport(M2TS) formats

• Toolsupportforcreating(annotationtools)andconsumption(players,Webplugins)� selab.itec.aau.at

• Devices:e.g.,amBX (AmbientExperience)system+SDK,Gameskunk,Scentscape,etc.


Howtocaptureandmeasure?• Subjectivequalityassessments

– Methodology:basedonstandardmethods– Testcontent:differentgenres,manuallyannotated(cf.QUALINETDB)

• ExperimentI– Aim:Demonstratesensoryeffectsasavitaltoolforenhancing thequalityofexperience

dependingontheactualgenre• ExperimentII

– Aim:investigatetherelationshipoftheQoEtovariousvideobit-ratesofmultimediacontentsannotatedwithsensoryeffects.

– Subjectivequalitygap betweenvideoresourcesannotatedwithandwithoutsensoryeffectsatdifferentbit-rates

• [ExperimentIII]ambientlights&differentcolorcalculationsettings• ExperimentIV

– Aim:investigatetheenhancementoftheQoEandhowusers’emotionsareelicited andinfluenced byWebvideosannotatedwithandwithoutsensoryeffects


ExperimentII:Results


Sequence BabylonA.D. EarthDuration 35s 21sResolution 1280x544 1280x720Motion High LowNr.ofEffects W:7;V:9 W:8;V:1Bit-rates Kbit/s PSNR Kbit/s PSNRLowQuality 2154 38.93 2204 38.11MediumQuality 3112 41.27 3171 40.65HighQuality 4044 42.95 4116 42.27HighestQuality 6315 N/A 6701 N/A

TestSequencesMOSvs.PSNR/bit-rateforEarth.

Howtoquantify?• ExperimentV

– Aim:towardsaquality/utilitymodelforQuaSE


• Stimuliwithallcombinationsofsensoryeffects– Vibrationhigherimpactthan

light&wind– HighestQoEwithalleffects

present• GeneralQuaSE model

Howtojudgeandexplain?• ExperimentVI

– Aim:understandQuaSE• Biosensor-basedQoEevaluationsystem


J.Donley, C.Ritz,M.Shujau, "Analysing theQualityofExperienceofMultisensory MediafromMeasurementsofPhysiologicalResponses,” QoMEX2014, Singapore,Sep.2014.

Howtojudgeandexplain?• ExperimentVII

– Aim:understandQuaSE• EEGCorrelatesofPleasantandUnpleasantOdorPerception


E.Kroupi,A.Yazdani,J.-M.Vesin,T.Ebrahimi,"EEGCorrelatesofPleasantandUnpleasantOdorPerception,"ACMTOMM,vol.11,no.1s,Sep.2014.

Howtojudgeandexplain?• ExperimentVIII

– Aim:understandQuaSE• Multiple-ScentEnhancedMultimediaSynchronization


N.Murray,B.Lee,Y.Qiao,andG.-M.Muntean,"Multiple-ScentEnhancedMultimediaSynchronization,"ACMTOMM,vol.11,no.1s,Sep.2014.

Generaltemporalboundaries:-10sto+15sare“in-sync”,skewvaluesbeyondare“out-of-sync”

Conclusions• Fromtheneedforascientificframeworktocapture,measure,quantify,judge,andexplain thequalityofexperience

• To…– Howtocreate,delivery,consume?– Howtocaptureandmeasure?– Howtoquantify?– Howtojudgeandexplain?

• Openissues?July2016 ICME2016Tutorial,C.Timmerer 38

�

�

�

�Many!

OpenIssues/Challenges• QoEassessmentisadelicatemixtureofingredientsandchoices

– Test&labenvironment– Testcontent– Testmethodology– Dataanalysis

• (Semi-)Automaticcontentcreation/annotation• Towardslargescaledeployment

– Lessonslearntfrom3D(disaster)– 4D,5D,xD – addinganotherdimensiondoesnotguaranteesuccess

• Holisticapproachnotfeasible– Needformuchmorespecialized QuaSEmodels

• QUALINETTaskForce:"ImmersiveMediaExperiences(IMEx)”– https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:imex


AssessingQualityofExperience…ABitLikeMeasuring‘Happiness’…

© F. Pereira, Instituto Superior Técnico, Univ. Lisboa, PortugalJuly2016 ICME2016Tutorial,C.Timmerer 40

3. TowardstheConceptofQualityofLife

A. BeyondQualityofExperience

B. Theageofwearables

C. IngredientsofamodernassessmentofQualityofLife

Touradj Ebrahimi:AboutMe…

• ProfessorofmultimediasignalprocessingatEPFL

• Activeinimage/videocompression,mediainterpretation

(segmentation, annotation, search,retrieval,qualityassessment,

braincomputerinterface,affectivecomputing, etc.)andmedia

security(privacyprotection,copyrightprotection,mediaintegrity

verification,etc.)

• MemberofMPEGstandardizationcommitteesince1992andactive

inmanyofitsvideostandardizationactivities:MPEG-4,H.264/AVC,

MVC,H.265/HEVC,MV-HEVC,3D-HEVC,SCC,HDRextensions.

• MemberofJPEGstandardizationcommitteesince1994andactivein

manyof itsimagestandardizationactivities:JPEG2000,JPSearch,

JPEGXR,JPEGAIC,JPEGXT,JPEGXS,JPEGPLENO.

• MemberoftheSteeringCommitteeofQoMEX andchairofitsfirst

edition in2009

• ConvenerofJPEGStandardizationCommitteesince2014

• ChairofCOSTActionIC1003Qualinet

• FirstcoinedinFebruary2001thetermQualityofExperience(QoE)as

auser-centricalternativetoQualityofService(QoS)

A.Beyond Quality ofExperience

WhatfutureforQualityofExperience

• Prediction isverydifficult, especially about thefuture

→ Niels Bohr (1885-1962): PhysicsNobelPrizeWinner1922

Trendsinuser-centricmultimedia

• Consequences ofMoore’s law

- Betterandrichercontent

- Largerbandwidthnetworks

- Biggerstoragecapacities

- Moresophisticatedcodecs/processing

• Betterintegration

- Art,design

- Psychology,psychophysics,neuroscience

- Sociology,humanities

- Technologyandengineering

Trendsinuser-centricexperiences

• Newmediaexperiences

• Personalwell-beingandpersonalhealth

• Bigdataandsocialmedia

Newmediaexperiences

• UHD,HDR,HFR,3D,…

• Lightfieldimaging

• Integralimaging

• Holographicimaging

• Haptics

• Virtual,Augmented,Mixedreality

• Immersivemedia

• Multi-sensorymedia

• …

Majortrendsinmultimedia

NewMediaWearables

InternetofThings

BigdataSocialmedia

ContentSensor

Processing

Multimediaexperiences

Majortrendsinmultimedia

NewMediaWearables

InternetofThings

BigdataSocialmedia

ContentSensor

Processing

Multimediaexperiences

Life

B.The ageofwearables

Generation-0(smart)wearables

Mobilephonesaswearables

Smartwatchesaswearables

Personalwell-beingwearables

Notanywearableshouldbesmart!

Smartwatchesversuswatches

Otherwearables forusersensing

Otherwearables forusersensing

Otherwearables forenvironmentsensing

Othervariantsofsmartglasses

Sony’srecentlyannouncedsmartglasses

MicrosoftHololens



Wearabledataanalysis

C.Ingredients of amodern assessmentof Quality of Life

QualityofLife

• Qualityoflife(QoL)isthegeneralwell-being

ofindividuals andsocieties.QoL hasawide

rangeofcontexts,includingthefieldsof

internationaldevelopment,healthcare,

politicsandemployment.

Whenpastbecomesfuture

• QualityofLife:Meaning,Measurement,andModels– ElyseW.Kerce,NavyPersonnelResearchandDevelopmentCenter

(U.S.)

– NavyPersonnelResearchandDevelopmentCenter,1992

• Originsoftheconceptdatebackto1725!

– FrancisHutcheson

MeasuringQualityofLife

IngredientsofamodernQoL assessment

• Usersensing

• Environmentsensing

• Contextextraction

• Bigdataanalytics

Users

Context

Content

Datais theKing!

• (Alotof)Datafromusers

andtheirenvironmentsis

neededtocarryout

researchonwearables:

– Publiclyavailable

– Reliable

– Richdata

– Generatedviacrowdsourcing

Bigdataandsocialmedia

1billionmonthlyactiveusers 5million

photos added every day

175ktweets posted every second

Devices andsensors areQueens!

• (Easilyavailable)Wearable

devicesandsensorsare

neededtogeneratethe

data:

– Affordablecomponentsand

sensorstobepurchasedby

interestedindividuals

– Reliable

– Easytoconfigureand

calibrate

– Userfriendly

SoftwareisthePresident!

• (Efficient)Software

tocontrolwearables

andsensors

– Opensource

– Extensible

– Reliable

– Easytoinstallonawide

varietyofplatforms

Datamanagementisamust!

• (Distributed)server

architecturefordata

synchronization,storage

andaccess:

– Cloudbased

– Reliable

– Scalable

– RespectfulofPrivacyandEthical

issues

Interoperabilityisessential!

• (Open)Standardsolutions

– Standardcomponents

– Standarddatasyntax

– StandardInterface

– Compliance/Certification

Compellingusecasesareimportant!

• Concreteusecases

– Dietaryassessment

– Lifelog

– …

Internationalconsortiumisneeded!

• AseedconsortiumfromUS,JapanandSwiss

universitiesinplaceandhasinitiatedworkon

thistopicaroundMultimediaDietary

AssessmentUsecase

ConsumerHealthApplications

• Mobiledeviceasadatacollectiontoolfordietaryassessment

• Increasingdemandinapplicationsformobiledevices

ConsumerHealthApplications

• Manualintakedataentry

– Tap&Track

• Barcodebasedintakeentry

– Fooducate

• Imagebasedintakeentry

– MealSnap

Wheredopeoplelookwhentheyeat?

Needtocoverawideangle

Didyoufinishtheplate?

Needtoconsidertemporalaspects

Before After

360/omnidirectionalvideocamera

Creationofanewdatabase

+…

Foodrecognition

Food

Deeplearning

Omnidirectional

image

Swallowingdetection

UseEMGtodetectswallowingandenable/disablecamera

• The sound of mastication (food crushing) has relation to physical properties of the food, but little relevance to energy content

Chewing sensors

O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005.S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012.

• The sound of mastication (food crushing) has relation to physical properties of the food, but little relevance to energy content

Chewing sensors

O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005.S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012.

Integrateothercontextualaspects

Time

Place

Context

Userpersonality

MeasuringQoL today…

MeasuringQoL tomorrow…

Take-HomeMessages

• QoL isthenaturalstepbeyondQoE !

• QoL notanewconceptbutitcantakeadvantageof

moderntechnologies.

• Afederatingprojectisneededinordertocreatethe

necessarycriticalmass(especiallyindata).

• MultimediaDietaryAssessmentisacompellinguse

case.

ForSomeoftheSlides...