JPEG Innovations Workshop

Daniel LeeDaniel Leeconvener@jpeg.orgconvener@jpeg.org

ConvenerISO/IEC JTC1/SC29/WG1“Coding of Still Pictures”

http://www.jpeg.org

• JPEG family of Imaging Standards– JPEG, JPEG 2000, JPEG XR, JPSearch, AIC

• Impact of JPEG to the industry– Digital Still Cameras, mobile devices, smart phones

– Commoditization of JPEG chips

• Impact of JPEG to the users community– Social networks – New social phenomenon

OutlineOutline

– Social networks – New social phenomenon

– Privacy protection – How standardization will help

– Finding your photos – JPSearch, Visual Search to the rescue

• JPEG Innovations– 3D Photography

– Social Media, Web 3.0

– Advanced Image Coding

• Success factors of Standardization– importance of timing, industry participation

JTC 1IEC ISO

International Organization for StandardizationInternational Organization for Standardization

SC29

WG 1

JPEG

JBIG

WG 11

MPEG

JPEGJPEG((Joint Photographic Experts GroupJoint Photographic Experts Group))

Joint ISO/IEC and ITU-T

Image Compression and JPEG Standards

• Technology and JPEG Family of Standards– 1948 Information Theory (Shannon)

– 1951 Huffman Code (Huffman)

– 1951 DPCM technology (Shannon)

– 1974 DCT technology (Ahmed)

– 1976 Arithmetic encoding technology (Rissanen, Pasco)

– 1980 Group 3 & 4 bi-level compression published (ITU-T Fax standards)

– 1984 The building blocks of wavelet transform technology (Mintzer)

– 1986 IBM Q Coder, Mitsubishi M Coder

– 1992 ISO JPEG standard published (ISO 10918)

– 1994 ISO JBIG standard published (ISO 11544)

– 1995 CREW – Compression with Reversible Embedded Wavelets (Zandi,Allen, Schwartz, Boliek)

– 1998 ISO JPEG – LS published (ISO 14495)

– 1998 EBCOT – Embedded Block Coding with Optimal Truncation (Taubman)

– 2000 ISO JBIG2 published (ISO 14492)

– 2000 ISO JPEG 2000 Part 1 published (ISO 15444)

JPEG 2000 Family of StandardsJPEG 2000 Family of Standards

• JPEG 2000 Image Coding System (ISO/IEC 15444)– Part 1, Core coding system

– Part 2, Extensions (includes multiple component compression)

– Part 3, Motion JPEG 2000

– Part 4, Conformance Testing

– Part 5, Reference software (Java and C implementations)

– Part 6, Compound image file format (document imaging)– Part 6, Compound image file format (document imaging)

– Part 7, not used

– Part 8, JPSEC: Secure JEPG 2000

– Part 9, JPIP: Interactivity tools, APIs and protocols

– Part 10, JP3D: Extensions for three dimensional data

– Part 11, JPWL: Wireless

– Part 12, ISO Media file format

– Part 13, Entry level encoder

– Part 14, JPXML: Structural representation and reference

JPSearchJPSearch, JPEG XR, AIC, JPEG XR, AIC

• JPSearch (ISO/IEC 24800)– Part 1, System Framework and Components

– Part 2, Registration, Identification and Management of Schema andOntology

– Part 3, JPSearch Query Format

– Part 4, File Format for Metadata Embedded in Image Data (JPEG and JPEG 2000)

– Part 5, Data Interchange Format Between Image Repositories

– Part 6, Reference Software– Part 6, Reference Software

• JPEG XR (Extended Range) (ISO/IEC 29199)– Part 1, System Architecture

– Part 2, Image Coding Specification

– Part 3, Motion JPEG XR

– Part 4, Conformance Testing

– Part 5, Reference software

• AIC - Advanced Image Coding and Evaluation Methodologies (ISO/IEC 29170)

JPEG 2000 SuccessJPEG 2000 Success

• Digital CinemaDigital CinemaDigital CinemaDigital Cinema

• Satellite ImagingSatellite ImagingSatellite ImagingSatellite Imaging

• GIS/Remote sensingGIS/Remote sensingGIS/Remote sensingGIS/Remote sensing• GIS/Remote sensingGIS/Remote sensingGIS/Remote sensingGIS/Remote sensing

• Digital Culture ImagingDigital Culture ImagingDigital Culture ImagingDigital Culture Imaging

• Archive applicationsArchive applicationsArchive applicationsArchive applications

• Medical ImagingMedical ImagingMedical ImagingMedical Imaging

Digital CinemaDigital Cinema

� JPEG 2000 is in use for all Digital Cinema movie distributions worldwide (for 2D and 3D movies!)

� The distribution form is a Digital Cinema Package (DCP) – JPEG 2000 code streams are wrapped with the MXF file format

� 2010 – 36,242 Digital Cinema screens deployed worldwide, 60,5% 3D screens*screens*

� 2015 – forecast 100,000 Digital Cinema screens*

� Digital Cinema Packages are using the Digital Cinema Profiles of JPEG 2000

� Many companies offering JPEG 2000 solutions: Analog Devices, Doremi, Dolby, Sony, Fraunhofer, …

Digital CinemaDigital Cinema

•• Standards OrganizationsStandards Organizations

•• Data ProvidersData Providers

•• Data Users/SoftwareData Users/Software

• Standards AdoptedStandards Adopted

– JPEG 2000 Part 1 Baseline

– Digital Cinema Profile

JPEG JPEG -- Impact to the industryImpact to the industry

• Commoditization of JPEG chips Commoditization of JPEG chips Commoditization of JPEG chips Commoditization of JPEG chips

– Moore’s law,Moore’s law,Moore’s law,Moore’s law,

– JPEG IP coreJPEG IP coreJPEG IP coreJPEG IP core

JPEG IP CoreJPEG IP Core

JPEG JPEG -- Impact to the industryImpact to the industry

• Commoditization of JPEG chips Commoditization of JPEG chips Commoditization of JPEG chips Commoditization of JPEG chips

– Moore’s law,Moore’s law,Moore’s law,Moore’s law,

– JPEG IP coreJPEG IP coreJPEG IP coreJPEG IP core

• Digital Still CamerasDigital Still CamerasDigital Still CamerasDigital Still Cameras

– Higher pixel counts, more image processing functionsHigher pixel counts, more image processing functionsHigher pixel counts, more image processing functionsHigher pixel counts, more image processing functions

– High dynamic range photography High dynamic range photography High dynamic range photography High dynamic range photography –––– JPEG Innovations can helpJPEG Innovations can helpJPEG Innovations can helpJPEG Innovations can help– High dynamic range photography High dynamic range photography High dynamic range photography High dynamic range photography –––– JPEG Innovations can helpJPEG Innovations can helpJPEG Innovations can helpJPEG Innovations can help

– Raw image format, CFA format Raw image format, CFA format Raw image format, CFA format Raw image format, CFA format –––– AIC can helpAIC can helpAIC can helpAIC can help

• Photo printingPhoto printingPhoto printingPhoto printing

– Home printing, store printing, kiosk printing …Home printing, store printing, kiosk printing …Home printing, store printing, kiosk printing …Home printing, store printing, kiosk printing …

– Printing of JPEG images Printing of JPEG images Printing of JPEG images Printing of JPEG images –––– 10918109181091810918----6 (proposed) can help6 (proposed) can help6 (proposed) can help6 (proposed) can help

• Mobile devicesMobile devicesMobile devicesMobile devices

– Mobile phones, Mobile phones, Mobile phones, Mobile phones, iPhonesiPhonesiPhonesiPhones, Android phones, . . ., Android phones, . . ., Android phones, . . ., Android phones, . . .

– New applications created everydayNew applications created everydayNew applications created everydayNew applications created everyday

Photo Printing ServicesPhoto Printing Services

JPEG JPEG -- Impact to the users communityImpact to the users community

• Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks –––– new social phenomenon new social phenomenon new social phenomenon new social phenomenon

– Photos sharing Photos sharing Photos sharing Photos sharing ---- #1 apps in social networks #1 apps in social networks #1 apps in social networks #1 apps in social networks –––– FacebookFacebookFacebookFacebook, Twitter, , Twitter, , Twitter, , Twitter, FlickrFlickrFlickrFlickr, …, …, …, …

– >100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day

– New applications created everydayNew applications created everydayNew applications created everydayNew applications created everyday

– InstagramInstagramInstagramInstagram ---- 1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months

FacebookFacebook

InstagramInstagram

JPEG JPEG -- Impact to the users communityImpact to the users community

• Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks –––– new social phenomenon new social phenomenon new social phenomenon new social phenomenon

– Photos sharing Photos sharing Photos sharing Photos sharing ---- #1 apps in social networks #1 apps in social networks #1 apps in social networks #1 apps in social networks –––– FacebookFacebookFacebookFacebook, Twitter, , Twitter, , Twitter, , Twitter, FlickrFlickrFlickrFlickr, …, …, …, …

– >100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day

– New applications created everydayNew applications created everydayNew applications created everydayNew applications created everyday

– InstagramInstagramInstagramInstagram ---- 1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months

• Privacy protectionPrivacy protectionPrivacy protectionPrivacy protection

– Image authenticity , unauthorized modification Image authenticity , unauthorized modification Image authenticity , unauthorized modification Image authenticity , unauthorized modification –––– JPSecJPSecJPSecJPSec can helpcan helpcan helpcan help

– Metadata protection, location, author information Metadata protection, location, author information Metadata protection, location, author information Metadata protection, location, author information –––– JPSearchJPSearchJPSearchJPSearch can helpcan helpcan helpcan help

Privacy ProtectionPrivacy Protection

JPEG JPEG -- Impact to the users communityImpact to the users community

• Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks Photos sharing in social networks –––– new social phenomenon new social phenomenon new social phenomenon new social phenomenon

– Photos sharing Photos sharing Photos sharing Photos sharing ---- #1 apps in social networks #1 apps in social networks #1 apps in social networks #1 apps in social networks –––– FacebookFacebookFacebookFacebook, , , , FlickrFlickrFlickrFlickr, …, …, …, …

– >100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day>100 millions of photos being shared each day

– New applications created everydayNew applications created everydayNew applications created everydayNew applications created everyday

– InstagramInstagramInstagramInstagram ---- 1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months1.7 million users in 4 months

• Privacy protectionPrivacy protectionPrivacy protectionPrivacy protection

– Image authenticity , unauthorized modification Image authenticity , unauthorized modification Image authenticity , unauthorized modification Image authenticity , unauthorized modification –––– JPSecJPSecJPSecJPSec can helpcan helpcan helpcan help

– Metadata protection, location, author information Metadata protection, location, author information Metadata protection, location, author information Metadata protection, location, author information –––– JPSearchJPSearchJPSearchJPSearch can helpcan helpcan helpcan help

• Finding your photosFinding your photosFinding your photosFinding your photos

– Films in a shoebox becomes digital photos in a shoebox Films in a shoebox becomes digital photos in a shoebox Films in a shoebox becomes digital photos in a shoebox Films in a shoebox becomes digital photos in a shoebox –––– how do you find how do you find how do you find how do you find

your photosyour photosyour photosyour photos

– JPSearchJPSearchJPSearchJPSearch –––– standardize on image query, metadata interoperabilitystandardize on image query, metadata interoperabilitystandardize on image query, metadata interoperabilitystandardize on image query, metadata interoperability

– Visual Search Visual Search Visual Search Visual Search –––– indexing images using image featuresindexing images using image featuresindexing images using image featuresindexing images using image features

– eCommerceeCommerceeCommerceeCommerce –––– finding the things you wantfinding the things you wantfinding the things you wantfinding the things you want

Image SearchImage Search

JPEG InnovationsJPEG Innovations

• 3D photography3D photography3D photography3D photography

– Photorealism codingPhotorealism codingPhotorealism codingPhotorealism coding

– MultiMultiMultiMulti----view image codingview image codingview image codingview image coding

– Stereo image codingStereo image codingStereo image codingStereo image coding

– Light field image codingLight field image codingLight field image codingLight field image coding

• Advanced Image CodingAdvanced Image CodingAdvanced Image CodingAdvanced Image Coding• Advanced Image CodingAdvanced Image CodingAdvanced Image CodingAdvanced Image Coding

– New application domains New application domains New application domains New application domains ---- HDRHDRHDRHDR

– CFA image codingCFA image codingCFA image codingCFA image coding

– Surveillance , Forensic applicationsSurveillance , Forensic applicationsSurveillance , Forensic applicationsSurveillance , Forensic applications

• Social Media, Web 3.0Social Media, Web 3.0Social Media, Web 3.0Social Media, Web 3.0

– Metadata interoperabilityMetadata interoperabilityMetadata interoperabilityMetadata interoperability

– Image searchImage searchImage searchImage search

– Visual searchVisual searchVisual searchVisual search

• Formal standards vs industry standard

– Interoperability

– Product life cycle management

• Standardization process

– ISO Consensus approach

– Avoid anti-competitive behavior

Importance of Standardization ProcessImportance of Standardization Process

• When to develop a standard

– Too early – non-realistic requirements

– Too late – lose market window

• Value of standards and standards participation

– Innovations

– Company/Institution credibility

– Professional/personal relationships

• Market requirements

• Understand how technology scales

• Time to market – keep project on schedule

• Keep members of committee interested

Manage consensus vs simple voting

Success FactorsSuccess Factors

• Manage consensus vs simple voting

• Importance of IPR issue – royalty free standard key to pervasive adoption

• Importance of reference software, compliance testing

• PR: seminars, workshops, tutorials, books

• Developers/users community

• Industry active participation

In Summary…In Summary…

• JPEG Family of Standards – a triumph of innovations, teamwork and good execution

• Social networks drive exponential growth in digital photos – photos sharing becomes a new social phenomenon

• How to find your photos – Image search is still a challenge• How to find your photos – Image search is still a challenge

• 3D photography – exciting new realism

• Success of standardization – active industry participation

• Over 100 experts from 17 national bodies, over 60 organizations

• Adobe, Aerospace, AlgoVision, Analog Devices, Apple, Aware, Booz Allen, Canon, Dolby Labs, eBay, Elysium, Ericsson, Fraunhofer, Fujifilm, HP, IBM, Image Power, ITT, Kodak, Los Alamos, LuraTech, Microsoft, Mitre, Mitsubishi, Motorola, NEC, NetImage, Nokia, Oki, Pegasus, Philips, Picture Element, Ricoh, SACD, SAIC, Sharp, Sony, Thales, TI, Xilinx, Yahoo!, Xerox, Zoran,

AcknowledgmentsAcknowledgments

SACD, SAIC, Sharp, Sony, Thales, TI, Xilinx, Yahoo!, Xerox, Zoran, …

• Univ. of New South Wales, Univ. of Arizona, EPFL, UBC, RPI, Tokyo Instit. Polytechnics, Vrije Univ. Brussel, University of Poitiers, University of Stuttgart . . .

• National Bodies: Australia, Belgium, China, Denmark, Finland, France, Germany, Israel, India, Italy, Japan, Korea, Portugal, Russian, Singapore, Spain, Sweden, Switzerland, UK, US

Thank you!