CS6640 Computational Photography - Cornell CS6640 Computational Photography 7. Digital camera processing

  • View
    0

  • Download
    0

Embed Size (px)

Text of CS6640 Computational Photography - Cornell CS6640 Computational Photography 7. Digital camera...

  • © 2012 Steve Marschner

    CS6640 Computational Photography

    7. Digital camera processing pipeline

    1

  • Cornell CS6640 Fall 2012

    Camera processing pipeline • read image out from sensor —see Sensors lecture • optional: HDR assembly —see Homework 2 • color balance —see Color lecture • demosaic • noise processing • color matrix —see Color lecture • tone map

    2

  • Cornell CS6640 Fall 2012

    Demosaicking • First question: how to spell it

    I cite “picnicking” • Each photosite

    senses only one color

    • We need three measurements at each pixel

    3 http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html

    http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html

  • Cornell CS6640 Fall 2012

    Bayer array • Simple solution: make a half-resolution image

    • Want sensor-resolution image? Make up 2/3 of the data!

    4

    ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 5

  • Cornell CS6640 Fall 2012 6

  • Cornell CS6640 Fall 2012 6

    bayer

  • Cornell CS6640 Fall 2012 6

    bayer bayer

  • Cornell CS6640 Fall 2012 7

  • Cornell CS6640 Fall 2012 8

  • Cornell CS6640 Fall 2012 8

    bayer

  • Cornell CS6640 Fall 2012 8

    bayer bayer

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic • Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this?

    9

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic • Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this? 1. throws away too much resolution

    9

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic • Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this? 1. throws away too much resolution 2. produces subpixel shifts in color planes!

    9

  • Cornell CS6640 Fall 2012 10

  • Cornell CS6640 Fall 2012 10

    bayer

  • Cornell CS6640 Fall 2012 10

    bayerblock

  • Cornell CS6640 Fall 2012 10

    bayerblock bayer

  • Cornell CS6640 Fall 2012 10

    bayerblock bayerblock

  • Cornell CS6640 Fall 2012 11

  • Cornell CS6640 Fall 2012 11

    bayer

  • Cornell CS6640 Fall 2012 11

    bayerblock

  • Cornell CS6640 Fall 2012 11

    bayerblock bayer

  • Cornell CS6640 Fall 2012 11

    bayerblock bayerblock

  • Cornell CS6640 Fall 2012

    Centered half-resolution • Average pixels in groups that all have the same

    “center of gravity” avoids major color fringing

    12

    ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    ?

    ? ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 13

  • Cornell CS6640 Fall 2012 13

    bayer

  • Cornell CS6640 Fall 2012 13

    bayerblock

  • Cornell CS6640 Fall 2012 13

    bayerblockcentered

  • Cornell CS6640 Fall 2012 13

    bayerblockcentered bayer

  • Cornell CS6640 Fall 2012 13

    bayerblockcentered bayerblock

  • Cornell CS6640 Fall 2012 13

    bayerblockcentered bayerblockcentered

  • Cornell CS6640 Fall 2012 14

    naïve full-resdcraw naïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayernaïve full-resdcraw naïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblocknaïve full-resdcraw naïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcraw naïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcraw bayernaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcraw bayerblocknaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcraw bayerblockcenterednaïve full-resdcraw

  • Cornell CS6640 Fall 2012

    Naïve full-resolution interpolation • What if we don’t want to throw away so much sharpness?

    15

    ? ? ? ? ?? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ?

    ? ?

    ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    ? ?

    ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

    ? ? ? ? ? ?

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 16

    dcraw dcraw

  • Cornell CS6640 Fall 2012 16

    bayerdcraw dcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockdcraw dcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcentereddcraw dcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcraw dcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcraw bayerdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcraw bayerblockdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcraw bayerblockcentereddcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcraw bayerblockcenterednaïve full-resdcraw

  • Cornell CS6640 Fall 2012 17

  • Cornell CS6640 Fall 2012 17

    bayer

  • Cornell CS6640 Fall 2012 17

    bayerblock

  • Cornell CS6640 Fall 2012 17

    bayerblockcentered

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res bayer

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res bayerblock

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res bayerblockcentered

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res bayerblockcenterednaïve full-res

  • slide by Frédo D urand, M

    IT

    Results of simple linear

  • slide by Frédo D urand, M

    IT

    Results - not perfect

  • slide by Frédo D urand, M

    IT

    Questions?

  • slide by Frédo D urand, M

    IT

    The problem • Imagine a black-on-white corner • Let’s focus on the green channel for now

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

  • slide by Frédo D urand, M

    IT

    The problem • Imagine a black-on-white corner

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 1 0 0 1

    0 0 1 0 0 1

    1 1 1 1 1 1

  • slide by Frédo D urand, M

    IT

    The problem • Imagine a black-on-white corner

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 1 0 0 1

    0 0 1 0 0 1

    1 1 1 1 1 1

    0 0

    0

    1

    .25 .75

    .25

    .75 1

    0

    .25

    1

    1

    .1

    1

    0 0 1 0 0 1

    0 0 1 0 0 1

    1 1 1 1 1 1

    .25 .75 .75

  • slide by Frédo D urand, M

    IT

    The problem • Imagine a black-on-white corner

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ? ?

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 1 0 0 1

    0 0 1 0 0 1

    1 1 1 1 1 1

    0 0

    0

    1

    .25 .75

    .25

    .75 1

    0

    .25

    1

    1

    .1

    1

    0 0 1 0 0 1

    0 0 1 0 0 1

    1 1 1 1 1 1

    .25 .75 .75

  • slide by Frédo D urand, M

    IT

    Yep, that’s what we saw

  • slide by Frédo D urand, M

    IT

    Green channel

  • slide by Frédo D urand, M

    IT

    Edge-based Demosaicking

  • slide by Frédo D urand, M

    IT

    Idea • Take into account structure