Copyright © Magnum Semiconductor, Unpublished Introduction to Deinterlacing by Mark Korhonen

Copyright © Magnum Semiconductor, UnpublishedCopyright © Magnum Semiconductor, Unpublished

Introduction to Deinterlacing

by Mark Korhonen

Copyright © Magnum Semiconductor, Unpublished

2

Overview:Introduction To Deinterlacing

Background• Progressive video• Interlaced video• Deinterlacing

Basic Deinterlacing Algorithms• Weave• Vertical Interpolation (aka Bob)

Advanced Deinterlacing Algorithms• Diagonal Interpolation• Cadence Detection• Motion Adaptive Deinterlacing (MADI)• Motion Compensated Deinterlacing (MCDI)

Deinterlacing Applications

Picture source: http://lurkertech.com/lg/fields/Copyright © Magnum Semiconductor, Unpublished

3

Background: Progressive Video

A complete video frame is displayed at regular intervals

View on computers and digital televisions

Example resolutions and frame rates:• SD: 720x480p30, 720x576p25• HD: 1280x720p60, 1920x1080p30

Sources: film (movies), animation, progressive cameras

Example: 30p = 30 frames/s

Picture source: http://lurkertech.com/lg/fields/Copyright © Magnum Semiconductor, Unpublished

4

Background: Interlaced Video

Frames of video are sampled at two time intervals• Even rows sampled at: 2t• Odd rows sampled at: 2t + 1

Terminology:• Top field: even rows in the frame• Bottom field: odd rows in the frame• Field polarity: indicates if a field is a top field or bot. field

Example: 60i = 60 fields/s


5

Background: Interlaced Video

Why interlaced video exists• Double frame rate w/ same bandwidth = smoother motion• Original television formats are all interlaced

– NTSC=720x480i60– PAL=720x576i50

• 1920x1080i60 exists because– Processing requirements for 1920x1080p60 are very high– Motion is smoother than 1920x1080p30– More detail than 1280x720p60


6

Background: Deinterlacing

Deinterlacing: convert interlaced video to progressive• Input: X field/s (e.g. i60)• Output: X frames/s (e.g p60)• This process generates X “missing” fields/s


7

Background: Deinterlacing

Fields typically used to generate the missing field• The current field is the opposite polarity of the field that

needs to be generated, but displayed at the same time– E.g. output = bottom field, curr = top field

• The previous and next fields are the same polarity as the field that needs to be generated

– E.g. output = bottom field, prev & next = bottom fields

Picture source: doom9.orgCopyright © Magnum Semiconductor, Unpublished

8

Basic Deinterlacing: Weave

Weave: use “prev” or “next” as the missing field• For still images and progressive content, no missing data• Very severe artifacts for moving interlaced content


9

Basic Deinterlacing: Weave

Extension• If stationary, average the previous and next fields

together as the missing field– Noise reduction (averaging two identical images reduces noise)– Handles fades better if luminance changing every field

Output[x,y] = m prev[x,y] + (1-m) next[x,y]• m is typically 0.0, 0.5 or 1.0


10

Basic Deinterlacing: Bob

Bob (aka vertical interpolation)• Generate the opposite polarity by interpolating along

vertical columns in the current field

Implementations• Line doubling – just use current field

– e.g. Output[x,y] = curr[x,y]

• 2-tap FIR filter – average the line above and below– e.g. Output[x,y] = 0.5 curr[x,y] + 0.5 curr[x,y+1]

• 8-tap FIR filter – better frequency response


11

Basic Deinterlacing: Bob

Bob Video Quality• Looks good for most video content

– Still content without a lot of detail– Moving content (hard to see detail on moving objects)

• Interpolation fails if a lot of detail – causes flickering

Picture source: Computer Desktop EncyclopediaCopyright © Magnum Semiconductor, Unpublished

12

Advanced Deinterlacing:Diagonal Interpolation

Problem: Bob isn’t ideal for diagonal edges

Solution: Apply FIR filter along a diagonal


13

Advanced Deinterlacing:Diagonal Interpolation

Diagonal Edge Detection Algorithm• Try a bunch of different angles and see which is best

– Pattern recognition problem: classify edges as a supported angle• Determine which angle is most correlated at each pel

• We can assume that the angle of the edge is wide

» Allows using neighbouring pels to improve accuracy

Picture source: http://www.infocellar.com/televisionCopyright © Magnum Semiconductor, Unpublished

14

Advanced Deinterlacing:Cadence Detection

Observation: If interlaced video was generated from a progressive source, we can safely weave it with no artifacts

Sample cadences


15


Detection Area• Size of cadence area can be variable

– e.g. entire field, 16x16 block, every pel

• Detecting entire fields sufficient for ~99% of video• Smaller areas only needed if video is a mixed source

– moving interlaced text over progressive video• e.g. weather warnings on a TV movie

– different frame rates of progressive video edited together• e.g. source of border was 30p, but source of contents were 24p

Picture source: http://www.infocellar.com/televisionCopyright © Magnum Semiconductor, Unpublished

16


Cadence Algorithm 1: detect regular repeated fields• For 3:2, every 5th field is a repeat

– SAD will be very low every 5th field called Inverse Telecine

• Very robust for a lot of content• Doesn't work for 2:2 (no repeats), or changing cadences

(e.g. slow-motion replays, edited video)


17


Cadence Algorithm 2: detect weaving artifacts• 1) Weave curr with prev, 2) Weave curr with next• Determine if weaving artifacts are less in a field

– Pattern recognition problem: classify into three states• Progressive – weave with previous

• Progressive – weave with next

• Interlaced – perform other deinterlacing


18


Cadence Algorithm 2: detect weaving artifacts• Tricky for stationary content and at scene changes

– Fairly easy to compensate

• Tricky for vertical motion of detailed video– e.g. pan of venetian blinds

• Can be tricky for video that looks like it already contains weaving artifacts

– Artifacts introduced by video compression can look like weaving artifacts

– Certain textures


19

Advanced Deinterlacing:Motion Adaptive Deinterlacing (MADI)

Idea: weave stationary areas, interpolate moving areas

Stationary detection = pattern recognition problem• Size of stationary area can vary (e.g. entire field, 16x16

block, every pel)– Smaller stationary areas = less flickering but more computation

• Possible implementations:– SAD of prev and next

• Watch out for periodic motion

– min SAD of last X fields• Watch out for how long it takes stationary regions to be detected


20

Advanced Deinterlacing:Motion Adaptive Deinterlacing (MADI)

Basic algorithm:• Inputs:

– Weave – generated from previous and/or next– Inter – generated from current (diagonal interpolation)

• Output[x,y] = (K) Weave[x,y] + (1-K) Inter[x,y]– K ~= 1 for stationary/progressive areas (weave)– K ~= 0 for moving areas (interpolation)

NOTES:• Fades may need special handling

– e.g. bias more towards interpolation

• x and y advance through regions in the video– Could be every pel, 16x16 blocks, or the entire field


21

Advanced Deinterlacing:Motion Compensated Deinterlacing (MCDI)

Idea:• If we can’t weave a particular region, could we weave a

motion compensated version of this region from curr, prev or next?

Comments:• Potential for preserving all of the detail• Very difficult to do correctly

– Especially rotations, zooms, morphing, lighting changes, etc.– Objects can get covered/uncovered– Even small errors can look very bad

• weaving artifacts, ghosting around edges, weird motion, etc.

• Motion compensation should be done for every pel– Very computationally demanding


22

Advanced Deinterlacing:Motion Compensated Deinterlacing (MCDI)

Basic Algorithm:• Input fields

– MC_curr[x,y] = curr[x + dx_c, y + dy_c]– MC_prev[x,y] = prev[x + dx_p, y + dy_p]– MC_next[x,y] = next[x + dx_n, y + dy_n]– MADI[x,y] = pel generated using motion adaptive deinterlacing

• Output:

• p + q + r ~= 1 if high confidence in motion compensation

• p + q + r ~= 0 if low confidence in motion compensation

– Confidence in the motion compensation can be based on• The smoothness of the motion vector field

• Estimated SAD (true SAD unknown because the field is missing)

• Consistency of motion from field to field

yxMADIrqpyxnextMCryxcurrMCqyxprevMCpyxMCDI ,)1(,_,_,_,


23


Frame rate conversion from interlaced to interlaced• No deinterlacing = motion judder

– Need to drop a pair of fields – increases motion judder

• Deinterlacing = smoother motion– Requires excellent quality deinterlacing

• Deinterlaced fields will be deinterlaced twice


24


Upscaling interlaced video requires deinterlacing• Perserves more detail• Minimizes aliasing


25


If the output frame rate is low, flicker can be severe

• If input frame rate is high enough– In the output, pick a polarity that only use fields from the input– The other polarity will always be generated (e.g. via interpolation)

• This works because flicker is caused by alternating rows between a) original and b) interpolated – we have removed the alternating


26

Appendices


27

Video Resolutions

Common video resolutions• SD: 720x480 (NTSC), 720x576 (PAL)• HD: 1280x720, 1920x1080• Sub SD: VGA, QVGA, CIF, QCIF, etc.

Definitions• pels = 1x1 area of video data• pixel = 1x1 area on video display• E.g. 720x480 video is 720 pels wide, but may be

displayed on a television with 1920 pixels


28

Color Formats

RGB vs. YUV• Y = greyscale, making deinterlacing decisions primarily

based on Y is simpler (but not as accurate)• Data in U and V isn't as critical so typically downsampled

– 4:2:0 vs 4:2:2 vs 4:4:4

Documents

Copyright © Magnum Semiconductor, Unpublished Introduction to Deinterlacing by Mark Korhonen