LightSync: Unsynchronized Visual Communication over
Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu
Slide 2
2D barcodes everywhere
Slide 3
http://www.windowsphone.com/zh-cn (Linking to online
information)
Slide 4
A video of barcodes instead Transmitting information (vs
linking) E.g., information on Gibraltarian attractions E.g., movie
show times for nearby theaters Receiver Transmitter
Slide 5
Visual communications on phones No need for additional
connectivity Free, directional, lightweight links Optical
software-defined radio Run as apps on commodity devices
Slide 6
Many challenges Frame synchronization Poor image quality
Limited computation power on phones Aesthetics
Slide 7
In the rest of the talk Frame synchronization challenges
LightSync design Implementation and evaluation
Slide 8
FRAME SYNCHRONIZATION CHALLENGES
Slide 9
Frame rate mismatch Screen side (TX) 30 fps video playback
Fairly stable Camera side (RX) 15 30 fps in practice Variable
Lighting conditions Device capability API/system factors High rate,
steady Low rate, unsteady
Slide 10
Original frame Received frame patterns
Slide 11
Original frame Single frame 2-frame mix Mixing pattern varies
by line Received frame patterns
Slide 12
2-frame mix Mixing pattern (& #components) varies by line
Due to Rolling Shutter CMOS camera sensor scans one line (of
pixels) at a time
Slide 13
Received frame patterns Original frame Single frame 2-frame mix
3-frame mix Consecutive frames indicating a missing frame Mixing
pattern varies by line
Slide 14
Display frame rate C Frame capture rate C/2 C 2C Occasional
missed frames 4-frame mixes 3-frame mix Occasional missed
frames
Slide 15
Synchronized communication Display frame rate C Frame capture
rate C/2 C 2C The only decodable pattern Frame synchronized
Under-utilized and inefficient! Carries information But undecodable
and dropped Cannot be used (of worst camera)
Slide 16
Unsynchronized communication Display frame rate (fps) 0 30
Works with any display and capture rates Higher camera capture
rate, better performance Supported frame capture rate
(fps)3015
Slide 17
Requirements Need to be able to decode mixed frames In-frame
color tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding Essentially a temporal code across
frames Works with any (monochrome) per-frame code
Slide 18
Our solution: LightSync Need to be able to decode mixed frames
In-frame grayscale tracking per line Need to guard against lost
frames/lines Inter-frame erasure coding Essentially a temporal code
across frames Works with any (monochrome) per-frame code
Slide 19
LIGHTSYNC DEMO
Slide 20
LightSync Need to be able to decode mixed frames In-frame
grayscale tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding
Slide 21
Per-line tracking: Mixing patterns 3 frames mixing 3 blocks
mixing Bl or w per block 8 mixing possibilities ??? Frame 1 Frame 2
Frame 3 RXTX
Slide 22
Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3
Frame 1 Frame 2 Frame 3
Slide 23
Per-line tracking: Mixing patterns Trivial cases RX TX Line 1
Line 2 Line 3 Frame 1 Frame 2 Frame 3
Slide 24
Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line
2 Line 3 Frame 1 Frame 2 Frame 3
Slide 25
Per-line tracking: Mixing patterns Tricky cases, to be tracked
per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
Slide 26
Code area Per-line tracking: Reference blocks Corner Scanning
direction Tracking bar orientation Tracking bars
Slide 27
Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4 4 reference blocks per line
Slide 28
Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4
Slide 29
Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference
colors of mixes
Slide 30
LightSync Need to be able to decode mixed frames In-frame
grayscale tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding
Slide 31
Inter-frame coding Simple pairwise XOR Group of 3 original
frames Coded frames
Slide 32
Inter-frame coding Looping display on screen Coded frames All
original frames Camera stops frame capture after message
decoded
Slide 33
PERFORMANCE
Slide 34
Implementation Current frame capture rate Inter-frame
intervals
Slide 35
Performance Goal recap: Works with any display/capture rate
combinations Metric: Effective frame capture rate Actual throughput
scales with per-frame capacity
Slide 36
Schematic performance For a 20fps phone camera
Slide 37
Schematic performance For a 20fps phone camera Ideal Peaks at
the camera rate Level at higher display rates Ideal
Slide 38
Schematic performance Peaks at ~ half the camera rate For a
20fps phone camera Conventional Drops to 0 at higher display rates
Ideal
Slide 39
Schematic performance Conventional LightSync For a 20fps phone
camera Peaks at the camera rate Slight drops at higher display
rates Ideal
Slide 40
LightSync performance Nokia Lumia 920 (~30 fps)
Slide 41
LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20
fps)
Slide 42
LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20
fps) Huawei Ascend C8812E (~15 fps) Works with any display and
capture rates Higher camera capture rate, better performance
Slide 43
Conclusion and outlook LightSync: Unsynchronized visual
communication Primitive for similar scenarios Lots of potential in
the space of visual communication
Related work Temporal barcode design (inter-frame)
Unsynchronized 4D Barcodes Basically slows down display, only for
TX up to 10fps Spatial barcode design (per-frame layout) COBRA:
Optimized for phone-to-phone PixNet: High-end camera,
computationally intensive, offline processing Other work on visible
light communications