Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
SocialSync: Synchronization for a
Smartphone Camera Network
Richard Latimer
Rice University
September 7, 2014
2
2
Intro
3
Why Smartphones?
• 1 billion smartphones sold in 2014
• Growth in computer vision applications
• Advanced development platform
Best camera is the one with you!
4
4
What if Smartphones Could Collaborate?
Refocusing Free Viewpoint Television
3D Depth
Lytro
PelicanPhilips
Kilner et al.
5
Prior Work: Camera Arrays
Traditional camera arrays are assumed to be embedded and tightly synchronized.
High-Speed VideographyUsing a Dense Camera Array
[Wilburn et al.]
High Performance ImagingUsing Large Camera Arrays
[Wilburn et al.]
6
6
A Smartphone Camera ArrayInvolves the OS Stack
Application
Application Framework
Library Runtime
Linux Kernel/Drivers
Hardware
7
7
Social Capture: Problem Statement
If you use multiple smartphone cameras to capture video simultaneously, there is an inherit synchronization challenge that manifests (due to OS stack) when the scene is moving.
8
8
Scene Setup: How Bad is Misalignment?
Scene Setup
1
0
0
1
0
0
1
0
0
1
0
0
Resolution Target
SmartphoneCamera Array
9
9
Outcome: How Bad is Misalignment?
Even when photos are captured simultaneously, misalignment of frames can
cause problems when target is moving.
Moving Resolution Target Capturedby 4 Phones Simultaneously
10
10
Source of Problem
Frame 0 Frame 1 Frame 2
Frame 0 Frame 1
Frame 1 Frame 2 Frame 3
time
Camera 1:
Camera 2:
Camera 3: Frame 2
Frame 0
11
11
Goal of SocialSync
Objective: Align frame sequences from smartphone cameras within tolerance ranges less
than t milliseconds, where t < frame duration.
Frame 0 Frame 1 Frame 2
Frame 0 Frame 1
Frame 1 Frame 2 Frame 3
time
Camera 1:
Camera 2:
Camera 3: Frame 2
Frame 0
12
12
1. Characterization
2. SocialSync Protocol
3. Evaluation
13
13
1. Characterization
• Measurement Test Apparatus
• Results
2. SocialSync Protocol
3. Evaluation
14
14
What We Use: Android 4.4
15
15
Network Setup
1
0
0
Hub and Spoke Network
•Server provides global reference clock
•Server broadcasts image capture request
1
0
0
1
0
0
1
0
0
1
0
0
16
System for Measuring Capture Timestamps
16
17
17
1. Characterization
• Measurement Test Apparatus
• Results
2. SocialSync Protocol
3. Evaluation
18
18
Random Variables Associatedwith Capturing a Frame
Setup Time: Delay to setup camera before capture of first frame
Delivery Time: After capture, there is delay to when frame reaches application and a timestamp can be recorded.
19
19
Camera Setup
Setup Frame 0 Frame 1 Frame 2
time
20
20
Camera 1: Is Setup Time Consistent?
Setup
Setup
Setup
time
21
21
Camera 1: Is Setup Time Consistent?
Setup
Setup
Setup
time
Frame 0
Frame 0
Frame 0
22
22
Variability in Camera Setup Makes Aligning Frames Challenging
Each time camera started, the start of the image sequence will be delayed by a variable amount of time.
•Setup can vary at least 35 ms
•StDev = 9.4 ms
•Not deterministic
23
23
Frame Delay
Setup Frame 0
time
Frame 1 Frame N
24
Measurement of Frame Delay
25
25
Delivery Time of Frame to Application
If we keep measuring capture times of frames, they are spaced evenly
Delivery time of frame to app is a proxy for estimating capture time with delay:
• μ = 67 ms• σ = 4.5 ms
26
26
1. Characterization
2. SocialSync Protocol
• Estimation
• Alignment
3. Evaluation
27
27
Setup Frame 0 Frame 1 Frame 2
Setup Frame 0 Frame 1
Setup Frame 0 Frame 1 Frame 2
time
Camera 1:
Camera 2:
Camera 3: Frame 2
Goal of Social Sync
28
Social Sync Protocol
Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames
Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment
28
29
29
Setup Frame 0 Frame 1 Frame 2
Setup Frame 0 Frame 1
Setup Frame 0 Frame 1 Frame 2
time
Camera 1:
Camera 2:
Camera 3: Frame 2
If Misalignment is Due to Camera Setup
Frame 2
30
30
Setup Frame 0 Frame 1 Frame 2
Setup Frame 0 Frame 1
Setup Frame 0 Frame 1 Frame 2
time
Camera 1:
Camera 2:
Camera 3:
Could We Estimate Capture Time By Calibrating for Setup Time?
31
31
Recall Wide Variability in Setup
If setup time had little variance, could align capture timestamps by offsetting
•Setup can vary at least 35 ms, when worst case frame misalignment was 33 ms (inverse of FPS)
•Can’t calibrate for setup time to reduce misalignment!
32
32
Does Frame Delay Provide a Better Estimate of the Capture Timestamp?
Setup Frame 0
time
Frame 1 Frame N
33
Measurement of Frame Delay
34
How does estimate of mean improve with number of samples?
Stats 101: Standard error of a sample mean is
Based on Central Limit Theorem, 95% confidence sample mean is within range:
35
35
Mean Frame Delay Recap
Setup Frame 0
time
36
36
Mean Frame Delay Recap
Setup Frame 0
time
Mean Delay Est.:
37
37
Mean Frame Delay Recap
Setup Frame 0
time
Frame 1
Mean Delay Est.:
38
38
Mean Frame Delay Recap
Setup Frame 0
time
Frame 1 Frame N
Mean Delay Est.:
39
Samples Required for Estimate
Since mean delay is , we can determine the capture image sequence position for milliseconds using:
•22 samples for 2 ms accuracy
•85 samples for 1 ms accuracy
39
40
Social Sync Protocol
Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames
Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment
40
41
41
Synchronization Tolerance
time
Goal: Want
synchronization to
a reference image
sequence to be
aligned within
some tolerance
42
42
Synchronization Attempt 1
Setup Frame 0
time
43
43
Synchronization Attempt 1
time
Frame N
Setup Frame 0
44
44
Synchronization Attempt k
Setup Frame 0
time
45
45
Synchronization Attempt k
time
Frame N
Setup Frame 0
46
Obtaining Synchronization
•Let T be the worst case misalignment between frames
•Let t be the range specified by the sync tolerance, where t ≤T
Let p = t/T, be the probability of synchronization error being reduced for a single phone
46
47
Single Camera Sync ProbabilityAfter k Attempts
Pk = 1-(1-p)k
Expected Number of Sync Cameras
n*Pk
47
48
Expected Attempts to Sync N Cameras
48
49
49
•Characterization
•SocialSync Protocol
•Evaluation
50
Evaluating Synchronization
NaïveSync SocialSync
4 Cameras
8 Cameras
23 ms
35 ms
5 ms
6 ms
50
Smartphones in array to reduceartifacts not related to timesynchronization error
51
8 Cameras
Naïve Sync
SocialSync
Reference Views Depth Map 51
52
4 Cameras
Reference NaïveSync SocialSync
52
53
4 Cameras
NaïveSync SocialSyncReference
53
54
Conclusion
1. Characterize Android Camera
2. Developed SocialSync Protocol
3. Demonstrated SocialSync
Future Work:
- Greater control of Android camera system
- In wild implementation
54
55
Acknowledgements
1. Ashok, Lin, Ashu and their students
2. Co-Author Jason Holloway
3. LF4CV Reviewers, Paper Accepted
4. National Science Foundation
55