Upload
nicholas-eu
View
148
Download
0
Embed Size (px)
Citation preview
1
Real-time Image Stitching with OpenCV
Advisor: Ching-Hwa ChengFeng Chia University
Priscilla ChengNicholas Eu
2
What is Image Stitching?
Panorama, Google Maps Streetview Minimally Invasive Surgery (MIS)
Image stitching provides one cohesive, enlarged view of surgical area
3
Our Task
Difficulties Existing research minimal Real time video, not still images
Use OpenCV library with C++ Develop and improve video stitching
application Stitch up to 4 cameras Improve stitching stability/speed
4
Stitching ProcessFeature Detection and Extraction
Feature Matching
Find Homography Matrix
Perspective Transformation
Output
5
Stitching Algorithm
6
Frame-by-frame Video Stitching Real-time video made of many
frames Stitching process repeated on each
frame Loop: extract frame stitch images
output
Left Frame Right Frame
7
Stitching Process1. Feature detection and extraction
• Use a feature detector to find feature points in overlapping regions
https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect6.pdf
8
Stitching Process
SURF Detector Algorithm
1. Feature detection and extraction
Box Filters2nd derivative of the Gaussian
9
Stitching Process2. Feature Matching
• BruteForceMatcher• Green – Good match• Red – Bad match
if distance between matches < threshold,push match pair into vector(save match pair)
10
findHomography function in OpenCV
Two images related by homography if on same plane, but different perspectives.
Describes mapping between two images with matching points using 3x3 matrix
Stitching Process3. Find Homography MatrixStitching Process
http://ieeexplore.ieee.org/ieee_pilot/articles/08tro05/tro-scaramuzza-2004490
11
Warpperspective function in OpenCV
Use the Homography matrix to combine multiple perspectives into one perspective
Stitching Process4. Perspective TransformationStitching Process
http://cebe.cf.ac.uk/learning/habitat/HABITAT1/QTVR.html
12
Stitching Process5. OutputStitching Process
+
13
Demo: 2 Videocameras
14
How can we improve?
1. Stitch up to 4 videos simultaneously Adding a camera = more processing
time = program slows down!2. Speed up program
Improve feature detection speed Reduce lag due to additional cameras
15
Stitching 3 Videos Take the output of first two stitched
frames Apply the stitching process to this
image and the 3rd frame1 2
3
16
Stitching 4 VideosProcess1. Stitch 1st and 2nd frames2. Stitch 3rd and 4th frames3. Stitch results from above steps together
12
34
17
4 Videocameras: Output
1 2
3 4
18
4 Videocameras: Output
1 2
3 4
19
Pipeline Parallelism with OpenMP
Parallel as opposed to sequential execution Utilize two threads on the CPU Thread 1 and Thread 2 execute
simultaneously Analogy: assembly line Resolve issues with speed
Before pipelining: ~1.2 seconds per frame
Pipeline ParallelismThread 1 Thread 2
21
Speed-up using OpenCL Feature detection and extraction
take up 70-80% of the total time Utilizes GPU, multiple coresSURF_OCL: OpenCL implementation
of SURF detector and extractor Improves speed and stability
nvidia.com
ctimes.com.tw
22
Other Improvements For fixed camera angles Limit feature detection to just half
the frame Essentially, don’t try to find matches
outside specified region
23
Locking Homography Calculations
Lock = fixed Homography matrix values
Unlock = recalculate values every frame
24
Accomplishments
Reduce each frame’s average processing time to ~0.6 seconds
About 2x speedup after parallelizing Avoid exponential time increase
Processing Time
Number of
Cameras
Before ParallelismAfter Parallelism
25
Demo: 4 Videocameras
26
Future Work
Improve accuracy Seam removal Optimize algorithm of feature
detection functions Test on endoscopic cameras at very
close range
27
Thank you for listening!