View
32
Download
4
Category
Preview:
DESCRIPTION
LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera. Supervisor: Prof. Michael R.Lyu Group Member: Leung Man Kin, Stephen Ng Ying Kit, Tommy. Outline. Introduction Approach Stitching Algorithm Geometric Transformation Progress Conclusion Future Plan. Outline. - PowerPoint PPT Presentation
Citation preview
11
LYU0503LYU0503Document Image RestorationDocument Image Restoration
on Mobile Phoneon Mobile PhoneUsing Onboard CameraUsing Onboard Camera
Supervisor:Supervisor:Prof. Michael R.LyuProf. Michael R.Lyu
Group Member:Group Member:Leung Man Kin, StephenLeung Man Kin, StephenNg Ying Kit, TommyNg Ying Kit, Tommy
22
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
33
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
44
IntroductionIntroduction
Use mobile phone to capture photo Use mobile phone to capture photo of document.of document.
The document can be stored in the The document can be stored in the phone or sent to other people by phone or sent to other people by several means.several means.
55
ObjectiveObjective
Use camera to take photo of different Use camera to take photo of different parts of a document.parts of a document.
Use suitable algorithm to merge the Use suitable algorithm to merge the photos back to a whole document.photos back to a whole document.
Transmit the document by several Transmit the document by several meansmeans• e.g. Fax, MMS, etc.e.g. Fax, MMS, etc.
66
MotivationMotivation Most of phones nowadays have built-in camerMost of phones nowadays have built-in camer
a.a.
Resolution of camera is not enough for documResolution of camera is not enough for documentent• The text on the document would be blurredThe text on the document would be blurred
Symbian phone can provide several transmissiSymbian phone can provide several transmission function. E.g. Fax, MMSon function. E.g. Fax, MMS
Fax at any time!Fax at any time!
77
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
88
ApproachApproach
Take photos of different part of Take photos of different part of documentdocument
Use stitching algorithm to merge the Use stitching algorithm to merge the photos into a photophotos into a photo
Stitching algorithm is our major Stitching algorithm is our major research area of our projectresearch area of our project
99
AssumptionAssumption
The document consists mainly of The document consists mainly of text.text.
The camera has macro mode.The camera has macro mode.• Sharp Close Up.Sharp Close Up.
1010
ApproachApproach
Procedure of our designProcedure of our design• Take a photo of full document first.Take a photo of full document first.
• Then ask the user to take photos of Then ask the user to take photos of different partsdifferent parts
Overlay a rough picture on the screen of the Overlay a rough picture on the screen of the phone.phone.
1111
ApproachApproach
A4 document
1212
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
1313
Stitching AlgorithmStitching Algorithm
Stitching needs to find corresponding Stitching needs to find corresponding points in two imagespoints in two images
two different approachestwo different approaches• Direct alignment Direct alignment
Compare images pixel by pixelCompare images pixel by pixel e.g.e.g.
• Feature-based registrationFeature-based registration Use some methods to find feature pointsUse some methods to find feature points Match features in two imagesMatch features in two images
images. two theare and , 10
2
01 IIxIuxIuEi
iiSSD
1414
Stitching AlgorithmStitching Algorithm Feature-based registration is chosen.Feature-based registration is chosen.
• Usually, direct alignment is less efficient.Usually, direct alignment is less efficient.• Need to consider different scale, rotation, etc.Need to consider different scale, rotation, etc.
There are several methods to extract There are several methods to extract feature.feature.• E.g. corner detector.E.g. corner detector.
Scalable Invariant Feature Transform Scalable Invariant Feature Transform (SIFT) is used to extract the feature.(SIFT) is used to extract the feature.• The features are invariant to image scale and The features are invariant to image scale and
rotation.rotation.
1515
Stitching AlgorithmStitching Algorithm
How to get the feature point?How to get the feature point?• Identifies key locations in scale space by Identifies key locations in scale space by
looking for locations that are maxima or looking for locations that are maxima or minima of a difference-of-Gaussian minima of a difference-of-Gaussian function.function.
2
22
222
1,,
yx
eyxG
yxIyxGkyxGyxD ,*,,,,,,
1616
Stitching AlgorithmStitching Algorithm
Varying Varying σσ
Repeat the above process using subsampled image Repeat the above process using subsampled image for several timesfor several times• Decrease the resolution by 2.Decrease the resolution by 2.
2
22
222
1,,
yx
eyxG
1717
Stitching AlgorithmStitching Algorithm
Find the maxima or minima by comparinFind the maxima or minima by comparing 26 neighbors.g 26 neighbors.
They are the key feature points.They are the key feature points.
1818
Stitching AlgorithmStitching Algorithm
Property of the feature keypointsProperty of the feature keypoints• Achieve partial invariance to local variations Achieve partial invariance to local variations
like affine or 3D projections.like affine or 3D projections.
• Need to filter feature keypoints furtherNeed to filter feature keypoints further Reject keypoints that have low contrast.Reject keypoints that have low contrast.
1919
Stitching AlgorithmStitching Algorithm
How to store the feature?How to store the feature?• For each keypoint, image gradient magnituFor each keypoint, image gradient magnitu
des and orientations are sampled to store tdes and orientations are sampled to store the feature.he feature.
2020
Stitching AlgorithmStitching Algorithm Compare the features in two imagesCompare the features in two images
• Get the position to stitchGet the position to stitch
However, the photo taken may suffer from However, the photo taken may suffer from geometric distortiongeometric distortion• e.g. Lens distortion, taking photo with different anglese.g. Lens distortion, taking photo with different angles
2121
Stitching AlgorithmStitching Algorithm
Use the matched feature to calculate thUse the matched feature to calculate the transformation parameters. e transformation parameters.
Transform images to the correct orientaTransform images to the correct orientation before mergingtion before merging
2222
SIFT
Stitching AlgorithmStitching Algorithm
Extract feature keypoint
Matching keypoints in two images
Findtransformation parameter
by matched keypoints
Transform imagesusing the parameter
Merge all the images
2323
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
2424
Lens CorrectionLens Correction
Transformation
2525
Transformation parameters ai
Transformation parameters bi
Image2
Image3
Image1
2626
Take several
overlapping images
of the document……
2727
High Resolution
Document Image
Restored!
2828
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
2929
Porting to Symbian PhonePorting to Symbian Phone
On the PC, the stitching algorithm is impOn the PC, the stitching algorithm is implemented using C# and Matlab.lemented using C# and Matlab.
We should port the algorithm to the SymWe should port the algorithm to the Symbian phone.bian phone.
Symbian OS use C++ as the language.Symbian OS use C++ as the language.
3030
Porting to Symbian PhonePorting to Symbian Phone
Difficulty:Difficulty:• Their syntaxes are not the same.Their syntaxes are not the same.
e.g. set / get keyword in C#e.g. set / get keyword in C#• Symbian and C# provide different API.Symbian and C# provide different API.
Math library, Array library.Math library, Array library.• Some of data types are different in Symbian Some of data types are different in Symbian
C++.C++.
3131
Porting to Symbian PhonePorting to Symbian Phone
Difficulty:Difficulty:• The processor on Symbian phone is not poThe processor on Symbian phone is not po
werful.werful.
• The memory space on Symbian phone is noThe memory space on Symbian phone is not large.t large.
We may need modify the algorithm so thWe may need modify the algorithm so that it would suit the phone.at it would suit the phone.
3232
OutlineOutline
IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
3333
ConclusionConclusion
We have studied different stitching algorWe have studied different stitching algorithmsithms
We have implemented the stitching algoWe have implemented the stitching algorithm on PCrithm on PC
We are porting the stitching algorithm oWe are porting the stitching algorithm on Symbian phonen Symbian phone
3434
OutlineOutline
IntroductionIntroduction ObjectiveObjective MotivationMotivation ApproachApproach Stitching algorithmStitching algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan
3535
Future PlanFuture Plan
Implement the stitching on the Symbian Implement the stitching on the Symbian phone.phone.
Modify SIFT or design other algorithms fModify SIFT or design other algorithms for more efficient performance on mobile or more efficient performance on mobile phonephone
3636
Future PlanFuture Plan
Improve the algorithm of merging.Improve the algorithm of merging.
Study super-resolution to make text Study super-resolution to make text clearer.clearer.
Recommended