A Fully-Automated Solver for Multiple Square …...A Fully-Automated Solver for Multiple Square Jigsaw Puzzles Using Hierarchical Clustering October 27, 2016 Zayd Hammoudeh [email protected]

A Fully-Automated Solver for Multiple SquareJigsaw Puzzles Using Hierarchical Clustering

October 27, 2016

Zayd [email protected]

Department of Computer ScienceSan José State University

mailto:[email protected]

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

1Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching


Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionThesis Goals

Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.

Additional Goals:

I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver

I Design visualizations for viewing the errors (if any) in asolver output

51




1Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


IntroductionThesis Goals

Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.

Additional Goals:

I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver

I Design visualizations for viewing the errors (if any) in asolver output

51




Thesis Goals

2IntroductionPrevious Work


Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


IntroductionJigsaw Puzzles

I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.

I First computational jigsaw puzzle solver introduced in1964

I Solving a jigsaw puzzle is NP-complete [1, 2]

I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing

I In most cases, the original, “ground-truth” image isunknown.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


IntroductionJig Swap Puzzles

Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult

Ground-Truth Image Randomized Jig Swap Puzzle

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions




Ground-Truth Image

Randomized Jig Swap Puzzle

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions




Ground-Truth Image Randomized Jig Swap Puzzle

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions








51




Thesis Goals

Introduction5Previous Work


Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Previous WorkPaikin & Tal

Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver

I Supports Type 1, Type 2, and Mixed-Bag puzzles

I Immune to missing pieces

Limitations:I Poor Seed Selection: All decisions are made at runtime

using as few as 13 pieces

I Externally Supplied Information: The solver must betold the number of input puzzles

51




Thesis Goals

Introduction5Previous Work


Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Previous WorkPaikin & Tal

Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver

I Supports Type 1, Type 2, and Mixed-Bag puzzles

I Immune to missing pieces

Limitations:I Poor Seed Selection: All decisions are made at runtime

using as few as 13 pieces

I Externally Supplied Information: The solver must betold the number of input puzzles

The Mixed-Bag Solver

51




Thesis Goals


6Mixed-Bag SolverAssembler

Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Mixed-Bag SolverBasic Structure

...

Mixed Bag

Final

Assembly

Hierarchical

Segment

Clustering

Segmentation Stitching

Final

Seed Piece

Selection

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Mixed-Bag SolverFoundations

Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece

I Place all pieces around the expanding kernel

Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)

I Iteratively merge smaller regions to form large ones

I Advantages of this Approach:

I Reduces the size of the problem

I Provides structure to the unordered set of puzzle pieces.

The alternate strategy is the basis of the Mixed-Bag Solver

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions











51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions











51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions











51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Mixed-Bag SolverOverview

I The Mixed-Bag Solver is fully-automated. It makes noassumptions concerning piece orientation, puzzledimensions, or number of puzzles.

I Input: A bag of puzzle pieces

I Output: One or more disjoint, solved puzzles.

I The Mixed-Bag Solver consists of five distinct stages:

...

Mixed Bag

Final

Assembly

Hierarchical

Segment

Clustering

Segmentation Stitching

Final

Seed Piece

Selection

51




Thesis Goals


Mixed-Bag Solver9Assembler

Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


AssemblerMixed-Bag Solver Component

I Role: Place the individual pieces in the solved puzzle.

I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.

I Assembler Used in this Thesis: Paikin & Tal

I Current state of the art

I Allows for more direct comparison of performance

I Natively supports Mixed-Bag puzzles

I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]

I Written in Python and fully open source [5]

51




Thesis Goals


Mixed-Bag Solver9Assembler

Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


AssemblerMixed-Bag Solver Component

I Role: Place the individual pieces in the solved puzzle.

I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.

I Assembler Used in this Thesis: Paikin & Tal

I Current state of the art

I Allows for more direct comparison of performance

I Natively supports Mixed-Bag puzzles

I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]

I Written in Python and fully open source [5]

51




Thesis Goals



10Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationMixed-Bag Solver Stage #1

I Segment: Partial puzzle assembly where this is a highdegree of confidence pieces are placed correctly.

I Role of Segmentation: Provide structure to the set ofpuzzle pieces by partitioning them into disjoint segments

I Input: A bag of puzzle pieces

I Output: Set of saved segments

I Relationship between Puzzle Pieces and Segments:

I Pieces from a single ground-truth input may be separatedinto multiple segments

I A piece can be assigned to at most one segment

51




Thesis Goals



11Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationAlgorithm Overview

I Iterative process consisting of one or more rounds

I In each round, all pieces not yet assigned to a segmentare assembled as if all are from the same input image

I Segments of sufficient size are saved to be used in futureMixed-Bag Solver stages

I Pieces in a saved segment are not placed in future rounds.

I Segmentation terminates if all pieces are assigned to asaved segment or when no segment is larger than theminimum allowed size

51




Thesis Goals



12Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationComposition of a Segment

I Starting a Segment: Each segment is created iterativelystarting with a single seed piece

I Definition of Best Buddies: Any pair of pieces that aremore similar to each other than they are to any other piece.

I Growing the Segment: Add to the segment any piecethat is a neighbor and best buddy of a segment member

I Trimming the SegmentI Articulation Point: Any piece whose removal disconnects

other pieces from the segment seed.I All articulation pieces pieces are removed from the segment.

I After the removal of the articulation points, any pieces nolonger connected to the seed are removed.

51




Thesis Goals



13Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationExample – Input Images

Image (a) – 805 Pieces [6]

Image (b) – 540 Pieces [7]

51




Thesis Goals



14Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationExample – First Segmentation Round Output Image

51




Thesis Goals



15Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


SegmentationExample – Segmented Output Image

51




Thesis Goals



Segmentation

16Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingMixed-Bag Solver Stage #2

I Role of Stitching: Quantify the extent that any pair ofsegments is related.

I Input: All puzzle pieces and the set of saved segments

I Output: Segment overlap matrix

I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.

I Segments should be allowed, but not forced, to expand in alldirections.

51




Thesis Goals



Segmentation

16Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingMixed-Bag Solver Stage #2

I Role of Stitching: Quantify the extent that any pair ofsegments is related.

I Input: All puzzle pieces and the set of saved segments

I Output: Segment overlap matrix

I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.

I Segments should be allowed, but not forced, to expand in alldirections.

51




Thesis Goals



Segmentation

17Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingStitching Piece Location

I Mini-assembly (MA): Same as a standard assemblyexcept only a fixed number of pieces are placed .

I Stitching Piece (ζx ): A piece near the boundary of asegment that is used as the seed of a mini-assembly

I Segment Overlap: Maximum overlap between anymini-assembly for segment, Φi and another segment Φj .

OverlapΦi ,Φj = arg maxζx∈Φi

|MAζx

⋂Φj |

min(|MAζx |, |Φj |)(1)

I Asymmetry: In most cases:

OverlapΦi ,Φj 6= OverlapΦj ,Φi (2)

51




Thesis Goals



Segmentation

18Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingExample – Input Image

51




Thesis Goals



Segmentation

19Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingExample – Two Segment Images

Segment #1 Segment #2

51




Thesis Goals



Segmentation

20Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingExample – Stitching Piece Locations

Segment #1 Segment #2

51




Thesis Goals



Segmentation

21Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


StitchingExample – Stitching Piece Locations

Stitching Result from Segment #1

Segment Overlap:

OverlapΦ1,Φ2 = 0.83 (3)

51




Thesis Goals



Segmentation

Stitching

22Hierarchical Clustering


Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Hierarchical Segment ClusteringMixed-Bag Solver Stage #3

I A single ground-truth image may be comprised of multiplesegments.

I Role of Hierarchical Clustering: Merge all segmentsfrom the same input image into a single segment cluster.

I Input: All saved segments and the segment overlap matrix

I Output: A set of segment clusters

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Hierarchical Segment ClusteringCalculating the Initial Similarity Matrix

I Segment Overlap Matrix: A hollow matrix quantifying therelationship between each pair of segments.

I Hierarchical Clustering Similarity Matrix: A diagonalmatrix quantifying the similarity between segment pairs.

I Quantifying Similarity: Given n segments, the similaritybetween segments Φi and Φj is:

ωi,j =OverlapΦi ,Φj + OverlapΦj ,Φi

2(4)

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Hierarchical Segment ClusteringMerging Clusters

I After two clusters are combined, the similarity between themerged cluster and all other clusters must be recalculated.

I Single Link Clustering: The similarity between any twoclusters is equal to the maximum similarity between anytwo members in the clusters [8]

I The Mixed-Bag Solver must use single link clustering astwo clusters may only have two member segments that areadjacent.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Hierarchical Segment ClusteringExample – Single Linking

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions



Segment 1 Segment 2 Segment 3 Segment 4

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions



Segment Cluster 1 Segment 3 Segment 4

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions



Segment Cluster 1 Segment Cluster 2

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Hierarchical Segment ClusteringTerminating Clustering

I The solver continues merging segment clusters until oneof two criteria is satisfied:

I Only a single segment cluster remains

I Maximum similarity between any segment clusters is belowa predefined threshold

I All remaining segment clusters are passed to the nextsolver stage.

51




Thesis Goals



Segmentation

Stitching


30Final Seed Piece Selection

Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Final Seed Piece SelectionMixed-Bag Solver Stage #4

Mixed-Bag Solver

I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.

I Input: Set of segment clusters

I Output: Final seed pieces

I A single seed piece is selected from each segment cluster

Paikin & Tal

I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.

51




Thesis Goals



Segmentation

Stitching


30Final Seed Piece Selection

Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Final Seed Piece SelectionMixed-Bag Solver Stage #4

Mixed-Bag Solver

I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.

I Input: Set of segment clusters

I Output: Final seed pieces

I A single seed piece is selected from each segment cluster

Paikin & Tal

I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.

51




Thesis Goals



Segmentation

Stitching



31Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Final Assembly StageMixed-Bag Solver Stage #5

I Role of Final Assembly: Generate the solved puzzlesthat are output by the Mixed-Bag Solver.

I Input: Set of puzzle pieces with the seeds marked

I Output: Final solved puzzles

I All pieces are placed around the seeds selected in theprevious stage.

I Assembly proceeds in this stage normally without anycustom modifications.

Quantifying Solver Quality

51




Thesis Goals



Segmentation

Stitching



Final Assembly

32Quantifying QualityDirect Accuracy

Neighbor Accuracy


Solver Comparison

Conclusions



I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)

I Metrics compare the quality of solver outputs

I Two Most Common Quality Metrics:

I Direct Accuracy

I Neighbor Accuracy

I Disadvantages of Current Metrics: Neither account for:

I Pieces misplaced in different puzzles

I Extra pieces from other puzzles

I Goal: Define new quality metrics for Mixed-Bag puzzles

51




Thesis Goals



Segmentation

Stitching



Final Assembly

32Quantifying QualityDirect Accuracy

Neighbor Accuracy


Solver Comparison

Conclusions



I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)

I Metrics compare the quality of solver outputs

I Two Most Common Quality Metrics:

I Direct Accuracy

I Neighbor Accuracy

I Disadvantages of Current Metrics: Neither account for:

I Pieces misplaced in different puzzles

I Extra pieces from other puzzles

I Goal: Define new quality metrics for Mixed-Bag puzzles

51




Thesis Goals



Segmentation

Stitching



Final Assembly

Quantifying Quality33Direct Accuracy

Neighbor Accuracy


Solver Comparison

Conclusions


Quantifying Solver QualityStandard and Enhanced Direct Accuracy

I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n

DA =cn

(5)

I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.

EDASPi = arg maxSj∈S

ci,j

ni +∑

k 6=i (mk,j )(6)

I Direct Accuracy Range: 0 to 1

I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions




DA =cn

(5)



ci,j

ni +∑

k 6=i (mk,j )(6)



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions




DA =cn

(5)



ci,j

ni +∑

k 6=i (mk,j )(6)



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Direct AccuracyExample – Effect of Shifts

Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed

Ground-Truth Image Solver Output

Conclusion: Direct accuracy can be overly punitive.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions




Ground-Truth Image

Solver Output


51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Direct AccuracyShiftable Enhanced Direct Accuracy Score (SEDAS)

I Solution: Allow the reference point for direct accuracy toshift beyond the upper left corner of the image

I Shiftable Enhanced Direct Accuracy Score (SEDAS):Select the reference point, l , within radius dmin of the upperleft corner of the solved puzzle

I dmin – Manhattan distance between the upper left corner ofthe solved image and the nearest puzzle piece

I Formal Definition of SEDAS:

SEDASPi = arg maxl∈L

(arg max

Sj∈S

ci,j,l

ni +∑

k 6=i (mk,j )

)(7)

I SEDAS Range: 0 to 1

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Direct AccuracyExample – Shiftable Reference Point

Solver Output

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions



Solver Output Direct Accuracy Reference Point

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions



Solver Output SEDAS Reference Points

51




Thesis Goals



Segmentation

Stitching



Final Assembly


37Neighbor Accuracy


Solver Comparison

Conclusions


Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy

I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q

NA =a

n · q(8)

I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.

ENASPi = arg maxSj∈S

ai,j

q(ni +∑

k 6=i (mk,j )(9)

I Neighbor Accuracy Range: 0 to 1

I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy

51




Thesis Goals



Segmentation

Stitching



Final Assembly


37Neighbor Accuracy


Solver Comparison

Conclusions




NA =a

n · q(8)



ai,j

q(ni +∑

k 6=i (mk,j )(9)



51




Thesis Goals



Segmentation

Stitching



Final Assembly


37Neighbor Accuracy


Solver Comparison

Conclusions




NA =a

n · q(8)



ai,j

q(ni +∑

k 6=i (mk,j )(9)



51




Thesis Goals



Segmentation

Stitching



Final Assembly


37Neighbor Accuracy


Solver Comparison

Conclusions




NA =a

n · q(8)



ai,j

q(ni +∑

k 6=i (mk,j )(9)



Experimental Results

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions



I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






# Puzzles 1 2 3 4 5



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






# Puzzles 1 2 3 4 5



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






# Puzzles 1 2 3 4 5



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions






# Puzzles 1 2 3 4 5



51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Experimental ResultsDetermining Input Puzzle Count

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles

I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.

I Single Puzzle Accuracy – Represents the solver’sperformance ceiling

I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Experimental ResultsDetermining Input Puzzle Count

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles

I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.

I Single Puzzle Accuracy – Represents the solver’sperformance ceiling

I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy

Experimental Results40Input Puzzle Count

Solver Comparison

Conclusions


Determining Input Puzzle CountSingle Input Puzzle Results

I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input

I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.

I All three images have large areas with little variation (e.g., ablue sky, smooth water)

I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]

I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountSingle Input Puzzle Results

I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input

I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.

I All three images have large areas with little variation (e.g., ablue sky, smooth water)

I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]

I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountVisual Comparison of a Misclassified Image

Perfectly ReconstructedImage (a) Misclassified Image (b)

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountMultiple Input Puzzles

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images

I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.

I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.

I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountMultiple Input Puzzles

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images

I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.

I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.

I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountMultiple Input Puzzles – Results

0 1 2 30

20

40

60

80

100

75

167

2

4448

4 4

50 50

0 0

60

20 20

0

Size of Input Puzzle Count Error

Freq

uenc

y(%

)

Mixed-Bag Solver’s Input Puzzle Count Error Frequency

2 Puzzles 3 Puzzles 4 Puzzles 5 Puzzles

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountMultiple Input Puzzles – Results Summary

I Overall Accuracy: 65%

I Iterations with Error Greater than One: 8%

I Accuracy did not significantly degrade as the number ofinput puzzles increased.

I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.

I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

Conclusions


Determining Input Puzzle CountMultiple Input Puzzles – Results Summary

I Overall Accuracy: 65%

I Iterations with Error Greater than One: 8%

I Accuracy did not significantly degrade as the number ofinput puzzles increased.

I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.

I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


45Solver Comparison

Conclusions


Experimental ResultsPerformance on Multiple Input Puzzles

I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm

I Procedure: Randomly select a specified number ofimages and input them into both solvers.

I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage

I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.

I This represents the performance ceiling for optimalhierarchical clustering.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


45Solver Comparison

Conclusions


Experimental ResultsPerformance on Multiple Input Puzzles

I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm

I Procedure: Randomly select a specified number ofimages and input them into both solvers.

I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage

I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.

I This represents the performance ceiling for optimalhierarchical clustering.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


46Solver Comparison

Conclusions


Performance on Multiple Input PuzzlesShiftable Enhanced Direct Accuracy Score (SEDAS)

2 3 4 50

0.2

0.4

0.6

0.8

1

Number of Input Puzzles

SE

DA

S

Effect of the Number of Input Puzzles on SEDAS

MBS Correct Puzzle Count MBS All Paikin & Tal

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


47Solver Comparison

Conclusions


Performance on Multiple Input PuzzlesEnhanced Neighbor Accuracy Score (ENAS)

2 3 4 50

0.2

0.4

0.6

0.8

1


EN

AS

Effect of the Number of Input Puzzles on ENAS


51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


48Solver Comparison

Conclusions


Performance on Multiple Input PuzzlesPerfect Reconstruction Percentage

2 3 4 50

5

10

15

20

25

30


Per

fect

Rec

onst

ruct

ion

(%)

Effect of the Number of Input Puzzles on Perfect Reconstruction


51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


49Solver Comparison

Conclusions


Performance on Multiple Input PuzzlesResults Summary

I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.

I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.

I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles

I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.

I Many of the extra puzzles were relatively insignificant in size

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


49Solver Comparison

Conclusions








51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


49Solver Comparison

Conclusions








Conclusions

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

50Conclusions


Conclusions

I This thesis presented a fully-automated solver forMixed-Bag puzzles.

I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.

I Introduced the first set of solver quality metrics forMixed-Bag puzzles.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

50Conclusions


Conclusions

I This thesis presented a fully-automated solver forMixed-Bag puzzles.

I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.

I Introduced the first set of solver quality metrics forMixed-Bag puzzles.

51




Thesis Goals



Segmentation

Stitching



Final Assembly


Neighbor Accuracy


Solver Comparison

51Conclusions


Summary of Topics

Thesis GoalsIntroduction

Previous WorkMixed-Bag Solver

AssemblerSegmentationStitchingHierarchical ClusteringFinal Seed Piece SelectionFinal Assembly

Quantifying QualityDirect AccuracyNeighbor Accuracy

Experimental ResultsInput Puzzle CountSolver Comparison

Conclusions

Appendix

51




Conclusions53Future Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results


ConclusionsFuture Work

I Improved AssemblerI Prioritize placement using multiple best buddies

I Address placement performance in regions with low bestbuddy density

I Dynamic determination of the segment clusteringthreshold

I Expanded stitching piece selection

51






Previous Work



Ten Puzzle Results







51






Previous Work



Ten Puzzle Results







51






Previous Work



Ten Puzzle Results


List of References I

[1] T. Altman, “Solving the jigsaw puzzle problem in linear time,” Applied Artificial Intelligence, vol. 3, pp. 453–462, Jan. 1990.

[2] E. D. Demaine and M. L. Demaine, “Jigsaw puzzles, edge matching, and polyomino packing: Connections andcomplexity,” Graphs and Combinatorics, vol. 23 (Supplement), pp. 195–208, June 2007.

[3] A. C. Gallagher, “Jigsaw puzzles with pieces of unknown orientation,” in Proceedings of the 2012 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), CVPR ’12, pp. 382–389, IEEE Computer Society, 2012.

[4] G. Paikin and A. Tal, “Solving multiple square jigsaw puzzles with missing pieces,” in Proceedings of the 2015 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’15, IEEE Computer Society, 2015.

[5] Z. Hammoudeh, “Github - puzzle solver thesis repository.” https://github.com/ZaydH/Thesis.(Accessed on 10/23/2016).

[6] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “Computational jigsaw puzzle solving.”https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/, 2011.(Accessed on 05/01/2016).

[7] A. Olmos and F. A. A. Kingdom, “McGill calibrated colour image database.” http://tabby.vision.mcgill.ca/, 2005.(Accessed on 05/01/2016).

[8] P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining, (First Edition).Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2005.

[9] T. S. Cho, S. Avidan, and W. T. Freeman, “A probabilistic image jigsaw puzzle solver,” in Proceedings of the 2010 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’10, pp. 183–190, IEEE Computer Society,2010.

[10] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “A fully automated greedy square jigsaw puzzle solver,” in Proceedingsof the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’11, pp. 9–16, IEEEComputer Society, 2011.

[11] H. Braxmeier and S. Steinberger, “Pixabay.” https://pixabay.com/.(Accessed on 05/15/2016).

https://github.com/ZaydH/Thesis

https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/

http://tabby.vision.mcgill.ca/

https://pixabay.com/

51





Introduction55Puzzle Types

Previous Work



Ten Puzzle Results



There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).


I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles


51






Previous Work



Ten Puzzle Results









51






Previous Work



Ten Puzzle Results









51






Previous Work



Ten Puzzle Results









51






Previous Work



Ten Puzzle Results









51






Previous Work



Ten Puzzle Results









51






56Previous Work



Ten Puzzle Results


Previous Work

I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver

I Graphical model-based Type 1 solver

I Puzzle dimensions are known

I Used one or more anchor pieces

I Defined quality metrics for Type 1 and Type 2 puzzles

I Established the standard comparative test conditions

I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver

I Eliminated the use of anchor pieces

I Created multiple solver benchmarks of various sizes

51






56Previous Work



Ten Puzzle Results


Previous Work

I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver

I Graphical model-based Type 1 solver

I Puzzle dimensions are known

I Used one or more anchor pieces

I Defined quality metrics for Type 1 and Type 2 puzzles

I Established the standard comparative test conditions

I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver

I Eliminated the use of anchor pieces

I Created multiple solver benchmarks of various sizes

51






Previous Work

57Best BuddiesBest Buddy Density


Ten Puzzle Results


IntroductionBest Buddies

I Basis of all Modern Jig Swap Solvers: The morecompatible two pieces are, the more likely they are to beadjacent.

I Best Buddies: A pair of puzzles pieces that are morecompatible with each other on their respective sides thanthey are to any other piece [10]

I Note: Not all puzzle pieces will have a best buddy.

∀pk∀sz ,C(pi , sx ,pj , sy ) ≥ C(pi , sx ,pk , sz)

and

∀pk∀sz ,C(pj , sy ,pi , sx ) ≥ C(pj , sy ,pk , sz)

(10)

I Importance of Best Buddies: Key adjacency indicator

51






Previous Work

Best Buddies58Best Buddy Density


Ten Puzzle Results


Quantifying Solver QualityBest Buddy Density

I Best Buddy Density (BBD): A metric for quantifying thebest buddy profile of an image that is independent ofimage size.

BBD =b

n · q(11)

I A greater BBD means the pieces are more differentiatedmaking the puzzle easier to solve.

51






Previous Work



Ten Puzzle Results


Best Buddy DensityVisualization

Visualizing Best Buddy DensityI Transform each puzzle piece into a square consisting of

four isosceles triangles.

I Color each triangle according to whether the adjacentpiece is a best buddy. The scheme used in this thesis:

No BestBuddy

Non-AdjacentBest Buddy

AdjacentBest Buddy

No PiecePresent

I Areas with higher best buddy density will have more greentriangles.

51






Previous Work



Ten Puzzle Results


Best Buddy DensityVisualization Example

Original Image [11]

Best Buddy Visualization

51






Previous Work



Ten Puzzle Results


Best Buddy DensityVisualization Example

Original Image [11] Best Buddy Visualization

51






Previous Work


Experimental Results61Single Input Puzzle

Ten Puzzle Results


Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images

Perfectly ReconstructedImage (a)

Best Buddy Visualization (a)

Misclassified Image (b)

Best Buddy Visualization (b)

51






Previous Work


Experimental Results61Single Input Puzzle

Ten Puzzle Results


Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images

Perfectly ReconstructedImage (a) Best Buddy Visualization (a)

Misclassified Image (b) Best Buddy Visualization (b)

51






Previous Work



62Ten Puzzle Results


Experimental ResultsSolving More than Five Puzzles

I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.

I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.

I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.

51






Previous Work









51






Previous Work









51






Previous Work





Ten Puzzle ResultsSummary

I Paikin & TalI Seed of nine images came from just three input images

I SEDAS and EDAS greater than 0.9 for only one image

I No perfectly reconstructed images

I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images

I Four images perfectly reconstructed

I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually

51






Previous Work





Ten Puzzle ResultsSummary

I Paikin & TalI Seed of nine images came from just three input images

I SEDAS and EDAS greater than 0.9 for only one image

I No perfectly reconstructed images

I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images

I Four images perfectly reconstructed

I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually

Documents

A Fully-Automated Solver for Multiple Square …...A Fully-Automated Solver for Multiple Square Jigsaw Puzzles Using Hierarchical Clustering October 27, 2016 Zayd Hammoudeh [email protected]