WCPM 1

Chang-Tsun Li

Department of Computer Science

University of Warwick

UK

Image Clustering Based on Camera Fingerprints

WCPM 2

Digital Image Acquisition Process

Scene Post-Processing

Lens

Sensor CFA InterpolationCFA Photo

CFA: Colour Filter Array

R G

G B

Bayer CFA R G

G B

mapping of CFA to sensor pixels

R G

G B

R G

G B

R G

G B

R G

G B

R G

G B

WCPM 3

What is Camera Fingerprint• Lens aberration

• Sensor pattern noise

• Colour filter array (CFA) interpolation artefacts

• Camera response function

• Quantisation table of JPEG compression

Scene Post-Processing

Lens

Sensor CFA InterpolationCFA Photo

WCPM 4

Camera Fingerprint for Multimedia ForensicsMultimedia Forensics: The use of “fingerprints” left in images by the imaging devices for

•source device identification

•source device linking

•content integrity verification

•image classification

WCPM 5

What is Sensor Pattern Noise Sensor Pattern Noise (SPN) is the noise left in the

images by the sensors of digital imaging devices such as cameras, camcorders and scanners.

SPN is mainly caused by – manufacturing imperfection and – different sensitivity of pixels to light due to the

inhomogeneity of silicon wafers. Sensors made from the same silicon wafer possess

unique SPN because of the non-uniform imperfection.

SPN can differentiate cameras of the same model.

WCPM 6

“Traditional” SPN Extraction Method Lukáš et al’s model for SPN extraction (IEEE TIFS

2006)

– I is the original image– I’ is the low-pass filtered version of I by the Weiner

filter applied in the wavelet transform domain– n is the extracted SPN

SPN is the high-frequency component of the image.

)(_' IfilterWeinerI

),('),( jiIjiIn

WCPM 7

Interference from Scene Details Scene details, e.g., brick walls, tree leaves, or other kinds of

textures, contribute to the high-frequency components of images.

a contaminated SPN

natural image

SPN

a clean SPN

WCPM 8

SPN Enhancement at Warwick• C.-T. Li, "Source Camera Identification Using Enhanced Sensor

Pattern Noise," IEEE Trans. on Information Forensics and Security, June 2010

• C.-T. Li and Y. Li, "Color-Decoupled Photo Response Non-Uniformity for Digital Image Forensics," IEEE Trans. on Circuits and Systems for Video Technology, 2012

• X. Lin and C.-T. Li, "Preprocessing Reference Sensor Pattern Noise via Spectrum Equalization," IEEE Trans. on Information Forensics and Security, 2016 .

• X. Lin and C.-T. Li, "Enhancing Sensor Pattern Noise via Filtering Distortion Removal," IEEE Signal Processing Letter, accepted for publication in 2016

WCPM 9

Image Classification/Clustering

Scenario: A forensic investigator •has a large set of images taken by an unknown number of unknown digital cameras and •wishes to cluster those images into a number of classes, each including the images acquired by the same camera.

Each data point represents one image Each cluster present one unknown device

WCPM 10

Challenges Facing Image Classification

The forensic investigator does not have the cameras that have taken the images to generate reference SNPs for comparison.

No prior knowledge about the number and types of the imaging devices are available.

With a large dataset, exhaustive fingerprint comparison is computationally prohibitive.

Given the shear number of images, analysing each image in its full size is computationally infeasible.

WCPM 11

Image Classification – a MRF ApproachStep 1. Extract and enhance the fingerprint of each block cropped from the images

Step 2. Establish a similarity matrix ρ for a Focus Set of M images

Step 3. Train the classifier based on the similarity matrix ρ. For each fingerprint i, which is treated as a random variable

3.1. Assign a unique random class label

3.2. Calculate a reference similarity (i.e., a “soft” threshold)

3.3. Establish a membership committee (neighbourhood)

3.4. Update the class label iteratively based on the information from the membership committee until there are no changes of class labels to any SPN throughout a entire iteration

Step 4. Classify the rest of the dataset using the classifier

WCPM 12

Establishing Similarity Matrix To establish an M × M similarity matrix ρ, the similarity between any two enhanced SPNs i and j in the Focus Set is calculated using

},...,3,2,1{, , )()(

),( Mjinnnn

nnnnji

jjii

jjii

i , j 1 2 3 4 …. M

1 1.00

2 1.00

3 1.00

4 1.00:: 1.00

M 1.00

WCPM 13

Classifier Training• Each fingerprint (SPN) is treated as a random variable. 3.1. Assign a unique random class label to each SPN 3.2. Calculate a reference similarity r Normally intra-class similarities > inter-class similarities.

•A similarity less than r indicates that the two images are taken by different devices, otherwise by the same device.

similarity Similarity

inter-class similarity intra-class similarity

μ1 r μ2

221

r

WCPM 14

3.3. Establish a membership committee For each SPN i , a membership committee Ci with c SPN

members from the focus set that are most similar to i is established.

Classifier Training

vvvvvv

×

WCPM 15

• fi : class label of SPN i

• C• ρ(i, Ci ) is the similarities between

SPN i and the members of Ci, i.e.,

3.4. Update the class label iteratively according to p(fi |ρ(i, Ci ), Li) until there is no change of class label to any SPN in x consecutive iterations

Classifier Training

• p(fi |ρ(i, Ci ), Li): probability of assigning fi given the conditions

• ri: reference similarity (“soft” threshold) of I

}|{}{ ijii CjffL

}|),({),( ii CjjiCi

)]),,(,(exp[1)),,(|( iiiii

iii LCifUZ

LCifp

)]),,(,(exp[

ii Lf

iiiii LCifUZ

WCPM 16

similarity Similarity

inter-class similarity intra-class similarity

μ1 r μ2

The combination of the s(.) and ρ(.) says, • a penalty (i.e. positive value) will be incurred

if ρ(i,j) > r and a different label than fi is to be assigned to i or

if ρ(i,j) < r and the same label as fj is to be assigned to i• a reward (i.e. negative value) will be given

if ρ(i,j) < r and a different label than fi is to be assigned to i or

if ρ(i,j) > r and the same label as fj is to be assigned to i

Objective Function / Cost Function

iCj

ijiiiii rjiffsLCifU ),(),()),,(,(

ji

jiji ff

ffffs

if , 1 if , 1

),(

WCPM 17

Image Classification

The centroids of the image clusters provided by the classifier training process at the end of Step 3.4 are used to classify the images.

To classify an image x, we compare the similarity of its SPN to the centroid of each identified cluster and classify it to the class with its centroid closest to the image.  

WCPM 18

Clustering in progress …..

Classifier Training - Simulation

Initial label configure: Each pattern is assigned an unique label / colour

WCPM 19

Final Classification

WCPM 20

Experimental Results

c

Block Size

256 × 256 256 × 512 512 × 512

Focus set size (M) Focus set size (M) Focus set size (M)

120 300 120 300 120 300M-1

M/2

M/3

M/4

M/5

Table 1. Classification error rate. c is the size of the membership committee.

c: the size of the membership committeeM: the size of the focus set

8.889

8.333

8.333

8.333

8.333

4.000

4.000

4.000

4.000

4.000

3. 778

2.333

2.333

2.333

3. 778

1.333

1.333

1.333

1.333

1.222

1.444

1.444

1.444

1.444

1.444

1.444

1.444

1.444

1.556

1.444

Misclassification rates: 1200 images taken by six cameras, each taking 200. Class identification stops when there are no class label changes throughout an iteration.

A misclassification rate in the range (1.2 ~ 1.6) is likely to be the best the system can achieve.

WCPM 21

Conclusions

Multimedia forensics using “fingerprint” left in the images by the imaging devices has emerged as a new area of research in the last few years.

Sensor pattern noise (SPN) is one of the most promising types of fingerprint.

The “traditional” SPN extraction method is unable to cope with the interference of scene details.

The proposed classifier is feasible, but is unable to classify images without clean SPNs provided by the proposed SPN enhancer.