1

30th AAAI Conference on Artificial Intelligence, Phoenix, Arizona - 2016

Differential Privacy Preservation for Deep Auto-Encoders: an Application of

Human Behavior Prediction

NhatHai Phan1, Yue Wang2, Xintao Wu3, and Dejing Dou1

1 University of Oregon, 2 University of North Carolina Charlotte, 3University of Arkansas

{haiphan,dou}@cs.uoregon.edu, ywang91@uncc.edu, xintaowu@uark.edu

2

Outline• Deep Learning and Deep Auto-Encoders

• Differential Privacy Preservation for Deep Auto-Encoders– Deep Private Auto-encoders (dPA)

• Application– YesiWell Health Social Network– Human Behavior Prediction

• Conclusions and Future Works

3

Deep Learning

Pixels

1st Layer“Edges”

2nd Layer“Object parts”

3rd Layer“Objects”

[Andrew Ng]iv

1h

2h

3h

y

1W

2W

3W

4W

v

4

Deep Auto-Encoders• Data reconstruction

• Softmax layer

Auto-encoder

y

0WTW0

v

1h

1

11WTW1

2h1

1

……

…

)(kW

TkW 1

Deep Auto-encoder

0WTW0

v

1h

1

1

)(h k

||

1 1

)~1log()1(~log),(D

i

d

jijijijij vvvvWDR

||

1

)ˆ1log()1(ˆlog),(TY

iiiiiT yyyyYC

v~

5

Motivation

• Deep learning– Social media, social network analysis,

bioinformatics, medicine and healthcare. • Privacy issues? – Users' personal and highly sensitive data, such as

clinical records, user profiles, photo, etc.• Differential privacy– Deep Private Auto-Encoders

6

- Differential Privacy Definition

• The goal of a privacy-preserving statistical database is to – learn properties of the population as a whole, – while protecting the privacy of the individuals in the

sample

• Differential privacy (preserving algorithm)– maximize the accuracy of queries from statistical

databases– minimize the chances of identifying its records

7

Challenges

• Unprecedented work

• A non-trivial task– R(D,W) is complicated– The algorithm must be

efficient on large datasets

• Guarantee the potential to use unlabeled data in a dPA model

Amount of DataPe

rform

ance

Most learning algorithms

New AI methods(deep learning)

[Andrew Ng, 2015]

Deep Private Auto-Encoders

• Functional Mechanism– injecting Laplace noise Lap(Δ/ε) into

polynomial coefficients of polynomial functions

y

0WTW0

v

1h

1

1

11WTW1

2h1

1

……

……

)(kW

1

Deep Private Auto-encoder

8

Polynomial Approximation

||

1 1 )~1log()1(

~log),(

D

i

d

j ijij

ijij

vvvv

WDR

v~

9

•

• Apply Functional Mechanism to inject Laplace noise Lap(Δ/ε)

Polynomial Approximation Taylor Expansion [Arfken 1985]

Arfken, G. 1985. In Mathematical Methods for Physicists (Third Edition). Academic Press.

||

1 1 22

1

)2(

2

1

2

1

)1()0(

))(!2)0(

(

))0(()0(),(ˆ

D

i

d

jij

l

lj

ijl l

ljlj

hWf

hWffWDR

||

1 1

)~1log()1(~log),(D

i

d

jijijijij vvvvWDR

Taylor Expansion Error?

10

Approximation Error Bounds

• Approximation error bounds

• Our algorithm can be applied on large datasets

2

2

2

2

)1(12)~,(~)ˆ,(~)1(12)~,(~)ˆ,(~

),(ˆminargˆ);,(~minarg~

eeeeYCYC

deeeeWDRWDR

WDRWWDRW

TT

WW

#input units - derror

11

Outline• Deep Learning and Deep Auto-Encoders

• Differential Privacy Preservation for Deep Auto-Encoders– Deep Private Auto-encoders (dPA)

• Application– YesiWell Health Social Network– Human Behavior Prediction

• Conclusions and Future Works

12

Semantic Mining of Activity, Social, and Health Data (NIH/NIGMS Funded in 2013, R01 GM103309) (PI: Dou)

13

Human Behavior Prediction

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

t1t 1tDecrease exercise Increase exercise

Dataset, Features, and Task• YesiWell dataset

– 254 users– Oct 2010 – Aug 2011

• BMI • Wellness Score

• Prediction Task: Try to predict whether a YesiWell user will increase or decrease exercises in the next week compared with the current week.

14

2))(()( mheightkgmass

)1()()/()(

3423

121

cHbAULDLUHDLTGUBMIUy

15

dPA-based Human Behavior Prediction (dPAH)

IndividualFeatures

IndividualPast Features

SocialCorrelations

1h

1

16

Human Behavior PredictionExperimental Results

• Do not enforce differential privacy– CRBM, SctRBM– Deep Auto-Encoder (dA)– Truncated Deep Auto-

Encoder (TdA)

• Do enforce differential privacy– Functional Mechanism (FM)– DPME, Filter-Priority (FP)

• dPAH: 83.39% – (ε, sampling rate) = (1, 0.4)

data

17

Conclusions

• Deep private auto-encoders– Human behavior prediction: 83.392%

• The proposed algorithm can work for– Deep Belief Networks– Convolutional Neural Networks

• Extract sensitive information from a deep private auto-encoder

18

30th AAAI Conference on Artificial Intelligence, Phoenix, Arizona - 2016

SMASH Project: http://aimlab.cs.uoregon.edu/smash/

YesiWell Health Social Network

Thank you!