1

Privacy-Preserving Collaborative Deep Learningwith Irregular Participants

Lingchen Zhao, Yan Zhang, Qian Wang, Member, IEEE, Yanjiao Chen, Member, IEEE,Cong Wang, Member, IEEE and Qin Zou, Member, IEEE

Abstract—With large amounts of data collected from massivesensors, mobile users and institutions becomes widely available,neural network based deep learning is becoming increasinglypopular and making great success in many application scenarios,such as image detection, speech recognition and machine transla-tion. While deep learning can provide various benefits, the datafor training usually contains highly sensitive information, e.g.,personal medical records, and a central location for saving thedata may pose a considerable threat to user privacy.

In this paper, we present a practical privacy-preserving collab-orative deep learning system that allows users (i.e., participants)to cooperatively build a collective deep learning model with dataof all participants, without direct data sharing and central datastorage. In our system, each participant trains a local modelwith their own data and only shares model parameters with theothers. To further avoid potential privacy leakage from sharingmodel parameters, we use functional mechanism to perturb theobjective function of the neural network in the training processto achieve ε-differential privacy. In particular, for the first time,we consider the possibility that the data of certain participantsmay be of low quality (called irregular participants), and proposea solution to reduce the impact of these participants whileprotecting their privacy. We evaluate the performance of oursystem on two well-known real-world data sets for regressionand classification tasks. The results demonstrate that our systemis robust to irregular participants, and can achieve high accuracyclose to the centralized model while ensuring rigorous privacyprotection.

I. INTRODUCTION

IN the past few years, deep learning has tremendouslyrevolutionized the fields of machine learning and artifi-

cial intelligence, achieving a much better performance thantraditional machine learning methods in various applications,e.g., image processing [1], speech recognition [2], [3], canceranalysis [4], [5], and the game of Go [6]. The great success ofdeep learning is largely owing to the availability of massivedata for training the neural networks. In general, the deeplearning model will be more accurate if trained with morediverse data, which motivates companies and institutions tocollect as much data as possible from their users, usuallygenerated by sensors on users’ personal devices, e.g., GPS,cameras, gyroscopes, and heart rate sensors. From the privacyperspective, however, user-generated data is usually highly

L. Zhao, Y. Zhang and Q. Wang are with the Cyber Science andEngineering, Wuhan University, China, email: {lczhaocs, stong, qian-wang}@whu.edu.cn.

Y. Chen and Q. Zou are with the School of Computer Science, WuhanUniversity, China, email: {chenyanjiao, qzou}@whu.edu.cn.

C. Wang is with the Computer Science Department at City University ofHong Kong, Kowloon, Hong Kong, China, email: congwang@cityu.edu.hk.

sensitive, e.g., location information, personal medical records,and social relationships. To gather these sensitive data at acentralized location will raise serious concerns about privacyleakage. A recent regulation of EU [7] stipulates that compa-nies should carefully collect and use users’ personal data, andusers have the right to require the company to permanently“forget” their data. The bill also prohibits any automatedindividual decision-making (e.g., personal financial situation,personal health condition, and location prediction) based onthe data, which may greatly affect the machine learningtasks by companies. In addition, in many sectors especiallymedical industry, sharing personal data is forbidden by lawsor regulations. recently, where multiple parties host some ofthe data, contribute their computing capacities and finally learna collective machine learning model which benefits from allparties.

To glean the benefit of machine learning while protectinguser privacy, there is a rising interest in designing privacy-assured machine learning algorithms from both academiaand industry. Existing solutions regarding traditional machinelearning algorithms mainly exploit the intrinsic features of thealgorithms, e.g., strictly convex objective functions. Privacy-preserving techniques such as secure multi-party computationor differential privacy have been applied to linear and logisticregression analysis [8], [9], k-means clustering [10], supportvector machines [11], and crowd machine learning [12]. In re-cent years, privacy-preserving deep learning has received muchattention from the research community. In [13], cryptographictool, namely homomorphic encryption, was first applied toconvolutional neural networks (CNNs), but the solution is acentralized one and requires extensive computation resources.Subsequently, many works tried to improve the performanceof deep learning on encrypted data, such as [14], [15], butall these schemes introduce heavy overheads to the originalcomputation on plaintext data. In [16], only the intermediaterepresentations obtained by a local neural network model arepublished to hide the private data, but the scheme did notprovide a rigorous privacy guarantee. In [17], a differentiallyprivate stochastic gradient descent algorithm and a mechanismto accurately track the privacy loss during training weredesigned, which could train deep neural networks with amodest privacy budget and a manageable model quality, butthe scheme still depends on the number of training epochsand some empirical parameters (e.g. lot size, clipping bound).In [18], differential privacy was applied to a specific deeplearning model, deep auto-encoder, and sensitivity analysisand noise insertion were conducted on data reconstruction and

arX

iv:1

812.

1011

3v1

[cs

.CR

] 2

5 D

ec 2

018

2

cross-entropy error objective functions. In [19], a frameworkcalled DSSGD was proposed to ensure differential privacyfor distributed deep networks. Some other attacks that tryto extract alternative information from the machine learningprocess are proposed in recent years, e.g., model inversionattacks [20], membership inference attacks [21], and modelextraction attacks [22]. In particular, by utilizing generativeadversarial network (GAN), it is claimed in [23] that adistributed deep learning approach cannot protect the trainingsets of honest participants even if the model is trained ina privacy-preserving manner by [19]. However, it is foundin [24] that [23] does not truly break the rigorous differentialprivacy.

In this paper, we investigate the problem of collaborativedeep learning with strong privacy protection while maintaininga high data utility. In collaborative deep learning, users (i.e.,participants) cooperatively learn a collective deep learningmodel that can benefit from the data of all users. Our work ismost related to [17]–[19], but is quite different in several ways.The proposed schemes in [17] and [18] were not designed forthe collaborative deep learning. In [19], participants only sharea subset of parameters with the others to reduce communi-cation costs and differential privacy is achieved by insertingnoises to truncated weights, but there are some limitations.The consumed privacy budget during the learning process isrelatively high for each single parameter. The total privacybudget is proportional to the number of parameters, whichmay be as high as tens of thousands in deep learning models.The parameter that tunes the fraction of uploaded gradientsis used to quantify the privacy, but each pixel in the trainingdata may be revealed by multiple gradients. Furthermore, it isnot considered that the data quality of certain participants maybe poor, which may degrade the performance of collaborativelearning.

To address the above issues, in our proposed mechanism,each participant (e.g., mobile user, medical institution) main-tains a local neural network model and a local dataset thatmay be highly sensitive. Instead of sharing local data withthe central server, the participant only uploads the updatedparameters of the local model based on the local dataset. Thecentral server derives the global parameters for the collectivemodel according to the updates from all participants. Param-eter sharing can prevent direct exposure of the local data, butmay indirectly disclose the information of the sensitive data.To solve this problem, we utilize differential privacy [25] toobtain the sanitized parameters to minimize privacy leakage.Unlike [19] where noise is directly injected to the gradients,we apply functional mechanism [26] to perturb the objectivefunction of the neural network, and obtain the sanitizedparameters by minimizing the perturbed objective function.In collaborative learning, the quality of data contributed bydifferent participants may be diversified. Different terminaldevices or persons may have different capacities to generatethe training data, and there may exist unpredictable randomerrors during data collection and storage. Participants with lowquality data are referred to as irregular participants (discussedin detail in Section II-A). To make the learning process fairand non-discriminative, we consider the “data quality” as one

LocalDataset

Participant

LocalDataset

Participant

LocalDataset

Participant Participant

LocalDataset

(Low quality)

Auxiliary Dataset

Server

Irregular

Fig. 1: Collaborative deep learning system with irregularparticipants.

of the privacy concerns of participants, which should not beinferred by other participants during the learning process. Weadopts exponential mechanism to protect this privacy whileeffectively learning an accurate model. Our main contributionsare summarized as follows.• We make the first attempt to investigate the problem of

privacy-preserving collaborative deep learning taking intoaccount the existence of irregular participants.

• We present a novel scheme called SecProbe, which allowsparticipants to share model parameters and deals withirregular participants by utilizing exponential mechanism.SecProbe can protect the privacy of data quality of eachparticipant while effectively learning an accurate model.

• We derive the approximate polynomial form of the objec-tive function in a neural network with two different lossfunctions, and use functional mechanism to inject noisesto the coefficients to achieve differential privacy withoutconsuming too much privacy budget. We show that it iseasy to extend and apply our method to networks withmore layers.

• We evaluate the performance of SecProbe on two well-known real-world data sets for regression and classifica-tion tasks. The results demonstrate that SecProbe is robustto irregular participants, and achieves high accuracy closeto the solution based on the centralized model, whileproviding a rigorous privacy guarantee.

II. PROBLEM STATEMENT AND PRELIMINARIES

In this section, we introduce our problem statement andsome preliminary knowledge of deep learning, differentialprivacy and functional mechanism used in our design.

A. Problem Statement

In this paper, we consider the problem of privacy-preservingdistributed collaborative deep learning. As shown in Figure 1,in our model, each collaborative participant may have theirown sensitive data and it would like to learn a model benefitingfrom both its own data and the others.

3

In particular, instead of making the assumption that allthe participants are “regular”, i.e., the data held by eachparticipant is balanced and has the same or similar quality, weconsider a more practical model where there may exist a smallgroup of participants who are “irregular” during some phasesof the whole learning process. That is, a portion of data heldby irregular participants is not always as accurate as others,and thus their uploaded parameters may disturb the learningaccuracy. In our daily lives, irregular participants are commonin a collaborative learning system. Consider a typical scenariowhere several hospitals aim to learn a model together forcancer prediction for patients. There may exist non-negligiblegaps in the quality of data among different hospitals sincea rich-experienced chief physician with advanced medicaldevices in a high-rate hospital will be more likely to produceaccurate data than an junior physician with low-end of devicesin an ordinary hospital. Note that, it does not mean that thedata in the ordinary hospital are all and/or always “bad”.Actually, every participant might have some bad data in somephases of their training process when more and more dataare being gathered into their local date set, because there areso many possibilities to go wrong in the data generation andstorage procedures. Consequently, the existence of irregularparticipants will bring non-ignorable disturbance during thecollaborative training process, which may finally result in aninaccurate or even useless model. In this paper, we aim toreduce the impact of inaccurate data on the accuracy of thelearned model in the presence of irregular participants. In ourscheme, we assume the irregular participants are not malicious,i.e., they do not deliberately inject false data into the system.Similar to the commonly-used curious-but-honest adversarialmodel, we refer to irregular participants as reckless-but-honest.In addition, we show that our scheme is secure against activeadversaries, i.e., malicious participants, in Section III-D.

We assume the server is honest, and there are two majorprivacy concerns in our model: 1) privacy of participants’local data, which may be revealed by the parameters of thelocal model of each participant and inferred by the server andother participants and 2) privacy of the quality of participants’data, i.e., it is necessary and important to hide a participant’sreal data quality from others in order to preserve its reputationand achieve a non-discriminative collaborative learning envi-ronment. The data quality of participants may be inferred byothers by observing the global parameters generated by theserver.

Keep the above performance requirements and privacy con-cerns in mind, our ultimate goal is to design a distributedcollaborative deep learning system that can jointly learn anaccurate model while protecting both the privacy of theparticipants’ local data and quality privacy of the participants’data.

B. Deep learning

Broadly speaking, deep learning, based on artificial neuralnetworks, aims to learn and extract high-level abstractions indata and build a network model to describe accurate relationsbetween inputs and outputs. Common deep learning models

Input Layer

Hidden Layers

Output Layer

x

jz

Fig. 2: A neural network with multiple hidden layers.

are usually constructed by multi-layer networks, where non-linear functions are embedded, so that more complicatedunderlying features and relations can be learned in differentlayers. Interested readers can refer to thorough surveys orreviews in [27], [28].

There are multiple forms of deep learning models, e.g.,multi-layer perceptron (MLP), convolutional neural network(CNN), recurrent neural network (RNN). Different modelsfit for different types of problems, and among all of thosemodels, MLP is a very common and representative form ofdeep learning architecture. Specifically, MLP is a kind offeed-forward neural network, where each neuron receives theoutputs of neurons from the previous layer. Figure 2 shows atypical MLP with multiple hidden layers. Each neuron has anactivation function which is usually non-linear. As shown, fora neuron in a hidden layer, say j, the output of the neuronis calculated by the equation hj = f(W

(1)j x), where W (1)

j isthe weight vector which determines the contribution of eachinput signal to the neuron j, x is the input of the modeland f is the activation function. The activation function isusually non-linear in order to capture the complicated non-linear relation between the output and input. Typical examplesare sigmoid function f(x) = (1 + e−x)−1, ReLU functionf(x) = max(0, x), and hyperbolic tangent f(x) = e2x−1

e2x+1 .In our work, we will focus on a MLP model, where ReLUfunction is applied.

Training a neural network, i.e., learning the parameters(weights) of the network, is a non-convex optimization prob-lem. The typical algorithms used to solve the problem aredifferent types of gradient descent methods [29]. In this paper,we will consider a supervised learning task, e.g., regressionanalysis, and assume the output of the network is z. Supposethe data we use to train the network is a tuple (xi, yi),where xi is used to be the network input and yi is thelabel. Consequently, we can use loss (objective) function tomeasure the difference between the network output and thereal training label, e.g., Errori = (zi−yi)2. We then can useback propagation [30] algorithm to propagate the error backto the neurons, compute the contribution of each neuron tothis error, and adjust the weights accordingly to reduce the

4

training error. The adjustment procedure for a weight, say wj ,is wj = wj − l ∂Errori∂wj

, where l is the learning rate.Among various gradient descent algorithms, stochastic gra-

dient descent (SGD) [31] is considered to be especially fit foroptimizing highly non-convex problem for its high efficiencyand effectiveness. This algorithm brings stochastic factors intothe training process, which helps the model to escape from lo-cal optimum. For a large training data set, SGD first randomlysamples a small subset (mini-batch) of the whole data set, thencomputes the gradients over the mini-batch and updates theweights, e.g., for weight wj . After one iteration on the mini-batch, the new wj is computed by wj = wj− l ∂Errorb∂wj

, whereErrorb is the loss function computed on the mini-batch b. Inour work, we will apply SGD to each participant to train itslocal model.

C. Differential Privacy

Differential privacy has become a de facto standard privacymodel for statistics analysis with provable privacy guarantee,and has been widely used in data publishing [32], [33] anddata analysis [34], [35]. Intuitively, a mechanism satisfiesdifferential privacy if its outputs are approximately the sameeven if a single record in the dataset is arbitrarily changed, sothat an adversary infers no more information from the outputsabout the record owner than from the dataset where the recordis absent.

Definition 1 (Differential Privacy [25]): A privacy mech-anism M gives ε-differential privacy, where ε > 0, if for anydatasets D and D

′differing on at most one record, and for

all sets S ⊆ Range(M),

Pr[M(D) ∈ S] ≤ exp(ε) · Pr[M(D′) ∈ S], (1)

where ε is the privacy budget representing the privacy levelthe mechanism provides. Generally speaking, a smaller εguarantees a stronger privacy level, but also requires a largerperturbation noise.

Definition 2 (Sensitivity [36]): For any function f : D →Rd, the sensitivity of f w.r.t. D is

∆(f) = maxD,D′∈D

||f(D)− f(D′)||1 (2)

for all D and D′

differing on at most one record.Laplace mechanism is the most commonly used mechanism

that satisfies ε-differential privacy. Its main idea is to addnoise drawn from a Laplace distribution into the datasets tobe published.

Theorem 1 (Laplace Mechanism [36]): For any functionf : D → Rd, the Laplace Mechanism M for any datasetD ∈ D

M(D) = f(D) + 〈Lap(∆(f)/ε)〉d (3)

satisfies ε-differential privacy, where the noise Lap(∆(f)/ε)is drawn from a Laplace distribution with mean zero and scale∆(f)/ε.

Obviously, Laplace mechanism only fits for numeric query.Thus, for the query whose outputs are not numeric, Mcsherryet al. [37] proposed Exponential mechanism that selects anoutput r from the output domain R.

Theorem 2 (Exponential Mechanism [37]): Let ∆u bethe sensitivity of the utility function u: (D × R) → R, themechanism M for any dataset D ∈ D,

M(D,u) = choose r ∈ R with probability ∝ exp(εu(D, r)

2∆u)

(4)gives ε-differential privacy.

This theorem implies that Exponential mechanism can makehigh utility outputs exponentially more likely at a rate thatmainly depends on the utility score such that the final outputwould be approximately optimum with respect to u, andmeanwhile give rigorous privacy guarantee.

The composition properties of differential privacy provideprivacy guarantee for a sequence of computations.

Theorem 3 (Sequential Composition [38]): Let M1, M2,· · · , Mr be a set of mechanisms and each Mi providesεi-differential privacy. Let M be another mechanism thatexecutesM1(D), · · · ,Mr(D) using independent randomnessfor each Mi. Then M satisfies (

∑i εi)-differential privacy.

Theorem 4 (Parallel Composition [38]): Let Mi eachprovides εi-differential privacy. A sequence of Mi(Di) overdisjoint datasets Di provides max(εi)-differential privacy.

These theorems allow us to distribute the privacy budgetamong r mechanisms to realize ε-differential privacy.

D. Functional Mechanism

Functional mechanism (FM) [26] is a general framework forregression analysis with differential privacy. It can be seen asan extension of the Laplace mechanism which ensures privacyby perturbing the optimization goal of regression analysisinstead of injecting noise directly into the regression results.

A typical regression analysis on data set D returns a modelparameter w that minimizes the optimization (objective) func-tion fD(w) =

∑xi∈D f(xi, w). However, directly releasing

w would raise privacy concern, since the parameters revealinformation about data set D and function fD(w). In orderto achieve differential privacy, we use FM to firstly perturbthe objective function fD(w) (by exploiting the polynomialrepresentation of fD(w)), and then release the parameter wthat minimizes the perturbed objective function fD(w).

We assume w is a vector containing d values w1, . . . , wd.Let φ(w) denote the product of w1, . . . , wd, i.e., φ(w) =wc11 · w

c22 · · ·w

cdd , where c1, . . . , cd ∈ N . Let Φj(j ∈ N)

denote the set of all products of w1, . . . , wd with degree j,i.e., Φj = {wc11 · w

c22 · · ·w

cdd |∑dl=1 cl = j}. By the Stone-

Weierstrass Theorem [39], any continuous and differentiablefunction f(w) can always be written as a polynomial ofw1, . . . , wd, i.e., f(xi, w) =

∑Jj=0

∑φ∈Φj

λφxiΦ(w), where

λφxi∈ R denotes the coefficient of φ(w) in the polynomial,

and J ∈ [0,∞]. Similarly, we can derive the polynomialfunction of fD(w) as

fD(w) =

J∑j=0

∑φ∈Φj

∑xi∈D

λφxiΦ(w). (5)

Lemma 1: ( [26]) Let D and D′ be any two neighboringdatabases. Let fD(w) and f ′D(w) be the objective functions of

5

Algorithm 1 A high-level description of SecProbe

1: Build the models and initialize all parameters2: for each communication round do3: for each participant i do4: for iteration j = 1 to I do5: Run SGD independently on the local data set using

the perturbed loss function6: end for7: Upload Wi to the server8: end for9: The sever chooses to accept Wi according to the com-

puted utility score.10: The sever conducts model average to obtain Wnew and

send it to each participant.11: end for

regression analysis on D and D′, respectively. Then, we havethe following inequality

∆ =

J∑j=1

∑φ∈Φj

∥∥∥∥ ∑xi∈D

λφxi−∑xi∈D′

λφx′i

∥∥∥∥1

≤ 2 maxx

J∑j=1

∑φ∈Φj

‖λφx‖1.

In order to achieve ε-differential privacy, FM perturbsfD(w) by injecting Laplace noise into its polynomial coeffi-cients. According to Lemma 1, fD(w) is perturbed by injectingLaplace noise with scale of Lap(∆/ε) into the polynomialcoefficients λ(φ), where ∆ = 2 max

x

∑Jj=1

∑φ∈Φj

‖λφx‖1.

Then we can derive the model parameter w which minimizesthe perturbed function fD(w). In our work, we propose toutilize functional mechanism in our design to protect theprivacy of participants’ local data.

III. SECPROBE: PRIVACY-PRESERVING COLLABORATIVEDEEP LEARNING SYSTEM

A. System Architecture

In Figure 1, we assume there are N participants, and each ofthem has a sensitive data set for local training. The participantsaim to learn a common model, i.e., the architectures of thelocal models are identical, and the learning objectives arethe same. There are many deficiencies and difficulties ofcollecting all the data from participants in advance and trainingon the entire data set. Such complicated process of datacollection usually incurs high communication overhead, andparticipants may not be willing to directly upload their datato a third party from the perspective of privacy or businessconsideration. Therefore, the participants only exchange theparameters (weights) with others, and a server (e.g., a cloudservice provider) undertakes the job of communicating withparticipants, exchanging and storing parameters. In our model,we assume there exists a global model and an auxiliaryvalidation data set on the server. This data set can be verysmall, and it is easy to be obtained in practice. For example,the data can be collected from participants which have already

Algorithm 2 SecProbe on the server side

1: Initialize parameters W0 and send them to each participant

2: Wait for participants to upload their weights until thereare already M participants’ weights W1, W2, . . . , WM

3: Calculate the accuracy score u(G,D,m) for each upload-ing participant by running the model with weights fromeach of them over the auxiliary data set

4: Sample K participants from M withoutreplacement such that Pr[Selecting participant m] ∝exp( ε

2K∆uu(G,D,m))

5: Average the K weights and obtain Wnew = 1K

∑Ki=1 Wi

6: Send the new averaged weights to all participants7: Repeat steps 2-6 until there is no participant in the system

been expired with no privacy concern or publicly well-testedhand-classified data sets (such as MNIST [40]).

Algorithm 1 gives the high-level steps of SecProbe. Theserver and participants build their own models and initial-ize all the parameters before the learning starts. For eachcommunication round, the participants locally train their ownmodels using SGD in a differentially private way. After I timesof iteration, the participants upload the perturbed parametersto the server. The server then uses the auxiliary data tocompute a utility score for each participant, and then choosesto accept the parameters with certain probability. Next, theaveraged model parameters are computed and distributed toeach participant for the next round of local training. Note thatthe participant can terminate its training procedure and dropout from the system at any time if it believes its model isaccurate enough, and meanwhile a new participant can alsojoin into the system at anytime. We next describe the detailedprocedures of SecProbe on the sever side and the participantside, respectively.

B. SecProbe: The Server Part

Algorithm 2 gives the pseudocode of SecProbe on the severside. The server first initializes the parameters and waits for thelocal training results by each participant. When the number ofparticipants who upload their weights to the server reaches apre-fixed threshold M, the server stops receiving the uploadeddata and sends a stop signal to notify the other participantsthat there is no need to upload weights (in step 2). Theparameter M is used to control the number of participants thatthe server plans to utilize per round and meanwhile saves a lotof communication costs. Alternatively, this procedure can alsobe achieved by randomly assigning a set of M participantsat the beginning of each round. These two approaches havetheir own advantages and both can be used in our design.The former can intrinsically deal with the occurrence of faileduploads, while the latter can save a lot of computation costson the participant side. Without loss of generality, we adoptthe first approach in the description of Algorithm 2.

As discussed above, the existence of irregular participantsindicates that the parameters uploaded by them may be dis-ruptive, and it may reduce the accuracy of the global model.

6

To reduce their effect on the model accuracy, we measure thedata quality of these irregular participants by calculating autility score for each participant. Specifically, the server runsthe model on the auxiliary validation data set D with theweights of each of M participants respectively and obtains autility score for participant m. Let G = [W1, . . . , Wm] denotethe set of uploaded weights, where each item can be usedto infer the data quality of each participant. For a regressiontask, suppose the data set has d samples, we define a scoringfunction u(G,D,m) as

u(G,D,m) =1

d

d∑i

(1− |zi − yiyi|), (6)

where zi is the output of the model with parameter G(m), andyi is the real value from the auxiliary data. Without loss ofgenerality, we assume that yi is in range [0, 1]. The scoringfunction calculates an accuracy score for each participant m.However, the sensitivity of Equation 6 is unscalable since theproportion of |zi| and yi is infinite in theory. We observethat the term of the summation in Equation 6 is in range[−1, 1] when zi ≤ 3yi, which normally holds in practicesince it is almost impossible for the predicted parameter zito deviate from the real value yi for more than three times1.Therefore, we add the additional restriction in Equation 6 thatthe proportion zi

yiis no more than 3, and the sensitivity can

be bounded by 1.Moreover, for a classification task, the scoring function u

can be defined as the correct prediction rate directly.Lemma 2: The sensitivity of prediction accuracy for clas-

sification is ∆u = 12 .

Proof 1: Let m and n denotes the number of correct predic-tions and the number of samples respectively. The sensitivity∆ = m+1

n+1 −mn = n−m

n(n+1) . Since n ≥ 1 and n ≥ m, themaximum of ∆u is 1

2 when n = 1 and m = 0.If the server then chooses the participants only according to

the scoring function without any uncertainty, the participantscould make inferences easily about which participants hold thelow quality data by comparing their own parameters and thenew parameters sent from the server. In SecProbe, we utilizeexponential mechanism to inject uncertainty into the samplingprocedure against this kind of inference. The server samplesK participants without replacement such that

Pr[Selecting participant m] ∝ exp(ε

2K∆uu(G,D,m)). (7)

Theorem 5: The sampling procedure in Algorithm 2 (line4) satisfies ε-differential privacy.

Proof 2: Proof sketch. Because sampling one participantconsumes ε

K budget and satisfies εK -differential privacy ac-

cording to Theorem 2, the sampling procedure which samplesK participants will satisfy Kε

K -differential privacy. 2Note that Theorem 5 only ensures that this sampling pro-

cedure satisfies ε-differential privacy at the current trainingiteration. Due to the composition properties of differentialprivacy, the privacy level provided by it may degrade during

1We find that the average of | zi−yiyi

|, i.e., the mean relative error, is alwaysless than 1 in the experimental results in Section IV

Algorithm 3 SecProbe on the participant side

1: Download the same initialized weights W0 from the server

2: Set the mini-batch size |S| and the number of iterations Ithat the participant performs local SGD in each commu-nication round

3: Decompose the loss function `(S,W ) and derive anapproximated polynomial form ˆ(S,W )

4: Obtain the perturbed loss function ¯(S,W ) by functionalmechanism

5: for iteration j = 1 to I do6: Run SGD with batch size |S| on the local data set using

the perturbed loss function7: end for8: Upload W to the server9: Receive the new averaged weight Wnew from the server

10: Repeat steps 5-9 until an acceptable small test error isobtained

11: Drop out of the system

training. We will discuss in detail which privacy level ourmechanism will provide during the whole training process inthe privacy analysis of this section.

Remarks. The above procedure samples a set of partici-pants at an exponential rate based on the scoring functionwhile preventing the sampling procedure from leaking pri-vacy. Therefore, the real quality of uploaded weights from aparticipant cannot be inferred by others since the new weightsare computed on a set of privately-chosen participants, andthe system can sample the approximately optimal weights andeliminate the disturbance of irregular participants as much aspossible. It is easy to see that the time complexity of thesampling step is O(KM). We can further significantly reducethe running time by implementing the sampling step on astatic balanced binary tree as suggested in [41]. The improvedsampling step can run in time O(M +K ln(M)).

After choosing the final accepted weights, the server con-ducts a model average operation that sets the new globalweights to be the average of all the accepted weights. Theserver finally sends the new weights to every participant, andwaits for the next round of uploading. We now briefly explainthe reason why model average operation works. The averageoperation to some extent consistently inherits the procedure ofSGD by randomly choosing a mini-batch of the training data toget the sum of errors on the mini-batch and then computing thegradients on the error. The average operation acts as choosinga mini-batch of the data from all the accepted participantsand computing the gradients on the overall error. Note that,our experiments show that this operation works well only ifthe parameters of each participant are randomly initializedby the same seed, which is easy to be implemented, e.g.,the participants can download the same initialized parametersfrom the server to replace their own initialization at the verybeginning.

It is worth noting that, while we assume the server is trustedto some extent (i.e., the server can know the data quality

7

Input LayerHidden Layer

Output Layer

Fig. 3: A neural network with one hidden layer.

of each participant after uploading), we can further easilyrelax this assumption by adopting techniques of anonymouscommunication with provable security to hide participantsfrom the server by using approaches suggested in [19].

C. SecProbe: The Participant Part

Algorithm 3 presents the pseudocode of SecProbe on theparticipant side. Each participant has its own local training dataset and conducts the standard SGD algorithm to train its localmodel. Let Wi denote the network weights of participant i.To protect the privacy of the participant’s sensitive data beingdisclosed by Wi, the participant applies differential privacyonto the training algorithm to get sanitized weights Wi, anduploads it to the server.

To achieve differential privacy, Laplace mechanism is uti-lized in [19] to directly inject noise to the weights. However,their scheme has to consume too much privacy budget for eachweight per epoch in order to achieve acceptable results. Insteadof directly injecting noise to the weights W , in our designwe propose to utilize functional mechanism [26] to perturbthe objective function of the network, train the model on theperturbed objective function and finally compute the sanitizedweights W . Since the structures of the neural networks maybe varied and often depend on specific application scenarios, itis impossible to design a one-size-fits-all differentially privatesolution for all deep learning models. In this paper, we focuson the most common neural network MLP. Specifically, wefirst consider a three-layer fully-connected neural network,design algorithms to train the model in a differentially privatemanner, and then show that more hidden layers can be stackedeasily by using our proposed scheme.

The regression problem usually uses mean square error(MSE) as the loss function. Suppose the training set D hasa set of n tuples κ1, κ2, . . . , κn. For each tuple κi = (Xi, yi),Xi contains d attributes (xi1, xi2, . . . , xid) and yi is the labelof κi. Without loss of generality, we assume each attributein Xi and yi is in the scope of [0, 1], which is easily to besatisfied by data normalization. The MLP takes Xi as inputand outputs a prediction zi of yi as accurate as possible. Then,the objective function can be given by

`(D,W ) =

n∑i=1

(zi − yi)2. (8)

Recall the calculations of MLP, we have zi =σ1(HW (2)) and H = σ2(XT

i W(1)), where W (1) and

W (2) are the weight matrixes of the network (as shown inFigure 3, σ1 is the sigmoid function, and σ2 is the ReLUfunction. Note that we bound the ReLU function by [0, 1] toavoid introducing an unbounded global sensitivity.

Consequently, for a mini-batch S sampled from the trainingset D, the objective function can be written into the followingform

`(S,W ) =

|S|∑i=1

(zi − yi)2

=

|S|∑i=1

[y2i − 2yi[1 + e(−(ReLU(XT

i W(1))W

(2)j ))]−1

+ [1 + e(−(ReLU(XTi W

(1))W(2)j ))]−2].

(9)

Recall that FM requires the objective function be thepolynomial representation of weights w, thus we need toapproximate Equation 9 and rewrite it into a polynomial form.Since the first term of Equation 9 is already in the polynomialform, we only consider the other two terms. To utilize TaylorExpansion to help approximate the functions as suggestedin [26], for any j ∈ [1, d], let f1j , f2j , g1j and g2j be fourfunctions defined as follows

f1 = −2yi[1 + exp(−z)]−1; f2 = [1 + exp(−z)]−2;

g1 = ReLU(XTi W

(1))W (2); g2 = ReLU(XTi W

(1))W (2).(10)

Then, we can rewrite Equation 9 into the following form

`(S,W ) =

|S|∑i=1

[y2i + f1(g1(κi,W )) + f2(g2(κi,W ))]. (11)

Given the above decomposition of the original loss function,we can then apply Taylor expansion in Equation 11 and obtainthe following equation

˜(S,W ) =

|S|∑i=1

[y2i +

2∑l=1

∞∑k=0

f(k)l (γl)

k!

(gl(κi,W )− γl

)k],

(12)where γlj is a real number and without loss of generality

we set it to zero for ease of analysis. As can be seen inEquation 12, the number of polynomial terms is infinite, whichmay result in an unacceptable large sensitivity. Thus, wepropose to truncate Equation 12 by cutting off all polynomialterms with order larger than 2, i.e., we set k ∈ [0, 2]. Thenwe can obtain the final polynomial objective function used fortraining as Equation 13.

Now we are ready to give the following lemma.Lemma 3: Let S and S′ be any two neighboring databases.

Let ˆ(S,W ) and ˆ(S′,W ) be the objective functions of MLP

8

ˆ(S,W ) =

|S|∑i=1

[y2i +

2∑l=1

2∑k=0

[f (k)l (γl)

k!

(gl(κi,W )−γl

)k]]

=

|S|∑i=1

[(y2i +

2∑l=1

f(0)l (0)

)+

2∑l=1

f(1)l (0)

(ReLU(XT

i W(1))W (2)

)+

2∑l=1

f(2)l (0)

2

(ReLU(XT

i W(1))W (2)

)2]

=

|S|∑i=1

[y2i − yi +

1

4+

1− 2yi4

(ReLU(XT

i W(1))W (2)

)+

1

16

(ReLU(XT

i W(1))W (2)

)2](13)

on S and S′ respectively. Then, the global sensitivity of theobjective function ˆ over S and S′ is

∆ ≤ 1

2b+

1

8b2,

where b is the number of hidden units in the hidden layer.Proof 3: Without loss of generality, we assume that S and

S′ differ in the last tuple, κ|S|(κ′

|S|). According to Lemma 1,we have

∆ ≤ 2 maxκ

(1

4

b∑p=1

hp +1

16

b∑p=1,q=1

hphq)

≤ 2(1

4b+

1

16b2) =

1

2b+

1

8b2,

where h is the value of hidden neurons at the hidden layer.As can be seen, the sensitivity of the objective functionˆ(S,W ) only depends on the model structure, which is inde-pendent with the cardinality of the data set S. Finally, we injectLaplace noise with scale ∆

ε to the coefficients of ˆ(S,W )and obtain the perturbed objective function ¯(S,W ), whichsatisfies ε-differential privacy.

The classification problem usually adopts cross-entropyerror as the loss function. We take a compact CNN to solvethe binomial classification problem with one convolution layer,one pooling layer and one fully-connected layer as an example.We choose sigmoid as the activation function. Similar to theregression problem, the objective function is

`(κi,W ) = −yi log zi + (1− yi) log(1− zi). (14)

Equation 14 can also be decomposed to four functions asfollows

f1 = yi log[1 + exp(−z)]−1;

f2 = (1− yi) log[1− (1 + exp(−z))−1];

g1 = g2 = Conv(κi)W(2),

(15)

note that Conv(κi) represents the output of the previousconvolution layer, and W (2) is the weight matrix of the fully-connected layer.

The loss function can be rewritten as Equation 16.

`(S,W ) = −|S|∑i=1

[f1(g1(κi,W )) + f2(g2(κi,W ))]. (16)

The expansion form of Equation 16 is

˜(S,W ) = −|S|∑i=1

2∑l=1

∞∑k=0

[f (k)l (γl)

k!

(gl(κi,W )− γl

)k]. (17)

The final polynomial objective function used for training is inEquation 18.

Lemma 4: Let S and S′ be any two neighboring databases.Let ˆ(S,W ) and ˆ(S′,W ) be the objective functions of MLPon S and S′ respectively. The global sensitivity of the objectivefunction ˆ over S and S′ is

∆ ≤ b+1

4b2,

Proof 4: Without loss of generality, we assume that S andS′ differ in the last tuple, κ|S|(κ

′

|S|). According to Lemma 1,we have

∆ ≤ 2 maxκ

(1

2

b∑p=1

hp +1

8

b∑p=1,q=1

hphq)

≤ 2(1

2b+

1

8b2) = b+

1

4b2.

We next revisit Algorithm 3. After obtaining the perturbedloss function ¯(S,W ), the participant performs standard SGDalgorithm with batch size |S| and iterates for I times. Thenit gets the sanitized weights W of the current round anduploads them to the server. The parameter I manages thecommunication cost of the system by controlling the frequencyof updates between the participants and the server. Simply put,the increase of I will decrease the frequency of updates andthus reduce the communication cost. But it will also depressthe benefits from collaborative learning at the same time sincethe “collaboration” decreases. We will evaluate the effect of Ithrough extensive experiments in the next section.

Scalability. In the above discussion, we focus on an MLPmodel with one hidden layer for regression analysis. Basedon the above calculations, it is easy to stack more hiddenlayers with ReLU function into the model to address morecomplicated problems. For example, if an additional hiddenlayer with b′ neurons is added, the only change of Equation 9 isadding the layers matrix multiplication and activation functionin the exponential. Because the output of the previous layeris bounded to [−1, 1], the sensitivity of the loss function willslightly change to 1

2b′ + 1

8b′2.

Moreover, the functional mechanism can also be appliedto other types of loss functions (e.g., huber-loss function),other activation functions (e.g., hyperbolic tangent), and othertypes of networks (e.g., Auto-Encoder or RNN) with certainadaptations, which are beyond the scope of this paper.

9

ˆ(S,W ) = −|S|∑i=1

2∑l=1

2∑k=0

[f (k)l (γl)

k!

(gl(κi,W )− γl

)k]=

|S|∑i=1

[log 2 + (

1

2− yi)Conv(κi)W

(2) +1

8[Conv(κi)W

(2)]2]

(18)

D. Security Analysis

Let the privacy budget used in sampling participants andperturbing objective functions be ε1 and ε2 respectively. Sincethe training procedure on each participant strictly follows theFunctional Mechanism, the parameters computed from theperturbed objective function satisfy ε-differential privacy ineach training iteration. Let Si be the training batch of aniteration. Since every batch is disjoint from each other ina training epoch (e.g. Si and Si−1 contains different tuplessampling randomly from the training data), where an epochis one full training process that consists of several iterationscovering the whole training data, we can conclude that thetraining process on each participant ensures ε2-differentialprivacy in each epoch according to Theorem 4.

Recall that the sampling procedure in Algorithm 2 alsoensures ε1-differential privacy on each sampling step. Sinceit can be seen that each step of sampling protect the privacyof part of the training data’s quality, we can conclude thatthe sampling procedure in Algorithm 2 satisfies ε1-differentialprivacy in each epoch.

Note that the two procedures above address two differentprivacy concerns respectively. In case of passive adversaries,the procedure of each participant aims to protect the privacyof the training data, which focuses on each single record ofthe training data, while the procedure on the server aims toprotect the privacy of the data quality, which takes all thecorresponding records as a whole. Therefore, we can finallyconclude that our scheme SecProbe satisfies max(ε1, ε2)-differential privacy in each training epoch.

We further consider stronger adversaries. If the adversariesare active, they may perform two kinds of behaviors: 1) sendfake parameters to the server; 2) steal the parameters fromthe communication process directly. For the first maliciousbehavior, thanks to the exponential mechanism that we haveintroduced, it is almost impossible for the fake parametersto significantly affect the model, thus there is no negativeeffect on the training process, and the adversaries cannotinfer the data quality of other participants. For the secondmalicious behavior, the adversary may eavesdrop on channelsbetween honest participants and the server. Some effectivecryptography tools can be used to encrypt the parameters (e.g.,AES) and verify the received data (e.g., SHA-256) to ensurecommunication security. Therefore, our scheme is robust andsecure when facing active adversaries, i.e., malicious partici-pants.

IV. EXPERIMENTAL EVALUATION

In this section, we evaluate the performance of SecProbe ona real-world data set for regression analysis. All experimentsare conducted on a machine with Intel Core i5-4460S CPU2.9GHz and 12GB RAM, running Ubuntu 14.04.

I 20 50 100 1000Communication round 443 190 152 193

TABLE I: The effect of iterations I which controls the fre-quency of updates. Each entry in the table gives the necessarynumber of communication rounds to achieve MRE of 0.15 (N= 60, M = 30, K = 30, P = 0, and ε = 1).

A. Datasets

For the regression task, we use the data set from IntegratedPublic Use Microdata Series [42], named US, which contains600,000 census records collected in US. There are 15 attributesin the data set, namely, Sex, Age, Race, Education, Filedof Degree, Marital Status, Family Size, Number of Children,Hours Work per Week, Ownership of Dwelling, Number ofChildren, Number of Rooms, Private Health Insurance, LivingDifficulty, Annual Income. Among all these attributes, there are6 attributes that are categorical, including Race, Education,Field of Degree, Marital Status, Private Health Insurance,Living Difficulty. For an attribute that can only be two possiblevalues (e.g., male and female for sex), we set it to be 0 or 1.For the remainings, we follow the common practice in machinelearning to transform these attributes by one-hot encoding. Wethen normalize the other numeric attributes into the scope of[0, 1]. Specifically, for the Annual Income, we apply log trans-formation before normalization in order to obtain a relativelystable distribution. After these transformations, our data setnow has 20 dimensions.

We randomly sample 90,000 records to be the test data set,and 10,000 records to be the auxiliary validation data set on theserver. The remaining 500,000 records are randomly dividedinto N parts, where N is the number of participants. The datais already shuffled before training.

We focus on a regression task predicting the value of AnnualIncome by using the other attributes as the input. The accuracyof the model is measured by mean relative error (MRE),

MRE =1

n

n∑i=1

|zi − yi|yi

, (19)

where yi is the real value, zi is the predicted value producedby the network, and n is the number of tuples in the test dataset.

For the binomial classification task, we use the MNIST dataset as the benchmark, which consists of 28x28 images ofhandwritten digits with 60,000 training samples and 10,000test samples. We use prediction accuracy to evaluate theperformance of the model.

B. Experimental Setup

We use the popular neural network architectures: multi-layerperceptron (MLP) with three fully-connected layers. For the

10

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.0

0.2

0.4

0.6

0.8

1.0M

RE

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(a) N = 30

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.0

0.2

0.4

0.6

0.8

1.0

MRE

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(b) N = 60

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.0

0.2

0.4

0.6

0.8

1.0

MRE

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(c) N = 100

Fig. 4: Training convergence for all schemes in regression task.

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(a) N = 30

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(b) N = 60

0 50 100 150 200 250 300 350 400 450 500Communication Round

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

CentralizedSecProbeStand-aloneDSSGD, u = 1DSSGD, u = 0.5DSSGD, u = 0.1DSSGD, u = 0.01

(c) N = 100

Fig. 5: Training convergence for all schemes in classification task.

regression task, the activation functions of hidden layer andoutput layer are ReLU and sigmoid function respectively. Thenumber of neurons in the hidden layer is 80. We use SGDas the learning algorithm, the learning rate is set to be 0.01and the mini-batch size is set to be 128. The weights of themodels are randomly initialized by normal distribution (withmean 0 and standard deviation 1).

For the classification task, we use a compact CNN withonly one convolution and one hidden layer, the activationfunctions of the output layer is sigmoid function, and thenumber of neurons in the fully-connected layer is 128. Theother hyperparameters are the same as the regression task.

Since the approaches proposed in [18] and [17] are not spe-cially designed for collaborative learning, we mainly compareall results with DSSGD [19], an existing work on privacy-preserving collaborative deep learning, and two baseline ap-proaches. The first is the centralized training on the entire dataset, which is the basic approach that does not consider privacyconcerns and ought to have the best performance of the modelaccuracy. The second is stand-alone training, which trainssolely on local data set without collaboration. We call thesetwo baselines Centralized and Stand-alone, respectively. Allapproaches are implemented on TensorFlow [43], a populardeep-learning library developed by Google. For SecProbe, weset the privacy budgets used in participants’ sampling andperturbing objective functions to be the same. We fine-tunethe parameters in DSSGD according to [19] and use the

settings with the best performance (the parameter downloadratio θd = 1, gradient bound γ = 0.001, gradient selectingthreshold τ = 0.0001).

To simulate the irregular participants, we randomly choosehalf of the participants and replace P fraction of their data withrandom noise in the scope of [0,1]. We vary P to evaluate therobustness of SecProbe against irregular participants.

C. Results

The effect of I . Table I shows the effect of parameter I .The results show that a larger I will speed up the trainingconvergence by increasing the computation loads on eachparticipant. However, a too large I will slow down the trainingconvergence since the collaboration decreases. Based on theseresults, we set I to be 100 in the following experiments.

Training convergence. Figure 4 and Figure 5 shows thetraining convergence of all schemes for regression and classi-fication tasks respectively. We vary the number of participantsN in SecProbe and DSSGD, and set M = N , K = M ,ε = 1, P = 0. The y-axis is the performance of trainedmodel, and the x-axis denotes the number of communicationrounds. The results show that although Centralized achievesthe best accuracy in the end, SecProbe can achieve almost thesame accuracy with a higher convergence rate, while providinga rigorous privacy guarantee. Note that our scheme also hasa better performance than DSSGD both on convergence rateand model accuracy in the regression task, and almost the

11

(a) N = 30 (b) N = 60 (c) N = 100

Fig. 6: The effect of number of participants (M ) utilized per round on the system performance.

1 0.1N 0.2NSpecial Participants Number

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

MRE

Regular testSpecial test

(a) N = 30

1 0.1N 0.2NSpecial Participants Number

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

MRE

Regular testSpecial test

(b) N = 60

1 0.1N 0.2NSpecial Participants Number

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

MRE

Regular testSpecial test

(c) N = 100

Fig. 7: The effect of number of participants (M ) utilized per round on the system performance.

same performance in the classification task with more rigorousprivacy guarantee. The main reason is that only perturbing theuploaded gradients is not enough to preserve the training data,and DSSGD consumes too much privacy budget in perturbingall the gradient values. Moreover, the noise directly injectedon each gradient independently may also make the trainingprocess unstable. In addition, SecProbe can always reach abetter final accuracy than DSSGD, and achieve a comparableconvergence rate with DSSGD when the uploaded ration is 1in CNN.

The effect of parallelism. The parameter M controls thenumber of participants the server choose to utilize per round,which can also be regarded as parallelism degree. We vary Mto be (0.1N, 0.3N, 0.5N, 0.7N, 1.0N ) and set K = M , ε = 1,and P = 0. Note that the total size of training data set does notchange with different values of M . As can be seen in Figure 6,the increase of parallelism will speed up the convergence oftraining, and it leads to a more accurate model while increasingthe communication loads. On the contrary, the decrease ofparallelism will reduce the number of accesses for each datapoint. In brief, the size of M affects the amount of data fortraining in each communication round and the convergencespeed. It is good to see that, when M = 0.1N it can stillachieve relatively accurate results. Based on the above results,we choose M = 0.5N for the following experiments to strikea good balance between efficiency and convergence rate.

The effect of auxiliary validation set. To show the effectof the auxiliary validation set on the system performance, we

re-design the training set and the test set to simulate oneor multiple special participants whose hypotheses are notinclude at the server. More specifically, we exclude all sampleswhose one attribute is within a certain range from the originaldatasets, and put these excluded data to the special participantsand a special test set. Here we choose the attribute Age dueto its numeric form and its impact on the income. We removeall samples with Age ≤ 0.25 or Age ≥ 0.7 from the originaldatasets. We set the number of special participants as 1, 0.1N,and 0.2N respectively, and demonstrate the results in Figure7.We can see that the performance of the special test set issignificantly worse than that of the regular test set. With morespecial participants, the performance of the special test setwill improve due to the increasing contribution of the specialparticipants to the collaborative learning process. In fact, ifa participant’s data is completely random noise, it is almostimpossible to be chosen by the server to calculate the averagethanks to the exponential mechanism. A special participantwith high-quality dataset that contains new findings that mightbe misjudged can also play a role in improving the model.

The robustness against irregular participants. Figure 9shows the results of the robustness against irregular partici-pants. We choose half of the participants and replace P pro-portion of their data with random noise, and set M = 0.5N ,K = 0.5M , and ε = 1. We vary the total number of participantsN and the proportion P of the noise data, which means thatone half of the participants are irregular with P fraction oftheir data to be random noise. We set N = (30, 60, 100) and

12

Fig. 8: Accuracy vs. privacy budget ε (N = 60, M = 30, K= 15, and P = 0.2).

P = (0.2, 0.4, 0.6). Correspondingly, for Centralized, we setP = (0.1, 0.2, 0.3). For the number of sampling participantsK, we set it to be the half of M , which follows the assumptionthat the majority of the participants are regular. As can be seenin Figure 9, the model accuracy of DSSGD, Centralized andStand-alone degrade quickly with the increase of noise, sincethey all lack in designing mechanisms against irregular partic-ipants. Meanwhile, Our approach SecProbe achieves very highaccuracy which is almost the same as the performance of thecase with no irregular participants, and it is also robust againstthe proportion of noise. The experimental results validate theeffectiveness of our scheme.

Accuracy vs. privacy. We evaluate the effect of differentvalues of privacy budget ε on the accuracy of the neuralnetwork. Figure 8 shows the results compared with the com-petitors. The x-axis represents the privacy budget per trainingepoch (an epoch contains several iterations over all trainingsamples). It is shown that, a larger value of ε results in highaccuracy while giving lower privacy guarantee. SecProbe canachieve almost the same results of Centralized and outperformStand-alone when ε ≥ 0.1.

V. CONCLUSIONS

In this paper, we took the first step to investigate the prob-lem of privacy-preserving collaborative deep learning systemwhile considering the existence of irregular participants, andpresented a new scheme called SecProbe. SecProbe utilizesexponential mechanism and functional mechanism to protectboth the privacy of the participants’ data and the quality oftheir data, the two major privacy concerns in such a system.The experimental results demonstrate that our system is robustto irregular participants, and can achieve high accurate resultswhich is close to the centralized model, while providingrigorous privacy guarantee.

REFERENCES

[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” arXiv preprint arXiv:1512.03385, 2015.

Fig. 9: The robustness of SecProbe against irregular partici-pants.

[2] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition withdeep recurrent neural networks,” in Proc. of ICASSP’13. IEEE, 2013,pp. 6645–6649.

[3] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly,A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neuralnetworks for acoustic modeling in speech recognition: The shared viewsof four research groups,” IEEE Signal Processing Magazine, vol. 29,no. 6, pp. 82–97, 2012.

[4] R. Fakoor, F. Ladhak, A. Nazi, and M. Huber, “Using deep learningto enhance cancer diagnosis and classification,” in Proc. of ICML’13,2013.

[5] M. Liang, Z. Li, T. Chen, and J. Zeng, “Integrative data analysis ofmulti-platform cancer data with a multimodal deep learning approach,”IEEE/ACM Transactions on Computational Biology and Bioinformatics,vol. 12, no. 4, pp. 928–937, 2015.

[6] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. VanDen Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam,M. Lanctot et al., “Mastering the game of go with deep neural networksand tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016.

[7] “General data protection regulation,” https://en.wikipedia.org/wiki/General Data Protection Regulation, 2016.

[8] W. Du, Y. S. Han, and S. Chen, “Privacy-preserving multivariatestatistical analysis: Linear regression and classification.” in Proc. ofSDM’04, vol. 4. SIAM, 2004, pp. 222–233.

[9] K. Chaudhuri and C. Monteleoni, “Privacy-preserving logistic regres-sion,” in Proc. of NIPS’09, 2009, pp. 289–296.

[10] G. Jagannathan and R. N. Wright, “Privacy-preserving distributed k-means clustering over arbitrarily partitioned data,” in Proc. of KDD’05.ACM, 2005, pp. 593–599.

[11] B. I. Rubinstein, P. L. Bartlett, L. Huang, and N. Taft, “Learning in alarge function space: Privacy-preserving mechanisms for svm learning,”arXiv preprint arXiv:0911.5708, 2009.

[12] J. Hamm, A. C. Champion, G. Chen, M. Belkin, and D. Xuan, “Crowd-ml: A privacy-preserving learning framework for a crowd of smartdevices,” in Proc. of ICDCS’15. IEEE, 2015, pp. 11–20.

[13] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, andJ. Wernsing, “Cryptonets: Applying neural networks to encrypted datawith high throughput and accuracy,” in Proc. of ICML’16, 2016, pp.201–210.

[14] J. Liu, M. Juuti, Y. Lu, and N. Asokan, “Oblivious neural networkpredictions via minionn transformations,” in Proc. of CCS’17. ACM,2017, pp. 619–631.

[15] P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-preserving machine learning,” in Proc. of S%P’17. IEEE, 2017, pp.19–38.

[16] M. Li, L. Lai, N. Suda, V. Chandra, and D. Z. Pan, “Privynet: A flexibleframework for privacy-preserving deep neural network training,” arXivpreprint arXiv:1709.06161, 2017.

[17] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Tal-war, and L. Zhang, “Deep learning with differential privacy,” in Proc.of CCS’16. ACM, 2016, pp. 308–318.

13

[18] N. Phan, Y. Wang, X. Wu, and D. Dou, “Differential privacy preservationfor deep auto-encoders: an application of human behavior prediction.”in Proc. of AAAI’16, vol. 16, 2016, pp. 1309–1316.

[19] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in Proc.of CCS’15. ACM, 2015, pp. 1310–1321.

[20] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks thatexploit confidence information and basic countermeasures,” in Proc. ofCCS’15. ACM, 2015, pp. 1322–1333.

[21] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membershipinference attacks against machine learning models,” in Proc. of S%P’17.IEEE, 2017, pp. 3–18.

[22] F. Tramer, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealingmachine learning models via prediction apis.” in Proc. of USENIXSecurity’16, 2016, pp. 601–618.

[23] B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under thegan: information leakage from collaborative deep learning,” in Proc.of CCS’17. ACM, 2017, pp. 603–618.

[24] “Deep learning and differential privacy,” https://github.com/frankmcsherry/blog/blob/master/posts/2017-10-27.md, 2016.

[25] C. Dwork, “Differential privacy,” in Proc. of ICALP’06, 2006, pp. 1–12.[26] J. Zhang, Z. Zhang, X. Xiao, Y. Yang, and M. Winslett, “Functional

mechanism: regression analysis under differential privacy,” Proc. ofVLDB’12, vol. 5, no. 11, pp. 1364–1375, 2012.

[27] J. Schmidhuber, “Deep learning in neural networks: An overview,”Neural Networks, vol. 61, pp. 85–117, 2015.

[28] L. Deng, “A tutorial survey of architectures, algorithms, and applicationsfor deep learning,” APSIPA Transactions on Signal and InformationProcessing, vol. 3, p. e2, 2014.

[29] M. Avriel, Nonlinear programming: analysis and methods. CourierCorporation, 2003.

[30] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning represen-tations by back-propagating errors,” Cognitive modeling, vol. 5, no. 3,p. 1, 1988.

[31] T. Zhang, “Solving large scale linear prediction problems using stochas-tic gradient descent algorithms,” in Proc. of ICML’04. ACM, 2004, p.116.

[32] Q. Wang, Y. Zhang, X. Lu, Z. Wang, Z. Qin, and K. Ren, “Real-time and spatio-temporal crowd-sourced social network data publishingwith differential privacy,” IEEE Transactions on Dependable and SecureComputing, vol. 15, no. 4, pp. 591–606, 2018.

[33] R. Chen, Q. Xiao, Y. Zhang, and J. Xu, “Differentially private high-dimensional data publication via sampling-based inference,” in Proc. ofKDD’15. ACM, 2015, pp. 129–138.

[34] B. Yang, I. Sato, and H. Nakagawa, “Bayesian differential privacy oncorrelated data,” in Proc. of SIGMOD’15. ACM, 2015, pp. 747–762.

[35] D. Proserpio, S. Goldberg, and F. McSherry, “Calibrating data tosensitivity in private data analysis: a platform for differentially-privateanalysis of weighted datasets,” Proc. of VLDB’14, vol. 7, no. 8, pp.637–648, 2014.

[36] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noiseto sensitivity in private data analysis,” in Theory of cryptography, 2006,pp. 265–284.

[37] F. McSherry and K. Talwar, “Mechanism design via differential privacy,”in Proc. of FOCS’07. IEEE, 2007, pp. 94–103.

[38] F. D. McSherry, “Privacy integrated queries: an extensible platform forprivacy-preserving data analysis,” in Proc. of SIGMOD’09. ACM, 2009,pp. 19–30.

[39] W. Rudin, Principles of mathematical analysis. McGraw-Hill NewYork, 1964, vol. 3.

[40] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learningapplied to document recognition,” Proceedings of The IEEE, vol. 86,no. 11, pp. 2278–2324, 1998.

[41] R. Bhaskar, S. Laxman, A. Smith, and A. Thakurta, “Discoveringfrequent patterns in sensitive data,” in Proc. of KDD’10. ACM, 2010,pp. 503–512.

[42] “Minnesota population center. public use microdata series, international:Version 6.4. university of minnesota, 2015.” https://international.ipums.org.

[43] “Tensorflow,” https://www.tensorflow.org.