Safe Machine Learning and Defeat-ing Adversarial AttacksBita Darvish Rouhani, Mohammad Samragh, Tara Javidi, and Farinaz Koushanfar | University ofCalifornia San Diego

Adversarial attacks have exposed the unreliability of machine learning models for decision making in autonomousagents. This article discusses recent research for ML model assurance in face of adversarial attacks.

The fourth industrial revolutionshaped by Machine Learning (ML)

algorithms is underway. ML algorithmshave provided a paradigm shift in de-vising automated systems that can evensurpass human performance in con-trolled environments. While advancedlearning technologies are essential forenabling interaction among autonomousagents and the environment, a charac-terization of their quality or carefulanalysis of the system reliability in thepresence of malicious entities are still intheir infancy.

Reliability and safety considerationis the biggest obstacle to the wide-scale adoption of emerging learning al-gorithms in sensitive scenarios such asintelligent transportation, health-care,warfare, and financial systems. Al-though ML models deliver high accu-racies in conventional settings with lim-ited simulated input samples, recent re-search in adversarial ML has shed lighton the unreliability of their decisions inreal-world scenarios. For instance, con-sider a traffic sign classifier used in self-driving cars. Figure 1 shows an exampleadversarial sample where the attackercarefully adds imperceptible perturba-tion to the input image to mislead theemployed ML model, and thus, jeopar-dizes the safety of the vehicle.

In light of the adversarial attacks,to pervasively employ autonomous ML

agents in sensitive tasks it is imperativeto answer the following two questions:

• What are the vulnerabilities ofmachine learning models that at-tackers can leverage for craftingadversarial samples?

• How can we characterize andthwart the adversarial space foreffective ML model assuranceand defense against adversaries?

In this article, we discuss our recentresearch results for adaptive ML modelassurance in face of adversarial attacks.In particular, we introduce, implement,and automate a novel countermeasurecalled Modular Robust Redundancy(MRR) to thwart the potential adversar-ial space and significantly improve thereliability of a victim ML model.1

Unlike prior defense strategies,MRR methodology is based upon unsu-pervised learning, meaning that no par-ticular adversarial sample is leveragedto build/train the modular redundan-cies. Instead, our unsupervised learningmethodology leverages the structure ofthe built model and characterizes thedistribution of the high dimensionalspace in the training data. Adoptingan unsupervised learning approach, inturn, ensures that the proposed detec-tion scheme can be generalized to awide class of adversarial attacks. Wecorroborate the effectiveness of our

method against the existing state-of-the-art adversarial attacks. In partic-ular, we open-source our API to en-sure ease of use by data scientists andengineers and invite the communityto attempt attacks against our pro-vided benchmarks in form of a chal-lenge. Our API is available at https://github.com/Bitadr/DeepFense

Adversarial samples have alreadyexposed the vulnerability of ML mod-els to malicious attacks; thereby under-mining the integrity of autonomous sys-tems built upon machine learning. Ourresearch, in turn, empowers coherent in-tegration of safety consideration into thedesign process of ML models. We be-lieve that the reliability of ML modelsshould be ensured in the early develop-ment stage instead of looking back withregret when the machine learning sys-tems are compromised by adversaries.

Figure 1: The left image is a legiti-mate “stop” sign sample that is clas-sified correctly by an ML model. Theright image, however, is an adversar-ial input crafted by adding a particu-lar perturbation that makes the samemodel classify it as a “yield” sign.

1

Adversary Models and Present Attacks

An adversarial sample refers to an input to the ML model that can deceive the model to make a wrong decision. Adversarialsamples are generated by adding carefully crafted perturbations to a legitimate input. In particular, an adversarial sample should atleast satisfy three conditions: (i) The ML model should perceive a correct decision on the original (legitimate) sample; for instance,in a classification task, the ML model should correctly classify the original sample. (ii) The ML system should make a wrongdecision on the perturbed adversarial sample; e.g., in a classification task, the model must misclassify the adversarial sample. (iii)The perturbation added to the original sample should be imperceptible, meaning that the perturbation should not be recognizablein the human cognitive system.Depending on the attacker’s knowledge, the threat model can be categorized into three classes:

• White-box attacks. The attacker knows everything about the victim model including the learning algorithm, model topol-ogy, defense mechanism, and model/defender parameters.

• Gray-box attacks. The attacker only knows the underlying learning algorithm, model topology, and defense mechanismbut has no access to the model/defender parameters.

• Black-box attacks. The attacker knows nothing about the pertinent machine learning algorithm, ML model, or defensemechanism. This attacker only can obtain the outputs of the victim ML model corresponding to input samples. In thissetting, the adversary can perform a differential attack by observing the output changes with respect to the input variations.

Henceforth, we consider the white-box threat model as it represents the most powerful attacker that can appear in real-worldsettings. We evaluate our proposed countermeasure against four different classes of attacks including Fast-Gradient-Sign (FGS),1

Jacobian Saliency Map Attack (JSMA),2 Deepfool,3 Basic Iterative Method (BIM),4 and Carlini and Wagner attack (CarliniL2),5,6

to corroborate the generalizability of our unsupervised approach. The aforementioned attacks cover a wide range of one-shot anditerative attack algorithms. The goal of each attack is to minimize the distance between the legitimate sample and the correspondingadversarial samples with a particular constraint such that the generated adversarial sample misleads the victim ML model. Pleaserefer to the technical papers for the details of each attack algorithm. For the realization of different attack strategies,7 we leveragethe well-known adversarial attack benchmark library known as Cleverhans.——————————-References1. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprintarXiv:1412.6572, 2014.2. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations ofdeep learning in adversarial settings. In Proceedings of the IEEE European Symposium on Security and Privacy (SP), 2016.3. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deepneural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.4. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. International Conference onLearning Representations, 2017.5. Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proceedings of the IEEE Sympo-sium on Security and Privacy (SP), 2017.6. Nicholas Carlini and David Wagner. Magnet and “efficient defenses against adversarial attacks" are not robust to adversarialexamples. arXiv preprint arXiv:1711.08478, 2017.7. Ian Goodfellow Reuben Feinman Fartash Faghri Alexander Matyasko Karen Hambardzumyan Yi-Lin Juang Alexey KurakinRyan Sheatsley Abhibhav Garg Yen-Chen Lin Nicolas Papernot, Nicholas Carlini. cleverhans v2.0.0: an adversarial machine learn-ing library. arXiv preprint arXiv:1610.00768, 2017.

Adversarial DefensesIn response to various adversarial at-

tacks proposed in the literature, severalresearch attempts have been made to de-sign ML models that are more robustin face of adversarial examples. The ex-

isting countermeasures can be classifiedinto two distinct categories:

(i) Supervised strategies which aim toimprove the generalization of the learn-ing models by incorporating the noise-corrupted version of inputs as train-

ing samples and/or injecting adversar-ial examples generated by different at-tacks into the DL training phase.2, 3, 4, 5

The proposed defense methods in thiscategory are particularly tailored forspecific perturbation patterns and can

2

(a) (b) (c)

Figure 2: (a) In this example, data points (denoted by green squares and blue triangles) can be easily separatedin one-dimensional space. Having extra dimensions adds ambiguity in choosing the pertinent decision boundaries.For instance, all the shown boundaries (dashed lines) are sufficient to classify the raw data with full accuracy in two-dimensional space but are not equivalent in terms of robustness to noise. (b) The rarely explored space (region specifiedby diagonal striped) in a learning model leaves room for adversaries to manipulate the nuisance (non-critical) vari-ables and mislead the model by crossing the decision boundaries. (c) In MRR methodology, a set of defender modulesis trained to characterize the data density distribution in the space spanned by the victim model. The defender modulesare used in parallel to checkpoint the reliability of the ultimate prediction and raise an alarm flag for risky samples.

only partially evade adversarial sam-ples generated by other attack scenarios(e.g., with different perturbation distri-butions) from being effective.6

(ii) Unsupervised strategies which aimto smooth out the decision bound-aries by incorporating a smoothnesspenalty7, 8 as a regularization term in theloss function or compressing the neuralnetwork by removing the nuisance vari-ables.9 These works have been devel-oped based on an implicit assumptionthat the existence of adversarial samplesis due to the piece-wise linear behav-ior of decision boundaries (obtained bythe gradient descent approach) in high-dimensional spaces. As such, their in-tegrity can be jeopardized by consider-ing a slightly higher perturbation at theinput space to cross the smoothed deci-sion boundaries.10

More recently, an unsupervisedmanifold projection approach (calledMagNet) is proposed in the literature toreform adversarial samples using auto-encoders.11 Unlike MRR countermea-

sure, MagNet is inattentive to the perti-nent data density in the latent space. Asshown by Carlini and Wagner,12 mani-fold projection methods including Mag-Net are not robust to adversarial samplesand can approximately increase the re-quired distortion to generate adversarialsample by only 37 percent.

To the best of our knowledge, ourproposed MRR methodology is the firstunsupervised learning countermeasurethat simultaneously considers both datageometry (density) and decision bound-aries for an effective defense against ad-versarial attacks. Our proposed coun-termeasure is able to withstand thestrongest known white-box attack todate by provably increasing the robust-ness of the underlying model. The MRRmethodology does not assume any par-ticular attack strategy and/or perturba-tion pattern. This obliviousness to theunderlying attack or perturbation mod-els demonstrates the generalizability ofthe proposed approach in face of poten-tial future adversarial attacks.

Furthermore, a recent line of re-search in adversarial ML has shown atrade-off between the robustness of amodel and its accuracy.13 To avoid thistrade-off, instead of learning a singlemodel that is both robust and accurate,our proposed countermeasure learns aset of complementary defender moduleswhile keeping the victim model intact;therefore, our defense mechanism doesnot impose any degradation of accuracyon the victim model.

What is the root causeof adversarial samples?

Our hypothesis is that the vulnera-bility of ML models to adversarial sam-ples originates from the relatively largesubsets of the data domain that remainmainly unexplored. This phenomenon islikely caused by the limited access tothe labeled data and/or inefficiency ofthe algorithms in terms of their general-ized properties. Figure 2 provides a sim-ple illustration of the partially explored

3

space in a two-dimensional setup. Weanalytically and empirically back up ourhypothesis by extensive evaluations1 onvarious benchmarks including the well-known MNIST, CIFAR10, and mini-ImageNet datasets.

Due to the curse of dimensionality,it is often not practical to fully coverthe underlying high-dimensional spacespanned by modern ML applications.What we can do, instead, is to con-struct statistical modules that can quan-titatively assess whether or not a certainsample comes from the subspaces thatwere exposed to the ML agent. To en-sure robustness against adversarial sam-ples, we argue that ML models shouldbe capable of rejecting samples that liewithin the rarely-explored regions.

How can we character-ize and thwart the ad-versarial space?

We formalize the goal of prevent-ing adversarial attacks as an optimiza-tion problem to minimize the rarely ob-served regions in the latent feature spacespanned by an ML model. To solve theaforementioned minimization problem,a set of complementary but disjoint re-dundancy modules are trained to capturethe Probability Density Function (PDF)of the legitimate (explored) subspaces.In MRR methodology, the victim modelis kept as is while separate defendermodules are trained to checkpoint the re-liability of the victim model prediction.

Each modular redundancy learns aPDF to explicitly characterize the ge-ometry (density) of a certain high-dimensional data abstraction within anML model. In a neural network, for ex-ample, each MRR module checkpointsa certain intermediate hidden (or input)layer (Figure 3). A DL layer may becheckpointed by multiple MRR mod-ules to provide a more robust defense

strategy. Each defender marks the com-plement of the space characterized bythe learned PDF as the rarely observedregion, enabling statistical tests to deter-mine the validity of new samples.

Once such characterizations are ob-tained, statistical testing is used at run-time to determine the legitimacy of newdata samples. The defender modulesevaluate the input sample probability inparallel with the victim model and raisealarm flags for data points that lie withinthe rarely explored regions. As such, theadversary is required to simultaneouslydeceive all defender modules in order tosucceed. Unlike the prior works, our ap-proach does not suffer from a degrada-tion of accuracy since the victim modelis untouched.

Latent Checkpoints

Input Checkpoint

Defender DefenderDefender

Figure 3: A high-level overview ofproposed MRR methodology. Theoutput layer of the victim neuralnetwork (the green neurons) is aug-mented with a single risk measure(the red neuron) determining the le-gitimacy of the prediction.

The outputs of MRR modules areaggregated into a single output node(the red neuron in Figure 3) that quan-titatively measures the reliability of theoriginal prediction. For any input sam-ple, the new neuron outputs a risk mea-sure in the unit interval [0,1], with 0 and1 indicating safe and highly risky sam-ples, respectively. The extra neuron in-

corporates a “don’t know” class into themodel: samples with a risk factor higherthan a certain threshold (a.k.a., securityparameter) are treated as adversarial in-puts. The threshold is determined basedon the safety-sensitivity of the applica-tion for which the ML model is em-ployed. This approach is beneficial in asense that it allows dynamic reconfigu-ration of the detection policy with mini-mal required recomputing overhead.

Adversarial and legitimate samplesdiffer in certain statistical properties.Adversarial samples are particularlycrafted by finding the rarely explored di-mensions in an `∞ ball of radius ε . InMRR methodology, samples whose fea-tures lie in the unlikely subspaces aremarked and identified as risky samples.Our conjecture is that a general MLmodel equipped with the side informa-tion about the density distribution of theinput data as well as the distribution ofthe latent feature vectors can be madearbitrary robust against adversarial sam-ples. Our proposed MRR methodologystrengthens the defense by training mul-tiple defenders that are negatively corre-lated. Informally, if two MRR modulesare negatively correlated, then an adver-sarial sample that can mislead one mod-ule will raise high suspicion in the othermodule and vice versa.

As an example, consider a classifi-cation task where a 4-layer neural net-work is used to categorize ten differ-ent classes of the popular digit recogni-tion dataset known as MNIST. Figure 4ademonstrates the feature vectors withinthe second-to-last layer of the pertinentvictim neural network in the Euclideanspace. Note that only three dimensionsof the feature vectors are shown for vi-sualization purposes. The feature vec-tors of samples corresponding to thesame class (same color) tend to be clus-tered in the Euclidean space. Each clus-ter has a center obtained by taking the

4

(a) (b) (c) (d)

Figure 4: Example feature samples in a 4-layer neural network trained for a digit recognition task. Latent featuresamples in the second-to-last layer of (a) the victim model and (c) its corresponding transformation in our defendermodule. The majority of adversarial samples (e.g., the red dot points in (c)) reside in the regions with low densityof training samples. Figures (b) and (d) show the histogram of the distance between samples and cluster centers forlegitimate and adversarial inputs in the victim and defender models, respectively.

average of the features of the corre-sponding class. For each input sampleidentified as a certain class by the vic-tim model, we compute the distance be-tween the feature vector and the corre-sponding center.

Figure 4b demonstrates the distribu-tion of the distance between data sam-ples and the center of the pertinentclass for legitimate (blue) and adver-sarial (red) samples. In this experiment,we generate the adversarial samples us-ing the FGS attack algorithm. It can beseen that the aforementioned distanceis higher for adversarial samples com-pared to legitimate samples. This, infact, validates our hypothesis that ad-versarial samples lie within the unex-plored subspaces (higher distance fromcluster centers in this case). The adver-sarial samples can be simply detectedby thresholding the aforementioned dis-tance. Nevertheless, building a detec-tion method based on this distance in itscurrent form will lead to a high prob-ability of false alarm: legitimate inputsmight be incorrectly marked as adver-sarial samples.

Each defender module is regular-ized based on a prior distribution (e.g.,

a Gaussian Mixture Model) to enforcedisentanglement between the featurescorresponding to different categoriesand be more robust against skewed fea-ture distributions.1 As an example, thecorresponding data distribution and dis-tance measure for a single defender areshown in Figures 4c and 4d, respec-tively. It can be seen that the clustersare well-separated, thus, the characteri-zation of the adversarial subspace incursa small probability of false alarms. Ta-ble 1 summarizes the Area Under Curve(AUC) score attained against four differ-ent attacks in a black-box setting.

Table 1: AUC score obtained by 16latent defenders that checkpoint thesecond-to-last layer of the victim modelfor MNIST and CIFAR-10 benchmarks.For ImageNet benchmark, we only used1 defender due to the high computa-tional complexity of the pertinent neuralnetwork and attacks.

MNIST CIFAR10 ImageNetFGS 0.996 0.911 0.881JSMA 0.995 0.966 -Deepfool 0.996 0.960 0.908CarliniL2 0.989 0.929 0.907BIM 0.994 0.907 0.820

Adaptive white-box attack. To fur-ther corroborate the robustness of MRRmethodology, we applied the state-of-the-art CarliniL2 attack in a white-boxsetting.12 A similar strategy was pre-viously used in the literature to breakthe state-of-the-art countermeasures in-cluding MagNet,11 APE-GAN,14 andother recently proposed efficient de-fenses methods.15 Table 2 summarizesthe success rate of the CarliniL2 attackalgorithm for different numbers of re-dundancy (defender) modules and riskthresholds (security parameters) for theMNIST benchmark.

Our MRR methodology offers atrade-off between robustness of the MLmodel and its computational complex-ity. On the one hand, increasing thenumber of MRRs enhances the robust-ness of the model as shown in Ta-ble 2. On the other hand, the compu-tational complexity grows linearly withthe number of MRRs (each redundancymodule incurs the same overhead as thevictim model). Our proposed MRR de-fense mechanism outperforms existingstate-of-the-art defenses both in termsof the detection success rate and theamount of perturbation required to fool

5

REFERENCES REFERENCES

Table 2: Evaluation of MRR methodology against adaptive white-box attack. We compare our results with prior-art worksincluding Magnet,11 Efficient Defenses Against Adversarial Attacks,15 and APE-GAN.14 For each evaluation, the L2 distor-tion is normalized to that of the attack without the presence of any defense mechanism. For fair comparison to prior work,we did not include our non-differentiable input defenders in this experiment. Note that highly disturbed images (with large L2distortions) can be easily detected using the input dictionaries/filters.

MRR Methodology (White-box Attack) Prior-Art Defenses (Gray-box Attack)Security Parameter SP=1% SP=5% Magnet Efficient Defenses APE-GANNumber of Defenders N=0 N=1 N=2 N=4 N=8 N=16 N=0 N=1 N=2 N=4 N=8 N=16 N=16 - -Defense Success - 43% 53% 64% 65% 66% - 46% 63% 69% 81% 84% 1% 0% 0%Normalized Distortion (L2) 1.00 1.04 1.11 1.12 1.31 1.38 1.00 1.09 1.28 1.28 1.63 1.57 1.37 1.30 1.06FP Rate - 2.9% 4.4% 6.1% 7.8% 8.4% - 6.9% 11.2% 16.2% 21.9% 27.6% - - -

the defenders in a white-box setting.We emphasize that training the de-

fender module is carried out in an un-supervised setting, meaning that no ad-versarial sample is included in the train-ing phase. We believe that leveragingan unsupervised learning approach isthe key to having a generalizable de-fense scheme that is applicable to awide class of adversarial machine learn-ing attacks. To the best of our knowl-edge, our proposed MRR approach1

is the first unsupervised countermea-sure to withstand the existing adver-sarial attacks for (deep) ML modelsincluding Fast-Gradient-Sign, JacobianSaliency Map Attack, Deepfool, andCarlini&WagnerL2 in both black-boxand white-box settings. Details aboutthe robustness of the MRR methodologyagainst the aforementioned attack meth-ods are available in our paper.1

TransferabilityIn the context of adversarial sam-

ples, transferability is defined as theability of adversarial samples to deceiveML models that have not been used bythe attack algorithm, i.e. their parame-ters and network structures were not re-vealed to the attacker. In other words,adversarial samples that are generatedfor a certain ML model can potentiallydeceive another model that has not beenexposed to the attacker. Our proposedMRR methodology is robust against

model transferability in a sense that theadversarial samples generated for thevictim model using the best-known at-tack methodologies are not transferredto the defender modules.1 This, in turn,guarantees the effective performance ofour MRR method against both white-box and black-box16 attacks.

Our key observation is that the ma-jority of adversarial samples that canbe easily transferred in between dif-ferent models are crafted from legiti-mate samples that are inherently hard-to-classify due to the closeness to deci-sion boundaries corresponding to suchclasses. For instance, in the MNISTdigit recognition task, such adversarialsamples mostly belong to class 5 that ismisclassified to class 3, or class 4 mis-classified as 9. These misclassificationsare indeed the model approximation er-ror which is well-understood due to thestatistical nature of the models.

Figure 5 shows example adversar-ial samples generated by such hard-to-classify examples. As demonstrated,even a human observer might make amistake in labeling such images. We be-lieve that a more precise definition ofadversarial samples is necessary to dis-tinguish malicious samples form thosethat simply lie near the decision bound-aries. Therefore, the notion of transfer-ability should be redefined to differen-tiate between hard-to-classify samplesand adversarial examples.

Figure 5: Example adversarial sam-ples for which accurate detection ishard due to the closeness of deci-sion boundaries for the correspond-ing data categories.

References

1. Bita Darvish Rouhani, Moham-mad Samragh, Mojan Javaheripi,Tara Javidi, and Farinaz Koushan-far. Deepfense: Online accel-erated defense against adversarialdeep learning. arXiv preprintarXiv:1709.02538, 2017.

2. Jonghoon Jin, Aysegul Dundar, andEugenio Culurciello. Robust con-volutional neural networks underadversarial noise. arXiv preprintarXiv:1511.06306, 2015.

3. Ruitong Huang, Bing Xu,Dale Schuurmans, and CsabaSzepesvári. Learning with a

6

REFERENCES REFERENCES

strong adversary. arXiv preprintarXiv:1511.03034, 2015.

4. Uri Shaham, Yutaro Yamada, andSahand Negahban. Understandingadversarial training: Increasing lo-cal stability of neural nets throughrobust optimization. arXiv preprintarXiv:1511.05432, 2015.

5. Christian Szegedy, WojciechZaremba, Ilya Sutskever, JoanBruna, Dumitru Erhan, Ian Good-fellow, and Rob Fergus. Intriguingproperties of neural networks.arXiv preprint arXiv:1312.6199,2013.

6. Shixiang Gu and Luca Rigazio.Towards deep neural networkarchitectures robust to adver-sarial examples. arXiv preprintarXiv:1412.5068, 2014.

7. Takeru Miyato, Shin-ichi Maeda,Masanori Koyama, Ken Nakae, andShin Ishii. Distributional smooth-ing with virtual adversarial training.arXiv preprint arXiv:1507.00677,2015.

8. Nicholas Carlini and David Wag-ner. Towards evaluating the robust-ness of neural networks. In Securityand Privacy (SP), 2017 IEEE Sym-posium on. IEEE, 2017.

9. Nicolas Papernot, Patrick Mc-Daniel, Xi Wu, Somesh Jha, andAnanthram Swami. Distillation asa defense to adversarial perturba-

tions against deep neural networks.2016.

10. Nicholas Carlini and David Wag-ner. Defensive distillation is not ro-bust to adversarial examples. arXivpreprint, 2016.

11. Dongyu Meng and Hao Chen. Mag-net: a two-pronged defense againstadversarial examples. In Proceed-ings of the 2017 ACM SIGSAC Con-ference on Computer and Commu-nications Security. ACM, 2017.

12. Nicholas Carlini and David Wag-ner. Magnet and" efficient defensesagainst adversarial attacks" are notrobust to adversarial examples.arXiv preprint arXiv:1711.08478,2017.

13. Aleksander Madry, AleksandarMakelov, Ludwig Schmidt, Dim-itris Tsipras, and Adrian Vladu.Towards deep learning modelsresistant to adversarial attacks.arXiv preprint arXiv:1706.06083,2017.

14. Shiwei Shen, Guoqing Jin, Ke Gao,and Yongdong Zhang. Ape-gan:Adversarial perturbation elimina-tion with gan. ICLR Submission,available on OpenReview, 2017.

15. Valentina Zantedeschi, Maria-IrinaNicolae, and Ambrish Rawat. Ef-ficient defenses against adversarialattacks. In Proceedings of the 10thACM Workshop on Artificial Intelli-gence and Security. ACM, 2017.

16. Nicolas Papernot, Patrick Mc-Daniel, Ian Goodfellow, SomeshJha, Z Berkay Celik, and Anan-thram Swami. Practical black-boxattacks against deep learning sys-tems using adversarial examples.arXiv preprint arXiv:1602.02697,2016.

Bita Darvish Rouhani is a Ph.D. candi-date in the Department of Electrical andComputer Engineering at the Universityof California San Diego. Contact her atbita@ucsd.edu.

Mohammad Samragh is a Ph.D. stu-dent in the Department of Electrical andComputer Engineering at the Universityof California San Diego. Contact him atmsamragh@ucsd.edu.

Tara Javidi is a Professor in the De-partment of Electrical and ComputerEngineering at the University of Cal-ifornia San Diego. Contact her at tja-vidi@ucsd.edu.

Farinaz Koushanfar is a Professor andHenry Booker Faculty Scholar in theDepartment of Electrical and ComputerEngineering at the University of Cali-fornia San Diego. Contact her at fari-naz@ucsd.edu.

7