MotherNets: Rapid Deep Ensemble Learning

MOTHERNETS: RAPID DEEP ENSEMBLE LEARNING

Abdul Wasay 1 Brian Hentschel 1 Yuze Liao 1 Sanyuan Chen 1 Stratos Idreos 1

ABSTRACTEnsembles of deep neural networks significantly improve generalization accuracy. However, training neuralnetwork ensembles requires a large amount of computational resources and time. State-of-the-art approacheseither train all networks from scratch leading to prohibitive training cost that allows only very small ensemble sizesin practice, or generate ensembles by training a monolithic architecture, which results in lower model diversityand decreased prediction accuracy. We propose MotherNets to enable higher accuracy and practical training costfor large and diverse neural network ensembles: A MotherNet captures the structural similarity across some orall members of a deep neural network ensemble which allows us to share data movement and computation costsacross these networks. We first train a single or a small set of MotherNets and, subsequently, we generate thetarget ensemble networks by transferring the function from the trained MotherNet(s). Then, we continue to trainthese ensemble networks, which now converge drastically faster compared to training from scratch. MotherNetshandle ensembles with diverse architectures by clustering ensemble networks of similar architecture and traininga separate MotherNet for every cluster. MotherNets also use clustering to control the accuracy vs. trainingcost tradeoff. We show that compared to state-of-the-art approaches such as Snapshot Ensembles, KnowledgeDistillation, and TreeNets, MotherNets provide a new Pareto frontier for the accuracy-training cost tradeoff.Crucially, training cost and accuracy improvements continue to scale as we increase the ensemble size (2 to 3percent reduced absolute test error rate and up to 35 percent faster training compared to Snapshot Ensembles). Weverify these benefits over numerous neural network architectures and large data sets.

1 INTRODUCTION

Neural network ensembles. Various applications increas-ingly use ensembles of multiple neural networks to scale therepresentational power of their deep learning pipelines. Forexample, deep neural network ensembles predict relation-ships between chemical structure and reactivity (Agrafiotiset al., 2002), segment complex images with multiple ob-jects (Ju et al., 2017), and are used in zero-shot as well asmultiple choice learning (Guzman-Rivera et al., 2014; Ye &Guo, 2017). Further, several winners and top performers onthe ImageNet challenge are ensembles of neural networks(Lee et al., 2015a; Russakovsky et al., 2015). Ensemblesfunction as collections of experts and have been shown, boththeoretically and empirically, to improve generalization ac-curacy (Drucker et al., 1993; Dietterich, 2000; Granitto et al.,2005; Huggins et al., 2016; Ju et al., 2017; Lee et al., 2015a;Russakovsky et al., 2015; Xu et al., 2014). For instance,by combining several image classification networks on theCIFAR-10, CIFAR-100, and SVHN data sets, ensemblescan reduce the misclassification rate by up to 20 percent,

1Harvard School of Engineering and Applied Sciences. Corre-spondence to: Abdul Wasay <[email protected]>.

Proceedings of the 3 rd MLSys Conference, Austin, TX, USA,2020. Copyright 2020 by the author(s).

training time

test

err

or ra

te

Independent (full data)

MotherNets

Snapshot Ensembles

TreeNets

Knowledge Distillation

Bagging

Number of clusters g navigate this tradeoff

g=1

g=ensemble size

Pareto frontier

Figure 1. MotherNets establish a new Pareto frontier for theaccuracy-training time tradeoff as well as navigate this tradeoff.e.g., from 6 percent to 4.5 percent for ensembles of ResNetson CIFAR-10 (Huang et al., 2017a; Ju et al., 2017).

The growing training cost. Training ensembles of mul-tiple deep neural networks takes a prohibitively largeamount of time and computational resources. Even on high-performance hardware, a single deep neural network maytake several days to train and this training cost grows lin-early with the size of the ensemble as every neural networkin the ensemble needs to be trained (Szegedy et al., 2015; Heet al., 2016; Huang et al., 2017b;a). This problem persistseven in the presence of multiple machines. This is becausethe holistic cost of training, in terms of buying or rentingout these machines through a cloud service provider, still in-

arX

iv:1

809.

0427

0v2

[cs

.LG

] 8

Mar

202

0

MotherNets: Rapid Deep Ensemble Learning

Table 1. Existing approaches to train ensembles of deep neuralnetworks are limited in speed, accuracy, diversity, and size.

Fast High Diverse Largetrain. acc. arch. size

Full data × ×Bagging ∼ × ×Knowledge Dist. ∼ × ×TreeNets ∼ ∼ × ×Snapshot Ens. × ×MotherNets

creases linearly with the ensemble size. The rising trainingcost is a bottleneck for numerous applications, especiallywhen it is critical to quickly incorporate new data and toachieve a target accuracy. For instance, in one use case,where deep learning models are applied to detect DiabeticRetinopathy (a leading cause of blindness), newly labelledimages become available every day. Thus, incorporatingnew data in the neural network models as quickly as possi-ble is crucial in order to enable more accurate diagnosis forthe immediately next patient (Gulshan et al., 2016).

Problem 1: Restrictive ensemble size. Due to this pro-hibitive training cost, researchers and practitioners can onlyfeasibly train and employ small ensembles (Szegedy et al.,2015; He et al., 2016; Huang et al., 2017b;a). In partic-ular, neural network ensembles contain drastically fewerindividual models when compared with ensembles of othermachine learning methods. For instance, random decisionforests, a popular ensemble algorithm, often has several hun-dreds of individual models (decision trees), whereas state-of-the-art ensembles of deep neural networks consist of aroundfive networks (He et al., 2016; Huang et al., 2017b;a; Oshiroet al., 2012; Szegedy et al., 2015). This is restrictive sincethe generalization accuracy of an ensemble increases withthe number of well-trained models it contains (Oshiro et al.,2012; Bonab & Can, 2016; Huggins et al., 2016). Theoreti-cally, for best accuracy, the size of the ensemble should beat least equal to the number of class labels, of which therecould be thousands in modern applications (Bonab & Can,2016).

Additional problems: Speed, accuracy, and diversity.Typically, every deep neural network in an ensemble isinitialized randomly and then trained individually using alltraining data (full data), or by using a random subset of thetraining data (i.e., bootstrap aggregation or bagging) (Juet al., 2017; Lee et al., 2015a; Moghimi & Vasconcelos,2016). This requires a significant amount of processingtime and computing resources that grow linearly with theensemble size.

To alleviate this linear training cost, two techniques havebeen recently introduced that generate a k network ensemble

from a single network: Snapshot Ensembles and TreeNets.Snapshot Ensembles train a single network and use its pa-rameters at k different points of the training process toinstantiate k networks that will form the target ensemble(Huang et al., 2017a). Snapshot Ensembles vary the learningrate in a cyclical fashion, which enables the single networkto converge to k local minima along its optimization path.TreeNets also train a single network but this network isdesigned to branch out into k sub-networks after the firstfew layers. Effectively every sub-network functions as aseparate member of the target ensemble (Lee et al., 2015b).

While these approaches do improve training time, they alsocome with two critical problems. First, the resulting en-sembles are less accurate because they are less diverse com-pared to using k different and individually trained networks.Second, these approaches cannot be applied to state-of-the-art diverse ensembles. Such ensembles may containarbitrary neural network architectures with structural differ-ences to achieve increased accuracy (for instance, such asthose used in the ImageNet competitions (Lee et al., 2015a;Russakovsky et al., 2015)).

Knowledge Distillation provides a middle ground betweenseparate training and ensemble generation approaches (Hin-ton et al., 2015). With Knowledge Distillation, an ensembleis trained by first training a large generalist network andthen distilling its knowledge to an ensemble of small spe-cialist networks that may have different architectures (bytraining them to mimic the probabilities produced by thelarger network) (Hinton et al., 2015; Li & Hoiem, 2017).However, this approach results in limited improvement intraining cost as distilling knowledge still takes around 70percent of the time needed to train from scratch. Even then,the ensemble networks are still closely tied to the samelarge network that they are distilled from. The result issignificantly lower accuracy and diversity when comparedto ensembles where every network is trained individually(Hinton et al., 2015; Li & Hoiem, 2017).

MotherNets. We propose MotherNets, which enable rapidtraining of large feed-forward neural network ensembles.The core benefits of MotherNets are depicted in Table 1.MotherNets provide: (i) lower training time and better gen-eralization accuracy than existing fast ensemble trainingapproaches and (ii) the capacity to train large ensembleswith diverse network architectures.

Figure 2 depicts the core intuition behind MotherNets: AMotherNet is a network that captures the maximum struc-tural similarity between a cluster of networks (Figure 2 Step(1)). An ensemble may consist of one or more clusters; oneMotherNet is constructed per cluster. Every MotherNet istrained to convergence using the full data set (Figure 2 Step(2)). Then, every target network in the ensemble is hatchedfrom its MotherNet using function-preserving transforma-


Specifications of ensemble networks

Step 1: Construct the MotherNet per cluster to capture the largest structural

commonality (shown in bold).

inpu

t lay

ers

output layers

Step 2: Train the MotherNet using the entire data set. This allows us to “share epochs” amongst ensemble networks.

Step 3: Hatch ensemble networks by function-preserving

transformations and further train.

MotherNet Hatched networks

neur

on

trained neuron

sele

cted

for

Mot

herN

ets clus

ter g

clus

ter

g-1

clus

ter

g+1

Figure 2. MotherNets train an ensemble of neural networks by first training a set of MotherNets and transferring the function to theensemble networks. The ensemble networks are then further trained converging significantly faster than training individually.

tions (Figure 2 Step (3)) ensuring that knowledge from theMotherNet is transferred to every network. The ensemblenetworks are then trained. They converge significantly fastercompared to training from scratch (within tens of epochs).

The core technical intuition behind the MotherNets designis that it enables us to “share epochs” between the ensemblenetworks. At a lower level what this means is that the net-works implicitly share part of the data movement and com-putation costs that manifest during training over the samedata. This design draws intuition from systems techniquessuch as “shared scans” in data systems where many queriesshare data movement and computation for part of a scanover the same data (Harizopoulos et al., 2005; Zukowskiet al., 2007; Qiao et al., 2008; Arumugam et al., 2010; Can-dea et al., 2011; Giannikis et al., 2012; Psaroudakis et al.,2013; Giannikis et al., 2014; Kester et al., 2017).

Accuracy-training time tradeoff. MotherNets do not traineach network individually but “source” all networks fromthe same set of “seed” networks instead. This introducessome reduction in diversity and accuracy compared to anapproach that trains all networks independently. There is noway around this. In practice, there is an intrinsic tradeoffbetween ensemble accuracy and training time. All existingapproaches are affected by this and their design decisionseffectively place them at a particular balance within thistradeoff (Guzman-Rivera et al., 2014; Lee et al., 2015a;Huang et al., 2017a).

We show that MotherNets, strike a superior balance betweenaccuracy and training time than all existing approaches. Infact, we show that MotherNets establish a new Pareto fron-tier for this tradeoff and that we can navigate this tradeoff.To achieve this, MotherNets cluster ensemble networks (tak-ing into account both the topology and the architecture class)and train a separate MotherNet for each cluster. The numberof clusters used (and thus the number of MotherNets) is

a knob that helps navigate the training time vs. accuracytradeoff. Figure 1 depicts visually the new tradeoff achievedby MotherNets.

Contributions. We describe how to construct MotherNetsin detail and how to trade accuracy for speed. Then througha detailed experimental evaluation with diverse data sets andarchitectures we demonstrate that MotherNets bring threebenefits: (i) MotherNets establish a new Pareto frontier ofthe accuracy-training time tradeoff providing up to 2 per-cent better absolute test error rate compared to fast ensembletraining approaches at comparable or less training cost. (ii)MotherNets allow robust navigation of this new Pareto fron-tier of the tradeoff between accuracy and training time. (iii)MotherNets enable scaling of neural network ensembles tolarge sizes (100s of models) with practical training cost andincreasing accuracy benefits.

We provide a web-based interactive demo as an additionalresource to help in understanding the training processin MotherNets: http://daslab.seas.harvard.edu/mothernets/.

2 RAPID ENSEMBLE TRAINING

Definition: MotherNet. Given a cluster of k neural net-works C = {N1, N2, . . . Nk}, where Ni denotes the i-thneural network in C, the MotherNet Mc is defined as thelargest network from which all networks in C can be ob-tained through function-preserving transformations. Moth-erNets divide an ensemble into one or more such networkclusters and construct a separate MotherNet for each.

Constructing a MotherNet for fully-connected net-works. Assume a cluster C of fully-connected neural net-works. The input and the output layers of Mc have the samestructure as all networks in C, since they are all trainedfor the same task. Mc is initialized with as many hidden

http://daslab.seas.harvard.edu/mothernets/

http://daslab.seas.harvard.edu/mothernets/


layers as the shallowest network in C. Then, we constructthe hidden layers of Mc one-by-one going from the inputto the output layer. The structure of the i-th hidden layerof Mc is the same as the i-th hidden layer of the networkin C with the least number of parameters at the i-th layer.Figure 2 shows an example of how this process works fora toy ensemble of two three-layered and one four-layeredneural networks. Here, the MotherNet is constructed withthree layers. Every layer has the same structure as the layerwith the least number of parameters at that position (shownin bold in Figure 2 Step (1)). In Appendix A we also includea pseudo-code description of this algorithm.

Constructing a MotherNet for convolutional networks.Convolutional neural network architectures consist of blocksof one or more convolutional layers separated by poolinglayers (He et al., 2016; Shazeer et al., 2017; Simonyan& Zisserman, 2015; Szegedy et al., 2015). These blocksare then followed by another block of one or more fully-connected layers. For instance, VGGNets are composedof five blocks of convolutional layers separated by max-pooling layers, whereas, DenseNets consist of four blocks ofdensely connected convolutional layers. For convolutionalnetworks, we construct the MotherNet Mc block-by-blockinstead of layer-by-layer. The intuition is that deeper orwider variants of such networks are created by adding orexpanding layers within individual blocks instead of addingthem all at the end of the network. For instance, VGG-C(with 16 convolutional layers) is obtained by adding onelayer to each of the last three blocks of VGG-B (with 13convolutional layers) (Simonyan & Zisserman, 2015). Toconstruct the MotherNet for every block, we select as manyconvolutional layers to include in the MotherNet as thenetwork in C with the least number of layers in that block.Every layer within a block is constructed such that it hasthe least number of filters and the smallest filter size of anylayer at the same position within that block. An exampleof this process is shown in Figure 3. Here, we construct aMotherNet for three convolutional neural networks block-by-block. For instance, in the first block, we include oneconvolutional layer in the MotherNet having the smallestfilter width and the least number of filters (i.e., 3 and 32respectively). In Appendix A we also include a pseudo-codedescription of this algorithm.

Constructing MotherNets for ensembles of neural net-works with different sizes and topologies. By construc-tion, the overall size and topology (sequence of layer sizes)of a MotherNet is limited by the smallest network in its clus-ter. If we were to assign a single cluster to all networks inan ensemble that has a large difference in size and topologybetween the smallest and the largest networks, there willbe a correspondingly large difference between at least oneensemble network and the MotherNet. This may lead to ascenario where the MotherNet only captures an insignifi-

3 : 64

3 : 64

3 : 32

1 : 64

3 : 64

3 : 64

3 : 64

3 : 64

3 : 64

5 : 64

3 : 64

3 : 64

3 : 72

3 : 64

5 : 64

1 : 64

3 : 32

3 : 64

3 : 64

Cluster of ensemble networks C MotherNet

a bl

ock

a po

olin

g la

yer

a co

nv.

laye

r

Net 1 Net 2 Net 3 Mc<latexit sha1_base64="Udm5xoxUxkeSJ78/C7WWjqvzZBU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MWLUNF+QBvKZjtpl242YXcjlNCf4MWDIl79Rd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHwzq7efUGkey0czSdCP6FDykDNqrPVw12f9csWtunORVfByqECuRr/81RvELI1QGiao1l3PTYyfUWU4Ezgt9VKNCWVjOsSuRUkj1H42X3VKzqwzIGGs7JOGzN3fExmNtJ5Ege2MqBnp5drM/K/WTU145WdcJqlByRYfhakgJiazu8mAK2RGTCxQprjdlbARVZQZm07JhuAtn7wKrYuqZ/n+slK/zuMowgmcwjl4UIM63EIDmsBgCM/wCm+OcF6cd+dj0Vpw8plj+CPn8wcV+I2n</latexit><latexit sha1_base64="Udm5xoxUxkeSJ78/C7WWjqvzZBU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MWLUNF+QBvKZjtpl242YXcjlNCf4MWDIl79Rd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHwzq7efUGkey0czSdCP6FDykDNqrPVw12f9csWtunORVfByqECuRr/81RvELI1QGiao1l3PTYyfUWU4Ezgt9VKNCWVjOsSuRUkj1H42X3VKzqwzIGGs7JOGzN3fExmNtJ5Ege2MqBnp5drM/K/WTU145WdcJqlByRYfhakgJiazu8mAK2RGTCxQprjdlbARVZQZm07JhuAtn7wKrYuqZ/n+slK/zuMowgmcwjl4UIM63EIDmsBgCM/wCm+OcF6cd+dj0Vpw8plj+CPn8wcV+I2n</latexit><latexit sha1_base64="Udm5xoxUxkeSJ78/C7WWjqvzZBU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MWLUNF+QBvKZjtpl242YXcjlNCf4MWDIl79Rd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHwzq7efUGkey0czSdCP6FDykDNqrPVw12f9csWtunORVfByqECuRr/81RvELI1QGiao1l3PTYyfUWU4Ezgt9VKNCWVjOsSuRUkj1H42X3VKzqwzIGGs7JOGzN3fExmNtJ5Ege2MqBnp5drM/K/WTU145WdcJqlByRYfhakgJiazu8mAK2RGTCxQprjdlbARVZQZm07JhuAtn7wKrYuqZ/n+slK/zuMowgmcwjl4UIM63EIDmsBgCM/wCm+OcF6cd+dj0Vpw8plj+CPn8wcV+I2n</latexit><latexit sha1_base64="Udm5xoxUxkeSJ78/C7WWjqvzZBU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MWLUNF+QBvKZjtpl242YXcjlNCf4MWDIl79Rd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHwzq7efUGkey0czSdCP6FDykDNqrPVw12f9csWtunORVfByqECuRr/81RvELI1QGiao1l3PTYyfUWU4Ezgt9VKNCWVjOsSuRUkj1H42X3VKzqwzIGGs7JOGzN3fExmNtJ5Ege2MqBnp5drM/K/WTU145WdcJqlByRYfhakgJiazu8mAK2RGTCxQprjdlbARVZQZm07JhuAtn7wKrYuqZ/n+slK/zuMowgmcwjl4UIM63EIDmsBgCM/wCm+OcF6cd+dj0Vpw8plj+CPn8wcV+I2n</latexit>

Figure 3. Constructing MotherNet for convolutional neu-ral networks block-by-block. For each layer, we selectthe layer with the least number of parameters from theensemble networks (shown in bold rectangles) (Notation:<filter_width>:<filter_number>).

cant amount of commonality. This would negatively affectperformance as we would not be able to share significantcomputation and data movement costs across the ensemblenetworks. This property is directly correlated with the sizeof the MotherNet.

In order to maintain the ability to share costs in diverseensembles, we partition such an ensemble into g clusters,and for every cluster, we construct and train a separateMotherNet. To perform this clustering, the m networks inthe ensemble E = {N1, N2, . . . Nm} are represented asvectors Ev = {V1, V2, . . . Vm} such that V j

i stores the sizeof the j-th layer in Ni. These vectors are zero-padded toa length of max({|N1|, |N2|, . . . |Nm|}) (where |Ni| is thenumber of layers in Ni). For convolutional neural networks,these vectors are created by first creating similarly zero-padded sub-vectors per block and then concatenating thesub-vectors to get the final vector. In this case, to fullyrepresent convolutional layers, V j

i stores a 2-tuple of filtersizes and number of filters.

Given a set of vectors Ev , we create g clusters using the bal-anced K-means algorithm while minimizing the Levenshteindistance between the vector representation of networks in acluster and its MotherNet (Levenshtein, 1966; MacQueen,1967). The Levenshtein or the edit distance between twovectors is the minimum number of edits – insertions, dele-tions, or substitutions – needed to transform one vector toanother. By minimizing this distance, we ensure that, forevery cluster, the ensemble networks can be obtained fromtheir cluster’s MotherNet with the minimal amount of editsconstrained on g. During every iteration of the K-means al-gorithm, instead of computing centers of candidate clusters,we construct MotherNets corresponding to every cluster.


Then, we use the edit distance between these MotherNetsand all networks to perform cluster reassignments.

Constructing MotherNets for ensembles of diverse ar-chitecture classes. An individual MotherNet is built fora cluster of networks that belong to a single architectureclass. Each architecture class has the property of function-preserving navigation. This is to say that given any memberof this class, we can build another member of this class withmore parameters but having the same function. Multipletypes of neural networks fall under the same architectureclass (Cai et al., 2018). For instance, we can build a sin-gle MotherNet for ensembles of AlexNets, VGGNets, andInception Nets as well as one for DenseNets and ResNets.To handle scenarios when an ensemble contains membersfrom diverse architecture classes i.e., we cannot navigatethe entire set of ensemble networks in a function-preservingmanner, we build a separate MotherNet for each class (or aset of MotherNets if each class also consists of networks ofdiverse sizes).

Overall, the techniques described in the previous paragraphsallow us to create g MotherNets for an ensemble, being ableto capture the structural similarity across diverse networksboth in terms of architecture and topology. We now describehow to train an ensemble using one or more MotherNetsto help share the data movement and computation costsamongst the target ensemble networks.

Training Step 1: Training the MotherNets. First, theMotherNet for every cluster is trained from scratch usingthe entire data set until convergence. This allows the Moth-erNet to learn a good core representation of the data. TheMotherNet has fewer parameters than any of the networksin its cluster (by construction) and thus it takes less time perepoch to train than any of the cluster networks.

Training Step 2: Hatching ensemble networks. Oncethe MotherNet corresponding to a cluster is trained, thenext step is to generate every cluster network through a se-quence of function-preserving transformations that allowus to expand the size of any feed-forward neural network,while ensuring that the function (or mapping) it learned ispreserved (Chen et al., 2016). We call this process hatch-ing and there are two distinct approaches to achieve this:Net2Net increases the capacity of the given network byadding identity layers or by replicating existing weights(Chen et al., 2016). Network Morphism, on the other hand,derives sufficient and necessary conditions that when satis-fied will extend the network while preserving its functionand provides algorithms to solve for those conditions (Weiet al., 2016; 2017).

In MotherNets, we adopt the first approach i.e., Net2Net.Not only is it conceptually simpler but in our experimentswe observe that it serves as a better starting point for further

training of the expanded network as compared to NetworkMorphism. Overall, function-preserving transformationsare readily applicable to a wide range of feed-forward neu-ral networks including VGGNets, ResNets, FractalNets,DenseNets, and Wide ResNets (Chen et al., 2016; Wei et al.,2016; 2017; Huang et al., 2017b). As such MotherNets isapplicable to all of these different network architectures. Inaddition, designing function-preserving transformations isan active area of research and better transformation tech-niques may be incorporated in MotherNets as they becomeavailable.

Hatching is a computationally inexpensive process that takesnegligible time compared to an epoch of training (Wei et al.,2016). This is because generating every network in a clus-ter through function preserving transformations requires atmost a single pass on layers in its MotherNet.

Training Step 3: Training hatched networks. To explic-itly add diversity to the hatched networks, we randomlyperturb their parameters with gaussian noise before furthertraining. This breaks symmetry after hatching and it is astandard technique to create diversity when training ensem-ble networks (Hinton et al., 2015; Lee et al., 2015b; Weiet al., 2016; 2017). Further, adding noise forces the hatchednetworks to be in a different part of the hypothesis spacefrom their MotherNets.

The hatched ensemble networks are further trained converg-ing significantly faster compared to training from scratch.This fast convergence is due to the fact that by initializingevery ensemble network through its MotherNet, we placedit in a good position in the parameter space and we needto explore only for a relatively small region instead of thewhole parameter space. We show that hatched networkstypically converge in a very small number epochs.

We experimented with both full data and bagging to trainhatched networks. We use full data because given the smallnumber of epochs needed for the hatched networks, baggingdoes not offer any significant advantage in speed while ithurts accuracy.

Accuracy-training time tradeoff. MotherNets can nav-igate the tradeoff between accuracy and training time bycontrolling the number of clusters g, which in turn controlshow many MotherNets we have to train independently fromscratch. For instance, on one extreme if g is set to m, thenevery network in E will be trained independently, yieldinghigh accuracy at the cost of higher training time. On theother extreme, if g is set to one then, all ensemble networkshave a shared ancestor and this process may yield networksthat are not as diverse or accurate, however, the trainingtime will be low.

MotherNets expose g as a tuning knob. As we show in ourexperimental analysis, MotherNets achieve a new Pareto


Table 2. We experiment with ensembles of various sizes and neural network architectures.

Ensemble Member networks Param. SE alternative Param.

V5 VGG 13, 16, 16A, 16B, and 19 from the VGGNet paper (Si-monyan & Zisserman, 2015)

682M VGG-16 × 5 690M

D5 Two variants of DenseNet-40 (with 12 and 24 convolutional filtersper layer) and three variants of DenseNet-100 (with 12, 16, and24 filters per layer) (Huang et al., 2017b)

17M DenseNet-60 × 5 17.3M

R10 Two variants each of ResNet 20, 32, 44, 56, and 110 from theResNet paper (He et al., 2016)

327M R-56 × 10 350M

V25 25 variants of VGG-16 with distinct architectures created byprogressively varying one layer from VGG16 in one of threeways: (i) increasing the number of filters, (ii) increasing the filtersize, or (iii) applying both (i) and (ii)

3410M VGG-16 × 25 3450M

V100 100 variants of VGG-16 created as described above 13640M VGG-16 × 100 13800M

frontier for the accuracy-training cost tradeoff which is awell-defined convex space. That is, with every step in in-creasing g (and consequently the number of independentlytrained MotherNets) accuracy does get better at the cost ofsome additional training time and vice versa. Conceptuallythis is shown in Figure 1. This convex space allows robustand predictable navigation of the tradeoff. For example, un-less one needs best accuracy or best training time (in whichcase the choice is simply the extreme values of g), they canstart with a single MotherNet and keep adding MotherNetsin small steps until the desired accuracy is achieved or thetraining time budget is exhausted. This process can furtherbe fine-tuned using known approaches for hyperparametertuning methods such as bayesian optimization, training onsampled data, or learning trajectory sampling (Goodfellowet al., 2016).

Parallel training. MotherNets create a new schedule for“sharing epochs” amongst networks of an ensemble but theactual process of training in every epoch remains unchanged.As such, state-of-the-art approaches for distributed train-ing such as parameter-server (Dean et al., 2012) and asyn-chronous gradient descent (Gupta et al., 2016; Iandola et al.,2016) can be applied to fully utilize as many machines asavailable during any stage of MotherNets’ training.

Fast inference. MotherNets can also be used to improveinference time by keeping the MotherNet parameters sharedacross the hatched networks. We describe this idea in Ap-pendix C.

3 EXPERIMENTAL ANALYSIS

We demonstrate that MotherNets enable a better trainingtime-accuracy tradeoff than existing fast ensemble trainingapproaches across multiple data sets and architectures. We

also show that MotherNets make it more realistic to utilizelarge neural network ensembles.

Baselines. We compare against five state-of-the-art methodsspanning both techniques that train all ensemble networksindividually, i.e., Full Data (FD) and Bagging (BA), aswell as approaches that generate ensembles by training asingle network, i.e., Knowledge Distillation (KD), SnapshotEnsembles (SE), and TreeNets (TN).

Evaluation metrics. We capture both the training cost andthe resulting accuracy of an ensemble. For the training cost,we report the wall clock time as well as the monetary cost fortraining on the cloud. For ensemble test accuracy, we reportthe test error rate under the widely used ensemble-averagingmethod (Van der Laan et al., 2007; Guzman-Rivera et al.,2012; 2014; Lee et al., 2015b). Experiments with alternativeinference methods (e.g., super learner and voting (Ju et al.,2017)) showed that the method we use does not affect theoverall results in terms of comparing the training algorithms.

Ensemble networks. We experiment with ensembles of var-ious convolutional architectures such as VGGNets, ResNets,Wide ResNets1, and DenseNets. Ensembles of these ar-chitectures have been extensively used to evaluate fast en-semble training approaches (Lee et al., 2015a; Huang et al.,2017a). Each of these ensembles are composed of networkshaving diverse architectures as described in Table 2.

To provide a fair comparison with SE (where the snapshotshave to be from the same network architecture), we cre-ate snapshots having comparable number of parameters toeach of the ensembles described above. This comparablealternatives we used for SE are also summarized in Table 2.

For TN, we varied the number of shared layers and found

1For experiments with Wide ResNets, see Appendix E.


5.5

6

6.5

7

7.5

8

8.5

0 4 8 12 16

test

err

or ra

te (%

)

training time (hrs)

single model

Pareto frontier

MN (g=1)

MN (g=2)

MN (g=3)

MN (g=4)MN (g=5)/FD

SE

KD

TN

(a) V5 (C-10)

4.7

4.8

4.9

5

5.1

5.2

5.3

4 8 12 16 20 24te

st e

rror

rate

(%)

training time (hrs)

single model

MN (g=1)

MN (g=2)

MN (g=3)MN (g=4)

MN (g=5)/FD

SE

KD

(b) D5 (C-10)

3

3.5

4

4.5

5

5.5

20 30 40 50 60 70

test

err

or ra

te (%

)

training time (hrs)

single model

MN (g=1)

MN (g=2)

MN (g=5)

MN (g=10)/FD

SE

KD

(c) R10 (C-10)

23

24

25

26

27

28

29

30

31

20 40 60 80 100

test

err

or ra

te (%

)

training time (hrs)

single model

MN (g=1)

MN (g=2)

MN (g=5)

MN (g=10)MN (g=25)/

FD

KD

SE

(d) V25 (C-100)

3.8

4

4.2

4.4

4.6

4.8

5

5.2

4 8 12 16 20 24

test

err

or ra

te (%

)

training time (hrs)

single model

MN (g=1)

MN (g=2)MN (g=5)

MN (g=10)MN (g=25)

SE

KD

(e) V25 (SVHN)

Figure 4. MotherNets provide consistently better accuracy-training time tradeoff when compared with existing fast ensemble trainingapproaches across various data sets, architectures, and ensemble sizes.

that sharing the 3 initial layers provides the best accuracy.This is similar to the optimal proportion of shared layersin the TreeNets paper (Lee et al., 2015a). TN is not appli-cable to DenseNets or ResNets as it is designed only fornetworks without skip-connections (Lee et al., 2015a). Weomit comparison with TN for such ensembles.

Training setup. For all training approaches we use stochas-tic gradient descent with a mini-batch size of 256 and batch-normalization. All weights are initialized by sampling froma standard normal distribution. Training data is randomlyshuffled before every training epoch. The learning rate is setto 0.1 with the exception of DenseNets. For DenseNets, weuse a learning rate of 0.1 to train MotherNets and 0.01 totrain hatched networks. This is inline with the learning ratedecay used in the DenseNets paper (Huang et al., 2017b).For FD, KD, TN, and MotherNets, we stop training if thetraining accuracy does not improve for 15 epochs. For SEwe use the optimized training setup proposed in the originalpaper (Huang et al., 2017a), starting with an initial learningrate of 0.2 and then training every snapshot for 60 epochs.

Data sets. We experiment with a diverse array of datasets: SVHN, CIFAR-10, and CIFAR-100 (Krizhevsky, 2009;Netzer et al.). The SVHN data set is composed of images ofhouse numbers and has ten class labels. There are a total of99K images. We use 73K for training and 26K for testing.The CIFAR-10 and CIFAR-100 data sets have 10 and 100class labels respectively corresponding to various images ofeveryday objects. There are a total of 60K images – 50Ktraining and 10K test images.

Hardware platform. All experiments are run on the sameserver with Nvidia Tesla V100 GPU.

3.1 Better training time-accuracy tradeoff

We first show how MotherNets strike an overall superioraccuracy-training time tradeoff when compared to existingfast ensemble training approaches.

Figure 4 shows results across all our test data sets and en-

semble networks. All graphs in Figure 4 depict the tradeoffbetween training time needed versus accuracy achieved. Thecore observation from Figure 4 is that across all datasets andnetworks, MotherNets help establish a new Pareto frontierof this tradeoff. The different versions of MotherNets shownin Figure 4 represent different numbers of clusters used (g).When g=1, we use a single MotherNet, optimizing for train-ing time, while when g becomes equal to the ensemble size,we optimize for accuracy (effectively this is equal to FD asevery network is trained independently in its own cluster).

The horizontal line at the top of each graph indicates the ac-curacy of the best-performing single model in the ensembletrained from scratch. This serves as a benchmark and, inthe vast majority of cases, all approaches do improve over asingle model even when they have to sacrifice on accuracyto improve training time. MotherNets is consistently andsignificantly better than that benchmark.

Next we discuss each individual training approach and howit compares to MotherNets.

MotherNets vs. KD, TN, and BA. MotherNets (with g=1)is 2× to 4.2× faster than KD and results in up to 2 percentbetter test accuracy. KD suffers in terms of accuracy becauseits ensemble networks are more closely tied to the basenetwork as they are trained from the output of the samenetwork. KD’s higher training cost is because distilling isexpensive. Every network starts from scratch and is trainedon the data set using a combination of empirical loss andthe loss from the output of the teacher network. We observethat distilling a network still takes around 60 to 70 percentof the time required to train it using just the empirical loss.

To achieve comparable accuracy to MotherNets (with g=1),TN requires up to 3.8× more training time on V 5. In thesame time budget, MotherNets can train with g=4 providingover one percent reduction in test error rate. The highertraining time of TN is due to the fact that it combines severalnetworks together to create a monolithic architecture withvarious branches. We observe that training this takes a


0

5

10

15

20

25

FD KD BA MN1

train

ing

time

(hrs

.)

training method

MNDN1DN2

DN3DN4DN5

Figure 5. MotherNets trainensemble networks signif-icantly faster after havingtrained the MotherNet(shown in black).

5 5.5

6 6.5

7 7.5

8 8.5

0 5 10 15 20 25 30 35 40 45

test

err

or ra

te (%

)training time (hrs)

SE

MN (g=1)

MN (g=8)

(k=5)

(k=10)

(k=25)

(k=50)

(k=100)

(k=5)

(k=10)(k=25) (k=50)(k=100)

(k=5)

(k=10)(k=25) (k=50) (k=100)

Figure 6. As the size of theensemble grows, MotherNetsscale better than SE both interms of training time and ac-curacy achieved.

significant time per epoch as well as requires more epochsto converge. Moreover, TN does not generalize to neuralnetworks with skip-connections.

Figure 4 does not show results for BA because it is anoutlier. BA takes on average 73 percent of the time FDneeds to train but results in significantly higher test errorrate than any of the baseline approaches including the singlemodel. Compared to BA, MotherNets is on average 3.6 ×faster and results in significantly better accuracy – up to 5.5percent lower absolute test error rate. These observations areconsistent with past studies that show how BA is ineffectivewhen training deep neural networks as it reduces the numberof unique data items seen by individual networks (Lee et al.,2015a).

Overall, the low test error rate of MotherNets when com-pared to KD, TN, and BA stems from the fact that trans-ferring the learned function from MotherNets to target en-semble networks provides a good starting point as well asintroduces regularization for further training. This also al-lows hatched ensemble networks to converge significantlyfaster, resulting in overall lower training time.

Training time breakdown. To better understand where thetime goes during the training process, Figure 5 providesthe time breakdown per ensemble network. We show thisfor the D5 ensemble and compare MotherNets (with g=1)with individual training approaches FD, BA, and KD. Whileother approaches spend significant time training each net-work, MotherNets, can train these networks very quicklyafter having trained the core MotherNet (black part in theMotherNets stacked bar in Figure 5). We observe similartime breakdown across all ensembles in our experiments.

3.2 MotherNets vs. SE and scaling to large ensembles

Across all experiments in Figure 4, SE is the closest baselineto MotherNets. In effect, SE is part of the very same Paretofrontier defined by MotherNets in the accuracy-training cost

Table 3. MotherNets (with g=1) give better oracle test accuracycompared to Snapshot ensembles.

V5C10

D5C10

R10C10

V25C100

V25SVHN

MN 96.71 97.43 98.61 87.5 97.17

SE 96.03 96.91 97.11 86.9 97.3

tradeoff. That is, it represents one more valid point that canbe useful depending on the desired balance. For example,in Figure 4a (for V5 CIFAR-10), SE sacrifices nearly onepercent in test error rate compared to MotherNets (with g=1)for a small improvement in training cost. We observe similartrends in Figure 4c and 4d). In Figure 4b, SE achieves abalance that is in between MotherNets with one and twoclusters. However, when training V25 on SVHN (Figure 4e)SE is in fact outside the Pareto frontier as it is both slowerand achieves worst accuracy.

Overall, MotherNets enables drastic improvements in eitheraccuracy or training time compared to SE by being able tocontrol and navigate the tradeoff between the two.

Oracle accuracy. Also, Table 3 shows that MotherNets(with g=1) enable better oracle test accuracy when comparedwith SE across all our experiments. This is the accuracy ifan oracle were to pick the prediction of the most accuratenetwork in the ensemble per test element (Guzman-Riveraet al., 2012; 2014; Lee et al., 2015b). Oracle accuracy is anupper bound for the accuracy that any ensemble inferencetechnique could achieve. This metric is also used to evaluatethe utility of ensembles when they are applied to solve Mul-tiple Choice Learning (MCL) problems (Guzman-Riveraet al., 2014; Lee et al., 2016; Brodie et al., 2018).

Scaling to very large ensembles. As we discussed before,large ensembles help improve accuracy and thus ideallywe would like to scale neural network ensembles to largenumber of models as it happens for other ensembles suchas random forests (Oshiro et al., 2012; Bonab & Can, 2016;2017). Our previous results were for small to medium en-sembles of 5, 10 or 25 networks. We now show that whenit comes to larger ensembles, MotherNets dominate SE inboth how accuracy and training time scale.

Figure 6 shows results as we increase the number of net-works up to a hundred variants of VGGNets trained onCIFAR-10. For every point in Figure 6, k indicates thenumber of networks. For MotherNets we plot results for thetime-optimized version with g=1, as well as with g=8.

Figure 6 shows that as the size of the ensemble grows, Moth-erNets scale much better in terms of training time. Towardthe end (for 100 networks), MotherNets train more than10 hours faster (out of 40 total hours needed for SE). The


training time of MotherNets grows at a much smaller ratebecause once the MotherNet has been trained, it takes 40percent less time to train a hatched network than the time ittakes to train one snapshot.

In addition, Figure 6 shows that MotherNets does not onlyscale better in terms of training time, but also it scales betterin terms of accuracy. As we add more networks to theensemble, MotherNets keeps improving its error rate bynearly 2 percent while SE actually becomes worse by morethan 0.5 percent. The declining accuracy of SE as the sizeof the ensemble increases has also been observed in thepast, where by increasing the number of snapshots above sixresults in degradation in performance (Huang et al., 2017a).

Finaly, Figure 6 shows that different cluster settings forMotherNets allow us to achieve different performance bal-ances while still providing robust and predictable navigationof the tradeoff. In this case, with g=8 accuracy improvesconsistently across all points (compared to g=1) at the costof extra training time.

3.3 Improving cloud training cost

0

250

500

750

1000

M1

V25 SVHNR10 C10

V25 C100

0

150

300

MN1 SE KD BA FD

train

ing

cost

(USD

)

training method

M2

Figure 7. Training cost (USD)

One approach to speedup training of large en-sembles is to utilizemore than one machines.For example, we couldtrain k individual net-works in parallel usingk machines. Whilethis does save time, theholistic cost in termsof energy and resourcesspent is still linear to theensemble size.

One proxy for capturing the holistic cost is to look at theamount of money one has to pay on the cloud for traininga given ensemble. In our next experiment, we compare allapproaches using this proxy. Figure 7 shows the cost (inUSD) of training on four cloud instances across two cloudservice providers: (i) M1 that maps to AWS P2.xlarge andAzure NC6, and (ii) M2 that maps to AWS P3.2xlarge andAzure NCv3. M1 is priced at USD 0.9 per hour and M2 ispriced at USD 3.06 per hour for both cloud service providers(Amazon, 2019; Microsoft, 2019).

Training time-optimized MotherNets provide significantreduction in training cost (up to 3×) as it can train a verylarge ensemble in a fraction of the training time comparedto other approaches.

0.025

0

0

0

0

0

0.025

0

0

0

0

0

0.027

0

0

0

0

0

0.026

− 0.001

0

0

0

− 0.001

0.026

0.027

0.01

0.009

0.009

0.009

0.01

0.027

0.009

0.009

0.008

0.009

0.009

0.028

0.009

0.008

0.009

0.009

0.009

0.028

0.008

0.009

0.008

0.008

0.008

0.028

0.034

0.008

0.008

0.007

0.008

0.008

0.028

0.01

0.009

0.009

0.008

0.01

0.027

0.01

0.01

0.007

0.009

0.01

0.028

0.01

0.008

0.009

0.01

0.01

0.027

Full Data MotherNets Snapshot Ensemble

Figure 8. MotherNets (with g=1) train ensembles with lower modelcovariances compared to Snapshot Ensembles.

3.4 Diversity of model predictions

Next, we analyze how diversity of ensembles produced byMotherNets compares with SE and FD.

Ensembles and predictive diversity. Theoretical resultssuggest that ensembles of models perform better when themodels’ predictions on a single example are less correlated.This is true under two assumptions: (i) models have equalcorrect classification probability and (ii) the ensemble usesmajority vote for classification (Krogh & Vedelsby, 1994;Rosen, 1996; Kuncheva & Whitaker, 2003). Under ensem-ble averaging, no analytical proof that negative correlationreduces error rate exists, but lower correlation between mod-els can be used to create a smaller upper bound on incorrectclassification probability. More precise statements and theirproofs are given in Appendix B.

Rapid ensemble training methods. For MotherNets, aswell as for all other compared techniques for ensembletraining, the training procedure binds the models together todecrease training time. This can have two negative effectscompared to independent training of models:

1. by changing the model’s architecture or training pat-tern, the technique affects each model’s prediction qual-ity (the model’s marginal prediction accuracy suffers)

2. by sharing layers (TN), attempted softmax values (KD),or training epochs (SE, MN), the training techniquecreates positive correlations between model errors.

We compare here the magnitude of these two effects for-MotherNets and Snapshot Ensembles when compared toindependent training of each model on CIFAR-10 using V5.

Individual model quality. For both SE and MN, theindividual model accuracy drops, but the effect is morepronounced in SE than MN. The mean misclassificationpercentage of the individual models for V5 using FD, MNand SE are 8.1%, 8.4% and 9.8% respectively. The poorperformance of SE in this area is due to its difficulty inconsistently hitting performant local minima, either becauseit overfits to the training data when trained for a long timeor because its early snapshots need to be far away from thefinal optimum to encourage diversity.


Model variance. Our goal in assessing variance is to seehow the training procedure affects how models in the en-semble correlate with each other on each example. To dothis, we train each of the five models in V 5 five times underMN, SE, and FD. Letting Yij be the softmax of the correctmodel on test example j using model i, we then estimateV ar(Yij) for each i, j and Cov(Yij , Yi′j) for each i, i′, jwith i 6= i′ using the sample variance and covariance. Toget a single number for a model, instead of one for eachtest example, we then average across all test examples, i.e.Cov(Yi, Yi′) =

1n

∑nj=1 Cov(Yij , Yi′j). For total variance

numbers for the ensemble, we perform the same procedureon Yj = 1

5

∑5i=1 Yij .

Figure 8 shows the results. As expected, independent train-ing between the models in FD makes their correspondingcovariance 0 and provides the greatest overall variance re-duction for the ensemble, with ensemble variance at 0.0051.For both SE and MN, the covariance of separate models isnon-zero at around 0.009 per pair of models; however, it isalso significantly less than the variance of a single model.As a result, both MN and SE provide significant variancereduction compared to a single model. Whereas a singlemodel has variance around 0.026, MN and SE provide en-semble variance of 0.0125 and 0.0130 respectively.

Takeaways. Since both SE and MN train nearly as fast as asingle model, they provide variance reduction in predictionat very little training cost. Additionally, for MN, at the costof higher training time, one can create more clusters andthus make the training of certain models independent of eachother, zeroing out many of the covariance terms and reduc-ing the overall ensemble variance. When compared to eachother, MN with g=1 and SE have similar variance numbers,with MN slightly lower, but MotherNets has a substantialincrease in individual model accuracy when compared toSnapshot Ensembles. As a result, its overall ensemble per-forms better.

Additional results. We demonstrate in Appendix C howMotherNets can improve inference time by 2×. In Ap-pendix D, we show how the relative behavior of MotherNetsremains the same when training using multiple GPUs. Fi-nally, in Appendix E we provide experiments with WideResNets and demonstrate how MotherNets provide betteraccuracy-training time tradeoff when compared with FastGeometric Ensembles.

4 RELATED WORK

In this section, we briefly survey additional (but orthogonal)ensemble training techniques beyond Snapshot Ensembles,TreeNets, and Knowledge Distillation.

Parameter sharing. Various related techniques share pa-rameters between different networks during ensemble train-

ing and, in doing so, improve training cost. One interpre-tation of techniques such as Dropout and Swapout is that,during training, they create several networks with sharedweights within a single network. Then, they implicitly en-semble them during inference (Wan et al., 2013; Srivastavaet al., 2014; Huang et al., 2016; Singh et al., 2016; Huanget al., 2017a). Our approach, on the other hand, captures thestructural similarity in an ensemble, where members havedifferent and explicitly defined neural network architecturesand trains it. Overall, this enables us to effectively com-bine well-known architectures together within an ensemble.Furthermore, implicit ensemble techniques (e.g., dropoutand swapout) can be used as training optimizations to im-prove upon the accuracy of individual networks trained inMotherNets (Srivastava et al., 2014; Singh et al., 2016).

Efficient deep network training. Various algorithmic tech-niques target fundamental bottlenecks in the training process(Niu et al., 2011; Brown et al., 2016; Bottou et al., 2018).Others apply system-oriented techniques to reduce mem-ory overhead and data movement (De Sa & Feldman, 2017;Jain et al., 2018). Recently, specialized hardware is beingdeveloped to improve performance, parallelism, and energyconsumption of neural network training (Prabhakar et al.,2016; De Sa & Feldman, 2017; Jouppi et al., 2017). Alltechniques to improve upon training efficiency of individualneural networks are orthogonal to MotherNets and in factdirectly compatible. This is because MotherNets does notmake any changes to the core computational componentsof the training process. In our experiments, we do utilizesome of the widely applied training optimizations such asbatch-normalization and early-stopping. The advantage thatMotherNets bring on top of these approaches is that we canfurther reduce the total number of epochs that are required totrain an ensemble. This is because a set of MotherNets willtrain for the structural similarity present in the ensemble.

5 CONCLUSION

We present MotherNets which enable training of large anddiverse neural network ensembles while being able to nav-igate a new Pareto frontier with respect to accuracy andtraining cost. The core intuition behind MotherNets is toreduce the number of epochs needed to train an ensemble bycapturing the structural similarity present in the ensembleand training for it once.

6 ACKNOWLEDGMENTS

We thank reviewers for their valuable feedback. We alsothank Chang Xu for building the web-based demo and allDASlab members for their help. This work was partiallyfunded by Tableau, Cisco, and the Harvard Data ScienceInstitute.


REFERENCES

Agrafiotis, D. K., Cedeno, W., and Lobanov, V. S. On theuse of neural network ensembles in qsar and qspr. Journalof chemical information and computer sciences, 42(4),2002.

Amazon. Aws pricing. https://aws.amazon.com/pricing/, 2019. (Accessed on 05/16/2019).

Arumugam, S., Dobra, A., Jermaine, C. M., Pansare,N., and Perez, L. The DataPath System: A Data-centric Analytic Processing Engine for Large Data Ware-houses. In Proceedings of the ACM SIGMOD Inter-national Conference on Management of Data, pp. 519–530, 2010. URL http://dl.acm.org/citation.cfm?id=1807167.1807224.

Bonab, H. R. and Can, F. A theoretical framework on theideal number of classifiers for online ensembles in datastreams. In Proceedings of the 25th ACM International onConference on Information and Knowledge Management,2016.

Bonab, H. R. and Can, F. Less is more: A comprehensiveframework for the number of components of ensembleclassifiers. IEEE Transactions on Neural Networks andLearning Systems, 2017.

Bottou, L., Curtis, F. E., and Nocedal, J. Optimizationmethods for large-scale machine learning. SIAM Review,60(2), 2018.

Brodie, M., Tensmeyer, C., Ackerman, W., and Martinez,T. Alpha model domination in multiple choice learning.In IEEE International Conference on Machine Learningand Applications (ICMLA), 2018.

Brown, K. J., Lee, H., Rompf, T., Sujeeth, A. K., Sa, C. D.,Aberger, C. R., and Olukotun, K. Have abstraction and eatperformance, too: Optimized heterogeneous computingwith parallel patterns. In Proceedings of the InternationalSymposium on Code Generation and Optimization, 2016.

Cai, H., Chen, T., Zhang, W., Yu, Y., and Wang, J. Efficientarchitecture search by network transformation. In AAAIConference on Artificial Intelligence, 2018.

Candea, G., Polyzotis, N., and Vingralek, R. Pre-dictable Performance and High Query Concurrencyfor Data Analytics. The VLDB Journal, 20(2):227–248, 2011. URL http://dl.acm.org/citation.cfm?id=1969331.1969355.

Chen, T., Goodfellow, I. J., and Shlens, J. Net2net: Accel-erating learning via knowledge transfer. In InternationalConference on Learning Representations (ICLR), SanJuan, Puerto Rico, May 2-4, 2016, Conference TrackProceedings, 2016.

De Sa, C. and Feldman, M. Understanding and optimizingasynchronous low-precision stochastic gradient descent.In Annual International Symposium on Computer Archi-tecture (ISCA), 2017.

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M.,Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al.Large scale distributed deep networks. In Advances inNeural Information Processing Systems, 2012.

Dietterich, T. G. Ensemble methods in machine learning. InInternational Workshop on Multiple Classifier Systems,2000.

Drucker, H., Schapire, R., and Simard, P. Improving per-formance in neural networks using a boosting algorithm.In Advances in Neural Information Processing Systems,1993.

Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., andWilson, A. G. Loss surfaces, mode connectivity, and fastensembling of dnns. In Advances in Neural InformationProcessing Systems, 2018.

Giannikis, G., Alonso, G., and Kossmann, D. SharedDB:Killing One Thousand Queries with One Stone. Pro-ceedings of the VLDB Endowment, 5(6):526–537,2012. URL http://dl.acm.org/citation.cfm?id=2168651.2168654.

Giannikis, G., Makreshanski, D., Alonso, G., and Koss-mann, D. Shared Workload Optimization. Pro-ceedings of the VLDB Endowment, 7(6):429–440,2014. URL http://www.vldb.org/pvldb/vol7/p429-giannikis.pdf.

Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning.MIT Press, 2016.

Granitto, P. M., Verdes, P. F., and Ceccatto, H. A. Neural net-work ensembles: Evaluation of aggregation algorithms.Artificial Intelligence, 163(2), 2005.

Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu,D., Narayanaswamy, A., Venugopalan, S., Widner, K.,Madams, T., Cuadros, J., et al. Development and valida-tion of a deep learning algorithm for detection of diabeticretinopathy in retinal fundus photographs. Jama, 316(22),2016.

Gupta, S., Zhang, W., and Wang, F. Model accuracy andruntime tradeoff in distributed deep learning: A system-atic study. In IEEE International Conference on DataMining (ICDM), 2016.

Guzman-Rivera, A., Batra, D., and Kohli, P. Multiple choicelearning: Learning to produce multiple structured outputs.In Advances in Neural Information Processing Systems,2012.

https://aws.amazon.com/pricing/

https://aws.amazon.com/pricing/

http://dl.acm.org/citation.cfm?id=1807167.1807224






http://www.vldb.org/pvldb/vol7/p429-giannikis.pdf

http://www.vldb.org/pvldb/vol7/p429-giannikis.pdf


Guzman-Rivera, A., Kohli, P., Batra, D., and Rutenbar, R.Efficiently enforcing diversity in multi-output structuredprediction. In Artificial Intelligence and Statistics, 2014.

Harizopoulos, S., Shkapenyuk, V., and Ailamaki, A.QPipe: A Simultaneously Pipelined Relational QueryEngine. In Proceedings of the ACM SIGMOD Inter-national Conference on Management of Data, pp. 383–394, 2005. URL http://dl.acm.org/citation.cfm?id=1066157.1066201.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residuallearning for image recognition. In IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2016.

Hinton, G., Vinyals, O., and Dean, J. Distilling the knowl-edge in a neural network. In NIPS Deep Learning andRepresentation Learning Workshop, 2015.

Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger,K. Q. Deep networks with stochastic depth. In EuropeanConference on Computer Vision, 2016.

Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., andWeinberger, K. Q. Snapshot ensembles: Train 1, getm for free. 5th International Conference on LearningRepresentations (ICLR), 2017a.

Huang, G., Liu, Z., van der Maaten, L., and Weinberger,K. Q. Densely connected convolutional networks. InIEEE Conference on Computer Vision and Pattern Recog-nition, (CVPR), 2017b.

Huggins, J., Campbell, T., and Broderick, T. Coresets forscalable bayesian logistic regression. In Advances inNeural Information Processing Systems, 2016.

Iandola, F. N., Moskewicz, M. W., Ashraf, K., and Keutzer,K. Firecaffe: Near-linear acceleration of deep neural net-work training on compute clusters. In IEEE Conferenceon Computer Vision and Pattern Recognition, 2016.

Jain, A., Phanishayee, A., Mars, J., Tang, L., and Pekhi-menko, G. Gist: Efficient data encoding for deep neuralnetwork training. In IEEE Annual International Sympo-sium on Computer Architecture, 2018.

Jouppi, N. P. et al. In-datacenter performance analysis of atensor processing unit. In Annual International Sympo-sium on Computer Architecture (ISCA), 2017.

Ju, C., Bibaut, A., and van der Laan, M. J. The relativeperformance of ensemble methods with deep convolu-tional neural networks for image classification. CoRR,abs/1704.01664, 2017.

Kester, M. S., Athanassoulis, M., and Idreos, S. AccessPath Selection in Main-Memory Optimized Data Systems:

Should I Scan or Should I Probe? In Proceedings of theACM SIGMOD International Conference on Managementof Data, pp. 715–730, 2017. ISBN 9781450341974. doi:10.1145/3035918.3064049. URL http://dl.acm.org/citation.cfm?doid=3035918.3064049.

Keuper, J. and Preundt, F.-J. Distributed training of deepneural networks: Theoretical and practical limits of paral-lel scalability. In 2016 2nd Workshop on Machine Learn-ing in HPC Environments (MLHPC), pp. 19–26. IEEE,2016.

Krizhevsky, A. Learning multiple layers of features fromtiny images. 2009.

Krogh, A. and Vedelsby, J. Neural network ensembles,cross validation and active learning. In InternationalConference on Neural Information Processing Systems,1994.

Kuncheva, L. I. and Whitaker, C. J. Measures of diversityin classifier ensembles and their relationship with theensemble accuracy. Machine Learning, 51(2), 2003.

Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D. J.,and Batra, D. Why M heads are better than one:Training a diverse ensemble of deep networks. CoRR,abs/1511.06314, 2015a.

Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D. J.,and Batra, D. Why M heads are better than one:Training a diverse ensemble of deep networks. CoRR,abs/1511.06314, 2015b.

Lee, S., Prakash, S. P. S., Cogswell, M., Ranjan, V., Cran-dall, D., and Batra, D. Stochastic multiple choice learningfor training diverse deep ensembles. In Advances in Neu-ral Information Processing Systems, 2016.

Levenshtein, V. I. Binary codes capable of correcting dele-tions, insertions, and reversals. 1966.

Li, Z. and Hoiem, D. Learning without forgetting. IEEETransactions on Pattern Analysis and Machine Intelli-gence, 2017.

MacQueen, J. Some methods for classification and analysisof multivariate observations. In Berkeley Symposium onMathematical Statistics and Probability, 1967.

Microsoft. Pricing - windows virtual ma-chines | microsoft azure. https://azure.microsoft.com/en-us/pricing/details/virtual-machines/windows/, 2019. (Accessedon 05/16/2019).

Moghimi, M. and Vasconcelos, N. Boosted convolutionalneural networks. In Proceedings of the British MachineVision Conference, 2016.



http://dl.acm.org/citation.cfm?doid=3035918.3064049

http://dl.acm.org/citation.cfm?doid=3035918.3064049

https://azure.microsoft.com/en-us/pricing/details/virtual-machines/windows/




Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,and Ng, A. Y. Reading digits in natural images withunsupervised feature learning.

Niu, F., Recht, B., Ré, C., and Wright, S. J. Hogwild!:A lock-free approach to parallelizing stochastic gradientdescent. In Advances in Neural Information ProcessingSystems, 2011.

Oshiro, T. M., Perez, P. S., and Baranauskas, J. A. Howmany trees in a random forest? In International Work-shop on Machine Learning and Data Mining in PatternRecognition. Springer, 2012.

Prabhakar, R., Koeplinger, D., Brown, K. J., Lee, H.,Sa, C. D., Kozyrakis, C., and Olukotun, K. Generat-ing configurable hardware from parallel patterns. InProceedings of the Twenty-First International Confer-ence on Architectural Support for Programming Lan-guages and Operating Systems, ASPLOS ’16, Atlanta,GA, USA, April 2-6, 2016, pp. 651–665, 2016. doi:10.1145/2872362.2872415. URL http://doi.acm.org/10.1145/2872362.2872415.

Psaroudakis, I., Athanassoulis, M., and Ailamaki, A.Sharing Data and Work Across Concurrent AnalyticalQueries. Proceedings of the VLDB Endowment, 6(9):637–648, 2013. URL http://dl.acm.org/citation.cfm?id=2536360.2536364.

Qiao, L., Raman, V., Reiss, F., Haas, P. J., and Lohman,G. M. Main-memory Scan Sharing for Multi-coreCPUs. Proceedings of the VLDB Endowment, 1(1):610–621, 2008. URL http://dl.acm.org/citation.cfm?id=1453856.1453924.

Rosen, B. E. Ensemble learning using decorrelated neuralnetworks. Connection Science, 1996.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., et al. Imagenet large scale visual recognition chal-lenge. International Journal of Computer Vision, 115(3),2015.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le,Q., Hinton, G., and Dean, J. Outrageously large neuralnetworks: The sparsely-gated mixture-of-experts layer.International Conference on Learning Representations(ICLR), 2017.

Simonyan, K. and Zisserman, A. Very deep convolutionalnetworks for large-scale image recognition. In Interna-tional Conference on Learning Representations (ICLR),2015.

Singh, S., Hoiem, D., and Forsyth, D. Swapout: Learningan ensemble of deep architectures. In Advances in NeuralInformation Processing Systems, 2016.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I.,and Salakhutdinov, R. Dropout: A simple way to preventneural networks from overfitting. The Journal of MachineLearning Research, 15(1), 2014.

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.,Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich,A. Going deeper with convolutions. In Computer Visionand Pattern Recognition (CVPR), 2015.

Van der Laan, M. J., Polley, E. C., and Hubbard, A. E. Superlearner. Statistical applications in genetics and molecularbiology, 6(1), 2007.

Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, R.Regularization of neural networks using dropconnect. InInternational Conference on Machine Learning, 2013.

Wei, T., Wang, C., Rui, Y., and Chen, C. W. Network mor-phism. In International Conference on Machine Learning,2016.

Wei, T., Wang, C., and Chen, C. W. Modularized morphingof neural networks. CoRR, abs/1701.03281, 2017.

Xu, L., Ren, J. S., Liu, C., and Jia, J. Deep convolutionalneural network for image deconvolution. In Advances inNeural Information Processing Systems, 2014.

Ye, M. and Guo, Y. Self-training ensemble networks forzero-shot image recognition. Knowl.-Based Syst., 123:41–60, 2017.

Zagoruyko, S. and Komodakis, N. Wide residual networks.Proceedings of the British Machine Vision Conference,2016.

Zukowski, M., Héman, S., Nes, N. J., and Boncz, P. A.Cooperative Scans: Dynamic Bandwidth Sharing ina DBMS. In Proceedings of the International Con-ference on Very Large Data Bases (VLDB), pp. 723–734, 2007. URL http://dl.acm.org/citation.cfm?id=1325851.1325934.

http://doi.acm.org/10.1145/2872362.2872415

http://doi.acm.org/10.1145/2872362.2872415








APPENDIX

Algorithm A Constructing the MotherNet for fully-connected neural networks

Input: E: ensemble networks in one cluster;Initialize: M: empty MotherNet;

// set input/output layer sizes

M.input.num_param← E[0].input.num_param;M.output.num_param← E[0].output.num_param;M.num_layers← getShallowestNetwork(E).num_layers;

// set hidden layer sizes

for i← 0 . . . M.num_layers-1 doM.layers[i].num_param← getMin(E,i);

return M;

// Get the min. size layer at posn

Function getMin(E,posn)min← E[0].layers[posn].num_param;for j ← 0 . . . len(E)-1 do

if E[j].layers[posn].num_param < min thenmin← E[j].layers[posn].num_param

return min;

A ALGORITHMS FOR CONSTRUCTINGMOTHERNETS

We outline algorithms for constructing the MotherNet givena cluster of neural networks. We describe the algorithms forboth fully-connected and convolutional neural networks.

Fully-Connected Neural Networks. Algorithm A de-scribes how to construct the MotherNet for a cluster offully-connected neural networks. We proceed layer-by-layerselecting the layer with the least number of parameters atevery position.

Convolutional Neural Networks. Algorithm B provides adetailed strategy to construct the MotherNet for a cluster ofconvolutional neural networks. We proceed block-by-block,where each block is composed of multiple convolutionallayers. The MotherNet has as many blocks as the networkwith the least number of blocks. Then, for every block, weproceed layer-by-layer and construct the MotherNet layerat every position as follows: First, we compute the leastnumber of convolutional filters and convolutional filter sizesat that position across all ensemble networks. Let thesebe Fmin and Smin respectively. Then, in MotherNet, weinclude a convolutional layer with Fmin filters of Smin sizeat that position.

B MODEL COVARIANCE AND ENSEMBLEPREDICTIVE ACCURACY

We can analyze how model covariance effects ensembleperformance by using Chebyshev’s Inequality to bound thechance that a model predicts an example incorrectly. Byshowing that lower covariance between models makes thisbound on the probability smaller, we give an intuitive rea-son why ensembles with lower covariance between modelsperform better. The proof shows as well that the averagemodel’s predictive accuracy is important; finally, no assump-tions need to be made for the proof to hold. The individualmodels can be of different quality and have different chancesof getting each example correct.

Given a fixed training dataset, let Yi be the softmax valueof model i in the ensemble for the correct class, and letY = 1

m

∑mi=1 Yi be the ensemble’s average softmax value

on the correct class. Both are random variables with therandomness of Y and Yi coming through the randomnessof neural network training. Under the mild assumption thatE[Y ] > 1

2 , so that the a one vs. all softmax classifier wouldsay on average that the correct class is more likely, thanChebyshev’s Inequality bounds the probability of incorrectprediction. Namely, the correct prediction is made with cer-tainty if Y ≥ 1

2 and so the probability of incorrect predictionis less than

P (|Y − E[Y ]| ≥ E[Y ]− 1

2) ≤ V ar(Y )

(E[Y ]− 12 )

2

From the form of the equation, we immediately see thatkeeping the average model accuracy E[Yi] high is impor-tant, and that degradation in model quality can offset reduc-tions in variance. Since the variance of Y decomposes into1

m2 (∑m

i=1 V ar(Yi)+∑

i6=i′ Cov(Yi, Yi′)), we see that lowmodel covariance keeps the variance of the ensemble low,and that models which have which have high covariancewith other models provides little benefit to the ensemble.

We explain how MotherNets improve the efficiency of en-semble inference.

Ensemble inference. Inference in an ensemble of neuralnetworks proceeds as follows: First, the data item (e.g., animage or a feature vector) is passed through every networkin the ensemble. These forward passes produce multiplepredictions – one prediction for every network in the ensem-ble. The prediction of the ensemble is then computed bycombining the individual predictions using some averagingor voting function. As the size of the ensemble grows, theinference cost in terms of memory and time required forinference increases linearly. This is because for every addi-tional ensemble network, we need to maintain its parametersas well as do an additional forward pass on them.


Algorithm B Constructing the MotherNet for convolutional neural networks block-by-block.

Input: E: ensemble of convolutional networks in one cluster;Initialize: M: empty MotherNet;

// set input/output layer sizes and number of blocks

M.input.num_param← E[0].input.num_param;M.output.num_param← E[0].output.num_param;M.num_blocks← getShallowestNetwork(E).num_blocks;

// set hidden layers block-by-block

for k ← 0 . . . M.num_blocks-1 doM.block[k].num_hidden← getShallowestBlockAt(E,k).num_hidden; // select the shallowest block

for i← 0 . . . M.block[k].num_hidden-1 doM.block[k].hidden[i]..num_filters, M.block[k].hidden[i]..filter_size← getMin(E,k,i)

return M;

// Get minimum number of filters and filter size at posn

Function getMin(E,blk,posn)min_num_filters← E[0].block[blk].hidden[posn].num_filters;min_filter_size← E[0].block[blk].hidden[posn].filter_size;for j ← 0 . . . len(E) do

if E[j].block[blk].hidden[posn].num_filters < min_num_filters thenmin_num_filters← E[j].block[blk].hidden[posn].num_filters;

if E[j].block[blk].hidden[posn].filter_size < min_filter_size thenmin_filter_size← E[j].block[blk].hidden[posn].filter_size;

return min_num_filters, min_filter_size;

Shared MotherNetsHatched ensemble networks

Shared param.Ensemble param.

Figure A. To construct a shared-MotherNet, parameters originatingfrom the MotherNet are combined together in the ensemble.

C SHARED-MOTHERNETS

Shared-MotherNets. We introduce shared-MotherNets toreduce inference time and memory requirement of ensem-bles trained through MotherNets. In shared-MotherNets, af-ter the process of hatching (step 2 from §2), the parametersoriginating from the MotherNet are incrementally trainedin a shared manner. This yields a neural network ensemblewith a single copy of MotherNet parameters reducing bothinference time and memory requirement.

Constructing a shared-MotherNet. Given an ensembleE of K hatched networks (i.e., those networks that areobtained from a trained MotherNet), we construct a shared-

MotherNet S as follows: First, S is initialized with K inputand output layers, one for every hatched network. This al-lows S to produce as many as K predictions. Then, everyhidden layer of S is constructed one-by-one going fromthe input to the output layer and consolidating all neuronsacross all of E that originate from the MotherNet. To con-solidate a MotherNet neuron at layer li, we first reduce thek copies of that neuron (across all K networks in H) to asingle copy. All inputs to the neuron that may originatefrom various other neurons in the layer li−1 across differenthatched networks are added together. The output of thisconsolidated neuron is then forwarded to all neurons in thenext layer li+1 (across all hatched networks) which wereconnected to the consolidated neuron.

Figure A shows an example of how this process works fora simple ensemble of three hatched networks. The filledcircles represent neurons originating from the MotherNetand the colored circles represent neurons from ensemblenetworks. To construct the shared-MotherNet (shown onthe right), we go layer-by-layer consolidating MotherNetneurons.

The shared-MotherNet is then trained incrementally. Thisproceeds similarly to step 3 from §2, however, now throughthe shared-MotherNet, the neurons originating from theMotherNet are trained jointly. This results in an ensemble


6

7

8

err.

rate

(%)

MN

Shared-MN

0

10

20

1 2 3 4 5

inf.

time

(ms)

number of clusters

Figure B. Shared MotherNetsimprove inference time by 2×for the V5 ensemble.

10

100

1000

10000

1 2 4 8tra

inin

g tim

e (m

in)

number of GPUs

V5

Figure C. MotherNets con-tinue to improve training costin parallel settings (V5).

10

100

1000

10000

1 2 4 8

train

ing

time

(min

)

number of GPUs

D5 FDSE

MN

Figure D. MotherNets is ableto utilize multiple GPUs effec-tively scaling better than SE.

0 2 4 6 8

10 12 14 16 18 20

C10 C100

test

err

or ra

te (%

) FGEMN

Figure E. MotherNets outper-form FGE on Wide ResNet en-sembles.

that has K outputs, but some parameters between the net-works are shared instead of being completely independent.This reduces the overall number of parameters, improvingboth the speed and the memory requirement of inference.

Memory reduction. Assume an ensemble E ={N0, N1, . . . NK−1} of K neural networks (where Ni de-notes a neural network architecture in the ensemble with|Ni| number of parameters) and its MotherNet M . Thenumber of parameters in the ensemble is reduced by a factorof χ given by:

χ = 1− k|M |∑K−1i=0 |Ni|

Results. Figure B shows how shared-MotherNets improvesinference time for an ensemble of 5 variants of VGGNetas described in Table 1. This ensemble is trained on theCIFAR-10 data set. We report both overall ensemble testerror rate and the inference time per image. We see animprovement of 2× with negligible loss in accuracy. Thisimprovement is because shared-MotherNets has a reducednumber of parameters requiring less computation duringinference time. This improvement scales with the ensemblesize.

D PARALLEL TRAINING

Deep learning pipelines rely on clusters of multiple GPUsto train computationally-intensive neural networks. Mother-Nets continue to improve training time in such cases whenan ensemble is trained on more than one GPUs. We showthis experimentally.

To train an ensemble of multiple networks, we queue allnetworks that are ready to be trained and assign them toavailable GPUs in the following fashion: If the number ofready networks is greater than free GPUs, then we assign aseparate network to every GPU. If the number of ensemblenetworks available to be trained are less than the number of

idle GPUs, then we assign one network to multiple GPUsdividing idle GPUs equally between networks. In suchcases, we adopt data parallelism to train a network acrossmultiple machines (Dean et al., 2012).

We train on a cluster of 8 Nvidia K80 GPUs and vary thenumber of available GPUs from 1 to 8. The training hyper-parameters are the same as described in Section 3. FigureC and Figure D show the time to train the V5 and D5 en-sembles respectively across FD, SE, and MotherNets. Weobserve that compared to Snapshot Ensembles, MotherNets(g=1) scale better as we increase the number of GPUs. Thereason for this is that after the MotherNet has been trained,the rest of the ensemble networks are all ready to be trained.They can then be trained in a way that minimizes communi-cation overhead by assigning them to as distinct set of GPUsas possible. Snapshot Ensembles, on the other hand, aregenerated one after the other. In a parallel setting this boilsdown to training a single network across multiple GPUs,which incurs communication overhead that increases as thenumber of GPUs increases (Keuper & Preundt, 2016).

E IMPROVING OVER FAST GEOMETRICENSEMBLES

Now we compare against Fast Geometric Ensembles (FGE),a technique closely related to Snapshot Ensembles (SE)(Huang et al., 2017a; Garipov et al., 2018). FGE also trainsa single neural network architecture and saves the network’sparameters at various points of its training trajectory. In par-ticular, FGE uses a cyclical geometric learning rate scheduleto explore various regions in a neural network’s loss surfacethat have a low test error (Garipov et al., 2018). As thelearning rate reaches its lowest value in a cycle, FGE saves‘snapshots’ of the network parameters. These snapshots arethen used in an ensemble.

We compare MotherNets to FGE using an ensemble of WideResidual Networks trained on CIFAR-10 and CIFAR-100


(Zagoruyko & Komodakis, 2016). Our experiment consistsof an ensemble with six WRN-28-10. For MotherNets, weuse six variants of this architecture having different numberof filters and filter widths. For FGE all networks in theensemble are the same as is required by the approach. For afair comparison, the number of parameters are kept identicalbetween the two approaches. We use the same traininghyperparameters as discussed in the FGE paper and train

for a full training budget of 200 epochs. MotherNets isalso allocated the same training budget. The MotherNetis trained for a 140 epochs and every ensemble networkis trained for 10 epochs after hatching. The experimentalhardware is the same as outlined in Section 3. Figure Eshows that for identical training budget MotherNets is moreaccurate than FGE across both data sets.

Documents

MotherNets: Rapid Deep Ensemble Learning