Embed Size (px)
Citation preview
Optimizing Millions of Hyperparameters by Implicit Differentiation
Jonathan Lorraine Paul Vicol David DuvenaudUniversity of Toronto, Vector Institute
{lorraine, pvicol, duvenaud}@cs.toronto.edu
AbstractWe propose an algorithm for inexpensivegradient-based hyperparameter optimizationthat combines the implicit function theorem(IFT) with efficient inverse Hessian approx-imations. We present results on the rela-tionship between the IFT and differentiat-ing through optimization, motivating our al-gorithm. We use the proposed approach totrain modern network architectures with mil-lions of weights and millions of hyperparame-ters. We learn a data-augmentation network—where every weight is a hyperparameter tunedfor validation performance—that outputs aug-mented training examples; we learn a distilleddataset where each feature in each datapointis a hyperparameter; and we tune millionsof regularization hyperparameters. Jointlytuning weights and hyperparameters with ourapproach is only a few times more costly inmemory and compute than standard training.
1 IntroductionThe generalization of neural networks (NNs) dependscrucially on the choice of hyperparameters. Hyperpa-rameter optimization (HO) has a rich history [1, 2], andachieved recent success in scaling due to gradient-basedoptimizers [3–11]. There are dozens of regularizationtechniques to combine in deep learning, and each mayhave multiple hyperparameters [12]. If we can scale HOto have as many—or more—hyperparameters as param-eters, there are various exciting regularization strategiesto investigate. For example, we could learn a distilleddataset with a hyperparameter for every feature ofeach input [4, 13], weights on each loss term [14–16],or augmentation on each input [17, 18].
When the hyperparameters are low-dimensional—e.g., 1-5 dimensions—simple methods, like randomsearch, work; however, these break down for medium-dimensional HO—e.g., 5-100 dimensions. We mayuse more scalable algorithms like Bayesian Optimiza-tion [19–21], but this often breaks down for high-dimensional HO—e.g., >100 dimensions. We can solve
Submitted to AISTATS 2020.
high-dimensional HO problems locally with gradient-based optimizers, but this is difficult because we mustdifferentiate through the optimized weights as a func-tion of the hyperparameters. In other words, we mustapproximate the Jacobian of the best-response functionof the parameters to the hyperparameters.
We leverage the Implicit Function Theorem (IFT) tocompute the optimized validation loss gradient withrespect to the hyperparameters—hereafter denoted thehypergradient. The IFT requires inverting the trainingHessian with respect to the NN weights, which is infea-sible for modern, deep networks. Thus, we propose anapproximate inverse, motivated by a link to unrolleddifferentiation [3] that scales to Hessians of large NNs,is more stable than conjugate gradient [22, 7], and onlyrequires a constant amount of memory.
Finally, when fitting many parameters, the amount ofdata can limit generalization. There are ad hoc rulesfor partitioning data into training and validation sets—e.g., using 10% for validation. Often, practitionersre-train their models from scratch on the combinedtraining and validation partitions with optimized hy-perparameters, which can provide marginal test-timeperformance increases. We verify empirically that stan-dard partitioning and re-training procedures performwell when fitting few hyperparameters, but break downwhen fitting many. When fitting many hyperparame-ters, we need a large validation partition, which makesre-training our model with optimized hyperparametersvital for strong test performance.
Contributions
• We propose a stable inverse Hessian approximationwith constant memory cost.
• We show that the IFT is the limit of differentiatingthrough optimization.
• We scale IFT-based hyperparameter optimizationto modern, large neural architectures, includingAlexNet and LSTM-based language models.
• We demonstrate several uses for fitting hyperpa-rameters almost as easily as weights, including per-parameter regularization, data distillation, andlearned-from-scratch data augmentation methods.
• We explore how training-validation splits shouldchange when tuning many hyperparameters.
arX
iv:1
911.
0259
0v1
[cs
.LG
] 6
Nov
201
9
Jonathan Lorraine, Paul Vicol, David Duvenaud
Parameter w Hyper
para
mete
r λL
oss L
Tra
in(w, λ
)
L(λ,w∗(λ))
w∗(λ) =Implictbest− responsefunction
w∗
λ=
Implicit derivativeL(λ0,w)
λ0,w∗(λ0) Parameter w Hyp
erpa
ramete
r λL
oss L
Valid.(w
)
L∗(λ)
L∗λ
=
Lλ
+ Lw
w∗
λ=
Hypergradient
Figure 1: Overview of gradient-based hyperparameter optimization (HO). Left: a training loss manifold; Right: avalidation loss manifold. The implicit function w∗(λ) is the best-response of the weights to the hyperparametersand shown in blue projected onto the (λ,w)-plane. We get our desired objective function L∗V(λ) when thebest-response is put into the validation loss, shown projected on the hyperparameter axis in red. The validationloss does not depend directly on the hyperparameters, as is typical in hyperparameter optimization. Instead, thehyperparameters only affect the validation loss by changing the weights’ response. We show the best-responseJacobian in blue, and the hypergradient in red.
2 Overview of Proposed AlgorithmThere are four essential components to understandingour proposed algorithm. Further background is pro-vided in Appendix A, and notation is shown in Table 5.
1. HO is nested optimization: Let LT and LVdenote the training and validation losses, w the NNweights, and λ the hyperparameters. We aim to find op-timal hyperparameters λ∗ such that the NN minimizesthe validation loss after training:
λ∗ :=argminλ
L∗V(λ) where (1)
L∗V(λ) :=LV(λ,w∗(λ)) and w∗(λ) :=argminw
LT(λ,w) (2)
Our implicit function is w∗(λ), which is the best-responseof the weights to the hyperparameters. We assumeunique solutions to argmin for simplicity.
2. Hypergradients have two terms: For gradient-based HO we want the hypergradient ∂L∗V(λ)
∂λ , whichdecomposes into:
∂L∗V(λ)∂λ︸ ︷︷ ︸
hypergradient
=(
∂LV∂λ + ∂LV
∂w∂w∗∂λ
)∣∣∣∣λ,w∗(λ)
=
∂LV(λ,w∗(λ))∂λ︸ ︷︷ ︸
hyperparam direct grad.
+
hyperparam indirect grad.︷ ︸︸ ︷∂LV(λ,w∗(λ))
∂w∗(λ)︸ ︷︷ ︸parameter direct grad.
× ∂w∗(λ)∂λ︸ ︷︷ ︸
best-response Jacobian
(3)
The direct gradient is easy to compute, but the indirectgradient is difficult to compute because we must ac-count for how the optimal weights change with respectto the hyperparameters (i.e., ∂w∗(λ)
∂λ ). In HO the directgradient is often identically 0, necessitating an approx-imation of the indirect gradient to make any progress(visualized in Fig. 1).
3. We can estimate the implicit best-responsewith the IFT: We approximate the best-responseJacobian—how the optimal weights change with respectto the hyperparameters—using the IFT (Thm. 1). Wepresent the complete statement in Appendix C, buthighlight the key assumptions and results here.Theorem 1 (Cauchy, Implicit Function Theorem). Iffor some (λ′,w′), ∂LT
∂w |λ′,w′ = 0 and regularity condi-tions are satisfied, then surrounding (λ′,w′) there is afunction w∗(λ) s.t. ∂LT
∂w |λ,w∗(λ) = 0 and we have:
∂w∗∂λ
∣∣∣λ′=−
[∂2LT
∂w∂wT︸ ︷︷ ︸training Hessian
]−1× ∂2LT
∂w∂λT︸ ︷︷ ︸training mixed partials
∣∣∣λ′,w∗(λ′)
(IFT)
The condition ∂LT∂w |λ′,w′ = 0 is equivalent to λ′,w′ be-
ing a fixed point of the training gradient field. Sincew∗(λ′) is a fixed point of the training gradient field,we can leverage the IFT to evaluate the best-responseJacobian locally. We only have access to an approxi-mation of the true best-response—denoted w∗—whichwe can find with gradient descent.
4. Tractable inverse Hessian approximations:To exactly invert a general m ×m Hessian, we oftenrequire O(m3) operations, which is intractable for thematrix in Eq. IFT in modern NNs. We can efficientlyapproximate the inverse with the Neumann series:[
∂2LT∂w∂wT
]−1= lim
i→∞
i∑j=0
[I − ∂2LT
∂w∂wT
]j(4)
In Section 4 we show that unrolling differentiation fori steps around locally optimal weights w∗ is equivalentto approximating the inverse with the first i terms inthe Neumann series. We then show how to use thisapproximation without instantiating any matrices byusing efficient vector-Jacobian products.
Jonathan Lorraine, Paul Vicol, David Duvenaud
∂L∗V∂λ︷︸︸︷
=
∂LV∂λ︷︸︸︷
+
∂LV∂w︷ ︸︸ ︷
∂w∗∂λ︷︸︸︷
=
∂LV∂λ︷︸︸︷
+
∂LV∂w︷ ︸︸ ︷
−[
∂2LT∂w∂wT
]−1︷ ︸︸ ︷︸ ︷︷ ︸
vector-inverse Hessian product
∂2LT∂w∂λT︷︸︸︷
=
∂LV∂λ︷︸︸︷
+
∂LV∂w ×−
[∂2LT
∂w∂wT]−1︷ ︸︸ ︷
∂2LT∂w∂λT︷︸︸︷
︸ ︷︷ ︸vector-Jacobian product
Figure 2: Hypergradient computation. The entirecomputation can be performed efficiently using vector-Jacobian products, provided a cheap approximation tothe inverse-Hessian-vector product is available.
Algorithm 1 Gradient-based HO for λ∗,w∗(λ∗)1: Initialize hyperparameters λ′ and weights w′2: while not converged do3: for k = 1 . . . N do4: w′ −= α · ∂LT∂w |λ′,w′5: λ′ −= hypergradient(LV,LT,λ′,w′)6: return λ′,w′ . λ∗,w∗(λ∗) from Eq.1
Algorithm 2 hypergradient(LV,LT,λ′,w′)1: v1 = ∂LV
∂w |λ′,w′2: v2 = approxInverseHVP(v1,
∂LT∂w )
3: v3 = grad(∂LT∂λ ,w, grad_outputs = v2)4: return ∂LV
∂λ |λ′,w′ − v3 . Return to Alg. 1
Algorithm 3 approxInverseHVP(v, f): Neumann ap-proximation of inverse-Hessian-vector product v[ ∂ f
∂w ]−1
1: Initialize sum p = v2: for j = 1 . . . i do3: v −= α · grad(f,w, grad_outputs = v)4: p −= v5: return p . Return to Alg. 2.
2.1 Proposed Algorithms
We outline our method in Algs. 1, 2, and 3, where αdenotes the learning rate. Alg. 3 is also shown in [22].We visualize the hypergradient computation in Fig. 2.
3 Related WorkImplicit Function Theorem. The IFT has beenused for optimization in nested optimization prob-lems [27, 28], backpropagating through arbitrarily longRNNs [22], or even efficient k-fold cross-validation [29].
Early work applied the IFT to regularization by ex-plicitly computing the Hessian (or Gauss-Newton) in-verse [23, 2]. In [24], the identity matrix is used to ap-proximate the inverse Hessian in the IFT. HOAG [30]uses conjugate gradient (CG) to invert the Hessianapproximately and provides convergence results giventolerances on the optimal parameter and inverse. IniMAML [9], a center to the weights is fit to performwell on multiple tasks—contrasted with our use of vali-dation loss. In DEQ [31], implicit differentiation is usedto add differentiable fixed-point methods into NN ar-chitectures. We use a Neumann approximation for theinverse-Hessian, instead of CG [30, 9] or the identity.
Approximate inversion algorithms. CG is diffi-cult to scale to modern, deep NNs. We use the Neu-mann inverse approximation, which was observed tobe a stable alternative to CG in NNs [22, 7]. Thestability is motivated by connections between the Neu-mann approximation and unrolled differentiation [7].Alternatively, we could use prior knowledge about theNN structure to aid in the inversion—e.g., by usingKFAC [32]. It is possible to approximate the Hessianwith the Gauss-Newton matrix or Fisher Informationmatrix [23]. Various works use an identity approxi-mation to the inverse, which is equivalent to 1-stepunrolled differentiation [24, 14, 33, 10, 8, 34, 35].
Unrolled differentiation for HO. A key difficultyin nested optimization is approximating how the opti-mized inner parameters (i.e., NN weights) change withrespect to the outer parameters (i.e., hyperparameters).We often optimize the inner parameters with gradientdescent, so we can simply differentiate through thisoptimization. Differentiation through optimization hasbeen applied to nested optimization problems by [3],was scaled to HO for NNs by [4], and has been appliedto various applications like learning optimizers [36]. [6]provides convergence results for this class of algorithms,while [5] discusses forward- and reverse-mode variants.
As the number of gradient steps we backpropagatethrough increases, so does the memory and computa-tional cost. Often, gradient descent does not exactlyminimize our objective after a finite number of steps—it only approaches a local minimum. Thus, to see howthe hyperparameters affect the local minima, we mayhave to unroll the optimization infeasibly far. Unrollinga small number of steps can be crucial for performancebut may induce bias [37]. [7] discusses connectionsbetween unrolling and the IFT, and proposes to un-roll only the last L-steps. DrMAD [38] proposes aninterpolation scheme to save memory.
We compare hypergradient approximations in Table 1,and memory costs of gradient-based HO methods inTable 2. We survey gradient-free HO in Appendix B.
Jonathan Lorraine, Paul Vicol, David Duvenaud
Method Steps Eval. Hypergradient Approximation
Exact IFT ∞ w∗(λ) ∂LV∂λ− ∂LV
∂w ×[
∂2LT∂w∂wT
]−1∂2LT∂w∂λT
∣∣∣∣w∗(λ)
Unrolled Diff. [4] i w0∂LV∂λ− ∂LV
∂w ×∑j≤i
[∏k<j I −
∂2LT∂w∂wT
∣∣∣wi−k
]∂2LT∂w∂λT
∣∣∣∣wi−j
L-Step Truncated Unrolled Diff. [7] i wL∂LV∂λ− ∂LV
∂w ×∑L≤j≤i
[∏k<j I −
∂2LT∂w∂wT
∣∣∣wi−k
]∂2LT∂w∂λT
∣∣∣∣wi−j
Larsen et al. [23] ∞ w∗(λ) ∂LV∂λ− ∂LV
∂w ×[∂LT∂w
∂LT∂w
T]−1
∂2LT∂w∂λT
∣∣∣∣w∗(λ)
Bengio [2] ∞ w∗(λ) ∂LV∂λ− ∂LV
∂w ×[
∂2LT∂w∂wT
]−1∂2LT∂w∂λT
∣∣∣∣w∗(λ)
T1− T2 [24] 1 w∗(λ) ∂LV∂λ− ∂LV
∂w × [I]−1 ∂2LT∂w∂λT
∣∣∣w∗(λ)
Ours i w∗(λ) ∂LV∂λ− ∂LV
∂w ×(∑
j<i
[I − ∂2LT
∂w∂wT
]j)∂2LT∂w∂λT
∣∣∣∣∣w∗(λ)
Conjugate Gradient (CG) ≈ - w∗(λ) ∂LV∂λ−
(argminx ‖x
∂2LT∂w∂wT −
∂LV∂w ‖
)∂2LT∂w∂λT
∣∣∣∣w∗(λ)
Hypernetwork [25, 26] - - ∂LV∂λ
+ ∂LV∂w ×
∂w∗φ∂λ
where w∗φ(λ) = argminφ LT(λ,wφ(λ))
Bayesian Optimization [19, 20, 7] ≈ - - ∂E[L∗V]
∂λwhere L∗V ∼ Gaussian-Process({λi,LV(λi,w∗(λi))})
Table 1: An overview of methods to approximate hypergradients. Some methods can be viewed as using anapproximate inverse in the IFT, or as differentiating through optimization around an evaluation point. Here,w∗(λ) is an approximation of the best-response at a fixed λ, which is often found with gradient descent.
Method Memory CostDiff. through Opt. [3, 4, 7] O(PI +H)
Linear Hypernet [25] O(PH)Self-Tuning Nets (STN) [26] O((P +H)K)
Neumann/CG IFT O(P +H)
Table 2: Gradient-based methods for HO. Differentia-tion through optimization scales with the number ofunrolled iterations I; the STN scales with bottlenecksize K, while our method only scales with the weightand hyperparameter sizes P and H.
4 Method
In this section, we discuss how HO is a uniquely chal-lenging nested optimization problem and how to com-bine the benefits of the IFT and unrolled differentiation.
4.1 Hyperparameter Opt. is Pure-Response
Eq. 3 shows that the hypergradient decomposes intoa direct and indirect gradient. The bottleneck in hy-pergradient computation is usually finding the indirectgradient because we must take into account how theoptimized parameters vary with respect to the hyperpa-rameters. A simple optimization approach is to neglectthe indirect gradient and only use the direct gradient.This can be useful in zero-sum games like GANs [39]because they always have a non-zero direct term.
However, using only the direct gradient does not workin general games [40]. In particular, it does not work forHO because the direct gradient is identically 0 when thehyperparameters λ can only influence the validation
loss by changing the optimized weights w∗(λ). Forexample, if we use regularization like weight decay whencomputing the training loss, but not the validation loss,then the direct gradient is always 0.
If the direct gradient is identically 0, we call the gamepure-response. Pure-response games are uniquely diffi-cult nested optimization problems for gradient-basedoptimization because we cannot use simple algorithmsthat rely on the direct gradient like simultaneous SGD.Thus, we must approximate the indirect gradient.
4.2 Unrolled Optimization and the IFT
Here, we discuss the relationship between the IFTand differentiation through optimization. Specifically,we (1) introduce the recurrence relation that ariseswhen we unroll SGD optimization, (2) give a formulafor the derivative of the recurrence, and (3) establishconditions for the recurrence to converge. Notably, weshow that the fixed points of the recurrence recoverthe IFT solution. We use these results to motivate acomputationally tractable approximation scheme tothe IFT solution. We give proofs of all results inAppendix D.
Unrolling SGD optimization—given an initializationw0—gives us the recurrence:
wi+1(λ)=T(λ,wi)=wi(λ)−α∂LT(λ,wi(λ))
∂w(5)
In our exposition, assume that α = 1. We provide aformula for the derivative of the recurrence, to showthat it converges to the IFT under some conditions.
Jonathan Lorraine, Paul Vicol, David Duvenaud
Lemma. Given the recurrence from unrolling SGDoptimization in Eq. 5, we have:
∂wi+1
∂λ=−
∑j≤i
∏k<j
I− ∂2LT∂w∂wT
∣∣∣λ,wi−k(λ)
∂2LT∂w∂λT
∣∣∣λ,wi−j(λ)
(6)
This recurrence converges to a fixed point if the transi-tion Jacobian ∂T
∂w is contractive, by the Banach Fixed-Point Theorem [41]. Theorem 2 shows that the recur-rence converges to the IFT if we start at locally optimalweights w0=w∗(λ), and the transition Jacobian ∂T
∂w iscontractive. We leverage that if an operator U is con-tractive, then the Neumann series
∑∞i=0 U
k=(Id−U)−1.
Theorem 2 (Neumann-SGD). Given the recurrencefrom unrolling SGD optimization in Eq. 5, if w0 =w∗(λ):
∂wi+1
∂λ= −
∑j<i
[I − ∂2LT
∂w∂wT
]j ∂2LT∂w∂λT
∣∣∣∣∣∣w∗(λ)
(7)
and if I + ∂2LT∂w∂wT is contractive:
limi→∞
∂wi+1
∂λ= −
[∂2LT
∂w∂wT
]−1∂2LT
∂w∂λT
∣∣∣∣w∗(λ)
(8)
This result is also shown in [7], but they use a differentapproximation for computing the hypergradient—seeTable 1. Instead, we use the following best-responseJacobian approximation, where i controls the trade-offbetween computation and error bounds:
∂w∗∂λ ≈ −
∑j<i
[I − α ∂2LT
∂w∂wT
]j ∂2LT∂w∂λT
∣∣∣∣∣∣w∗(λ)
(9)
Shaban et al. [7] use an approximation that scalesmemory linearly in i, while ours is constant. We savememory because we reuse last w i times, while [7] needsthe last iw’s. Scaling the Hessian by the learning rate αis key for convergence. Our algorithm has the followingmain advantages relative to other approaches:
• It requires a constant amount of memory, unlikeother unrolled differentiation methods [4, 7].
• It is more stable than conjugate gradient, likeunrolled differentiation methods [22, 7].
4.3 Scope and Limitations
The assumptions necessary to apply the IFT are asfollows: (1) LV : Λ×W → R is differentiable, (2)
LT : Λ×W→R is twice differentiable with an invertibleHessian at w∗(λ), and (3) w∗ : Λ→W is differentiable.
We need continuous hyperparameters to use gradient-based optimization, but many discrete hyperparame-ters (e.g., number of hidden units) have continuousrelaxations [42, 43]. Also, we can only optimize hy-perparameters that change the loss manifold, so ourapproach is not straightforwardly applicable to opti-mizer hyperparameters.
To exactly compute hypergradients, we must find(λ′,w′) s.t. ∂LT∂w |λ′,w′ =0, which we can only solve to atolerance with an approximate solution denoted w∗(λ).[30] shows results for error in w∗ and the inversion.
5 ExperimentsWe first compare the properties of Neumann inverseapproximations and conjugate gradient, with experi-ments similar to [22, 4, 7, 30]. Then we demonstratethat our proposed approach can overfit the validationdata with small training and validation sets. Finally,we apply our approach to high-dimensional HO tasks:(1) dataset distillation; (2) learning a data augmenta-tion network; and (3) tuning regularization parametersfor an LSTM language model.
HO algorithms that are not based on implicit differen-tiation or differentiation through optimization—suchas [44–48, 20]—do not scale to the high-dimensionalhyperparameters we use. Thus, we cannot sensiblycompare to them for high-dimensional problems.
5.1 Approximate Inversion Algorithms
In Fig. 3 we investigate how close various approxima-tions are to the true inverse. We calculate the distancebetween the approximate hypergradient and the truehypergradient. We can only do this for small-scale prob-lems because we need the exact inverse for the truehypergradient. Thus, we use a linear network on theBoston housing dataset [49], which makes finding thebest-response w∗ and inverse training Hessian feasible.
We measure the cosine similarity, which tells us how ac-curate the direction is, and the `2 (Euclidean) distancebetween the approximate and true hypergradients. TheNeumann approximation does better than CG in cosinesimilarity if we take enough HO steps, while CG alwaysdoes better for `2 distance.
In Fig. 4 we show the inverse Hessian for a fully-connected 1-layer NN on the Boston housing dataset.The true inverse Hessian has a dominant diagonal,motivating identity approximations, while using moreNeumann terms yields structure closer to the true in-verse.
Jonathan Lorraine, Paul Vicol, David Duvenaud
0.0
0.2
0.4
0.6
0.8
1.0
400 60010 3
10 1
101
103 20 CG Steps20 Neumann5 Neumann1 Neumann0 Neumann
CosineSimila
rity
` 2Distance
Optimization Iter.0 20 40
CGNeumann
# of Inversion Steps
Figure 3: Comparing approximate hypergradients forinverse Hessian approximations to true hypergradients.The Neumann scheme often has greater cosine similaritythan CG, but larger `2 distance for equal steps.
1.0
0.5
0.0
0.5
1.0
1 Neumann1.0
0.5
0.0
0.5
1.0
5 Neumann1.0
0.5
0.0
0.5
1.0
True Inverse1.0
0.5
0.0
0.5
1.0
Figure 4: Inverse Hessian approximations preprocessedby applying tanh for a 1-layer, fully-connected NN onthe Boston housing dataset as in [50].
5.2 Overfitting a Small Validation Set
In Fig. 5, we check the capacity of our HO algorithmto overfit the validation dataset. We use the same re-stricted dataset as in [5, 6] of 50 training and validationexamples, which allows us to assess HO performanceeasily. We tune a separate weight decay hyperparam-eter for each NN parameter as in [33, 4]. We showthe performance with a linear classifier, AlexNet [51],and ResNet44 [52]. For AlexNet, this yields more than50 000 000 hyperparameters, so we can perfectly classifyour validation data by optimizing the hyperparameters.
Algorithm 1 achieves 100% accuracy on the trainingand validation sets with significantly lower accuracyon the test set (Appendix E, Fig. 10), showing that wehave a powerful HO algorithm. The same optimizer isused for weights and hyperparameters in all cases.
5.3 Dataset Distillation
Dataset distillation [4, 13] aims to learn a small, syn-thetic training dataset from scratch, that condenses theknowledge contained in the original full-sized trainingset. The goal is that a model trained on the synthetic
100 101 102 103 1040.0
0.2
0.4
0.6
0.8
1.0
LinearAlexNetResNet44
ValidationError
IterationFigure 5: Algorithm 1 can overfit a small validationset on CIFAR-10. It optimizes for loss and achieves100% validation accuracy for standard, large models.
data generalizes to the original validation and test sets.Distillation is an interesting benchmark for HO as itallows us to introduce tens of thousands of hyperparam-eters, and visually inspect what is learned: here, everypixel value in each synthetic training example is a hyper-parameter. We distill MNIST and CIFAR-10/100 [53],yielding 28×28×10 = 7840, 32×32×3×10 = 30 720, and32×32×3×100 = 300 720 hyperparameters, respectively.For these experiments, all labeled data are in our vali-dation set, while our distilled data are in the trainingset. We visualize the distilled images for each class inFig. 6, recovering recognizable digits for MNIST andreasonable color averages for CIFAR-10/100.
5.4 Learned Data Augmentation
Data augmentation is a simple way to introduce invari-ances to a model—such as scale or contrast invariance—that improve generalization [17, 18]. Taking advantageof the ability to optimize many hyperparameters, welearn data augmentation from scratch (Fig. 7).
Specifically, we learn a data augmentation networkx = fλ(x, ε) that takes a training example x and noiseε ∼ N (0, I), and outputs an augmented example x.The noise ε allows us to learn stochastic augmentations.We parameterize f as a U-net [54] with a residual con-nection from the input to the output, to make it easyto learn the identity mapping. The parameters of theU-net, λ, are hyperparameters tuned for the validationloss—thus, we have 6659 hyperparameters. We traineda ResNet18 [52] on CIFAR-10 with augmented exam-ples produced by the U-net (that is simultaneouslytrained on the validation set).
Results for the identity and Neumann inverse approxi-mations are shown in Table 3. We omit CG because itperformed no better than the identity. We found thatusing the data augmentation network improves vali-dation and test accuracy by 2-3%, and yields smallervariance between multiple random restarts. In [55], adifferent augmentation network architecture is learnedwith adversarial training.
Jonathan Lorraine, Paul Vicol, David Duvenaud
CIFAR-10 DistillationPlane Car Bird Cat Deer Dog Frog Horse Ship Truck
MNIST Distillation
CIFAR-100 DistillationApple Fish Baby Bear Beaver Bed Bee Beetle Bicycle Bottle
Bowl Boy Bridge Bus Butterfly Camel Can Castle Caterpillar Cattle
Figure 6: Distilled datasets for CIFAR-10, MNIST, and CIFAR-100. For CIFAR-100, we show the first 20classes—the rest are in Appendix Fig. 11. We learn one distilled image per class, so after training a logisticregression classifier on the distillation, it generalizes to the rest of the data.
Original Sample 1 Sample 2 Pixel Std.
Figure 7: Learned data augmentations. The originalimage is on the left, followed by two augmented samplesand the standard deviation of the pixel intensities fromthe augmentation distribution.
Inverse Approx. Validation Test
0 92.5 ±0.021 92.6 ±0.0173 Neumann 95.1 ±0.002 94.6 ±0.001
3 Unrolled Diff. 95.0 ±0.002 94.7 ±0.001I 94.6 ±0.002 94.1 ±0.002
Table 3: Accuracy of different inverse approximations.Using 0 means that no HO occurs, and the augmenta-tion is initially the identity. The Neumann approachperforms similarly to unrolled differentiation [4, 7] withequal steps and less memory. Using more terms doesbetter than the identity, and the identity performedbetter than CG (not shown), which was unstable.
5.5 RNN Hyperparameter Optimization
We also used our proposed algorithm to tune regulariza-tion hyperparameters for an LSTM [56] trained on thePenn TreeBank (PTB) corpus [57]. As in [58], we useda 2-layer LSTM with 650 hidden units per layer and650-dimensional word embeddings. Additional detailsare provided in Appendix E.4.
Overfitting Validation Data. We first verify thatour algorithm can overfit the validation set in a small-data setting with 10 training and 10 validation se-quences (Fig. 8). The LSTM architecture we use has13 280 400 weights, and we tune a separate weight decayhyperparameter per weight. We overfit the validationset, reaching nearly 0 validation loss.
0k 25k 50k 75k 100kIteration
012345678
Loss
TrainValid
Loss
Iteration
Figure 8: Alg. 1 can overfit a small validation set withan LSTM on PTB.
Jonathan Lorraine, Paul Vicol, David Duvenaud
Large-Scale HO. There are various forms of regular-ization used for training RNNs, including variationaldropout [59] on the input, hidden state, and output;embedding dropout that sets rows of the embeddingmatrix to 0, removing tokens from all sequences in amini-batch; DropConnect [60] on the hidden-to-hiddenweights; and activation and temporal activation reg-ularization. We tune these 7 hyperparameters simul-taneously. Additionally, we experiment with tuningseparate dropout/DropConnect rate for each activa-tion/weight, giving 1 691 951 total hyperparameters.To allow for gradient-based optimization of dropoutrates, we use concrete dropout [61].
Instead of using the small dropout initialization as in[26], we use a larger initialization of 0.5, which preventsearly learning rate decay for our method. The resultsfor our new initialization with no HO, our method tun-ing the same hyperparameters as [26] (“Ours”), and ourmethod tuning many more hyperparameters (“Ours,Many”) are shown in Table 4. We are able to tunehyperparameters more quickly and achieve better per-plexities than the alternatives.
Method Validation Test Time(s)
Grid Search 97.32 94.58 100kRandom Search 84.81 81.46 100kBayesian Opt. 72.13 69.29 100k
STN 70.30 67.68 25kNo HO 75.72 71.91 18.5kOurs 69.22 66.40 18.5k
Ours, Many 68.18 66.14 18.5k
Table 4: Comparing HO methods for LSTM trainingon PTB. We tune millions of hyperparameters fasterand with comparable memory to competitors tuning ahandful. Our method competitively optimizes the same7 hyperparameters as baselines from [26] (first fourrows). We show a performance boost by tuning millionsof hyperparameters, introduced with per-unit/weightdropout and DropConnect. “No HO” shows how thehyperparameter initialization affects training.
5.6 Effects of Many Hyperparameters
Given the ability to tune high-dimensional hyperpa-rameters and the potential risk of overfitting to thevalidation set, should we reconsider how our trainingand validation splits are structured? Do the sameheuristics apply as for low-dimensional hyperparame-ters (e.g., use ∼ 10% of the data for validation)?
In Fig. 9 we see how splitting our data into trainingand validation sets of different ratios affects test per-formance. We show the results of jointly optimizingthe NN weights and hyperparameters, as well as theresults of fixing the final optimized hyperparametersand re-training the NN weights from scratch, which isa common technique for boosting performance [62].
We evaluate a high-dimensional regime with a separateweight decay hyperparameter per NN parameter, anda low-dimensional regime with a single, global weightdecay. We observe that: (1) for few hyperparameters,the optimal combination of validation data and hy-perparameters has similar test performance with andwithout re-training, because the optimal amount ofvalidation data is small; and (2) for many hyperparam-eters, the optimal combination of validation data andhyperparameters is significantly affected by re-training,because the optimal amount of validation data needsto be large to fit our hyperparameters effectively.
For few hyperparameters, our results agree with thestandard practice of using 10% of the data for val-idation and the other 90% for training. For manyhyperparameters, our results show that we should uselarger validation partitions for HO. If we use a largevalidation partition to fit the hyperparameters, it iscritical to re-train our model with all of the data.
0 .1 .25 .5 .75 .9Proportion data in valid
.7
.8
Test
Acc
urac
y
MNIST with Logistic Regression
TestAccuracy
Without re-training
% Data in Validation0 .1 .25 .5 .75 .9
Proportion data in valid
.7
.8
Test
Acc
urac
y
MNIST with Logistic Regression with Re-Training
Global DecayDecay per Weight
With re-training
% Data in Validation
Figure 9: Test accuracy of logistic regression onMNIST, with different size validation splits. Solid linescorrespond to a single global weight decay (1 hyperpa-rameter), while dotted lines correspond to a separateweight decay per weight (many hyperparameters). Thebest validation proportion for test performance is dif-ferent after re-training for many hyperparameters, butsimilar for few hyperparameters.
6 Conclusion
We present a gradient-based hyperparameter optimiza-tion algorithm that scales to high-dimensional hyper-parameters for modern, deep NNs. We use the implicitfunction theorem to formulate the hypergradient asa matrix equation, whose bottleneck is inverting theHessian of the training loss with respect to the NNparameters. We scale the hypergradient computationto large NNs by approximately inverting the Hessian,leveraging a relationship with unrolled differentiation.
We believe algorithms of this nature provide a pathfor practical nested optimization, where we have Hes-sians with known structure. Examples of this includeGANs [39], and other multi-agent games [63, 64].
Jonathan Lorraine, Paul Vicol, David Duvenaud
Acknowledgements
We thank Chris Pal for recommending we investigatere-training with all the data, Haoping Xu for discussingrelated experiments on inverse-approximation variants,Roger Grosse for guidance, and Cem Anil & ChrisCremer for their feedback on the paper. Paul Vicolwas supported by a JP Morgan AI Fellowship. We alsothank everyone else at Vector for helpful discussionsand feedback.
References[1] Jürgen Schmidhuber. Evolutionary principles in
self-referential learning, or on learning how tolearn: The meta-meta-... hook. PhD thesis, Tech-nische Universität München, 1987.
[2] Yoshua Bengio. Gradient-based optimization of hy-perparameters. Neural Computation, 12(8):1889–1900, 2000.
[3] Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence andStatistics, pages 318–326, 2012.
[4] Dougal Maclaurin, David Duvenaud, and RyanAdams. Gradient-based hyperparameter optimiza-tion through reversible learning. In InternationalConference on Machine Learning, pages 2113–2122,2015.
[5] Luca Franceschi, Michele Donini, Paolo Frasconi,and Massimiliano Pontil. Forward and reversegradient-based hyperparameter optimization. InInternational Conference on Machine Learning,pages 1165–1173, 2017.
[6] Luca Franceschi, Paolo Frasconi, Saverio Salzo,Riccardo Grazzi, and Massimiliano Pontil. Bilevelprogramming for hyperparameter optimizationand meta-learning. In International Conferenceon Machine Learning, pages 1563–1572, 2018.
[7] Amirreza Shaban, Ching-An Cheng, NathanHatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In Inter-national Conference on Artificial Intelligence andStatistics, pages 1723–1732, 2019.
[8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.Model-agnostic meta-learning for fast adaptationof deep networks. In International Conference onMachine Learning, pages 1126–1135, 2017.
[9] Aravind Rajeswaran, Chelsea Finn, Sham Kakade,and Sergey Levine. Meta-learning with implicitgradients. arXiv preprint arXiv:1909.04630, 2019.
[10] Hanxiao Liu, Karen Simonyan, and Yiming Yang.Darts: Differentiable architecture search. arXivpreprint arXiv:1806.09055, 2018.
[11] Edward Grefenstette, Brandon Amos, DenisYarats, Phu Mon Htut, Artem Molchanov,Franziska Meier, Douwe Kiela, Kyunghyun Cho,and Soumith Chintala. Generalized inner loopmeta-learning. arXiv preprint arXiv:1910.01727,2019.
[12] Jan Kukačka, Vladimir Golkov, and Daniel Cre-mers. Regularization for deep learning: A taxon-omy. arXiv preprint arXiv:1710.10686, 2017.
[13] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba,and Alexei A Efros. Dataset distillation. arXivpreprint arXiv:1811.10959, 2018.
[14] Mengye Ren, Wenyuan Zeng, Bin Yang, andRaquel Urtasun. Learning to reweight examplesfor robust deep learning. In International Con-ference on Machine Learning, pages 4331–4340,2018.
[15] Tae-Hoon Kim and Jonghyun Choi. ScreenerNet:Learning self-paced curriculum for deep neuralnetworks. arXiv preprint arXiv:1801.00904, 2018.
[16] Jiong Zhang, Hsiang-fu Yu, and Inderjit S Dhillon.AutoAssist: A framework to accelerate train-ing of deep neural networks. arXiv preprintarXiv:1905.03381, 2019.
[17] Ekin D Cubuk, Barret Zoph, Dandelion Mane,Vijay Vasudevan, and Quoc V Le. Autoaugment:Learning augmentation policies from data. arXivpreprint arXiv:1805.09501, 2018.
[18] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-ThangLuong, and Quoc V Le. Unsupervised data aug-mentation. arXiv preprint arXiv:1904.12848, 2019.
[19] Jonas Močkus. On Bayesian methods for seekingthe extremum. In Optimization Techniques IFIPTechnical Conference, pages 400–404. Springer,1975.
[20] Jasper Snoek, Hugo Larochelle, and Ryan PAdams. Practical Bayesian optimization of ma-chine learning algorithms. In Advances in NeuralInformation Processing Systems, pages 2951–2959,2012.
[21] Kirthevasan Kandasamy, Karun Raju Vysyaraju,Willie Neiswanger, Biswajit Paria, Christopher RCollins, Jeff Schneider, Barnabas Poczos, andEric P Xing. Tuning hyperparameters withoutgrad students: Scalable and robust Bayesianoptimisation with Dragonfly. arXiv preprintarXiv:1903.06694, 2019.
[22] Renjie Liao, Yuwen Xiong, Ethan Fetaya, LisaZhang, KiJung Yoon, Xaq Pitkow, Raquel Urta-sun, and Richard Zemel. Reviving and improvingrecurrent back-propagation. In International Con-ference on Machine Learning, pages 3088–3097,2018.
Jonathan Lorraine, Paul Vicol, David Duvenaud
[23] Jan Larsen, Lars Kai Hansen, Claus Svarer, andM Ohlsson. Design and regularization of neuralnetworks: The optimal use of a validation set. InNeural Networks for Signal Processing VI. Proceed-ings of the 1996 IEEE Signal Processing SocietyWorkshop, pages 62–71, 1996.
[24] Jelena Luketina, Mathias Berglund, Klaus Greff,and Tapani Raiko. Scalable gradient-based tuningof continuous regularization hyperparameters. InInternational Conference on Machine Learning,pages 2952–2960, 2016.
[25] Jonathan Lorraine and David Duvenaud. Stochas-tic hyperparameter optimization through hyper-networks. arXiv preprint arXiv:1802.09419, 2018.
[26] Matthew MacKay, Paul Vicol, Jon Lorraine, DavidDuvenaud, and Roger Grosse. Self-tuning net-works: Bilevel optimization of hyperparametersusing structured best-response functions. In Inter-national Conference on Learning Representations,2019.
[27] Peter Ochs, René Ranftl, Thomas Brox, andThomas Pock. Bilevel optimization with nons-mooth lower level problems. In International Con-ference on Scale Space and Variational Methodsin Computer Vision, pages 654–665, 2015.
[28] Anonymous. On solving minimax optimizationlocally: A follow-the-ridge approach. Interna-tional Conference on Learning Representations,2019. URL https://openreview.net/pdf?id=Hkx7_1rKwS.
[29] Ahmad Beirami, Meisam Razaviyayn, ShahinShahrampour, and Vahid Tarokh. On optimalgeneralizability in parametric learning. In Ad-vances in Neural Information Processing Systems,pages 3455–3465, 2017.
[30] Fabian Pedregosa. Hyperparameter optimizationwith approximate gradient. In International Con-ference on Machine Learning, pages 737–746, 2016.
[31] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.Deep equilibrium models. arXiv preprintarXiv:1909.01377, 2019.
[32] James Martens and Roger Grosse. Optimizingneural networks with Kronecker-factored approxi-mate curvature. In International Conference onMachine Learning, pages 2408–2417, 2015.
[33] Yogesh Balaji, Swami Sankaranarayanan, andRama Chellappa. Metareg: Towards domain gen-eralization using meta-regularization. In Advancesin Neural Information Processing Systems, pages998–1008, 2018.
[34] Ira Shavitt and Eran Segal. Regularization learn-ing networks: Deep learning for tabular datasets.
In Advances in Neural Information Processing Sys-tems, pages 1379–1389, 2018.
[35] Alex Nichol, Joshua Achiam, and John Schulman.On first-order meta-learning algorithms. arXivpreprint arXiv:1803.02999, 2018.
[36] Marcin Andrychowicz, Misha Denil, Sergio Gomez,Matthew W Hoffman, David Pfau, Tom Schaul,Brendan Shillingford, and Nando De Freitas.Learning to learn by gradient descent by gradi-ent descent. In Advances in Neural InformationProcessing Systems, pages 3981–3989, 2016.
[37] Yuhuai Wu, Mengye Ren, Renjie Liao, and RogerGrosse. Understanding short-horizon bias instochastic meta-optimization. arXiv preprintarXiv:1803.02021, 2018.
[38] Jie Fu, Hongyin Luo, Jiashi Feng, Kian HsiangLow, and Tat-Seng Chua. DrMAD: Distillingreverse-mode automatic differentiation for opti-mizing hyperparameters of deep neural networks.In International Joint Conference on ArtificialIntelligence, pages 1469–1475, 2016.
[39] Ian Goodfellow, Jean Pouget-Abadie, MehdiMirza, Bing Xu, David Warde-Farley, Sherjil Ozair,Aaron Courville, and Yoshua Bengio. Generativeadversarial nets. In Advances in Neural Informa-tion Processing Systems, pages 2672–2680, 2014.
[40] David Balduzzi, Sebastien Racaniere, JamesMartens, Jakob Foerster, Karl Tuyls, and ThoreGraepel. The mechanics of n-player differentiablegames. In International Conference on MachineLearning, pages 363–372, 2018.
[41] Stefan Banach. Sur les opérations dans les ensem-bles abstraits et leur application aux équationsintégrales. Fundamenta Mathematicae, 3:133–181,1922.
[42] C Maddison, A Mnih, and Y Teh. The concretedistribution: A continuous relaxation of discreterandom variables. In International Conference onLearning Representations, 2017.
[43] Eric Jang, Shixiang Gu, and Ben Poole. Cate-gorical reparameterization with Gumbel-softmax.arXiv preprint arXiv:1611.01144, 2016.
[44] Max Jaderberg, Valentin Dalibard, Simon Osin-dero, Wojciech M Czarnecki, Jeff Donahue, AliRazavi, Oriol Vinyals, Tim Green, Iain Dun-ning, Karen Simonyan, et al. Population basedtraining of neural networks. arXiv preprintarXiv:1711.09846, 2017.
[45] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperpa-rameter optimization. In Artificial Intelligenceand Statistics, pages 240–248, 2016.
Jonathan Lorraine, Paul Vicol, David Duvenaud
[46] James Bergstra and Yoshua Bengio. Randomsearch for hyper-parameter optimization. Journalof Machine Learning Research, 13:281–305, 2012.
[47] Manoj Kumar, George E Dahl, Vijay Vasudevan,and Mohammad Norouzi. Parallel architecture andhyperparameter search via successive halving andclassification. arXiv preprint arXiv:1805.10255,2018.
[48] Lisha Li, Kevin Jamieson, Giulia DeSalvo, AfshinRostamizadeh, and Ameet Talwalkar. Hyperband:a novel bandit-based approach to hyperparameteroptimization. The Journal of Machine LearningResearch, 18(1):6765–6816, 2017.
[49] David Harrison Jr and Daniel L Rubinfeld. He-donic housing prices and the demand for cleanair. Journal of Environmental Economics andManagement, 5(1):81–102, 1978.
[50] Guodong Zhang, Shengyang Sun, David Duve-naud, and Roger Grosse. Noisy natural gradientas variational inference. In International Con-ference on Machine Learning, pages 5847–5856,2018.
[51] Alex Krizhevsky, Ilya Sutskever, and Geoffrey EHinton. ImageNet classification with deep convo-lutional neural networks. In Advances in NeuralInformation Processing Systems, pages 1097–1105,2012.
[52] Kaiming He, Xiangyu Zhang, Shaoqing Ren, andJian Sun. Deep residual learning for image recog-nition. In Conference on Computer Vision andPattern Recognition, pages 770–778, 2016.
[53] Alex Krizhevsky. Learning multiple layers of fea-tures from tiny images. Technical report, 2009.
[54] Olaf Ronneberger, Philipp Fischer, and ThomasBrox. U-Net: Convolutional networks for biomed-ical image segmentation. In International Confer-ence on Medical image Computing and Computer-Assisted Intervention, pages 234–241, 2015.
[55] Saypraseuth Mounsaveng, David Vazquez, Is-mail Ben Ayed, and Marco Pedersoli. Adversariallearning of general transformations for data aug-mentation. International Conference on LearningRepresentations, 2019.
[56] Sepp Hochreiter and Jürgen Schmidhuber. Longshort-term memory. Neural Computation, 9(8):1735–1780, 1997.
[57] Mitchell P Marcus, Mary Ann Marcinkiewicz, andBeatrice Santorini. Building a large annotatedcorpus of English: The Penn Treebank. Computa-tional Linguistics, 19(2):313–330, 1993.
[58] Yarin Gal and Zoubin Ghahramani. A theoreti-cally grounded application of dropout in recurrent
neural networks. In Advances in Neural Informa-tion Processing Systems, pages 1027–1035, 2016.
[59] Durk P Kingma, Tim Salimans, and Max Welling.Variational dropout and the local reparameteri-zation trick. In Advances in Neural InformationProcessing Systems, pages 2575–2583, 2015.
[60] Li Wan, Matthew Zeiler, Sixin Zhang, YannLe Cun, and Rob Fergus. Regularization of neu-ral networks using Dropconnect. In InternationalConference on Machine Learning, pages 1058–1066,2013.
[61] Yarin Gal, Jiri Hron, and Alex Kendall. Con-crete dropout. In Advances in Neural InformationProcessing Systems, pages 3581–3590, 2017.
[62] Ian Goodfellow, Yoshua Bengio, and AaronCourville. Deep Learning. MIT Press, 2016.http://www.deeplearningbook.org.
[63] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, andIgor Mordatch. Learning with opponent-learningawareness. In International Conference on Au-tonomous Agents and MultiAgent Systems, pages122–130, 2018.
[64] Alistair Letcher, Jakob Foerster, David Balduzzi,Tim Rocktäschel, and Shimon Whiteson. Stableopponent shaping in differentiable games. arXivpreprint arXiv:1811.08469, 2018.
[65] Andrew Brock, Theodore Lim, James MillarRitchie, and Nicholas J Weston. SMASH: One-Shot Model Architecture Search through Hyper-Networks. In International Conference on Learn-ing Representations, 2018.
[66] Adam Paszke, Sam Gross, Soumith Chintala, Gre-gory Chanan, Edward Yang, Zachary DeVito, Zem-ing Lin, Alban Desmaison, Luca Antiga, and AdamLerer. Automatic differentiation in PyTorch. 2017.
[67] Diederik P Kingma and Jimmy Ba. Adam: Amethod for stochastic optimization. InternationalConference on Learning Representations, 2014.
[68] Geoffrey Hinton, Nitish Srivastava, and KevinSwersky. Neural networks for machine learning.Lecture 6a. Overview of mini-batch gradient de-scent. 2012.
[69] Yann LeCun, Léon Bottou, Yoshua Bengio, PatrickHaffner, et al. Gradient-based learning applied todocument recognition. Proceedings of the IEEE,86(11):2278–2324, 1998.
[70] Stephen Merity, Nitish Shirish Keskar, andRichard Socher. Regularizing and optimizingLSTM language models. International Confer-ence on Learning Representations, 2018.
Jonathan Lorraine, Paul Vicol, David Duvenaud
Optimizing Millions of Hyperparameters by Implicit DifferentiationAppendix
A Extended Background
In this section we provide an outline of our notation(Table 5), and the proposed algorithm. Here, we assumewe have access to to a finite dataset D = {(xi,yi)|i =1 . . . n}, with n examples drawn from the distributionp(x,y) with support P . We denote the input and targetdomains by X and Y , respectively. Assume y : X → Yis a function and we wish to learn y : X ×W → Ywith a NN parameterized by w ∈ W, s.t. y is closeto y. We measure how close a predicted value is to atarget with the prediction loss L : Y × Y → R. Ourgoal is to minimize the expected prediction loss orpopulation risk: argminw Ex∼p(x)[L(y(x,w),y(x))].Since we only have access to a finite num-ber of samples, we minimize the empirical risk:argminw 1/n
∑x,y∈D L(y(x,w),y(x)).
Due to a limited size dataset D, there may be a signifi-cant difference between the minimizer of the empiricalrisk and the population risk. We can estimate thisdifference by partitioning our dataset into training andvalidation datasets— Dtrain,Dvalid. We find the min-imizer over the training dataset Dtrain, and estimateits performance on the population risk by evaluatingthe empirical risk over the validation dataset Dvalid.We introduce modifications to the empirical trainingrisk to decrease our population risk, parameterized byλ ∈ Λ. These parameters for generalization are calledthe hyperparameters. We call the modified empiricaltraining risk our training loss for simplicity and denoteit LT(λ,w). Our validation empirical risk is calledvalidation loss for simplicity and denoted by LV(λ,w).Often the validation loss does not directly depend onthe hyperparameters, and we just have LV(w).
The population risk is estimated by plugging thetraining loss minimizer w∗(λ) = argminw LT(λ,w)into the validation loss for the estimated populationrisk L∗V(λ) = LV(λ,w∗(λ)). We want our hyperpa-rameters to minimize the estimated population risk:λ∗ = argminλ L∗V(λ). We can create a third partitionof our dataset Dtest to assess if we have overfit thevalidation dataset Dvalid with our hyperparameters λ.
B Extended Related Work
Independent HO: A simple class of HO algorithmsinvolve making a number of independent hyperparame-
ter selections, and training the model to completion onthem. Popular examples include grid search and ran-dom search [46]. Since each hyperparameter selection isindependent, these algorithms are trivial to parallelize.
Global HO: Some HO algorithms attempt to find aglobally optimal hyperparameter setting, which can beimportant if the loss is non-convex. A simple exampleis random search, while a more sophisticated exampleis Bayesian optimization [19–21]. These HO algorithmsoften involve re-initializing the hyperparameter andweights on each optimization iteration. This allowsglobal optimization, at the cost of expensive re-trainingweights or hyperparameters.
Local HO: Other HO algorithms only attempt to finda locally optimal hyperparameter setting. Often thesealgorithms will maintain a current estimate of the bestcombination of hyperparameter and weights. On eachoptimization iteration, the hyperparameter is adjustedby a small amount, which allows us to avoid excessivere-training of the weights on each update. This is be-cause the new optimal weights are near the old optimalweights due to a small change in the hyperparameters.
Learned proxy function based HO:Many HO algo-rithms attempt to learn a proxy function for optimiza-tion. The proxy function is used to estimate the lossfor a hyperparameter selection. We could learn a proxyfunction for global or local HO . We can learn a usefulproxy function over any node in our computationalgraph including the optimized weights. For example,we could learn how the optimized weights change w.r.t.the hyperparameters [25], how the optimized predic-tions change w.r.t. the hyperparameters [26], or howthe optimized validation loss changes w.r.t. the hyper-parameters as in Bayesian Optimization. It is possibleto do gradient descent on the proxy function to findnew hyperparameters to query as in Bayesian optimiza-tion. Alternatively, we could use a non-differentiableproxy function to get cheap estimates of the validationloss like SMASH [65] for architecture choices.
Jonathan Lorraine, Paul Vicol, David Duvenaud
Table 5: Notation
HO Hyperparameter optimizationNN Neural networkIFT Implicit Function Theorem
HVP / JVP Hessian/Jacobian-vector productλ,w Hyperparameters and NN parameters/weightsn,m Hyperparameter and NN parameter dimensionality
Λ⊆Rn,W⊆Rm Hyperparameters and NN parameter domainsλ′,w′ Arbitrary, fixed hyperparameters and weights
LT(λ,w),LV(λ,w) Training loss & validation lossw∗(λ) Best-response of the weights to the hyperparametersw∗(λ) An approximate best-response of the weights to the hyperparameters
L∗V(λ)= LV(λ,w∗(λ)) The validation loss with best-responding weightsRed (Approximations to) The validation loss with best-responding weights
W∗ = w∗(Λ) The domain of best-responding weightsλ∗ The optimal hyperparametersx,y An input and its associated targetX ,Y The input and target domains respectivelyD A data matrix consisting of tuples of inputs and targets
y(x,w) A predicted target for a input data and weights∂LV∂λ ,
∂LV∂w The (validation loss hyperparameter / parameter) direct gradient
Green (Approximations to) The validation loss direct gradient.∂w∗∂λ The best-response JacobianBlue (Approximations to) The (Jacobian of the) best-response of the weights
to the hyperparameters∂LV∂w
∂w∗∂λ The indirect gradient
∂L∗V∂λ A hypergradient: sum of validation losses direct and indirect gradient[
∂2LT∂w∂wT
]−1The training Hessian inverse
Magenta (Approximations to) The training Hessian inverse∂LV∂w
[∂2LT
∂w∂wT
]−1The vector - Inverse Hessian product.
Orange (Approximations to) The vector - Inverse Hessian product.∂2LT
∂w∂λTThe training mixed partial derivatives
I The identity matrix
Jonathan Lorraine, Paul Vicol, David Duvenaud
C Implicit Function Theorem
Theorem (Augustin-Louis Cauchy, Implicit Function Theorem). Let ∂LT∂w (λ,w) : Λ×W→W be a continuously
differentiable function. Fix a point (λ′,w′) with ∂LT∂w (λ′,w′) = 0. If the Jacobian J
∂LT∂w
w (λ′,w′) is invertible, thereexists an open set U ⊆ Λ containing λ′ s.t. there exists a continuously differentiable function w∗ : U →W s.t.:
w∗(λ′) = w′ and ∀λ ∈ U, ∂LT∂w
(λ,w∗(λ))) = 0
Moreover, the partial derivatives of w∗ in U are given by the matrix product:
∂w∗
∂λ(λ) = −
[J
∂LT∂w
w (λ,w∗(λ))
]−1J
∂LT∂wλ (λ,w∗(λ)))
Typically the IFT is presented with ∂LT∂w = f , w∗ = g, Λ = Rm, W = Rn, λ = x, w = y, λ′ = a, w′ = b.
Jonathan Lorraine, Paul Vicol, David Duvenaud
D Proofs
Lemma (1). If the recurrence given by unrolling SGD optimization in Eq. 5 has a fixed point w∞ (i.e.,0 = ∂LT
∂w |λ,w∞(λ)), then:∂w∞∂λ
= −[
∂2LT∂w∂wT
]−1∂2LT
∂w∂λT
∣∣∣∣w∞(λ)
Proof.
⇒ ∂∂λ
(∂LT∂w
∣∣∣λ,w∞(λ)
)= 0 given
⇒(
∂2LT∂w∂λT
I + ∂2LT∂w∂wT
∂w∞∂λ
)∣∣∣∣λ,w∞(λ)
= 0 chain rule through |λ,w∞(λ)
⇒ ∂2LT∂w∂wT
∂w∞∂λ
∣∣∣λ,w∞(λ)
= − ∂2LT∂w∂λT
∣∣∣λ,w∞(λ)
re-arrange terms
⇒ ∂w∞∂λ
∣∣∣λ= −
[∂2LT
∂w∂wT
]−1∂2LT
∂w∂λT
∣∣∣∣λ
left-multiply by[
∂2LT∂w∂wT
]−1∣∣∣∣λ,w∞(λ)
Lemma (2). Given the recurrence from unrolling SGD optimization in Eq. 5 we have:
∂wi+1
∂λ= −
∑j≤i
∏k<j
I − ∂2LT∂w∂wT
∣∣∣λ,wi−k(λ)
∂2LT∂w∂λT
∣∣∣λ,wi−j(λ)
Proof.
∂wi+1
∂λ
∣∣∣λ= ∂
∂λ
(wi(λ)− ∂LT
∂w
∣∣∣λ,wi(λ)
)take derivative w.r.t. λ
= ∂wi∂λ
∣∣∣λ− ∂
∂λ
(∂LT∂w
∣∣∣λ,wi(λ)
)chain rule
= ∂wi∂λ
∣∣∣λ−(
∂2LT∂w∂wT
∂wi∂λ + ∂2LT
∂w∂λT
)∣∣∣∣λ,wi(λ)
chain rule through |λ,wi(λ)
= − ∂2LT∂w∂λT
∣∣∣λ,wi(λ)
+(I − ∂2LT
∂w∂wT
)∂wi∂λ
∣∣∣∣λ,wi(λ)
re-arrange terms
= − ∂2LT∂w∂λT
∣∣∣λ,wi(λ)
+(I − ∂2LT
∂w∂wT
)∣∣∣∣λ,wi(λ)
·
((I − ∂2LT
∂w∂wT
)∂wi−1
∂λ − ∂2LT∂w∂λT
)∣∣∣∣∣λ,wi−1(λ)
expand ∂wi∂λ
= − ∂2LT∂w∂λT
∣∣∣λ,wi(λ)
−(I − ∂2LT
∂w∂wT
∣∣∣λ,wi(λ)
)∂2LT
∂w∂λT
∣∣∣λ,wi−1(λ)
+∏k<2
I − ∂2LT∂w∂wT
∣∣∣λ,wi−k(λ)
∂wi−1
∂λ
∣∣∣λ
re-arrange terms
= · · ·
So, ∂wi+1
∂λ = −∑j≤i
∏k<j
I − ∂2LT∂w∂wT
∣∣∣λ,wi−k(λ)
∂2LT∂w∂λT
∣∣∣λ,wi−j(λ)
telescope the recurrence
Jonathan Lorraine, Paul Vicol, David Duvenaud
Theorem (Neumann-SGD). Given the recurrence from unrolling SGD optimization in Eq. 5, if w0 = w∗(λ):
∂wi+1
∂λ= −
∑j≤i
[I − ∂2LT
∂w∂wT
]j ∂2LT∂w∂λT
∣∣∣∣∣∣w∗(λ)
and if I + ∂2LT∂w∂wT is contractive:
limi→∞
∂wi+1
∂λ= −
[∂2LT
∂w∂wT
]−1∂2LT
∂w∂λT
∣∣∣∣w∗(λ)
Proof.
limi→∞
∂wi+1
∂λ
∣∣∣λ
take limi→∞
= limi→∞
−∑j≤i
∏k<j
I − ∂2LT∂w∂wT
∣∣∣λ,wi−k(λ)
∂2LT∂w∂λT
∣∣∣λ,wi−j(λ)
by Lemma 2
= − limi→∞
∑j≤i
∏k<j
I − ∂2LT∂w∂wT
∂2LT∂w∂λT
∣∣∣∣∣∣∣λ,w∗(λ)
w0 = w∗(λ) = wi
= − limi→∞
∑j≤i
[I − ∂2LT
∂w∂wT
]j ∂2LT∂w∂λT
∣∣∣∣∣∣λ,w∗(λ)
simplify
= −[I −
(I − ∂2LT
∂w∂wT
)]−1∂2LT
∂w∂λT
∣∣∣∣∣λ,w∗(λ)
contractive & Neumann series
= −[
∂2LT∂w∂wT
]−1∂2LT
∂w∂λT
∣∣∣∣λ,w∗(λ)
simplify
Jonathan Lorraine, Paul Vicol, David Duvenaud
E Experiments
We use PyTorch [66] as our computational framework.All experiments were performed on NVIDIA TITANXp GPUs.
For all CNN experiments we use the following optimiza-tion setup: for the NN weights we use Adam [67] witha learning rate of 1e-4. For the hyperparameters weuse RMSprop [68] with a learning rate of 1e-2.
E.1 Overfitting a Small Validation Set
We see our algorithm’s ability to overfit the validationdata (see Fig. 10). We use 50 training input, and 50validation input with the standard testing partitionfor both MNIST and CIFAR-10. We check perfor-mance with logistic regression (Linear), a 1-Layer fully-connected NN with as many hidden units as input size(ex., 28× 28 = 784, or 32× 32× 3 = 3072), LeNet [69],AlexNet [51], and ResNet44 [52]. In all examples wecan achieve 100% training and validation accuracy,while the testing accuracy is significantly lower.
100 101 102 1030.0
0.2
0.4
0.6
0.8
1.0
Classification
Error MNIST
Iteration100 101 102 103 1040.0
0.2
0.4
0.6
0.8
1.0TrainingValidationTestLinear1-LayerLeNetAlexNetResNet44
CIFAR-10
Iteration
Figure 10: Overfitting validation data. Algorithm 1can overfit the validation dataset. We use 50 traininginput, and 50 validation input with the standard testingpartition for both MNIST and CIFAR-10. We checkthe performance with logistic regression (Linear), a1-Layer fully-connected NN with as many hidden unitsas input size (ex., 28×28 = 784, or 32×32×3 = 3072),LeNet [69], AlexNet [51], and ResNet44 [52]. Separatelines are plotted for the training, validation, and test-ing error. In all examples we achieve 100% trainingand validation accuracy, while the testing accuracy issignificantly lower.
E.2 Dataset Distillation
With MNIST we use the entire dataset in validation,while for CIFAR we use 300 validation data points.
E.3 Learned Data Augmentation
Augmentation Network Details: Data augmenta-tion can be framed as an image-to-image transformation
problem; inspired by this, we use a U-Net [54] as thedata augmentation network. To allow for stochastictransformations, we feed in random noise by concatenat-ing a noise channel to the input image, so the resultinginput has 4 channels.
E.4 RNN Hyperparameter Optimization
Our base our implementation on the AWD-LSTM codebase https://github.com/salesforce/awd-lstm-lm. Similar to [58] we used a 2-layer LSTMwith 650 hidden units per layer and 650-dimensionalword embeddings.
Overfitting Validation Data: We used a subsetof 10 training sequences and 10 validation sequences,and tuned separate weight decays per parameter. TheLSTM architecture we use has 13 280 400 weights, andthus an equal number of weight decay hyperparameters.
Optimization Details: For the large-scale experi-ments, we follow the training setup proposed in [70]:for the NN weights, we use SGD with learning rate 30and gradient clipping to magnitude 0.25. The learn-ing rate was decayed by a factor of 4 based on thenonmonotonic criterion introduced by [70] (i.e., whenthe validation loss fails to decrease for 5 epochs). Tooptimize the hyperparameters, we used Adam withlearning rate 0.001. We trained on sequences of length70 in mini-batches of size 40.
Jonathan Lorraine, Paul Vicol, David Duvenaud
CIFAR-100 Distillation
Figure 11: The complete dataset distillation for CIFAR-100. Referenced in Fig. 6.
