AdapterHub: A Framework for Adapting Transformers

Jonas Pfeiffer∗1, Andreas Ruckle∗1, Clifton Poth∗1,Aishwarya Kamath2, Ivan Vulic4, Sebastian Ruder5,

Kyunghyun Cho2,3, Iryna Gurevych1

1Technical University Darmstadt2New York University 3CIFAR Associate Fellow

4University of Cambridge5DeepMind

AdapterHub.ml

Abstract

The current modus operandi in NLP involvesdownloading and fine-tuning pre-trained mod-els consisting of millions or billions of pa-rameters. Storing and sharing such largetrained models is expensive, slow, and time-consuming, which impedes progress towardsmore general and versatile NLP methods thatlearn from and for many tasks. Adapters—small learnt bottleneck layers inserted withineach layer of a pre-trained model— amelioratethis issue by avoiding full fine-tuning of theentire model. However, sharing and integrat-ing adapter layers is not straightforward. Wepropose AdapterHub, a framework that allowsdynamic “stiching-in” of pre-trained adaptersfor different tasks and languages. The frame-work, built on top of the popular HuggingFaceTransformers library, enables extremely easyand quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Download-ing, sharing, and training adapters is as seam-less as possible using minimal changes to thetraining scripts and a specialized infrastructure.Our framework enables scalable and easy ac-cess to sharing of task-specific models, partic-ularly in low-resource scenarios. AdapterHubincludes all recent adapter architectures andcan be found at AdapterHub.ml.

1 Introduction

Recent advances in NLP leverage transformer-based language models (Vaswani et al., 2017), pre-trained on large amounts of text data (Devlin et al.,2019; Liu et al., 2019; Conneau et al., 2020). Thesemodels are fine-tuned on a target task and achievestate-of-the-art (SotA) performance for most nat-ural language understanding tasks. Their perfor-mance has been shown to scale with their size (Ka-plan et al., 2020) and recent models have reached

∗*Equal contribution.

billions of parameters (Raffel et al., 2019; Brownet al., 2020). While fine-tuning large pre-trainedmodels on target task data can be done fairly effi-ciently (Howard and Ruder, 2018), training themfor multiple tasks and sharing trained models isoften prohibitive. This precludes research on moremodular architectures (Shazeer et al., 2017), taskcomposition (Andreas et al., 2016), and injectingbiases and external information (e.g., world or lin-guistic knowledge) into large models (Lauscheret al., 2019; Wang et al., 2020).

Adapters (Houlsby et al., 2019) have been in-troduced as an alternative lightweight fine-tuningstrategy that achieves on-par performance to fullfine-tuning (Peters et al., 2019) on most tasks.They consist of a small set of additional newlyinitialized weights at every layer of the transformer.These weights are then trained during fine-tuning,while the pre-trained parameters of the large modelare kept frozen/fixed. This enables efficient pa-rameter sharing between tasks by training manytask-specific and language-specific adapters for thesame model, which can be exchanged and com-bined post-hoc. Adapters have recently achievedstrong results in multi-task and cross-lingual trans-fer learning (Pfeiffer et al., 2020a,b).

However, reusing and sharing adapters is notstraightforward. Adapters are rarely released in-dividually; their architectures differ in subtle yetimportant ways, and they are model, task, and lan-guage dependent. To mitigate these issues and fa-cilitate transfer learning with adapters in a range ofsettings, we propose AdapterHub, a framework thatenables seamless training and sharing of adapters.

AdapterHub is built on top of the populartransformers framework by HuggingFace1

(Wolf et al., 2019), which provides access to state-of-the-art pre-trained language models. We en-

1https://github.com/huggingface/transformers

arX

iv:2

007.

0777

9v1

[cs

.CL

] 1

5 Ju

l 202

0

hance transformers with adapter modules thatcan be combined with existing SotA models withminimal code edits. We additionally provide a web-site that enables quick and seamless upload, down-load, and sharing of pre-trained adapters. Adapter-Hub is available online at: AdapterHub.ml.

AdapterHub for the first time enables NLP re-searchers and practitioners to easily and efficientlyshare and obtain access to models that have beentrained for particular tasks, domains, and languages.This opens up the possibility of building on andcombining information from many more sourcesthan was previously possible and makes researchsuch as intermediate task training (Pruksachatkunet al., 2020), composing information from manytasks (Pfeiffer et al., 2020a), and training modelsfor very low-resource languages (Pfeiffer et al.,2020b) much more accessible.

Contributions. 1) We propose an easy-to-useand extensible adapter training and sharing frame-work for transformer-based models such as BERT,RoBERTa, and XLM(-R); 2) we incorporate it intothe HuggingFace transformers framework, re-quiring as little as two additional lines of code totrain adapters with existing scripts; 3) our frame-work automatically extracts the adapter weights,storing them separately to the pre-trained trans-former model, requiring as little as 1Mb of stor-age; 4) we provide an open-source framework andwebsite that allows the community to upload theiradapter weights, making them easily accessiblewith only one additional line of code; 5) we in-corporate adapter composition as well as adapterstacking out-of-the-box and pave the way for awide range of other extensions in the future.

2 Adapters

While the predominant methodology for transferlearning is to fine-tune all weights of the pre-trainedmodel, adapters have recently been introduced asan alternative approach, with applications in com-puter vision (Rebuffi et al., 2017) as well as theNLP domain (Houlsby et al., 2019; Bapna and Firat,2019; Wang et al., 2020; Pfeiffer et al., 2020a,b).

2.1 Adapter Architecture

Adapters are neural modules with a small amountof additional newly introduced parameters Φ withina large pre-trained model with parameters Θ. Theparameters Φ are learnt on a target task while keep-ing Θ fixed; Φ thus learn to encode task-specific

representations in intermediate layers of the pre-trained model. Current work predominantly fo-cuses on training adapters for each task separately(Houlsby et al., 2019; Bapna and Firat, 2019; Pfeif-fer et al., 2020a,b), which enables parallel trainingand subsequent combination of the weights.

In NLP, adapters have been mainly used withindeep transformer-based architectures (Vaswaniet al., 2017). At each transformer layer l, a set ofadapter parameters Φl is introduced. The place-ment and architecture of adapter parameters Φwithin a pre-trained model is non-trivial and mayimpact their efficacy: Houlsby et al. (2019) experi-ment with different adapter architectures, empiri-cally validating that a two-layer feed-forward neu-ral network with a bottleneck works well. Whilethis down- and up-projection has largely beenagreed upon, the actual placement of adapterswithin each transformer block, as well as the in-troduction of new LayerNorms2 (Ba et al., 2016)varies in the literature (Houlsby et al., 2019; Bapnaand Firat, 2019; Stickland and Murray, 2019; Pfeif-fer et al., 2020a). In order to support standardadapter architectures from the literature, as well asto enable easy extensibility, AdapterHub providesa configuration file where the architecture settingscan be defined dynamically. We illustrate the dif-ferent configuration possibilities in Figure 3, anddescribe them in more detail in §3.

2.2 Why Adapters?

Adapters provide numerous benefits over fully fine-tuning a model such as scalability, modularity, andcomposition. We now provide a few use-cases foradapters to illustrate their usefulness in practice.

Task-specific Layer-wise Representation Learn-ing. Prior to the introduction of adapters, in orderto achieve SotA performance on downstream tasks,the entire pre-trained transformer model needs tobe fine-tuned (Peters et al., 2019). Adapters havebeen shown to work on-par with full fine-tuning,by adapting the representations at every layer. Wepresent the results of fully fine-tuning the modelcompared to two different adapter architectureson the GLUE benchmark (Wang et al., 2018) inTable 1. The adapters of Houlsby et al. (2019,Figure 3c) and Pfeiffer et al. (2020a, Figure 3b)comprise two and one down- and up-projection

2Layer normalization learns to normalize the inputs acrossthe features. This is usually done by introducing a new set offeatures for mean and variance.

Full Pfeif. Houl.

RTE (Wang et al., 2018) 66.2 70.8 69.8MRPC (Dolan and Brockett, 2005) 90.5 89.7 91.5STS-B (Cer et al., 2017) 88.8 89.0 89.2CoLA (Warstadt et al., 2019) 59.5 58.9 59.1SST-2 (Socher et al., 2013) 92.6 92.2 92.8QNLI (Rajpurkar et al., 2016) 91.3 91.3 91.2MNLI (Williams et al., 2018) 84.1 84.1 84.1QQP (Iyer et al., 2017) 91.4 90.5 90.8

Table 1: Mean development scores over 3 runs onGLUE (Wang et al., 2018) leveraging the BERT-Basepre-trained weights. We present the results with fullfine-tuning (Full) and with the adapter architectures ofPfeiffer et al. (2020a, Pfeif., Figure 3b) and Houlsbyet al. (2019, Houl., Figure 3c) both with bottleneck size48. We show F1 for MRPC, Spearman rank correlationfor STS-B, and accuracy for the rest. RTE is a combi-nation of datasets (Dagan et al., 2005; Bar-Haim et al.,2006; Giampiccolo et al., 2007).

within each transformer layer, respectively. Theformer adapter thus has more capacity at the costof training and inference speed. We find that forall settings, there is no large difference in termsof performance between the model architectures,verifying that training adapters is a suitable andlightweight alternative to full fine-tuning in orderto achieve SotA performance on downstream tasks.

Small, Scalable, Shareable. Transformer-basedmodels are very deep neural networks with mil-lions or billions of weights and large storage re-quirements, e.g., around 2.2Gb of compressed stor-age space is needed for XLM-R Large (Conneauet al., 2020). Fully fine-tuning these models foreach task separately requires storing a copy of thefine-tuned model for each task. This impedes bothiterating and parallelizing training, particularly instorage-restricted environments.

Adapters mitigate this problem. Depending onthe model size and the adapter bottleneck size, asingle task requires as little as 0.9Mb storage space.We present the storage requirements in Table 2.This highlights that > 99% of the parameters re-quired for each target task are fixed during trainingand can be shared across all models for inference.For instance, for the popular Bert-Base model witha size of 440Mb, storing 2 fully fine-tuned modelsamounts to the same storage space required by 125models with adapters, when using a bottleneck sizeof 48 and adapters of Pfeiffer et al. (2020a). More-over, when performing inference on a mobile de-vice, adapters can be leveraged to save a significantamount of storage space, while supporting a large

Base LargeCRate #Params Size #Params Size

64 0.2M 0.9Mb 0.8M 3.2Mb16 0.9M 3.5Mb 3.1M 13Mb2 7.1M 28Mb 25.2M 97Mb

Table 2: Number of additional parameters and com-pressed storage space of the adapter of Pfeiffer et al.(2020a) in (Ro)BERT(a)-Base and Large transformerarchitectures. The adapter of Houlsby et al. (2019) re-quires roughly twice as much space. CRate refers to theadapter’s compression rate: e.g., a. rate of 64 meansthat the adapter’s bottleneck layer is 64 times smallerthan the underlying model’s hidden layer size.

number of target tasks. Additionally, due to thesmall size of the adapter modules—which in manycases do not exceed the file size of an image—newtasks can be added on-the-fly. Overall, these factorsmake adapters a much more computationally—andecologically (Strubell et al., 2019)—viable optioncompared to updating entire models. Easy access tofine-tuned models may also improve reproducibil-ity as researchers will be able to easily rerun andevaluate trained models of previous work.

Modularity of Representations. Adapters learnto encode information of a task within designatedparameters. Due to the encapsulated placement ofadapters, wherein the surrounding parameters arefixed, at each layer an adapter is forced to learnan output representation compatible with the sub-sequent layer of the transformer model. This set-ting allows for modularity of components such thatadapters can be stacked on top of each other, orreplaced dynamically. In a recent example, Pfeifferet al. (2020b) successfully combine adapters thathave been independently trained for specific tasksand languages. This demonstrates that adapters aremodular and that output representations of differ-ent adapters are compatible. As NLP tasks becomemore complex and require knowledge that is not di-rectly accessible in a single monolithic pre-trainedmodel (Ruder et al., 2019), adapters will provideNLP researchers and practitioners with many moresources of relevant information that can be easilycombined in an efficient and modular way.

Non-Interfering Composition of Information.Sharing information across tasks has a long-standing history in machine learning (Ruder, 2017).Multi-task learning (MTL), which shares a set ofparameters between tasks, has arguably receivedthe most attention. However, MTL suffers from

problems such as catastrophic forgetting where in-formation learned during earlier stages of trainingis “overwritten” (Masson et al., 2019), catastrophicinterference where the performance of a set of tasksdeteriorates when adding new tasks (Hashimotoet al., 2017), and intricate task weighting for taskswith different distributions (Sanh et al., 2019).

The encapsulation of adapters forces them tolearn output representations that are compatibleacross tasks. When training adapters on differentdownstream tasks, they store the respective infor-mation in their designated parameters. Multipleadapters can then be combined, e.g., with atten-tion (Pfeiffer et al., 2020a). Because the respectiveadapters are trained separately, the necessity ofsampling heuristics due to skewed data set sizesno longer arises. By separating knowledge extrac-tion and composition, adapters mitigate the twomost common pitfalls of multi-task learning, catas-trophic forgetting and catastrophic interference.

Overcoming these problems together with theavailability of readily available trained task-specificadapters enables researchers and practitioners toleverage information from specific tasks, domains,or languages that is often more relevant for a spe-cific application—rather than more general pre-trained counterparts. Recent work (Howard andRuder, 2018; Phang et al., 2018; Pruksachatkunet al., 2020; Gururangan et al., 2020) has shown thebenefits of such information, which was previouslyonly available by fully fine-tuning a model on thedata of interest prior to task-specific fine-tuning.

3 AdapterHub

AdapterHub consists of two core components:1) A library built on top of HuggingFacetransformers, and 2) a website that dynam-ically provides analysis and filtering of pre-trainedadapters. AdapterHub provides tools for the entirelife-cycle of adapters, illustrated in Figure 1 and dis-cussed in what follows: ¬ introducing new adapterweights Φ into pre-trained transformer weights Θ;­ training adapter weights Φ on a downstreamtask (while keeping Θ frozen); ® automatic extrac-tion of the trained adapter weights Φ′ and open-sourcing the adapters; ¯ automatic visualizationof the adapters with configuration filters; ° on-the-fly downloading/caching the pre-trained adapterweights Φ′ and stitching the adapter into the pre-trained transformer model Θ; ± performing infer-ence with the trained adapter transformer model.

1

23

4

5

6

ΘΦ

Θ,Φ’

Φ’

Θ

Φ’

Training Adapters Inference

1

2 3

4

5

6

ΘΦ

Θ,Φ’

Φ’

Θ

Φ’

Training adapters Inference

Loading m

odel

& adding new

adap

ters

Extracting and uploading adapters

Loading model

& pre-trained adapters

Finding adapters

Figure 1: The AdapterHub Process graph. Adapters Φare introduced into a pre-trained transformer Θ (step¬) and are trained (­). They can then be extractedand open-sourced (®) and visualized (¯). Pre-trainedadapters are downloaded on-the-fly (°) and stitchedinto a model that is used for inference (±).

¬ Adapters in Transformer Layers

We minimize the required changes to existingHuggingFace training scripts, resulting in onlytwo additional lines of code. In Figure 2 wepresent the required code to add adapter weights(line 3) and freeze all the transformer weightsΘ (line 4). In this example, the model is pre-pared to train a task adapter on the binary ver-sion of the Stanford Sentiment Treebank (SST;Socher et al., 2013) using the adapter architec-ture of Pfeiffer et al. (2020a). Similarly, languageadapters can be added by setting the type parameterto AdapterType.text language, and otheradapter architectures can be chosen accordingly.

While we provide ready-made configuration filesfor well-known architectures in the current litera-ture, adapters are dynamically configurable, whichmakes it possible to define a multitude of architec-tures. We illustrate the configurable components asdashed lines and objects in Figure 3. The config-urable components are placements of new weights,residual connections as well as placements of Lay-erNorm layers (Ba et al., 2016).

The code changes within the HuggingFacetransformers framework are realized throughMixIns, which are inherited by the respectivetransformer classes. This minimizes the amount ofcode changes of our proposed extensions and en-capsulates adapters as designated classes. It furtherincreases readability as adapters are clearly sepa-

1 from adapter_transformers import AutoModelForSequenceClassification, AdapterType2 model = AutoModelForSequenceClassification.from_pretrained("roberta-base")3 model.add_adapter("sst-2", AdapterType.text_task, config="pfeiffer")4 model.train_adapters(["sst-2"])5 # Train model ...6 model.save_adapter("adapters/text-task/sst-2/", "sst")7 # Push link to zip file to AdapterHub ...

Figure 2: ¬ Adding new adapter weights Φ to pre-trained RoBERTa-Base weights Θ (line 3), and freezing Θ (line4). ® Extracting and storing the trained adapter weights Φ′ (line 7).

FeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

FeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

FF Down

FF Up

Add & Norm

(a) Configuration Possibilities

FeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

FeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

FF Down

FF Up

Add & Norm

(b) Pfeiffer ArchitectureFeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

Add & Norm

LayerNorm

LayerNorm

FF Down

FF Up

FeedForward

Add & Norm

Multi-HeadAttention

Add & Norm

FF Down

FF Up

Add & Norm

(c) Houlsby Architecture

Figure 3: Dynamic customization possibilities wheredashed lines in (a) show the current configuration op-tions. These options include the placements of newweights Φ (including down and up projections as wellas new LayerNorms), residual connections, bottlenecksizes as well as activation functions. All new weights Φare illustrated within the pink boxes, everything outsidebelongs to the pre-trained weights Θ. In addition, weprovide pre-set configuration files for architectures inthe literature. The resulting configurations for the archi-tecture proposed by Pfeiffer et al. (2020a) and Houlsbyet al. (2019) are illustrated in (b) and (c) respectively.We also provide a configuration file for the architectureproposed by Bapna and Firat (2019), not shown here.

rated from the main transformers code base,which makes it easy to keep both repositories insync as well as to extend AdapterHub.

­ Training Adapters

Adapters are trained in the same manner as full fine-tuning of the model. The information is passedthrough the different layers of the transformerwhere additionally to the pre-trained weights at ev-ery layer the representations are additionally passedthrough the adapter parameters. However, in con-trast to full fine-tuning, the pre-trained weights Θare fixed and only the adapter weights Φ and theprediction head are trained. Because Θ is fixed, theadapter weights Φ are encapsuled within the trans-former weights, forcing them to learn compatiblerepresentations across tasks.

® Extracting and Open-Sourcing Adapters

When training adapters instead of full fine-tuning,it is no longer necessary to store checkpoints of theentire model. Instead, only the adapter weights Φ′,as well as the prediction head need to be stored, asthe base model’s weights Θ remain the same. Thisis integrated automatically as soon as adapters aretrained, which significantly reduces the requiredstorage space during training and enables storing alarge number of checkpoints simultaneously.

When adapter training has completed, the param-eter file together with the corresponding adapterconfiguration file are zipped and uploaded to a pub-lic server. The user then enters the metadata (e.g.,URL to weights, user info, description of train-ing procedure, data set used, adapter architecture,GitHub handle, Twitter handle) into a designatedYAML file and issues a pull request to the Adapter-Hub GitHub repository. When all automatic checkspass, the AdapterHub.ml website is automaticallyregenerated with the newly available adapter, whichmakes it possible for users to immediately findand use these new weights described by the meta-data. We hope that the ease of sharing pre-trainedadapters will further facilitate and speed up newdevelopments in transfer learning in NLP.

1 from adapter_transformers import AutoModelForSequenceClassification, AdapterType2 model = AutoModelForSequenceClassification.from_pretrained("roberta-base")3 model.load_adapter("sst", config="pfeiffer")

Figure 4: ° After the correct adapter has been identified by the user on the explore page of AdapterHub.ml, theycan load and stitch the pre-trained adapter weights Φ′ into the transformer Θ (line 3).

¯ Finding Pre-Trained Adapters

The website AdapterHub.ml provides a dynamicoverview of the currently available pre-trainedadapters. Due to the large number of tasks in manydifferent languages as well as different transformermodels, we provide an intuitively understandablehierarchical structure, as well as search options.This makes it easy for users to find adapters thatare suitable for their use-case. Namely, Adapter-Hub’s explore page is structured into three hier-archical levels. At the first level, adapters can beviewed by task or language. The second level al-lows for a more fine-grained distinction separatingadapters into data sets of higher-level NLP tasksfollowing a categorization similar to paperswith-code.com. For languages, the second level distin-guishes the adapters by the language they weretrained on. The third level separates adapters intoindividual datasets or domains such as SST forsentiment analysis or Wikipedia for Swahili.

When a specific dataset has been selected, theuser can see the available pre-trained adapters forthis setting. Adapters depend on the transformermodel they were trained on and are otherwise notcompatible.3 The user selects the model architec-ture and certain hyper-parameters and is shown thecompatible adapters. When selecting one of theadapters, the user is provided with additional infor-mation about the adapter, which is available in themetadata (see ® again for more information).

° Stitching-In Pre-Trained Adapters

Pre-trained adapters can be stitched into the largetransformer model as easily as adding randomly ini-tialized weights; this requires a single line of code,see Figure 4, line 3. When selecting an adapter onthe website (see ¯ again) the user is provided withsample code, which corresponds to the configura-tion necessary to include the specific weights.4

3We plan to look into mapping adapters between differentmodels as future work.

4When selecting an adapter based on a name, we allow forstring matching as long as there is no ambiguity.

± Inference with Adapters

Inference with a pre-trained model that relies onadapters is in line with the standard inference prac-tice based on full fine-tuning. Similar to trainingadapters, during inference the active adapter nameis passed into the model together with the text to-kens. At every transformer layer the informationis passed through the transformer layers and thecorresponding adapter parameters.

The adapters can be used for inference in thedesignated task they were trained on. To this end,we provide an option to upload the prediction headstogether with the adapter weights. In addition,they can be used for further research such as trans-ferring the adapter to a new task, stacking multi-ple adapters, fusing the information from diverseadapters, or enriching AdapterHub with adaptersfor other modalities, among many other possiblemodes of usage and future directions.

4 Conclusion and Future Work

We have introduced AdapterHub, a novel easy-to-use framework that enables simple and effectivetransfer learning via training and community shar-ing of adapters. Adapters are small neural modulesthat can be stitched into large pre-trained trans-former models to facilitate, simplify, and speedup transfer learning across a range of languagesand tasks. AdapterHub is built on top of the com-monly used HuggingFace transformers, and itrequires only adding as little as two lines of code toexisting training scripts. Using adapters in Adapter-Hub has numerous benefits such as improved re-producibility, much better efficiency compared tofull fine-tuning, easy extensibility to new modelsand new tasks, and easy access to trained models.

With AdapterHub, we hope to provide a suit-able and stable framework for the community totrain, search, and use adapters. We plan to continu-ously improve the framework, extend the composi-tion and modularity possibilities, and support othertransformer models, even the ones yet to come.

Acknowledgments

Jonas Pfeiffer is supported by the Hessian researchexcellence program “Landes-Offensive zur En-twicklung Wissenschaftlich-Okonomischer Exzel-lenz” (LOEWE) as part of the research centerEmergenCITY. Andreas Ruckle is supported bythe German Federal Ministry of Education and Re-search (BMBF) under the promotional reference03VP02540 (ArgumenText) and by the EuropeanRegional Development Fund (ERDF) and the Hes-sian State Chancellery – Hessian Minister of Dig-ital Strategy and Development under the promo-tional reference 20005482 (TexPrax). AishwaryaKamath is supported in part by a DeepMind PhDFellowship. The work of Ivan Vulic is supportedby the ERC Consolidator Grant LEXICAL: Lex-ical Acquisition Across Languages (no 648909).Kyunghyun Cho is supported by Samsung Ad-vanced Institute of Technology (Next GenerationDeep Learning: from pattern recognition to AI)and Samsung Research (Improving Deep Learningusing Latent Structure).

We would like to thank Isabel Pfeiffer for theillustrations.

ReferencesJacob Andreas, Marcus Rohrbach, Trevor Darrell, and

Dan Klein. 2016. Learning to Compose Neural Net-works for Question Answering. In Proceedings ofNAACL 2016.

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hin-ton. 2016. Layer normalization. arXiv preprint.

Ankur Bapna and Orhan Firat. 2019. Simple, scalableadaptation for neural machine translation. In Pro-ceedings of EMNLP-IJCNLP 2019.

Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro,Danilo Giampiccolo, Bernardo Magnini, and IdanSzpektor. 2006. The second pascal recognising tex-tual entailment challenge. In Proceedings of thePASCAL@ACL 2006.

Tom B. Brown, Benjamin Mann, Nick Ryder, MelanieSubbiah, Jared Kaplan, Prafulla Dhariwal, ArvindNeelakantan, Pranav Shyam, Girish Sastry, AmandaAskell, Sandhini Agarwal, Ariel Herbert-Voss,Gretchen Krueger, Tom Henighan, Rewon Child,Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,Clemens Winter, Christopher Hesse, Mark Chen,Eric Sigler, Mateusz Litwin, Scott Gray, BenjaminChess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and DarioAmodei. 2020. Language models are few-shot learn-ers. arXiv preprint.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017task 1: Semantic textual similarity multilingual andcrosslingual focused evaluation. In Proceedings ofSemEval-2017.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal,Vishrav Chaudhary, Guillaume Wenzek, FranciscoGuzman, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervisedcross-lingual representation learning at scale. InProceedings of ACL 2020.

Ido Dagan, Oren Glickman, and Bernardo Magnini.2005. The pascal recognising textual entailmentchallenge. In PASCAL MLCW 2005.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of NAACL 2019.

William B Dolan and Chris Brockett. 2005. Automati-cally constructing a corpus of sentential paraphrases.In Proceedings of IWP@IJCNLP 2005.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan,and Bill Dolan. 2007. The third pascal recognizingtextual entailment challenge. In Proceedings of thePASCAL@ACL 2007.

Suchin Gururangan, Ana Marasovic, SwabhaSwayamdipta, Kyle Lo, Iz Beltagy, Doug Downey,and Noah A Smith. 2020. Don’t stop pretraining:Adapt language models to domains and tasks. InProceedings of ACL 2020.

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsu-ruoka, and Richard Socher. 2017. A Joint Many-Task Model: Growing a Neural Network for Mul-tiple NLP Tasks. In Proceedings of EMNLP 2017.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkeb-ski, Bruna Morrone, Quentin de Laroussilhe, An-drea Gesmundo, Mona Attariyan, and Sylvain Gelly.2019. Parameter-efficient transfer learning for NLP.In Proceedings of ICML 2019.

Jeremy Howard and Sebastian Ruder. 2018. UniversalLanguage Model Fine-tuning for Text Classification.In Proceedings of ACL 2018.

Shankar Iyer, Nikhil Dandekar, and Kornel Csernai.2017. First quora dataset release: Question pairs.

Jared Kaplan, Sam McCandlish, Tom Henighan,Tom B. Brown, Benjamin Chess, Rewon Child,Scott Gray, Alec Radford, Jeffrey Wu, and DarioAmodei. 2020. Scaling Laws for Neural LanguageModels. arXiv preprint.

Anne Lauscher, Ivan Vulic, Edoardo Maria Ponti, AnnaKorhonen, and Goran Glavas. 2019. Specializingunsupervised pretraining models for word-level se-mantic similarity. arXiv preprint.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019.Roberta: A robustly optimized bert pretraining ap-proach. arXiv preprint.

Cyprien De Masson, Sebastian Ruder, Lingpeng Kong,and Dani Yogatama. 2019. Episodic Memoryin Lifelong Language Learning. In Advances inNeurIPS 2019.

Matthew E. Peters, Sebastian Ruder, and Noah A.Smith. 2019. To tune or not to tune? adapting pre-trained representations to diverse tasks. In Proceed-ings of the RepL4NLP@ACL 2019.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle,Kyunghyun Cho, and Iryna Gurevych. 2020a.AdapterFusion: Non-destructive task compositionfor transfer learning. arXiv preprint.

Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Se-bastian Ruder. 2020b. MAD-X: An Adapter-basedFramework for Multi-task Cross-lingual Transfer.arXiv preprint.

Jason Phang, Thibault Fevry, and Samuel R Bowman.2018. Sentence encoders on stilts: Supplementarytraining on intermediate labeled-data tasks. arXivpreprint.

Yada Pruksachatkun, Jason Phang, Haokun Liu,Phu Mon Htut, Xiaoyi Zhang, Richard YuanzhePang, Clara Vania, Katharina Kann, and Samuel R.Bowman. 2020. Intermediate-Task Transfer Learn-ing with Pretrained Models for Natural LanguageUnderstanding: When and Why Does It Work? InProceedings of ACL 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, KatherineLee, Sharan Narang, Michael Matena, Yanqi Zhou,Wei Li, and Peter J. Liu. 2019. Exploring the Lim-its of Transfer Learning with a Unified Text-to-TextTransformer. arXiv preprint.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. SQuAD: 100,000+ Questionsfor Machine Comprehension of Text. In Proceed-ings of EMNLP 2016.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and AndreaVedaldi. 2017. Learning multiple visual domainswith residual adapters. In Advances in NeurIPS2017.

Sebastian Ruder. 2017. An Overview of Multi-TaskLearning in Deep Neural Networks. arXiv preprint.

Sebastian Ruder, Matthew E Peters, SwabhaSwayamdipta, and Thomas Wolf. 2019. Trans-fer learning in natural language processing. InProceedings of NAACL 2019: Tutorials.

Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019.A Hierarchical Multi-task Approach for LearningEmbeddings from Semantic Tasks. In Proceedingsof AAAI 2019.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz,Andy Davis, Quoc Le, and Jeff Dean. 2017. Out-rageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In Proceedings ofICLR 2017.

Richard Socher, Alex Perelygin, Jean Wu, JasonChuang, Christopher D. Manning, Andrew Ng, andChristopher Potts. 2013. Recursive deep modelsfor semantic compositionality over a sentiment tree-bank. In Proceedings of EMNLP 2013.

Asa Cooper Stickland and Iain Murray. 2019. BERTand PALs: Projected attention layers for efficientadaptation in multi-task learning. In Proceedings ofICML 2019.

Emma Strubell, Ananya Ganesh, and Andrew McCal-lum. 2019. Energy and policy considerations fordeep learning in NLP. In Proceedings of ACL 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention Is AllYou Need. In Advances in NeurIPS 2017.

Alex Wang, Amanpreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R. Bowman. 2018.GLUE: A multi-task benchmark and analysis plat-form for natural language understanding. In Pro-ceedings of BlackboxNLP@EMNLP 2018.

Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xu-anjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang,and Ming Zhou. 2020. K-adapter: Infusing knowl-edge into pre-trained models with adapters. arXivpreprint.

Alex Warstadt, Amanpreet Singh, and Samuel R Bow-man. 2019. Neural network acceptability judgments.Transactions of ACL 2019.

Adina Williams, Nikita Nangia, and Samuel R. Bow-man. 2018. A broad-coverage challenge corpus forsentence understanding through inference. In Pro-ceedings NAACL-HLT 2018.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi an-dArt Pierric Cistac, Tim Rault, Remi Louf, Mor-gan Funtowicz, and Jamie Brew. 2019. Hugging-Face’s Transformers: State-of-the-art Natural Lan-guage Processing. arXiv preprint.