CLIP-Adapter: Better Vision-Language Models with Feature

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Peng Gao∗1, Shijie Geng∗2, Renrui Zhang∗1, Teli Ma1, Rongyao Fang3,Yongfeng Zhang2, Hongsheng Li3, Yu Qiao1

1Shanghai AI Laboratory 2Rutgers University3The Chinese University of Hong Kong

{gaopeng,zhangrenrui,qiaoyu}@[email protected], [email protected]

Abstract

Large-scale contrastive vision-language pre-training has shown significant progress in vi-sual representation learning. Unlike traditionalvisual systems trained by a fixed set of dis-crete labels, a new paradigm was introduced in(Radford et al., 2021) to directly learn to alignimages with raw texts in an open-vocabularysetting. On downstream tasks, a carefully cho-sen text prompt is employed to make zero-shot predictions. To avoid non-trivial promptengineering, context optimization (Zhou et al.,2021) has been proposed to learn continuousvectors as task-specific prompts with few-shottraining examples. In this paper, we show thatthere is an alternative path to achieve bettervision-language models other than prompt tun-ing. While prompt tuning is for the textualinputs, we propose CLIP-Adapter to conductfine-tuning with feature adapters on either vi-sual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layerto learn new features and performs residual-style feature blending with the original pre-trained features. As a consequence, CLIP-Adapter is able to outperform context opti-mization while maintains a simple design. Ex-periments and extensive ablation studies onvarious visual classification tasks demonstratethe effectiveness of our approach.

1 Introduction

Visual understanding tasks, such as classifica-tion (Krizhevsky et al., 2012; He et al., 2016;Howard et al., 2017; Dosovitskiy et al., 2021;Touvron et al., 2021; Gao et al., 2021a; Maoet al., 2021), object detection (Ren et al., 2015;Carion et al., 2020; Gao et al., 2021b), and se-mantic segmentation (Long et al., 2015), havebeen improved significantly based on the betterarchitecture designs and large-scale high-quality

∗ Indicates equal contributions

datasets. Unfortunately, collecting large-scale high-quality datasets for every visual task is labor-intensive and too expensive to scale. To solve theproblem, the “pretraining-finetuning” paradigm,namely pretraining on large-scale datasets like Im-ageNet (Krizhevsky et al., 2012) and then fine-tuning on a variety of downstream tasks, has beenwidely adopted in vision domain. However, suchapproaches still need a huge amount of annota-tions for fine-tuning on many downstream tasks.Recently, Contrastive Language-Image Pretrain-ing (CLIP) (Radford et al., 2021) was proposedfor solving vision tasks by exploiting contrastivelearning with large-scale noisy image-text pairs.It achieves inspirational performances on variousvisual classification tasks without any annotations(i.e., zero-shot transfer) by putting visual categoriesinto suitable hand-crafted template as prompts.

Although prompt-based zero-shot transfer learn-ing showed promising performances, designinggood prompts remains an engineering problemthat demands substantial time and domain knowl-edge. To address the issue, Context Optimization(CoOp) (Zhou et al., 2021) further proposed tolearn continuous soft prompts with few-shot ex-amples for replacing the carefully-chosen hardprompts. CoOp brings about significant improve-ment on few-shot classification over both zero-shotCLIP and linear probe CLIP settings, exhibiting thepotential of prompt tuning on large-scale pretrainedvision-language models.

In this paper, we propose a different approachfor better adapting vision-language models withfeature adapters instead of prompt tuning. Differ-ent from CoOp that performs soft prompt opti-mization, we simply conduct fine-tuning on thelight-weight additional feature adapters. Becauseof the over-parameterization of CLIP and lack ofenough training examples, naive finetuning wouldlead to overfitting on specific datasets and thetraining process would be very slow owing to

arX

iv:2

110.

0454

4v1

[cs

.CV

] 9

Oct

202

1

CNN

Trans

form

er

carplane

dogflower

Visual Embedding Classifier Weight

MatMul

BERT[CLS] this is a dog

car plane dog flower

Naive Classification

carplane

dogflower

CLIPBERT[CLS] [V]1 [V]2 [V]n dog


carplane

dogflower

CoOp

BERT[CLS] this is a dog


CLIP-Adapter

carplane

dogflower

Figure 1: Comparison of different visual classification architectures. The image in the top row with a green re-gion shows the naive pipeline for image classification (Krizhevsky et al., 2012), where f and W represents thefeature and classifier weight respectively. The following pink, yellow and blue regions represent the pipeline ofCLIP (Radford et al., 2021), CoOp (Zhou et al., 2021), and our proposed CLIP-Adapter respectively.

the forward and backward propagation across allCLIP layers. Motivated by the adapter modulesin parameter-efficient transfer learning (Houlsbyet al., 2019), we propose CLIP-Adapter, which onlyfinetunes a small number of additional weights in-stead of optimizing all parameters of CLIP. CLIP-Adapter adopts a lightweight bottleneck architec-ture to prevent the potential overfitting problemof few-shot learning by reducing the number ofparameters. Meanwhile, CLIP-Adapter is differ-ent from Houlsby et al. (2019) in two importantaspects: CLIP-Adapter only adds two additionallinear layers following the last layer of vision or lan-guage backbone. In contrast, the original adaptermodules are inserted into all layers of the languagebackbone; In addition, CLIP-Adapter mixes theoriginal zero-shot visual or language embeddingwith the corresponding finetuning feature via resid-ual connection. Through such a “residual-styleblending”, CLIP-Adapter can simultaneously ex-ploit the knowledge stored in the original CLIP andthe freshly learned knowledge originated from thefew-shot training examples. Overall, our contribu-

tions can be summarized as follows:

• We propose CLIP-Adapter that conducts residual-style feature blending to achieve efficient few-shot transfer learning via fine-tuning.

• Compared with CoOp, CLIP-Adapter achievesbetter few-shot classification performance whilehas a much simpler design, demonstrating thatCLIP-Adapter is a promising alternative toprompt tuning.

• We perform extensive ablation studies ofCLIP-Adapter on eleven classification datasetsto analyze its characteristics. The code willbe released at https://github.com/gaopengcuhk/CLIP-Adapter.

2 Related Work

2.1 Model Fine-Tuning

Deep neural network is data-hungry. However, col-lecting and annotating large amount of high-qualitydata is costly and even impossible for some specialdomains. The “pretraining-finetuning paradigm”offers a good solution to different computer vi-

https://github.com/gaopengcuhk/CLIP-Adapter

https://github.com/gaopengcuhk/CLIP-Adapter

sion (Krizhevsky et al., 2012; Simonyan and Zis-serman, 2015; He et al., 2016) and natural lan-guage processing (Kenton and Toutanova, 2019;Dong et al., 2019; Conneau et al., 2020) tasks andhas been widely adopted for many years. For data-efficient finetuning over downstream tasks, adaptermodules (Houlsby et al., 2019) is proposed tofreeze the weight of backbones and insert learnablelinear layers to each Transformer layer. Differentfrom adapter modules, the proposed CLIP-Adapterapplies a simple residual transformation layer overthe feature embedding or classifier weight gen-erated by CLIP. Thanks to the residual connec-tion and bottleneck linear layer, CLIP-Adaptercan improve the performance of CLIP on few-shot learning setting and achieve superior perfor-mance than the recently proposed CoOp. To alle-viate the performance gap under distribution shift-ing, WiSE-FT (Wortsman et al., 2021) proposes apost-ensemble method for improving CLIP’s out-of-distribution robustness. While WiSE-FT frozethe weight of image branch during fine-tuning, ourCLIP-Adapter can be applied to both image andtext branches with a learnable gating ratio to dy-namically balance and mix the knowledge from theoriginal features and CLIP-Adapter’s outputs.

2.2 Prompt Design

Prompt design (Liu et al., 2021a) are popularizedby the success of GPT series (Radford et al., 2019;Brown et al., 2020). GPT-3 showed that a hugeautoregressive language model trained on a large-scale dataset can perform any NLP tasks in a zero-shot or few-shot style without finetuning the basearchitecture. Following the brand new “pre-train,prompt, and predict” paradigm, various prompt de-sign approaches are proposed recently. One typeof them focus on prompt engineering by miningor generating proper discrete prompts (Jiang et al.,2020; Shin et al., 2020; Gao et al., 2021c). In con-trast, continuous prompts circumvent the restrictionfrom pretrained language models and are adoptedby Li and Liang (2021); Liu et al. (2021b); Lesteret al. (2021); Gu et al. (2021) on NLP tasks. Mo-tivated by GPT-3, CLIP trains a large contrastivelearning model over 400 million image-text pairsand demonstrates the potential for prompt-basedzero-shot visual classification. With CLIP as back-bone, CoOp (Zhou et al., 2021) and CPT (Yaoet al., 2021) further shows that optimizing continu-ous prompts can largely surpass manually-designed

discrete prompts on vision tasks. In this paper, wedemonstrate that prompt tuning is not the only pathto better vision-language models. Fine-tuning witha small portion of parameters can also achieve com-parable or even better performance on vision tasksyet with much simpler design.

2.3 Vision-Language Models

Exploring the interaction between vision and lan-guage is a core research topic in artificial intelli-gence. Previously, attention-based approaches suchas bottom-up top-down attention (Anderson et al.,2018), BAN (Kim et al., 2018), Intra-Inter (Gaoet al., 2019), and MCAN (Yu et al., 2019) haddominated visual-language tasks. Inspired by thesuccess of BERT (Kenton and Toutanova, 2019),ViLBERT (Lu et al., 2019), LXMERT (Tan andBansal, 2019), UNITER (Chen et al., 2020), andOscar (Li et al., 2020) further push the boundaryof multimodal reasoning. Recently, CLIP (Rad-ford et al., 2021) and ALIGN (Jia et al., 2021)demonstrates the power of visual-language con-trastive representation learning. They achieve as-tonishing results on a wide spectrum of vision taskswithout any fine-tuning. To further close the gapbetween CLIP and supervised training, CoOp pro-poses a continuous prompt optimization methodfor improving the performance on visual classifica-tion tasks. While CoOp improves vision-languagemodels from the perspective of prompt design, ourCLIP-Adapter explores simple finetuning with thehelp of lightweight feature adapters.

3 Our Approach

In this section, we introduce the proposed CLIP-Adapter. In Section 3.1, we first revisit CLIP andCoOp from the perspective of classifier weight gen-eration. In Section 3.2, we elaborate the detailsof the proposed CLIP-Adapter. In Section 3.3, weprovide several variants of CLIP-Adapter.

3.1 Classifier Weight Generation forFew-Shot Learning

Let us first review the basic framework for imageclassification using deep neural networks: Givenan image I ∈ RH×W×3, where H and W standsfor the height and width of the image respec-tively, a neural network backbone that consists ofcascade of basic components (e.g., CNN, Trans-former (Vaswani et al., 2017) or the mixture ofboth) takes I and transforms it into a feature man-

ifold f ∈ RD, where D represents the feature di-mensionality. To perform classification, the imagefeature vector f is then multiplied with a classifierweight matrix W ∈ RD×K , where K representsthe number of classes to be classified. After ma-trix multiplication, we can obtain a K-dimensionallogit. A Softmax function is used to convert thelogit into a probability vector p ∈ RK over the Kclasses. The whole process can be written as thefollowing equations:

f = Backbone(I), pi =exp(WT

i f)/τ∑Nj=1 exp(W

Tj f)/τ

,

(1)where τ stands for the temperature of Softmax, Wi

represents the prototype weight vector for class i,and pi denotes the probability of category i.

Different from supervised training, in the paper,we are interested in image classification with few-shot examples. Training the backbone and clas-sifier together from scratch with a small numberof samples is prone to over-fit certain datasets andmight suffer from severe performance drop on thetest split. Typically, the representative paradigmon few-shot learning is to first pretrain the back-bone on a large-scale dataset, and then transfer thelearned knowledge to downstream tasks by eitherconducting zero-shot prediction directly or furtherfine-tuning on few-shot examples.

CLIP adheres to the zero-shot transfer style – itfirst pretrains the visual backbone and textual en-coder through contrastive learning on large-scalenoisy image-text pairs, and then after pretraining,CLIP directly performs image classification with-out any finetuning. Given an image classificationdownstream dataset that containsK categories withtheir natural language name {C1, . . . , Ck}, CLIPconstructs to place each category name Ci into thepre-defined hard prompt template H . Then thelanguage feature extractor encodes the resultingprompt as a classifier weight Wi. We denote theclassifier weight generation process as below:

Wi = BERT(Tokenizer([H;Ci])). (2)

Alternatively, CoOp adopts continuous promptsinstead of hand-crafted hard prompts. CoOp cre-ates a list of random-initialized learnable soft to-kens S ∈ RL×D, where L stands for the length ofthe soft token sequence. The soft token sequence Sis then concatenated to each class nameCi and thusform a prompt. We represent the whole process as

Wi = BERT([S; Tokenizer(Ci)]). (3)

For both CLIP and CoOp, with the generatedclassifier weight Wi, where i ∈ {1, · · · ,K}, wecan thus calculate the prediction probability pi forclass i by the previously mentioned Eq. (1).

3.2 CLIP-AdapterUnlike CoOp’s prompt tuning, we present an al-ternative framework for achieving better vision-language models on few-shot image classifica-tion by fine-tuning additional feature adapters. Weclaim that the previous widely-adopted “pretrain-finetuning” paradigm would fail in finetuning thewhole CLIP backbone under the few-shot settingdue to the enormous amount of parameters and theshortage of training examples. Hence, we proposeCLIP-Adapter, which only appends a small num-ber of additional learnable bottleneck linear lay-ers to CLIP’s language and image branches whilekeep the original CLIP backbone frozen duringfew-shot fine-tuning. However, naive fine-tuningwith additional layer may still fall into over-fittingon the few-shot examples. To deal with over-fittingand improve the robustness of CLIP-Adapter, wefurther adopt residual connections to dynamicallyblend the fine-tuned knowledge with the originalknowledge from CLIP’s backbone.

Specifically, given the input image I and a setof categories’ natural language names {Ci}Ki=1, theimage feature f and classifier weight W from theoriginal CLIP backbone are computed with Equa-tions (1) and (2). Afterwards, two learnable featureadapters, Av(·) and At(·), each of which containstwo layers of linear transformations, are integratedto transform f and W, respectively. We adopt aresidual connection for the feature adapter to avoidforgetting the original knowledge encoded by thepretrained CLIP. Two constant values α and β areemployed as “residual ratio” to help adjust the de-gree of maintaining the original knowledge for bet-ter performance. In summary, the feature adapterscan be written as

Av(f) = ReLU(fTWv1)W

v2, (4)

At(W) = ReLU(WTWt1)W

t2. (5)

The new knowledge captured via finetuning isadded with the original features via residual con-nections:

f? = αAv(f)T + (1− α)f, (6)

W? = βAt(W)T + (1− β)W. (7)

After obtaining new image feature f? and classifierweight W?, we also adopt Equation (1) to calculate

the category probability vector P = {pi}Ki=1 andpredict the image category by selecting the class ithat has the highest probability: i = argmaxi pi.

During the few-shot training, the weights ofAv(·) and At(·) are optimized with the cross-entropy loss:

L(θ) = − 1

N

N∑n=1

K∑i=1

y(n)i log y

(n)i , (8)

where N is the total number of training examples;yi = 1 if i equals to the ground-truth categorylabel i, otherwise yi = 0; yi = pi is the predictedprobability for class i; θ = {Wv

1,Wv2,W

t1,W

t2}

represents all learnable parameters.

3.3 Variants of CLIP-Adapter

Our CLIP-Adapter has three structural variants: 1)only fine-tuning the feature adapter for the imagebranch while keep the text branch frozen; 2) onlyfine-tuning the feature adapter for the text branchwhile keeping the image branch frozen; 3) fine-tuning both the image and text branches of CLIPbackbone. In terms of the hyperparameters α andβ, we observe that different datasets have differentoptimal α and β values. Choosing the hyperparam-eters manually is time-consuming and laborious.Thus we also explore learning α and β in a differ-entiable manner by setting them as learnable pa-rameters. In this way, α and β can be dynamicallypredicted from either visual feature or classifierweight via a hypernetwork Q: α, β = Q(f,W).

4 Experiments

4.1 Few-Shot Learning

4.1.1 Training SettingsFollowing CLIP (Radford et al., 2021) andCoOp (Zhou et al., 2021), we select 11 image clas-sification datasets to validate CLIP-Adapter’s ef-fectiveness: ImageNet (Deng et al., 2009), Stan-fordCars (Krause et al., 2013), UCF101 (Soomroet al., 2012), Caltech101 (Fei-Fei et al., 2004),Flowers102 (Nilsback and Zisserman, 2008),SUN397 (Xiao et al., 2010), DTD (Cimpoi et al.,2014), EuroSAT (Helber et al., 2019), FGVCAir-craft (Maji et al., 2013), OxfordPets (Parkhi et al.,2012), and Food101 (Bossard et al., 2014). Specif-ically, we train our CLIP-Adapter under the few-shot setups of 1, 2, 4, 8, 16 shots and then test thetuned models on full test splits. We conduct allexperiments on a single Nvidia A100 GPU.

The first variant of CLIP-Adapter is adopted bydefault if not specified, which finetunes the imagefeature while freezes the classifier weight. In otherwords, it only implements CLIP-Adapter for thevisual adapter. The results of other variants that acti-vate text adapter are presented in Section 4.1.5. Weuse the same training hyperparameters as CoOp,including a batch size of 32 and a learning rateof 1 × 10−5 for all datasets except for the resid-ual ratio α. We perform hyperparameter search-ing over different value selections of α for eachdataset and report the best performance among allsearching spaces. We use ResNet-50 (He et al.,2016) as the visual backbone (visual encoder) andBERT (Kenton and Toutanova, 2019) as classifierweight generator (textual encoder). The hiddenembedding dimensionality of both visual and textbottleneck layers is set to 256, which is a quarterof the original embedding dimensionality. In con-trast to the learnable continuous prompts in CoOp,simple hand-crafted hard prompts are utilized asthe text inputs of CLIP-Adapter, which is the sameas CLIP. For generic-category image datasets, suchas ImageNet, we adopt “a photo of a {CLASS}” asthe hard prompt template. For fine-grained clas-sification datasets, we specify its correspondingdomain keyword in the template for a better per-formance, for instance, “a centered satellite photoof {CLASS}" for EuroSAT, and similarly for otherfine-grained datasets.

4.1.2 Baseline ModelsWe compare our CLIP-Adapter with three base-line models – the Zero-shot CLIP (Radford et al.,2021), Linear probe CLIP (Radford et al., 2021),and CoOp (Zhou et al., 2021). In our implementa-tion, CLIP-Adapter shares the same hand-craftedhard prompts with Zero-shot CLIP (Radford et al.,2021) for fair comparisons. CoOp (Zhou et al.,2021) substitutes discrete tokens with learnablecontinuous vectors. Thus there are multiple candi-date positions to place the class token in the prompttemplate, namely at the front, in the middle, or atthe end. Here, we choose CoOp’s best-performancevariant – placing the class token at the end of the 16-token soft prompt and shares such a context amongdifferent classes. Linear probe CLIP (Radford et al.,2021) trains an additional linear classifier on top ofits visual encoder and follows a few-shot trainingmanner. It is different from our bottleneck adapterthat finetunes both the image feature and classifierweight in a dynamic and residual fashion.

0 2 4 6 8 10 12 14 16Number of labeled training examples per class

35

40

45

50

55

60

65

70

75

Scor

e (%

)

Zero-shotCLIP

Average over 11 datasets

CLIP-AdapterCoOpLinear probe CLIPZero-shot CLIP


20

25

30

35

40

45

50

55

60

Scor

e (%

)

Zero-shotCLIP

ImageNet



70

75

80

85

90

Scor

e (%

)

Zero-shotCLIP

Caltech101



30

40

50

60

70

80

Scor

e (%

)

Zero-shotCLIP

OxfordPets



30

40

50

60

70

Scor

e (%

)Zero-shotCLIP

StanfordCars



60

65

70

75

80

85

90

95

Scor

e (%

)

Zero-shotCLIP

Flowers102



30

40

50

60

70

Scor

e (%

)

Zero-shotCLIP

Food101



10

15

20

25

30

35

40

Scor

e (%

)

Zero-shotCLIP

FGVCAircraft



35

40

45

50

55

60

65

70

Scor

e (%

) Zero-shotCLIP

SUN397



25

30

35

40

45

50

55

60

65

Scor

e (%

)

Zero-shotCLIP

DTD



20

30

40

50

60

70

80

Scor

e (%

)

Zero-shotCLIP

EuroSAT



40

45

50

55

60

65

70

75

Scor

e (%

)

Zero-shotCLIP

UCF101


Figure 2: Main results of few-shot learning on 11 datasets. CLIP-Adapter consistently shows better performanceover previous baselines across different training shots.

4.1.3 Performance Comparison & Analysis

The main results are presented in Figure 2. Fromthe average accuracy over the 11 datasets shownat the top-left corner, CLIP-Adapter clearly outper-forms the other three baseline models on all differ-ent shot setups, demonstrating its superior few-shotlearning capacity. It is especially worth noticingthat, under extreme conditions such as 1-shot or 2-shot training setup, CLIP-Adapter achieves largerperformance improvements against the baselines,which indicates a better generalization ability indata-deficient training circumstances.

Compared with Zero-shot CLIP (Radford et al.,2021), our CLIP-Adapter achieves significant per-

formance gains over all 11 datasets. The rankedabsolute performance improvements for all 11datasets under the 16-shot training setup are shownin Figure 3. For the first five fine-grained datasets,from EuroSAT to FGVCAircraft, CLIP-Adapterachieves huge performance boosts ranging from20% to 50%. The improvements become smalleron more challenging and generic datasets, such asCaltech101 and ImageNet. As for OxfordPets andFood101, CLIP-Adapter shows relatively limitedimprovements, since the original results of Zero-shot CLIP are already quite decent.

Compared with Linear probe CLIP (Radfordet al., 2021), which follows a similar style to fine-

tune the pretrained vision-language models, CLIP-Adapter also shows comprehensive performanceadvantages. Under 1-shot and 2-shot training se-tups, Linear probe CLIP barely reaches the perfor-mance of Zero-shot CLIP, but CLIP-Adapter canalways surpass Zero-shot CLIP and exceed Linearprobe CLIP by a large margin. For instance, theabsolute margin of 1-shot and 2-shot training se-tups are 53.6% and 42.16% for OxfordPets, and37.17% and 27.58% for ImageNet, respectively.

Compared with CoOp (Zhou et al., 2021), al-though it has already gained huge improvementsover Zero-shot CLIP, CLIP-Adapter still outper-forms CoOp on all datasets and different shot set-tings. Note that CLIP-Adapter handles few-shotlearning from a totally different perspective (i.e.,fine-tuning) instead of CoOp’s prompt tuning. Thissuggests that finetuning lightweight adapters withresidual connections for prompt-fixed pretrainedvision-language models can achieve better perfor-mance than prompt engineering (Liu et al., 2021a)approaches.

EuroS

AT

Flowers

102

DTD

Stanfo

rdCars

FGVCAirc

raft

UCF101

SUN39

7

Caltech

101

Imag

eNet

OxfordP

ets

Food1

010

10

20

30

40

50

60

Abso

lute

Impr

ovem

ent (

%)

51.32

31.89

25.4324.8921.96

15.4511.9

9.015.83 4.19

2.02

CLIP-Adapter vs. Zero-shot CLIP

Figure 3: Absolute performance gain of CLIP-Adapteragainst hand-crafted prompts on different datasets.

4.1.4 Observation on Optimal Residual RatioInterestingly, we observe the best residual ratioα, to some extent, reflects the characteristics ofdifferent datasets under the “pretrain-finetuning”paradigm. A larger semantic gap between pre-trained and finetuning datasets requires CLIP-Adapter to learn a higher portion of knowledgefrom the newly adapted feature compared to theoriginal CLIP’s output, thus resulting in a largeroptimal residual ratio, and vice versa. For fine-grained datasets on specialized domains, like Eu-roSAT of satellite images and DTD of detailed tex-tures, the optimal residual ratio α is usually locatedwithin the range from 0.6 to 0.8. By contrast, the

best α value of comprehensive and generic imagedatasets (e.g., Caltech-101 and ImageNet) is oftenaround 0.2.

4.1.5 Variants with Text AdapterHere, we investigate the other two variants of CLIP-Adapter mentioned in Section 3.3 – finetuning thetext adapter while keeping the visual adapter frozenand finetuning both the text and visual adapters.Rather than manually selecting the residual ratiosfor each dataset, we utilize learnable parameters αand β since it is time-efficient and can also achievesatisfactory performance. We compare their perfor-mances on four datasets that can be divided intotwo categories – fine-grained datasets (EuroSAT& DTD) and generic datasets (Caltech101 & Ima-geNet). As shown in Figure 4, we can conclude thatthe text adapter and visual adapter performs compa-rably and both improve the classification accuracygreatly over Zero-shot CLIP. In addition, adopt-ing visual adapter only is better than text adapteronly. This indicates that it is more important toconduct image feature adaption than text featureadaption for few-shot image classification, sincethe semantic gap between visual features in pre-trained and finetuning datasets is larger than that oftext features. Surprisingly, combining both adapterstogether does not observe a better performance thanvisual adapter only. This demonstrates that the textand visual adapters might capture redundant infor-mation or even conflict with each other.

60 70 80 90 100Score (%)

ImageNet

Caltech101

EuroSAT

DTD

61.33

93.43

82.85

66.06

60.06

92.78

81.85

63.63

60.7

93.1

82.74

64.78

Variants of CLIP-Adaptervisual adaptertext adaptervisual + text adapter

Figure 4: Comparison among different variants ofCLIP-Adapter.

4.2 Visualization of Manifold

We use t-SNE (Van der Maaten and Hinton, 2008)to visualize the manifold of CLIP, CoOp, CLIP-Adapter without residual connections, and CLIP-Adapter with residual connections after trainingthem on the EuroSAT dataset. The t-SNE visual-

Figure 5: Visualization of different learned feature manifolds via t-SNE.

ization results are presented in Figure 5, wherethe numbers 0 to 9 stand for the categories of An-nualCrop, Forest, Herbaceous Vegetation Land,Highway or Road, Industrial Buildings, PastureLand, Permanent Crop Land, Residential Build-ings, River, Sea or Lake, respectively. It is clearlyillustrated that in high-dimensional classificationspace, the CLIP-Adapter with residual connectionsin sub-figure (d) shows much more obvious separa-tion of image features belong to different categories.As for the confusing categories such as Highwayor Road (red points), Permanent Crop Land (pinkpoints), and Pasture Land (brown points), com-pared with other methods, our CLIP-Adapter ismore effective in detecting the similarities amongthe image manifolds from the same class. In sum-mary, the visualization results prove that CLIP-Adapter is good at learning better feature manifoldsunder few-shot setups.

4.3 Ablation Studies

In this section, we perform several ablation studiesfor CLIP-Adapter. We choose the best-performancevariant which only activates the visual adapter, andselect two datasets – DTD & ImageNet, servingas the representatives of fine-grained and genericdatasets, to perform the ablation studies.

4.3.1 Dimension of Bottleneck LayerWe first conduct ablations by varying the hidden di-mension of bottleneck layers. The results are shownin Table 1, where D represents the dimensionalityof the original image feature. By reducing the hid-den dimension from D to D/32, we observe thateither too small or too large intermediate dimen-sionality will deteriorate the performance signifi-cantly and the best bottleneck dimension is D/4,which is able to preserve enough semantics withoutredundancy.

Dimension D D/2 D/4 D/8 D/16 D/32

DTD (%) 65.03 65.62 66.06 64.93 63.75 63.50ImageNet (%) 59.78 60.03 61.33 60.06 60.02 59.45

Table 1: Ablations on varying the hidden dimension ofbottleneck layers.

4.3.2 Residual Ratio αMoreover, we perform ablation study of the resid-ual ratio α. From Table 2, we can see that the bestresidual ratio of fine-grained dataset DTD is 0.6,and that of generic dataset ImageNet is 0.2. Thisverifies our observation in Section 4.1.4 that adapt-ing fine-grained dataset requires more new knowl-edge than old knowledge, and the case is oppositefor generic dataset. Note that when α equals to

0, it is equivalent to Zero-shot CLIP since no newknowledge is learned. When α is set to 1.0, theclassification is fully rely on the adapted feature(CLIP-Adapter w/o Res). However, this is not opti-mal because CLIP-Adapter tends to over-fit in suchcondition. Combining Table 2 and Figure 5, wecan also conclude the advantages of residual con-nections in CLIP-Adapter: 1) avoids over-fitting onfew-shot examples and improves the generalizationability of CLIP-Adapter with the help of zero-shotknowledge; 2) preserves the freedom for learningbetter image feature or classifier weight throughfew-shot fine-tuning.

Ratio α 0 0.2 0.4 0.6 0.8 1.0

DTD (%) 40.72 54.59 64.84 66.06 65.96 63.79ImageNet (%) 60.46 61.33 61.17 60.77 59.79 59.05

Table 2: Ablations on varying the residual ratio α.

5 Conclusions and Future Work

We present CLIP-Adapter as an alternative ofprompt-based approaches for few-shot image clas-sification. The CLIP-Adapter revives the “pretrain-finetuning” paradigm by only fine-tuning a smallnumber of additional bottleneck layers. To furtherimprove the generalization ability, we adopt resid-ual connections parameterized by a residual ratio todynamically blend zero-shot knowledge with newadapted features. According to the experimental re-sults, CLIP-Adapter outperforms competitive base-lines on eleven image classification datasets underdifferent few-shot setups. Extensive ablation stud-ies confirm our design and prove CLIP-Adapter’sability in learning better feature manifolds. In thefuture, we plan to extend CLIP-Adapter to morevision-language applications. We will also combineCLIP-Adapter with soft prompts together to furtherunleash the power of CLIP backbone.

References

Peter Anderson, Xiaodong He, Chris Buehler, DamienTeney, Mark Johnson, Stephen Gould, and LeiZhang. 2018. Bottom-up and top-down attention forimage captioning and visual question answering. InCVPR.

Lukas Bossard, Matthieu Guillaumin, and LucVan Gool. 2014. Food-101–mining discrimina-tive components with random forests. In Euro-pean conference on computer vision, pages 446–461.Springer.

Tom Brown, Benjamin Mann, Nick Ryder, MelanieSubbiah, Jared D Kaplan, Prafulla Dhariwal,Arvind Neelakantan, Pranav Shyam, Girish Sastry,Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, RewonChild, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu,Clemens Winter, Chris Hesse, Mark Chen, EricSigler, Mateusz Litwin, Scott Gray, Benjamin Chess,Jack Clark, Christopher Berner, Sam McCandlish,Alec Radford, Ilya Sutskever, and Dario Amodei.2020. Language models are few-shot learners. InNeurIPS.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve,Nicolas Usunier, Alexander Kirillov, and SergeyZagoruyko. 2020. End-to-end object detection withtransformers. In ECCV.

Yen-Chun Chen, Linjie Li, Licheng Yu, AhmedEl Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, andJingjing Liu. 2020. Uniter: Learning universalimage-text representations. In ECCV.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos,Sammy Mohamed, and Andrea Vedaldi. 2014. De-scribing textures in the wild. In Proceedings of theIEEE Conference on Computer Vision and PatternRecognition, pages 3606–3613.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal,Vishrav Chaudhary, Guillaume Wenzek, FranciscoGuzmán, Édouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervisedcross-lingual representation learning at scale. InACL.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, KaiLi, and Li Fei-Fei. 2009. Imagenet: A large-scalehierarchical image database. In CVPR.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi-aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou,and Hsiao-Wuen Hon. 2019. Unified languagemodel pre-training for natural language understand-ing and generation. In NeurIPS.

Alexey Dosovitskiy, Lucas Beyer, AlexanderKolesnikov, Dirk Weissenborn, Xiaohua Zhai,Thomas Unterthiner, Mostafa Dehghani, MatthiasMinderer, Georg Heigold, Sylvain Gelly, JakobUszkoreit, and Neil Houlsby. 2021. An imageis worth 16x16 words: Transformers for imagerecognition at scale. In ICLR.

Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004.Learning generative visual models from few trainingexamples: An incremental bayesian approach testedon 101 object categories. In 2004 conference oncomputer vision and pattern recognition workshop,pages 178–178. IEEE.

Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu,Steven CH Hoi, Xiaogang Wang, and Hongsheng Li.2019. Dynamic fusion with intra-and inter-modalityattention flow for visual question answering. InCVPR.

Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mot-taghi, and Aniruddha Kembhavi. 2021a. Container:Context aggregation network. In NeurIPS.

Peng Gao, Minghang Zheng, Xiaogang Wang, JifengDai, and Hongsheng Li. 2021b. Fast convergenceof detr with spatially modulated co-attention. arXivpreprint arXiv:2101.07448.

Tianyu Gao, Adam Fisch, and Danqi Chen. 2021c.Making pre-trained language models better few-shotlearners. In ACL-IJCNLP.

Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang.2021. Ppt: Pre-trained prompt tuning for few-shotlearning. arXiv preprint arXiv:2109.04332.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep residual learning for image recog-nition. In CVPR.

Patrick Helber, Benjamin Bischke, Andreas Dengel,and Damian Borth. 2019. Eurosat: A novel datasetand deep learning benchmark for land use and landcover classification. IEEE Journal of Selected Top-ics in Applied Earth Observations and Remote Sens-ing, 12(7):2217–2226.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski,Bruna Morrone, Quentin De Laroussilhe, AndreaGesmundo, Mona Attariyan, and Sylvain Gelly.2019. Parameter-efficient transfer learning for nlp.In ICML.

Andrew G Howard, Menglong Zhu, Bo Chen, DmitryKalenichenko, Weijun Wang, Tobias Weyand,Marco Andreetto, and Hartwig Adam. 2017. Mo-bilenets: Efficient convolutional neural networksfor mobile vision applications. arXiv preprintarXiv:1704.04861.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, ZaranaParekh, Hieu Pham, Quoc V Le, Yunhsuan Sung,Zhen Li, and Tom Duerig. 2021. Scaling up visualand vision-language representation learning withnoisy text supervision. In ICML.

Zhengbao Jiang, Frank F Xu, Jun Araki, and GrahamNeubig. 2020. How can we know what languagemodels know? Transactions of the Association forComputational Linguistics, 8:423–438.

Jacob Devlin Ming-Wei Chang Kenton andLee Kristina Toutanova. 2019. Bert: Pre-trainingof deep bidirectional transformers for languageunderstanding. In NAACL-HLT.

Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang.2018. Bilinear attention networks. In NIPS.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grainedcategorization. In Proceedings of the IEEE inter-national conference on computer vision workshops,pages 554–561.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hin-ton. 2012. Imagenet classification with deep convo-lutional neural networks. In NIPS.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.The power of scale for parameter-efficient prompttuning. In EMNLP.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:Optimizing continuous prompts for generation. InACL.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xi-aowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu,Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantics aligned pre-training for vision-languagetasks. In ECCV.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang,Hiroaki Hayashi, and Graham Neubig. 2021a. Pre-train, prompt, and predict: A systematic survey ofprompting methods in natural language processing.arXiv preprint arXiv:2107.13586.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding,Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gptunderstands, too. arXiv preprint arXiv:2103.10385.

Jonathan Long, Evan Shelhamer, and Trevor Darrell.2015. Fully convolutional networks for semanticsegmentation. In CVPR.

Jiasen Lu, Dhruv Batra, Devi Parikh, and StefanLee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-languagetasks. In NeurIPS.

Subhransu Maji, Esa Rahtu, Juho Kannala, MatthewBlaschko, and Andrea Vedaldi. 2013. Fine-grainedvisual classification of aircraft. arXiv preprintarXiv:1306.5151.

Mingyuan Mao, Renrui Zhang, Honghui Zheng, PengGao, Teli Ma, Yan Peng, Errui Ding, and ShuminHan. 2021. Dual-stream network for visual recogni-tion. arXiv preprint arXiv:2105.14734.

Maria-Elena Nilsback and Andrew Zisserman. 2008.Automated flower classification over a large numberof classes. In 2008 Sixth Indian Conference on Com-puter Vision, Graphics & Image Processing, pages722–729. IEEE.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman,and CV Jawahar. 2012. Cats and dogs. In 2012IEEE conference on computer vision and patternrecognition, pages 3498–3505. IEEE.

Alec Radford, Jong Wook Kim, Chris Hallacy, AdityaRamesh, Gabriel Goh, Sandhini Agarwal, GirishSastry, Amanda Askell, Pamela Mishkin, Jack Clark,et al. 2021. Learning transferable visual modelsfrom natural language supervision. In ICML.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, Ilya Sutskever, et al. 2019. Lan-guage models are unsupervised multitask learners.OpenAI blog.

Shaoqing Ren, Kaiming He, Ross Girshick, and JianSun. 2015. Faster r-cnn: Towards real-time objectdetection with region proposal networks. In NIPS.

Taylor Shin, Yasaman Razeghi, Robert L Logan IV,Eric Wallace, and Sameer Singh. 2020. Autoprompt:Eliciting knowledge from language models with au-tomatically generated prompts. In EMNLP.

Karen Simonyan and Andrew Zisserman. 2015. Verydeep convolutional networks for large-scale imagerecognition. In ICLR.

Khurram Soomro, Amir Roshan Zamir, and MubarakShah. 2012. Ucf101: A dataset of 101 human ac-tions classes from videos in the wild. arXiv preprintarXiv:1212.0402.

Hao Tan and Mohit Bansal. 2019. Lxmert: Learningcross-modality encoder representations from trans-formers. In EMNLP-IJCNLP.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran-cisco Massa, Alexandre Sablayrolles, and Hervé Jé-gou. 2021. Training data-efficient image transform-ers & distillation through attention. In ICML.

Laurens Van der Maaten and Geoffrey Hinton. 2008.Visualizing data using t-sne. Journal of machinelearning research, 9(11).

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NIPS.

Mitchell Wortsman, Gabriel Ilharco, Mike Li,Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi,Hongseok Namkoong, and Ludwig Schmidt. 2021.Robust fine-tuning of zero-shot models. arXivpreprint arXiv:2109.01903.

Jianxiong Xiao, James Hays, Krista A Ehinger, AudeOliva, and Antonio Torralba. 2010. Sun database:Large-scale scene recognition from abbey to zoo.In 2010 IEEE computer society conference on com-puter vision and pattern recognition, pages 3485–3492. IEEE.

Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu,Tat-Seng Chua, and Maosong Sun. 2021. Cpt: Col-orful prompt tuning for pre-trained vision-languagemodels. arXiv preprint arXiv:2109.11797.

Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, andQi Tian. 2019. Deep modular co-attention networksfor visual question answering. In CVPR.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, andZiwei Liu. 2021. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134.

Documents

CLIP-Adapter: Better Vision-Language Models with Feature