Conditional Self-Supervised Learning for Few-Shot

Conditional Self-Supervised Learning for Few-Shot Classification

Yuexuan An1,2 , Hui Xue1,2∗ , Xingyu Zhao1,2 and Lu Zhang1,2

1School of Computer Science and Engineering, Southeast University, Nanjing, 210096, China2MOE Key Laboratory of Computer Network and Information Integration (Southeast University), China

{yx an, hxue, xyzhao, lu Zhang}@seu.edu.com

Abstract

How to learn a transferable feature representationfrom limited examples is a key challenge for few-shot classification. Self-supervision as an auxiliarytask to the main supervised few-shot task is con-sidered to be a conceivable way to solve the prob-lem since self-supervision can provide additionalstructural information easily ignored by the maintask. However, learning a good representation bytraditional self-supervised methods is usually de-pendent on large training samples. In few-shot sce-narios, due to the lack of sufficient samples, theseself-supervised methods might learn a biased rep-resentation, which more likely leads to the wrongguidance for the main tasks and finally causes theperformance degradation. In this paper, we pro-pose conditional self-supervised learning (CSS) touse prior knowledge to guide the representationlearning of self-supervised tasks. Specifically, CSSleverages inherent supervised information in la-beled data to shape and improve the learning fea-ture manifold of self-supervision without auxil-iary unlabeled data, so as to reduce representationbias and mine more effective semantic information.Moreover, CSS exploits more meaningful informa-tion through supervised learning and the improvedself-supervised learning respectively and integratesthe information into a unified distribution, whichcan further enrich and broaden the original repre-sentation. Extensive experiments demonstrate thatour proposed method without any fine-tuning canachieve a significant accuracy improvement on thefew-shot classification scenarios compared to thestate-of-the-art few-shot learning methods.

1 IntroductionDifferent from previous deep learning based methods that re-quire a large number of manually labeled training data, few-shot learning methods can mimic human to recognize newclasses with very few examples. The recent work of visual

∗Contact Author

few-shot learning can learn a transferable feature representa-tion from training a collection of tasks on base classes andgeneralize the representation to novel (unseen) classes withvery few examples [Chen et al., 2019b]. However, due to thedata scarcity, the obtained supervised information mainly fo-cuses on the differences of the base class samples and ignoresthe semantic information within samples valuable for novelclasses. Therefore, more semantic information for good rep-resentation from limited samples should be extracted for few-shot classification problems.

Self-supervised learning has emerged as an importanttraining paradigm for exploring a strong visual representa-tion, without relying on pre-defined annotations [He et al.,2020; Chen et al., 2020]. Usually, the self-supervised repre-sentation is learned from a pretext task by constructing cor-related views with augmentation operations (e.g., rotation orexemplar) on original data and making a distinction betweenaugmented views and the original one. Another method ofself-supervision adopts contrastive loss, which brings repre-sentations of different views of the same data closer (“positivepairs”), and spreads representations of views from differentdata (“negative pairs”) apart [He et al., 2020].

Recently, self-supervised learning considered to offer addi-tional semantic information is applied to the few-shot classifi-cation [Gidaris et al., 2019; Su et al., 2020; Lee et al., 2020].These self-supervision based few-shot learning methods useself-supervised tasks as auxiliary tasks and original few-shotclassification tasks as main tasks to jointly learn a same repre-sentation. However, self-supervision is usually dependent onthe availability of large training samples and unsuitable forfew-shot scenarios. The direct application of previous self-supervision learning tasks for few-shot scenarios might eas-ily fall into the learning of undesirable shortcuts [Noroozi andFavaro, 2016a], e.g., color histograms and edge continuity in-stead of key semantic information. Therefore, the learningof self-supervision might be biased, which might lead to thewrong guidance for the main classification tasks and finallycause the performance degradation.

In this paper, we propose conditional self-supervised learn-ing (CSS) that can be more resilient to few-shot classifi-cation. CSS treats supervised few-shot and self-supervisedtasks equally and learns two feature representations by thesetwo paradigms respectively. For self-supervised part, CSSleverages supervised information as a teacher to guide the

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)

2140

(a) Pre-training stage (b) Self-supervised training stage

(c) Meta-training stage

Figure 1: Architecture of conditional self-supervised learning for few-shot classification.

learning of self-supervision. Finally, all the information isintegrated into a unified distribution which can further en-rich and broaden the original representation. Therefore, theknowledge learned by CSS can infer other things from onefact and further improve the generalization performance. Itis worth noting that our approach is fundamentally differentfrom semi-supervised learning methods, and does not requireany auxiliary unlabeled data.

The contributions of our work are:• We explore a new pattern of self-supervision for few-

shot learning. CSS learns two different representationsby supervised and self-supervised learning respectivelyinstead of the previous shared representation pattern,and fuses these representations into a new and aug-mented one to further enrich and broaden the semanticrepresentation capacity for few-shot learning.• We design a novel conditional module applied to self-

supervised learning. The module can shape the learningfeature manifold of self-supervision and reduce repre-sentation bias. To the best of our knowledge, it is thefirst time to leverage supervised information to guide thetraining of self-supervision in few-shot learning.• We propose CSS with a 3-stage training pipeline: super-

vised pre-training stage, self-supervised training stageand meta-training stage. In the last stage, we design anovel knowledge distillation (FD) to reinforce the fusionof representations learning from the first two stages.• We perform adequate experiments to demonstrate that

our approach is totally superior to original and self-supervision-based few-shot classification methods. Sys-tematic studies by varying the combinations of differentstages are conducted, which verifies the effectiveness of

the proposed three-stage training method.

2 Related WorkFew-shot classification. Few-shot classification aims to ac-quire transferable visual representation by learning to rec-ognize unseen novel classes from few samples with abun-dant training on base classes [Chen et al., 2019b]. Manyefforts have been devoted to overcoming the data efficiencyissue and can be mainly divided into two categories: thegradient-based model and metric-based model. The gradient-based model [Andrychowicz et al., 2016; Finn et al., 2017;Rusu et al., 2019] can rapidly adapt a model to a given taskvia a small number of gradient update steps. MAML [Finnet al., 2017] is a representative as the gradient-based model.This method advocates learning a suitable initialization ofmodel parameters by learning from base classes, and trans-fers these parameters to novel classes in a few gradient steps.Metric-based model that leverages similarity information be-tween samples to identify novel classes with few samples[Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017;Sung et al., 2018]. Prototypical Network [Snell et al., 2017]is a representative of metric-based model. It takes the meanof support samples i.e., training samples of a class as its classprototype and classifies a query, i.e., test sample according tothe distance to each class prototype. In our work, we mainlyconsider the classification model combined with PrototypicalNetworks and cosine classifiers.

Self-supervised learning. Without the expensive manualannotations, self-supervision learns a feature representationby constructing pseudo-labels for annotation-free pretexttasks with only input signals and learning to predict them.


2141

The recent advances of self-supervised learning usually con-sist of two components: image generation and contrastivelearning. The former dedicates to design the pretext tasksto exploit compact semantic visual representation such as rel-ative rotation prediction [Chen et al., 2019a], jigsaw puzzles[Noroozi and Favaro, 2016b], relative patch location predic-tion [Doersch et al., 2015] and colorization [Zhang et al.,2016]. However, these pretext tasks rely on specific transfor-mations and heuristics and harm the generalization of learnedrepresentations [Chen et al., 2020]. Contrastive learning[Chen et al., 2020; Grill et al., 2020] currently achieves state-of-the-art performance in self-supervised learning. Con-trastive learning aims to learn the feature representation ofsamples by bringing the representations of positive pairscloser, and spreading representations of negative pairs apart.Wu et al. [Wu et al., 2018] adopts a memory bank to storethe instance class representation vector. Momentum Contrast(MoCo) [He et al., 2020] trains a visual representation en-coder by matching an encoded query q to a dictionary of en-coded keys with contrastive loss. SimCLR [Chen et al., 2020]uses the normalized temperature-scaled cross-entropy loss ascontrast loss. BYOL [Grill et al., 2020] only relies on positivepairs rather than negative pairs to learn the feature representa-tion. SimSiam [Chen and He, 2020] removes the momentumencoder and designs a simpler one based on BYOL. In this pa-per, we use the state-of-the-art SimSiam to extract semanticinformation of samples for simplicity and flexibility, which ismore resilient to few-shot classification.

Some researchers are devoted to using self-supervision tosqueeze out generalizable features and structural informationfrom low data regimes. [Gidaris et al., 2019] proposes amulti-task method combining self-supervised pretext as anauxiliary loss with the main few-shot loss. [Su et al., 2020]uses self-supervised learning as a regularizer and uses im-portance weights to select data for self-supervision. SLA[Lee et al., 2020] treats the pretext task as augmented labelsand avoids the optimization of the weighted summation lossbetween supervised and self-supervised tasks. These meth-ods directly apply self-supervision as an auxiliary task to themain few-shot task. However, due to limited samples, unde-sirable shortcuts might be learned by self-supervision, whichinevitably hurts the learning of key semantic information.

Different from the previous work, we use different featurerepresentations for different tasks. Firstly, a supervised repre-sentation is trained by the few-shot classification task. Then,we use the supervised representation as a teacher to guide theself-supervision to avoid exploiting shortcuts to solve self-supervised tasks. Finally, we learn a reinforced representationfrom the above representations with a novel fusing distillationmethod (FD). In that case, our method can benefit from thesedifferent paradigms respectively and learn a better semanticrepresentation.

3 Method3.1 PreliminaryFew-shot learning uses the episode training process and eachepisode samples a task. The meta train dataset and test datasetare represented by Dtr = {Ti}Ii=1 and Dte = {T ∗i }

I∗

i=1 re-

spectively, where T denotes the task. Each task instance con-tains a support set and a query set. The goal of each task is toestimate the class of samples in the query set with support set.For the meta train dataset Dtr = {Ti}Ii=1 = {Si, Qi}Ii=1, thesupport set Si is composed of N×K samples that N classesare randomly selected from all classes of Dtr and K samplesare extracted for each class. The query set Qi is composedof N×M samples that M samples for each class are un-seen in Si. The model is trained by randomly selecting tasksfrom the task set {Ti}Ii=1. Similar to Dtr, the test datasetDte = {T ∗i }

I∗

i=1 is considered as a task set, from which a newtask T ∗i = {S∗i , Q∗i } is sampled and the classes of S∗i and Q∗iare from Dte but unseen in Dtr.

3.2 MethodologyAs illustrated in Figure 1, the proposed CSS uses a 3-stagetraining pipeline. Firstly, in the pre-training stage, CSS learnsan initial feature extractor fθ (·) through the original super-vised learning method. In the next self-supervised trainingstage, CSS uses the learned fθ as a prior condition to op-timize the learning of self-supervision model gξ (·). In thefinal meta-training stage, CSS distills the knowledge of fθ (·)and gξ (·) learned in the first two stages into the final fea-ture embedding network hϕ (·) through a novel fusion distil-lation method. It is worth noting that in all the three trainingstages, only Dtr is needed, without introducing any auxiliarydatasets. In the remaining part of the section, we will describethese training stages in detail.

3.3 The Pre-Training StageIn the pre-training stage as shown in Figure 1(a), feature ex-tractor fθ (·) is learned with original few-shot classificationtasks. For N -way K-shot problem, in each training episode,a few-shot task Ti = {Si, Qi} is sampled. Given the featureextractor fθ (·), the prototype for each class k can be com-puted as

ck =1∣∣Ski ∣∣

∑(xt,yt)∈Sk

i

fθ(xt), (1)

where∣∣Ski ∣∣ denotes the number of support samples of the k-

th class. Then, given a new sample xq from Qi, the classifieroutputs the normalized classification score for each class k

pθ,ω (y = k |xq, Si ) =exp (simω (fθ (xq) , ck))∑k′ exp (simω (fθ (xq) , ck′))

,

(2)where simω (·, ·) is a similarity function parameterized by ω.In CSS, we consider the following similarity function:

simω (A,B) = λ cos (Fω (A) , Fω (B)) , (3)where Fω (·) is a single layer neural network parameterizedby ω and λ is the inverse temperature parameter. Then CSScan update θ and ω according to the following classificationloss:

Lpre (θ, ω) = E(xq,yq)∈Qi

[− log pθ,ω (y = yq |xq, Si )] . (4)


2142

3.4 The Self-Supervised Training StageThe core idea of the self-supervised training stage is to lever-age the knowledge learned from the supervised pre-trainingstage to reduce representation bias and improve the learningof self-supervised tasks. For the above consideration, we pro-pose the conditional self-supervision model, which consistsof a self-supervised module and a conditional module. Moredetails are illustrated in Figure 1(b).

In this stage, CSS trains the conditional self-supervisionmodel by using all the data (removing labels) in Dtr withoutauxiliary unlabeled data. CSS trains a self-supervised fea-ture extractor from scratch, rather than fine-tuning the fea-ture extractor trained in the pre-training stage, which enablesself-supervision to learn a diverse feature representation forfew-shot learning.

Self-supervised module. We consider SimSiam [Chen andHe, 2020] as a self-supervised task for simplicity and flexi-bility, other self-supervised methods are equally acceptable.Given a input image x, two augmented views x1 and x2 aregenerated according to random augment methods. The twoviews are processed by an encoder network consisting of theself-supervised feature extractor gξ (·) and a projection MLPhead σ. A prediction MLP head, denoted as δ, encode theoutput of one view and matches it to the other view. Rep-resenting the two output vectors as p1 , δ (σ (gξ (x1))) andz2 , σ (gξ (x2)), the negative cosine similarity of two em-bedding vectors is obtained:

D (p1, z2) = −p1‖p1‖2

· z2‖z2‖2

, (5)

where ‖·‖2 is l2-norm. Besides, a stop-gradient [Grill et al.,2020] operation is implemented on z2. Similarly, switchingthese two views and obtaining the distance between the twovectors, the symmetrized loss can be defined as:

Lself (x) = D (p1, z2) +D (p2, z1) . (6)

Conditional module. CSS takes the features learned in thepre-training stage as a prior condition to optimize the featuremanifold learned in the self-supervised module. CSS mini-mizes the negative cosine similarity between fθ(x) and gξ(x)as the condition of self-supervised training:

Lcond(x) = −fθ (x)

‖fθ (x)‖2· gξ (x)

‖gξ (x)‖2. (7)

Then, the final loss function combining condition loss withself-supervised loss can be obtained by:

Lcss (x) = Ex∈Dtr

[Lself (x) + γLcond(x)], (8)

where γ is a positive constant trading off the importance ofthe self-supervised and the conditional terms of the loss.

3.5 The Meta-Training StageIn the meta-training stage, shown in Figure 1(c), CSS dis-tills the knowledge learned in the first two stages with anovel fusion distillation (FD) to improve the quality of therepresentation. Specifically, the knowledge extracted from

fθ (·) and gξ (·) is adopted to augment the representations ofsamples with graph convolution network (GCN) [Kipf andWelling, 2017]. In particular, for the sample xi, CSS firstcalculates its two corresponding embedding vectors fθ (xi)and gξ (xi), and then uses an augmentation operation zi =C (fθ (xi) , gξ (xi)) (e.g. concatenation) to connect them. Bycalculating zi of different samples, we can obtain the corre-sponding feature matrix Z = [z1, · · · , zn]T of these samples,where n is the number of samples. After that, we can calcu-late the cosine similarity between two samples and generatea graph S, where each vertex represents feature of a sample

Si,j =

Zi,:Z

Tj,:

‖Zi,:‖2‖Zj,:‖2, i 6= j

0 , i = j

, (9)

where Si,j represents feature similarity between the i-th andj-th sample. Then CSS normalizes the graph matrix to obtainan adjacency matrix:

E = D−12SD−

12 , (10)

where D is the degree diagonal matrix and Dii =∑j Si,j .

Inspired by [Li and Liu, 2020], CSS takes E as the Lapla-cian matrix in GCN to aggregate information among vertices.Then for the sample xi, the embedding vector of xi with FDmethod can be obtained by

z∗i =∑n

j=1(αI + E)

νi,j hϕ (xj), (11)

where I is the identity matrix, ν is the number of times foraggregating feature, α is a trade-off parameter to trade off therepresentation of neighbors and itself one. (αI + E)

νi,j is the

j-th element of the i-th row of (αI + E)ν .

After that, for a support set Si and a query sample xq , theclassification score for class k can be obtained:

p∗ϕ,ω (y = k |xq, Si ) =exp

(simω

(z∗q ,Ck

))∑k′ exp

(simω

(z∗q ,Ck′

)) , (12)

whereCk =

1∣∣Ski ∣∣∑

(xt,yt)∈Ski

z∗t (13)

is the prototype of class k. Besides, the classification scorefor class k without FD can also be caculated:

pϕ,ω (y = k |xq, Si ) =exp (simω (hϕ (xq) ,Ck))∑k′ exp (simω (hϕ (xq) ,Ck′))

,

(14)where

Ck =1∣∣Ski ∣∣

∑(xt,yt)∈Sk

i

hϕ (xt). (15)

Then the final loss is given by:

Lmeta (ϕ, ω) = E(xq,yq)∈Qi

[− log p∗ϕ,ω (y = yq |xq, Si )

− η log pϕ,ω (y = yq |xq, Si )](16)


2143

where η is a positive constant trading off the importance ofthe first and the second terms of the loss.

When predicting, CSS uses hϕ as the feature extractor toextract the features of samples. Then CSS calculates the aver-age feature of the support set samples corresponding to eachclass as a prototype of the class. Then the similarity betweenthe query sample and the prototype of each class is calculatedby the similarity function simω (·, ·). Finally, the class withthe highest similarity score is obtained as the classificationresult.

4 ExperimentsIn this section, the effectiveness of CSS is verified by variousexperiments. Three standard few-shot classification datasets(CIFAR-FS [Bertinetto et al., 2019], CUB-200 [Wah et al.,2011], mini-ImageNet [Vinyals et al., 2016]) are selected tocompare the performance of our approach with previous few-shot learning methods. All the computations are performedon a GPU server with NVIDIA TITAN RTX, Intel Core i7-8700 CPU 3.20GHz processor and 32 GB memory.

4.1 Implementation DetailsFor the fair comparison, we implement all methods by Py-Torch. Since [Chen et al., 2019b] shows that the gap amongdifferent methods drastically reduces as the backbone getsdeeper, all algorithms use the Conv-4-64 [Vinyals et al.,2016] as the backbone and the feature embedding dimen-sion is set to 1600. In CSS, fθ, gξ and hϕ are implementby the Conv-4-64 backbone, and the output dimension of Fωis 2048. The projection MLP head σ is a composite func-tion, which is composed of Fω and a multi-layer neural net-work with {1600, 2048, 2048} units and batch normalizationin each layer. The prediction MLP head δ is parameterizedby a three-layer neural network with 512 hidden units andbatch normalization in the hidden layer. All hidden layers useReLU function [Glorot et al., 2011] as the activation function.

4.2 Comparison with Prior WorkIn order to verify the effectiveness of CSS, several competi-tive and state-of-the-art fully supervised few-shot classifica-tion methods are compared, including Baseline [Chen et al.,2019b], Baseline++ [Chen et al., 2019b], MAML [Finn etal., 2017], Matching networks [Vinyals et al., 2016], Pro-totypical networks [Snell et al., 2017], Relation networks[Sung et al., 2018], FEAT [Ye et al., 2020] and Deep KernelTransfer (DKT) [Patacchiola et al., 2020]. Besides, we alsocompare the performance of CSS with several state-of-the-art self-supervision-based few-shot learning methods. Proto-Transfer [Medina et al., 2020] pre-trains a representation onself-supervision and adapts it to original few-shot classifica-tion tasks. CC+rot, PN+rot, SLA and ProtoNet+rot [Gidariset al., 2019; Lee et al., 2020; Su et al., 2020] are multi-taskfew-shot learning and its varient methods.

We perform experiments to illustrate the average classifi-cation accuracies over 600 test episodes with 95% confidenceintervals of different methods on CIFAR-FS, CUB-200 andmini-ImageNet datasets under 5-way 5-shot and 1-shot set-tings respectively. For each setting, the best result is high-lighted in bold. CSS (pre-training), CSS(SSL-training) and

Mehods 5-shot 1-shotBaseline [Chen et al., 2019b] 68.43±0.73 47.01±0.78Baseline++ [Chen et al., 2019b] 72.85±0.72 51.55±0.80ProtoNet. [Snell et al., 2017] 68.78±0.77 43.65±0.86MatchingNet. [Vinyals et al., 2016] 68.93±0.74 51.32±0.85RelationNet. [Sung et al., 2018] 65.01±0.79 46.76±0.86MAML [Finn et al., 2017] 70.30±0.77 52.42±0.91FEAT [Ye et al., 2020] 69.39±0.77 50.69±0.86DKT [Patacchiola et al., 2020] 67.81±0.73 50.94±0.85ProtoTransfer [Medina et al., 2020] 67.95±0.76 42.46±0.77CC+rot [Gidaris et al., 2019] 69.75±0.76 52.02±0.87PN+rot [Gidaris et al., 2019] 70.52±0.72 48.53±0.88SLA [Lee et al., 2020] 68.62±0.75 45.94±0.87ProtoNet+rot [Su et al., 2020] 69.46±0.75 46.81±0.87CSS(pre-training) 70.64±0.72 51.07±0.89CSS(SSL-training) 69.96±0.76 51.48±0.89CSS(meta-training) 74.59±0.72 56.49±0.93

Table 1: Comparison with prior work on CIFAR-FS. Average 5-wayaccuracies (%) with 95% confidence intervals (600 episodes).


Table 2: Comparison with prior work on CUB-200. Average 5-wayaccuracies (%) with 95% confidence intervals (600 episodes).

CSS(meta-training) compose the 3-stage training pipelineproposed by us, which represent the classification model ofCSS using fθ, gξ and hϕ as feature extractors respectively.

From Tables 1, 2 and 3, we can find that in most ofthe few-shot classification settings, CSS already achieves al-most the same performance as state-of-the-art methods onlythrough the pre-training stage and self-supervised trainingstage, while the meta-learning stage can further improve theaccuracy of our model. In all cases, after the meta-trainingfinishes, we achieve state-of-the-art results surpassing pre-vious methods with a significant margin. For instance, onCIFAR-FS, CUB-200, and mini-ImageNet datasets, CSS hasaround 6%, 7% and 4% performance improvements com-pared with the vanilla prototypical networks under the 5-shotsetting, while it has 13%, 15% and 6% performance improve-ments under the 1-shot setting. At the same time, in all set-


2144


Table 3: Comparison with prior work on mini-ImageNet. Average5-way accuracies (%) with 95% confidence intervals (600 episodes).

Figure 2: Classification results in different cases (5-way 5-shot).

tings, our approach outperforms around 2% to 5% higher thanall previous leading methods.

4.3 Ablation StudyNow we explore the importance of the condition module inself-supervised learning and the effects of different stages.We design five cases respectively to investigate the perfor-mance when applying different combinations of stages. Then,we test the performance of these methods in the standard few-shot classification settings. To be brief, we use S1, S2 andS3 to represent CSS(pre-training), CSS(SSL-training) andCSS(meta-training) respectively.

• SSL: Discard S1 and S3 and replace S2 with a vanillaself-supervised learning without a conditional module.• SL+SSL+FD: Change S2 and abandon the conditional

module in our self-supervised learning method.• SL (S1): Only execute supervised learning, i.e., using fθ

in CSS as the feature extractor for classification.• CSS (S1+S2): execute the first two stages without S3,

i.e., using gξ in CSS as the feature extractor.• CSS (S1+S2+S3): Execute the whole three stages of

CSS, using hϕ as the feature extractor for classification.

Figure 3: Classification results in different cases (5-way 1-shot).

The classification performance in different cases are shownin Figures 2 and 3. From Figures 2 and 3, we find that the per-formance of vanilla self-supervised learning (SSL) degradessignificantly compared with other settings, which proves itdifficult to study an effective representation in the absenceof supervised learning with limited samples. The combina-tion of supervised learning, self-supervised learning and fu-sion distillation (SL+SSL+FD) can improve the classificationaccuracy compared with SSL. However, the performance ofSL+SSL+FD almost can not surpass the supervised learningmodel (SL) and the negative impact of SSL with limited sam-ples still continues. In addition, by comparing the resultsof CSS(S1+S2) with SSL and SL, we find that the condi-tional module can significantly improve the performance ofself-supervised learning, which can achieve comparable re-sults with the supervised model (SL). Furthermore, CSS withentire process can achieve an obvious performance improve-ment and surpass other cases with large margin. The experi-ments illustrate that the condition module plays a crucial rolein self-supervision and the effective feature fusion can furtherimprove the model performance.

5 ConclusionIn this work, we propose conditional self-supervised learn-ing (CSS) with a 3-stage training pipeline: the supervisedpre-training stage, the self-supervised training stage and themeta-training stage, and each training stage is conducive tothe improvements of the model performance. For the self-supervised training stage, CSS leverages the supervised infor-mation learned by the pre-training stage to guide the learningof self-supervision, thus more resilient to low data regimes.For the meta-train stage, CSS utilizes a novel fusion distilla-tion (FD) to integrate the information from the first two stagesinto a unified distribution so as to enrich and broaden the orig-inal representation. Detailed experiments reveal that our ap-proach indeed leads to significant performance improvementsover recent approaches and achieves state-of-the-art results.

AcknowledgmentsThis work was supported by National Natural Science Foun-dation of China (Grant No.62076062) and National Key R&DProgram of China (Grant No.2017YFB1002801). Further-more, the work was also supported by Collaborative Innova-tion Center of Wireless Communications Technology.


2145

References[Andrychowicz et al., 2016] Marcin Andrychowicz, Misha

Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman,David Pfau, Tom Schaul, and Nando de Freitas. Learningto learn by gradient descent by gradient descent. In NIPS,pages 3981–3989, 2016.

[Bertinetto et al., 2019] Luca Bertinetto, Joao F. Henriques,Philip H. S. Torr, and Andrea Vedaldi. Meta-learning withdifferentiable closed-form solvers. In ICLR, 2019.

[Chen and He, 2020] Xinlei Chen and Kaiming He. Ex-ploring simple siamese representation learning. CoRR,abs/2011.10566, 2020.

[Chen et al., 2019a] Ting Chen, Xiaohua Zhai, Marvin Rit-ter, Mario Lucic, and Neil Houlsby. Self-supervised gansvia auxiliary rotation loss. In CVPR, pages 12154–12163,2019.

[Chen et al., 2019b] Wei-Yu Chen, Yen-Cheng Liu, ZsoltKira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closerlook at few-shot classification. In ICLR, 2019.

[Chen et al., 2020] Ting Chen, Simon Kornblith, MohammadNorouzi, and Geoffrey E. Hinton. A simple framework forcontrastive learning of visual representations. In ICML, vol-ume 119, pages 1597–1607, 2020.

[Doersch et al., 2015] Carl Doersch, Abhinav Gupta, andAlexei A. Efros. Unsupervised visual representation learn-ing by context prediction. In ICCV, pages 1422–1430,2015.

[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and SergeyLevine. Model-agnostic meta-learning for fast adaptationof deep networks. In ICML, volume 70, pages 1126–1135,2017.

[Gidaris et al., 2019] Spyros Gidaris, Andrei Bursuc, NikosKomodakis, Patrick Perez, and Matthieu Cord. Boostingfew-shot visual learning with self-supervision. In ICCV,pages 8058–8067, 2019.

[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, andYoshua Bengio. Deep sparse rectifier neural networks. InAISTATS, volume 15, pages 315–323, 2011.

[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, FlorentAltche, Corentin Tallec, Pierre H. Richemond, ElenaBuchatskaya, Carl Doersch, Bernardo Avila Pires, Zhao-han Guo, Mohammad Gheshlaghi Azar, Bilal Piot, KorayKavukcuoglu, Remi Munos, and Michal Valko. Bootstrapyour own latent - A new approach to self-supervised learn-ing. In NeurIPS, 2020.

[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, SainingXie, and Ross B. Girshick. Momentum contrast for un-supervised visual representation learning. In CVPR, pages9726–9735, 2020.

[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling.Semi-supervised classification with graph convolutionalnetworks. In ICLR, 2017.

[Koch et al., 2015] Gregory Koch, Richard Zemel, and Rus-lan Salakhutdinov. Siamese neural networks for one-shotimage recognition. In ICML deep learning workshop, 2015.

[Lee et al., 2020] Hankook Lee, Sung Ju Hwang, and JinwooShin. Self-supervised label augmentation via input transfor-mations. In ICML, volume 119, pages 5714–5724, 2020.

[Li and Liu, 2020] Jianyi Li and Guizhong Liu. Few-shot im-age classification via contrastive self-supervised learning.CoRR, abs/2008.09942, 2020.

[Medina et al., 2020] Carlos Medina, Arnout Devos, andMatthias Grossglauser. Self-supervised prototypicaltransfer learning for few-shot classification. CoRR,abs/2006.11325, 2020.

[Noroozi and Favaro, 2016a] Mehdi Noroozi and PaoloFavaro. Unsupervised learning of visual representationsby solving jigsaw puzzles. In ECCV, volume 9910, pages69–84, 2016.

[Noroozi and Favaro, 2016b] Mehdi Noroozi and PaoloFavaro. Unsupervised learning of visual representationsby solving jigsaw puzzles. In ECCV, volume 9910, pages69–84, 2016.

[Patacchiola et al., 2020] Massimiliano Patacchiola, JackTurner, Elliot J. Crowley, Michael F. P. O’Boyle, andAmos J. Storkey. Bayesian meta-learning for the few-shotsetting via deep kernels. In NeurIPS, 2020.

[Rusu et al., 2019] Andrei A. Rusu, Dushyant Rao, JakubSygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin-dero, and Raia Hadsell. Meta-learning with latent embed-ding optimization. In ICLR, 2019.

[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard S.Zemel. Prototypical networks for few-shot learning. InNIPS, pages 4077–4087, 2017.

[Su et al., 2020] Jong-Chyi Su, Subhransu Maji, and BharathHariharan. When does self-supervision improve few-shotlearning? In ECCV, volume 12352, pages 645–666, 2020.

[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, TaoXiang, Philip H. S. Torr, and Timothy M. Hospedales.Learning to compare: Relation network for few-shot learn-ing. In CVPR, pages 1199–1208, 2018.

[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, TimLillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match-ing networks for one shot learning. In NIPS, pages 3630–3638, 2016.

[Wah et al., 2011] Catherine Wah, Steve Branson, PeterWelinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X. Yu,and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pages 3733–3742, 2018.

[Ye et al., 2020] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan,and Fei Sha. Few-shot learning via embedding adapta-tion with set-to-set functions. In CVPR, pages 8805–8814,2020.

[Zhang et al., 2016] Richard Zhang, Phillip Isola, andAlexei A. Efros. Colorful image colorization. In ECCV,volume 9907, pages 649–666, 2016.


2146

Documents

Conditional Self-Supervised Learning for Few-Shot