Embed Size (px)
Citation preview
![Page 1: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/1.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERTA Joint Model for Video and Language Representation Learning
Pascale Walters
CS 886University of Waterloo
March 9, 2020
P. Walters VideoBERT March 9, 2020
![Page 2: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/2.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Motivation
Good performance in computer vision tasks tends to require hugeamounts of labelled dataOnline platforms, such as YouTube, have huge amounts of unlabelledvideo dataCan we get high level representations of videos without the need formanual annotation?
P. Walters VideoBERT March 9, 2020
![Page 3: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/3.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Motivation
Humans can describe objects and events in a video at a high levelwith language
Natural source of self-supervisionUse BERT to learn a joint linguistic and visual representation
p(x, y), x is a series of visual words and y is a series of spoken wordsApplications: text-to-video prediction, dense video captioning,long-range forecasting
P. Walters VideoBERT March 9, 2020
![Page 4: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/4.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
BERT
BERT (bidirectional encoding representations from transformers) getsrepresentations of words from text by looking to the left and rightIt is pre-trained in an unsupervised manner with a large corpus of textand fine-tuned for specific tasksPre-training is done in two steps:
1 Masked language model2 Next sentence prediction
P. Walters VideoBERT March 9, 2020
![Page 5: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/5.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
VideoBERT
Extend BERT to videos by generating “visual words” with hierarchicalvector quantization
Can still use pretrained models and existing implementationsThe model focuses on high level semantics and long-range temporaldynamics
P. Walters VideoBERT March 9, 2020
![Page 6: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/6.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Combining Visual and Linguistic Sentences
Figure: VideoBERT token prediction (cloze task)
In VideoBERT, visual and linguistic sentences are combined with a specialtoken
[CLS] orange chicken with [MASK] sauce [>]
v01 [MASK] v08 v72 [SEP]
P. Walters VideoBERT March 9, 2020
![Page 7: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/7.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Combining Visual and Linguistic Sentences
Replace the next sentence prediction task from BERT with alinguistic-visual alignment task
Predict whether linguistic sentence is temporally aligned with visualsentence
P. Walters VideoBERT March 9, 2020
![Page 8: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/8.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Dataset
Instructional videos: spoken words are more likely to refer to thevisual contentCooking videos: existing annotated dataset for evaluation
YouCook II: 2K videos, 176 hoursTraining dataset collected from YouTube
312K videos, 23,186 hoursText extracted with YouTube’s automatic speech recognition toolkit
P. Walters VideoBERT March 9, 2020
![Page 9: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/9.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Video Tokenization
Just like with BERT, a tokenized representation of the input video isrequiredExtract representations for 30-frame clips with S3D pretrained onKinetics datasetTokenize the visual features with hierarchical k-means clusteringText is tokenized with WordPieces (as with BERT)To break the video into sentences,
Start and end timestamps of a text sentence, or16 tokens
P. Walters VideoBERT March 9, 2020
![Page 10: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/10.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Video Tokenization
Figure: Examples of frame and sentence pairs. The images on the left are framesextracted from the video and the images on the right are visual centroids.
P. Walters VideoBERT March 9, 2020
![Page 11: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/11.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Downstream Tasks
VideoBERT proposes two downstream tasks:1 Zero-shot classification2 Feature extraction for video captioning
P. Walters VideoBERT March 9, 2020
![Page 12: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/12.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Zero-Shot Classification
Zero-shot: test model with different data or labels that were used fortrainingWant to compute p(y | x), x is the sequence of visual tokens, y is afixed sentence (extract verb and noun):
now let me show you how to [MASK] the [MASK].
Model trained on YouTube dataset is tested on the YouCook IIdataset
VideoBERT achieves a comparable top-5 accuracy to a fully supervisedclassifier trained on YouCook II
P. Walters VideoBERT March 9, 2020
![Page 13: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/13.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Zero-Shot Classification
Figure: Performance of the VideoBERT model on the zero-shot classification tasktested on the YouCook II dataset.
P. Walters VideoBERT March 9, 2020
![Page 14: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/14.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Zero-Shot Classification
Figure: Examples of VideoBERT zero-shot classification.
P. Walters VideoBERT March 9, 2020
![Page 15: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/15.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Transfer Learning for Video Captioning
VideoBERT can be used to extract features from video input:
now let's [MASK] the [MASK] to the [MASK],
and then [MASK] the [MASK].
Use the output as input for a supervised downstream modelConcatenate average of video tokens and masked out text tokensModel is a transformer encoder-decoder that maps video segments tocaptions
Best performance is achieved on cross-modal pretraining
P. Walters VideoBERT March 9, 2020
![Page 16: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/16.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Transfer Learning for Video Captioning
Figure: Performance on the video captioning task on the YouCook II dataset. Allmodels are based on that of Zhou et al., but with varying inputs.
P. Walters VideoBERT March 9, 2020
![Page 17: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/17.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Transfer Learning for Video Captioning
Figure: Examples of VideoBERT captioning.
P. Walters VideoBERT March 9, 2020
![Page 18: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/18.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
VideoBERT
Effects of Dataset Size
Is there an effect on the size of the training dataset on theperformance of the model?Performance increases with increased training dataset size with noappearance of saturation
Figure: Performance of VideoBERT on the zero-shot classification task withincreasing dataset size.
P. Walters VideoBERT March 9, 2020
![Page 19: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/19.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
Limitations of Vector Quantization
The vector quantization in VideoBERT can lose fine-grained detailsthat may be critical for downstream tasksThe authors propose another method for encoding video sequencesfor training bidirectional transformer models
P. Walters VideoBERT March 9, 2020
![Page 20: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/20.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
Contrastive Bidirectional Transformer
CBT doesn’t need a pre-trained visual encoderUse softmax version of noise contrastive estimation loss in BERT,since images and videos have real-valued vectors as inputs instead ofa fixed discrete vocabularyBetter at utilizing long temporal context
P. Walters VideoBERT March 9, 2020
![Page 21: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/21.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
Contrastive Bidirectional Transformer
Try to maximize mutual information between the video and languagesequences at a sequence level, rather than at a frame level
P. Walters VideoBERT March 9, 2020
![Page 22: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/22.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
CBT Loss Function
Lcbt = wbertLbert + wvisualLvisual + wcrossLcross (1)
wbert is set to 0, since a pretrained BERT is usedwvisual is set to 1wcross is either 1 or 0
P. Walters VideoBERT March 9, 2020
![Page 23: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/23.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
CBT Loss Function
Softmax version of the noise contrastive estimation
Lvisual = −Ex∼D∑t
log NCE(xt|x−t) (2)
NCE(xt|x−t) =exp
(eTt et
)exp
(eTt et
)+∑
j∈neg(t) exp(eTj et)(3)
where et = fenc(xt) is the output of the S3D model applied to a smallwindow around frame t, et = gcontext(e−t) is the output of the visualtransformer, and neg(t) is a set of negative examples.
P. Walters VideoBERT March 9, 2020
![Page 24: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/24.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Contrastive Bidirectional Transformer
Performance on Video Captioning
Figure: The CBT method performs better than VideoBERT on video captioningand achieves state-of-the-art on the YouCookII dataset. The only differencebetween the methods is the video encoder.
P. Walters VideoBERT March 9, 2020
![Page 25: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/25.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
References
References
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert:Pre-training of deep bidirectional transformers for languageunderstanding. arXiv preprint arXiv:1810.04805 (2018).Sun, C., Baradel, F., Murphy, K. & Schmid, C. Learning Video
Representations using Contrastive Bidirectional Transformer. 2019.arXiv: 1906.05743 [cs.LG].Sun, C., Myers, A., Vondrick, C., Murphy, K. & Schmid, C.
Videobert: A joint model for video and language representationlearning. in Proceedings of the IEEE International Conference onComputer Vision (2019), 7464–7473.Zhou, L., Zhou, Y., Corso, J. J., Socher, R. & Xiong, C. End-to-End
Dense Video Captioning with Masked Transformer. in Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition(2018), 8739–8748.
P. Walters VideoBERT March 9, 2020
![Page 26: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/26.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Thank You
P. Walters VideoBERT March 9, 2020
![Page 27: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/27.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Dense Video Captioning
Video Captioning
End-to-end video captioningThe encoder encodes the video frames (features) into the properrepresentation. The proposal decoder then decodes this representationwith different anchors to form event proposals, i.e., start and end timeof the event, and a confidence score. The captioning decoder thendecodes the proposal specific representation using a masking network,which converts the event proposal into a differentiable mask.
P. Walters VideoBERT March 9, 2020
![Page 28: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/28.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Dense Video Captioning
Dense Video Captioning
Figure: Dense video captioning.
P. Walters VideoBERT March 9, 2020
![Page 29: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/29.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
S3D
S3D Feature Extraction
Separable 3D CNN (S3D) is a network that finds an optimal balancebetween complexity and accuracy for video classification
P. Walters VideoBERT March 9, 2020
![Page 30: VideoBERT - A Joint Model for Video and Language ...mli/Pascale.pdf · VideoBERT can be used to extract features from video input: now let's [MASK] the [MASK] to the [MASK], and then](https://reader034.pdfslide.us/reader034/viewer/2022050507/5f987d614c01711238724f96/html5/thumbnails/30.jpg)
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
S3D
S3D Feature Extraction
Figure: S3D architecture.
P. Walters VideoBERT March 9, 2020
![DS-9016HFI-S11].pdf · Advanced Features Advanced motion detection and video analytics, including private mask, tampering and video loss alerts pre/post-recording Of alarm and motion](https://img.pdfslide.us/doc/110x75/5f3d776fa7ecfd48bb143f51/ds-9016hfi-s1-1pdf-advanced-features-advanced-motion-detection-and-video-analytics.jpg)
