The Quest for Video Understanding

Xiaofeng Ren

Institute of Data Science and Technologies (iDST), Alibaba Group

Alibaba

2

Alibaba Ecosystem

3CLOUD COMPUTING OPERATING SYSTEM

DATATECHNOLOGY TEAM

Online and Mobile Commerce Platform

$BUYERSSELLERS

Marketing

Financing & Payment Solution

ProfessionalService Providers

DigitalEntertainment

Logistics

Mobile Browser

Location-BasedSocial MediaWeibo

Alibaba Mission

4CLOUD COMPUTING OPERATING SYSTEM

DATATECHNOLOGY TEAM

Online and Mobile Commerce Platform

$BUYERSSELLERS

Marketing

Financing & Payment Solution

ProfessionalService Providers

DigitalEntertainment

Logistics

Mobile Browser

Location-BasedSocial MediaWeibo

make it easy to do business anywhere

(E-)(M-)(O-) Commerce

5

Text Image Video Interactive Media

Taobao @ 2009

7

Taobao @ 2017

8

Taobao Search: Text Text

9

Title：茜舞2017春装新品韩版修身公

主裙红色连衣裙无袖打底背心裙春秋

Q：无袖蓬蓬裙连衣裙夏 a型

Title：美特斯邦威卫衣女士春装字母拼色针织套头衫

Q：百搭灰色套头无帽卫衣女

Taobao Search: Text Text

10

Title：茜舞2017春装新品韩版修身公

主裙红色连衣裙无袖打底背心裙春秋

Q：无袖蓬蓬裙连衣裙夏 a型

Title：美特斯邦威卫衣女士春装字母拼色针织套头衫

Q：百搭灰色套头无帽卫衣女

Taobao Search: Text Image

11

Title：茜舞2017春装新品韩版修身公

主裙红色连衣裙无袖打底背心裙春秋

Q：无袖蓬蓬裙连衣裙夏 a型

• Joint text and image embedding for description• Leverage user click data to learn query-description similarity

Taobao Search: Image Image

12

3 billion images over 100 million products, 12 million daily active users

Videos on Taobao

13

Videos on Taobao

14

• Promotional video• Video guide on how to use product• Documentary on how a product is made

… …

Live Streaming on Taobao

15

• Many top-selling stores have live streaming • Allow user interaction (chat, FAQ)• Conversion rate much higher than normal

Videos on Taobao

16

• Short videos are abundant on Xianyu (second-hand trading platform)• Demonstrations of products; sometimes people showing skills• Much richer information than text descriptions

Videos on Taobao

17

• Authoring tools• Video enhancements• Content analysis and tagging• Video-to-text matching

… …

AR/VR on Taobao

18

Youku & Tudou: “Youtube-like”

19

Founded 2006

Founded 2005

IPO 2010

IPO 2011

Merged

2012Joined Alibaba

2016

Youku: Video Hosting

20

30M paid members

~400M active users

700-min per month

Youku: Product Identification

21

Youku: Product Identification

22

Youku: Product Placement

23

Tudou: UPGC Sharing

24

Tudou: UPGC Sharing

25

Copyright protection

Authoring tools

Search & recommendation

… …

Cainiao Logistics

27

24 hours in China

72 hours globally

1B packages on 11/11Mission:

Robots @ Cainiao

Delivery Robots Delivery Drones

Picking Robots

Sample Video Applications

30

DigitalEntertainment

Logistics

CLOUD COMPUTING

Videos Everywhere

31CLOUD COMPUTING OPERATING SYSTEM

DATATECHNOLOGY TEAM

Online and Mobile Commerce Platform

$BUYERSSELLERS

Marketing

Financing & Payment Solution

ProfessionalService Providers

DigitalEntertainment

Logistics

Mobile Browser

Location-BasedSocial MediaWeibo

DAMO Academy (including iDST): $15B investment on R&D to solve “big problems”

Thank You!

iDST@Alibaba