C.BYREGOWDA INSTITUTE OF TECHNOLOGY, KOLAR-563101

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

Technical Seminar on

“ TagSense: Approach to automatic image tagging” Under the guidance of Presented by, Mr. Raja A Usha V N Asst. Professor 1CK10CS049 Dept.of CSE, CBIT, Kolar

ContentsBig PictureMotivationTagSenseProblem SpaceSystem ArchitectureDesign and ImplementationPerformance EvaluationLimitations/Future workConclusions

1

Big Picture: Application category

Built-in sensors in Smartphone : Accelerometer, Compass, Light sensor, Camera(Back-illuminated sensor), Microphone, GPS, Gyroscope, Gesture sensor, Barometer, Hygrometer, Thermometer, Magnetometer …

2

Motivation

Digital pictures are undergoing an explosionImage retrieval becomes crucial and they use tagsHuman tagging is accurate but slowImage based auto-tagging still has many constraintsHow to approximate the human tagging ability?

3

Smartphone the wildcardToday's Smartphone have powerful built-in sensorsPeople always carry their phones

4

Existing problem of auto-taggingAutomatic image tagging has improved by research in image processing and face recognition But,

cannot recognize individuals moving fastcan only identify individuals who have well-defined facial features

Picasa iphoto

5

TagSenseMain points of new automatic image tagging system

Better than image processing/face recognitionCreates tag including the people, activity and context in a picture

Tagsense: A Smartphone-based Approach to Automatic Image Tagging Leverages multiple sensing domains of Smartphone“Tag” Definition : keywords that describe the on-going scenario/event/occasion during which the picture was taken“Tag” Format :when-where-who-what 6

Problem Space

Sensing multiple dimensionsaccelerometer, compass, light sensor, camera, microphone, GPS, gyroscope

Basis for Comparison with iPhoto and Picasa good under bad lighting conditions

Because it does not depend on the physical features of a person’s face

TagSense generated the following tags :

November 21st afternoon, Nasher Museum, indoor, Romit, Sushma, Naveen, Souvik, Justin, Vijay, Xuan, standing, talking

7

System Architecture

People enter a common password of TagSense in respective phonesThis password acts as a shared session key, ensuring that sensed information is assimilated only from group members.

8

Example Scenario

Bob’s phone immediately broadcasts an active-sensor bacon, encrypted with the shared keyPhones in the group activate their respective sensorsOnce Bob clicks the picture, Bob’s camera sends a beacon with its local times-tamp and the phones record it 9

Example Scenario (contd…)

After a threshold time from the click, the phones deactivate their sensors, perform basic activity recognition on the sensed information, and send them back to Bob’s phone

Bob’s phone assimilates these per-person activities, and also infers some contextual information from its own sensors

10

PowerPoint TemplateSubtitle color

Example of a slide with a subheadSet the slide title in “title case”Set subheads in “sentence case”Generally set subhead to 36pt or smaller so it will fit on a single lineThe subhead color is defined for this template but must be selected. In PowerPoint 2007, it is the fourth font color from the left

Tag GenerationTag Generation

11

Design & ImplementationWho are in the picture?- includes only those in camera view 3 possible techniques enabled by multi-dimensional sensing

Accelerometer based motion signaturesComplementary compass directionsCorrelating visual and acceleration

12

Accelerometer based motion signature

People move into a specific posture during picture-click

Accelerometer based signature 13

Accelerometer based motion signature (contd..)

People inside the picture

The variance of accelerometer readings From 20pictures at different times and Locations

people outside the picture

picture 14

Complementary compass directionsPeople behave naturally when the picture is being taken

Complementary compass directions15

Complementary compass directions (contd..)

People in picture likely face cameraPersonal Compass Offset (PCO)

Use posing picture to calibrate PCO

16

Correlating visual and acceleration

People move actively like playing ping-ping, dancing, running

Correlating visual and acceleration17

Moving Subjects

TagSense matches the optical velocity with each of the phone’s accelerometer reading to identify the moving subjectsBasic idea1. Taking multiple snapshots from the camera2. Deriving the subject’s motion vector from these snapshots3. Correlating it to the accelerometer measurements recorded by different phone

18

Moving Subjects (contd..)

Extracting motion vectors of people from two successive snapshotsThe optical flow field showing the velocity of each pixelThe motion vectors form the two detected moving objects 19

Combining the oppurtunity

20

What are they doing ?Activity recognition with the aid of mobile phones has been an active area of research lately.Ex: SoundSense, Sensing Meets Mobile Social Networks The focus of this paper not on devising new activity recognition schemesSo, they start with a limited vocabulary of tags to represent a basic set of activities.

21

What are they doing (Contd..)Usage of Accelerometer Standing, Sitting, Walking, Jumping, Biking, Playing Clear signature from accelerometer Sitting Or Standing Accelerometer readings & location information walking, jumping, biking, playing

22

What are they doing (Contd..)Usage of Acoustic : Talking, Music, SilencePhoto + Audio Sample From acoustic sensor Easier to differentiate between two cases In TagSense prototype, it provide basic information regarding ambient sound when the picture is taken

23

Where is the picture takenLocation of a picture conveys semantic information about the picture It also enables location based photo search.GPS based location coordinates are suitable for these purposes.TagSense leverages mobile phone sensors and cloud services to approach these goals TagSense utilizes the light sensor on the camera phone to detect indoor/outdoor

24

Where is the picture taken (Contd..)The variation of light intensity measured at 400 different times across days and nights in outdoor and indoor environments.Feasible to compute light intensity thresholds Using the light intensity measurement (from the camera) during the picture-click And uses this information to tag the picture as “indoors” or “outdoors”.

25

Where is the picture taken (Contd..)Location + Phone Compasses combinationTo tag the backgrounds

California beach + Westward = Infer the ocean in the background

26

When is the picture taken?

27

AdvantagesEnvisioning an alternative opportunity towards automatic image tagging.Designing TagSense, an architecture for coordinating the mobile phone sensors, and processing the sensed information to tag images.

28

Limitations

TagSense does not generate captions and cannot tag pictures taken in the past.TagSense requires users to input a group password at the beginning of a photo session.Tag Sense vocabulary of tags is quite limited

29

Future Work

Combine with facial recognition, robust systemVideo-taggingAugmented Reality

30

ConclusionsTagSense leverages trend to automatically tag pictures with people and their activities.Mobile phones are Replacing traditional cameras. TagSense has somewhat lower precision and comparable fall-out but significantly higher recall than iPhoto/PicasaLimited vocabulary of tags to represent a basic set of activities like what they are doing.GPS-based location coordinates are used to tell where the picture is taken

31

References[1] “TagSense: Leveraging Smartphones for Automatic Image

Tagging”, IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 13, NO. 1, JANUARY 2014

[2] H. Lu and et al, “SoundSense: scalable sound sensing for people centric applications on mobile phones,” in ACM MobiSys, 2009.

[3] A. Engstrom and et al., “Mobile collaborative live video mixing,”Mobile Multimedia Workshop (with MobileHCI), Sep 2008.

[4] M. Azizyan and et al., “Surround Sense: mobile phone localization via ambience fingerprinting,” in ACM MobiCom, 2009.

32

PowerPoint TemplateSubtitle color

Example of a slide with a subheadSet the slide title in “title case”Set subheads in “sentence case”Generally set subhead to 36pt or smaller so it will fit on a single lineThe subhead color is defined for this template but must be selected. In PowerPoint 2007, it is the fourth font color from the left

PowerPoint Template

Example of a slide with a subheadSet the slide title in “title case”Set subheads in “sentence case”Generally set subhead to 36pt or smaller so it will fit on a single lineThe subhead color is defined for this template but must be selected. In PowerPoint 2007, it is the fourth font color from the left