Discovering the digital identity through face recognition on mobile devices

7/31/2019 Discovering the digital identity through face recognition on mobile devices

1/6

Discovering digital identities through face recognition on

mobile devices

Gerry Hendrickx

Faculty of Engineering, Department of Computer Science

Katholieke Universiteit Leuven

[email protected]

Abstract

This paper covers the current status and past work

of the creation of an iPhone application. The applica-

tion uses face recognition provided by Face.com [1] torecognize faces captured with the camera of the iPhone.

The recognized faces will be named and enable the user

to get access to the information of the person from dif-

ferent social networks, his digital identity. This paper

describes the research, the related work, the user in-

terface prototyping and the current state of the imple-

mentation. It covers 3 iterations of paper prototyping

in which a user interface gets established and the initial

implementation of the application.

1. INTRODUCTION

The internet has revolutionized the way people

interact with each other. Different social networks have

been created to support different levels of communica-

tions. The presence of a person on these networks is

called his digital identity. The goal of this thesis is to

find an efficient way to discover the digital identity of

a person, by using face recognition and a smartphone.

This paper describes the process of creating a face

recognition application for the iPhone. It uses face

recognition to offer the user extra information about

the persons seen through the camera. This extra in-

formation will come from various social networks, the

most important being Facebook, Twitter and Google+.

It aims to offer users access to online data and informa-

tion that is publicly available on the internet. This can

be used in a private context, to enhance conversations

and find common ground with your discussion partner,

or in an academic context, to be able to easily find

information, like slides or publications of the speaker

at an event youre attending. The app will be created

for iOS because the SDK offers a built in face detection

mechanism since iOS5 [2]. Android, the other option,

does not have this built in feature.

A brainstorm and a survey resulted in a list of re-

quirements for the face recognition application. The

survey asked about which information users would like

to get if they were able to recognize a person by using

a smartphone. The survey was answered by 34 persons.

14 out of 34 voters would respect the privacy of the rec-

ognized person and thus would not want any informa-

tion. 9 wanted contact information, 6 wanted links to

the social network profiles of the recognized person and

3 of the voters wanted the last status update of Facebook

or Twitter. The 2 remaining votes went to pictures of the

recognized person and the location where they last met.

There was a strong need for privacy, so a policy was de-cided upon: The app could be limited to the recognition

of the users Facebook friends, but the need to recog-

nize your Facebook friends is lower than the need to

recognize strangers. To broaden the scope of recogniz-

able people, the other users of the application will also

be recognizable. The general policy will be: if you use

the application, you can be recognized. An eye for an

eye.

The brainstorm resulted in a list of functionalities or

characteristics. First of all the application should work

fast. Holding your smartphone up and pointed to an-

other person is quite cumbersome, so the faster the

recognition works, the smaller this problem becomes.

The information about the person should be displayed

in augmented reality (AR) [3]. This is a technology that

augments a users view of the world. AR can add extra

information to the screen, based on GPS data or image

processing. AR could place information about a person

around his face in real-time. The functionality require-

ments from the poll and brainstorm and thus the goals to

achieve efficient discovery of digital identities by using

face recognition are the following:


2/6

Detection and recognition of all the faces in the

field of view.

Once recognized, the name and other options (like

available social networks) will appear on screen

with the face.

Contact info will be fetched from the networks and

can be saved in the contacts app.

Quick access to all social networks will be avail-

able, along with some basic information such as

the last status update/tweet.

Information available will differ from person to

person. To add an extra layer of privacy settings, a user

will be able to link his networks and choose to enable or

disable them. When the user gets recognized, only his

enabled networks will show up.

2. RELATED WORK

For this masterthesis we went to search for related

mobile applications. No existing application was found

that does exactly the same as this project. However,

some similarities were found with the following appli-

cations:

Viewdle [5]: Viewdle is a company focusing on

face recognition. They have several projects on-

going, and have already created an application for

Android, called Viewdle Social Camera. It recog-nizes faces in images based on Facebook. Viewdle

has a face recognition iOS SDK available.

Animetrics [6]: Animetrics created applications to

help government and law enforcing agencies. It

has multiple products like FaceR MobileID, which

can be used to get the names and percentage of the

match of any random person. FaceR CredentialME

can be used for authentication on your own smart-

phone. It recognizes your face and if it matches,

it unlocks your data. Animetrics also focuses on

home security face recognition. However they donot seem to have a public API, since their focus is

not on the commercial market.

TAT Augmented ID [7]: TAT Augmented ID is a

concept app. It recognizes faces in real-time and

uses augmented reality to display icons around the

face. The concept is the same, but the resulting

user interface is different. Section 3 discusses why

a fully augmented user interface is not preferred on

mobile devices.

Another non-commercial related work is a mas-

terthesis of 2010 at the Katholieke Universiteit Leuven

[10]. The author created a head-mounted-display-based

application to recognize faces and get information.

From his work we learned that HMDs are not the ideal

practical setup (his app required a backpack with a lap-

top and heavy headgear), and that the technology used(OpenGL) is a cumbersome way to develop. Using iOS

simplifies these aspects. The author used Face.com as

face recognition API and was very satisfied with it.

A comparison of face recognition APIs was made

in order to find the one most suited to our goals. A quick

summary of the positive and negative points:

Viewdle: As said above, weve tried to contact

Viewdle to get more information about the API.

Sadly, they did not respond, therefore Viewdle is

no option.

Face.com: Face.com offers a well documented

REST API. It offers Facebook and Twitter integra-

tion and a private namespace that will allow the

application to apply the previously stated privacy

policy . There is a rate limit on the free version of

Face.com.

Betaface [8]: The only API to work with images

and video. However, it is Windows only, hasnt

been used with iOS yet and is not free.

PittPatt[9]: PittPatt was a very promising service,

but sadly it got acquired by Google. The servicecannot be used at this time.

It seemed that Face.com is the only, but the best

option found. It has an iOS SDK and social media inte-

gration, which are both very useful in the context of this

application.

3. PAPER PROTOTYPING

Paper prototyping is the process of designing the

user interface based on quick drawings of all the differ-

ent parts of the user interface [11]. By doing this withpaper parts, it becomes easy to quickly evaluate and

adapt the interface. The prototyping phase consisted of

3 iterations: the interface decision, the interface evalua-

tion and the expert opinion.

3.1. Phase one: interface decisions

The first phase of the paper prototyping was to de-

cide which of the 3 possible interfaces would be used.

The interfaces were:


3/6

Interface 1: A box of information attached to the

head of the recognized person. This is the best use

of augmented reality, but you have to keep your

phone pointed to the person in order to be able to

read his information. See figure 1a.

Interface 2: A sidebar with information, whichtakes about 1/4th of the screen. This way, users

lower their phone when a person is selected, but

they can still use the camera if they want. See fig-

ure 1b.

Interface 3: A full screen information screen. This

makes minimal use of augmented reality but offers

a practical way to view the information. Users see

the name of the recognized person in the camera

view, and once tapped, they get referred to the full

screen information window. See figure 1c.

These interfaces are evaluated using 11 test sub-

jects, aged 18 to 23, with mixed smartphone experience.

The tests were done using the think aloud technique,

which means they have to say what they think is going

to happen when they click a button. The interviewer

plays computer and changes the screens. The same sim-

ple scenario was given for all interfaces where the test

subject needed to recognize a person and find informa-

tion about him. After the test, a small custom-made

questionnaire was given to poll the interface preference.

None of the users picked interface 1 as their favourite.

The fact that you should keep your phone pointed to the

person in order to read and browse through the infor-

mation, proved to be a big disadvantage. The choicebetween interface 2 and 3 was not unanimously de-

cided. 27% chose interface 2, 73% chose interface 3.

Thus interface 3 was chosen and elaborated for the sec-

ond phase of paper prototyping. People liked the sec-

ond idea where you could still see the camera, but then

again they realized that, if interface 1 thought us that

you would not keep your camera pointed at the crowd,

you wouldnt do this in interface 2, so the camera would

be showing your table or pants. This reduces the use of

the camera feed. The smartphone owners also pointed

out that using only a part of the screen would be too

small to actually put readable information on it.

3.2. Phase two: interface evaluation

For the second phase, 10 test subjects were used

between the ages of 20 and 23, again with mixed smart-

phone experience. An extended scenario was created

which explored the full scope of functionality in the ap-

plication. The testers needed to recognize people, ad-

just settings, link social networks to their profile, indi-

cate false positives and manage the recognition history.

(a) Interface 1

(b) Interface 2

(c) Interface 3

Figure 1: The different scenarios


4/6

The think aloud technique was applied once again. At

the end of each prototype test, the test subject needed to

fill in a USE questionnaire [4]. This is a questionnaire

consisting of 30 questions, divided into 4 categories to

poll to different aspect of the evaluated application. The

questions can be answered on a 7-point Likert rating

scale, 1 representing totally disagree and 7 representingtotally agree. The categories are usefulness, ease of use,

ease of learning and satisfaction.

3.2.1. Usefulness. The overall results were good. The

question about whether the application is useful re-

ceives a median score of 5 out of 7, with no result below

4. People seem to understand why the app is useful and

it does what they would expect. The scores on whether

it meets the need of the users were divided with scores

going from 1 to 7, because some users (especially the

users with some privacy concerns or without a smart-

phone) did not see the usefulness of the application.However, the target audience, the smartphone users, did

see its usefulness, resulting in higher scores in this sec-

tion.

3.2.2. Ease of use. From the ease of use questions the

need to add or rearrange buttons became clear. Users

complained about the number of screen transitions it

took to get from one screen to somewhere else in the

application and this could be seen in the results. The

question whether using the application is effortless re-

ceived scores ranging from 2 to 7 with a median of 5.

The path through the application should be revised to

see whether missing link can be found and corrected toimprove the navigation. For instance, the home button

to go to the main screen will be added on several other

screens, instead of having to navigate back through all

the previous screens. Using the application was effort-

less for all iPhone users, because the user interface was

built using the standard iOS interface parts.

3.2.3. Ease of learning. None of the ease of learning

questions scored below 5 on the 7-point Likert rating

scale. This is also due to the standard iOS interface,

which is developed by Apple to be easy to work with

and easy to learn.

3.2.4. Satisfaction. Most people were satisfied with

the functionality offered by the application and how

it was presented in the user interface. Especially the

iPhone users were very enthusiastic, calling it an inno-

vative, futuristic application. The question whether the

user was satisfied with the application received scores

ranging from 5 to 6, with a median at 6 out of 7. Non-

smartphone users were more skeptical and did not see

the need for such an application. Aside from this, the

application was fun to use and the target audience was

satisfied.

3.2.5. Positive and negative aspects. The users were

asked to give the positive and negative aspects of the

application. The positive aspects were the iOS-style of

working and the functionality and concept of the appli-cation. The negative aspects were more user interface

related, such as not enough home buttons, and the sug-

gested method to indicate a false positive. This button

was placed on the full screen information window of a

person. Everybody agreed that this was to late, because

all the data of the wrongfully tagged person would then

be displayed. So the incorrect tag-button should be

placed on the camera view. Some useful ideas like en-

abling the user to follow a recognized person on Twitter

were suggested.

3.3. Phase three: expert opinions

For this phase, the supervisor and 6 advisers of the

thesis were used in the paper prototype test. The proto-

type was adjusted to the results of the second iteration.

More home-buttons were placed and the incorrect tag-

button was placed on the camera view. The test sub-

jects all had extensive experience in the field of human-

computer interaction and can thus be seen as experts.

They took the tests and filled in the same questionnaire

and gave their opinion on several aspects of the pro-

gram. A small summary:

There were concerns about the image quality ofthe different iPhones. Tests should be done to test

from what distance a person can be recognized.

The application should be modular enough. In a

rapidly evolving 2.0 world, social network may

need to be added or deleted from the application. If

all the networks are implemented as modules, this

will be a simpler task.

The incorrect tag-button could be implemented in

the same way as the iPhoto application asks the

user if an image is tagged correctly.

The social network information should not just be

static info. The user should be able to interact di-

rectly from the application. If this is not possible,

it would be better to refer the user directly to the

Facebook or Twitter app.

More info could be displayed in the full screen in-

formation window. Instead of showing links to all

networks, the general information about the person

could already be displayed there.


5/6

When asked which social networks they would like

to see in the application, nearly everybody said Face-

book, Twitter and Google+. In an academic context,

they would like to see Mendeley and Slideshare.

4. IMPLEMENTATION

Apart from some suggestions, the third paper

prototype was received positively. The next step in

the process is the implementation. The application is

currently in development, and a small base is working.

The main focus so far is on the crucial functionality,

the face recognition. It is important to get this part

up and running as fast as possible, because the entire

application depends on it. So far the application is

able to track faces using the iOS5 face detection. A

temporary box frames the faces and follows them asthe camera or person moves. This functionality could

be used to test the quality of the face detection API. As

you can see in figure 2, the face detection algorithm

of iOS5 can detect multiple faces at ones, and in such

depth that the smallest recognized face is barely bigger

than a button. These are great results, because the

algorithm appeared to be fast and reliable and detailed

enough for our purpose. Detection of even smaller

faces is not necessary, because the boxes will become

harder to click if they are smaller than a button.

The next step was the face recognition. This is han-

dled by Face.com. An iOS SDK is available on the web-

site [12]. This SDK contains the functionality to send

images to the Face.com servers, and receive a JSON re-

sponse with the recognized faces. It also covers the nec-

essary Facebook login, as Face.com requires the user to

log in using his Facebook account. This login is only

needed one time. One problem was that Face.com only

accepts images, not video. To be able to test the face

recognition as fast as possible, a recognize-button was

added to the camera view. Once clicked, a snapshot

is taken with the camera. This snapshot is send to the

servers of Face.com and analyzed for different faces.

The JSON response gets parsed and the percentage of

the match and the list of best matches can be fetched

from it. At the moment, only one face can be recog-

nized at the same time, because there is no algorithm

provided to check which part of the response should be

matched to which face on the camera. This is temporar-

ily solved by limiting the program to one face at a time.

Figure 3 shows the current status of the application. A

face is recognized and its Facebook ID is printed above.

Figure 2: Face tracking results.

Figure 3: Face recognized and matched with the correct

Facebook ID.

5. NEXT STEPS AND FUTURE WORK

The next step in development is the further de-

velopment of the user interface. Now that we have


6/6

a basic implementation of the main functionality, it

is important to finish the mock-up of the application.

This way, a dummy implementation of several screens

can be used to test the interface in several iterations

using the digital prototype. When these tests happen,

the underlying functionality can be extended and

implemented in parallel.

Several big problems need to be solved. The

biggest being the matching of faces detected by iOS5

face detection and the faces recognized by Face.com.

Because Face.com recognizes faces by using images,

a way needs to be found to match these results to the

faces on screen. If the user moves the camera to other

people after pressing the recognize-button, the results

from Face.com will not match the faces on screen.

The solution in mind is to use an algorithm to match

the Face.com results with the face detection based on

proportions. If we succeed in finding a correlationbetween for instance the eyes and the nose of a person

in both services, it should be possible to find which

detected face matches which Face.com result. Another

way to match the faces to the reply is to keep track

of the faces based on their coordinates on the screen.

A suitable algorithm for this problem needs to be found.

Another smaller problem is the use of the Face.com

SDK. It has a limited Facebook graph API built into it.

However, this API can not be used to fetch the name

of an ID or to get status updates. Therefore the real

Facebook iOS SDK should be used. To prevent the appfrom working with two separate APIs, the Face.com

API needs to be adapted so that it uses the real Facebook

iOS SDK instead of the limited graph API.

6. CONCLUSION

This masterthesis is still a work in progress. We al-

ready have good results from paper prototyping, and the

core of the application has already been implemented.

In the following months, some problems will have to

be solved and user testing is still required to make the

application match its goal: a fast, new way to discoverpeople using the newest technologies and networks.

References

[1] http://www.face.com

[2] https://developer.apple.com/library/mac/documentation/

CoreImage/Reference/CIDetector Ref/Reference/Reference.html

[3] R. Azuma, Y. Baillot, R. Behreinger, S. Feiner, S.

Julier, B. MacInture, Recent developments in augmented

reality, IEEE Computer Graphics And Applications,

21(6):34-47, 2001

[4] Arnold M. Lund, Measuring Usability with

the USE Questionnaire, in Usability and User

Experience, vol. 8, no. 2, October 2001,

http://www.stcsig.org/usability/newsletter/0110

measuring with use.html[5] http://www.viewdle.com/

[6] http://www.animetrics.com/Products/FACER.php

[7] http://www.youtube.com/watch?v=tb0pMeg1UN0

[8] http://www.betaface.com/

[9] http://www.pittpatt.com/

[10] Niels Buekers, Social Annotations in Augmented Real-

ity, Masterthesis at KULeuven, 2010-2011

[11] Erik Duval, paper prototyping,

http://www.slideshare.net/erik.duval/paper-prototyping-

12082416, last checked on April 29, 2012

[12] Sergiomtz Losa, FaceWrapper for iPhone,

https://github.com/sergiomtzlosa/faceWrapper-iphone

Documents

Discovering the digital identity through face recognition on mobile devices