Mobile Biometrics: Towards A Comprehensive Evaluation ... · New generation devices: personal, connected and powerful! I Beyond classical communication I Taking pictures & making

Mobile Biometrics: Towards A ComprehensiveEvaluation Methodology

Attaullah Buriro1, Zahid Akhtar3, Bruno Crispo1,2,Sandeep Gupta1

1DISI University of Trento, Italy

2DistriNet - KULeuven, Belgium

3INRS-EMT, University of Quebec, Montreal, Canada

{attaullah.buriro, sandeep.gupta, bruno.crispo}@[email protected]

[email protected],

October 26, 2017

Outline

I Motivation

I Problem statement

I Guidelines/Recommendations

I Conclusion

New Generation Devices

I The UK is now a smartphone society1

I Computer usage falls as 20% of millennials go mobile-only2

1https://www.ofcom.org.uk/about-ofcom/latest/media/media-releases/2015/cmr-uk-2015

2https://www.extremetech.com/computing/226867-comscore-computer-usage-falls-as-20-of-millennials-go-

mobile-only.



I Computer usage falls as 20% of millennials go mobile-only2



mobile-only.



I Computer usage falls as 20% of millennials gomobile-only2



mobile-only.

New generation devices: personal, connected and powerful!

I Beyond classical communicationI Taking pictures & making movies and sharing with othersI Social Networking

I Facebook, Viber, whatsApp, Skype, Twitter, etc.I Online Transactions

I Google Wallet, Paypal, XOOM, etc.

I They continuously track the user location and have fullcontrol over user’s emails

I All of these apps generate and store very personal userinformation which needs to be protected

New generation devices: personal, connected and powerful!

I Beyond classical communicationI Taking pictures & making movies and sharing with othersI Social Networking

I Facebook, Viber, whatsApp, Skype, Twitter, etc.I Online Transactions

I Google Wallet, Paypal, XOOM, etc.

I They continuously track the user location and have full controlover user’s emails

I All of these apps generate and store very personal userinformation which needs to be protected

Authentication

I What is Authentication?I Being able to prove a user is who she claims to be

I Authentication was introduced for

I Protecting long term sessionsI One-shot - TOCTOU problemI Binary decision

I Authentication is required for

I Long, short & frequent sessionsI Repeatable (as and when required)I Risk-based and adaptiveI Continuous

Authentication


I Authentication was introduced for

I Protecting long term sessionsI One-shot - TOCTOU problemI Binary decision



Authentication


I Authentication was introduced forI Protecting long term sessionsI One-shot - TOCTOU problemI Binary decision



Authentication





Authentication



I Authentication is required forI Long, short & frequent sessionsI Repeatable (as and when required)I Risk-based and adaptiveI Continuous

Usage pattern shift

I Usability study show that users interact with theirsmartphones every 6.5 minutes (in 24 h).

I Passwords, PINs and physical biometrics do not fit the currentinteraction model for newer devices.

I Research has been diverted to design new acceptable & securemetaphors for user authentication.

I Evaluation is mainly based on security (under zero-effort) noton the other operational issues, i.e., usability, robustnessagainst attacks, and computational overhead.

I We present a set of guidelines for designing, implementation,and evaluating newer user authentication methods for apositive impact on future technological developments.

Usage pattern shift


I Passwords, PINs and physical biometrics do not fit thecurrent interaction model for newer devices.




Usage pattern shift



I Research has been diverted to design new acceptable &secure metaphors for user authentication.



Usage pattern shift




I Evaluation is mainly based on security (under zero-effort)not on the other operational issues, i.e., usability,robustness against attacks, and computational overhead.


Usage pattern shift





I We present a set of guidelines for designing,implementation, and evaluating newer userauthentication methods for a positive impact on futuretechnological developments.

Guidelines!

Data Collection Protocol

I Higher number of users (preferably from diversebackground) and samples are always better.

I Data should be collected anonymously or their data privacyshould be ensured.

I We recommend to collect data in a natural way, i.e., datashould be collected in multiple sessions so that the participantshould not be able to memorize the behavior.


I Higher number of users (preferably from diverse background)and samples are always better.

I Data should be collected anonymously or their dataprivacy should be ensured.

I We recommend to collect data in a natural way, i.e., datashould be collected in multiple sessions so that the participantshould not be able to memorize the behavior.


I Higher number of users (preferably from diverse background)and samples are always better.

I Data should be collected anonymously or their data privacyshould be ensured.

I We recommend to collect data in a natural way, i.e.,data should be collected in multiple sessions so that theparticipant should not be able to memorize the behavior.

Classification Protocol

I The mobile user authentication problem essentially is aone-class classification problem, it is thus unreasonableto formulate mobile user authentication as the binaryclass classification problem.

Classification Protocol

I The mobile user authentication problem essentially is aone-class classification problem, it is thus unreasonable toformulate mobile user authentication as the binary classclassification problem.

Cross-validation / Training-testing Protocol

I We consider classifier training with initial set ofobservations, e.g., first 5 or 10, more realistic ascompared to using a large fraction for the classifiertraining.

Cross-validation / Training-testing Protocol

I We consider classifier training with initial set of observations,e.g., first 5 or 10, more realistic as compared to using a largefraction for the classifier training.

Success Metric

I New mechanisms are normally evaluated in terms ofFalse Reject Rate (FRR), False Acceptance Rate (FAR),Equal Error Rate (EER).

I Failure to Acquire Rate (FTAR)

I Failure to Enroll Rate (FTER)

Success Metric

I New mechanisms are normally evaluated in terms of FalseReject Rate (FRR), False Acceptance Rate (FAR), Equal ErrorRate (EER).

I Failure to Acquire Rate (FTAR)

I Failure to Enroll Rate (FTER)

Usability Study

1. Sample Acquisition TimeI Biometric researchers should minimize the required

sample acquisition time in order to increase theacceptability of their proposed scheme.

2. Classifier’s Training/Testing Times

I Larger testing time could end up in annoying the user andwon’t get the wide user acceptability.

Figure: Time Consuming (15-20s)3

3http://www.ibtimes.co.uk/unlocking-phone-your-eyes-fujitsu-iris-recognition-tech-coming-smartphones-2015-

1490297

Usability Study

1. Sample Acquisition TimeI Biometric researchers should minimize the required sample

acquisition time in order to increase the acceptability of theirproposed scheme.

2. Classifier’s Training/Testing TimesI Larger testing time could end up in annoying the user

and won’t get the wide user acceptability.

Figure: Time Consuming (15-20s)3

3http://www.ibtimes.co.uk/unlocking-phone-your-eyes-fujitsu-iris-recognition-tech-coming-smartphones-2015-

1490297

Usability Study (2)

1. Applicability to all users of all age-groups

2. Applicability in different situations

I The newly proposed authentication scheme needs to beevaluated in multiple common activities in order to obtain aclear picture of their final accuracy.

Usability Study (2)

1. Applicability to all users of all age-groups2. Applicability in different situations

I The newly proposed authentication scheme needs to beevaluated in multiple common activities in order to obtain aclear picture of their final accuracy.

Usability Study (2)

1. Applicability to all users of all age-groups2. Applicability in different situations

I The newly proposed authentication scheme needs to beevaluated in multiple common activities in order toobtain a clear picture of their final accuracy.

Usability Study (3)

1. Roll of Hardware VariabilityI It would be worth investigating to evaluate the newly

proposed authentication scheme on different devicesand/or multiple models and reporting the resultsaccordingly.

2. Software Usability Scale (SUS)

I Research proposing new mobile biometric should also includeinitial usability evaluation to get an impression of useracceptability of their scheme.

19.58%

BestImaginable(x ≥ 92)

10.30%

Excellent(92 < x ≥ 85)

28.86%

Good(85 < x ≥ 72)

24.74%

OK(72 < x ≥ 52)

15.46%

Poor(52 < x ≥ 38)

1.06%Worst

Imaginable(38 < x ≥ 25)

Usability Study (3)

1. Roll of Hardware VariabilityI It would be worth investigating to evaluate the newly proposed

authentication scheme on different devices and/or multiplemodels and reporting the results accordingly.

2. Software Usability Scale (SUS)I Research proposing new mobile biometric should also

include initial usability evaluation to get an impression ofuser acceptability of their scheme.

19.58%

BestImaginable(x ≥ 92)

10.30%

Excellent(92 < x ≥ 85)

28.86%

Good(85 < x ≥ 72)

24.74%

OK(72 < x ≥ 52)

15.46%

Poor(52 < x ≥ 38)

1.06%Worst

Imaginable(38 < x ≥ 25)

Performance Analysis

1. Power OverheadI Any proposed mobile biometric recognition system must

not consume much power to be adopted in the real-worldapplications.

2. Computational Overhead

I It is strongly recommended to report CPU and memoryoverhead usage estimation for the proposed mechanism(s) toavoid any bad user-experience.

3. Some Bench-Mark Applications

I AnTuTuI GreekBenchI Quadrant Standard


1. Power OverheadI Any proposed mobile biometric recognition system must not

consume much power to be adopted in the real-worldapplications.

2. Computational OverheadI It is strongly recommended to report CPU and memory

overhead usage estimation for the proposedmechanism(s) to avoid any bad user-experience.

3. Some Bench-Mark Applications

I AnTuTuI GreekBenchI Quadrant Standard


1. Power OverheadI Any proposed mobile biometric recognition system must not

consume much power to be adopted in the real-worldapplications.

2. Computational OverheadI It is strongly recommended to report CPU and memory

overhead usage estimation for the proposed mechanism(s) toavoid any bad user-experience.

3. Some Bench-Mark ApplicationsI AnTuTuI GreekBenchI Quadrant Standard

Adversarial Analysis

1. Random AttacksI To obtain such attacks samples, the participants should

be asked to try randomly unlocking the device withoutknowing the implemented authentication mechanism.

2. Mimic Attacks

I To obtain such attacks samples, a genuine user could be askedto use the mechanism in front of the test-adversaries as manytimes as possible. In this way the adversaries may get a betteroverview of the implemented mechanism as well as legitimateuser’s behaviors that is to be mimicked.

3. Engineered Attacks

I We admit that executing this type of attack is a bit timetaking, cumbersome, and tricky, but the claims regarding therobustness of their proposed schemes should only be madeafter such evaluation.


1. Random AttacksI To obtain such attacks samples, the participants should be

asked to try randomly unlocking the device without knowingthe implemented authentication mechanism.

2. Mimic AttacksI To obtain such attacks samples, a genuine user could be

asked to use the mechanism in front of thetest-adversaries as many times as possible. In this waythe adversaries may get a better overview of theimplemented mechanism as well as legitimate user’sbehaviors that is to be mimicked.

3. Engineered Attacks

I We admit that executing this type of attack is a bit timetaking, cumbersome, and tricky, but the claims regarding therobustness of their proposed schemes should only be madeafter such evaluation.


1. Random AttacksI To obtain such attacks samples, the participants should be

asked to try randomly unlocking the device without knowingthe implemented authentication mechanism.

2. Mimic AttacksI To obtain such attacks samples, a genuine user could be asked

to use the mechanism in front of the test-adversaries as manytimes as possible. In this way the adversaries may get a betteroverview of the implemented mechanism as well as legitimateuser’s behaviors that is to be mimicked.

3. Engineered AttacksI We admit that executing this type of attack is a bit time

taking, cumbersome, and tricky, but the claims regardingthe robustness of their proposed schemes should only bemade after such evaluation.

Conclusions

I MotivationI Recent years have witnessed a lot of effort targeting the

development newer (secure and usable) authenticationsolutions for smart devices.

I Guidelines

I We presented some guidelines, particularly targetingresearchers of smart-devices authentication domain, for helpingthem in designing, implementation, and evaluation of theirproposed schemes.

I Objective

I In order to maximize the impact and usability of the proposedschemes, it becomes extremely important to design, developand evaluate, comprehensively, the upcoming schemes diversecriterion.

Conclusions



I GuidelinesI We presented some guidelines, particularly targeting

researchers of smart-devices authentication domain, forhelping them in designing, implementation, andevaluation of their proposed schemes.

I Objective

I In order to maximize the impact and usability of the proposedschemes, it becomes extremely important to design, developand evaluate, comprehensively, the upcoming schemes diversecriterion.

Conclusions



I GuidelinesI We presented some guidelines, particularly targeting

researchers of smart-devices authentication domain, for helpingthem in designing, implementation, and evaluation of theirproposed schemes.

I ObjectiveI In order to maximize the impact and usability of the

proposed schemes, it becomes extremely important todesign, develop and evaluate, comprehensively, theupcoming schemes diverse criterion.

Thank You!

Documents

Mobile Biometrics: Towards A Comprehensive Evaluation ... · New generation devices: personal, connected and powerful! I Beyond classical communication I Taking pictures & making