Drag-and-type: a new method for typing with virtual keyboards on small touchscreens

Contributed Paper Manuscript received 12/31/13 Current version published 03/20/14 Electronic version published 03/20/14. 0098 3063/14/$20.00 © 2014 IEEE

Drag-and-Type: A New Method for Typing with Virtual Keyboards on Small Touchscreens

Taekyoung Kwon, Member, IEEE, Sarang Na, Student Member, IEEE, and Sang-ho Park, Non-member, IEEE

Abstract — Small touchscreens are widely used in consumer

electronics, such as smartphones and mobile electronic devices. However, typing on the small touchscreen is still worth studying. In fact, smartphone users are experiencing difficulties and also many errors in typing alphanumeric keys with their thumbs because a small virtual keyboard even with the reduced set of touchable keys can only provide tiny size keys to the users. This paper studies a new style of typing method called Drag-and-Type, which leverages the dragging action instead of direct tapping on the touchscreen to ease more accurate typing on the small virtual keyboard. Although the typing speed is controversial, the consumers can choose this method when an accurate typing is more required, for example, for a password entry that is quite more sensitive to erroneous key inputs. In that sense, the proposed method is further explored to the extension called Secure Drag-and-Type for securing the password entry against shoulder-surfing and spyware attacks under the Drag-and-Type paradigm. In the user study, it was found that the proposed method could be used for secure and accurate password entry on the small touchscreen regarding the security-sensitive consumer electronics applications1.

Index Terms — Smartphone, touchscreen, virtual keyboard, shoulder-surfing, spyware.

I. INTRODUCTION

A smartphones is now becoming a part of electronics consumer’s lives and turns out to be one of the most popularly used consumer electronic devices. Its small flat touchscreen enables those consumers to navigate various kinds of services and applications very easily, promptly, and intuitively with their fingers. The small touchscreen is also changing the way of typing alphanumeric characters on those devices. Without a physical keyboard, today’s smartphones popularly present virtual keyboards, aka software keyboards based on the high-resolution of small touchscreens, e.g., 4.8″ 1280 720 pixels (306 ppi) and

1 This is the extended full manuscript version of the paper presented at the 2013 IEEE ICCE [4]. This work was supported by the IT R&D program of MSIP/KEIT [10039180, Intuitive, convenient and secure HCI-based usable security technologies for mobile authentication and security enhancement in mobile computing environments].

T. Kwon is with the Graduate School of Information, Yonsei University, Seoul, 120-749, Korea (e-mail: [email protected]).

S. Na is with the Graduate School of Information, Yonsei University, Seoul, 120-749, Korea (e-mail: [email protected]).

S. Park is with the Dept. of Computer Engineering, Sejong University, Seoul, 143-747, Korea (e-mail: [email protected]).

3.5″ 640 960 pixels (326 ppi) in commodities. To input alphanumeric keys, for example, consumers may tap their fingers on the small virtual keyboard through the small touchscreen but there exist at least two concerns that strongly motivate this study.

Fig. 1. Visual echo problems. (a) An entered key and its visual echo are occluded under the thumb. (b) Bigger echo of entered key b can be more easily observed not only by the user but also by the adversaries.

First, the smartphone users are frequently experiencing difficulties and also many errors in typing alphanumeric keys with their thick thumbs because a small virtual keyboard even with the reduced set of touchable keys can only provide tiny size keys to the users [1], [2]. Although the higher resolution of touchscreens can facilitate much smaller keys for constructing a full size keyboard layout, users may prefer a larger key so as to type characters with thumbs more easily. Unfortunately, such a larger key may only allow a partial keyboard layout having the reduced set of keys on the small touchscreen, e.g., separate layouts for alphabets and numeric (and/or special) characters, and pop-up keys for rendering more characters on the keys at best. Note that the partial keyboard layout requires a number of switches between distinct layouts. As illustrated in Fig. 1-(a), even worse, a visual echo, i.e., the most widely used response method on the virtual keyboard, can be occluded and hidden under the thick thumb with blunt touch. This tendency could reduce the benefits from the recent and future advance in the high-resolution touchscreens and hinder the consumers from being aware of the real key entry and eventually correct key entry on the touchscreen.

Second, the consumers are susceptible to malicious people nearby or spyware inside because they can capture the key input, particularly secret input such as a password, in mobile environments. As illustrated in Fig. 1-(b), when the visual echo is eminently shown bigger, the malicious people nearby can read what actually was entered by the consumer. This is called a shoulder-surfing attack that is more effective in a crowded place. Also, spyware can capture and exploit the

T. Kwon et al.: Drag-and-Type: A New Method for Typing with Virtual Keyboards on Small Touchscreens 99

touch event and its geometric data if a user types secret characters regardless of the visual echo [3].

In this paper, the two concerns regarding accuracy and security motivated the authors to develop a new style of typing method called Drag-and-Type, on the full layout of the virtual keyboard presented on the small touchscreens. The Drag-and-Type method leverages the dragging action instead of direct tapping on the touchscreen to ease more accurate typing on the small virtual keyboard. In particular, two kinds of Drag-and-Type methods are proposed: Drag-and-Tap and Drag-and-Drop on the full layout of the virtual keyboard. The Drag-and-Tap method works with separate tapping actions on the full size keyboard, whereas the Drag-and-Drop method works with dragging actions only [4]. Although the typing speed is controversial in both methods, the consumers can choose the Drag-and-Type methods when an accurate typing is more required, for example, for a password entry that is quite more sensitive to erroneous key inputs. In that sense, the Drag-and-Drop method is further explored to secure the password entry against shoulder-surfing and spyware attacks as well. The extended method is called Secure Drag-and-Type (Secure DnT when they say only briefly). Two user studies were conducted for both basic and secure methods, and it was found that the proposed method can particularly be used for accurate and secure typing on the small touchscreen regarding security-sensitive consumer electronics applications.

The remainder of this paper is organized as follows. The related work of this paper is reviewed in Section II. The Drag-and-Type method and its evaluation results are discussed in Section III. Subsequently, Secure DnT and its evaluation results are described in Section IV. Finally, this paper is concluded in Section V.

II. RELATED WORK

A virtual keyboard is commonly used to type characters into a touchscreen-based electronic device. To enter a character, a user must tap a finger on the corresponding software key instead of pressing the hardware key. There have been various keyboard designs regarding usability and security issues.

A. Virtual Keyboards for Usability

A number of virtual keyboards with distinct layouts, such as OPTI, ATOMIK, Metropolis and FITALY [5]-[7], have been proposed by rethinking the standard QWERTY keyboard with regard to usability issues in mobile electronic devices that incorporate a small touchscreen. CATKey [8] was developed to provide customizable and adaptable functions using QWERTY arrangement. However, due to the small size of the keys on a small touchscreen, it was hard for users to type characters correctly on those virtual keyboards. To cope with this problem, there have been various attempts. One is to overlay larger split-keys in a pie menu represented on a virtual keyboard [9] but, on the other hand, it causes two layered typing, which may be undesirable for the fast and/or consecutive typing of characters. M. Klima et al. [10]

proposed a vector keyboard that is composed of three major clusters containing 9 characters, respectively. A user can type characters with their thumbs by drawing a vector from one of the clusters. There still remains a problem that character keys can be visually occluded. S. Zhai et al. [11] proposed SHARK (Shorthand Aided Rapid Keyboarding) in which an ATOMIK keyboard is used to type characters by shorthand symbols, the symbols drawn with a stylus on the touchscreen [6]. Although there have been a number of virtual keyboard designs not limited to the above, it is interesting that the most widely used virtual keyboard is the QWERTY virtual keyboard but with the reduced set of touchable keys on the commodity devices.

B. Virtual Keyboards for Security

The virtual keyboards have been studied regarding the security as well. To defeat shoulder-surfing and spyware attacks on a secret key entry, such as a password for authentication, researchers designed various kinds virtual keyboards. Tan et al. [12] proposed Spy-Resistant Keyboard that consists of 42 character tiles and 2 indicator tiles. Each character tile is assigned three characters in random order. To type a password, a user must set a shift state and move one of the indicators over the target character tile. Bai et al. [13] proposed PAS (Predicate-based Authentication Services) in which the user can indirectly enter the password through the predicates generated by two secret values and the CAPTCHA table. Zhao and Li [14] developed S3PAS (A Scalable Shoulder-Surfing Resistant Textual-Graphical Password Authentication Scheme). In this method, the user can enter a graphical password by constructing pass-triangles based on the password and clicking the inside of the triangle. A variety of authentication methods have also been developed to resist spyware that exploits touch-based screenshot captures. J. Lim [15] proposed an anti-screen capture method based on a partial image on each input key. In this method, the keypad is arranged at random after every mouse click. Interestingly, each key alternates three images very quickly on it: one blank and two partial images of the real key value. Agarwal et al. [16] studied Dynamic Virtual Keyboard. The keyboard layout of this method is similar to the random virtual keyboard that incorporates a dynamic random arrangement. In this method, the user clicks a specific key called hide keys and then a real input key when all keys are already hidden. The common limitation of these improved security measures is undoubtedly their input performance. It takes long to enter passwords, about 49s for 8-text passwords in the Spy-Resistant Keyboard [12] and about 33s for 6-text passwords in the Dynamic Virtual Keyboard [16]. Readers are referred to Table I in the end of this paper for more comparisons.

III. DRAG-AND-TYPE METHODS

On the flat touchscreen, finger touch actions can be classified into two actions, i.e., tapping and dragging. The former is activated usually for a click event, whereas the latter is done for scrolls and/or more functions, such as pointing and navigating. Multiple touch actions may involve simultaneous

100 IEEE Transactions on Consumer Electronics, Vol. 60, No. 1, February 2014

and/or consecutive actions of tapping and dragging. These actions are considered for devising a new typing method. First of all, it is pointed out that the dragging action enables more accurate targeting to a tiny key on the virtual keyboard. Another point is that the user’s thumb typing on a small touchscreen is done eventually as like the hunt and peck typing, aka two-fingered typing, on a real keyboard. Thus, it is expected that if a small touchscreen represents a full size keyboard on which tiny keys are located close to each other, then the dragging actions of pointing would be quite familiar as well as more accurate than the direct tapping actions. Although the accuracy is obtained at the cost of dragging time, it would be reasonable to think that less erroneous typing is also attractive in a large number of applications. So two sorts of Drag-and-Type methods are devised in that sense.

Fig. 2. Prototype designs of Drag-and-Type methods. (Large circles indicate places to touch, whereas red and blue dots signify the pointers.) (a) Drag-and-Tap. (b) Drag-and-Drop. (It shows the split QWERTY keyboard layout for two fingers.)

A. Drag-and-Tap

The first method presumes a full layout of standard QWERTY keyboard in small size and makes a user navigate the virtual keyboard by dragging one finger, e.g., the left thumb, and type a highlighted (selected) character by tapping on any blank area with another finger, e.g., the right thumb. Fig. 2-(a) illustrates a prototype layout of Drag-and-Tap keyboard. The small red dot located among the keys, y, u, and h, is used to navigate and select a target key while the larger grey circle below the keyboard indicates an actual place for dragging. Deep grey keys are used for rendering more functions onto the keyboard, such as tab, language, shift, backspace, space, and enter. Fig. 3-(a) is a snapshot of the Drag-and-Tap method in actual use.

B. Drag-and-Drop

The second method also presumes a full layout of standard QWERTY keyboard in small size, and makes a user navigate the virtual keyboard by dragging one or two fingers simultaneously, e.g., the left and right thumbs, and type a highlighted (selected) character by releasing (dropping) the corresponding finger. Fig. 2-(b) illustrates a prototype layout of Drag-and-Drop keyboard using a split QWERTY layout for two fingers. The small red and blue dots are used to navigate

and select target keys, respectively, while larger grey circles indicates a place for dragging. Deep grey keys are also split for rendering more functions onto the keyboard. Fig. 3-(b) is a snapshot of the Drag-and-Drop method in use. Note that the Drag-and-Drop method can be used in one hand.

Fig. 3. Screenshots of Drag-and-Type methods in use. (a) Drag-and-Tap. The left thumb is used for dragging while the right thumb is used for tapping. (b) Drag-and-Drop. A single hand can be used for typing.

C. Usability Evaluation

Prototype systems of the Drag-and-Type methods were implemented on the smartphone, as illustrated in Fig. 3 and a user study was conducted for evaluating the usability of each method. In the user experiment, the split keyboard layout is used in Drag-and-Drop. Drag-and-Type methods were compared to the regular virtual keyboard with respect to the speed and accuracy for typing characters.

1) Design. The user experiment was designed as a within group study using 2 3 Repeated Measures-ANOVA. In the user study, first independent variable is character type (alphabets, decimals). Second independent variable is virtual keyboard (regular keyboard, Drag-and-Tap, and Drag-and-Drop). The participants conducted one combination of independent variables randomly to reduce learning effects for character and method type. To evaluate the performance of each typing method, the entry time and error rates were measured in the evaluation session.

2) Participants. 12 participants (9 males, 3 females) with academic education were recruited. Their average age was 26.9 and the average period of using smartphone (cellphone) was 2.3 (10.9). The participants were comprised of 2 left-handers and 10 right-handers. All of them had normal eyesight and experience of using regular virtual keyboard. The participants received a small gratuity for the user experiment.

3) Procedure. The participants conducted three methods in the within group study. The order of method and character type was counterbalanced (3! 2! = 12). They received an explanation about how to type the characters with each method. They were asked to type alphabets in sequence, i.e., a to z, for 5 times, and decimals in sequence, i.e., 1 to 0, for another 5 times, after training themselves up to twenty minutes. After finishing the experiment of each method, they responded the questionnaire. Likert-type scales were used for rating 1 (strongly disagree) to 5 (strongly agree).


4) Hypothesis. The following hypotheses were stated for the user experiments:

(H1) Drag-and-Type is slower than the regular keyboard.

(H2) Drag-and-Drop is slower than Drag-and-Tap.

(H3) Drag-and-Type is less error-prone than the regular keyboard.

5) Results. The entry time was measured for each method from the beginning of the application execution to the last release of the pressed key in the successful session. The average entry time was 18.963s (sd: 2.558), 23.496s (sd: 2.251), and 23.691s (sd: 2.182) for alphabets, respectively, in regular, Drag-and-Tap, and Drag-and-Drop methods. As for decimals, the average entry time was 6.552s (sd: 1.171), 8.192s (sd: 1.114), and 10.085s (sd: 1.464), respectively. Fig. 4 illustrates the graphical results of user study regarding entry time. In 2 3 Repeated Measures-ANOVA, there was a significant main effect for virtual keyboard (F(2, 22) = 43.039, p < 0.001) and character type (F(1, 11) = 1433.931, p < 0.001). However, the interaction effect of virtual keyboard and character type was not significant (F(2, 22) = 3.144, n.s.(p = 0.063)). Post-hoc analysis showed that there was a significant difference between regular keyboard and Drag-and-Type method (p < 0.001). Regarding these results, H1 can be accepted. There was no significant difference for two Drag-and-Type methods (p = 0.181). Thus, H2 has to be rejected. In the questionnaire, the participants evaluated the regular keyboard (mean: 4.5, sd: 0.674) more fast to use than Drag-and-Tap (mean: 2.583, sd: 0.996) and Drag-and-Drop (mean: 2.417, sd: 0.9). When considering that participants are already used to the regular keyboard, the experimental results are fairly remarkable.

The error rates were measured with the number of mistyped trials and backspace counts to correct the entered characters in the evaluation session. The results were impressive in the user study. There was one mistyped trial (8.3%) for alphabets in the regular keyboard. However there was no mistyped trial for decimals in the three methods. Thus, there was no significant main effect for virtual keyboard about failed sessions. The average error rates regarding backspace counts for alphabets were 3.04%, 0.31%, and 0.31%, respectively, in the regular keyboard, Drag-and-Tap, and Drag-and-Drop. As for decimals, the average error rates were 2.14%, 0.31%, and 0.0%, respectively. Fig. 5 shows the graphical results of the error rates. It was found that there was a significant main effect for virtual keyboard (F(1.233, 13.564) = 11.192, p < 0.005). However, there was no significant difference for character type (F(1, 11) = 3.287, n.s.(p = 0.097)) and the interaction effect of virtual keyboard and character type (F(2, 22) = 0.553, n.s.(p = 0.583)). Post-hoc analysis showed that there was a significant difference between the regular keyboard and Drag-and-Type (p < 0.05). Regarding these results, H3 can be accepted.

It is important to enter passwords without mistyped trials and typos so that users cannot provide attackers with opportunities to observe more entered sensitive information. In addition, there is a concern that the authentication system

locks out users’ account if they exceed the fixed trials for entering the passwords. Therefore, the extension of Drag-and-Type is explored as a password input method.

Fig. 4. Average entry time of each method (regular keyboard, Drag-and-Tap, Drag-and-Drop) in the user experiment.

Fig. 5. Average error rates regarding backspace counts of each method (regular keyboard, Drag-and-Tap, Drag-and-Drop) in the user study.

IV. SECURE DRAG-AND-TYPE METHODS Most of applications and web services in the smartphone

provide a regular virtual keyboard, using QWERTY arrangement, when users enter even their secret characters. However, the regular virtual keyboard is not appropriate as a password input method because it is possible for shoulder-surfing attackers and spyware to intercept the user’s sensitive information from the mobile computing devices. To cope with this problem, some applications offer their own secure virtual keyboards and a number of authentication methods have been also proposed. However, those authentication methods can’t properly defend shoulder-surfing and spyware attacks at the same time. The extended method of Drag-and-Type, called Secure DnT, is designed to be secure against those attacks in the smartphone.

A. Threat Model

In this paper, it is assumed that there are two kinds of adversaries observing the entered secret characters in the mobile computing devices. First adversary is a human shoulder-surfing attacker, trying to look over someone’s typing. Second adversary is a touch-based spyware attack that gathers consumers’ sensitive information without their consent by exploiting the touch event information and screenshots.


Fig. 6. Prototype design of Secure DnT method. (a) Keyboard layout before a user drags a pointer. (b) Blank keyboard layout when a user drags a pointer.

B. Basic Concept

The keyboard layout of Secure DnT is composed of alphanumeric characters in random arrangement. Fig. 6 shows the prototype design of Secure DnT method. The characters of all keys are hidden when a user begins to drag a pointer on the touchscreen. The keyboard layout remains blank until a character key is entered. After the character key is entered, the keyboard layout is rearranged in random sequences and the hidden keys reappear. These mechanisms enable to prevent efficiently shoulder surfing and spyware attack from stealing users’ secret characters. A user can find out easily the location of own password, whereas it is hard for observers to identify it. Although adversaries may guess the secret characters, they could not find out exactly whole of them because of hiding promptly all keys in the keyboard layout.

C. Input Interface

Secure DnT uses an input interface of Drag-and-Drop that uses dragging for navigation and releasing for typing. It differs from Drag-and-Drop in using without visual echo. However, it is possible to enter the character accurately with visual selected key echo and vibration feedback. A user has to verify the location of the target character keys before touching on the touchscreen. When the touch event, e.g., ACTION_DOWN, is occurred, Secure DnT hides all keys automatically without an additional action, e.g., pressing the hide key button. So a user has no extra burden to enter the characters with this method.

D. Usability Evaluation

Prototype applications (random keyboard and Secure DnT) were implemented on the smartphone and a user study was conducted. In this study, the Secure DnT method was compared to the regular and the random keyboards.

1) Design. The experiment was designed as a within group user study for evaluating usability using 2 3 Repeated Measures-ANOVA. In the user study, first independent variable is password type (system-chosen password, user-chosen password). Second independent variable is password input method (regular keyboard, random keyboard, and Secure DnT). The participants performed one combination of

independent variables randomly to reduce learning effects for password type and password system. To evaluate the performance of each typing method, the entry time and error rates were measured in the evaluation session.

2) Participants. New 18 participants (11 males, 7 females) with academic education were recruited in the local university. Their average age was 27.7 years and the average period of using smartphone (cellphone) was 2.6 (11.4) years. The participants were comprised of 2 left-handers and 16 right-handers. All of them had normal eyesight and had experience of using regular virtual keyboard. The participants received a small gratuity in return for the user experiment.

3) Procedure. The participants performed three methods in random sequences. The order of method and character type was counterbalanced based on a Latin square design. They received an explanation about the instruction of each method and were allowed training time for entering “abcd1234” three times, respectively. In the evaluation test, participants were asked to enter two types of passwords twice, one for practice and the other for test. The participants had system-chosen passwords and user-chosen passwords that consist of 8 alphanumeric characters. System-chosen passwords are generated by software program. User-chosen passwords are made by the users avoiding very simple passwords, e. g., “qwer0987”. It is familiar to users, but it is hard for attackers to guess. They memorized two passwords before entering the passwords. After finishing the experiment of each method, they responded the questionnaire.

4) Results. The password entry time for each method was measured from the beginning of the application execution to the last release of the pressed key. The fastest password input method was regular keyboard with user-chosen passwords (mean: 5.913, sd: 0.994). Regular keyboard with system-chosen passwords (mean: 6.138, sd: 1.061), random keyboard with system-chosen passwords (mean: 20.361, sd: 2.682), random keyboard with user-chosen passwords (mean: 20.729, sd: 2.846), Secure DnT with user-chosen passwords (mean: 21.940, sd: 2.106), Secure DnT with system-chosen passwords (mean: 22.435, sd: 2.661) followed it. Fig. 7 shows the average entry time for each method. In 2 3 (password type password input system) Repeated Measures-ANOVA, there was a significant main effect for password input system (F(2, 34) = 959.466, p < 0.001). However, there was no significant main effect for password type (F(1, 17) = 0.129, n.s.(p = 0.724)). The interaction effect of password type and password input system was not significant (F(2, 34) = 0.706, n.s.(p = 0.501)). Regarding these results, Secure DnT was slower than the regular keyboard and random keyboard. It was reasonable, considering the security level of each method (more detailed in the security evaluation). In the questionnaire, the participants evaluated regular keyboard (mean: 4.39, sd: 0.698) more fast to use than random keyboard (mean: 1.67, sd: 0.686) and Secure DnT (mean: 1.78, sd: 0.808).


Fig. 7. Average entry time of each method (regular keyboard, random keyboard, Secure DnT) in the user experiment.

The failed sessions and backspace counts for each method were measured in the evaluation session. There was no failed session in the regular keyboard and Secure DnT. However, for the random keyboard, two participants succeeded in second trial (5.6%) with system-chosen passwords. The number of backspace was one with user-chosen password in the random keyboard and Secure DnT, respectively. Thus, there was no significant main effect for password input methods.

E. Security Evaluation

1) Shoulder-surfing Resilience. Participants were asked to mount a shoulder surfing attack by observing the user’s password entry on the smartphone. The attack experiment was conducted with 10 participants (7 males, 3 females) whose average of age was 27.2 years and had joined the usability experiment before. All of the participants had normal eyesight. They were given 5 video records for authentication sessions that an operator enters the 8-text passwords with the regular keyboard (no echo mode and visual echo mode), random keyboard and Secure DnT method, respectively. Each video was adjusted the entry time of usability results and recorded with the digital camcorder. The participants planned their own strategy for shoulder-surfing attacks before starting the experiment. The 23-inch computer monitor (1920 1200 pixels) was used for playing the recorded videos. The size of played smartphone was set similar to real smartphone size. The entered passwords were shown as in the shape of asterisk. So, the participants could observe just keyboard layout for each method.

In the attack experiment, the results were surprising. The highest success rate of guessing the passwords is regular keyboard with echo (86.8%). Regular keyboard without echo (73.5%), Random keyboard (12.3%), and Secure DnT (2.8%) followed it. Fig. 8 illustrates the results of attack experiments. In case of regular keyboard with echo, the success rate of identifying 8-text passwords was 32% and missed one text was 42%. Regular keyboard without echo was harder than regular keyboard with echo to find out the passwords due to the hidden keys under the thumb. On the other hand, the failed rate of identifying all 8-text passwords with Secure DnT was 82%. The participants had difficulty in identifying the passwords in the random keyboard layout both random

keyboard and Secure DnT. Random keyboard is more vulnerable than Secure DnT, because random keyboard can give some hints about moving direction of a user’s finger.

Fig. 8. Graphical results of the shoulder-surfing attack experiments for each password input method.

2) Spyware Resilience. An attacker who is familiar with the regular keyboard (with visual echo), random keyboard, and Secure DnT simulated a spyware attack, recording the data of touch events and capturing screenshots for each method. The attacker implemented malicious applications that have two additional functions, gathering coordinates of touch events and screenshots when a user touches up on the smartphone. Also, the pointer location option in the developer options of system settings was checked to present visually current touch actions.

The results of the spyware attack were impressive. The attacker tried to find out the entered key by exploiting both screenshots and touch coordinates in the keyboard layout. The attacker could identify the typed character in the regular keyboard and random keyboard. It is easy to distinguish the pressed key both methods. In case of the regular keyboard, it uses QWERTY arrangement and appears certain location in the application each time. Thus, only one screenshot was needed for analyzing the regular keyboard to get information for keyboard location. On the other hand, multiple screenshots were needed for analyzing random keyboard. Fig. 9-(a) and Fig. 9-(b) illustrate one of the screenshots together with touched area of each method, respectively. It is possible for the regular keyboard to find the typed character using only information of touch events due to the fixed keyboard layout. Furthermore, the attacker could just identify the entered key using only screenshots in the regular keyboard and random keyboard, as illustrated in Fig. 9.

However, it was unsuccessful to identify the typed character in Secure DnT. That is why it hides all keys in the keyboard layout when a user touches down on the smartphone, as illustrated in Fig. 10. Although attackers may gain the screenshots and coordinates of touch events for Secure DnT, they can’t find out the character. In the questionnaire regarding security for shoulder-surfing and spyware attacks, the participants rated regular keyboard (mean: 1.39, sd: 0.608) less secure than random keyboard (mean: 2.89, sd: 0.832) and Secure DnT (mean: 4.67, sd: 0.594).


TABLE I COMPARISON OF THE USABILITY AND SECURITY FOR EACH PASSWORD INPUT METHOD*.

* Security level is categorized according to how each method is resilient to shoulder-surfing and spyware attack (e.g., weak, moderate, and strong).

Fig. 9. Screenshots including pointer location for each method. Upper screenshots are captured in an integrated development environment and lower screenshots are captured in the malicious application. (a) Regular keyboard with visual echo. (b) Random keyboard.

Fig. 10. Screenshots for Secure DnT. (a) Screenshots without pointer location option. (b) Screenshots with pointer location option.

F. Comparison

Secure DnT was compared with other authentication methods regarding its usability and security. TABLE I summarizes the comparison results of each authentication method. The fastest is the regular keyboard (5.91s – 6.14s), but it is vulnerable to shoulder-surfing and spyware attacks. The random keyboard is a little faster than the Secure DnT,

but it has low security for spyware attack. The security of Dynamic Virtual Keyboard is similar to Secure DnT method, but its entry time (about 5.48s for one-text password) is about twice as long as Secure DnT (2.74s - 2.81s for one-text password). Similarly to Dynamic Virtual Keyboard, Spy-resistant Keyboard is resilient to both attacks and it has also long entry time (about 49s). The user experiment result of CHC [18] was referred for comparing the entry time of S3PAS method. Its execution time is slowest (71.66s) among other authentication methods and PAS followed it (55.53s). Moreover, S3PAS and PAS are vulnerable to intercept attacks analyzing the multiple authentication sessions [17].

V. CONCLUSION

In this paper, the new Drag-and-Type method (Drag-and-Tap and Drag-and-Drop) and its extension called Secure DnT were proposed. The Drag-and-Type was a novel typing method based on the dragging actions on a small flat touchscreen. The prominent feature of the Drag-and-Type method was accuracy. The consumers are able to type more accurately but more slowly on the full-size virtual keyboards than on the existing virtual keyboard. The Drag-and-Type method was extended to its secure virtual keyboard version called Secure DnT to deal with shoulder-surfing and spyware attacks. The Secure DnT method was more efficient and/or more secure compared to the related authentication methods. The user studies and the attack experiments conducted in this paper confirm that it would be promising to adapt the Drag-and-Type method when a more accurate typing is preferred, and the Secure DnT method when a more accurate and securing typing is required on the consumer electronic devices. Specifically, a secure (and accurate) password entry can be achieved by the Secure DnT method. The limitation is that the Secure DnT can only resist a touch-based spyware attack. In the future study, a new method will be explored to resist an advanced spyware attack based on recording the whole interactions between consumer and electronic device through the small high-resolution touchscreens.

Method Password Length Rounds Entry Time (s) Shoulder-surfing Spyware

Regular Virtual Keyboard 8 1 5.91-6.14 Weak Weak

Random Virtual Keyboard 8 8 20.36-20.73 Moderate Weak

PAS [13] Two secret strings 2 55.53 Strong Moderate

S3PAS [14] 3-5 5 71.66 Strong Moderate

Dynamic Virtual Keyboard [16] 6 6 32.87 Strong Strong

Spy-resistant Keyboard [12] 8 8 49 Strong Strong

Secure Drag-and-Type 8 8 21.94-22.44 Strong Strong


REFERENCES

[1] Y. Yoon and G. Lee, “Square: 33 keypad mapped to geometric elements of a square,” IEEE Trans. on Consumer Electronics, vol. 54, pp. 1274-1280, Aug. 2008.

[2] V. Balakrishnan and P. Yeow, “A study of the effect of thumb sizes on mobile phone texting satisfaction,” Journal of Usability Studies, vol. 3, pp. 118-128, May 2008.

[3] L. Cai and H. Chen, “TouchLogger: Inferring keystrokes on touch screen from smartphone motion,” in Proc. USENIX Conference on Hot Topics in Security, San Francisco, USA, Aug. 2011.

[4] T. Kwon, S. Na, and S. Park, “Drag-and-Type: A new method for typing with virtual keyboards on small touchscreens,” in Proc. IEEE International Conference on Consumer Electronics, Las Vegas, USA, pp. 460-461, Jan. 2013.

[5] I. S. Mackenzie and S. X. Zhang, “The design and evaluation of a high-performance soft keyboard,” in Proc. SIGCHI Conference on Human Factors in Computing Systems, Pittsburgh, USA, ACM press, pp. 25-31, May 1999.

[6] S. Zhai, M. Hunter, and B. A. Smith, “Performance optimization of virtual keyboards,” Human-Computer Interaction, vol. 17, 2002.

[7] J. D. Ichbian, “Method for designing an ergonomic one-finger keyboard and apparatus therefor,” In US patent 5487616, 1996.

[8] K. Go and Y. Endo, “CATKey: Customizable and adaptable touchscreen keyboard with bubble cursor-like visual feedback,” in Proc. IFIP TC 13 International Conference on Human-Computer Interaction, Rio de Janeiro, Brazil, LNCS 4662, pp. 493-496, Sept. 2007.

[9] K. Go and L. Tsurumi, “Arranging touch screen software keyboard split-keys based on contact surface,” in Proc. CHI’10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, USA, ACM press, Apr. 2010.

[10] M. Klima and V. Slovacek, “Vector keyboard for touch screen devices,” in Proc. International Conference on Ergonomics and Health Aspects of Work with Computers, San Diego, USA, LNCS 5624, pp. 250-256, July 2009.

[11] S. Zhai and P. O. Kristensson, “Shorthand writing on stylus keyboard,” in Proc. SIGCHI Conference on Human Factors in Computing Systems, Lauderdale, USA, ACM press, pp. 97-104, Apr. 2003.

[12] D. S. Tan, P. Keyani, and M. Czerwinski, “Spy-Resistant Keyboard: More secure password entry on public touch screen displays,” in Proc. Australia Conference on Computer-Human Interaction, Canberra, Australia, Nov. 2005.

[13] X. Bai, W. Gu, S. Chellappan, X. Wang, D. Xuan, and B. Ma, “PAS: Predicate-based authentication services against powerful passive adversaries,” in Proc. IEEE Annual Computer Security Applications Conference, Anaheim, USA, pp. 433-442, Dec. 2008.

[14] H. Zhao and X. Li, “S3PAS: A scalable shoulder-surfing resistant textual-graphical password authentication scheme,” in Proc. IEEE International Conference on Advanced Information Networking and Applications Workshops, Niagara Falls, USA, vol. 2, pp. 467-472, May 2007.

[15] J. Lim, “Defeat spyware with anti-screen capture technology using visual persistence,” in Proc. Symposium on Usable Privacy and Security, ACM press, July 2007.

[16] M. Agarwal, M. Mehra, R. Pawar, and D. Shah, “Secure authentication using dynamic virtual keyboard layout,” in Proc. International Conference & Workshop on Emerging Trends in Technology, ACM press, pp. 288-291, Feb. 2011.

[17] Q. Yan, J. Han, Y. Li, and R. H. Deng, “On limitations of designing leakage-resilient password systems: Attacks, principles and usability,” in Proc. Network and Distributed System Security, San Diego, USA, Feb. 2012.

[18] S. Wiedenbeck, J. Waters, L. Sobrado, and J. C. Birget, “Design and evaluation of a shoulder-surfing resistant graphical password scheme,” in Proc. Advanced Visual Interfaces, Venezia, Italy, ACM press, pp. 177-184, May 2006.

BIOGRAPHIES

Taekyoung Kwon (M’02) received his B.S., M.S., and Ph.D. degrees in computer science from Yonsei University, Seoul, Korea, in 1992, 1995, and 1999, respectively. He is currently an Associate Professor of information at Yonsei University, Seoul, Korea. From 1999 to 2000, he was a Post-Doctoral Research Fellow at the University of California, Berkeley, CA, USA, and developed a

cryptographic protocol, which was later standardized by IEEE P1363.2 and ISO/IEC JTC1 SC27 11770-4, respectively. From 2001 to 2013, he was a professor of computer engineering at Sejong University, Seoul, Korea. From 2007 to 2008, he was on sabbatical at the University of Maryland, College Park. In 2013, he returned to Yonsei University, Seoul, Korea. His current research interests include information security and privacy, applied cryptography, cryptographic protocol, network protocol, usable security, and human–computer interactions.

Sarang Na (S’13) received her B.S. and M.S. degrees in computer science and engineering from Sejong University, Seoul, Korea, in 2011 and 2013, respectively. She is currently pursuing a Ph.D. degree at the Graduate School of Information, Yonsei University, Seoul, Korea. Her current research interests include cryptographic protocol, computer network security, mobile security, usable security, and

human-computer interactions.

Sang-ho Park received his B.S. and M.S. degrees in computer science and engineering from Sejong University, Seoul, Korea, in 2004 and 2006, respectively. He is currently pursuing a Ph.D. degree at the department of computer science and engineering, Sejong University, Seoul, Korea. His current research interests include computer network security and mobile security.


Documents

Drag-and-type: a new method for typing with virtual keyboards on small touchscreens