A Preliminary Field Study of Game Programming on Mobile Devices

A Preliminary Field Study of Game Programming on Mobile Devices

Sihan Li Tao Xie

University of Illinois at Urbana-Champaign

Eric Anderson North Carolina State University

1

http://arxiv.org/abs/1310.3308

Mining App Stores: Mobile App Markets

Apple App Store Google Play Microsoft Windows (Phone) Apps

Microsoft Research TouchDevelop

o Previous approaches look at permissions code (runtime behaviors)

o What does the users expect?

Mining App Stores: Program Analysis + Text Analytics

3

App Description

App Code

App Permissions

Pandita et al. WHYPER: Towards Automating Risk Assessment of Mobile Applications. USENIX Security 2013 http://web.engr.illinois.edu/~taoxie/publications/usenixsec13-whyper.pdf

o Previous approaches look at permissions code (runtime behaviors)

o What does the users expect?

Mining App Stores: Program Analysis + Text Analytics

3

App Description

App Code

App Permissions

Pandita et al. WHYPER: Towards Automating Risk Assessment of Mobile Applications. USENIX Security 2013 http://web.engr.illinois.edu/~taoxie/publications/usenixsec13-whyper.pdf

o Previous approaches look at permissions code (runtime behaviors)

o What does the users expect?

Mining App Stores: Program Analysis + Text Analytics

3

App Description

App Code

App Permissions

Pandita et al. WHYPER: Towards Automating Risk Assessment of Mobile Applications. USENIX Security 2013 http://web.engr.illinois.edu/~taoxie/publications/usenixsec13-whyper.pdf

o Previous approaches look at permissions code (runtime behaviors)

o What does the users expect?

Mining App Stores: Program Analysis + Text Analytics

3

App Description

App Code

App Permissions

Pandita et al. WHYPER: Towards Automating Risk Assessment of Mobile Applications. USENIX Security 2013 http://web.engr.illinois.edu/~taoxie/publications/usenixsec13-whyper.pdf

o Previous approaches look at permissions code (runtime behaviors)

o What does the users expect?

Mining App Stores: Program Analysis + Text Analytics

3

App Description

App Code

App Permissions

Pandita et al. WHYPER: Towards Automating Risk Assessment of Mobile Applications. USENIX Security 2013 http://web.engr.illinois.edu/~taoxie/publications/usenixsec13-whyper.pdf

Perhaps it's my ignorance, but would anything related to storage, system tools, your location, camera, development tools, phone calls, and network communication be ridiculously more accessible to the smartphone capabilities than what a flashlight app would require?

Mining App Stores: Open Source Mobile App Repo

https://f-droid.org/

http://research.microsoft.com/en-us/projects/touchdevelop/

Rapid Increase of TouchDevelop Popularity

5

vs. 17322 Feb this year

Mining App Stores: Educational/Learning Analytics

6

TouchDevelop Team@MSR

TouchDevelop Users

SE/HCI/Education … Researchers

Mining App Stores: Educational/Learning Analytics

6

TouchDevelop Team@MSR

TouchDevelop Users

SE/HCI/Education … Researchers

Mining App Stores: Educational/Learning Analytics

6

TouchDevelop Team@MSR

TouchDevelop Users

SE/HCI/Education … Researchers

Data-Driven Decision Making

(not just gut feeling)

Mining App Stores: Educational/Learning Analytics

6

TouchDevelop Team@MSR

TouchDevelop Users

SE/HCI/Education … Researchers

Data-Driven Decision Making

(not just gut feeling)

Our Previous Study on TouchDevelop Scripts and Users

• Characteristic and longitudinal study – 17,322 scripts and 4,275 users (July 2011-Feb 2013)

• Scripts – Structural features

– Code reuse

– Change of features over time

• Users – Expertise

– Publishing and learning activities over time

7

Sihan Li, Tao Xie, and Nikolai Tillmann. A Comprehensive Field Study of End-User Programming on Mobile Devices. VL/HCC 2013. http://web.engr.illinois.edu/~taoxie/publications/vlhcc13-study.pdf

Previous Characteristic Study

What do TouchDevelop scripts look like?

What is the code-reuse ratio of TouchDevelop scripts?

What kinds of TouchDevelop users are there?

8

How are TouchDevelop scripts changing over time?

What is the users’ progress of developing TouchDevelop scripts?

Structural Features: Findings

9

EMC: # external method call sites

What do TouchDevelop scripts look like?

Structural Features: Findings

73% less than 100 LOC, avg. 133 LOC

Small in size

9

EMC: # external method call sites

What do TouchDevelop scripts look like?

Structural Features: Findings

73% less than 100 LOC, avg. 133 LOC

106 EMC out of 114 MC per script Every two lines contain an EMC

Small in size

Heavily use of external code

9

EMC: # external method call sites

What do TouchDevelop scripts look like?

Structural Features: Findings

73% less than 100 LOC, avg. 133 LOC

106 EMC out of 114 MC per script Every two lines contain an EMC

Implications: Heavily rely on external methods to achieve functionalities. Enriching the built-in APIs and library scripts.

Small in size

Heavily use of external code

9

EMC: # external method call sites

What do TouchDevelop scripts look like?

Types of Code Reuse

• Inherit code from a parent script

• Use the existing library scripts

• Use TouchDevelop built-in APIs

• Copy and paste

10

Types of Code Reuse

• Inherit code from a parent script

• Use the existing library scripts

• Use TouchDevelop built-in APIs

• Copy and paste

10

Code Reuse: Findings

11

What is the code-reuse ratio of TouchDevelop scripts?

Code Reuse: Findings

58% scripts have a parent High code-reuse ratio

11

What is the code-reuse ratio of TouchDevelop scripts?

Code Reuse: Findings

58% scripts have a parent

74% parent-child pairs are written by the same user

High code-reuse ratio

Reuse their own code

11

What is the code-reuse ratio of TouchDevelop scripts?

Code Reuse: Findings

58% scripts have a parent

74% parent-child pairs are written by the same user

Quite a lot less than 10 LOC

High code-reuse ratio

Reuse their own code

Small modifications

11

What is the code-reuse ratio of TouchDevelop scripts?

Code Reuse: Findings

58% scripts have a parent

74% parent-child pairs are written by the same user

Quite a lot less than 10 LOC

Implications: Users tend to reuse code. Lack of code reuse across different users. Improve tool support for code reuse.

High code-reuse ratio

Reuse their own code

Small modifications

11

What is the code-reuse ratio of TouchDevelop scripts?

Expertise of Users: Metrics

• Activity of script-publishing – # scripts published

• Quality of scripts and comments – # positive reviews received

• Grasp of the TouchDevelop language – # TouchDevelop language features used (e.g.,

programming concepts, built-in APIs)

12

Clusters of Users

74.2% 21.5% 4.2%

13

Clusters of Users

74.2% 21.5% 4.2%

13

Expertise of Users: Findings

14

What kinds of TouchDevelop users are there?

Expertise of Users: Findings

74% novices, 22% ordinary users, 4% experts Big differences between users’ expertise

Most users are novices

14

What kinds of TouchDevelop users are there?

Expertise of Users: Findings

74% novices, 22% ordinary users, 4% experts Big differences between users’ expertise

Implications: Give novices high priority when designing tools and services. Provide customized services (e.g., UI) to different kinds of users.

Most users are novices

14

What kinds of TouchDevelop users are there?

Changes of Scripts: Findings

15

How are TouchDevelop scripts changing over time?

Changes of Scripts: Findings

# scripts , # large scripts , # library , code-reuse ratio .

Important increasing trends

15

How are TouchDevelop scripts changing over time?

Changes of Scripts: Findings

# scripts , # large scripts , # library , code-reuse ratio .

Implications: Enriching TouchDevelop language and Improving the management of libraries to better accommodate these trends.

Important increasing trends

15

How are TouchDevelop scripts changing over time?

Evolution of Users: Findings

16

What is users’ progress of developing TouchDevelop scripts?

Evolution of Users: Findings

22.1% users active initially, less active later 9.6% users not active initially, more active later 68.3% users publish one or two scripts, and then leave

Three patterns of publishing scripts

16

What is users’ progress of developing TouchDevelop scripts?

Evolution of Users: Findings

22.1% users active initially, less active later 9.6% users not active initially, more active later 68.3% users publish one or two scripts, and then leave

Implications: Useful to know why users leave or become less active and why they become more active. Design better strategies to retain users.

Three patterns of publishing scripts

16

What is users’ progress of developing TouchDevelop scripts?

Evolution of Users: Findings

17

Evolution of Users: Findings

16.3% users learn most initially, and only a few later 12.4% users learn some initially, and quite a lot at certain point 71.3% users learn a few initially, and stop learning

Three patterns of learning language features

17

Evolution of Users: Findings

16.3% users learn most initially, and only a few later 12.4% users learn some initially, and quite a lot at certain point 71.3% users learn a few initially, and stop learning

Implications: Users have different learning behaviors. Provide an adaptive tutoring system based on the history of users’ language-feature usage.

Three patterns of learning language features

17

Overview of This Study

• Motivation – As mobile gaming grows, it is important to analyze

game applications and how it compares to other applications.

• Characteristic study – 31,699 scripts, including 539 game scripts

• Goal – To discover how TouchDevelop games compare to

non-games – To discover what makes some games more popular

18

Characteristic study on Games

• RQ1: What do TouchDevelop game scripts look like when compared to non-game scripts?

• RQ2: How do games compare to non-games in terms of popularity?

• RQ3: What features do more popular games have that less popular games lack?

19

Research Question 1

• RQ1: What do TouchDevelop game scripts look like when compared to non-game scripts?

20

Structural Comparison

• Finding: Game scripts are longer than non-game scripts.

• Implication: Expanding IDE support could enhance the creation of large scripts.

21

Mean #LOC

Median #LOC

Game 369 272

Non-Game 210 50

Structural Comparison cont.

• Finding: Game scripts typically have more actions and events than non-game scripts

• Implication: IDE expansion could be highly beneficial.

22

Mean # action

Mean # event

Game 11.3 3.2

Non-Game 6.1 1.6

Structural Comparison cont.

• Finding: Game scripts invoke more methods than non-game scripts.

• Implication: Games include more features than non-games. Emphasis on games in marketing could be beneficial.

23

Mean # MC

Mean MC density

Game 347 0.94

Non-Game 165 0.79

Research Question 2

• RQ2: How do games compare to non-games in terms of popularity?

24

Comparison of Popularity

• Finding: Game scripts are more popular than non-game scripts

• Implication: TouchDevelop should market games as important types of scripts.

25

Mean # run

Mean # install

Mean # likes

Game 216 35 1.80

Non-Game 23 3 0.32

Research Question 3

• RQ3: What features do more popular games have that less popular games lack?

26

Game Popularity Comparison

• Metric: Top 25% of frequently run games as popular

• Finding: Popular games are longer than unpopular games

• Implication: Expansion of the IDE could encourage larger scripts

27

Games Mean # LOC

Median # LOC

Popular 462 384

Unpopular 363 255

Game Popularity Comparison cont.

• Finding: Popular and unpopular game scripts have similar numbers of actions and events

• Implication: Large numbers of actions and events may not contribute to popularity

28

Games Mean # action

Mean # event

Popular 14.1 3.6

Unpopular 11.1 3.2

Comparison of Actions and Events

0

2

4

6

8

10

12

14

16

Mean Actions Mean Events

Non-Games Games Popular Games Unpopular Games

29

# action and events

Comparison of Mean and Median Length

30

0

50

100

150

200

250

300

350

400

450

500

Mean Length Median Length

Non-Games Games Popular Games Unpopular Games

# LOC

Conclusion

• Previous study [VL/HCC 13] reports a number of valuable findings

• This study focuses on game vs. non-games popular vs. unpopular games – Games are typically longer and more feature-rich than

non-games – Popular games are typically longer but not necessarily

more feature rich than unpopular games

31

Thank you!

ACKNOWLEDGMENTS. This work is supported in part by NSF grants CCF-0845272, CCF- 0915400, CNS-0958235, CNS-1160603, CCF-1349666, CNS-1318419, a Microsoft Research award, and NSF of China No. 61228203. We thank Nikolai Tillmann for his help and feedback in conducting the study described in this paper.

32

Conclusion

• Previous study [VL/HCC 13] reports a number of valuable findings

• This study focuses on game vs. non-games popular vs. unpopular games – Games are typically longer and more feature-rich than

non-games – Popular games are typically longer but not necessarily

more feature rich than unpopular games

33