Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Augmented Reality with Phones for Way Finding
Zhibo Zhang
U6015337
COMP4560
The Research School of Computer Science, CECS
Australian National University
28 May 2019
The Australian National University | 2
Acknowledgment
Finishing this report takes a long time in coding and evaluation. I am writing to extend
my sincere gratitude to those people who helped me with this project.
Firstly, I have my best wishes to my supervisor, Henry Gardener, and my tutor, Wanqi
Zhao. They took a long time to help me to get familiar with AR environment and
Unity3D. They also provided lots of instructions about project management and report
writing skills. I also want to thank Penny who gives a lot of ideas about how to evaluate
an AR application.
Finally, I want to thank my parents gave me chance to have this great experience in
ANU and encouraged me during the project. Also, I want to say thanks to my friend
Yukun who was willing to be my listener by my side.
The Australian National University | 3
Abstract
Nowadays, many kinds of map applications can be found in phones. A map application
helps people to navigate in the real world. With the development of Augmented Reality
(AR), it is possible to combine such applications with three dimensional graphical
overlays on top of a real image. Such AR technology offers the prospect of making the
map applications more convenient to use.
This report is about designing and implementing an application on an Android phone
for wayfinding by implementing Augmented Reality using Unity3D. Current research
about Augmented Reality and wayfinding system are outlined, including how
Augmented Reality technology has developed and what other researchers have done to
combine it with wayfinding problems. An AR phone application prototype on Android
is then implemented, using Unity3D, ARKit, and Mapbox, to find directions to
locations outside and inside of buildings. Based on the AR map prototype, a Heuristic
Evaluation is performed, and its results are used to improve the prototype.
The Australian National University | 4
Content
1 Introduction ......................................................................................................... 9
1.1 Outline of the project ................................................................................... 9
1.2 Motivation ................................................................................................... 9
1.3 Structure of the report ................................................................................ 10
2 Background ....................................................................................................... 11
2.1 Augmented Reality .................................................................................... 11
2.2 GPS ............................................................................................................ 13
2.3 AR in wayfinding ...................................................................................... 14
3 Project Plan ....................................................................................................... 18
4 Implementation ................................................................................................. 19
4.1 Development environment ........................................................................ 19
4.2 Import Mapbox-plugin .............................................................................. 20
4.3 Match GPS and initialize the map ............................................................. 21
4.4 Navigation in 2D map................................................................................ 22
4.5 Navigation outside the building ................................................................. 23
4.6 Navigation inside the building ................................................................... 24
The Australian National University | 5
4.7 NavMesh .................................................................................................... 25
4.8 User interface design ................................................................................. 28
5 Evaluation ......................................................................................................... 30
5.1 Heuristic Evaluation Principles ................................................................. 30
5.2 Heuristic Evaluation Process ..................................................................... 32
5.3 Improvement based on evaluation ............................................................. 42
6 Limitations and Future Work ............................................................................ 43
6.1 Limitations ................................................................................................. 43
6.2 Future Work ............................................................................................... 44
7 Conclusion ........................................................................................................ 45
8 Bibliography ..................................................................................................... 46
9 Appendix 1 Product Screenshot ........................................................................ 48
10 Appendix 2 Contract ..................................................................................... 57
11 Appendix 3 Evaluation .................................................................................. 61
The Australian National University | 6
Figure Content
Figure 1: 2D mobile map and AR map (Google ARMap) ........................................ 10
Figure 2: Span in Milgram's Reality-Virtuality (Milgram et al., 1995) ....................... 12
Figure 3: How Virtual Information Display(Report et al., 2016) ................................ 13
Figure 4: Narzt’s program (Narzt et al., 2006) ............................................................ 15
Figure 5: Narzt's program workflow(Narzt et al., 2006) ............................................. 15
Figure 6: Kim & Jun's Program(Kim and Jun, 2008) .................................................. 16
Figure 7: Kim & Jun program workflow ..................................................................... 17
Figure 8: Mapbox Setup in Unity3D ........................................................................... 21
Figure 9: GeoToWorldPosition Method. ...................................................................... 22
Figure 10: Screenshot from 2D map ............................................................................ 23
Figure 11: Workflow for 2D and AR outdoor map ...................................................... 24
Figure 12: AR indoor map workflow ........................................................................... 25
Figure 13: NavMesh without path rendering ............................................................... 26
Figure 14: NavMesh with rendering path .................................................................... 27
Figure 15: Screenshot of indoor map prototype .......................................................... 29
Figure 16: Task device and materials ........................................................................... 39
The Australian National University | 7
Figure 17: Check point when during task .................................................................... 39
Figure 18: Screenshot of the new prototype ................................................................ 42
Figure 19: Screenshot of new prototype with reference check point ........................... 43
Figure 20: Workspace in Unity3D ............................................................................... 48
Figure 23: Original prototype screenshot 1 ................................................................. 49
Figure 24: Original prototype screenshot 2 ................................................................. 50
Figure 25: Original prototype screenshot 3 ................................................................. 51
Figure 26: Original prototype screenshot 4 ................................................................. 52
Figure 27: Original prototype screenshot 5 ................................................................. 53
Figure 28: New prototype screenshot 1 ....................................................................... 54
Figure 29: New prototype screenshot 2 ....................................................................... 55
Figure 30: New prototype screenshot 3 ....................................................................... 56
The Australian National University | 8
Table Content
Table 1: Development environment ............................................................................. 20
Table 2: Usability Principles for AR Application (Ko et al., 2013) ............................. 31
Table 3: HCI principles(Dünser et al., 2007) ............................................................... 32
Table 4: Principles used in evaluation ......................................................................... 33
Table 5: Serious problems found from the prototype .................................................. 37
Table 6: Problem list from task-based evaluation ........................................................ 40
Table 7: Feedbacks on principles ................................................................................. 41
The Australian National University | 9
1 Introduction
1.1 Outline of the project
The difficulty in finding directions from a 2D-map, and matching these directions to
the real world, has been a big problem in wayfinding for personal navigation. To solve
this problem, a prototype application using Augmented Reality (AR) technology is
presented in this report. Not just outdoor navigation, this application will also support
wayfinding inside of a building. Heuristic Evaluation and expert-based evaluation will
be done based on the prototype.
The prototype application will be designed on the Android system and will be run on a
Google Pixel Phone. Unity3D will be used as the main platform to implement the
application and the Unity3D Mapbox package will be used as a map database.
The prototype application is based on a map of Australian National University campus
and provides navigation around and inside of the Hanna Neumann building (building
145). People can use this prototype to find offices and meeting rooms in an area of the
first and second floor of the Hanna Neumann building.
1.2 Motivation
Nowadays, it is common to use a 2D map for people who use the navigation system.
However, there are always some troubles in finding directions when people use a 2D
map. Besides, hardly any map application provides an indoor map navigation service
when people are more likely to get lost inside a building.
The potential of Augmented Reality to enhance navigation is shown in Figure 1.
Compared with the 2D map shown on the left of this Figure, the “AR map” on the right
side is much easier to use to find a direction. It takes time to find the correct direction
The Australian National University | 10
to go in a 2D map navigation system. However, with implementing AR on a map, the
direction is shown obviously, restaurants and shops are marked. In addition to the
“outside” case shown in Figure 1, a 2D map does not support navigation inside of a
building and people are often likely to get lost inside of large buildings because of the
complex building structure. Thus, this project aims to design an AR map application
includes inside and outside building and evaluate how it helps people for wayfinding in
the real world. The challenge in this project will be a low accuracy of GPS service
inside a building. Finding a good solution will be main problem for indoor map.
Figure 1: 2D mobile map and AR map (Google ARMap)
1.3 Structure of the report
There will be five parts to this report. Firstly, related works are outlined: what other
researchers have done about Augmented Reality technology, how they implemented it
on devices and problems in wayfinding will be discussed. The project plan will be
illustrated in the second part; detailed plans will be explained, and this will be an order
of implementation. The third part describes implementation, according to the project
plan; what technologies are used in AR map and how it was implements will be written
here. Evaluation will be discussed in the next part; some Heuristic evaluation and
The Australian National University | 11
expert-based evaluation was performed, and their results will be described. Based on
the result of the evaluation, some improvements were made to the prototype. The last
part will be future work; what other techniques can be used to improve the map will be
discussed here.
2 Background
2.1 Augmented Reality
Augmented Reality (AR) is an interactive technology concerned with combining data
from the real world with computer generated data. Typically, objects and information
from the real world are augmented by virtual objects and other information generated
by a computer (Schueffel, 2017). This technology makes it possible to interact with the
virtual world by matching it with the real world directly, rather than simply display
information in the screen, relevant information or objects can be labeled and highlighted,
and this makes interacting more convenient. Augmented Reality is shown in Milgram's
Reality-Virtuality Continuum as a span between the real world and virtual world which
is showed in Figure 2 (Milgram et al., 1995). As shown in this figure, Augmented
Reality is much closer to the real world while Augmented Virtuality is closer to the
virtual world. Virtual information is brought to users' real-world and surroundings,
which enhances the perception of the real world and provides a more convenient way
of life to the user(Carmigniani et al., 2011).
The Australian National University | 12
Figure 2: Span in Milgram's Reality-Virtuality (Milgram et al., 1995)
Figure 3 (Report et al., 2016) shows how AR displays information on an object. Three
ways are shown in the figure: (1) Spatial AR projects information onto the real object
by a projector. It shows all information in the real-world. In spatial AR people do not
need to wear or hold a device; simple examples are teaching-used projector and
holographic display. (2) The second way is to display information in a hand-held device
such as a phone, all information is shown in a screen. The real object will be detected
by camera, then the information is processed by a phone or computer and then display
the information in the screen. (3) The third way is to have a head-attached device such
as a Microsoft HoloLens (ref). All information is displayed on the glass lenses of such
a display. In the present project, only the second way will be discussed and used.
The Australian National University | 13
Figure 3: How Virtual Information Display(Report et al., 2016)
2.2 GPS
To implement a navigation system, the most important part is to locate device and
update location when the device is moving. To obtain location, Global Positioning
System (GPS) is used. GPS is a system provides position, navigation and timing
services based on space. It uses satellites and ground monitoring stations to provide
signals for people to track, measure and locate(Hegarty, 2017). GPS technology has
been applied in many fields such as vehicles and aircraft navigation and mobile phone
track. In this project, mobile phone is a GPS receiver, location information is obtained
to implement a navigation system.
The Australian National University | 14
2.3 AR in wayfinding
Nowadays, AR has been applied in many different navigation systems. A vehicle based
AR navigation system was designed by Narzt et al. (2006); they pointed out the
advantage of AR in navigation is that user can easily perceive virtual information and
the real-world because they are in the same place. They also pointed out some AR
design concepts such like the path line should be in a semi-translucent color to avoid
covering the road.
The workflow of this program is shown in Figure 5. The program uses map data to
calculate a planning route, then uses location information to track vehicles for further
route planning. Finally, applications render an AR path aligned with a real-life scene
captured by a camera with video input. As shown in Figure 4, the AR renderer calculates
information from route, GPS and topography to render a virtual 3D model path, then
matches it with a live view. Finally, a navigation path is shown in the screen of a vehicle
navigation device, augmented reality navigation is implemented.
The Australian National University | 15
Figure 4: Narzt’s program (Narzt et al., 2006)
Figure 5: Narzt's program workflow(Narzt et al., 2006)
The Australian National University | 16
An indoor AR wayfinding application was designed by Kim and Jun (2008) with using
a head-wear device. Three different positioning ways were introduced, including GPS-
based, Sensor-based and RFID tag-based. They found that while GPS-based works not
well and Sensor-based needs extensive wiring, the RFID tag-based was possible to use
well.
In Kim and Jun’s paper, indoor location was calculated by marker detection and image
matching. As shown in Figure 6, their application shows a wayfinding path is quite
different from other AR navigation application. As many AR navigation application use
a line match road, such as the example in Figure 4, while his application uses a small
2D map and arrow to implement wayfinding shown in Figure 7. Furthermore, they use
a marker detection to fix position solve the problem that GPS has low accuracy indoors.
Figure 6: Kim & Jun's Program(Kim and Jun, 2008)
The Australian National University | 17
Figure 7: Kim & Jun program workflow
The Australian National University | 18
3 Project Plan
The plan for this project was divided into 4 parts:
1. The first part was to become familiar with the Unity3D platform and to match
locations from the virtual world to real-world. Unity3D is an excellent platform
to develop an AR environment and is widely used by researchers in AR. It
includes many AR Application Program Interfaces (APIs) such as Mapbox and
ARCore from Google and Vuforia respectively. This stage was thought to take
2 to 3 weeks.
2. In order to develop a map, matching the user’s present location is important.
Stage 2 of the project combined an outdoor 2-D map with an AR map. Although
this project focuses on an AR map, when the user chooses where they want to
go, an AR map by itself will not provide a specific location, so a 2D map is
necessary for the user to confirm their destination. This stage was thought to
take 4 to 5 weeks.
3. Stage 3 of the project involved the construction of an indoor AR map. This map
provides navigation inside of a building. Because of the complexities of a
building structure, it is sometimes difficult to reach a specific room or reception.
This part of the project implemented navigation from the ground in front of a
school building on the ANU Campus to an office inside the building. This stage
was thought to take 4 to 5 weeks
4. The final stage of the project was evaluation and improvement based on the
results of that evaluation. This stage was thought to take 3 to 4 weeks.
The Australian National University | 19
4 Implementation
This section introduces the implementation of this project, according to its project plan
and describes the technology and tools. Unity3D is a cross-platform game engine
released in June 2005 developed by Unity3D Technologies(Contributors, 2019). Three-
dimensional and two-dimensional games can be created by Unity3D, and by using an
API called ARCore develop by Google(Amadeo, 2017). AR can also be implemented
in Unity3D. The Unity3D API Mapbox is used in wayfinding in this project, Mapbox
provides custom online maps data(The New Cartographers, 2013). In this project the
map prototype is design based on the frame in Mapbox named ‘WorldScaleAR’.
Additional function will be added to this frame for an outdoor map. As for indoor map,
the indoor map will be rendered manually, navigation will be implemented by using
NavMesh in Unity3D. Different map will use different camera in Unity3D as different
camera will have different layer view levels.
4.1 Development environment
All the development environment and software version are listed in Table 1 below.
The Australian National University | 20
Name Version Note
Unity3D 2018.2f or
later
Game Engine, support Google AR
Core, main development platform in
this project
Unity Mapbox Plug-in v.1.4.5 or later Map database, contains Google AR
Core, contains function to get device
GPS location, contains function to
transfer GPS location to coordinate in
Unity
OS Windows 10 Main operation system
Android SDK 24 or later Support AR Core
Android System 7.0 or later Support AR Core
Microsoft Visual Studio 2017 Platform to edit code in Unity3D
Google Pixel Phone 3 Device to run and test prototype, it
should support AR Core
Table 1: Development environment
4.2 Import Mapbox-plugin
The Mapbox plug-in (v.1.4.5) is downloaded from Mapbox Unity(Mapbox.com, 2019),
then import the package in Unity. After importing the package, in Mapbox setup menu,
an access token should be input to Mapbox and then obtain permission to get map data.
Access token can be obtained from Mapbox website after registering an
account(Mapbox.com, 2019). As shown in Figure 8, after install Mapbox package and
input the access code obtained from website, a program frame can be setup.
The Australian National University | 21
Figure 8: Mapbox Setup in Unity3D
4.3 Match GPS and initialize the map
This AR Map matches location by using the GPS service in Unity3D. It gets GPS
information from a device by using script on GameObject called ‘LocationProvider’.
This script uses build-in class named ‘LocationService’ to obtain GPS information from
device(Unity - Scripting API: LocationService, 2019). After getting GPS information,
a function called ‘GeoToWorldPosition’ will be used. This transfers the GPS location
to Unity3D world location and then initializes map by setting this GPS location as the
world center and then the program gets map data from Unity3D’s Map Box. As shown
in Figure 9, this method takes the latitude and longitude map’s GPS information and a
scale (in meters) as input and returns a 2D vector as the location in the Unity3D world.
This initializes the map at Unity3D’s location refPoint and sets this point with known
longitude and latitude.
The Australian National University | 22
Figure 9: GeoToWorldPosition Method.
4.4 Navigation in 2D map
2D map workflow is shown in Figure 11. Firstly, program gets location information
from device by using LocationService in Unity3D. Then the 2D map will be initialized
based on start and end points. The start point will be set default by the location of the
device, and the endpoint can be set by a user using the screen or inputting a GPS
location in the program in advance. Then the location of these points will be converted
from function introduced in 4.3. Then these locations will be used to translate these
points to their location in Unity3D world. After initializing map, the possible path will
be calculated based on map data from Mapbox. Then a navigation line will be drawn
from start point to destination on walkable roads and shown on the screen. The line will
be drawn by the Unity3D in-built function LineRenderer. This function inputs start and
end positions and will draw a line from start to end position. The line type and color
can be set at the LineRenderer setting interface in Unity3D. From Figure 10, a blue line
from device to the destination can be seen. Implementing a 2D map lets people confirm
their destination, because in an AR map only surrounding environment can be seen on
the screen, when users is far away from the destination, it will be hard to confirm the
destination.
The Australian National University | 23
Figure 10: Screenshot from 2D map
4.5 Navigation outside the building
In the prototype, AR outdoor navigation will be implemented based on a 2D map. Based
on the 2D map data, building models will be rendered according to their location by
Mapbox. For those building which are new and not contained in Mapbox database, an
API called Points of Interest (POIs) will make it possible for a designer to render a
building or add a tag to building (POI placement | Maps SDK for Unity | Mapbox, 2019).
In the prototype, the Hanna Neumann building (Building 145) at ANU was built by POI
builder in this way. The navigation path will be the same as the navigation line rendered
in the 2D map because it is based on the same map data. Besides, this reduces repeating
calculation and rendering. The workflow of the AR outdoor map is shown in Figure 11.
The Australian National University | 24
Figure 11: Workflow for 2D and AR outdoor map
4.6 Navigation inside the building
To fix low accuracy of GPS problem, the application will use button to synchronize
location. It is implemented by moving camera to the check points to match the location
with real world.
In the application, the AR map inside a building is based on the Unity API called
NavMesh. NavMesh can let Unity render and set walkable areas and obstacles and
navigation can be implemented(Unity - Manual: NavMesh building components, 2019).
By rendering the model based on blueprint of the building, an indoor map is formed.
The floor areas are set to be “walkable” and walls are set to be “obstacles”. The
destination location of offices and rooms in building are set with tags. As Unity3D
employs a first-person game-playing paradigm, it is necessary to create a “player” and
to add components called NavMesh Agent for making the player as a start point for this
navigation. To make sure that the path line can be rendered and remaining for
pathfinding, it is necessary to set the player's speed to zero. Then a destination point
can be set by clicking the button. After setting the start point and destination point,
LineRenderer will be used to render the path. For multiple floor navigation, multiple
players will be set for each floor. Direction arrows will be rendered under the player to
make user know the direction they should go. The workflow of AR indoor map is shown
in Figure 12.
The Australian National University | 25
Figure 12: AR indoor map workflow
4.7 NavMesh
NavMesh is a singleton Class in Unity3D, it helps to achieve pathfinding and
walkability tests in a 3D space(Unity - Manual: NavMesh building components, 2019).
To build a NavMesh pathfinding system, a walkable plane must be set as a pathfinding
area, then it must be set to static and walkable. Several destination points are set on the
plane. According to the building structure in the real-world, walls are rendered and set
to be static and unwalkable. To set the walkability of objects, Navigation page will be
used, it can be opened by select menu bar- Window -> Navigation. After rendering all
staffs in the map, a player should be set on the plane, and then add a component called
‘NavMeshAgent’ on this player. This makes the player be a start point in NavMesh
system. Before the program be tested, the map should be baked in Unity3D in the bake
bar in Navigation page. The player size can also be set here by setting height and radius
by selecting Agent bar. As shown in Figure 13, after baking the map, there will be a
blue plane as walkable area formed according to the walkable place and player’s size.
As shown in Figure 14, when the program runs, a blue line is rendered from the player
to destination.
To achieve pathfinding, there must be a script adding on the player to set its destination.
Destination can be set by obtain an object’s location on the plane and then set it as
destination with build-in function ‘NavMeshAgent.SetDestination()’. After setting
The Australian National University | 26
destination, when the program is initialized, the player will move to the destination on
walkable area. In order to render a pathfinding line, all corner points on a path are gotten
from NavMeshAgent and use LineRenderer, a class in Unity3D for rendering line, to
render line from corner point to corner point. The player’s movement speed should be
set as 0 to remain a complete path line. As a result, there will be a path line on the plane
from player to the destination.
Figure 13: NavMesh without path rendering
The Australian National University | 27
Figure 14: NavMesh with rendering path
The Australian National University | 28
4.8 User interface design
Controls for the user interface are divided into four parts:
(i) Map type selection,
(ii) Set checkpoint,
(iii) Location debug,
(iv) Set destination.
As shown in Figure 15, controls to set a checkpoint are shown inside the red boundary,
controls to set the destination are shown inside the yellow boundary, the green boundary
contains controls debugging and the blue boundary contains controls to select different
types of map. In the prototype there are three options for map type: ARO for AR map
outdoor, ARI for AR map indoor and Map for 2D map. Buttons are designed by code
and each button is linked to target function, and button location is based on the width
and height of the screen. Buttons in red boundary will enable user to link the present
device location with target points for in the real-world. Buttons in the green boundary
will rotate the screen in order to debug direction. Buttons in the yellow boundary will
set a destination to a target point. As shown in Figure 15, the room “MSI-235” can be
seen, and a blue line as a wayfinding path for navigation.
The Australian National University | 29
Figure 15: Screenshot of indoor map prototype
The Australian National University | 30
5 Evaluation
Usability principles proposed by Ko et al. (2013), and Human Centered Interaction
(HCI) principles proposed by Dünser et al. (2007) will be described in section 5.1.
Based on these principles, Heuristic Evaluation was designed and conducted to evaluate
the prototype application construction described above in section 5.2. Both design
process and the evaluation will be discussed. Reflecting on the results of Heuristic
Evaluation, several adjustments were made to improve the prototype application, and
these are also discussed in section 5.3.
5.1 Heuristic Evaluation Principles
Ko et al. (2013) categorized several AR usability principles into five groups, which are
User-information, User-cognitive, User-support, User-interaction, User-usage. Each
group has several usability principles making up 22 principles in total as shown in Table
2 below.
Group Name Principle Name Principle Definition
User-information Defaults The application should be easy for a user to use and
provide an interaction interface that includes input
buttons and that shows corresponding output.
Enjoyment Use colourful and interesting interface components,
such as colourful buttons, icons, or text.
Familiarity Use familiar interface layout, as well as familiar icons
and familiar language.
Hierarchy Divide up large information display into different
levels to make it easier for a user to process.
Multi-modality Provide different modalities for information content
by combining visual display with sounds, feelings.
Visibility Properly design graphic features so that they can be
easily understood and recognized.
The Australian National University | 31
User-Cognitive Consistency Interface and terms use should be consistent to
prevent confusion.
Learnability Application functions and use should be easy and
effective for the user to learn
Predictability How a user reacts to the interface should be
predictable.
Recognition Necessary information should be provided to avoid
user using short-term memory.
User-Support Error
management
Errors occur while using application should be
prevented where they can be and have ready solutions
to fix when they cannot
Help and
documentation
Proper help should appear to help the user, such as
guidelines or tips
Personalization The user should be able to modify the interface style,
such as changing icon shape or text font and color.
User control The user should feel that they are controlling the
system.
User-Interaction Direct
manipulation
Graphical controls should be directly amenable to
user actions.
Feedback System state and process sequence should be shown
to the user visibly, immediately and consistently.
Low physical
effort
Minimize the effort of operating the application and
the tiredness of users.
Responsiveness React quickly to users’ action.
User-Usage Availability Rapid initialization, previous working state, and
options should remain when re-operating an
application.
Context-based UI design should adapt to a different environment.
Exiting The application should be easy to stop and exit.
Navigation Navigate freely in the application.
Table 2: Usability Principles for AR Application (Ko et al., 2013)
Dünser et al. (2007) also applied other some Human Computer Interaction (HCI)
principles to AR systems that help to evaluate an AR application. These principles
include affordance, reducing cognitive overhead, low physical effort, learnability, user
satisfaction, flexibility in use, responsiveness and feedback and error tolerance. Clearly
The Australian National University | 32
some of these principles are quite similar to the principles in Table 2. Those HCI
principles which are different from principles in Table 2 are listed in Table 3.
Principle name Description
Affordance Affordance in an AR system focuses on the subject-object
relationship model. In other words, use the functional shape of
the icon or button to make metaphor and let a user know the
function of the button.
Reducing cognitive
overhead
Reduce unnecessary cognitive features on the interface. Too
much non-automatic cognitive efforts can lead to users’
distraction and inefficiency.
User satisfaction Users’ feeling about interaction is important.
Table 3: HCI principles(Dünser et al., 2007)
5.2 Heuristic Evaluation Process
The principles shown in Table 4 will be used to perform a heuristic evaluation as
follows:
a) Task based evaluation will be undertaken by independent experts and by the
developer. During task-based evaluation, the evaluator will perform some
scripted tasks using the interface and make notes when and where they find
difficulties.
b) A static audit will be performed by examining and interacting with at least two
representative screenshots of the application. Once again, the developer and two
independent experts will undertake this evaluation.
The Australian National University | 33
Principle name Description Task-Based or Static-
Audit
Defaults The application should be easy for a user to
use and provide an interaction interface
that includes input buttons and that shows
corresponding output.
Task-based and static
audit
Hierarchy Divide up large information display into
different levels to make it easier for a user
to process.
Static audit
Visibility Properly design graphic features so that
they can be easily understood and
recognized
Static audit
Learnability Application functions and use should be
easy and effective for the user to learn
Task based.
Error
management
Errors occur while using application
should be prevented where they can be and
have ready solutions to fix when they
cannot
Task based and static
audit
Help and
documentation
Proper help should appear to help the user,
such as guidelines or tips
Static audit
User control The user should feel that they are
controlling the system.
Task based
Feedback System state and process sequence should
be shown to the user visibly, immediately
and consistently.
Task based
Responsiveness React quickly to users’ action. Task based
Affordance Affordance in an AR system focuses on the
subject-object relationship model. In other
words, use the functional shape of the icon
or button to make metaphor and let a user
know the function of the button.
Static audit
Table 4: Principles used in evaluation
The Australian National University | 34
5.2.1 Static audit Heuristic Evaluation
Based on principles in Table 4, principles marked with ‘static audit’ will be explained
and applied in this section, evaluation is based on several screenshots from prototype.
Defaults
Defaults needs application provide interface with input buttons and output display in
order to make it easy for users to use. For an AR wayfinding system, buttons should be
showed in screen and can be easily found by users. The output should be displayed with
matching the real-world and then show it in screen.
By applying this principle in the prototype, as shown in Figure 15, buttons are easily
found in screen and information for pathfinding can be seen matching with the real
building path.
Hierarchy
Hierarchy deals with large number of information shown on screen. An AR wayfinding
application including both the indoor map and the outdoor map will have much
information about locations both inside and outside a building.
To apply this principle, different maps selection is implemented in the prototype.
Information on an indoor and outdoor map is divided into different levels, the user can
select to use indoor or outdoor map. As shown in Figure 15, a selection module is in
the blue frame. By implementing hierarchy, it is much easier for the user to utilize map
and find target information they want.
The Australian National University | 35
Visibility
Visibility requires that graphic factors be designed properly, including using
understandable modeling and easily recognizable staffs. AR system shows all
information on the screen matched with the real world, so it is important to make
information understandable for the user. An AR map needs to match virtual information
with the real world, so it is important to render a building or landmark to help people
find the correct information and locate themselves.
By implementing visibility for the outdoor AR map, building models are rendered based
on real building models and an important building is tagged with that building's name.
Because the same model is shown in the device and real world and important building
will be tagged, users can easily find their destination. For the indoor AR map, there is
a flag called ‘synchronize point’ on the wall at each checkpoint for a user to synchronize
location, rooms, and offices are tagged with a name. All tags are visible and
recognizable easily. As shown in Figure 15, room name and navigation path are visible
for the user and it is easily recognized.
Error management
Some errors can be anticipated. AR wayfinding applications work based on GPS
services, and for a short-distance movement, the GPS coordinates might be incorrect as
there is an error associated with them. Besides, when a map renders a building and tags
with its name, this name might appear at the wrong location if the GPS coordinates are
stuck. Thus, implementing error management is important for AR way-finding systems.
Error management is applied by designing function to prevent and fix errors in the
application.
In the prototype, errors are fixed by adjusting the information location. To make sure
that information is at the right location, a rotate function is designed to rotate the camera
The Australian National University | 36
in Unity3D world. As shown in Figure 15, buttons in the green frame are used to fix
GPS errors.
Help and documentation
A guideline or instruction should be appeared when user use the application in order to
help user learn and use the application. For an AR system, a guideline should appear
when user open the application to teach user every function in the application.
Instructions should appear when complex function is used.
After applying this principle, there is no features that reflect this principle and this
problem will be listed in next section in Table 5.
Affordance
Affordance matches button’s function with its shape to make metaphor and lets a user
know the function of the button easily. In an AR system, this principle is linked with
button shape. For example, in the famous AR application game Pocket Mongo, when
player find a Pokémon, there is a Pocket ball on the screen and player can click and
move the ball on the Pokémon and then catch it. The pocket ball is linked with its
function.
By applying this principle, there is no features that reflect this principle and problem
will be listed in Table 5.
5.2.2 Problems found from static audit heuristic evaluation
Problems found during evaluation were majorly caused by limited or missed
application of some principles. Both description and reason of the problems and related
principles are listed in Table 5 below.
The Australian National University | 37
Problem Description Reason Related
Principles
Limited
information
There is no
information about
which button links
to which target
function
A user may get confused
because they do not know
which button to push to
implement their
requirements. This will
make it difficult for a user
to learn.
Defaults,
Learnability
Useless
buttons
The indoor map
button is showed on
all types of map. In
other maps, it will
not work and not
give feedback.
When a user uses the
outdoor map or 2D map, it
may be confusing because
these buttons do not work,
and the user may use it and
have no feedback.
Hierarchy,
Feedback, User
control,
Affordance
No guidelines
or
instructions
There are no
guidelines and
instructions telling
the user how to use
application
People who use the
application will get
confused about how to use
this map.
Help and
documentation
Limited error
management
No reference object
for a user to
confirm the debug
result.
To debug errors in the
prototype, the user has to
rotate information in the
device to match the real
world, but there is no
reference object to confirm
whether it is correct.
Error
management,
Help, and
documentation
No feedback
from the
system
There is no
feedback after user
push buttons.
A user may get confused
when they push buttons but
get no feedback, even the
prototype has finished the
button function.
Feedback, User
control,
Responsiveness
Table 5: Serious problems found from the prototype
The Australian National University | 38
5.2.3 Heuristic task-based evaluation
To do more evaluation, task-based evaluation is designed. Two tasks are designed,
include Task 1: Outside to Inside Task, and Task 2: Inside Multi-floor Task. Evaluation
principles in Table 4 which is marked with ‘Task based’ will be regard as reference
principles for giving feedback. Two task instructions are listed below.
Task-1: Outside to Inside Task
Instructions:
- Go outside as directed
- Switch APP on
- Have Outside AR mode selected
- Find MSI building in graphic display
- Rotate Left/Right to align MSI building graphic with the actual MSI building
- Choose target destination
- Walk into MSI along the blue line and enter MSI building to Synchronize
Point 1
Task-2: Inside Multi-Floors Task
Instructions:
- Follows on from Task 1 at Synchronize Point 1 with App running
- Select Inside AR mode
- Push point button with the same number point on the paper
- Look around use Phone to find the synchronize point in phone
- Use RotateLeft and RotateRight button to make synchronize point in the
phone same location with the synchronize paper in the real-world.
- Select a destination
- Walk to the destination along the blue line up the stairs
- Find next synchronize point
- Push point button with the same point number on it.
The Australian National University | 39
- Use RotateLeft and RotateRight button to make synchronize point in the
phone same location with the synchronize paper in the real-world.
- Walk to the destination along the blue line to next synchronize point
- Push point button with the same point number on it.
- Use RotateLeft and RotateRight button to make synchronize point in the
phone same location with the synchronize paper in the real-world.
- Walk to the destination along the blue line to next synchronize point
To do this task evaluation, two experts from CECS in ANU were invited, tasks were
done outside and inside level 1 and level 2 in Building 145 on ANU Campus.
Figure 16: Task device and materials
Figure 17: Check point when during task
The Australian National University | 40
After doing the tasks and analyses the results, lots of problems are found. According to
the feedback, serious and significative problems are listed in Table 6.
Problem Description
Menu Button is hard to click, and there is no feedback to tell user’s
finger press has been detected.
Text hard to read It is difficult to read white text without background.
Path not continuous Path flashes when user walk through.
UI text errors UI text has errors in synchronize point and Rotate buttons.
Check point issues Cannot identify checkpoints, distance between device and
checkpoints is confusing for synchronizing.
Table 6: Problem list from task-based evaluation
After analyzing the problems, the menu buttons need to be larger and need feedback
when finger is pressing it. The feedback text should have a background and use a clear
color to make it easy to read. Path flashes because the GPS has low accuracy, so there
should be a function to fix it. Any text errors should be fixed. There is no way to find
and identify check points and there is no reference shows how far users should stand in
front of the check point.
The feedbacks on task-based principles from two experts are also analyzed. The task-
based principles and feedbacks after analyzed are shown in Table 7.
The Australian National University | 41
Principle Name Principle Definition COMMENTS
Defaults The application should be easy
for a user to use and provide an
interaction interface that includes
input buttons and that shows
corresponding output.
Generally satisfied. But there
is unused button, and output
text is hard to read.
Visibility Properly design graphic features
so that they can be easily
understood and recognized
Clear visibility.
Learnability Application functions and use
should be easy and effective for
the user to learn
Application is difficult to
learn without help for a
person who uses this
application for the first time.
Error
management
Errors occur while using
application should be prevented
where they can be and have ready
solutions to fix when they cannot
No fix error function.
User control The user should feel that they are
controlling the system.
Well user control but it still
needs instructions.
Feedback System state and process
sequence should be shown to the
user visibly, immediately and
consistently.
Application only has
feedback on Mode selection
and Point section. No
feedback on system state.
Responsiveness React quickly to users’ action. Fairly quick.
Table 7: Feedbacks on principles
By analyzing the feedbacks on principles, error management needs more improvement
and feedbacks from system need to be more. Some instructions and guideline should
appear to help user learn how to use the application.
The Australian National University | 42
5.3 Improvement based on evaluation
The prototype is improved based on the results from heuristic evaluation. The user
interface is redesigned, references to synchronize check point and more feedbacks are
added. The new prototype is shown in Figure 18 and Figure 19. The buttons are much
larger for easily clicking. When user press button, the phone will vibrate. To make the
text easily read, all texts are in red. New references for check point are shown in Figure
19, the red circular plane is for user to synchronize check point. The path line is changed
to use a semi-transparent color for avoid covering road.
Figure 18: Screenshot of the new prototype
The Australian National University | 43
Figure 19: Screenshot of new prototype with reference check point
6 Limitations and Future Work
6.1 Limitations
This report describes the design and evaluation of an AR wayfinding prototype. The
major novel part of this prototype is the implementation of an indoor wayfinding system.
Its major limitations are the GPS accuracy inside of a building, the difficulty of
manually synchronizing device location, and the rendering of the indoor map. With the
present accuracy of GPS position systems, it is difficult to locate the user device in
particular places of a building in real time. In any case, the GPS coordinates do not
account for elevation and cannot distinguish different floors of the one building. This
leads to wrong locations of tags and path in AR system and make it hard to achieve
wayfinding. To cope with these deficiencies of GPS tracking, users have to manually
synchronize at each check point to adjust device locations. Manual synchronizing takes
The Australian National University | 44
time to finish and provides a potentially bad experience when using the prototype.
Though many excellent outdoor map databases exist already, indoor maps still need to
be rendered according to a building structure by designers. It is hard to popularize an
indoor map because it takes time and resources to render an accurate indoor map, and
proper rendering needs to use building blueprints. Design of the prototype does not
cover all the principles due to limited time. Task -based evaluation was done by people
who was familiar with the environment.
6.2 Future Work
Future work on prototypes such as the one presented in this report should look towards
implementing a better positioning method for indoor (and outdoor) maps and designing
a good model to render building according to building blueprints.
To achieve a better positioning method, computer vision technology can be used for
recognizing building tags for relocation. This not only reduces manual synchronization
time, but also has a more accurate location than a manually-adjusted location. A better
positioning method is to combine recognition and GPS, when GPS is not accurate. Real
scenes should be input to the program for a camera to recognize and relocate.
To achieve a good model to render an indoor map, there should be an algorithm to
recognize a blueprint image of a building. After recognition, an indoor map will be
rendered automatically according to the results of its input. Important rooms and space
should be marked and set as destination points.
More principles should be covered to design the prototype and have more evaluation
based on these principles. Task-based evaluation should invite more users to do tasks
to give more feedbacks.
The Australian National University | 45
7 Conclusion
This project has designed and evaluated a mobile wayfinding application that has been
implemented using AR technology. The application has implemented a simple
navigation system on the campus of ANU. GPS position and tag-based position were
used. The Mapbox and NavMesh packages in Unity3D were used to implement
navigation system. Heuristic Evaluation was designed and applied to the prototype.
Static audit and task-based evaluation was performed by two experts. Following the
evaluation results, problems found in the prototype were corrected to improve the
prototype. Future work in this area will need to design and implement better positioning,
particularly, inside of buildings and it will also need to further improve the user
interface.
The Australian National University | 46
8 Bibliography
Amadeo, R. (2017) ‘Google’s ARCore brings augmented reality to millions of Android
devices’, Ars Technica. Available at: https://arstechnica.com/gadgets/2017/08/googles-
arcore-brings-augmented-reality-to-millions-of-android-devices/ (Accessed: 14
October 2018).
Carmigniani, J. et al. (2011) ‘Augmented reality technologies, systems and
applications’, Multimedia Tools and Applications, 51(1), pp. 341–377. doi:
10.1007/s11042-010-0660-6.
Contributors, W. (2019) Unity (game engine), Wikipedia, The Free Encyclopedia.
Available at:
https://en.wikipedia.org/w/index.php?title=Unity_(game_engine)&oldid=892297050
(Accessed: 4 April 2019).
Dünser, A. et al. (2007) ‘Applying HCI Principles to AR Systems Design’, Proceedings
of 2nd International Workshop on Mixed Reality User Interfaces: Specification,
Authoring, Adaptation (MRUI ’07), (March 11), pp. 37–42.
Hegarty, C. J. (2017) ‘The Global Positioning System (GPS)’, Springer Handbook of
Global Navigation Satellite Systems, 50(3), pp. 197–218. doi: 10.1007/978-3-319-
42928-1_7.
Kim, J. and Jun, H. (2008) ‘Vision-Based Location Positioning using Augmented
Reality for Indoor Navigation’, 54(3), pp. 954–962.
Ko, S. M. et al. (2013) ‘Usability Principles for Augmented Reality Applications in a
Smartphone Environment Usability Principles for Augmented Reality Applications in
a Smartphone Environment’, 7318. doi: 10.1080/10447318.2012.722466.
The Australian National University | 47
Mapbox (no date) Unity | Mapbox, 2019. Available at: https://www.mapbox.com/unity/
(Accessed: 27 April 2019).
Milgram, P. et al. (1995) ‘<title>Augmented reality: a class of displays on the
reality-virtuality continuum</title>’;, (December 1995), pp. 282–292. doi:
10.1117/12.197321.
Narzt, W. et al. (2006) ‘Augmented reality navigation systems’, Universal Access in
the Information Society, 4(3), pp. 177–187. doi: 10.1007/s10209-005-0017-5.
POI placement | Maps SDK for Unity | Mapbox (2019). Available at:
https://docs.mapbox.com/unity/maps/examples/poi-placement/ (Accessed: 17 April
2019).
Report, T. et al. (2016) ‘Augmented Reality : Technologies , Applications , and
Limitations’, (April 2007). doi: 10.13140/RG.2.1.1874.7929.
Schueffel, P. (2017) The Concise Fintech Compendium. Fribourg: School of
Management Fribourg/Switzerland. Available at: http://www.heg-fr.ch/EN/School-of-
Management/Communication-and-
Events/events/Pages/EventViewer.aspx?Event=patrick-schuffel.aspx (Accessed: 14
October 2018).
The New Cartographers (2013). The Washington Post. Available at:
https://www.washingtonpost.com/sf/brand-connect/wp/2013/07/22/the-new-
cartographers (Accessed: 14 October 2018).
Unity - Manual: NavMesh building components (2019). Available at:
https://docs.unity3d.com/2019.1/Documentation/Manual/NavMesh-
BuildingComponents.html (Accessed: 17 April 2019).
The Australian National University | 48
Unity - Scripting API: LocationService (2019). Available at:
https://docs.unity3d.com/ScriptReference/LocationService.html (Accessed: 28 May
2019).
9 Appendix 1 Product Screenshot
Figure 20: Workspace in Unity3D
The Australian National University | 49
Figure 21: Original prototype screenshot 1
The Australian National University | 50
Figure 22: Original prototype screenshot 2
The Australian National University | 51
Figure 23: Original prototype screenshot 3
The Australian National University | 52
Figure 24: Original prototype screenshot 4
The Australian National University | 53
Figure 25: Original prototype screenshot 5
The Australian National University | 54
Figure 26: New prototype screenshot 1
The Australian National University | 55
Figure 27: New prototype screenshot 2
The Australian National University | 56
Figure 28: New prototype screenshot 3
10 Appendix 2 Contract
INDEPENDENT STUDY CONTRACT
PROJECTS
Note: Enrolment is subject to approval by the course convenor
SECTION A (Students and Supervisors)
UniID: ___u6015337 _________
SURNAME: ___Zhang_____________ FIRST NAMES: ___Zhibo__________________
PROJECT SUPERVISOR (may be external): ________Henry Gardner___________________________
FORMAL SUPERVISOR (if different, must be an RSSCS academic): ___________________________________
COURSE CODE, TITLE AND UNITS: ________COMP 4560___12 unit_________________________
COMMENCING SEMESTER S1 S2 YEAR: _____ _ Two-semester project (12u courses only): X
PROJECT TITLE:
Augmented Reality with Phones for Way Finding and Help
The Australian National University | 58
LEARNING OBJECTIVES:
Students will build an augmented reality application that will run on a modern smartphone. This application will
provide assistance with way finding in a local area of campus or its surrounds. Students will learn how to evaluate
their application and other applications using appropriate data and heuristics. Students will learn how to write a
formal report.
PROJECT DESCRIPTION:
Augmented reality typically involves overlaying graphical objects on top of a live video stream. When effective
geolocation is possible, the graphical objects can be used to provide information about the scene or with reference
to the scene. A popular application is the idea of “way finding” where a user will point a device such as a smart
phone at a scene and the scene will be overlayed with an arrow pointing where they might want to go next. This
project will implement a useful way-finding application for use in CECS, the ANU or the near vicinity.
There are various issues to do with the usability of augmented reality applications that need to be kept in mind
when building applications such as way finding. The project will employ a form of “heuristic evaluation” to
evaluate its effectiveness and suggest improvements. If time allows, the project will be revised following this
evaluation.
The Australian National University | 59
ASSESSMENT (as per the project course’s rules web page, with any differences noted below).
Assessed project components: % of mark Due date Evaluated by:
Report: style: ______________45_________________
(e.g. research report, software description...,)
(min 45, def 60)
Ben Swift
Artefact: kind: _____________45________
(e.g. software, user interface, robot...,)
(max 45, def 30) (supervisor)
Presentation :
(10) (course convenor)
MEETING DATES (IF KNOWN):
STUDENT DECLARATION: I agree to fulfil the above defined contract:
Signature Date
The Australian National University | 60
SECTION B (Supervisor):
I am willing to supervise and support this project. I have checked the student's academic record
and believe this student can complete the project. I nominate the following examiner, and have obtained
their consent to review the report (via signature below or attached email)
Signature Date: 27/07/2018
Examiner:
Name: Ben Swift Signature
(Nominated examiners may be subject to change on request by the supervisor or course convenor)
REQUIRED DEPARTMENT RESOURCES:
SECTION C (Course convenor approval)
………………………………………………….. ………………………..
Signature Date
The Australian National University | 61
11 Appendix 3 Evaluation
The Australian National University | 62
The Australian National University | 63
TASK BASED HEURISTIC EVALUATION
Completed by Henry Gardner on Thursday 2 May, 2019
TASK 1 NOTES
Issues
- I did not see the graphical buildings at first.
- I did not know that I needed to rotate to find the MSI building
- What was the “log” button there for?
- I could not read some menu items that had white writing and no background. Perhaps these could be
black writing or black writing with white shading?
- I found the menu items to be hard to click
- Menu items did not change shape or color to tell me that my thumb/finger press had been detected
- I needed to rotate left and right with repeated clicks. Perhaps have a click and hold?
- The location seems to drift. I have it sometimes and then it moves.
- The orange blocks seem to grow and compress. Perhaps this is because the display is actually
drifting.
- Note: it is hard to make notes while I am performing this test. It would be better to record my
thoughts and then play them back later.
TASK 2 NOTES
Issues:
- White text menu items as before.
- White text feedback printed to screen; perhaps the screen has too much information on it?
- Cannot click buttons easily. Particularly when buttons are between other buttons.
- Path is not continuous. It flashes into view as you move.
- Path did not go up the stairs
- Synchronise point 2 was swapped with the graphic for synchronise point 3
- Rotate left and right moved the images the wrong way! When I clicked RotateLeft the image moved
right.
- I found myself wanting to zoom the display repeatedly but this facility does not seem to exist.
- The path can completely disappear. There should be some graphic that references it so that you know
that you need to rotate the images
The Australian National University | 64
- I touched the mode menu by mistake with my palm and the app swapped to Outside mode.
- What is Map mode? I did not use it at all. Why is it there?
Principle Name Principle Definition COMMENTS
Defaults The application should be easy for a user to
use and provide an interaction interface
that includes input buttons and that shows
corresponding output.
Generally satisfied. But there are
some issues. Input buttons are
unclear and difficult to press. No
indication that a button has detected
a press. Corresponding output (the
path) can be behind the observer.
There are unused buttons (LOG and
2D Map?)
The inside AR control set seemed to
be cluttered.
The use of white text with no
background was poor when outside.
Make it black or mixed colours (with
shading)?
Visibility Properly design graphic features so that
they can be easily understood and
recognized
Generally OK. I liked the graphic for
the path. It was larger near the screen
and it was clear what direction you
needed to go. The graphics for the
external buildings were confusing;
they did not really match the building
shapes that I could see.
The path can be completely invisible
if it is behind you.
Learnability Application functions and use should be
easy and effective for the user to learn
I really needed help to use this
application for the first time. But I
think that it would be easy to
remember for the next time.
Error
management
Errors occur while using application should
be prevented where they can be and have
ready solutions to fix when they cannot
There were several errors. It was not
clear how to fix them.
The Australian National University | 65
User control The user should feel that they are
controlling the system.
This was ok and it could be a lot better
with more feedback that buttons had
been pressed. Also help is needed to
locate the path if it is behind you.
Feedback System state and process sequence should
be shown to the user visibly, immediately
and consistently.
What are the system states? I think
that there are several. They are not all
communicated
- Mode – Inside/Outside (OK)
- Point selected (hard to read)
- Path out of range (not shown)
- Destination found (not
shown)
Responsiveness React quickly to users’ action. OK. But we need to have more
feedback.
Any other
observations:
The Australian National University | 66
TASK BASED HEURISTIC EVALUATION
Completed by Wanqi Zhao on 3rd May , 2019
TASK 2 NOTES
Issues:
1) Cannot identify the ‘checkpoint’ in the AR scene
2) Directions for synchronising the AR scene with the real-life scene (RotateLeft and RotateRight)
seem to be opposite, which is confusing
3) Not sure how far I should stand away from the synchronize paper to match the distance in the
AR scene
4) Need some notifications when I need to climb upstairs
Principle Name Principle Definition COMMENTS
Defaults The application should be easy for a user to
use and provide an interaction interface
that includes input buttons and that shows
corresponding output.
Buttons are provided for users to
interact with application. The
output is almost graphical, some
confusion may arise if the mismatch
happens.
Visibility Properly design graphic features so that
they can be easily understood and
recognized
Buttons are grouped as function.
The text labels have clear meaning.
Learnability Application functions and use should be
easy and effective for the user to learn
Button is very straight ward, and
the graphics (e.g. arrows and lines)
are easy to understand.
Error
management
Errors occur while using application should
be prevented where they can be and have
ready solutions to fix when they cannot
N/A
User control The user should feel that they are
controlling the system.
Kind of. I don’t feel quite
confidence to use without any
instructions.
The Australian National University | 67
Feedback System state and process sequence should
be shown to the user visibly, immediately
and consistently.
The system rarely provides
feedback. Some scenarios do need
some feedback such as when users
manage to synchronise the point or
when the user arrive one check
point
Responsiveness React quickly to users’ action. Fairly quick
Any other
observations:
1) Good to have checkpoints as I don’t have to hold the device all the time, but needs to consider that I
am an expert user instead of novice user in MSI navigation
2) AR has a huge advantage in navigation as the nature of the technology enables users know the reality
as they are navigated. So they can avoid obstacles.