Trust and the Internet of Things

Jon Robinson, Ian Wakeman, Dan Chalmers, and Ben Horsfall

Department of Informatics, University of Sussex, UK

Abstract. We present the design of the shoppingLense and its surrounding infra-structure. The shoppingLense is designed to allow open collaborative tagging ofpatterns within the environment, and for users to browse the environment usingaugmented reality. We use trust to control the presentation of the patterns andanchors within the augmented reality, building upon trust relationships that aredynamically created and maintained between the users of the system, yet maintainprivacy.We evaluate our approach through an experimental evaluation of our prototypein a program committee experiment, where the shoppingLense is used to browsethe virtual reviews attached to submitted papers. We show that performance issignificantly improved using a structured presentation based upon trust over anunstructured random presentation.

1 Introduction

The world is increasingly a virtual place. In the developed world, it is rare tofind a business that does not have a virtual presence on the Internet. But if wewish to visit the web-sites relevant to our current location, in most cases, wemust transcribe URLs into the browser, notwithstanding the increasing use ofBluetooth advertisements and the availability of QR Codes [1] in Japan. Onepromising approach to browsing reality is through the use of augmented reality,where graphical anchors on patterns within displayed video provide links off tothe relevant web-sites and other services.Our vision is of a world where everything and anything can be used as a pattern,and where everyone can both create patterns, and can add anchors to any pattern.If such a free-for-all is possible, how are people to distinguish the patterns andanchors that are of interest to them? Our view is that this should be through theuse of trust relationships, and ensuring that each pattern and anchor can have apseudonymous owner so that the identities can be bound within statements oftrust. We use pseudonyms to provide privacy to the users, whilst allowing ac-countability in the event of malfeasance. Further, there should be an additionallevel of indirection relating pseudonyms to long-lived groups, so providing theshadow of the future over the actions taken under cover of the pseudonyms, asdescribed in [2].In this paper we outline our basic architecture for an open platform to act asa pattern repository, and to allow annotation of patterns with anchors. We havebuilt a prototype system to prove our design, based around a standard Tomcatweb server, and a browser based on the ARToolkit which we have named theshoppingLense. Finally, we describe a short experiment to both demonstrate ourconcepts, and motivate our use of trust in the interface.

2 Overall Design

The AR Toolkit from [3] is a mature toolkit for overlaying graphics upon a streamof video. The toolkit uses fiducial patterns, which are pre-compiled into recog-nisable patterns that the ARToolkit can recognise within frames of video. Thesepatterns currently use a thick black border, but can use any pattern within theborder such as in figure 1.

Fig. 1. An example ARToolkit pattern

Given that any pattern within the ARToolkit border can be compiled into a recog-nisable pattern, the question arises as to who should be allowed to compile pat-terns. In a similar manner to the level playing field of the web, we believe thatanyone should be able to compile patterns, and that it should be left to the userwhich patterns they choose to display. Similarly, the anchors associated with eachpattern can be installed by anyone, again relying on the user to choose which an-chors to display on their device.If anyone can install patterns, and anyone can associate anchors with a pattern,how are we to prevent the AR-enhanced web falling prey to the same tragedy ofcommons as happened to the spam-deluged1 email system? Our approach is tobuild trust mechanisms into the system. We provide a priori policies based on the

1 We define spam in our system as unwanted or irrelevant patterns and anchors, rather thanunsolicited email.

group memberships of the pattern or anchor owner, and a more dynamic mech-anism based on ratings. By providing an extra level of indirection between theidentity and the trust, we provide more privacy, ensuring that only the informa-tion necessary to make trust judgements is communicated.Our aim is to deploy a robust system within a local shopping precinct, allow-ing the public to download and install a viewer on their mobile phone [4]. Thepatterns are printed and placed in shop windows or any other suitable location,and anchors are attached to the patterns. The patterns are uploaded and compiledthrough the precinct’s web-site, and anchors are likewise attached to the com-posed patterns within the web-site. We have developed a web-services based sys-tem to provide the pattern registry, trust model, pseudonym management, and an-chor and rating database. Importantly, each upload is identified with a pseudonymheld within the web-site, and so each pattern or anchor is digitally signed andowned by an identity e.g. the shopkeepers will own the patterns located withintheir shop windows, whilst customers will own the anchors taggged to the pat-tern.Our design is based on the centralised scheme described in [2]. Each pattern oranchor is cryptographically linked to a pseudonymous or real identity, using thestandard approach of an identity consisting of a public/private key pair, and sign-ing the digital material with the private key. Each identity can then be a memberof groups known to the user.The web-site provides the group abstraction, where each group is managed byone or more identifiable and real people. Users apply through the web-site tothe group manager to have their pseudonym accepted as a group member, and toprovide the necessary digitally signed signature that this is so. Since each grouphas their own criteria for deciding whether management of the group is appro-priate, the actual mechanism is supplemented by several side channels, such astelephones and email. Users can choose to provide group membership creden-tials along with the ownership signature over the pattern or anchor when the pat-tern/anchor is downloaded.Users can form policies based on group memberships, using the credentials of theowning identity to order the trust placed in each pattern and anchor. An exampleof a simple policy can be seen in figure 2. The default trust level is 0, representingno information. If multiple policies apply, then the following rule is applied: ifany are negative then the lowest trust value is taken (to highlight risk), otherwisethe highest positive value is used. This assumes that the magnitude of a reportedvalue comes about through building multiple experiences weighted by the level oftrust experienced (through exposure to risk), so that larger values convey a weightof experience. In this way, the complete policy specification can be applied to alltags but those groups which have little view on some tag have no impact. In thisway terms of reference for each group do not need to be defined. The memberOffunction tests the membership of the owner of the object, whilst the ratingfunction retrieves the set of ratings that match the membership argument. We canthen apply functions such as max, min and mean to this set of ratings.It is clear that not all users would write the policies from figure 2 in a text editorand maintain them. However, some common patterns of behaviour could be built-in with switches, and additional policies might be written and shared in the sameway other small scripts, plug-ins and applications are shared (e.g. firefox addonsor iphone applications). Much of this user-specific information could be extractedfrom other data, further simplifying policy construction: Traders’ associationswill control their group membership and users may choose to place default levelsof trust in such organisations (the example of 1.0 may not be the value chosen)and particularly familiar organisations. Groupings of friends could be established

if memberOf(NorthLaineTradersAssociation)then trust := 1.0

if memberof(IansFriends)then trust := 1.0

if memberOf(EthicalConsumers)then trust := 0.8

if max(rating(EthicalConsumers)) > 0.8then trust = 0.5

Fig. 2. Example policy declaration

from social networking sites, phone books etc. Similarly, consumer and interestgroups would be found by the user through other activities and membership mightcause a trust level question to be raised. We have not yet designed a process ofexploring available groups, but many familiar possibilities from the web, e.g.search, keyword indexes, tags, maps of common interest, and suggesting groupsbased on aggregate data are all possible.A user would thus interact with the system in the following way:

1. On entering the shopping precinct, they would be greeted with signs remind-ing them that they can download and install the application.

2. The application would then contact the precinct’s website, and download allthe patterns and anchors, along with their associated meta-information, suchas the certificate chains.

3. The user can install policies about which pseudonyms and groups are trusted,using simple membership predicates.

4. The application applies the policies over the downloaded patterns and an-chors, generating an ordering based on trust.

5. The user in strolling through the precinct can use the application to recog-nise the pattern from the video feed and display relevant anchors. Whichpatterns and anchors that are displayed on the screen is limited to the mosttrustworthy items under consideration, as in figure 3.

6. By clicking through into a conventional list and view screen (figure 4), theuser can see all the anchors associated with a given pattern, ordered by theirtrust.

We also provide the capability for registered pseudonyms to add ratings to a pat-tern and anchors, allowing more complex policies based on transitive trust re-lationships, and the ratings given by a particular group. So, having experienceda shop a user may feel moved to provide a rating – this is not new and manysites rate particular goods, manufacturers or shops from user feedback. To thiswe have added a mechanism to describe trust in those ratings through the expe-rience of individuals known by the user and also through individuals who havesimilar concerns to the user as expressed in group memberships. This informa-tion, which expresses trust in tagged entities (shop, goods) is itself subject to trustrelationships. The trust in the information expediates the process of forming anopinion while in-situ.

2.1 Interface Design

The shoppingLense interface is based around a video view, highlighting patternsthat are recognised, and annotating the pattern with the name that was attached to

the pattern by its owner, as illustrated in figure 3. We then display the most trustedanchors according to the user’s policy around the pattern. By trial and error, giventhe expected viewing distance of up to ten meters, we decided to limit the numberof anchors displayed around a pattern to four.When an anchor is selected, the mode of the viewer is changed to that of a stan-dard list box, displaying the text or URL associated with the anchor, as illustratedin figure 4. All the other anchors linked to the pattern are displayed in a list,ordered by the trust rating of the user.The level of trust in an anchor is indicated through colour coding (green beingmost trusted, orange neutral and red least trusted), the font weight of the text, andthe size of the text. Anchors have a title and a body text. The body text is currentlyrestricted to be either plain text or a URL. If the anchor is linked to a URL, thenthe URL is transferred to the appropriate browser when the anchor is clicked.Our initial prototype is built on a tablet PC (as seen in figure 5, and work isunder way to port the shoppingLense to Windows CE and other mobile telephonyplatforms. The pattern repository is fully functional, based around Tomcat. Weare currently improving the robustness of the repository for deployment in a realshopping precinct.

3 Validation Experiment: A Shadow ProgramCommittee

To validate our design, we have constructed an experiment to determine whetherusing trust to order the patterns and anchors is better than providing the anchorsin a random order. Our experiment is based around the idea of using the shoppin-gLense to browse the reviews upon a set of candidate papers for a conference soas to decide which papers are to be accepted, and which rejected.We chose 6 papers from a recent workshop, 3 of which were accepted and 3which were rejected. We used a population of 12 PhD students working withinthe general area of the workshop to generate 2 reviews per paper. The authors thengenerated one antagonistic review and one spam anchor for each paper, providinga total of 4 genuine reviews, 1 antagonistic review and 1 spam comment perpaper2. The four reviews from the student were given a trust rating of 1, theantagonistic review was given a trust rating of 0, and the spam assigned a valueof -1. The six papers were printed out and attached to display boards, along withan abstract pattern, as shown in figure 6.The experimental setup was used in the following two trials using unpaired andpaired evaluations. In both trials, the subject had a preliminary briefing and periodto familiarise themselves with the tool before undertaking the experiment in an-other room. Each participant was asked to enter the room, use the shoppingLenseto access the potted reviews and decide which three were accepted, and whichwere rejected. The total time taken to come to a decision, along with logs ofclicks and currently viewed image were recorded. The shoppingLense was con-figured to present the anchors either in an random order (unstructured), or in anordering based on the trust rating of the review (structured). In the unstructuredview the reviews were still presented with colour and font variations to indicatetrust levels in the review. Hence it is only the ordering of the reviews due to trustwhich might effect the outcome. The spatial positioning of the display boards waspermuted across participants.

2 e.g. Greetings for you! Someone has sent you a greeting card. Visit our web-site to find outwho!

Fig. 3. The Browser Display augmented video view

Fig. 4. The Browser Display - list view

Fig. 5. Paper examination usage

Unpaired Trial The original pool of PhD students that generated the reviewswere asked to review the six papers, deciding which of the six were to be acceptedor rejected. Of course there would be some familiarity, but not with sufficientpapers to make an immediate selection. The students were primed on the use ofthe shoppingLense, and were then allowed to investigate the papers freely usingthe shoppingLense. Ten students were able to complete the trial (the other twobeing unavailable while the trial was running). Five used the structured view,whilst the other five used the unstructured view. Four students were native Englishspeakers, whilst the others had been in England for at least two years.Whilst there was too much variation between subject competencies in languageand computer efficiency to allow direct comparison in either total time taken ornumber of clicks, all the students reported that the tool was easy to use. We tooknote of some suggestions about font and colour usage, and refined the shoppin-gLense before undertaking the following paired trial.

Paired Trial In this experiment, we asked 14 academics and post-doctoral re-searchers to evaluate two sets of three papers using both structured and unstruc-tured orderings of the shoppingLense. Which ordering came first was balancedacross participants. All participants were well-acquainted with the program com-mittee concept. Two of the participants were excluded from analysis, since theymisconstrued the task they were undertaking, based on their comments duringand after the experiment. In the analysis, the dependent variable was the timetaken to come to a decision, whilst the independent variables were the orderings.On average, participants completed the reviewing task significantly faster usingthe structured presentation (mean=220s, standard error=22s) than using the un-structured presentation (mean=250s, standard error=24s) at a significance levelof (p = 0.079, t(11) = −1.934, r = 0.503).

Fig. 6. The Experimental Layout

This experiment showed considerable practice effects. We undertook a mixed de-sign ANOVA, comparing the first and second trials as the repeated measures, andthe order of the structured/unstructured presentations as a between-group variablewithin SPSS. The effect of trials shows significance F (1, 10) = 9.28, p < 0.012,whilst the combined interaction shows significance F (1, 10) = 6.561, p < 0.028,as illustrated in figure 7. It can be seen that the structured / unstructured differ-ence was most obvious upon initial presentation, and that practice effects thendominated upon the second trial.

3.1 Lessons

What does this tell us about moving forward with the shop based system? Thefirst phase of the trial provided some important usability testing within a small-scale trial. The second phase tells us that the approach of using trust to influencedisplay is quickly understood (faster than learning effects) and does not hinderthe user. While a stronger positive effect might be more exciting, this is a usefulpositive outcome. We believe that the benefit of the ordering would become moreapparent with a greater volume of different opinions to process. If we assume thata large volume of largely untrusted reports might be generated over time in theshopping scenario, the ability to find trusted sources (directly trusted or throughsocial / interest networks) would be valuable, whilst retaining the open authorshipmodel. Given this assumption, our experiment design presents a low complexitysituation in which any benefit would quickly be hidden by any negative aspects ofthe system. However, we see a clear (if not large) and significant benefit, whichgives us confidence to proceed with introducing greater data volume, more com-plex policies and more complex experimental situations. Further, it was only the

150

170

190

210

230

250

270

290

310

Time1 Time2

Mar

gin

al M

ean

Tas

k T

ime

Structured/Unstructured

Unstructured/Structured

Fig. 7. Marginal means for trials versus ordering

ordering of the reviews which was varied. If we had removed other visual indica-tions of trust in the review it is likely that greater effort would have been put intothe different reviews by the participants and so strengthening our result further.

4 Related Work

Much of the work on adaptive hypermedia uses some measure of relevance to auser to determine link selection [5]. For instance, the work by Sinclair et al in[6] displays the links using augmented reality, based on the conventions existingin standard hypermedia systems. Our approach differs from adaptive hypermedianorms in that we use trust in a public authoring model, building from the workon rating schemes in ubiquitous computing, such as Quercia et al [7]. This itselfbuilds on work in the WWW, where ratings are used to build trust in contribu-tions to the web-site [8], and more recent work in peer to peer networks such asEigentrust [9] and the early work by Aberer [10].There is a large body of work on mobile augmented reality eg [11–15]. Of par-ticular interest to our goal of creating an augmented shopping precinct is workusing AR to supplement the consumer process. Most such work has focused onleaving control to the retailer and the location administrator, such as the shopbased vision in [16], or has concentrated on presenting work as a narrative [17,18]. Whilst such systems may be initially attractive, without open input, they maynot grow and continue to be useful after the original designers have left.

5 Conclusion

We have presented the design of an open augmented reality tagging system builtupon the ARToolkit, using the group memberships of the pattern and anchor own-ers to derive the level of trust that the user should invest in the viewed object. We

believe that this will provide an effective mechanism to control the pollution ef-fects of spam on collaborative AR.We have presented an experiment showing that using trust to structure the pre-sentation of the data results in higher task performance within the confines ofdisplayed paper browsing, but that practice effects dominate after users becomeaccustomed to the technology. We are deploying the system for further user trials.

Acknowledgements

We are very grateful to Ann Light for many useful discussions, the members ofour lab for reviewing and using the tool, to the willing participant academicswithin our department, to the members of the program committee of the Mid-dleware for Pervasive and Ad-Hoc Computing workshop, and to the anonymousauthors who allowed re-reviewing of their papers.

References

1. ISO, ed.: Information technology – Automatic identification and data cap-ture techniques – QR Code 2005 bar code symbology specification. 2 edn.Number 18004. ISO/IEC (2006)

2. Wakeman, I., Chalmers, D., Fry, M.: Reconciling privacy and security inpervasive computing. In: 5th International Workshop on Middleware forPervasive and Ad-Hoc Computing (MPAC07), Los Angeles, Ca (November2007)

3. Billinghurst, M., Kato, H.: Collaborative augmented reality. Communica-tions of the ACM 45(7) (July 2002) 64–70

4. Wagner, D., Schmalstieg, D.: First steps towards handheld augmented reality.In: Seventh IEEE International Symposium on Wearable Computers. (2003)

5. Kobsa, A., Koenemann, J., Pohl, W.: Personalised hypermedia presentationtechniques for improving online customer relationships. Knowl. Eng. Rev.16(2) (2001) 111–155

6. Sinclair, P., Martinez, K., Millard, D.E., Weal, M.J.: Links in the palm ofyour hand: tangible hypermedia using augmented reality. In: HYPERTEXT’02: Proceedings of the thirteenth ACM conference on Hypertext and hyper-media, New York, NY, USA, ACM (2002) 127–136

7. Quercia, D., Hailes, S., Capra, L.: Mobirate: making mobile raters stick totheir word. In: UbiComp ’08: Proceedings of the 10th international confer-ence on Ubiquitous computing, New York, NY, USA, ACM (2008) 212–221

8. Dellarocas, C.: Immunizing online reputation reporting systems against un-fair ratings and discriminatory behavior. In: EC ’00: Proceedings of the2nd ACM conference on Electronic commerce, New York, NY, USA, ACM(2000) 150–157

9. Kamvar, S.D., Schlosser, M.T., Garcia-Molina, H.: The eigentrust algorithmfor reputation management in p2p networks. In: Proceedings of the TwelfthInternational World Wide Web Conference. (2003)

10. Aberer, K., Despotovic, Z.: Managing trust in a peer-2-peer informationsystem. In Paques, H., Liu, L., Grossman, D., eds.: Proceedings of theTenth International Conference on Information and Knowledge Management(CIKM01), ACM Press (2001) 310–317

11. Feiner, S., MacIntyre, B., Hollerer, T., Webster, A.: A touring machine: Pro-totyping 3d mobile augmented reality systems for exploring the urban en-vironment. In: ISWC ’97: Proceedings of the 1st IEEE International Sym-posium on Wearable Computers, Washington, DC, USA, IEEE ComputerSociety (1997) 74

12. Wagner, D., Pintaric, T., Ledermann, F., Schmalstieg, D.: Towards massivelymulti-user augmented reality on handheld devices. In: Third InternationalConference on Pervasive Computing (Pervasive 2005), Munich, Germany(May 2005)

13. Newman, J., Ingram, D., Hopper, A.: Augmented reality in a wide area sen-tient environment. In: ISAR ’01: Proceedings of the IEEE and ACM In-ternational Symposium on Augmented Reality (ISAR’01), Washington, DC,USA, IEEE Computer Society (2001) 77

14. Reitmayr, G., Schmalstieg, D.: Location based applications for mobile aug-mented reality. In: AUIC ’03: Proceedings of the Fourth Australasian userinterface conference on User interfaces 2003, Darlinghurst, Australia, Aus-tralia, Australian Computer Society, Inc. (2003) 65–73

15. Kimura, H., Tokunaga, E., Nakajima, T.: System support for mobile aug-mented reality services. In: SAC ’07: Proceedings of the 2007 ACM sympo-sium on Applied computing, New York, NY, USA, ACM (2007) 1616–1623

16. Zhu, W., Owen, C.B., Li, H., Lee, J.H.: Personalized in-store e-commercewith the promopad: an augmented reality shopping assistant. Electronic Jour-nal for E-commerce Tools and Applications 1(3) (2004)

17. Hansen, F.A., Bouvin, N.O., Christensen, B.G., Gronbaek, K., Pedersen,T.B., Gagach, J.: Integrating the web and the world: contextual trails on themove. In: HYPERTEXT ’04: Proceedings of the fifteenth ACM conferenceon Hypertext and hypermedia, New York, NY, USA, ACM (2004) 98–107

18. van Doorn, M., van Loenen, E., de Vries, A.P.: Deconstructing ambient intel-ligence into ambient narratives: the intelligent shop window. In: Ambi-Sys’08: Proceedings of the 1st international conference on Ambient media andsystems, ICST, Brussels, Belgium, Belgium, ICST (Institute for ComputerSciences, Social-Informatics and Telecommunications Engineering) (2008)1–8