CollARt: a Tool for Creating 3D Photo Collages Using Mobile Augmented Reality

Asier Marzo Public University of Navarre

Pamplona, Spain

asier.marzo@unavarra.es

Oscar Ardaiz Public University of Navarre

Pamplona, Spain

oscar.ardaiz@unavarra.es

ABSTRACT

A collage is an artistic composition made by assembling different

parts to create a new whole. This procedure can be applied for

assembling tridimensional objects. In this paper we present

CollARt, a Mobile Augmented Reality application which permits

to create 3D photo collages. Virtual pieces are textured with

pictures taken with the camera and can be blended with real

objects. A preliminary user study (N=12) revealed that

participants were able to create interesting works of art. The

evaluation also suggested that the possibility of itinerantly mixing

virtual pieces with the real world increases creativity.

Categories and Subject Descriptors

H.5.m [Information Interfaces and Presentation]: Misc

Keywords

Collages; mobile; augmented reality; creativity

1. INTRODUCTION The collage technique consists in sticking together different

pieces. This technique has been used by some famous artists such

as Pablo Picasso and Marcel Duchamp. Nowadays it is properly

established and used by both art professional and amateurs.

Collages are composed of pieces from other works of art, pictures,

newspaper clips or bits of colored papers. All of these elements

are glued on a canvas forming a shallow creation. Nonetheless,

pieces may be glued tilted or perpendicularly in order to create a

tridimensional work. In the same way, stacking clippings of

pictures gives some depth to collages. Extending this idea, some

collages are made with physical objects or mock-ups; however,

their assemblage is more difficult.

Digital technologies can be used for reducing required technical

level to execute certain creative activities or to increase the

effectiveness and quality of creations. Technology lowers barriers

or gives new possibilities to artistic creations. It makes very

interesting the idea of creating digital counterparts of traditional

artistic techniques. For instance, Pottery [16] permits to mold and

decorate different kinds of virtual pots, vases or jugs. Digital

adaptations may not have the same haptic response as the real

activity. Nevertheless, they are endowed with additional features.

Options such as undo, redo and quick preview of outcomes

improve exploration. Additionally, collaboration with other users

and sharing through social networks are common features of

present applications. Exploration, collaboration and open

interchange are essential for creativity support tools [17].

Augmented reality is a visualization technique which overlaps

virtual content over the real world [1]. Advances made in the

recent years permit to implement this technique in handheld multi-

touch devices.

We have developed CollARt, a Mobile Augmented Reality

(MAR) application for creating 3D photo collages. A paper

marker is used for determining the device position. The paper is

the canvas in the real and in the virtual world. Therefore, the

application provides intuitive navigation and placement of new

pieces for the user. Furthermore, mobile platforms are widely

available and support an itinerant way of creating works of art.

2. RELATED WORK A number of applications allow the users to create collages

automatically. For instance, CollageMachine [11] assembles

images fetched from the Internet based on the preferences and

interests of the user. Similarly, StainedGlass [7] extracts frames

from a video and creates a digest of it in form of a photo

composition. Setting a target illustration an algorithm can place a

set of images in an empty canvas trying to fit the shapes, colors

and textures of the target picture [9]. In the same way as

Arcimboldo did in his paintings. The same principle can be

applied for assembling a set of tridimensional objects to conform

a target 3D volume [6]. PhotoPieces is a mobile application [15]

in which the users create photo collages filling with pictures a

template made of different regions. Templates are chosen from a

library or made by segmenting a photo.

Augmented Reality has been used for painting virtually over the

surface of real objects using a head mounted display (HMD) and

an electronic pencil [8]. Different textures can be used for the

stroke as well as solid colors or patterns captured from the real

world. In addition, virtual objects with premade textures can be

attached to the real entities. A field study conducted with a similar

system in an architecture school [10] revealed several benefits

derived from its tangibility; however a projector was used instead

of a HMD. Additionally, the system permitted to attach different

virtual objects to tangible markers improving the configurability

[2]. Small planar markers or panoramic views can be augmented

with virtual objects [12] and shared with other users. That system

is composed of a mobile augmented reality application and a

server which stores the creations. ColAR [5] recognizes the

outlines of a picture from a set of predefined ones. It can

reproduce a 3D animation using the real filling that the users

painted as textures for the objects. ARPaint [13] can augment a

picture permitting to trace strokes on the plane which contains it.

Our work differs from previous ones as CollARt is a multi-touch

mobile augmented reality application which merges virtual objects

with real objects. Furthermore, it permits to create tridimensional

photo collages naturally and its workflow encourages exploration.

Permission to make digital or hard copies of all or part of this work for personal

or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice

and the full citation on the first page. Copyrights for components of this work

owned by others than ACM must be honored. Abstracting with credit is

permitted. To copy otherwise, or republish, to post on servers or to redistribute

to lists, requires prior specific permission and/or a fee. Request permissions

from permissions@acm.org.

MM’13, October 21–25, 2013, Barcelona, Spain. Copyright © 2013 ACM 978-1-4503-2404-5/13/10...$15.00.

http://dx.doi.org/10.1145/2502081.2502154

585

3. COLLART CollARt is a mobile application which allows the user to create

3D compositions. Using augmented reality, virtual objects can be

blended with the real world. Pieces are textured with pictures

taken with the camera (Figure 1). When the paper marker is

detected by the application it shows the live feedback of the

camera and the virtual pieces overlapped with it. By touching the

screen the menu icons toggle visible, permitting to undo, redo,

clear, select or add a new piece.

Pressing the add icon makes the camera view to come out for

taking a picture. Afterwards the placement options permit you to

rotate, scale, adjust the texture or select between different shapes

for the current piece (Figure 2). Planar pieces such as squares,

circles or rectangles as well as tridimensional volumes, namely

cubes, spheres or pyramids are available. For the rotation mode

we used the interaction technique defined in [14]. The position

and rotation of the piece relative to the device will remain

constant. Additionally, one finger can be used to perform an

Arcball rotation in view space, and two fingers to rotate or change

the depth of the object across the view axis. Adjusting the texture

allows the user to modify the part of the previously taken picture

which is mapped to the current piece.

Whenever the marker is detected by the camera and placing mode

is active, the tick button appears. Pressing it will place a piece on

the current position. As a result, the marker is only needed to

place pieces or to observe the ongoing creation. Pressing the finish

button will return to the main menu. By pressing the select icon

the application switches to select mode. In this mode a crosshair is

shown, permitting to select any object for modifying it.

Figure 1. Tridimensional pieces textured with pictures taken

with the camera.

Figure 2. Placing one piece using the interface as seen by a

user on the mobile device screen.

Virtual objects are blended with real objects. Therefore, the real

pieces occlude and clip virtual objects (Figure 3). Visual

recognition in real time of the world objects would require a

considerable time of processing or to paint them with markers.

Consequently, several premade scenes were created, defining

virtual replicates of the real objects.

CollARt has been implemented using C++, OpenGL and GLSL.

The target platform is iOS and the library Vuforia [17] has been

used for tracking the marker.

Figure 3. A creation which blends virtual pieces with a real

canister.

4. USER EVALUATION A preliminary evaluation was conducted with 12 people (3 women

and 9 men), aged between 18 and 43 (M=27.75, SD=7.08).

Participants had heterogeneous professions and computer

experience in some cases. Two users were left-handed. Four of the

participants had formal artistic education and two of them worked

as professional artists.

The main purpose of the evaluation was to check if blending

virtual elements with real objects and itinerant creations are

interesting properties to the users. As a general objective we tried

to determine whether users were able to create original collages

with artistic value. Additionally, we aimed to detect usability

problems and annotate interesting behaviors of the users.

Searching for more concrete uses of the tools was another goal of

the study.

Participants performed the evaluation individually and without

time constraints. At the beginning they had to complete the

training activity. In it, users were asked to place four cubes at each

corner of the canvas and two squares perpendicularly between

them on the middle. The purpose was to teach the users how to

use the basic functions of the application. Accordingly, they were

assisted during it.

Four tasks were designed: table, vertical, cylinder and on the

move. In the table and vertical tasks the users had to create a

collage having no restriction on shapes, pictures or time. In the

table task the canvas rested horizontally on a table whereas in the

vertical task it was stuck to a wall, this simple orientation change

may affect usability and creations. The cylinder task consisted of

placing a real canister in the middle of the canvas. The real

cylinder clipped and hid virtual pieces influencing the collage as it

586

was another existing piece. In the on the move task, using indoor

and outdoor pictures was mandatory.

Users were split randomly into two groups. The first group

performed the training activity followed by the table and vertical

tasks. The second group did the training activity and subsequently

the cylinder and the on the move tasks. Therefore, group 2 was

more focused on exclusive features of the application such as

blending virtual pieces with reality and on the move use.

Figure 4. Various collages made by the participants.

Afterwards, all the participants filled in a questionnaire for

measuring the system usability (SUS) [3] and a series of question

for determining CollARt’s value as a creativity support tool (CSI)

[4]. Both SUS and CSI range from 0 to 100, the higher the value

the more usable or creativity supportive. In addition, they had to

score the ease and originality of each task using a one to five

Likert scale. Various open questions were answered by the users,

namely: what aspects have been more interesting?, what

difficulties have arisen using the tool?, in what places or context

would you use the tool? and can you devise a game to be played

with the tool?. T-tests were used for comparing both CSI and SUS

scores between the first group and second group. One-way-

ANOVA was used for the Ease and Originality scores between the

tasks as each participant performed only two of them.

The average CSI score reported by the participants was 71.66

(SD=7.52) for the group 1 and 73.88 (SD=7.57) for the group 2.

The score is higher in the group 2; however, there was not a

significant difference between the groups (t(10) = -0.50, p > 0.05).

The average SUS of the questionnaires was 72.08 (SD=11.77) for

the group 1 and 80.83 (SD=15.46) for the group 2. The score is

higher in the group 2. Nonetheless, there was not a significant

difference between the groups (t(9) = -1.10, p > 0.05).

The easiness of completing each task was 4.08 (SD=1.24) for the

training, 4.16 (SD=0.40) for the table task, 4.16 (SD=0.75) for the

vertical task, 4.16(SD=1.16) for the cylinder and 4.33 (SD=0.81)

for the on the move task; there was no significant difference

between the four tasks (F3,20 = 0.06, p > 0.05). The perceived

originality of the creation for each task was 2.25 (SD=1.05) for

the training, 3.5 (SD=1.37) for the table task, 3.66 (SD=1.36) for

the vertical task, 3.83 (SD=0.98) for the cylinder and 4.33

(SD=1.03) for the on the move task; there was no significant

difference between the four tasks (F3,20= 0.43, p > 0.05).

Mean time to perform the evaluation was approximately half an

hour. Users spent in average 10 minutes per task. Creations

represented buildings, creatures or abstract shapes (Figure 4). The

training period lasted five minutes in average and all users

managed to work with the application. Nevertheless, one user

created collages without sense of space as pieces were placed

randomly. In the same way five users did not use all the features,

specifically they did not scale pieces or adjusted the textures. All

of the participants took more than one picture and expressed to

enjoy the activity. Furthermore, six of them asked if the

application was available for later use.

5. DISCUSSION Features such as mixing virtual content with the real world and on

the move creations seem to provide better support for creativity

than other features of the application as the group 2 obtained

higher scores. Nonetheless, the difference was not significant and

since the evaluation was preliminary, CSI results were not

weighted. SUS scores were higher in the second group. The

reason is that the real cylinder intersects the virtual objects.

Therefore, it provides a better perception of the position and

orientation of the ongoing piece placement. According to the

questionnaires, placing the objects and perceiving their true

position was the most difficult aspect of using the application.

During the on the move task the marker was smaller, consequently

the users did not need to be so far from it. In that task they did not

have to connect anything with the cylinder. Those may be the

reasons of why they reported the task as the easiest one to

perform. The training was slightly more difficult as it required

placement of objects with a certain layout. Owing to the fact that

placing objects is the most problematic aspect, future

improvements will add snap to grid or vertex features. Another

possibility would be to add physic simulation or shadows to the

collage pieces. Professional 3D software may give more precise

results; however, the time required for teaching the users to

perform the same tasks is more elevated.

Originality was higher for the on the move task. Probably the

users had more objects to photograph. The cylinder task was not

the most original. Apparently the cylinder shape conditioned the

placement of the pieces. All the users agreed that the most

interesting aspect was to take pictures and apply them to

587

tridimensional objects using a quick workflow. Several of them

pointed out that integrating virtual objects with the cylinder was

interesting. Furthermore, one said that it would be interesting to

do it with famous buildings or paintings.

Suggested games could be grouped as map editors, interactive

simulations or replacement of images in traditional board games.

Map editors for platform games, Minecraft (Figure 5) or Sims’

homes and interactive simulations such as Rude-Goldberg

machines or Jenga using the pieces. CollARt could be used for

playing Pictionary or Taboo but using tridimensional collages

instead of pictures. Two users attached the marker to some part of

their body and used it for playing (Figure 6). General

improvements recommended were to add multiplayer support or

the ability to share creations on Facebook.

Figure 5. A user tried to replicate a Minecraft landscape.

Figure 6. This user made a skull and performed Hamlet.

6. CONCLUSION CollARt is a mobile augmented reality application which permits

to create tridimensional photo collages. It can be used itinerantly

to create works of art that blend virtual objects with real world

elements. These features increase the creativity support index of

the tool. New techniques for improving the tridimensional

placement of objects should be evaluated. Nonetheless, users

generally used the application properly, enjoyed it and created

interesting works of art.

7. ACKNOWLEDGEMENTS This work has been partially funded by the Government of

Navarra through a research grant.

8. REFERENCES [1] Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S.

and MacIntyre, B. Recent advances in augmented reality.

Computer Graphics and Applications, vol. 21, no. 6, pp. 34-

47. 2001.

[2] Binder, T., De Michelis, G., Gervautz, M., Jacucci, G.,

Matkovic, K., Psik, T. and Wagner, I. Supporting

configurability in a mixed-media environment for design

students. Personal and Ubiquitous Computing, vol. 8, no. 5,

pp. 310-325. 2004.

[3] Brooke, J. SUS: A 'quick and dirty' usability scale. Usability

Evaluation in Industry 189. Taylor and Francis. 1996.

[4] Carroll, E. A., Latulipe C., Fung, R. and Terry, M. Creativity

Factor Evaluation: Towards a Standardized Survey Metric

for Creativity Support. Creativity & Cognition Berkeley, pp.

127-136. 2009.

[5] Clark, A. and Dunser, A. An interactive augmented reality

coloring book. 3D User Interfaces (3DUI), pp. 7-10. 2012.

[6] Gal, R., Sorkine, O., Popa, T., Sheffer, A. and Cohen-Or, D.

3D collage: expressive non-realistic modeling. Non-

photorealistic animation and rendering, pp. 7-14. 2007.

[7] Girgensohn, A. and Chiu, P. Stained glass photo collages.

IEEE International Conference on Image Processing, vol.2,

pp. 871-874. 2003.

[8] Grasset, R., Boissieux, L., Gascuel, J.D. and Schmalstieg, D.

Interactive mediated reality. Australasian conference on User

interface, vol. 40, pp. 21-29. 2005.

[9] Huang, H., Zhang, L. and Zhang, H. Arcimboldo-like collage

using internet images. ACM Transactions on Graphics, vol.

30, no. 6, p. 155. 2011.

[10] Jacucci, G., Oulasvirta, A., Salovaara, A., Psik, T. and

Wagner, I. Augmented reality painting and collage:

evaluating tangible interaction in a field study. Human-

Computer Interaction-INTERACT, pp. 43-56. 2005.

[11] Kerne, A. Collage Machine: Temporality and Indeterminacy

in Media Browsing via Interface Ecology. SIGCHI Extended

Abstracts, vol. 2. 1997.

[12] Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer,

C., Reitmayr, G. and Schmalstieg, D. Sketching up the

world: in situ authoring for mobile augmented reality.

Personal and Ubiquitous Computing, pp. 1-8. 2011.

[13] Lee, S., Jung, J., Hong, J., Ryu, J.B. and Yang, H. AR Paint:

A Fusion System of a Paint Tool and AR. Entertainment

Computing, pp. 122-129. 2012.

[14] Marzo, A., Bossavit, B. and Ardaiz, O. Interacting with

multi-touch handheld devices in augmented reality spaces:

effects of marker and screen size. Interacción Persona-

Ordenador, p. 8. 2012.

[15] Marzo, A. and Ardaiz, O. A Smart Phone Tool to Create

Photo Collages, Procedia Computer Science, vol. 15, pp.

159-162. 2012.

[16] Pottery. 2010. http://www.idreams.pl/Pottery/

[17] Qualcomm. Vuforia. https://www.vuforia.com/

[18] Shneiderman, B., Fischer, G., Czerwinski, M., Resnick, M.,

Myers, B. and 13 others. Creativity Support Tools: Report

from a U.S. National Science Foundation Sponsored

Workshop. Int. J. of Human–Computer Interaction vol. 20,

no. 2, pp. 61-77. 2006

588