184 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 16, NO. 1, JANUARY 2014
Socialized Mobile Photography: Learning toPhotograph With Social Context via Mobile DevicesWenyuan Yin, Student Member, IEEE, Tao Mei, Senior Member, IEEE, Chang Wen Chen, Fellow, IEEE, and
Shipeng Li, Fellow, IEEE
AbstractThe popularity of mobile devices equipped with var-ious cameras has revolutionized modern photography. People areable to take photos and share their experiences anytime and any-where. However, taking a high quality photograph via mobile de-vice remains a challenge for mobile users. In this paper we investi-gate a photography model to assist mobile users in capturing highquality photos by using both the rich context available frommobiledevices and crowdsourced social media on the Web. The photog-raphymodel is learned from community-contributed images on theWeb, and dependent on users social context. The context includesusers current geo-location, time (i.e., time of the day), and weather(e.g., clear, cloudy, foggy, etc.). Given a wide view of scene, our so-cialized mobile photography system is able to suggest the optimalview enclosure (composition) and appropriate camera parameters(aperture, ISO, and exposure time). Extensive experiments havebeen performed for eight well-known hot spot landmark locationswhere sufficient context tagged photos can be obtained. Throughboth objective and subjective evaluations, we show that the pro-posed socialized mobile photography system can indeed effectivelysuggest proper composition and camera parameters to help theuser capture high quality photos.
Index TermsCamera parameters, mobile photography, socialcontext, social media, view enclosure.
T HE recent popularity of mobile devices and the rapiddevelopment of wireless network technologies haverevolutionized the way people take and share multimediacontent. With the pervasiveness of mobile devices, more andmore people are taking photos to share their experiences usingtheir mobile devices anytime and anywhere. Market researchindicates that more than 27% of photos were captured bysmartphones in 2011, while the number was merely 17% in theprevious year . The booming development of built-in mobilecameras (such as the advanced eight megapixel resolution andthe large aperture) has triggered a trend that may lead tomobile cameras replacing traditional handheld cameras.
Manuscript received October 09, 2012; revised February 08, 2013 and May18, 2013; accepted June 27, 2013. Date of publication September 25, 2013;date of current version December 12, 2013. This work was supported by NSFGrant 0964797 and a Gift Funding fromKodak. Part of this work was performedwhen the first author visited Microsoft Research Asia as a research intern. Theassociate editor coordinating the review of this manuscript and approving it forpublication was Dr. Vasileios Mezaris.W. Yin and C. W. Chen are with the State University of New York at Buffalo,
Buffalo, NY 14260 USA (e-mail: firstname.lastname@example.org).T. Mei and S. Li are with Microsoft Research Asia, Beijing 100080, China
(e-mail: email@example.com).Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TMM.2013.2283468
However, mobile cameras cannot guarantee perfect photos.Although mobile cameras harness a variety of technologiesto take care of many camera settings (e.g., auto-exposure) forpoint-and-shoot ease, capturing high quality photos is still achallenging task for amateur mobile users, not to mention thoselacking photography knowledge and experience. Therefore,assisting amateur users to capture high quality photos viatheir mobile devices becomes a demanding task. While mostexisting research has predominantly focused on how to retrieveand manage photos on mobile devices , , or how to adaptmedia considering unique characteristics of mobile devices, there have been few attempts to address this topic before.To obtain high quality photos, various types of commercial
software such as Photoshop have been developed for post-pro-cessing to adjust photo quality. However, most of them are de-signed for desktop PC, which require intensive computation. Al-though there exist some mobile applications for image post-pro-cessing, they are only able to conduct simple operations, such ascropping and contrast adjustment. These post-processing toolscannot always fix poorly captured images. For example, the in-formation lost in an over-exposed image is usually unrecover-able by any post-processing technique. Therefore, it is desirableto assist mobile users in obtaining high quality photos while thephotos are being taken. For example, if we can suggest the op-timal scene composition and suitable camera settings (i.e., aper-ture, ISO, and exposure time) based on the users current con-text (i.e., geo-location, time, and weather condition) and inputscene, the users ability to capture high quality photos via mo-bile devices will be improved significantly.On the other hand, computational aesthetics of photography
have emerged as a hot research area. It aims to automatically as-sess or enhance image quality with computational models basedon various visual features, such as lighting (e.g., light, color,and texture) and composition (e.g., the rule of thirds). A com-prehensive survey on computational aesthetics can be found in. However, aesthetic assessment or enhancement methods inexisting works are also designed for desktop PCs, and focus onevaluating or enhancing image quality as post-processing tools.Moreover, they only apply some simple and general photog-raphy rules to the aesthetics evaluations. For example, whenassessing the photo composition, they tend to place objects ac-cording to the rule of thirds or put the horizontal line lower in theframe, according to the golden ratio . In many cases, insteadof applying these simple rules slavishly, professional photogra-phers usually adapt the composition and the camera exposureparameters for photographing, e.g., the scene and the lightingconditions. Therefore, the existing general photo aesthetics as-sessment and enhancement approaches are far from enough toguide mobile user to capture high quality photos on the fly.
1520-9210 2013 IEEE
YIN et al.: SOCIALIZED MOBILE PHOTOGRAPHY: LEARNING TO PHOTOGRAPH WITH SOCIAL CONTEXT VIA MOBILE DEVICES 185
Based on the above analysis, it is desirable to develop anintelligent built-in tool to assist mobile users in capturing highquality photos. Since composition and exposure parameters(i.e., aperture, ISO, and exposure time) are two key factorsaffecting photo quality , the system should be able to pro-vide suggestions to mobile users in terms of these two aspects.Therefore, we propose an intelligent mobile photographysystem, which can not only assist mobile user to find good viewenclosures, but also help them set correct exposure parameters.
A. Challenges and Opportunities
Despite recent development on computational aesthetics, in-telligent mobile photography still remains a challenge becauseno unified knowledge can always be applied to various scenesand contexts. Various composition rules and exposure prin-ciples apply adaptively and flexibly when capturing differentobjects or scenes with diverse perspectives and arrangementsunder various lighting conditions. Professional photographerstake years of training to obtain sufficient photography knowl-edge. They usually skillfully adopt the domain knowledge tocapture scene under different conditions. Photographers areartists whose knowledge is difficult to model and representwith simple rules. The descriptions on the complexity of photocomposition and camera settings are given in Section III.Therefore, assisting mobile users to take professional photosunder various shooting situations is a challenging task.Fortunately, the availability of rich context sensors on mo-
bile devices and the explosion of images associated with theircapture contexts, camera parameters and social information onsocial media community bring us good opportunities to solvethis challenging problem. On mobile devices, the GPS and timeof the camera can be obtained and some other photography re-lated context information like weather condition at the shootingtime can be further inferred. On the other hand, a large volumeof photos on social media websites are associated with meta-data such as GPS and timestamp, as well as EXIF includingthe camera parameters. Moreover, in the social media commu-nity, the photo quality can be implicitly inferred from numberof views, number of favorites, and comments, or even explicitlyobtained from ratings. Despite some noise in the media meta-data, the aggregated photographs captured nearby containingthe same scene with their metadata from the media commu-nity can provide significant insight into relevant photographyrules in terms of composition and exposure parameters for mo-bile photography assistance.For example, as shown in Fig. 1, when capturing a photo
for Statue of Liberty from the perspective as shown in theinput wide-view image, using the content and context of theinput image, photos with the same content from similar per-spectives and their associated social information which can re-flect their qualities can be crowdsourced. By analyzing the com-position and their aesthetic quality of these crowdsourced im-ages, we find that pictures with the Statue on the right third lineare usually highly preferred; hence the optimal composition ofthe input image can be inferred. Moreover, with the input timeand weather condition, we can also estimate the optimal cameraexposure parameters from the crowdsourced photos with sim-ilar content and lighting conditions. Here, the first two crowd-sourced similar photos with high ratings were captured under
Fig. 1. Illustration of how crowdsourced images help for view suggestion andexposure parameter suggestion.
similar season and time of the day, and their weather conditionis also overcast as the current weather condition, so by settingthe camera parameters to exposure time (ET): 1/200, aperture:8.0 and ISO: 100 as the two images, the user should be able toobtain high quality photos with proper exposure.Motivated by the above observations of mobile devices and
social media, we propose a socialized mobile photographysystem leveraging both the context and crowdsourced photog-raphy knowledge to aid mobile users in acquiring high-qualityphotos. We predominantly focus on outdoor landscape pho-tography on mobile devices. To achieve intelligent mobilephotography, the following problems need to be solved. First,the system should be able to suggest view enclosure withoptimal composition by mining the scene specific compositionrules from the crowdsourced community-contributed photos.Second, the optimal exposure parameters have to be recom-mended given the suggested composition and the lightingcondition.The proposed socialized mobile photography system is
shown in Fig. 2. Given the input wide-view image and theshooting context, i.e., geo-location, time, and weather, thesystem is able to suggest view enclosures with optimal com-position as marked in red rectangle by mining the relevantcomposition rules from crowdsourced photos captured nearby.Then, the system suggests optimal exposure parameters,i.e., ISO, aperture, and exposure time (ET) as shown on theupper-right corner of the screen, by mining exposure rules fromthe crowdsourced images with similar content and context.Therefore, the optimal view enclosure and exposure parameterscan be suggested by the proposed system to capture high quality
186 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 16, NO. 1, JANUARY 2014
Fig. 2. Demonstration of the proposed socialized mobile photography system.
photos in the given scene and context, as long as sufficientimages containing similar content are crowdsourced nearby.To intelligently aid mobile users in capturing high quality
photos considering the complex scene and context dependentcomposition and exposure principles, two fundamental com-ponents are developed in the socialized mobile photographysystem: 1) offline composition and exposure parameter learning,and 2) online photography suggestion.Considering various views in a given scene due to the differ-
ence of capture location, perspective, and content, viewpointclusters within a scope of location are discovered based onimage local features as well as their geo-locations by unsu-pervised methods. Relevant composition knowledge is minedby view cluster ranking and view cluster specific compositionlearning. To learn the effects of various contexts and differentshooting content to exposure, metric learning is carried outfor exposure parameter suggestion. The time consuming viewclustering, composition and exposure learning processes areall performed offline, which make the system applicable andpractical for mobile applications. In the online process, aportion of view enclosure candidates are discarded by clusterrankings obtained offline, which make the online view decisionquite efficient. Then top ranked view enclosure is selected fromthe remaining candidates based on the learned view specificcomposition model. Finally, the optimal exposure parametersare suggested according to the content of the optimal view andthe shooting context based on the learned exposure model.
We make the following three major contributions: We propose a general framework for mobile photographyusing the rich social context from mobile devices. To thebest of our knowledge, little research has been conductedon this topic before.
We solve the problem ofmobile photography by leveragingboth rich context and crowdsourced images on the Web.To overcome the photography challenge due to the com-plex scene and context-dependent characteristics, a viewcluster discovering followed by a view specific compo-sition learning and exposure parameter learning schemeis developed to suggest the optimal view and parametersbased on the discovered photography rules.
We develop a mobile photography system and evaluatethrough objective and subjective evaluations.
The remainder of the paper is organized as follows. InSection II related works are discussed. In Section III, we
present the challenges of mobile photography to justify theneed of the proposed mobile photography system. The proposedsocialized mobile photography system overview and the detailson offline photography learning and online mobile photographysuggestion are introduced in Section IV. Experiment designsand evaluations are demonstrated in Section V. Finally, weconclude this paper in Section VI.
II. RELATED WORKRecently, considerable research efforts have been made on
photo aesthetics computation. Based on some heuristic lowlevel features which are expected to discriminate betweenpleasing and displeasing images, linear regression is appliedto predict numerical aesthetics ratings in . In , bydetermining...