Embed Size (px)
Citation preview
A Platform for Large-Scale MachineLearning on Web Design
Arvind SatyanarayanSAP Stanford Graduate FellowDept. of Computer ScienceStanford University353 Serra MallStanford, CA 94305 [email protected]
Maxine LimDept. of Computer ScienceStanford University353 Serra MallStanford, CA 94305 [email protected]
Scott R. KlemmerDept. of Computer ScienceStanford University353 Serra MallStanford, CA 94305 [email protected]
Copyright is held by the author/owner(s).CHI’12, May 5–10, 2012, Austin, Texas, USA.ACM 978-1-4503-1016-1/12/05.
AbstractThe Web is an enormous and diverse repository of designexamples. Although people often draw from extantdesigns to create new ones, existing Web design tools donot facilitate example reuse in a way that captures thescale and diversity of the Web. To do so requires usingmachine learning techniques to train computationalmodels which can be queried during the design process. Inthis work-in-progress, we present a platform necessary fordoing such large-scale machine learning on Web designs,which consists of a Web crawler and proxy server toharvest and store a lossless and immutable snapshot ofthe Web; a page segmenter that codifies a page’s visuallayout; and an interface for augmenting the segmentationswith crowdsourced metadata.
Author KeywordsWeb design; examples; machine learning.
ACM Classification KeywordsH.5.4 [Information interfaces and presentation]:Hypertext/ Hypermedia - Theory.
IntroductionDesign is difficult and designers typically spend most oftheir time ideating rather than executing. As a result,they often rely on examples for inspiration [4] and to
facilitate design work [8]. In the scope of Web design, theseveral billion pages that comprise the Web provide alarge corpus of diverse design examples. While tools suchas a browser’s “view source” functionality [3], templategalleries, and curated design repositories have begun toallow designers to harness these resources, they are limitedin both usefulness and scale. For example, onlinerepositories are manually curated by designers; thereforethey usually contain a few thousand pages at most, whichinadequately capture the diversity of Web designs.Furthermore, browsing designs is inefficient, typicallyrestricted to page-by-page browsing or filtering on somebasic, manually identified attributes such as color.
The unstructured, constantly changing nature of the Web,and its sheer size, pose challenges that call for analgorithmic approach to example-based Web design tools.Algorithms would not only allow these tools to pull from alarger pool of examples, but they would also enable a newclass of applications. For example, consider a tool thatcould abstract a designer’s partially-formed design idea asa search specification and extract from an extensivecorpus design completions containing the inputparameters. Such an application would be difficult tobuild with a manually curated and classified corpus.
Hard-coded, formulaic algorithms, however, would alsonot account for the diversity of Web design patterns, norcould they simulate non-rule-based dimensions —creativity, for example. A more suitable approach ismachine learning: an algorithmic method that “learns”the characteristics of a domain, Web design in our case,from a “training set” of examples and makes predictionsabout these characteristics on new data.
To perform machine learning on Web designs, our trainingset would consist of a corpus of Web pages such that:
1. A page renders the same way every time it isrequested — to generate accurate predictors,machine learning algorithms need to be trained on aconstant, unchanging training set.
2. All pages within the corpus are complete copies oftheir online counterparts: all content they requestmust be stored in our corpus, and source codecannot be modified — any missing content, ormodified source code, could introduce rendering orbehavioral discrepancies into our corpus and corruptclassifiers trained on this data, causing them toperform poorly when run on live Web pages.
3. A representation of a page that captures thestructural relationships in its visual layout isavailable — classifiers that rely solely on pagemarkup would perform poorly as designers can usescripting and advanced CSS techniques to arbitrarilyreposition content such that visually salient regionsno longer correspond to the Document ObjectModel (DOM) [6].
4. The corpus is scalable and preserves the graphstructure of the Web (links between pages and theirresources) is key — it prevents duplicates beingsaved to the corpus, and the data could be usefulfor certain learning applications.
These requirements make it impossible to use the Webdirectly, as the DOM is the only representation readilyavailable, and content changes as frequently as every pageload. Thus, we need create a snapshot of the Web by
compiling a corpus that meets these requirements. Therehave been several projects to build such corpora [9, 7, 2]by using Web crawlers to identify and download pages andresources such as images, stylesheets and scripts.
Proxy Server
Corpus Database
Web Crawler
Browser Thread
Browser Thread ...
URLs
The Web Bento
Raw Content
Bento Nodes
Bento Features
Machine Learning
Algorithms
Applications
Interfaces
Web Mode Corpus-Only Mode
Figure 1: An overview of the system architecture.
Although Web crawlerspreserve the Web’s graphstructure, thus scaleablybuilding our corpus,they rely on heuristicsto identify URLsto crawl [5] and may,for example, miss thoseembedded in CSS orrequested dynamically inJavascript. This behaviorwould violate our seconddesign goal. Additionally,existing crawlerstreat URLs as uniquelyidentifying content [5],and hence only crawlthem once. This does notaccount for cases wheredifferent content mayshare a common URL —for example, a Javascriptfile that is dynamicallygenerated — and wouldagain violate our second
design goal. Page archivers [1] are an alternative to Webcrawlers. They leverage a Web browser to download apage and package it together with an individual copy ofall resources it uses. With this approach, we lose theWeb’s graph structure and also begin to store duplicatesof resources, violating our fourth design goal. To ensure
consistent offline rendering, archivers rewrite the sourceHTML to point to these local copies and strip anyscripting, contrary to our second design goal.
To meet these design goals, we developed a platform forlarge-scale machine learning on Web designs. As Fig. 1shows, we use a Web crawler and proxy servercombination to build a complete corpus of raw Web pageand resource content. This raw content is then processedby Bento [6], a page segmenting algorithm whichproduces a representation of the visual structure of apage. Finally, a crowdsourced label gathering applicationaugments our corpus with keywords which could serve asa seed for machine learning.
The InfrastructureWeb Crawler and Proxy ServerThe lightweight, multithreaded Web crawler maintains alist of URLs to crawl and then recursively loads a URLand analyzes it for additional URLs to add to the list;URLs are not treated as uniquely identifying content.Each Web crawler thread loads URLs in an embeddedWebkit1 browser, and the task of downloading content isdelegated by configuring the browser to use a customproxy server. Every request made by the browser, andevery response it receives from the Internet, goes throughthe proxy server, which writes this information to ourcorpus. As URLs are no longer considered unique, toprevent the corpus from burgeoning, we implement a“content-seen” test [5] to check if it already exists in ourcorpus, and only saved if it does not. With this pipeline,we are guaranteed that the proxy server automatically seesevery request a page makes for a resource, without anycustom heuristics, and yet scalably builds our corpus.
1http://www.webkit.org/
The proxy server, however, simply sees a series of requestsand their corresponding responses — as HTTP is astateless protocol, it is not able to identify which pagerequested what resource, a problem exacerbated bymultiple Web crawler threads making concurrent requeststhrough the proxy. This relationship is crucial topreserving the Web’s graph within our corpus.
Web Crawler Browser Proxy Server Corpus
DatabaseThe Web
GET http://google.com/Bricolage-ISAPage: 1
HTTP/1.1 200 OK<html><head>...
storePage()
page_id: 13
HTTP/1.1 200 OKBricolage-PageID: 13
<html><head>...
GET http://google.com/Bricolage-ISAPage: 1
GET http://google.com/logo.pngBricolage-PageID: 13
GET http://google.com/styles.cssBricolage-PageID: 13
Figure 2: Custom headers are used to overcome HTTP’s statelessness byidentifying which page made what request. This allows us to preserve theWeb’s graph structure in our corpus.
To easily identifythese relationships,we introduceda set of customHTTP headers intothe pipeline as shownin Fig. 2. The Webcrawler requests apage and supplies theBricolage-ISAPage
header to identifythe URL as a “page”as opposed to a“resource”. When theproxy server receivesthe page’s contentfrom the Internet, andsaves it to our corpus,a unique identifieris generated, whichis transmitted backto the Web crawler’sbrowser through theBricolage-PageID
header. This header is then attached to subsequentresource requests, which allows the proxy server torecreate the graph within our corpus.
As Fig. 1 shows, the system also contains a “corpus-onlymode.” In this mode, the proxy server does not passrequests to the Web, but rather serves content directlyfrom our corpus; if it is not able to find the URL in thecorpus, it responds with a 404 error. This mode,facilitated by adding a custom Bricolage-FromCache
header to requests, allows consistent offline renderingwithout modifying any source code.
Bento: A Page Segmenting AlgorithmEach page is passed through the Bento pagesegmenter [6] which wraps all inline content with <span>
tags, rearranges the DOM tree relationships to correspondto visual containment and finally augments the tree toremove nodes that do not contribute to the visual layout,and add those that do. Fig. 3 shows an example Bentotree, and highlights that all content is a leaf node in thetree, and non-leaf nodes correspond to screen space. Formaximum granularity, each Bento node, and its associatedstyle features, are stored as individual records within ourcorpus database.
body
#header
#logo #nav
#content
h1.section
.col .col .col
#footer
Figure 3: Bento nodes better correspond to visually salientregions of a page.
Crowdsourced Label-Collecting
Figure 4: Label gathering interface.
Correctly and effectivelycollecting Web pagesinto a corpus providesthe constant trainingset required for machinelearning, but supplyingvisual, design-orientedfeatures associated withthe collected pages forthe learning algorithmsto use is also criticalfor usefulness. To allowfor this functionality,labels corresponding tothe features in questionmust be stored with thepages with which they areassociated. These labels
may express the stylistic features of a Web page, itscontent and purpose, or any other characteristics that maybe of interest to designers. While labels that can be drawnfrom the lower-level representation of a saved Web pagecan be applied to the page automatically, less concretecharacteristics, which may be based on a holistic visualassessment of a Web design, do not lend themselves wellto automated classification. For example, automaticallyand accurately judging whether a design is minimalist orelegant is challenging because there does not exist a set ofcriteria that can be easily and methodically checked andthat will also consistently result in an accurateclassification. Therefore a viable solution must makeconsistent and accurate classifications as well as maintainthe scalability that comes with automated labeling. Wepropose a label-collecting interface, shown in Fig. 4, thatallows us to take advantage of crowdsourcing. If label
application is delegated to Web designers themselves, wecan attain accurate style labels; if a large enough audienceis targeted, scalability can be achieved as well.
To allow designers to apply labels to designs, we built alabel-collecting interface. Designers were sent to thisinterface and asked to submit stylistic keywords describinga number of Web pages in our corpus. The interfaceincludes a set of directions for entering labels into thedatabase, a screenshot of the current page, a list ofkeywords the designer has already applied to the page,and appropriate fields for label submission and pagenavigation. Upon access to the interface, a unique ID israndomly generated for the user, and this ID is maintainedthroughout the task. The corpus of Web pages to belabeled loads, and pages are presented to the user inrandom order. For each page, the user enters and submitsa number of keywords to describe it. The keyword field isenhanced with a jQuery-backed autocomplete featuredrawing from a dictionary of valid English keywords thathave been applied to pages in the database. Thedictionary is refreshed after every page entry. Uponsubmission each new keyword, along with the associateduser ID and Web page, is entered into a SQLite databaseand appears on the screen so that users can track whichkeywords they have already entered. Keyword entries andpage navigation are implemented using AJAX calls toavoid refreshing the interface page.
Discussion and ConclusionTo assess the feasibility of the label-collecting interface,we ran a preliminary trial to gather style labels for a300-page corpus by hiring crowdsourced designers to applylabels to the pages. The interface allowed us to collectover 3,000 style labels, shown as a tag cloud in Fig. 5,that appropriately described their corresponding pages.
However, though we were able to pay designers to performthis task for the purposes of this trial, providing suchcompensation does not allow for scalability. If designershad incentive to apply labels themselves, then paymentwould not be necessary. By building a searchable interfacefor our corpus based on the collected labels, we would beable to offer designers a powerful exploratory tool forinspiration unlike any design repository currently available.The usefulness of the tool would allow for scalability, sincedesigners would have an incentive to supply labels for theunderlying corpus in order to improve their experiencewith the interface.
Figure 5: A tag cloud of the 3,000 labelscollected in our preliminary trial. Noticethat some style words are used morecommonly than others.
The interface described is only one of the manyapplications that could stem from our corpus. Trainedprograms may eventually be able to automatically andaccurately classify designs based on stylistic features.Design tools in the same vein as the searchable interfacedescribed may provide designers with relevant inspirationalexamples when given a partially complete design or anexisting design example as input. They might evengenerate original designs based on a set of requirementsspecified by the designer. Additionally, browser extensionsto crowdsource corpus-building can be developed tofacilitate augmenting the corpus itself and increase therange of features it can support. To more thoroughlygrasp the power of machine learning in the context ofWeb design, we look forward to building these applicationsand investigating their implications for designers.
References[1] Mozilla archive file format. http://maf.mozdev.org/.[2] Cathro, W., Webb, C., and Whiting, J. Archiving the
web: The pandora archive at the national library
australia. National Library of Australia Staff Papers, 0(2009).
[3] Gibson, D., Punera, K., and Tomkins, A. The volumeand evolution of web page templates. In Specialinterest tracks and posters of the 14th internationalconference on World Wide Web, ACM (2005),830–839.
[4] Herring, S., Chang, C., Krantzler, J., and Bailey, B.Getting inspired!: understanding how and whyexamples are used in creative design practice. InProceedings of the 27th international conference onHuman factors in computing systems, ACM (2009),87–96.
[5] Heydon, A., and Najork, M. Mercator: A scalable,extensible web crawler. World Wide Web 2, 4 (1999),219–229.
[6] Kumar, R., Talton, J., Ahmad, S., and Klemmer, S.Bricolage: Example-based retargeting for web design.In Proceedings of the 2011 annual conference onHuman factors in computing systems, ACM (2011),2197–2206.
[7] Lampos, C., Eirinaki, M., Jevtuchova, D., andVazirgiannis, M. Archiving the greek web. In 4thInternational Web Archiving Workshop (IWAW04)(2004).
[8] Lee, B., Srivastava, S., Kumar, R., Brafman, R., andKlemmer, S. Designing with interactive examplegalleries. In Proceedings of the 28th internationalconference on Human factors in computing systems,ACM (2010), 2257–2266.
[9] Mohr, G., Stack, M., Rnitovic, I., Avery, D., andKimpton, M. Introduction to heritrix. In 4thInternational Web Archiving Workshop (2004).
