Embed Size (px)
Citation preview
A System for Efficient Cleaning and Transformation ofGeospatial Data Attributes (Demo Paper)
Yao-Yi ChiangUniversity of Southern
CaliforniaSpatial Sciences Institute
Los Angeles, CA 90089, [email protected]
Bo WuUniversity of Southern
CaliforniaDepartment of ComputerScience and Information
Sciences Institute4676 Admiralty Way, Marina
del Rey, CA 90292, [email protected]
Akshay AnandKetan Akade
University of SouthernCalifornia
Department of ComputerScience and Spatial Sciences
InstituteLos Angeles, CA 90089, USA
[akshayan,akade]@usc.edu
ABSTRACTA significant challenge in handling geographic datasets isthat the datasets can come from heterogeneous sources withvarious data qualities and formats. Before these datasets canbe used in a Geographic Information System (GIS) for spa-tial analysis or to create maps, a typical task is to clean theattribute data and transform the data into a unified format.However, conventional GIS products focus on manipulatingthe spatial component of geographic features and only offerbasic tools for editing the attribute data (e.g., one row at atime). This limits the capability for handling large datasetsin a GIS since manually editing and transforming attributedata between different formats does not scale for thousandsof geographic features. In this demo, we present ArcKarma,which is built on our previous work on data transforma-tion to efficiently clean and transform data attributes ina GIS. ArcKarma generates transformation programs froma few user-provided examples and applies the programs totransform individual attribute columns to the desired for-mats. We show that ArcKarma produces accurate resultsand eliminates the need for laborious manual data cleaningand scripting tasks.
Categories and Subject DescriptorsH.4.m [Information Systems Applications]: Miscella-neous—Geographic Information Systems; H.2.8 [DatabaseManagement]: Database Applications—Spatial Databasesand Geographic Information Systems
General TermsAlgorithms, Design, Human Factors
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SIGSPATIAL ’14 Dallas, Texas USACopyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
KeywordsGeographic information system, data cleaning, data trans-formation
1 INTRODUCTIONA key problem facing geographic information system (GIS)users is that the geographic datasets used in a project veryoften come from multiple sources and the attribute dataof these datasets can be in a variety of data formats (e.g.,telephone numbers stored as “213-740-2311” or “(213) 740-2311”). In addition, even individual geographic featureswithin one data source can have different formats for anattribute because of the datasets were collected at differ-ent time periods or a consistent attribute format was notenforced during data entry. While a GIS supports a widerange of operations to integrate the spatial component ofgeographic datasets (e.g., data overlay and spatial join),linking and joining geographic features using attribute datagenerally relies on manual data preparation steps or requirestask dependent programming work to ensure attributes ofgeographic features are in a unified format. These datapreparation steps can be laborious and require scripting orprogramming skills.
This paper presents a system called ArcKarma, which isan interactive data transformation tool integrated with aconventional GIS software (Esri ArcMap). ArcKarma em-ploys a training-by-example strategy to generate transfor-mation programs and automatically transforms attribute datato a unified format. ArcKarma does not require expertknowledge in scripting or programming. In the reminderof this paper, Section 2 presents the ArcKarma user inter-face and workflow, Section 3 describes the ArcKarma datacleaning and transformation algorithm, Section 4 presentsrelated systems, and Section 5 discusses the future work.
2 ArcKarma – WORKFLOW AND USERINTERFACE
Figure 1 shows the ArcKarma architecture and a typical usecase. The “Well Information” layer in this ArcMap projectcontains an attribute table of oil well information for main-taining the records of leased wells from a petroleum produc-tion company. This list is an integration of well data from
Well Code Depth Leasing Company Local Contact Number … 6C 3040 Smith Father and Sons (432) 877-‐9521 …
A 30 3102 Texas United 432-‐924-‐7106 …
A 33 3165 Texas United 432-‐924-‐7106 …
A 34 3120 Texas United 432-‐924-‐7106 …
A 36 3014 Texas United 432-‐924-‐7106 …
52 3015 Texas United 432-‐924-‐7106 …
53C 3191 Smith Father and Sons (432) 877-‐9521 …
66C 3192 Smith Father and Sons (432) 912-‐0326 …
…
Well Code User Provided Example
6C
A 30 30A
A 33
…
TransformaAon Result
6C
30A
33A
Local Contact Number
User Provided Example
(432) 877-‐9521
432-‐924-‐7106 (432) 924-‐7106
432-‐924-‐7106
…
TransformaAon Result
(432) 877-‐9521
(432) 924-‐7106
(432) 924-‐7106
Well Information Attribute Table
ArcKarma Cleaning and TransformaCon Service
USC ArcKarma
Figure 1: ArcKarma workflow
multiple sources that contain data created at different timeperiods. The spatial component of the “Well Information”layer contains the well locations (in this case, point data)and the attribute table stores auxiliary information aboutindividual wells. Before the year 2003, the petroleum com-pany used one alphabetic character, a space, and a numericstring to identify the well types (e.g., “A 30”) and the cur-rent system uses a new naming convention of one or morenumeric characters followed by an alphabetic character, e.g.,“6C”). As the information of the wells leased from the com-pany Texas United exists before the system change, theirwell codes follow the old naming convention. In addition,the previous system user interface does not enforce domainintegrity during data entry and hence the contact numberscan be in various formats and do not follow the current stan-dard.
To unify the data format for both the “Well Code” and“Local Contact Number” attributes, the user enters exam-ple values of the desired format (the blue columns in Fig-ure 1). ArcKarma takes the original data (the black columnsin Figure 1) and the example values to invoke the cleaningand transformation service for transforming individual at-tribute values. Finally, the transformation results (the greencolumns in Figure 1) are pushed back to the ‘Well Informa-tion” attribute table in ArcMap.
Figure 2 shows the ArcKarma user interface. Within theArcMap environment, the user clicks on “Karma Transfor-mation” and a table view shows up for the user to selectthe desired attribute for transformation (by clicking on thecolumn header) (steps 1 and 2). In this example, the userselects to transform the “WELL CODE” attribute and pro-vides an example value of “30A” for the original value “A 30”(step 3). After the user finishes providing the transforma-tion example(s), the user clicks on “Transform” to generatethe transformation results. The user can then click on“Save”to fetch the results back to the attribute table in ArcMap(steps 4, 5, and 6). In addition to directly saving the trans-formation results, the user can edit, discard, or add moreexamples if the results are not satisfactory. During the en-
tire process, the user stays in ArcMap and does not need tomanually import/export datasets from/to different softwareplatform.
3 ArcKarma – DATA CLEANING ANDTRANSFORMATION SERVICE
The ArcKarma data cleaning and transformation service(DCTS) is an implementation of our previous work in [6]that synthesizes transformation programs using examples.This previous work is based on the programming-by-exampleapproach developed by Gulwani [2], which defines a stringtransformation language that supports a restricted, but ex-pressive form of regular expressions including conditionalsand loops. In this section, we first present a sample trans-formation program and then describe how DCTS generatesthe program.
Figure 3 shows a sample transformation program gener-ated by DCTS using an example with the original value as“A 30” and the target value as “30A”. For each input rawdata, the program first uses a classifier to identify the formatof the input string and then selects a transformation branchto apply (i.e., the switch label statement). Each transfor-mation branch contains the code that can convert the inputstring to a target format (e.g., case “format1′′). The trans-formation code is essentially a concatenation of several sub-string expressions (e.g., value.substring). A substring ex-pression can either be (1) a constant string or (2) an extrac-tion rule describing how to extract specific string parts fromraw data. The program in Figure 3 contains two substringexpressions: the extraction rule (e.g., substring(pos3, pos4))to extract uppercase alphabetic characters and the extrac-tion rule (e.g., substring(pos1, pos2)) to extract numericcharacters.
In DCTS, an extraction rule has two expressions to iden-tify the start and end positions of a substring (e.g., thevalue.indexOf function). The positions can be expressed us-ing (1) an absolute position or (2) restricted regular expres-sions that identify the context of the given position, which
Step 1 Step 2
Step 3 Step 4
Step 5 Step 6
Figure 2: ArcKarma user interface
Transformation Branch
Figure 3: Program generated by the Data transfor-mation Module
can be represented as (“leftcxt”, “rightcxt”, “occ”). “leftcxt”describes the left context of the position, “rightcxt”describesthe right context, and“occ” is the occurrence of the position.For example, the start position of “A” in “A 30“ can be spec-ified as an absolute position 0 (pos3) or (START, UWRD, 1).Here, “START” represents the beginning of the raw value.“UWRD” represents an uppercase letter. Other token typesare that “END” is for the end of the raw value; “LWRD”
means a continuing sequence of lower letters; “NUM” refersto a continuing sequence of digits; the “BNK” means a blankspace. Therefore, “(START,UWRD, 1)” means the first oc-currence of a position (“occ” = 1), which is at the beginningof the raw data (“leftcxt” = “START”) and has a upper-case token at its right (“rightcxt” = ‘UWRD”). With a listof position expressions pos1, pos2, pos3 and pos4, the trans-formation program can specify the start and end positionsof the specific substrings. The program then extracts thesesubstrings and concatenates them to form the output.
To learn a transformation program, DCTS first separatesthe target values into string segments. For example, the rawdata “30A” is separated into two segments “30” and “A” asshown in Figure 4. DCTS then tries to independently gen-erate these segments by either extracting a substring fromthe input or inserting a new string. For instance, the “30” inthe target string can be extracted from the original input.With these alignments, DCTS then identifies the locationsused to extract the substrings and provides a variety of waysspecifying these locations such as using absolute position orusing its context.
A 30
30A
Input:
Target:
Figure 4: Learning the transformation programs
If there are multiple examples, DCTS uses an efficient al-gorithm to construct a version space [4] for each exampleand to merge them into a version space consistent with themaximum number of examples. A version space is essen-tially a hypothesis space that contains the programs thatare consistent with the examples. However, if the examplescannot be covered by a single version space, DCTS partitionsthe examples and generates a version space for each parti-tion individually. Each partition corresponds to a differentinput format. A conditional expression can be learned todistinguish multiple different formats. DCTS also supportsloop expressions by detecting whether continuous segments
of one program can be merged. To efficiently generate thetransformation program from the version space, DCTS de-fines a partial order over the version space. This partialorder helps to generate the simpler program first, which ismore likely to be correct according to Occam’s principle.
4 RELATED WORKOpenRefine1 and Potter’s wheel [5] allow the user to spec-ify string edit operations. OpenRefine is a data cleaningtool that supports regular expression style of string trans-formation and data layout transformation. Potter’s Wheelhas predefined transformation operations and users gradu-ally build transformation programs by adding or undoingtransformation operations using an interactive user inter-face. Many programming-by-demonstration approaches canlearn edit operations by asking the user to demonstrate theediting process [1]. Lau [4] presents a system that learnsfrom a user’s edit operations to generate a sequence of textediting programs. Data Wrangler [3] is an interactive toolthat uses the transformation operations defined in Potter’swheel for creating data transformation programs. In addi-tion to supporting string level transformation, Data Wran-gler also supports data layout transformation including col-umn split, column merge, fold, and unfold. In contrast, Ar-cKarm only requires the user to enter a few examples of thedesired data value without asking the user to demonstratethe format conversion steps.
5 DISCUSSION AND FUTURE WORKWe plan to integrate two additional capabilities from [6]to enhance ArcKarma usability: (1) recommending rows toprovide examples and (2) highlight the corresponding sub-strings between the raw and transformed values, which helpsthe user to understand the applied transformation.
6 ACKNOWLEDGMENTSThis research is based upon work supported in part by theChevron Corporation and the Center for Interactive SmartOilfield Technologies at University of Southern California.
References[1] A. Cypher, D. C. Halbert, D. Kurlander, H. Lieberman,
D. Maulsby, B. A. Myers, and A. Turransky, editors.Watch what I do: programming by demonstration. MITPress, 1993.
[2] S. Gulwani. Automating string processing in spread-sheets using input-output examples. In POPL, pages317–330, 2011.
[3] S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer. Wran-gler: interactive visual specification of data transforma-tion scripts. In CHI, pages 3363–3372, 2011.
[4] T. Lau, S. A. Wolfman, P. Domingos, and D. S. Weld.Programming by demonstration using version space al-gebra. Mach. Learn., pages 111–156, 2003.
[5] V. Raman and J. M. Hellerstein. Potter’s wheel: Aninteractive data cleaning system. In VLDB, pages 381–390, 2001.
[6] B. Wu, P. Szekely, and C. A. Knoblock. Minimizing usereffort in transforming data by example. In IUI, pages317–322, 2014.
1http://openrefine.org
