Data Quality and Neogeography

1blog.telemapics.com


Before We StartBefore We Start

I am not here to persuade you about the I am not here to persuade you about the usefulness or limitations of Neogeography usefulness or limitations of Neogeography or User Generated Contentor User Generated Content

I am here to share my views on issues I am here to share my views on issues relating to the topic of spatial data quality relating to the topic of spatial data quality and neogeographyand neogeography

Disclaimer - In general, my observations Disclaimer - In general, my observations derive from my familiarity with mapping, derive from my familiarity with mapping, navigation and local searchnavigation and local search


My BackgroundMy Background PhD in Geography, specializing in CartographyPhD in Geography, specializing in Cartography Attended AutoCarto 1 in 1974 (and gave the Attended AutoCarto 1 in 1974 (and gave the

keynote in 2008)keynote in 2008) Associate Professor of mapping and geography at Associate Professor of mapping and geography at

SUNY Albany (1972–1985)SUNY Albany (1972–1985) Associate at Spad SystemsAssociate at Spad Systems Chief Cartographer, Chief Technologist and VP of Chief Cartographer, Chief Technologist and VP of

BizDev for Rand McNally (1986-1999)BizDev for Rand McNally (1986-1999) CTO and EVP of Engineering for go2 Systems (YP CTO and EVP of Engineering for go2 Systems (YP

over cell phones)over cell phones) Now run a consulting business focused on Now run a consulting business focused on

geospatial, especially local search, mapping and geospatial, especially local search, mapping and navigation applicationsnavigation applications

Data Quality and Data Quality and NeogeographyNeogeography

Dr. Mike DobsonDr. Mike Dobson

PresidentPresident

TeleMapics LLCTeleMapics LLC

[email protected]@telemapics.com


Spatial Data Quality?Spatial Data Quality? Overall concern regarding the Overall concern regarding the

“fitness” of data for a particular use“fitness” of data for a particular use Accuracy of positionAccuracy of position

resolutionresolution Accuracy of AttributionAccuracy of Attribution

Logical ConsistencyLogical Consistency CompletenessCompleteness

Including spatial coverageIncluding spatial coverage Temporal relevanceTemporal relevance MetadataMetadata


Spatial Data’s Emerging Spatial Data’s Emerging PopularityPopularity

World of spatial data is explodingWorld of spatial data is exploding Accessibility to spatial data increasingAccessibility to spatial data increasing Availability of spatial data increasingAvailability of spatial data increasing Today’s online environment providesToday’s online environment provides

Easy-to-use tools for collecting spatial dataEasy-to-use tools for collecting spatial data Easy-to-use tools for analyzing spatial dataEasy-to-use tools for analyzing spatial data Easy-to-use tools for presenting spatial dataEasy-to-use tools for presenting spatial data


Why Is This of Concern?Why Is This of Concern? The quality of spatial data mitigates the The quality of spatial data mitigates the

success of communicating spatial success of communicating spatial conceptsconcepts Could this explosive growth have an influence Could this explosive growth have an influence

on the quality of spatial data?on the quality of spatial data?


Why Data Quality Is KeyWhy Data Quality Is Key


No Integrity!No Integrity!


NeogeographyNeogeography

Neogeography Neogeography ““new” geography using non-traditional new” geography using non-traditional

toolstools NeogeographersNeogeographers

Want to communicate/share their interests Want to communicate/share their interests in geography and are willing to do in geography and are willing to do something about itsomething about it


NeoGeosNeoGeos

What Roles do Neogeographers play in the What Roles do Neogeographers play in the process of communicating spatial data?process of communicating spatial data? Data collectors – database creatorsData collectors – database creators Data analyzersData analyzers Data PresentersData Presenters

While all three roles impact or are While all three roles impact or are influenced by “data quality”, today I will influenced by “data quality”, today I will focus on neogeographers and data focus on neogeographers and data collection /database creationcollection /database creation


Spatial Data Quality and Spatial Data Quality and NeogeographyNeogeography

In order to help you understand my In order to help you understand my persuasion on data quality and persuasion on data quality and neogeography, I would like to explore neogeography, I would like to explore User Generated ContentUser Generated Content UGC is one of the primary means that UGC is one of the primary means that

neogeographers use to express their neogeographers use to express their interest in Geographyinterest in Geography On this journey we will loop outside of On this journey we will loop outside of

geography and then fall back in through geography and then fall back in through mapping and other uses of spatial data.mapping and other uses of spatial data.


UUser ser GGenerated enerated CContent?ontent?

Content that is produced by users Content that is produced by users of web sites and digital mediaof web sites and digital media Contrasted with traditional media Contrasted with traditional media

producers such as broadcasters, producers such as broadcasters, production companies publishing production companies publishing companies and map database companies and map database companiescompanies


So What’s Important About So What’s Important About UGC?UGC?

Equality of opportunity to publishEquality of opportunity to publish Coupled with one of the most Coupled with one of the most

significant demographic trends in the significant demographic trends in the last century:last century: ““It’s about me” (e.g. use of It’s about me” (e.g. use of YouTube, YouTube,

MySpace, FacebookMySpace, Facebook) ) ““Especially in respect to the streets, roads Especially in respect to the streets, roads

and trails I travel, as well as the POIs I and trails I travel, as well as the POIs I frequent and the frequent and the spatial topics of interest to spatial topics of interest to meme””


Social NetworkingSocial Networking


How Did This Happen?How Did This Happen?

Technology that allows you to be Technology that allows you to be “connected”, as well as to “connected”, as well as to communicate and collaborate on communicate and collaborate on your own termsyour own terms InternetInternet Cellular telephonyCellular telephony

Development of comprehensive Development of comprehensive spatial databasesspatial databases Pushing geospatial into the mainstream Pushing geospatial into the mainstream

-Neogeography-Neogeography


How Did This Happen?How Did This Happen?

Networks provide forNetworks provide for Collective intelligence – the hive Collective intelligence – the hive

mentality or perhaps the Borgmentality or perhaps the Borg Aggregated knowledge from Aggregated knowledge from

decentralized sources (Wikipedia – decentralized sources (Wikipedia – Wikinomics)Wikinomics)

Low cost collaborationLow cost collaboration


UGC Potential BenefitsUGC Potential Benefits Linus’s law Linus’s law

With enough eyes all bugs (With enough eyes all bugs (spatial errorsspatial errors) become trivial) become trivial Contributors exhibitContributors exhibit

Self selectionSelf selection FocusFocus Self benefitSelf benefit

NumerousnessNumerousness There should be more interested spatial data There should be more interested spatial data

contributors than professional map editorscontributors than professional map editors Spatial distributionSpatial distribution

The distribution of UGCers is more ubiquitous than that The distribution of UGCers is more ubiquitous than that of professional map editors.of professional map editors.


Criticisms Of UGCCriticisms Of UGC

Some error situations are too Some error situations are too complex to be understood real-timecomplex to be understood real-time

Usability may be lowUsability may be low May require extensive error checkingMay require extensive error checking User priorities may lead to User priorities may lead to

unreliabilityunreliability Prejudice in responsesPrejudice in responses


Lake What Road?Lake What Road?


Not enough Contributors -Not enough Contributors -Data Points?Data Points?


User Priorities - OooopsUser Priorities - Oooops


Prejudice in Response?Prejudice in Response?


Prejudice in ResponsePrejudice in Response


UGC And Spatial DatabasesUGC And Spatial Databases


Spatial Database CreationSpatial Database Creation


What’s Being Optimized In What’s Being Optimized In The Previous Process?The Previous Process?

spatial data qualityspatial data quality Accuracy of positionAccuracy of position

resolutionresolution Accuracy of AttributionAccuracy of Attribution

Logical ConsistencyLogical Consistency CompletenessCompleteness

Including spatial coverageIncluding spatial coverage Temporal relevanceTemporal relevance MetadataMetadata


How Optimized?How Optimized? Data Quality is an integral part of the processData Quality is an integral part of the process

InitiallyInitially Data collected according to specificationsData collected according to specifications

Bad data re-collected or placed in the update queueBad data re-collected or placed in the update queue OngoingOngoing

Every year significant spatial changes are accommodated.Every year significant spatial changes are accommodated. Areas of high change are identified and updated.Areas of high change are identified and updated. Other changes are found by systematically working Other changes are found by systematically working

research teams through the entire coverage over timeresearch teams through the entire coverage over time The overall assignment is designed toThe overall assignment is designed to maximize the maximize the

time value of money, time value of money, while increasingwhile increasing the integrity of the integrity of the database.the database.


HarmonizationHarmonization

It is this attempt to It is this attempt to actively actively harmonize all dataharmonize all data that distinguishes that distinguishes database building efforts.database building efforts.

Important IssuesImportant Issues Who directs crowdsourced data from an Who directs crowdsourced data from an

editorial perspective?editorial perspective? Who sets standards for crowdsourced data?Who sets standards for crowdsourced data? Who Quality Controls crowdsourced data?Who Quality Controls crowdsourced data? What external guidance exists in What external guidance exists in

crowdsourced systems ? crowdsourced systems ?


Three Categories of Spatial Three Categories of Spatial DataData

Controlled dataControlled data OS, Navteq, TeleAtlas, INFOusaOS, Navteq, TeleAtlas, INFOusa

Hybrid (a mix of controlled and Hybrid (a mix of controlled and uncontrolled data)uncontrolled data)

Google, Yahoo, MSN, TomTomGoogle, Yahoo, MSN, TomTom

Crowdsourced (uncontrolled)Crowdsourced (uncontrolled) OSM, Flickr, etcOSM, Flickr, etc


IssueIssue

It is possible to manage It is possible to manage controlled data controlled data qualityquality to meet specific requirements to meet specific requirements

It is possible to manage It is possible to manage hybrid data hybrid data qualityquality to meet specific requirements to meet specific requirements

But can you manage But can you manage crowdsourced data crowdsourced data qualityquality to meet specific requirements on a to meet specific requirements on a reliable basis?reliable basis?

Let’s look at database compilation for Let’s look at database compilation for some insightssome insights


CompilationCompilation

CommercialCommercial Training in Training in

compilationcompilation SpecializationSpecialization Staff size limitedStaff size limited Research limitedResearch limited Sweat of the browSweat of the brow

But salaried sweat of But salaried sweat of the browthe brow

WikiWiki Self SelectionSelf Selection Local experienceLocal experience Staff size Staff size

potentially potentially unlimitedunlimited

Research hours Research hours potentially potentially unlimitedunlimited

AvocationAvocation


Compare and ContrastCompare and Contrast CommercialCommercial

What are my coverage What are my coverage goals?goals?

What are my accuracy What are my accuracy goals?goals?

How Much can I spend How Much can I spend on updating?on updating?

What size of capable What size of capable staff can I afford?staff can I afford?

How well can I pay How well can I pay them?them?

How can I otherwise How can I otherwise incent them to create incent them to create the best database the best database possible?possible?

WIKIWIKI How many people will How many people will

contribute?contribute? How many are How many are

capable?capable? Where are they Where are they

located?located? Does this match areas Does this match areas

of weak coverage?of weak coverage? How long will it take to How long will it take to

get good results over get good results over large coverages?large coverages?

How to motivate these How to motivate these collaborators over long collaborators over long periods?periods?


What Are The Potential What Are The Potential Weaknesses of WIKI?Weaknesses of WIKI?

Common issuesCommon issues Not enough data gatherers to validate the data Not enough data gatherers to validate the data

or a method to redeploy themor a method to redeploy them Not enough coverage to meet the need (the Not enough coverage to meet the need (the

distribution of the UGCers)distribution of the UGCers) Or a method to redeploy themOr a method to redeploy them

Lack of StandardsLack of Standards Lack of Quality ControlLack of Quality Control

But all of these limitation can be But all of these limitation can be accommodatedaccommodated


Getting Around Some UGC Getting Around Some UGC IssuesIssues


Are Other Types of Spatial Are Other Types of Spatial Databases Superior?Databases Superior?

Even with the benefits of Moolah ($) -Even with the benefits of Moolah ($) -Major navigation databases areMajor navigation databases are Out of dateOut of date InaccurateInaccurate Non-comprehensiveNon-comprehensive Variable qualityVariable quality Too expensive to maintainToo expensive to maintain

Navteq database extension and update Navteq database extension and update costs in 2007 were over $300,000,000costs in 2007 were over $300,000,000


www.refnum.com/osm/gmaps/www.refnum.com/osm/gmaps/Haywards HeathHaywards Heath


And That’s Why UGC and And That’s Why UGC and NeogeographersNeogeographers

Will become an integral part of Will become an integral part of building spatial databasesbuilding spatial databases

Hybrid data collection systems using Hybrid data collection systems using UCG and controlled data are where UCG and controlled data are where geospatial is goinggeospatial is going Let’s lookLet’s look


Old Information SharingOld Information Sharing


New Information SharingNew Information Sharing


What’s The New ProcessWhat’s The New Process


Social Networking Tools Of Social Networking Tools Of Interest in CompilationInterest in Compilation


Spatial Data CollectionSpatial Data Collection

Some UGC will be activeSome UGC will be active User connects to an app and enters relevant User connects to an app and enters relevant

spatial data for updating or extending a spatial data for updating or extending a spatial databasespatial database

Some UGC will be passiveSome UGC will be passive Device tracks and reports (anonymously) Device tracks and reports (anonymously)

user paths, builds database by merging path user paths, builds database by merging path information over timeinformation over time

Passive is particularly useful in building navigation Passive is particularly useful in building navigation databasesdatabases


Relative CostRelative Cost

Data Compilation Techniques

0

20

40

60

80

100

120

140

160

C C + D C +D +A C +D +P C+D+P+A

Rel

ativ

e C

ost


Relative AccuracyRelative Accuracy

Data Compilation Techniques

020406080

100120140160180200

C C + D C +D +A C +D +P C+D+P+A

Rel

ativ

e A

ccu

racy


Summing UPSumming UP

Data Collection SystemsData Collection Systems Closed – commercial compilation efforts, Closed – commercial compilation efforts,

no UGCno UGC Open – WIKI approaches, no proprietary Open – WIKI approaches, no proprietary

datadata Hybrid – where geospatial is goingHybrid – where geospatial is going

Advantages spatial data accuracy by Advantages spatial data accuracy by contributing the best of both approaches.contributing the best of both approaches.


Raises These QuestionsRaises These Questions

Will the winners beWill the winners be Established commercial companies that Established commercial companies that

capitalize on UGC to augment their capitalize on UGC to augment their data?data?

New competitors that commercialize New competitors that commercialize UGC and augment these data to UGC and augment these data to compete with established commercial compete with established commercial systems?systems?


PND Data Flow – A PND Data Flow – A WinnerWinner


UGC Open Street Data UGC Open Street Data Flow – No MedalFlow – No Medal


Commercializing UGCCommercializing UGC


Relative Benefits Of Types Relative Benefits Of Types Of UGC By DeviceOf UGC By Device


Why We Need UGC and Why We Need UGC and NeogeographersNeogeographers


ThanksThanks