22
Broadband Session Michael Byrne

Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Embed Size (px)

Citation preview

Page 1: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Broadband Session

Michael Byrne

Page 2: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Broadband Map Technical Details • Data Integration

• Map Presentation

• Since Launch

Page 3: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Data Integration – What we did

• Review data for format and content• Poured data into 1 container• Summary Statistics – Feedback to awardees• Analyze data for completeness• Incorporating speed data from Overview• Wireline Statistics (any geography, population, hhs)• Wireless Statistics (any geography, population, hhs)

Page 4: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Data Integration – Challenges / Innovations

• Challenge #1 – Data integrity problems– Feedback; communication; client side cleaning scripts

• Challenge #2 – Speed in a different geography– Substantial hand matching of providers to all records

• Challenge #3 – Spatial integrity of data– Parallel software processing; OGC Validation tools

• Challenge #4 – Spatial data complexity– Custom code for dissolving features (2/1/11)

• Challenge #5 – Calculate availability (pop / hhs)– Custom code for spatially distributing households

Page 5: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map Presentation – What we did• 1st requirement was entirely built on RESTful services• Wanted to embody the spirit of the internet (innovation,

fast response, full volume of data)• Wanted to serve multiple users• Required to serve out 100% of data (that we could)• Wanted to make data (all of it) easily consumable• Wanted to make a .gov look like a .com• “Searchable and interactive”

Page 6: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map – Challenges / Innovations• Challenge #1 – Time (we, like the awardees didn’t have

any)– Build a prototype; re-use code;– Agile development cycle– Rapid Application Development; – Separate the application development into component parts;– Focus on foundation architecture;– Use the right pieces for the solution

Page 7: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map – Challenges / Innovations• Challenge #2 – Provide multiple points of entry

– REST– Develop an architecture to translate codes to english– Put english words in the REST url– Develop an API information architecture to support ANY

combination of geography and metric– Build on standards (e.g. no plug-ins); this site is only html, CSS

and JavaScript– Build reusable modules in Web Tier (e.g. WordPress) – 475

modules

Page 8: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map – Challenges / Innovations• Challenge #3 – how can you serve this volume of data

this quickly (Some will say we didn’t)– Cache, cache, cache

• We pre-cache the geospatial tiles to level 10, then cache on the fly for every spatial zoom

• We implemented an open source caching software to cache queries (Varnish – it returns 89% of all hits)

– Develop right approach for search; and map• Search is point in poly• Map is dissolve

– Single color maps– Focus on what we control; leave basemap to others

Page 9: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map – Challenges / Innovations• Challenge #3a – how do you serve this fast

Page 10: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Map – Challenges / Innovations• Challenge #3a – how do you architect this back end

– Focus on the data to make it consumable for your audience (23 tables; ~120 view; views => cached tiles)

– Table structure• 22 million records over 23 tables• 2 tables make up 93% of all records• Organized by unique values

– Type of geography; Type of technology; Number of providers; Holding companies

• Reduce data to common denominators– Provider by Block; Technology type; download speed;

upload speed– Know someone else can do it better than us

Page 11: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch• First 11 hours

– 158,123,884 hits (89 served by cache)– 862 GB served– Peak requests (8,970 per second; average 1,095 per second)– ~500,000 visits

Page 12: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch

Page 13: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch

Page 14: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch

Page 15: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch

Page 16: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Site Launch• 30,000 user feedback votes

Page 17: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Since Launch

• http://www.coldfusionjedi.com/index.cfm/2011/2/17/National-Broadband-Map-jQuery-Demo

• http://www.Broadband.com • Code for America• US Department of Education

Page 18: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch
Page 19: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

• The bottom line• Developers who are thinking about integrating with your

platform want three things. First, they want to improve their application. Second, they want to get more users. Third, they want the first two things to take as little time as possible.

Page 20: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Scaling to billions …• Search Engine Optimization• Level 1 – create enough buzz around title of project –

keyword national broadband map• Level 2 – optimize for combination of key words +

location (broadband map and new york)• Level 3 – educational and consumer oriented

(broadband map speed 10 mbs)• Level 4 – national and strategic keywords + provider

name (national broadband map + verizon)

Page 21: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Scaling to billions …• Level 1 – keyword national broadband map

– Reached #1 w/i 48 hours of launch

• Level 2 – combination of key words + location– Reached spots 1 – 10 in first week of launch

• Level 3 – educational and consumer oriented– Reached #1 and 1st page for 4 of our pages (Classroom, Rank,

map)

• Level 4 – national and strategic keywords + provider – Reached #1 w/ our APIs and first page w/ Search

Page 22: Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

Thanks

Michael Byrne

[email protected]

Geographic Information Officer