Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation Since Launch

  • View

  • Download

Embed Size (px)

Text of Broadband Session Michael Byrne. Broadband Map Technical Details Data Integration Map Presentation...

  • Slide 1

Broadband Session Michael Byrne Slide 2 Broadband Map Technical Details Data Integration Map Presentation Since Launch Slide 3 Data Integration What we did Review data for format and content Poured data into 1 container Summary Statistics Feedback to awardees Analyze data for completeness Incorporating speed data from Overview Wireline Statistics (any geography, population, hhs) Wireless Statistics (any geography, population, hhs) Slide 4 Data Integration Challenges / Innovations Challenge #1 Data integrity problems Feedback; communication; client side cleaning scripts Challenge #2 Speed in a different geography Substantial hand matching of providers to all records Challenge #3 Spatial integrity of data Parallel software processing; OGC Validation tools Challenge #4 Spatial data complexity Custom code for dissolving features (2/1/11) Challenge #5 Calculate availability (pop / hhs) Custom code for spatially distributing households Slide 5 Map Presentation What we did 1 st requirement was entirely built on RESTful services Wanted to embody the spirit of the internet (innovation, fast response, full volume of data) Wanted to serve multiple users Required to serve out 100% of data (that we could) Wanted to make data (all of it) easily consumable Wanted to make look like Searchable and interactive Slide 6 Map Challenges / Innovations Challenge #1 Time (we, like the awardees didnt have any) Build a prototype; re-use code; Agile development cycle Rapid Application Development; Separate the application development into component parts; Focus on foundation architecture; Use the right pieces for the solution Slide 7 Map Challenges / Innovations Challenge #2 Provide multiple points of entry REST Develop an architecture to translate codes to english Put english words in the REST url Develop an API information architecture to support ANY combination of geography and metric Build on standards (e.g. no plug-ins); this site is only html, CSS and JavaScript Build reusable modules in Web Tier (e.g. WordPress) 475 modules Slide 8 Map Challenges / Innovations Challenge #3 how can you serve this volume of data this quickly ( Some will say we didnt ) Cache, cache, cache We pre-cache the geospatial tiles to level 10, then cache on the fly for every spatial zoom We implemented an open source caching software to cache queries (Varnish it returns 89% of all hits) Develop right approach for search; and map Search is point in poly Map is dissolve Single color maps Focus on what we control; leave basemap to others Slide 9 Map Challenges / Innovations Challenge #3a how do you serve this fast Slide 10 Map Challenges / Innovations Challenge #3a how do you architect this back end Focus on the data to make it consumable for your audience (23 tables; ~120 view; views => cached tiles) Table structure 22 million records over 23 tables 2 tables make up 93% of all records Organized by unique values Type of geography; Type of technology; Number of providers; Holding companies Reduce data to common denominators Provider by Block; Technology type; download speed; upload speed Know someone else can do it better than us Slide 11 Since Launch First 11 hours 158,123,884 hits (89 served by cache) 862 GB served Peak requests (8,970 per second; average 1,095 per second) ~500,000 visits Slide 12 Since Launch Slide 13 Slide 14 Slide 15 Slide 16 Site Launch 30,000 user feedback votes Slide 17 Since Launch nal-Broadband-Map-jQuery-Demo nal-Broadband-Map-jQuery-Demo Code for America US Department of Education Slide 18 Slide 19 The bottom line Developers who are thinking about integrating with your platform want three things. First, they want to improve their application. Second, they want to get more users. Third, they want the first two things to take as little time as possible. Slide 20 Scaling to billions Search Engine Optimization Level 1 create enough buzz around title of project keyword national broadband map Level 2 optimize for combination of key words + location (broadband map and new york) Level 3 educational and consumer oriented (broadband map speed 10 mbs) Level 4 national and strategic keywords + provider name (national broadband map + verizon) Slide 21 Scaling to billions Level 1 keyword national broadband map Reached #1 w/i 48 hours of launch Level 2 combination of key words + location Reached spots 1 10 in first week of launch Level 3 educational and consumer oriented Reached #1 and 1 st page for 4 of our pages (Classroom, Rank, map) Level 4 national and strategic keywords + provider Reached #1 w/ our APIs and first page w/ Search Slide 22 Thanks Michael Byrne Geographic Information Officer