45
Big Data Tap into Cloud Infrastructure with FME March 18, 2014

Big Data – Tap into Cloud Infrastructure with FME

Embed Size (px)

DESCRIPTION

See how to easily migrate to the cloud with FME, and take advantage of the corresponding benefits including unlimited resources, scalability, and zero hardware to maintain. You'll see how you can use new FME 2014 support to move data to Big Data handling tools and services such as Amazon RDS, Amazon S3, Amazon DynamoDB, Amazon RedShift, and Google BigQuery. Plus, learn about the benefits of being close to the data, and how FME Server and FME Cloud can help power the flow of data, whether it's hosted, on-site, or somewhere in between.

Citation preview

  • 1. Big Data Tap into Cloud Infrastructure with FME March 18, 2014

2. Meet the presenters. Don Murray President and Co-Founder @DonAtSafe Dean Hintz Senior Product Specialist @DeanAtSafe 3. Ask us. And join the discussion. Please submit using the GoToWebinar panel. We will follow-up with unanswered questions. 4. Agenda. What is Big Data Big Data Challenges FME and Big Data FME Demos: Loading and Extracting from MarkLogic Spatial Indexing and Loading to DynamoDB 5. What we do. www.safe.com 6. Poll: Which version of FME are you using? 7. New to FME? Get your bearings from our Getting Started Page: www.safe.com/fme/getting-started Learn from our crew in one of the weekly FME Overview webinars: safe.com/WeeklyIntro 8. What is Big Data? 9. Big Data and Cloud Big Data needs big resources Big datastores Big processing power Big bandwidth Cloud technology gives you this for fraction of traditional cost! 10. Big Data and FME Big Data is a new data classification for FME. Big Data is no different than other data to FME FME Cloud is a natural fit for data in the Cloud FME makes it easy to leverage the power of Big Data. 11. Big Data and FME Support Amazon S3 Limitless internet based storage Amazon RDS See blog article on Amazon RDS (PostGIS) Amazon DynamoDB NoSQL limitless database service Amazon RedShift Petabyte scale database warehouse service. Google BigQuery Superfast append only tables MarkLogic Large XML based database 12. Poll: How are you currently working with Big Data? 13. Big Data Challenges Loading Data Lacks spatial support Big Data Analysis Querying and Exporting Data 14. Demo #1 MarkLogic Demo #2 Limitless Spatial Database 15. Why Demo FME with MarkLogic and DynamoDB? Different from other databases supported by FME. 16. What is ? NoSQL database XML optimized Powerful search and analysis Native Spatial Support XML based data model (GML, XML, etc.) Deploy on Hadoop HDFS 17. FME and MarkLogic A Natural Fit Convert data to XML/GML* Easily Load XML into MarkLogic with FME Process and convert XML results FME 2014: New schema based GML Writer 18. Demo #1a Loading MarkLogic Convert GIS / CAD data to GML (XML) Compose REST request to PUT to MarkLogic database 19. 1. Convert GIS / CAD data into Valid GML 2.Generate Key Fields 3. Build insert message 4. Execute PUT REST call MarkLogic accepts any valid XML just PUT it! Loading GIS to MarkLogic 20. Loading GIS to MarkLogic with FME 21. What Big Data technology are you most interested in? 22. Demo #1b Exporting from MarkLogic GET Query to find URIs for features of interest GET Query using URIs to get feature XML/GML, then Conversion to format of choice (CAD, GIS ) /WFS 23. Exporting XML from MarkLogic 1. Query database via GET request 2. Parse search result and compose GET feature request 3. Extract attributes and geometry from result 4. Validate and Write XML Result 24. Exporting XML from MarkLogic Search GET request: http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago Retrieval GET request: http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c- 49735dd29356.xml 25. AIXM from MarkLogic via FMEServer http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.fmw ?Element=airportCode&Value=CYVR /AIXM 26. AIXM from MarkLogic via FMEServer 27. MarkLogic -> Anything (JSON, KML, GML ) 28. MarkLogic to ArcGIS via FME Server: 1. Submit search to MarkLogic as described earlier 2. Extract attributes and geometry from result 3. Generate update ESRIJSON message from feature 4. Post update ESRIJSON to ArcGIS Server MarkLogic / ArcGIS Integration 29. ArcGIS Server to MarkLogic via FME Server 1. Retrive JSON data from ArcGIS Server 2. Generate output GML 3. Write data to MarkLogic via PUT REST call 30. ArcGIS Server to MarkLogic 31. Demo #2 Limitless Spatial Database 32. DynamoDB NoSQL SSD-based database service No limit on size of Database Specify the needed performance Autoscale thru Dynamic DynamoDB Amazon EMR (Hadoop) integration 33. Demo # 2 Index Strategy Generate GeoHash Index for each feature and Write to GeoHashSpatialIndex 34. Demo #2a Vector, Raster, Lidar Write small features to DynamoDB Write large features to Amazon S3, link to DynamoDB 35. Demo #2b Geocoded Images Generate Geohash record of picture location Write Image to S3, link to DynamoDB 36. Demo #2c Spatially Store Anything Generate Geohash index Write Document to S3 and Link to DynamoDB location 37. Demo #2d Spatially Locate any internet resource Write URI Link to DynamoDB Generate Geohash index location 38. What data types are you planning to store in Big Data? 39. Save the date. Webinar: How to Automate Practically Anything with FME Server (March 25th) Webinar: How to Load Data into Google Maps Engine (April 16th) FME World Tour 2014 (April June 2014) FME International User Conference 2014 (20th Anniversary Celebration) June 10 13, 2014 in Vancouver, Canada 40. Free and fun to learn. Online Courses - Live & Hands-On Feb 18-19: FME Desktop Tutorials & Recorded Courses 41. Stay informed. fmepedia.com/community fmepedia.com/knowledge @SafeSoftware youtube.com/FMEChannel blog.safe.com 42. Summary Big Data = big new opportunities FME great for working with Big Data Cloud model is a natural fit for Big Data This is just the beginning - more to come! 43. Hand raising has now been enabled. If youd like to ask a question over the air, please click the hand icon and ensure your audio input is set up. 44. Thank you! Sales [email protected] Support www.safe.com/support (604) 501-9985 ext. 278 Don Murray [email protected] Dean Hintz [email protected]