MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB

MongoDB Days Silicon Valley

Winning the Dreamforce Hackathon with MongoDB

db.mongoDaysSV.insert( {

“init”: “Welcome Message”, “author”: “Greg Deeds”,

“twitter”: “@TEGBIZ”, setInterval: 162000

} );

RAPID PROTOTYPING with MongoDB

You want to change the game?

You have 24 hours...

Two True Tales Of MongoDB

MongoDB & Force // GR • API driven rapid prototyping • HEROKU CONTAINERIZATION • MASS IMPORT DATA SCIENCE

DRONEY EXPRESS // MongoDB your range • Geospatial & Autonomous

MongoDB or RDBMS

Winning with MongoDB

• Developing Rapid Prototypes

• Winning Hackathons with MongoDB

• Drones / MongoDB {Range Management}

MongoDB or

stay in the RDBMS Matrix

Give Up The Old And Win

• Traditional DevOps Practitioner moved to Micro-services

{Microsoft .NET & SQL Server}

{Oracle APEX-PL/SQL & Oracle Server} {Any Linux... MySQL/POSTGRES}

• Teamwork to overworked evolutions

• My SQL Dreams never materialized - {WHATS NEXT!}

• AGILITY TRAIN YOUR MIND AS A LIFESTYLE

The Early Days

MY BACKGROUND

• Started in 1979 {Ti99/Tape Deck} • Wrote first database on Commodore • BBS Author/ISP {One of the young ones} • Multimedia Producer // Software Engineer • Worked for corporations to startups

{ Military / Federal / Education } • The Writing is on the Wall - MongoDB Time!

My Computing History

TI-99/4A {circa 1982}

Monuments to Awesome-ness

Languages & Coding {Berkeley Logo}

Languages & Coding {Learning to Communicate}

Languages & Coding {Standing on the Shoulders of Giants to Communicate}

MongoDB Queries Support Projection!

PROBLEM: The data is available but in various formats, not easy to use and is changing

• Accessing your government should not be data science or outside of SalesForce.com

• PLATFORM HEROKU - {the app processing grid}

Our target user ranges from the socially conscious individual to Publicly Traded Companies and Governments. Our comprehensive tool will allow…

• Grassroots Organizations, Associations • Publicly Traded Companies • Lobbyists, PACs and Governments

Manage information

• Up-to-Date Congressional Legislator Directories

• Congressional / State Legislation Tracking

• Easy Meetings Management

• Bill and Amendments Tracking

• Committees and more

We are most proud of the tool because it allows both individuals and organizations to understand the landscape of our Government, politics, and social activism.

MongoDB & Droney Express

ForceGR / SPEC

• FORCE.COM APP Generator • APEX & PL/SQL Classes • Restful NODEJS HEROKU • Mongo for Import, Reduce and Production

A Global Competition

Data Sets Matter• Variety Of Formats

• YAML // XML // JSON // CSV // TAB

• Normalized only for XML / DTD files.

• Multiple websites crawled/parsed {Using RSYNC || PHP To Bulk Load Objects}

• Aggregate Pipeline

• Build Words / Terms from triples

• Semantic to understand {What you say}

• Latent Semantic to understand {What you mean}

• MongoDB // Provides both when used in pipeline

with other word bag or terminology generators

• The results are we find correlation in unnormalized

data

Data Miner Mongo DB

Over 500k Objects

• Monitor the query in MongoDB reactively {Check to see how the indexes need to improve with a statistics collection of performance and counters for the API}

• Compound indexes need to be built on geojson first then other fields when used

• Consider using elastic search to boost the Hackathon Team to victory if free text is slow

Use Conditional Indexes {add Agility_new ways working with this data?}

• Find source files over APIs //

{ to maintain high speed with no throttles. Hackathon bandwidth is varied sometimes when you need it. It gets worse with time }

• Use Old School Shell Tools // { NodeJS is not ideal for all challenges }

• PHP • PERL • PyThon • RSYNC

MISSION: Data Capture

• The focus of TEG’s FORCEGR tool is to present information into SalesForce users Desktop

• Then to make decisions about responses to these decisions

• Further guiding the NodeJS learning engine

APEX CLASSES / INTERACTIONS

question =

(my_department, has, what)

knowledge = (

(I, own, my_apartment),

(my_department, has, my_computer),

(my_department, has, my_vote),

(my_department, is_in, Philadelphia)

)

‘$push’ ‘$pop’

‘$unset’ ‘$set’

bye bye SQL...

Sew the Dataset

The Team

• .NET/Oracle APEX/PLSQL is heavy Spiral DEVOPS

• The Browser Wars- We Won {HTML5/Javascript Rocks}

• SUBVERSION TO GIT DEVOPS TO NEW CHATOPS {Your Dated If You Can’t GIT With It…}

• Shifts in web computer science

• Multitouch surface the GPU and JavaScript are the next big shift

Recognizing Shift

• Practice writing specifications for NOSQL

• Use New tools to trap ideas

• CHECK YOUR ROADMAP? { are you on course & time? }

• Gain momentum by solving problems quicker

• GIT more done { Practice like you play the game }

• Organization to Reproduce Results {use automated ‘Agents’ when possible & makes sense} {containerize your app with docker or Heroku}

Next Generation DevOps

question = (my_department, has, what)knowledge = ( (I, own, my_apartment), (my_department, has, my_computer), (my_department, has, my_vote), (my_department, is_in, Philadelphia) )for each statement in knowledge { if ((statement.subject == question.subject or question.subject == what) { and (statement.predicate == question.predicate or question.predicate == what) and (statement.object == question.object or question.object == what)) call FoundAnswer(statement) }}

Output: Answer: my_department has my_computer Answer: my_department has my_vote

Pseudocode for Question- Answering Logic in NodeJS

• Ask yourself on each project freshly {What else is aggregation good for?}

• HINT: You can use the $out pipeline operator and it will create or replace the collection it spits out, after the results are ready....

Aggregation Tips { Use the Pipeline, Luke! }

if (! Links.find().count()) { // init data var nodeNames = []; var data = ['Harry,Sally,1.2', 'Rubio,Mario,1.3', 'Obama,Barrack,0.2', 'Thompson,Fred,0.5', 'Peter,Johan,0.7'];

_.each(data, function (val) { var comp = val.split(',');

Links.insert({ source: comp[0], target: comp[1], compute: comp[2] }); nodeNames.push(comp[0]); nodeNames.push(comp[1]);

});

// remove duplicates nodeNames = _.uniq(nodeNames); _.each(nodeNames, function (name) { Things.insert({ _id: name }); }); }

Finesse by using new tools

VICTORY It never lasts long enough....

MongoDB & Droney Express

• HTML5 & WebGL ODM Tools

• MongoDB Powering Geoserver Raster

• Realtime 3-D Model Scenes In A Browser

Hibernate ORM is Dead {long live MongoDB ODM}

Hibernate ORM VS MongoDB ODM • Hibernate Maps Java Objects To RDBMS And Is

Proven Not Scalable

• RDBMS load times for a single object graph (Hibernate/SQL Server) from 3 to 4 seconds

• Relational model is not suited for complex object graphs.

• Even with massive parallel processing is not economical

Hibernate Search • Cannot Beat Elastic-search & MongoDB In

Response Speed

Business Opportunity

• ORM Hibernate JBOSS - Object/Relational Model {storage has reached the end of life period…}

• MongoDB is the best object store on the market today • Hibernate is complex when the world needs BSON • MongoDB is Built For Replication • Hibernate Replacement Is An Opportunity.

Market Need {MongoDB Sensor ODM}

Project Requirements • Open Source/No Internet/Limited WiFI

• Runs on variety of low cost hardware

• Auto-deploys Via Docker On IoT Appliance

• Plugin To Existing GeoServer/ESRI/GRASS

• Replace Hibernate With MongoDB

Droney Express {Range management System}

• Interchange GeoJSON set of Shapes & Waypoints

• RasberryPi Flight Computer running 2d Queries

• Objects Linked To External GIS Engine (GRASS/ESRI)

• AutoPilot Configuration In Flight/Over-The-Air

• MongoDB Monitors Situational Awareness Of Missions

• GIS Query Results Stream As BSON Objects

What is a Mission?

Geographic Forensics {sensor data store/search}

The work of Droney Express

• Planned Geofence Scanning

• 2D/3D indexed sensor data

• GIS & MongoDB Is Powerful

• Integrated GIS Remote Monitoring

• Connect Using DDP To Sync Collections Adhoc

Sensor Range Management

Async Data Sync -> Use Oplog For Synchronization

Serve GIS markup with pretty maps using Geoserver...

MongoDB is perfect for Sensors // sorry Oracle

Flight Plan For Best Routes

GIS Engines Accelerated By MonogDB Can Handle

more ecological range management sensors faster coordinated ground and drone assets

GEOJSON / MONGODB 2d DRAINAGE MODELING

GIS ENGINE & PLUGIN Droney Express Flight Planner

Least Cost Power & Route Calculator Export Tools For 3DR Ardupilot.

Replica Sets Are Ideal For Network Segregation

HQ is Always Primary - Drones Always Secondary

Force Constant State of Disaster Recovery - Dedup

Drones Are Small Autonomous Collection Vehicles

Eventual Consistency In Loosely Coupled Networks

MongoDB Makes Life Easy

Secondaries DeDup Secondaries To Update HQ GIS Backends For Mission Planning & Sensor Datasets

Droney Heartbeat modeled from mongoDB University

Airborne MongoDB Prevent Election Of New Primary

Droney Flight Computer Always Have Priority Of 0

They Never Get Elected To Primary

[Priority, Optime, Connections]

Sync Using Oplog To HQ After Weeks In Field

Network Environment Spotty For MongoDB Servers

Statement Replication ...Getting The Data To HQ

Droney Acquires Forensic Sensor Data - ‘A Data Silo’

Facilitates Greater Independence

Replicate Based Oplog For Forensic Time-coding

Member Replica Sync Stored Records Using Oplog

Use Oplog To Journal AutoPilot For Flight Review

MongoDB Secondaries Apply Unique ID Vs Primary

Keeps Mobile Workers From Overwriting Datasets

Cheap IoT Devices MongoDB 32bit Flight Computers & Range Sensors With Embedded MongoDB

In Flight Computers Are Low Power/LINUX Threaded

Sensors Streaming Data To/From MongoDB

Autopilot Flight Recorder Is MongoDB/NodeJS

IoT SD Cards - Ideal For Circular Buffers In Flight

Circular Buffers Makes Room For New Documents

Overwrite The Oldest Documents In A Collection

Final Thought.. NO CAMPING … KEEP CLIMBING!

CONSIDER THE FUTURE OF YOUR TIME STUDY NEXT GENERATION DEVOPS SAY “WHY CANT WE FIRST” SHARPEN UP REGULARLY

TEST YOUR SKILLS OR TEACH A SKILL

Greg Deeds Technology Exploration Group

SALES@TEGbiz.com www.TEGbiz.com

@TEGbiz @DRONEYEXPRESS @3dVRviewers