Building a Private Social Network with MongoDBJustin Jenkins@LearnMongo

Learn Mongo (.com that is)

MongoDB resources aimed at not scaring off the beginner.

Twitter: @LearnMongo

Saddleback Church

21,000+ per weekend @ 10 locations. 4,000 online.

4,300 “small groups”20k – 32k people

~1 million user records

3.2 million e-mails a week

Service trips to ~180 countries

Food bank serving Orange County

Features, easy of use …

Project Goals

• Provide a private way for groups to interact and share online.

• Tie into other social networks but keep group data private.

• Provide groups with video & audio resources for growth.

• Create a simple & fast user experience.

MongoDB SQL Server

• Flexible schema• Fairly easy to scale• Optimized for read / inserts

Uh Ohs• New, little past use• Can be harder to query• Different data format

• Solid, durable• Years of use• Very easy to query

Uh Ohs• Not optimized for the

web• Can be complex to scale• Stringent schema

Playing nice with SQL Server

Playing nice with SQL Server

• Keep users accounts in SQL• Use the same user id’s in both SQL and

MongoDB.• Store app specific user data in MongoDB.

• Keep group data and hierarchy in SQL• Extract group data from SQL when any user of

the group logs in, compare.• Push any changes back to SQL in the

background.

Problem Solved?

The Problems MongoDB Solved

• Flexible schema design • Each feed item can have totally different

attributes.• New features can be added quickly.

• Reduces impact on production SQL Server• Focus server resources on internal staff.

• BSON to JSON• Ideal for a highly AJAX / JavaScript site.

The Feed

The Feed

• The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc.

• We need something very flexible …• Discussions• Videos• Events• Meetings• Prayers• Web Links• Pictures

Schema Design

• Flexible Schema• Less clutter.• Cleaner, leaner storage.• Simpler queries.• JOINs largely unneeded.

• BSON Format• Converts into JSON with very little effort to make

JavaScript / AJAX centric apps happy.• Move from the client to the database, back to the

client easily.

{ "_id" : "4d19fb939ac0900274000005", "_t" : "Discussion", "created" : "Tue, 28 Dec 2010 07:00:35 GMT", "GroupID" : 129242, "eIndividualID" : "22872F9", "firstName" : "Bill", "lastName" : "Finch", "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.", "comments": [ { "_id" : "4d19fe329ac090027400000a", "eIndividualID" : "FFE4251", "created" : "2010-12-28T15:11:46.6760000Z", "firstName" : "Robin", "lastName" : "Tally", "commentText" : "I hope that everything is OK, whatever it is!" }, { "_id" : "4d1df3b29ac0900d64000003", ... } ]}

Models (the pretty kind)

using System;using System.Collections.Generic;using System.Linq;using System.Web;using Newtonsoft.Json;using MySmallGroup.Helpers.JsonConverters;

namespace MySmallGroup.Models{ public abstract class Feed : IFeed, IUserCreatable { [JsonConverter(typeof(MongoOidConverter))] public MongoDB.Oid Id { get; set; }

public DateTime created { get; set; } public int GroupID { get; set; } public string eIndividualID { get; set; } public string firstName { get; set; } public string lastName { get; set; } public abstract string shortDescription { get; } public int commentCount { get; set; } public IList<comment> comments { get; set; } public abstract void ReceivedJsonEncode(); public Feed Get() { return this; } }

public interface IFeed : IUserCreatable { MongoDB.Oid Id { get; set; } int GroupID { get; set; } string eIndividualID { get; set; } string firstName { get; set; } string lastName { get; set; } string shortDescription { get; } int commentCount { get; set; } IList<comment> comments { get; set; } void ReceivedJsonEncode(); Feed Get(); }}

using System;using System.Collections.Generic;using System.Linq;using System.Web;using MySmallGroup.Helpers.ExtensionMethods;

namespace MySmallGroup.Models{ public class Discussion : Feed { public string discussionBody { get; set;} public Discussion() { this.commentCount = new int(); this.comments = new List<comment>(); this.created = DateTime.Now; }

public override void ReceivedJsonEncode() { discussionBody = discussionBody.ReceivedJsonEncode(); }

public override string shortDescription { get { return discussionBody; } } }}

Live Chat & Video

Live Chat & Live Video{ "_id": "4cd261549ac0900c24000006", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT", "groupID": 119420, "message": "way faster then dev tho"}{ "_id": "4cd261a69ac0900c2400000a", "senderEIndividualID": "B656641", "senderFirstName": "Steve", "senderLastName": "Chen", "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT", "groupID": 119420, "message": "oh"} { "_id": "4cd2621b9ac0900c2400001f", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT", "groupID": 119420, "message": "I think it&#39;s the facebook pic reload voodoo Jeff made"}

User Settings

User Settings{ "eIndividualID": "A0BE87A", "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false, "GroupSettings": [ { "GroupId": 119420, "FeedNotificationEmail": false, "CommentNotificationEmail": false }, { "GroupId": 128834, "FeedNotificationEmail": true, "CommentNotificationEmail": false }, { "GroupId": 125680, "FeedNotificationEmail": true, "CommentNotificationEmail": true } ]}

Email Tracking …

E-mail Tracking (the boring stuff.)

Emails Sent• 350,000 normal e-mails sent a week.• 460,000 daily subscription e-mails sent a day.

Tracking Needs• Track each e-mail open, per individual.• Date• User Agent• IP

• If a user opens an e-mail more than once, don’t double count but still track.

• Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.

Email Tracking{ "EmailLogID": 14, "EmailLogDocumentID": "4d12a55e00ae2611f8000002", "EIndividualID": “ZZZZZZZ", "Email": "jxxx@zzzzzzzzz.com", "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT", "_id": "4d12a56a18bee80a7c00243d", "Opens": [ { "OpenDate": "2010-12-23T01:59:45", "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) ...", "IP": "72.111.250.111" }, { "OpenDate": "2010-12-23T16:22:19", "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) ...", "IP": "70.111.66.11" } ]}

> db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();

141762

LearnMongo.comTwitter@LearnMongo @MongoQuestion @JustinJenkins