Embed Size (px)
Citation preview
25/11/08
Mike [email protected]
Mailtrust, a division of Rackspacewww.mailtrust.com | www.rackspace.com
Contributing to OSS in a commercial non-OSS environment
2
Overview
□ What is Mailtrust□ What OSS does Mailtrust use□ How has Mailtrust contributed to Funambol□ Design challenges□ Review of integration strategies
3
□ Formerly Webmail.us□ Now a division of Rackspace
◊ Recently went public□ Business-class email hosting□ Noteworthy Webmail
◊ Full-featured AJAX-y webmail application◊ Competitive collaboration suite
4
Mailtrust and Open Source
□ Heavily Used◊ PHP◊ mySQL◊ Hadoop◊ Postfix◊ amavisd◊ policyd◊ Many others
□ Main Contributions◊ Funambol◊ Dovecot
5
Contributions to Funambol
□ DS-Server◊ Webmail Connector
□ Outlook Client◊ Support for 'custom fields'
□ Blackberry PIM Client◊ Initial development in-house◊ Now part of Funambol's core clients
□ iPhone client◊ 2 weeks in Italy
6
Goals for Sync Service
□ Synchronize shared data◊ Read-only
□ Give back to open source community◊ But keep proprietary systems private
□ Focus on a few highly refined clients◊ Outlook◊ Blackberry◊ Windows Mobile
7
System Architecture Evolution, Part 1
DS-Server WebmailModule
Webmail
Data
Initial design:● Very Simple● Connector hit DBs directly
UserDatabase
WebmailDatabase
Authentication
8
System Architecture Evolution, Part 1
DS-Server WebmailModule
Webmail
Lessons Learned:● 2 code bases to maintain!● Data validation duplicated● Proprietary DB schemas in open
source code!● We could not launch with this
architecture
Data
UserDatabase
WebmailDatabase
Authentication
9
System Architecture Evolution, Part 2
Webmail
Introducing the Webmail-Sync API● Implementation of SyncSource
interface in API form● Data parsing now in php
● Implemented a large Vobj library● Hope to open-source it!
● Uses existing structures and validation● HTTP + jsON
● Connector generic enough for other uses● Clear separation between Funambol
and Webmail
APIDS-Server WebmailModule
WebmailDatabase
10
System Architecture Evolution, Part 2
Lessons Learned● PHP has an execution time limit
● Don't do too much at once● Know your technologies intimately
● Distance matters● Reliability, speed deteriorates● Retry failed requests
● getSyncItemByKey is slow● Page calls and cache data
DS-Server WebmailModule Webmail
2000Miles!
API
WebmailDatabase
11
System Architecture Evolution, Part 3
Webmail
Batching calls to API● Can't batch calls w/ standard code
● addItem requires returning a GUID● Went one layer higher
● New BatchedSyncStrategy● New BatchedSyncSource● Completely compatible with original
● Reduced calls by 50-90%
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
12
System Architecture Evolution, Part 3
Webmail
Lessons Learned:● The Strategy is complicated● Test, Test, Test● Funambol is very flexible to these
kind of changes
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
13
System Architecture Evolution, Part 4
Webmail
New Problems● Cannot upload to webmail and
sync at the same time!● Do not want load from API
processing affecting webmail users
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
14
System Architecture Evolution, Part 4
DedicatedWebmail
ForSync
Lessons Learned:● Dedicate systems whenever possible● Scale services independently
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
Webmail
15
System Architecture Evolution, Part 5
DedicatedWebmail
ForSync
New Problems, Again!● Network connectivity issues
caused duplicates● Requests to addItems failed, and
were retried● Sync was hitting webmail
databases too hard ● 50 calendars can have a lot of data
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
Webmail
16
System Architecture Evolution, Part 5
DedicatedWebmail
ForSync
Lessons Learned:● Don't retry 'dangerous' commands● Database connections are limited
resources● Cache everything reasonable!
● Database requests● API requests!
● Open source has a solution
APIDS-Server WebmailModule
Str
ate
gy
WebmailDatabase
Webmail
memcached
mem
cach
ed
17
System Architecture – Misc details
□ SyncSourceRedirectSynclet◊ Replaces remote URI in requests◊ Uses content types sent by client, has defaults◊ 'contacts' => 'contacts-vcard21' or 'contacts-sif'◊ Easier for users to configure devices◊ Guarantees clients get the preferred type, if
available
18
System Architecture – Misc details
□ SessionConnectionLimiterSynclet◊ Disconnects a session that has lasted 'too long'◊ Protects service from runaway clients
19
System Architecture – Misc details
□ Authentication◊ Moved queries to mysql stored procedures◊ Keeps DB schema out of code and configuration◊ Allows installations to redefine these procedures
20
Summary
□ Use abstractions to hide proprietary details
◊ Write open logic, and share it!□ Physically and logically separate open and
closed source programs◊ API separated synchronization logic from
our proprietary systems and data◊ Stored procedures hide our database
behind an interface
21
Summary, cont.
□ Rely on configurations for details◊ Funambol does this already – model after
them◊ Requires flexible code, just the kind open
source needs□ Work at a great company
◊ The kind that values open source◊ The kind that lets you give back!
22
Summary, cont.
□ Coordinate with your community◊ Communication is key◊ Work out a way to share code and plans
often◊ Still working on this one ;)
□ Go to Italy!
