Enterprise OSS Collaboration · Architectural reference for how Red Hat IT has ... Quick rate of provision/deploy

Will Foster, Red Hat Inc.

Enterprise OSS Collaboration: RHEV3, RHEL6, Zimbra and iSCSI

Will FosterSr Sysadmin and IT Storage Lead, Red HatFebruary 18, 2013


This is a talk on..

● Architectural reference for how Red Hat IT has deployed Zimbra for collaboration services

● Design and layout● Usage, footprint,

● Inherent benefits of the design● Config management/provisioning● Modularity● Storage snapshot-based recovery

● Future enhancements in automation/admin● 'SnapCreator' framework


This is not a talk on..

● Migration guide from Exchange → Zimbra

● A setup guide or sales demo for Zimbra

● Informed discussion about licensing

● Anything specific to Red Hat internal services

● A guide for bathtub plutonium enrichment


Evolution of Email


Evolution of Email

● 1971 – First ARPANET email was set

● 1971 – Mailbox Protocol

● 1980+ SMTP protocol: start of wide adoption● 1983+ Sendmail and lots of beards starting to grow..

● 1987-1991 – Microsoft and MAPI● Proprietary protocol● Spec's finally released in 2007

● 2001: 42% of all public mail servers are Sendmail● 2012: 12% of all public mail servers are Sendmail● 2012: 43% of all public mail servers are Exim● 2012: 23% of all public mail servers are Postfix● 2012: 12% of all public mail servers are Microsoft Exchange

Source: E-soft survey: http://tinyurl.com/avykuk3


Need for Centralized Mail Services

● Legal discovery/retention

● Widespread use of calendaring

● Administration overhead in rapid growth

● Decentralized/departmental email servers still popular in academia


Why Zimbra?

● Need for 'enterprise' collaboration● Activesync/mobility (Android, IOS)

● Exchange/Notes/etc compatible Calendaring

● OSS stack components

● Avoid lock-in

● Multiple access methods and choice of client

– IMAP, POP, web interface, external webmail, pushmail

Fun Tip #2343: The name “Zimbra” was derived from the Talking Heads1979 album “Fear of Music”


Why use it?

● Stack built on OSS technologies● Separate MX infrastructure

● Modular vs. Monolithic

– Isolated vs. Glorious Failure

● Postfix, MySQL, Nginx, JettyApache, OpenLDAP, memcached

● Less Evil.. ● "All mail clients suck. This one just sucks less." - Mutt (1995)


Vendor Lock-in and risks of proprietary standards

Microsoft Exchange 5.5 → 2000/2003 ● Requirement of full Active Directory infrastructure● “This is my cousin, Frank. He can crash here right?”

Client Access: MAPI (limited to MS Outlook variantsor typically unsupported plugins for MAPI (Evolution)

● IMAP4/POP3 can be enabled but not default

Mailbox Format: proprietary● However some vendors have migration utilities

(google apps, Zimbra, Zarafa, others)


Zimbra Software Components


Components. Experience.

RHEL bare-metal IBM blades

Red Hat Cluster Suite

Zimbra 6.x (network edition)

4Gb Fibrechannel SAN LUNs

RHEL5.x Sendmail MX tier

Nginx/Apache Proxy Tier

OpenLDAP Backend

Bare-metal & clustered setup takes longer to provision/scale

DM-Multipath experiences on EL4 could be better

Zimbra/RHCS only supported on EL4 at the time

RHEL4/5 Bare Metal

Initial Zimbra Architecture (2008-2010)

Cisco MDS FC SANon NetappRed Hat Cluster Suite


Components. Benefits.

RHEV3.0 Hypervisors

RHEL6.3 Mail and MTA Nodes

Zimbra 7.2.0 (network edition)

Netapp 10GbE iSCSI LUNs

RHEL6.3 Sendmail MX tier

Nginx/Apache Proxy Tier

OpenLDAP Backend

OSS stack components

Snapshot based backups

Fast provisioning/deployment

Modular Design

RHEV3.x

Zimbra Architecture Overview Today

10GbE iSCSI Storage(Netapp)RHEL6 Mail Nodes


Zimbra Architecture: Application Overview

● 13 Mailbox nodes

● 1 Failover/reserved mailbox node● Restores, reserved capacity, etc

● 2 Utility Nodes (1 x archive , 1 x admin)

● 2 Proxy Nodes (Apache/Nginx)

● 3 LDAP nodes (2 x replica, 1 x master)

● 3 MTA nodes (load balanced)

● 5 Internal MX servers (sendmail)


Zimbra Architecture: Usage and Footprint

● 7,500 User accounts● Includes team calendars, shared accounts, meeting rooms● 300 mailboxes per node on average

● 1,216,058 inbound mails handled per work day● Average 965,388 inbound mails per day

● 41TB iSCSI LUN mailbox storage

● 37TB iSCSI off-site replication

● 49TB in NFS local mailbox backups


Zimbra Architecture: Application Overview


Zimbra Architecture: Mail Flow Overview


Zimbra Architecture: Backend Storage & DR


Why iSCSI ?

● Operationally cheap SAN block storage

● Great pricepoint (no need for expensive $$ FC SAN)

● First moved to RHEV at version 2.2: no direct FC SAN support

● Introduction of 10GbE Networking into Red Hat Infrastructure

● Transactional workloads, Lots of small files = better on block

● Performance onpar with FC: TOE, Jumbo frames, 10GbE


Netapp Backend Technology

● 3 x HA-pairs of FAS3270A (primary)- Mail nodes/resources spread evenly across HA pairs

● 1 x HA pair of FAS3270A (off-site)

● 10GbE iSCSI LUN connectivity: Cisco Nexus 5k

● Zimbra Mailbox deduplication average: 27%

● Zimbra off-site DR accomplished by 'Flexclones'- instant, innode pointer based volumes created from snapshot


Spam and Filtering

● Milter based application (MimeDefang)● Scans email and assigns spam score before we taking responsibility for message.

Do not have to “bounce” or “discard” spam messages● Calls Anti-Virus and Spam Scoring apps during the SMTP dialog

● Subscription based RBLs (commercial)● SpamHaus● Pattern matching in message content● SpamAssassin for scoring


Automation and Provisioning Concepts

● Cobbler/Puppet help automate VM provisioning and configs

● Keep nothing important on VM's – always use shared storage

● Virtual machines/servers are cheap commodities more easily replaced than fixed in most cases

● Quick rate of provision/deploy = faster scale

● ISCSI storage = one command away from block storage


Future Improvements: Snap Creator

● Snap creator is an open framework that integrates with Netappsnapshot commands and 3rd party applications

● “Snap Creator for Zimbra” allows full and incremental snapshotbased backups of multi-node Zimbra environments while online.

● Application and Crash consistent, Application-aware

● OSS software supported by Netapp

● Our future improvement: automated snapshot backups


Email Musings

● More than 97% of all emails sent over the net are unwanted

● Re: Re: Re: Re: Re: Re: Kittens ← most popular subjects

● The first known spam email, advertising a DEC product presentation, was sent in 1978 by Gary Thuerk

● Avoid HTML email when possible, it can get out of hand..(if you use HTML email make sure to use <blink> tags :)


How I view HTML email being taken too far


Summary

● Proprietary solutions can cause you lock-in and headache

● Build solutions which are modular, avoid monolithic design

● Aim for ease and automation in deployment

● Block-level storage tends to work better with small filesand transactional workloads (iSCSI, SAN)

● Many (mostly) OSS collaboration solutions out there..Zarafa, Zimbra, Atmail, etc.

● Every organization is different


● http://riser38.deviantart.com/art/Clone-Army-348790671

● https://en.wikipedia.org/wiki/Microsoft_Exchange_Server

● http://wikipedia.org (multiple)

● http://ko.fotopedia.com/items/flickr-3189975360

● (cc) JD Lasica/Socialmedia.biz

● http://www.securityspace.com/s_survey/data/man.201112/mxsurvey.html

● http://singletrackworld.com/forum/forum/off-topic

● https://www.zimbra.com/docs/os/6.0.8/administration_guide/2_Overview%20System%20Architecture.03.4.html

● https://en.wikipedia.org/wiki/Email_spam#Statistics_and_estimates

● http://www.netapp.com/us/products/management-software/snapcreator-framework.aspx

References, Attributions and Usage

http://www.securityspace.com/s_survey/data/man.201112/mxsurvey.html

https://www.zimbra.com/docs/os/6.0.8/administration_guide/2_Overview%20System%20Architecture.03.4.html

https://en.wikipedia.org/wiki/Email_spam#Statistics_and_estimates

Documents

Enterprise OSS Collaboration · Architectural reference for how Red Hat IT has ... Quick rate of provision/deploy