IndiMail - The Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

Manvendra Bhangui

IndiMailThe Flexible Messaging Platform

Strengths & AdvantagesComponentsFeature ListArchitecture

Installation & ConfigurationAdministrationMaintenance

Long-term GoalsSupport

Hardware RequirementsQuestions

IndiMailThe Flexible Messaging Platform

Strengths & Advantages–All parameters customizable through environment variables (around 200 parameters).

–Customizable globally, locally, dynamically or specifically for a user or group of users.

–Mail can be intercepted before injection, after queuing or before final delivery

–Single instance multiple queues. Concurrency customizable

–Horizontally scalable architecture.

–Crash proof

–Faster than qmail, postfix (the only two noteworthy competitors)

–Full featured (SMTP, POP3, IMAP, SPAM Filter, Virus Scanning, SSL, Domainkeys, DKIM, BATV, etc)

–Highly secure. Can protect vulnerable messaging servers like MS Exchange by acting as frontend SMTP, IMAP and POP3 servers.

–Open Standards – Compliant with most of the messaging RFCs

–Open Source – Source code available (GNU GPL V3).

–accesslist – control email exchanges between senders & recipients. Control exchanges between internet and users. (unique to indimail & satisfies corporate requirements)

–High speed virus/spam filtering by using inline scanning.

IndiMailThe Flexible Messaging Platform

ComponentsMTA -Based on qmail. Extensive changes to make it a modern

MTA and achieve highest possible delivery speed.

IMAP/POP3 - Based on courier-imap. Changes made to support multiple open standards authentication modules (v4.10.x)

Virus Filter - clamAV with automatic signature updates using freshclam (v0.96.x)

SPAM Filter- Bogofilter – robinson fisher algorithm (v1.2.2)

IndiMailThe Flexible Messaging Platform

Feature List– Speed

● Probably the fastest MTA. Multi-queue architecture allows 1.5 millions+ deliveries on a intel commodity hardware

– Setup● automatic adaptation to your UNIX variant

● Linux, SunOS, Solaris, and more

● automatic per-host configuration - gnu autoconf

● High degree of automation of configuration through svctool

● RPM packages for multiple Linux Distros.

IndiMailThe Flexible Messaging Platform

Feature List (cont...)– Security

● clear separation between addresses, files, and programs

● minimization of setuid code (qmail-queue, qhspi, qscanq, systpass)

● minimization of root code (qmail-start, qmail-lspawn)

● five-way trust partitioning---security in depth

● optional logging of one-way hashes, entire contents, etc. (EXTRAQUEUE, mailarchive control file)

● virus scanning through qscanq, clamav.

● Extensible plugin feature for virus scanners

● Inbuilt virus scanner

● sender/recipient accesslist, hostaccess using tcprules

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Message construction

● RFC 822, RFC 1123

● full support for address groups

● automatic conversion of old-style address lists to RFC 822 format

● sendmail hook for compatibility with current user agents

● header line length limited only by memory

● host masquerading (control/defaulthost)

● user masquerading ($MAILUSER, $MAILHOST)

● automatic Mail-Followup-To creation ($QMAILMFTFILE)

● ability to add signature/content to messages using altermime

● Abuse report format (ARF) generator using qarf

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– SMTP

● RFC 2821, RFC 1123, RFC 1651, RFC 1652, RFC 1854, RFC 1870, RFC 1893

● 8-bit clean

● 931/1413/ident/TAP callback

● relay control---stop unauthorized relaying by outsiders (control/rcpthosts)

● no interference between relay control and aliases

● automatic recognition of local IP addresses

● per-buffer timeouts

● hop counting * parallelism limit (tcpserver)

● per host limit (tcpserver - MAXPERIP)

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– SMTP (cont…)

● refusal of connections from known abusers(tcpserver, badmailfrom, badmailpatterns, badhelo, blackholedsender, blackholedpatterns, badhost, badip)

● goodrcptto, goodrcptpatterns which override the above

● blackholercpt, blackholercptpatterns for blackholing mails to specific senders.

● Control files spamignore, blackholedsender, badmailfrom, relaymailfrom, badrcptto, chkrcptdomains, goodrcptto, blackholercpt, badip can be specified in plain text, cdb format as well as stored in MySQL tables.

● relaying and message rewriting for authorized clients

● authenticated SMTP PLAIN, LOGIN, CRAM-MD5, CRAM-SHA1, CRAM-RIPEMD, DIGEST-MD5 HMAC (RFC 1321, RFC 2104, RFC 2554, RFC 2617)

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– SMTP (cont…)

● STARTLS extension, TLS

● Support for SMTPS

● POP/IMAP before SMTP

● ETRN (RFC 1985)

● ODMR (RFC 2645)

● RBL/ORBS support (rblsmtpd)

● DNSBL support using plugin

● SURBL (SURBL Blacklist) using surblfilter

● SPAM Control (Reject/Tag/Accept) using Bayesian techniques

● High Performance MS Virus Control via control file viruscheck andcontrol file signatures

● Content Filtering and blocking of prohibited attachments via control file bodycheck

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– SMTP (cont…)

● Ability to reject/bounce mails for unknown/inactive users (CHECKRECIPIENT)

● ability to have the RECIPIENT check for selective domains using control file chkrcptdomains

● Antispoofing mode (turned on by environment variable ANTISPOOFING)

● Masquerading ability.

● Multiline greetings via control file smtpgreeting

● Message Submission Agent – MSA (RFC 2476)

● Domain IP address pair access control via control file hostaccess

● Per User accesslist via control file accesslist

● SPF – Sender Permitted From

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– SMTP (cont…)

● Per User control of environment variable by envrules(rules file set by environment variable FROMRULES)

● Greylisting[3] capability using qmail-greyd or greydaemon

● Bounce Address Tag Validation (BATV)[4]

● Notify recipient when message size exceeds databyte limits (by setting environment variable DATABYTES_NOTIFY)

● SMTP Plugins using external plugins in /var/indimail/plugins

– Queue management● instant handling of messages added to queue

● parallelism limit (control/concurrencyremote, control/concurrencylocal)

● split queue directory---no slowdown when queue gets big

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Queue Management (cont…)

● quadratic retry schedule---old messages tried less often

● independent message retry schedules

● automatic safe queueing---no loss of mail if system crashes

● automatic per-recipient checkpointing

● automatic queue cleanups (qmail-clean)

● queue viewing (qmail-qread)

● detailed delivery statistics (qmailanalog)

● Ability to hold local, remote or both deliveries (holdlocal, holdremote control file)

● Qmail Queue Extra Header – Ability to pass extra headers to local and remote deliveries via qmail-queue (Environment variable QQEH).

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Queue Management (cont…)

● Configurable number of queues and time slicing algorithm for load balancing via qmail-multi. A queue in indimail is configurable by three environment variables QUEUE_BASE, QUEUE_COUNT, and QUEUE_START. A queue in IndiMail is a collection of queues.Each queue in the collection can have one or more SMTP listener but a single or no delivery (qmail-send) process. It is possible to have the entire queue collection without a delivery process (e.g. SMTP on port 366 – ODMR). The QUEUE_COUNT can be defined based onhow powerful your host is (IO bandwidth, etc).NOTE: This configurable number of queues is possibe with a single installation and does not require you to install multiple instances of qmail.

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Queue Management (cont…)

● External Virus scanning via QHPSI – Qmail High Performance Scanner Interface

● Ability to extend QHPSI interface through plugins. The keyword plugin:shared_libdefined in the environment variable QHPSI denotes 'shared_lib' to be loaded.

● Virus scanner qscanq. Ability to detect virus via a third party scanner defined by SCANCMD environment variable (clamscan, clamdscan, etc)

● Blocking of prohibited filename extensions via qscanq program

● Domainkeys (qmail-dk) RFC 4870 * DKIM[5] with ADSP/SSP (qmail-dkim) RFC 4871

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Queue Management (cont…)

● Set all header values listed in envheader control file as environment variables.

● Log all headers listed in control file logheaders to stderr.

● Remove all headers listed in control file removeheaders from email.

● Ability to do line processing instead of block processing.

● qmail-nullqueue – blackhole the mail silently.

● rule based mail archival using control file mailarchive (SOX, HIPAA compliance)

● Added additional recipients for a message using extraqueue or mailarchive control file.

● X-Originating-IP header to record the original IP from which the mail originates

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Bounces

● QSBMF bounce messages---both machine-readable and human-readable

● HCMSSC support---language-independent RFC 1893 error codes

● double bounces sent to postmaster

● Ability to discard double bounces

● Ability to preserve MIME format when bouncing.

● Control of bounce process via envrules (rules file controlled by environment

● variable BOUNCERULES or control files bounce.envrules)

● limit size of bounce using control file bouncemaxbytes

● Ability to process bounces using external bounce processor (environment variable BOUNCEPROCESSOR)

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Routing by domain

● any number of names for local host (control/locals)

● any number of virtual domains (control/virtualdomains)

● domain wildcards (control/virtualdomains)

● configurable percent hack support (control/percenthack)

● Clustered Domain. Same virtual domain can exist on multiple hosts, each having its own set of users. Provides Load Balancing and infinite scalability.

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Remote SMTP delivery

● RFC 2821, RFC 974, RFC 1123, RFC 1870

● 8-bit clean

● automatic downed host backoffs

● Configurable tcp timeouts for downed host backoffs.

● automatic switchover to next best MX

● artificial routing---smarthost, localnet, mailertable (control/smtproutes)

● Support for jumbo ISP (control/smtproutes.cdb)

● per-buffer timeouts

● passive SMTP queue---perfect for SLIP/PPP (serialmail)

● AutoTURN support (serialmail)

● Spam control (SPAMFILTER environment variable)

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Remote SMTP delivery (cont…)

● Authenticated SMTP (userid/passwd in control/smtproutes) - PLAIN, LOGIN, CRAM-MD5, CRAM-SHA1, CRAM-RIPEMD, DIGEST-MD5

● STARTTLS, TLS

● Static and Dynamic Routing. (SMTPROUTES environment variable)

● Environment variable control via envrules (rules file controlled byenvironment variable RCPTRULES) * QMAILREMOTE environment variable to run any executable/script instead of qmail-remote

● QMTP support, artificial routing using (control/qmtproutes)

● ONSUCCESS_REMOTE, ONFAILURE_REMOTE scripts run on successful or failed remote deliveries environment variables SMTPTEXT, SMTPCODE, ERRTEXT available for these scripts

● IP address binding on domain, sender address, recipient address and random selection from a pool of IP addresses

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Local delivery

● user-controlled address hierarchy : fred controls fred-anything

● mbox delivery

● reliable NFS delivery (maildir)

● user-controlled program delivery: procmail etc. (qmail-command)

● optional new-mail notification (qbiff)

● detailed Delivered-To Headers

● optional NRUDT return receipts (qreceipt)

● autoresponder RFC 3834 compliance (provide Auto-Submitted,In-Reply-To, References fields (RFC 3834))

● conditional filtering (condredirect, bouncesaying, vfilter)

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Local delivery (cont…)

● Environment variable control via envrules (rules file controlled by environment variable RCPTRULES)

● Eliminate duplicate messages

● QMAILLOCAL environment variable to run any executable/script instead of qmail-local

● X-Forwarded-To, X-Forwarded-For headers

– Other● Unix Client Server Program Interface (UCSPI) through

programs tcpserver and tcpclient

● Change concurrency of tcpserver without restart

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Other

● TLS/SSL Support in tcpserver

● STARTTLS extension in IMAP, STLS extension in POP3

● Ability to restrict connection per IP (MAXPERIP)

● run shutdown script if present on svc –d

● ability to log svscan output using multilog

● nssd Name Service Switch which allows extending of the system passwd database to IndiMail's database.

● pam-multi - Generic PAM module allows any external programs to authenticate against IndiMails database.

● multiple checkpassword modules sys-checkpwd, ldap-checkpwd, pam-checkpwd, vchkpass, systpass

● Proxy for IMAP/POP3 Protocol

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Other

● inlookup – High Performance User Lookup Daemon.

● indisrvr – Indimail Administration Daemon.● spawn-filter - Ability to add disclaimer, run multiple filters before

local/remote delivery.

● Post Execution Handle - Allows functionality of indimail to be extended by writing simple scripts

● On the fly migration of users by defining MIGRATEUSER environment variable.

● ready to use QMQP service

● ability to distribute QMQP traffic across multiple servers

● sslerator - TLS/SSL protocol wrapper for non-tls aware applications

IndiMailThe Flexible Messaging Platform

Feature List (cont…)– Other

● svctool – Configuration tool for IndiMail.

● adminclient protocol – Interface for external programs to administer IndiMail

● mrtg graphs for detailed statistics

● ability to specify commands in control files

IndiMail Architecture

Online Replication

Mailstoreimap/pop3/smtp

Inlookupvdelivermail

WebmailProxy

imap/pop3inlookup

WebmailProxy

imap/pop3inlookup

WebmailProxy

imap/pop3inlookup

IncomingRelayInlookupClamd

bogofilter

IncomingRelayInlookupClamd

bogofilter

Internetmx.xxxx.com (25)

imap.xxxx.com (110, 143)mail.xxxx.com (80)

Load Balancer 1

IncomingRelayInlookupClamd

bogofilter

Load Balancer 2

OutgoingRelayInlookup

clamd

OutgoingRelayInlookup

clamd Mailstoreimap/pop3/smtp

Inlookupvdelivermail

Mailstoreimap/pop3/smtp

Inlookupvdelivermail

InternetSMTPPort 25

SMTP

SMTP(for local domains)

MySQLHost Control

DB(Master)

MySQLHost Control

DB(Slave)

MySQL MySQLMySQL

SMTP(for authenticated

userto submit theiroutgoing email)

Internet SMTP

qmail-remote(external domain)

IMAP/POP3

MySQL Connection

MySQL ConnectionMySQL

Connection

smtp.xxxx.com (25/587)

imap.xxxx.com (110/143)mail.xxxx.com (80)

mx.xxxx.com (25)

MySQL Connection

SMTP(delivery from

mailstore)

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

Installation

– RPM for both 32 / 64 Bit

– rpm -ivh rpm_filename

– openSUSE

• openSUSE 11.3• openSUSE 11.2• openSUSE 11.1• SUSE Linux Enterprise 11 SP1• SUSE Linux Enterprise 11• SUSE Linux Enterprise 10• OpenSUSE Factory

– Red Hat

• Fedora14• Fedora13• Fedora12• RHEL6• RHEL5• Centos 5

– Mandriva Linux

• Mandriva 2010.1• Mandriva 2010

– Yum Repository

– http://download.opensuse.org/repositories/home:/indimail/

– Copy repo file to /etc/yum.repos.d for hands-free installation/upgrade

● % sudo yum install indimail.x86_64● % sudo yum install indimail.i386

●

IndiMailThe Flexible Messaging PlatformConfiguration

– svctool (service Tool)• SMTP, IMAP, POP3, ssl, fetchmail, certificates, verification of

installation

• Backups of data, configuration

• Repair of Database, queue

• Configuration for MySQL, Spamfilter, virus filter, qmail

• Reports

– GUI• Iwebadmin, indium, shit, cindimai, osh

– Supervise for 100% uptime• Envdir

• Logging

● Administration● Web Administration – iwebadmin

● ncurses GUI – Secure Host Interface

– Configurable menu & commands● Indium – TCL/TK GUI

– Powerful & provides in-depth administration● Operator Shell – osh

– Bash like shell with restricted access to commands

– Access to commands based on user roles● cindimail – Small shell providing access to all indimail commands

with tool tips

IndiMailThe Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

Maintenance– Backup of Configuration

● /var/indimail/control● /var/indimail/etc● /var/indimail/users● /var/indimail/share

– Mail Backup

– Update of Clamav Software

– Update of IndiMail (rpm, yum)

IndiMailThe Flexible Messaging Platform

Long Term Goals●To provide a modern technologically superior technology and solutions specifically related to Messaging Protocols (SMTP, IMAP, POP3)

●Provide a highly scalable, bug free, MSP class mailing solution using the Bazaar Model under GNU GPL V3

●Provide configuration for any property that can potentially affect the behaviour of messaging (Install the software once).

●FHS 2.3 compliance

●Build a community and get shipped with Linux distros by default

IndiMailThe Flexible Messaging Platform

Support● indimail-support - You can subscribe for Support at

● https://lists.sourceforge.net/lists/listinfo/indimail-support. You can email indimail-support@lists.sourceforge.net for posting messages to this list.

● indimail-devel - You can subscribe at https://lists.sourceforge.net/lists/listinfo/indimail-devel

● indimail-announce - This is only meant for announcement of New Releases or patches. You can subscribe at http://groups.google.com/group/indimail. You can email indimail-devel@lists.sourceforge.net for posting messages to this list.

● Archive at Google - http://groups.google.com/group/indimail. This group acts as a remote archive. Any discussions posted here goes to indimail-support.

● There is also a Project Tracker for IndiMail (Bugs, Feature Requests, Patches, Support Requests) at http://sourceforge.net/tracker/?group_id=230686

IndiMailThe Flexible Messaging Platform

Documentation– INSTALL, INSTALL-RPM

– IndiMail Documentation – indimail.pdf

– WIKI - http://en.wikipedia.org/wiki/User:Mbhangui/IndiMail

– Frequently Answered Questions – FAQ.pdf

– Man Pages - /var/indimail/man

– All docs in /var/indimail/docs

IndiMailThe Flexible Messaging Platform

Hardware Requirements (100,000 users)– Messaging Server

• DL 360 (Relay Server) – 2 Nos

• DL 380 (Message Store) – 1 Nos

• Storage (MSA 1000/MSA 2000)

- Groupware & Chat• DL380 (Egroupware & Jabber) – 1 Nos

- Load Balancer – 1 Nos

IndiMailThe Flexible Messaging Platform

Timelines– Messaging Server – 1 day installation + 4 weeks

training

– Groupware & chat - 6 weeks installation & configuration + 2 weeks training