Three cool innovations from ZarafaSteve Hardy

• Renamed ‘twidget’• Will parse email body to find

twitter references and show it in a widget

48h feature run update: twitter

• An existing PDF previewer library seems quite good• PDF previewer can work like JPG/PNG viewer• No screenshots yet

48h feature run: PDF preview

• Was already available for WebAccess• Send entire email to sugarcrm• Reuse of some WebAccess code • Looks promising

48h feature run: SugarCRM archive

• Delegates most interesting• Python script

48h feature run: commandline permissions

setpermissions.py [OPTIONS] mailbox

Options: Permission folders:

--calendar [none|readonly|secretary|owner] Set the permissions for the Calendar folder. Default <none>--tasks [none|readonly|secretary|owner] Set the permissions for the Tasks folder. Default <none>--inbox [none|readonly|secretary|owner] Set the permissions for the Inbox folder. Default <none>--contacts [none|readonly|secretary|owner] Set the permissions for the Contacts folder. Default <none>--notes [none|readonly|secretary|owner] Set the permissions for the Notes folder. Default <none>--journal [none|readonly|secretary|owner] Set the permissions for the Journal folder. Default <none>--store [none|readonly|secretary|owner] Set the permissions for the Store folder. Default <readonly>

Delegate options:--seeprivate Delegator can see private items

--sendcopy Delegator receives copies of the meeting-related messages sent to mailbox owner.--send-only-to-delegators Send meeting requests and response only to the delegator, not to the mailbox owner.

Users or group:--users usernames User which get the permissions for the given

mailbox.--groups groupnames Group which get the permissions for the given

mailbox.

Global options: [-h|--host path] -h path Connect through <path>, e.g. file:///var/run/socket -V Print version info. --help Show this help text.

• It’s done!

• Will be in 7.1

• Any byte size in any applications can be specified with

– “cache_cell_size = 16M” instead of

– “cache_cell_size = 16777216”

48h feature run: Sizes in GB, MB, KB

• Pro’s– ICS is the same (synchronization mechanism)

– Generic synchronization should work

• Con’s– Address conversion needed in many places

– ID conversion needed in some cases

48h feature run: Exchange / Zarafa replication

• BONUS FEATURE

• Apparently someone made some kind of dropbox integration

• Actual details are a surprise

48h feature run: dropbox

zarafa-search

• Performance of zaraf-indexer could be improved

• Clucene storage removed

• Switched to kyotocabinet-based storage system

• Renamed from zarafa-indexer to zarafa-search

• Included in Zarafa 7.1

LuceneText analysis

LuceneText indexer

Lucene Database file

• One-size-does-not-fit-all– Clucene optimized for web-like searches

– Top-N results shown, others ignored

• MAPI requires– All results

– Often many results

• I/O access patterns not optimized for MAPI-like searches

Why not CLucene ?

• Generic database– Kyoto cabinet

– Key/value database

– High performance

– Compression

– Low overhead

– Crash-safe

• Lucene still used for text analysis

I/O solution: re-engineer it yourself

LuceneText analysis

Zarafa-search text indexer

Kyoto cabinet database file

• Search is local to server• Multiple servers using single indexer no longer possible• Document ID’s are local• No more mapping needed between document ID and database ID

– Saves IOPS during search

Improved search protocol

Zarafa-server

Zarafa-search

node1

Zarafa-server

Zarafa-search

node2

IOPS Old New

Search with 10 hits 11 1

Search with 100 hits 101 5

Search with 10000 hits 10001 5

IOPS comparison

Indexing many users

Old method

-Scan all users-Scan all folders-Update index

New method

-Wait for signal from zarafa-server-Index only items received

Latency: > 30 minutes Latency: < 10 seconds

• Old indexes are not compatible with new indexes• New indexes must be generated• Just start zarafa-search and let it index your data

Upgrade path

• Zarafa-server now optimized for streaming output– Uses MySQL stored procedures

– Parallel processing between MySQL and zarafa-server

– Increases message throughput

– > 150 messages (emails) per second

• New protocol feature: streaming de-stub– For indexing archived messages

– Streams data from archive server just like ‘normal’ indexing

– Only possible in zarafa-server 7.1 (will fail if your archive server is running 7.0)

• Compressed indexes– Indexes are now about ½ size of previous Lucene indexes

Other indexer optimizations

• Uses libpff (open source library)• Able to read .PST files from linux commandline• Currently imports entire content into MAPI store• Supports unicode and non-unicode PSTs (97-2002 PSTs vs 2003

PSTs)

Things it does not do:- Email address conversion- Folder mapping

Future nice-to-have:- Import directly into archive

Zarafa-import