Quality of Experience

Performance Benchmark on Messaging solutions Microsoft Office 365, IBM LotusLive and Google Apps

Webinar October 6 th 2011

Alexis Blandin, Cloud & Social Business Leader | Synergie Informatique

Arnaud Bécart, International Presales Engineer| ip-label

David Wais, Business Development Leader| ip-label

Synergie Informatique

Cloud Services Provider, France

Social Business Specialist

Expert in messaging and collaboration solutions integration

IBM, Microsoft and Google partner

Around the world, ip-label.newtest solutions measure the quality of applications from the user’s point of view: business applications as well as web, voice and even TV over IP platforms. Over 1500 direct and indirect customers rely on ip-label.newtest to validate the end-to-end reliability of their websites and critical applications worldwide

ip-label.newtest

Cloud goes Mainstream

Innovation = Public versus Enterprise Social Network Mobility, highly connected, convergence Increasing Data SaaS, a serious consideration

“ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research “10% of Enterprise Email Will Be Based on a Cloud/SaaS Model in 2014” – Gartner

Messaging in the line of sight

Cloud Usage Scenarios

1. Prototyping

2. Test & Development

3. Messaging

4. Collaboration

5. Web Applications

6. Backup, Archiving

CIO Expected Advantages Commodities delegation Industrialisation Fast deployment Savings, ROI Self-service, flexibility / PAYG Innovation (« beta »)

Standardization, integration New challenges (mobility, …)

The CIO Evolution

Guarantees, disponibility (SLA)

Warranty and « Quality » improvement

Security, externalisation, reversibility

Sales relationships (B2B)

Contracting and legal management, (Patriot Act, CNIL)

Going from CAPEX to OPEX (subscription)

Operational and organisational restrains

Adapted Needs

Governance, « Cloud » Best Practices

Adapted procedures flowchart

More straight to the point (support, specifics, infrastructures)

Dependant to the service/ network

User experience

Metrology Tools

Editor’s Focus / Collaborative Suite

Identity, Strategy

1911 Services / Softwares

Social Business

1975 Windows / Office

« All In »

1998 Search

Pure Player

Product LotusLive Office 365 Google Apps For Business

Acquisition By modules

Yes Progressive (options) No

€/U/Y (public prices)

34 to 113 € 43 to 306 € 40 €

Advantages • Social and Collaborative

Network • External Opening

• Pack Office Integrated • Sharepoint Framework

• Web Experience • Google ecosystem

Weaknesses • Some missing features

(online office suite, blogs/wikis, call audio/video)

• SharePoint Out-of-the-Box (ergonomy, features)

• Document-Centric

• No web conference • Very basic integrated social and

collaborative tools

E-Mail Systems Focus

Product • LotusLive iNotes/Notes • Exchange Online • GMail

Features

• Classic codes

• Native Lotus Notes included

• Classic codes • Native Outlook integrated

• Codes different than a classic messaging suite

• GMail offline (Google Chrome)

Technical • Two offers: basic(web mail) +

advanced (Domino) • Hybrid Mode

• Hybride Mode • Full Web

Analysis, trend

• Lack of visibility, can look strong enough just for Domino clients

• Domino/LotusLive future vision available

• Leader On-Premise • Late Coming-Out Cloud

• Difficult move from BPOS to Office 365 : which

consequeces ?

• In position of conquest • Big efforts made in some

points (sécurity / administration)

Space 25 Go 25 Go 25 Go

SLA, Credit according to the available

%

99,9 % 2 à 10% monthly cost

99,9 % 25 à 100% monthly cost

99,9 % 3 to 15 days

“ Inbox = Tetris getting zero lines doesn’t win it just makes the next move easy “

Methodology Data Solution Pilot for all web applications , rich web or client (

Office 365, Outlook…) run like a user: mouse click, filling application forms, text control, click on image etc.

Fast launch on one of the 70 measuring points in the world (5 operators are tested in France, Orange, Neuf-SFR, Iliad-Free, Colt, Cogent)

Mode « Saas » on demand, from set up tests to operational robot and servers maintenance

Access to all measures via an extranet, reports sent via mail and warnings on incidents

Access to back up for all set up , expertise and training requests

Scenari recording tools for clients wanted to be autonomous

Solution Architecture

Tests Description 3 platforms tested

3 types of tests

– Access to the portal from 5 spots in France • https://portal.microsoftonline.com • https://apps.lotuslive.com • https://www.google.com/a/synergie-informatique.fr/ • Tests origins: France Telecom, Neuf-SFR, Iliad-Free, Colt, Cogent • Tests frequency: 15 minutes • Tests duration: 21/07/11 au 30/09/11 (71 jours)

– Scenario SENDING from 1 spot in France • Sending a mail from the Cloud platform to a external messging address • Delivery mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)

– Scenario RECEIVING from 1 spot inFrance • Sending a mail from an external messaging address rto the Cloud platform • Receiving mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)

Tests Description Scenario SENDING

– Connexion to the portal

– Authentication

– Access to the e-mail

– New online message to mailXX@ip-label.net

– Send the message and wait for POP from ip-label (mail.ip-label.com) – waiting time max 60 seconds

Scénario RECEIVING – Connexion to the portal

– Authentication

– Access to the e-mail

– Send a mail via SMTP from a server from ip-label (mail.ip-label.com)

– Wait for mail receipt on the tested platform – waiting time max 60 seconds

Each sent mail is automatically deleted after each test

Each portal account uses a distinctive external messaging account

Tests Description When the portal we are trying to access is considered unavailable?

– Non loading of the portal in less than 60 seconds

– Network incidents detected ( DNS faulty resolution , failed connexion) and server(loading slowness)

When a scenario is considered unavailable ?

– Non complete loading of a page in less than 30 seconds linked to network or server incidents

– Keywords not found

• Username to validate the loading of the portal

• Username/Password, connexion button, then button ‘new mail to validate the authentication step

• SEND : button new mail and send

• RECEIVING : refresh button to verify the mail had been received

– Non transmission of the e-mail ( external SMTP online messaging or online messaging SMTP) in less than 60 seconds

Portal Access Availibility and average access time to each portal

Results from 21/07/11 to 30/09/11 – Around 34000 tests per portal

– Google has the best portal loading time

– Lotus Live is quite fast (1,5 seconds) but the page is longer to start (600ms), with longer DNS resolution, connexion and server time

– Office 365 is the longest to load it is also the heaviest (1 Mo, versus 17 ko for google and 32 ko for Lotus Live)

– Lotus Live shows the best availibility with 7 failures, Google (8 échecs) and Office 365 (11 failures)

1,097 seconds 99.977 %

3,552 seconds 99.968 %

1,541 seconds 99.980 %

0 500 1000 1500 2000 2500 3000 3500 4000

Office 365

lotus live

google appsRésolution DNS

Temps de connexion

Temps de réponse du serveur

Chargement du contenu

Portal Access Evolution over a 3 months period

3474 3750

3432

1286 1357

1982

1119 1113 1061

0

500

1000

1500

2000

2500

3000

3500

4000

Juillet Aout Septembre

Office 365

lotus live

google apps

99

,98

99

,83

10

0

99

,98

99

,98

99

,98

99

,98

99

,96

99

,99

99,7

99,75

99,8

99,85

99,9

99,95

100

Juillet Aout Septembre

Access to portal performance Access to portal availibility

Conclusion

- For the LotusLive portal , the answering time network and server has increased since 31/08/11 ( for an identical content)

- Main errors encountered

- Google : 5 DNS resolution errors and 3 time outs

Lotus Live : 4 interrupted connections and 2 unreachable servers

Office 365 : 4 DNS resolution errors, 5 unreachable servers and 2 timeouts

- A maintenance intervention on the Office 365 portal on 9/9/11 from 5am to 8;15am was not taken into account

Authentication Authentication availibility and online messaging access

Results from 01/09/11 to 30/09/11 –About 2800 tests per portal

– Google is the strongest plaftform in terms of authentication 99.86 % in 24/24

– Office 365 99.76% in 24/24 on authentication (mainly linked to some timeouts and DNS errors); those incidents usually occur at night and from 8h to 23h the availibility is the same as Google

– LotusLive has the most errors at the authentication (99.17%); even if those errors occur mainly at night time, (99.56% availibility between 8h and 23h), one incident last September 18th between 8h and 10h penalise their position on this scale

– Google is the fastest platform to authentify yourself and access to messaging : 11,5 seconds, LotusLive 15 seconds ( in one page load in opposition to the others)

– Office 365 is the slowest to reach online Outlook : 19 seconds

98,52

99,56

99,17

99,81

99,89

99,86

99,54

99,89

99,76

97,50 98,00 98,50 99,00 99,50 100,00

DisponibilitéHNO

23h-8h

DisponibilitéHeures Ouvrées

8h-23h

Disponibilité24h/24

Office 365

google apps

lotus Live

19054

15238

11438

0 5000 10000 15000 20000

Performance Authentication and online messaging access

Authentication Authentication Incident for Lotus Live on 18/09

l 18

Timeout : incomplete page after login (88k out of 212ko expected)

Page error after login

Authentication Authentification errors examples

Authentification timeout on Office 365

Authentification timeout on Google Apps

Authentication Examples : Authentication errors on Google Apps portal

Erreur du serveur (HTTP 500) sur l’objet Dashboard qui devrait rediriger sur le portail

Impact utilisateur : le champ login absent de la page et l’utilisateur ne peut pas s’authentifier

Erreur

Cas normal

Sending/Receiving mail Performance

Average length of time to send and receive e-mails

– On average, to send a mail from a Cloud platform is faster than receiving a mail from an external platform

– Google is the platform which transmits mails the fastest, as well as sending to an external account (12,2 seconds on average) and receiving from an external account(7.5 seconds)

– Office 365 is faster than Lotus Live for receiving mail, but slower than LotusLive to send mails

15095

18895

10049

23213

7558

12271

0 5000 10000 15000 20000 25000

Temps d'envoi

Temps de réception

google apps

lotus live

Office 365

Sending/Receiving mail Stability

Average availibility for sending and receiving mails

– Lotus Live is the most available platform in terms of sending mails, if we consider as unavailable a length of time superior to 60 seconds ,to send a mail CloudExterne or ExterneCloud

– Office 365 (~75%) and Google Apps (~85%) : This means even with a fast sending and receiving mail process (<15 secondes), engorgements regularly slow down the e-mail sending or receiving process

– Sending and receiving mail process is more unstable for Google and Office 365

74,81

71,66

97,62

99,18

85,17

86,17

0 10 20 30 40 50 60 70 80 90 100

Disponibilité d'envoi (<60s)

Disponibilité de réception (<60s)

google apps

lotus live

Office 365

Google Apps Gmail can be considered as a reference in terms of online messaging access speed and you are guaranteed fast sending and receiving mail

IBM Lotus Live is a stable platform in terms of sending and receiving mails but can be a disappointment for users because of access unavailibility and average access time (15 sec)

Microsoft Office 365 has a good messaging availibility (99.89% during working hours) but some progress need to be made on page loading time ( particularly on authentication and access to Outlook) and on sending stability

1 3 2 Google Apps Lotus Live

Office 365

Portal performance 3 2 1

Portal availibility 2 3 1

Authentication availability and online messaging access 3 2 1

Authentifcation availability (working hours) 3 1 3

Receiving Time 3 1 2

Sending Time 3 2 1

Receiving Stability 2 3 1

Sending Stability 2 3 1

Total 21 17 11

Scores

Conclusions Results Details

Gmail Google

– Best Portal Access Performance

– Best authentication performance and availibity

– Best sending time to an external e-mail address and receiving from an external e-mail address

– Relative unstability in transfert time frames

• 15 mails out of 100 are sent in more than 60 seconds

• 14 mails out of 100 are received in more than 60 seconds

IBM Lotus Live

– Look out on the highest portal loading time since the beginning of September

– Best portal access availibility

– Best transfert stability

1 mails out of 100 are sent in more than 60 seconds

3 mails out of 100 are received in more than 60 seconds

– Authentication performance and availibility is not good

Microsoft Office 365

– Very good authentication disponibility during working hours.

– Portal access performance and access to messaging to be improved (19 seconds)

– Stabilité des durées de transmission des mails à améliorer

• 25 mails out of 100 are sent in more than 60 seconds

• 28 mails out of 100 are sent in more than 60 seconds

Conclusions Each editor commits himself to 99.9% availibility

– « Microsoft provides a financially-backed 99.9% uptime guarantee »

• http://www.microsoft.com/fr-fr/office365/what-is-office365.aspx

– « LotusLive Notes delivers the capabilities your business users need to increase efficiency while reducing IT complexity and associated bottom line costs - all backed by a dedicated IBM team with 24x7 support and a 99.9% up-time service level objective »

• https://www.lotuslive.com/styles/tours/transcripts/LLNotes_datasheet.pdf

– « Google Apps includes a 99.9% uptime SLA and synchronous replication »

• http://www.google.com/apps/intl/en/business/messaging.html

As a reminder data for September 2011…

According to you, what is the quality really

perceived by the online messaging users ?

Q/A

Were mails send with attachments ?

– No , only text content; the add on of attachments of different sizes could be a criteria to do more tests

Which internet navigator was used for the tests?

– Internet Explorer 8

Can those tests be run on a client (Lotus Notes, outlook) ?

Yesthe robots from ip-label can simulate any user running windows

Can this benchmark be testing with files /document sharing?

– Sharing files tools can also be tested by the robots ip-label

What about other navigators , especially if Microsoft Exchange is losing out on fonctionnality ?

– The robots can test Explorer or Firefox; the multi navigators cans be given through other solutions like Cloud Observer for instance

Can we audit audio / video flow?

– Ip-label can measure those flows using different measuring solutions on a user standpoint (streaming, voice quality analysis, video tests /audio conference, etc.)

Thank You

Synergie-informatique.fr

@SynergieInfo