Embed Size (px)
Citation preview
1
CMS Review Report
June 2012
A presentation to The Department of Finance and Deregulation (AGIMO)
Version 2.0
2
1. Project Overview
2. Methodology
3. Research Results
4. Discussion / Review / Adjustments?
5. Further Considerations
6. Moving Forward
7. Appendix
Contents
3
Project Overview
4
Project Overview and Goals The Department of Finance and Deregulation (Finance) has decided to replace their Web Content Management System (WCMS) due to concerns around extendibility, usability, difficulties in content editing across all sections of the site and the cost of support.
The objective was to review and select an appropriate Open Source WCMS that meets Finance’s needs. An evaluation methodology was applied to the three shortlisted CMS's to present a ranking.
This report will present an outline of the processes, Finance's specific requirement priorities, research results, scoring and future considerations.
5
Methodology
6
CMS Review Framework The framework based on the QSOS (open source) method was used. It provides an iterative approach with varying levels of granularity. Key processes include:
• Definition – Requirements
• Evaluation – Scoring and Weighting Criteria
• Qualification – Research and Evaluation
• Selection – 3 CMS’s for Evaluation
7
Requirements Definition
Your requirements criteria
Requirements workshop
CMS Evaluation Criteria (by category)
Categorised Requirements
(functional &
non-functional)
Requirement definition is a key part of understanding your business needs and how these translate into CMS requirements.
Normalisation
8
Web Content Management
- Content Authoring & Collaboration
- Content Presentation
- Navigation - Publishing - Workflow &
Lifecycle Management
User Management - Admin & Content
Editor Management
- Public User Management
Search
Modules
e-Commerce Support
Digital Asset Management
Analytics &
Reporting
Functional Requirement Categories
Content Personalisation
- Content Personalisation
- Content Targeting
Requirements are grouped into the following functional categories and sub-categories.
9
Company & Product - Company - Product /
Developer Community
Performance - Performance - Availability - Scalability
Interoperability
CMS Usability
Integration
Management
System Requirements
Security
Non-Functional Requirement Categories
These are the non-functional categories and sub-categories.
10
Each requirement is evaluated as a priority and these priorities are weighted.
Description Priority Weighting
Mandatory 3
Desirable 2
Optional 1
Not required 0
Evaluation Approach – Requirement Prioritisation
11
Score Description
3 The CMS meets the requirement out of the box.
2 The CMS requires slight customisation and/or configuration to meet the requirement. (e.g. integration of an existing plug-in or module).
1 The CMS requires considerable customisation and/or configuration to meet the requirement. (e.g. development effort required / no plug-in available).
0 The CMS does not meet the requirement.
For each CMS to be reviewed, every requirement is evaluated and scored (out of 3):
Evaluation Approach – Requirement Scoring
12
Research and scoring is based on:
Scoring
Eminent reports (e.g. Gartner, Forrester) / CMS documentation / desktop research / product support / sizing / history and roadmap
Subject Matter Experts’ input and review / Previous CMS selection project results
Speaking to the vendors
Requirements were scored twice independently then crosschecked to obtain an agreed score.
13
Raw Score
(out of 3)
Priority
(out of 3)
Weighted Score 1
(out of 9)
The scores are then multiplied by the priority that was agreed upon during the requirements gathering stage.
This provides us with the Weighted Score.
Evaluation Approach – Requirement Scoring
14
The Weighted Scores are added together to find the Category Score, weighted against its scaled value.
Weighted Score 1
(out of 9)
Weighted Score
2, 3, 4, 5… (out of 9)
Weighted Category
Score (each category has a
scaled value)
Scoring
15
Weighted Category
Score
Weighted Category
Score
Weighted Category
Score
Total Score for CMS 1 (out of 100)
Functional Requirements
The Weighted Category Scores are added together to calculate the Final Score for each CMS.
Scoring
Weighted Category
Score
Weighted Category
Score
Weighted Category
Score
Weighted Community
Score
Non-Functional Requirements
16
Research Results – Requirements
17
Normalised Requirements
CMS Review
Methodology
Finance Draft Business Requirements
(DBR’s) &
Requirement Workshops
We normalised the Functional and Non-Functional Requirements from various inputs, resulting in 401 requirements to be evaluated.
401 Requirements
103 Non-Functional
32 Community/Company
+ 13 Exclusions
266 Functional
18
Research Results – Scoring and Weightings
19
Functional Requirements (50%)
Category Breakdown %
Web Content Management 30%
User Management 15%
Search 15%
Content Personalisation 10%
Analytics and Reporting 10%
Modules 10%
Digital Asset Management 5%
E-Commerce 5%
Total 100%
Non-Functional Requirements (30%)
Category Breakdown %
Performance 15%
Performance: Availability 15%
Performance: Scalability 15%
CMS Usability 15%
Integration 10%
Management 10%
Security 10%
System Requirements 5%
Interoperability 5%
Total 100%
Company/Product/Community (20%)*
The requirements were categorised and the importance of each requirement category was evaluated and expressed as a value percentage.
*Community/Product/Company was considered a non-functional requirement category but we agreed to give it a separate scaling system due to its importance to Finance’s needs.
Requirements Categories and Rankings
20
Research Results – Shortlisted CMS’s
Project management plan, Project schedule, Project budget tracking, issues and risk register, status report
21
Shortlisting We began with a screening process and ended with a detailed scoring of requirements.
Level of detail in research increases
Number of CMS possibilities reduces
Final scoring
High-level research and shortlisting
Development of long list
22
This resulted in a ‘long list’ of 18 CMS:
Shortlisting Two key requirements helped to quickly narrow down the field further:
1. The WCMS must be truly Open Source.
2. The WCMS must not be a .NET or Ruby based solution.
Alfresco Daisy DotClear Drupal Expression Engine
eZ Publish Liferay Joomla! Magnolia Mambo
OpenCMS Hippo CMS PHP-Fusion PHP-Nuke Pixie
RavenNuke CMS Squiz WordPress
23
• A high-level view of our requirements criteria.
• The CMS’s presence and reputation in the market.
• Eminent reports such as the Gartner Magic Quadrant for Web Content Management and Forrester WCMS material.
• Previous WCMS selection project results and Subject Matter Experts’ input.
Shortlisting In order to narrow down the long list, each CMS was ranked based on the below criteria:
24
91 91 89
87 86 85
83
80
77
67 65
60
70
80
90
100
High-Level Scoring Results Points
CMS’s
Shortlisting High-level research results:
25
These were the final three Open Source CMS’s that were shortlisted for evaluation.
Shortlisting
26
The Results
27
Results
75%
73.5%
70%
The CMS’s ranked as follows:
Magnolia
Liferay
Drupal
28
Results (without portal requirements)
Liferay dropped from 73.5 to 72 points.
Magnolia
Liferay
Drupal * All following scores include portal requirements
75%
72%
70%
29
Results
75 73.5 70
33 33.5 31
28 26
23
14 14 16
0
10
20
30
40
50
60
70
80
Magnolia Liferay Drupal
Total
Functional
Non-Functional
Community
CMS’s
Points
Weighted score overview.
30
Functional Requirements - Breakdown
CMS
Web Content
Mgt
Content Personal-
isation User Mgt e-Commerce
Support Search Digital
Asset Mgt Analytics & Reporting Modules Total
Magnolia 24.9 3.3 12.9 2.1 9.9 3.7 3.5 5.8 66.1
Liferay 22.8 3.3 15.0 2.9 9.1 3.2 3.9 6.6 66.8
Drupal 19.7 5.8 10.4 2.9 8.0 3.2 5.6 6.7 62.3
Max Score 30 10 15 5 15 5 10 10 100
0
10
20
30
40
50
60
70
CMS A CMS B CMS C
Web Content Management
Content Personalisation
User Management
e-Commerce Support
Search
Digital Asset Management
Analytics & Reporting
Modules
Magnolia Liferay Drupal
31
Non-Functional Requirements Breakdown
CMS System
Requirements Security Performance Integration Manage-
ment CMS Usability Interoper-
ability Total
Magnolia 5.0 8.3 43.1 8.7 9.3 13.7 4.2 92.3
Liferay 4.8 8.9 40.4 7.4 9.4 12.2 4.1 87.2
Drupal 4.3 7.9 38.5 6.7 7.9 9.7 3.8 78.8
Max Score
5 10 45 10 10 15 5 100
0
20
40
60
80
100
CMS A CMS B CMS C
System Requirements
Security
Performance
Inegration
Management
CMS Usability
Interoperability
Magnolia Liferay Drupal
32
Company & Product Requirements Breakdown
CMS
Company & Product / Developer
Community
Magnolia 172
Liferay 171
Drupal 188
Max Score
240 160
165
170
175
180
185
190
195
200
Magnolia Liferay Drupal
33
The following are some additional breakdowns that were requested by Finance.
Requirements for metadata appeared 38 times, representing 9.5% of the unweighted evaluation criteria.
Additional Breakdowns – Metadata
CMS Metadata
Requirements
Magnolia 212
Liferay 205
Drupal 179
Max Score
285
0
50
100
150
200
250
Metadata Requirements
Magnolia
Liferay
Drupal
34
Requirements for forms appeared 3 times, representing 0.7% of the unweighted evaluation criteria.
Additional Breakdowns – Forms
Raw Score Weighted Score
Req. ID Category Requirement Priority
Mag
no
lia
Life
ray
Dru
pal
Mag
no
lia
Life
ray
Dru
pal
FW.FW.052 Web Content Management
The solution must provide the ability to manage user feedback generated through the submission of forms.
Optional 2 2 2 2 2 2
FW.CA.035 Web Content Management
The solution must provide the ability to create and modify forms within the CMS, and the subsequent validation, capture and management of this form data (e.g. sending emails on form submission, exporting to / from a database in various formats).
Desirable 2 2 2 4 4 4
NF.MA.011 Management The solution must provide the ability to manage user-generated content.
Desirable 2 3 2 4 6 4
Total Score (out of 15) - Forms 10 12 10
35
Requirements for multiple sites appeared 3 times, representing 0.7% of the unweighted evaluation criteria.
Additional Breakdowns – Multiple Sites
Raw Score Weighted Score
Req. ID Category Requirement Priority
Mag
no
lia
Life
ray
Dru
pal
Mag
no
lia
Life
ray
Dru
pal
FW.PB.003 Web Content Management
The solution must republish dependant and associated content automatically to maintain synchronisation, automatically updating appropriate content used by multiple pages or sites so that it remains consistent.
Mandatory 3 3 3 9 9 9
FW.CP.006 Web Content Management
The solution must provide the ability for content (including pages and single page objects) to be authored once, but published in multiple locations on the site, and on other websites.
Mandatory 2 2 2 6 6 6
FW.PB.002 Web Content Management
The solution must provide the ability to publish/depublish content items to one or many locations.
Mandatory 2 2 1 6 6 2
Total Score (out of 27) – Multiple Sites 21 21 17
36
Requirements for modules appeared 15 times, representing 3.7% of the unweighted evaluation criteria.
Additional Breakdowns – Modules
Raw Scores Weighted Scores
Req. ID Category Module Priority Magnolia Liferay Drupal Magnolia Liferay Drupal
FM.MO.001 Modules Blogs Optional 1 3 2 1 3 2
FM.MO.002 Modules Ratings / Tagging Optional 2 3 2 2 3 2
FM.MO.003 Modules Chat Desirable 1 2 2 2 4 4
FM.MO.004 Modules Classifieds / Advertising Desirable 1 1 2 1 2 4
FM.MO.005 Modules Forums Optional 2 3 2 2 1 2
FM.MO.006 Modules Document Management Optional 2 3 2 2 1 2
FM.MO.007 Modules Events Calendar Desirable 3 3 2 6 6 4
FM.MO.008 Modules FAQ Management Mandatory 3 1 2 9 3 6
FM.MO.009 Modules Graphing and Charting Tools Optional 1 2 2 1 2 2
FM.MO.011 Modules Error Reporting Mandatory 1 1 2 3 3 6
FM.MO.012 Modules Link Management Mandatory 1 1 2 3 3 6
FM.MO.013 Modules Newsletter Management Optional 3 3 2 3 3 2
FM.MO.014 Modules Polls Desirable 1 2 2 2 4 4
FM.MO.015 Modules Site Maps Mandatory 3 3 2 9 9 6
FM.MO.016 Modules Surveys Desirable 1 2 2 1 4 4
Total Score (out of 84) – Modules 47 51 56
37
Requirements for CMS extendibility appeared 6 times, representing 1.5% of the unweighted evaluation criteria.
Additional Breakdowns – CMS Extendibility
Raw Scores Weighted Scores
Req. ID Category Requirement Priority
Mag
no
lia
Life
ray
Dru
pal
Mag
no
lia
Life
ray
Dru
pal
NF.IG.003 Integration The solution must provide an API extensibility. Mandatory 3 2 2 9 6 6
NF.IG.004 Integration The solution must provide Web Services integration to support custom widgets.
Mandatory 2 3 2 6 9 6
NF.IG.011 Integration The solution must provide the ability to develop modules that may be incorporated in the CMS to modify functions of the CMS.
Mandatory 3 3 3 9 9 9
NF.IG.012 Integration The solution must provide the ability to develop modules that may be incorporated in the CMS to add functions to the CMS.
Mandatory 3 3 3 9 9 9
NF.IG.013 Integration The solution must ensure that the process to develop to modules is well supported and understandable.
Desirable 3 3 3 9 9 9
NF.IO.001 Interoperability The solution must provide an extensible, fully-documented Application Programming Interface (API) and architecture to connect and interact with external systems.
Mandatory 2 2 2 6 6 6
Total Score (out of 51) – CMS Extendibility 48 48 45
38
Scoring Overview Key findings:
• If implemented, all three shortlisted CMS’s would require customisation to meet Finance’s needs completely.
• Each CMS has good support for module extendibility (page 37) – i.e. the ability to develop or plugin modules for specific functionality. This is separate from the ‘Modules’ category (page 36) evaluated which covers specific functional requirements around blogging, chat, forums etc.
• While the final scores are close, some categories resulted in large ranges across the three CMS’s (e.g. User Management / CMS Usability).
• By excluding portal requirements, the results show no significant difference.
39
Further Considerations
40
Portal vs. CMS Considerations In our research we also looked at the influence of portals on the CMS.
• How much can we take portal requirements into consideration?
• Our findings show that the portal requirements did not affect the end ranking.
• The industry trend is to use one product to cover both.
• All of the shortlisted CMS’s are dynamic environments that render in real time.
41
Implementation Partners Cost of implementation is usually greater than license cost so the right implementation partner is crucial for project success:
• Providing a complete technical solution including necessary customisation.
• Non-technical build activity:
– User experience
– Creative design
– Project management
• Ability to provide extendibility and maintenance.
42
Is Open Source the only solution? There are some points to consider before committing to a CMS.
• Licensing vs development costs?
• Core functionality vs customisation (and upgrades)?
• Product support (SLA) advantages vs limitations?
• Community growth (not just relevant to Open Source)?
• The end-to-end user experience?
43
Moving Forward
44
Next Steps The following actions are expected following this presentation.
Finalising the CMS Selection
(Prototype / POC)
Selection of an implementation
partner
Rebuilding australia.gov.au
45
Appendix
46
Release version reviewed: Magnolia CMS 4.5 – Enterprise Edition (released Mar 2012)
Technology: Java Open Source solution. JSR-170 (and JSR-168).
Spring Framework and Struts integration, RSS feed support, standard templates, data caching and backup,
also supports: JSR-168, CMIS, JCR 2.0, HTML5 and Java 6.0
Company information: Founded in Basel, Switzerland, first release 2003.
Community channels: Magnolia Conference, Documentation, Forums, Forge, Mailing List, Blogs, Social
Networks and Training and Events (typically outside Australia).
Reference sites (3): US Navy, Travel.com.au, Digital UK
Australian SI’s:
Certified (2): Ventiv (VIC) – www.ventiv.com.au, HotHouse (NSW) – www.hothouse.com.au
Other (2): Hyro (NSW) – www.hyro.com, Permeance (VIC, WA) – www.permeance.com.au
Freelancers on LinkedIn: 2+
About Magnolia
47
Magnolia presents itself as a robust and reliable CMS, with strong potential moving forward. It outperformed in: Web Content Management, Search, Digital Asset Management, System Requirements, Performance, Integration, CMS Usability and Interoperability.
Although its community presence is more active than Liferay, it is still a way off that of Drupal.
Magnolia: Scoring overview
Strengths:
Strong usability
Enterprise support
Strong core based on open standards
Weaknesses:
Steep learning curve
Smallest SI presence in Australia
48
Release version reviewed: Liferay Portal 6.1 – Enterprise Edition (released Jan 2012)
Technology: Java Open Source solution based on JSR-286. Platform independent.
Also complies with JSR-168 and JSR-170.
Company information: Headquarters in LA, California, USA. Founded in 2004.
Community channels: Community, Documentation, Forums, Blogs, Wiki, User Groups, Marketplace,
Liferay Projects, Community Projects, Training and Events
Reference sites (3): Barcelona Turistic Bus, Terrassa City Hall, Allianz Insurance
Australian SI’s:
Certified (3): Permeance (VIC, WA) – www.permeance.com.au, Editure (VIC, SA) – www.editure.com.au,
Mexuz (NSW) – www.mexuz.com
Other (1): NSW Technology (NSW, VIC) – www.nsi.com.au
Freelancers on LinkedIn: 5+
About Liferay
49
Liferay’s WCMS is strong in its core capabilities but these aren’t as extensive as the other CMS’s. It is highly extensible however and has the advantage of being a highly regarded Portal. Its biggest downfall is that its current community support is the poorest of the CMS’s evaluated.
Liferay: Scoring overview
Strengths:
Solid core functionality
Robust – open standards
Potential for customisation of UX
Building momentum
Enterprise support
Weaknesses:
Learning curve
Limited community
50
Release version reviewed: Drupal 7 (7.14 released May 2012)
Technology: PHP solution which runs any platform that supports both a web server capable of running
PHP and a server. Requires PHP 5.2.5 or higher.
Company information: Founded in the Netherlands in 2001.
Community channels: Community, Documentation, Support, Marketplace, Forums, Chat, Mailing Lists,
Member Directory, Training and Events.
Reference sites (3): Australian Law Reform Commission, data.gov.uk, whitehouse.gov
Australian SI’s:
Official community: http://groups.drupal.org/australia
ACT based SI’s (2): Agileware (ACT) – http://agileware.net, OPC IT (ACT) – www.opc.com.au,
Other Australian SI’s: 10+ in Victoria, 3+ in New South Wales
Freelancers on LinkedIn: 200+
Drupal: Scoring overview
51
Drupal is the clear winner when it comes to community support. Arguably its strong modularity makes up for its lack of core functionality compared to Magnolia and Liferay. However this can make upgrades and support more difficult. Detailed user documentation is difficult to produce although this is recognised within the community and moves to improve this are underway.
Drupal: Scoring overview
Strengths:
Proven capability
Extensive community
Strong modularity (including a large selection of pre-developed modules)
Third party support
Weaknesses:
Dependency on modules
User experience
