Upload
jonathan-ralton
View
429
Download
0
Tags:
Embed Size (px)
Citation preview
SharePoint Saturday New York City 2015#SPSNYC
Fotopedia.com
An Independent Evaluation of Third-Party SharePoint Analytics Offerings
Jonathan RaltonAll trademarks and registered
trademarks are the property of their respective owners.
Jonathan RaltonBlueMetal
An Independent Evaluation of Third-Party SharePoint Analytics Offerings
AgendaSetting the StageClient StoryMeasuring and Trending
DefinitionsFrameworkExamples
SharePoint Analytics
Third-Party Tool CandidatesWhich would you choose?
EpilogueWhich did I choose?How does the story end?
Wrapping UpQuestions
Setting the StageAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
YOU
Audience
What roles are you in?
What SharePoint phase are you in?
What are you hoping to learn?
ME
Presenter
Jonathan Ralton• Senior Information Architect• SharePoint IT Pro since 2005
(WSS/SPS)• No coding!• Document Management,
Content Management, Knowledge Management…
blog.jonralton.net
Client StoryAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
Overview• Food and Beverage Manufacturing Company
• One of the top industry leaders in the country• One of the top industry leaders in the world
• Employees in dozens of countries• Hundreds of household brand names
• Some with $1b market shares
Intranet• Redesigned by BlueMetal and relaunched in late 2014• Heavy on video content• Social focus• Personalization
Challenges
Challenges
Communication Justification• Need to prepare for new
management oversight• Need to publish appropriate
news at appropriate times• Want to maximize investment in
the new facility of the intranet
Objectives
Content and Visitors• Learn what content is being
consumed and by whom• Learn which types of content
are most engaging• Gain better insight into what
kinds of content employees will respond well to, and in turn continue to return to myCorptown for
Employee Communications• Help to justify team’s value in
managing myCorptown• Demonstrate higher employee
satisfaction in being informed• Demonstrate higher employee
engagement
Objectives
AdoptionEngagement
Visitors/Visits/Constituencies
New Employees
New Users
Lapsed Users
Executive Employees
Mobile Users
Tablet Users
Laptop Users
Desktop Users
Kiosk Users
Current Tool
Measuring and TrendingMust Love Term Sets
WEB ANALYTICS
def·i·ni·tion [dèffə nísh'n]
Web Analytics
Web analytics is the measurement, collection, analysis, and reporting
of web data for purposes of understanding and optimizing web usage.
METRIC
def·i·ni·tion [dèffə nísh'n]
Metric
A metric is a something that can be measured based on quantifiable data,
or at the very least, estimated based on anecdotal data.
Metrics w/Tools
Web analytics tools mainly provide metrics for consumption via reports and APIs.
KEY PERFORMANCE INDICATOR(KPI)
def·i·ni·tion [dèffə nísh'n]
Key Performance Indicator
A key performance indicator (KPI) is a visual indicator that tells a story
about one or more measurements, in context.
Key Performance Indicators w/Tools
Web analytics tools may already provide some KPIs in dashboards.
Metrics and KPI Framework
Goals
Signals
Metrics KPIs
Goals
Adoption
Engagement
RetentionTask Success
Happiness
Sample Goals – Adoption
Ensure new employees and new users are
engaging with myCorptown content for the first time, including
the email newsletter
Build employee buy-in by telling
our story through our vehicles
Reach our employees ‘where they are’ including
via mobile and kiosks
‘Give them what they want’ by being able to respond to
popularity of content
Sample Goals – Engagement
Get our employees to give feedback on content
including likes and comments, submitting
questions, etc.
Have our employees be conversant in an informed manner
about company news with fellow colleagues
Get our executive team to choose
myCorptown as their communication
vehicle
Get our employees to help each other
(Employees as Communicators)
Sample Goals – Retention
Get our users to return to
myCorptown more often (more than 1x
per week)
Encourage our lapsed users to
return to myCorptown
Reduce stale content in areas on
myCorptown where dynamic content is
expected by our users
Sample Goals – Task Success
‘Give them what they need’ in
addition to what our users want, including
via search
Solve our users’ problems via self-
service where possible (without a
help desk call)
Sample Goals – Happiness
Ensure our employees are able to articulate the company story•What we stand for•How does my work fit into the story•How does my work touch consumers
Support a more transparent
culture at the company
Decrease frustration during
activities on myCorptown
Support our employees
through times of change
Sample Signals
GoalGet our users to return to myCorptown more often (more than 1x per week)
Signals• Providing content relevant to
users• Identify users who return to visit
the site within a week period
Sample Signals
GoalGet our employees to give feedback on content including likes and comments, submitting questions, etc.
Signals• Identifying posts that have
higher numbers of views• Identifying posts that have high
numbers of comments• Monitoring submitted questions
Sample Signals
Goal‘Give them what they need’ in addition to what our users want, including via search
Signals• Identifying usage of tools and
links that have typically been difficult to find• Monitoring search results for
successful and failed queries
Sample KPI w/Trending #1• Number of retained user home
page views minus bounces as percentage of all views• Over time period of 7 days• Indication of current metric
against target w/in a certain window• Trend
57%
StickinessHome Page
Target
70%Window
7 Days
Sample KPI w/Trending #2• Number of views during off-
work hour segments• Over time period of 5 days• Indication of current metric
against target w/in a certain window• Trend
1.6k
Time-Based VisitsOff-Work
Target
2kWindow
5 Days
Sample KPI w/Trending and History #1
• Number of posts related to influencers• Over time period of 30 days• Indication of current metric
against target w/in a certain window• Performance against last
window and previous year’s window• Example of where higher metric
is better
38
InfluencersPosts
Target
50Window
30 Days
YoY
+70.1%Last Month
-23.9%
Sample KPI w/Trending and History #2
• Number of questions submitted to portal team• Over time period of 30 days• Indication of current metric
against target w/in a certain window• Performance against last
window and previous year’s window• Example of where lower metric
is better
12
PortalQuestions Submitted
Target
15Window
30 Days
YoY
-156.5%Last Month
-10.4%
Sample KPI w/Snapshot #1
Average Length of Visit
> 5 m > 4 m > 3 m > 2 m > 1 m
• Average visit length• Segmented into groups• Current metric for one window• Shows spread/breakdown detail• Does not show trend
Sample KPI w/Snapshot #1• Average visit length in minutes• Over time period of 7 days• Indication of current metric
against target w/in a certain window• Trend
1.8
Length of VisitHome Page
Target
2.0Window
7 Days
Sample KPI w/Snapshot #2• Rating of frustration level on
portal from survey• Current metric for one window• Shows spread/breakdown detail• Does not show trend
Low Medium High0
1000
2000
3000
4000
5000
6000
Frustration Level
Frustration Level
Sample KPI w/Snapshot #2• Most common rating of
frustration level from survey• Over time period of 1 fiscal
quarter• Indication of current metric
against target w/in a certain window• Performance against last window
and previous year’s window• Example of where metric is
abstracted
M
PortalFrustration Level
Target
MWindow
Quarter
YoY
+27.6%Last Quarter
+5.8%
SharePoint AnalyticsAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
Changes ⓮ → ⓯
Changes ⓮ → ⓯• Web Analytics Web Part is not supported anymore• Reports Removed
• Browser Traffic• Top Users• Referring URL
Changes ⓮ → ⓯• Engine finds relevant information based on clicks, views, etc.• There are now visual indicators and usage numbers based on number
of views and number of unique visitors• Clicks and views are counted for each document• Content can be recommended• Search results influenced by the priority of an item• Ability to sort search results by "hit"• Engine is now extensible for third-party solutions
SharePoint Usage ReportingUsage
This report shows historical usage information about the site collection/site, such as the number of views and unique users. Use this report to identify usage trends and to determine times of high and low activity.
SharePoint Search ReportingNumber of Queries
This report shows the number of search queries performed. Use this report to identify search query volume trends and to determine times of high and low search activity.
Top Queries by Day/MonthThis report shows the most popular search queries. Use this report to understand what types of information visitors are seeking.
Abandoned Queries by Day/MonthThis report shows popular search queries that received low click-through. Use this report to identify search queries that might create user dissatisfaction and to improve the discoverability of content. Then, consider using query rules to improve the query's results.
No Result Queries by Day/MonthThis report shows popular search queries that returned no results. Use this report to identify search queries that might create user dissatisfaction and to improve the discoverability of content. Then, consider using query rules to improve the query's results.
Query Rule Usage by Day/MonthThis report shows how often query rules trigger, how many dictionary terms they use, and how often users click their promoted results. Use this report to see how useful your query rules and promoted results are to users.
SharePoint Library ReportingMost ViewsMost Views by Unique UsersMost Recommendation Clicks
QUICK EXPLORATION
Third-Party Tool CandidatesAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
Requirements – Compatibility
A tool for myCorptown analytics will require compatibility with (or extensibility to support) traffic delivered through…
Requirements – Awareness
In order to provide the most visibility into activity on myCorptown, a tool for analytics should be specifically aware of traffic, content, and actions related to… • Social Intranet
• Video Cloud
Requirements – Content Segmentation
Intranet administrators would like to segment analytics data by the types of content consumed.This would require support for…
• Metadata• File Extension
Requirements – Visitor Segmentation
Along with the desire to segment the content consumed within myCorptown, intranet administrators will need to examine various properties of the visits and visitor as well.
These attributes of a visit or visitor should be supported…
• Platform• Mobile Device• Browser• Date and Time• Session• Microsoft SharePoint 2013
• User Profile Property• User Audience Membership
• Microsoft Active Directory• User• User Attribute• User Group Membership
Requirements – Extensibility
Reusable MetricsIn order for a dashboard or other display and recalculation of raw metric data into KPIs and reports, an API or REST interface should ideally be provided by the tool.
Email CampaignsIn addition to native web traffic on myCorptown, the tool should support UTM tags commonly used in conjunction with email campaigns and the hyperlinks that refer visitors to the site.
Requirements – Pricing
For the purposes of evaluation, the following data points were used:
• Page Views~1,000,000/month• Users
~30,000
Google AnalyticsGoogle Analytics helps analyze visitor traffic and paint a picture of your audience and their needs. Mainly designed for e-commerce websites, it is a cloud-only/SaaS offering.
Google Analytics
Trackable Attributes/Actions• Page Views• Clickpaths• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Tracking occurs via JavaScript-
based page tags
Google Analytics• Data is often based on a subset, or percentage of sessions
• This is due to the practice of reporting trends in the subset to reduce processing time• This can occur in the reports, during the data collection itself, or in both
places
Google Analytics
Key Benefits Key Drawbacks In-Page Analytics Clickpath Visualization Sparkline KPIs Real-time Cloud-based
Google takes ownership of your metrics data and may use it how they see fit
Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization Shared login/Google account required
ActualMetrics AngelfishWith its roots in Urchin, Angelfish analyzes IIS logs as a completely separate application, usually installed on a server apart from SharePoint.
ActualMetrics Angelfish
Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Angelfish relies on the IIS logs• Although tagging with JavaScript
is supported, it is not required
ActualMetrics Angelfish• This tool appears to be in its infancy at this point in time
• A lot of features are in the works/slated for a future release
• The audience for this tool seems to be more technical users than business users• The interface seems like it would be more difficult to navigate for someone
outside of IT
ActualMetrics Angelfish
Key Benefits Key Drawbacks Spike analysis—second-level time
segmentation Unlimited nested segments Bandwidth analysis Broken link analysis Six different tracking methods for visitors
Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization Reports are only exportable in unformatted
form (CSV, XML) Separate hardware recommended
Adobe AnalyticsFormerly known under the brand Omniture, Adobe purchased this analytics suite and it is now part of their Marketing Cloud offering. Mainly designed for e-commerce websites, it is a cloud-only/SaaS offering.
Adobe Analytics
Trackable Attributes/Actions• Page Views• Clickpaths• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Tracking occurs real-time via
JavaScript-based page tags
Adobe Analytics• Adobe has put a lot of effort into the user interface to enable
visualization of data
Adobe Analytics
Key Benefits Key Drawbacks Sparkline KPIs Real-time Cloud-based; low overhead
Not ‘SharePoint-aware’ Does not analyze search queries No video metrics No Yammer integration No Sitrion integration Visitor attributes require customization
HarePoint Analytics for SharePointUsing its own analytics engine separate from SharePoint and any server logs, HarePoint Analytics installs as part of SharePoint while using its own SQL databases.
HarePoint Analytics for SharePoint
Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Not requiring JavaScript tags,
the tool utilizes its own tracking within SharePoint to provide metrics
HarePoint Analytics for SharePoint• HarePoint seems committed to this product, as it has adapted it from
SharePoint 2007 through 2010 and 2013, however they have yet to integrate social metrics
HarePoint Analytics for SharePoint
Key Benefits Key Drawbacks Workflows tracked Web part for analytics display within
SharePoint Aware of content within SharePoint, not
just actions by visitorso Unused content can be identified
Real-time No additional hardware required
Must access reports from Central Administration
Can’t track external links No social tracking
Intlock CardioLog AnalyticsDesigned specifically for SharePoint and covers all aspects of usage including portals, extranets, collaboration sites, and social. It is installed on a separate SQL server.
Intlock CardioLog Analytics
Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Tracking occurs real-time via
JavaScript-based page tags• IIS logs can be imported to
integrate historic data• It does not rely on SharePoint’s
built-in analytics or logs
Intlock CardioLog Analytics• This tool has great in-page feedback options including:
• Message bar• Survey• A/B multivariate testing
Cardio
Key Benefits Key Drawbacks Custom dashboards Direct integration with Active Directory and
SharePoint user profiles Display within SharePoint Page metadata Automatically generated reports Identify unused content Sitrion integration
Requires separate hardware
Piwik SharePoint AnalyticsWith a highly evolved UI for reports, Piwik offers solid SharePoint analytics by activating a feature per site collection.
Piwik SharePoint Analytics
Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Tracking occurs via JavaScript-
based page tags
Piwik SharePoint Analytics• The installation of Piwik is very streamlined
• A pre-configured server image is delivered and the tool automatically populates the tracking code into all relevant areas
Piwik SharePoint Analytics
Key Benefits Key Drawbacks Real-time viewing of visitor actions Sparkline KPIs Ease of setup
Lack of social tracking Separate hardware required
WebTrends SharePoint AnalyticsClosely aligned with Microsoft, Webtrends is a marquee vendor for web analytics.
Piwik SharePoint Analytics
Trackable Attributes/Actions• Page Views• Clickpaths• File Downloads• Search Queries• Visitors• Browser/Platform/Device• Date and Time
Analysis Method• Embedded JavaScript tags in the
page send information (encrypted in transit) to the tool
Webtrends SharePoint Analytics• Webtrends seems to have a particular focus on adoption and
engagement.• Their pitch is “consumable and actionable insights for business users”
• It has a well-polished UI
Webtrends SharePoint Analytics
Key Benefits Key Drawbacks Cloud-based; no overhead Targeted for consumption by business users Plug-ins for Brightcove Sitrion integration
Feature-heavy product; only small subset of feature set is required
Which would you choose?An Independent Evaluation of Third-Party SharePoint Analytics Offerings
EpilogueAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
Comparison Matrix
Areas of Comparison• Compatibility• Tracking Method• On-Premise vs. SaaS• Content and Process Awareness• Content Segmentation• Visits/Visitor Segmentation• Reporting• Feedback• Extensibility• Price
Which did I choose?An Independent Evaluation of Third-Party SharePoint Analytics Offerings
Recommended Tool
Why?• This recommendation is based largely on the feature set offered by
CardioLog• How those features align with the needs of the client in tracking activity on
myCorptown
• Content as well as activity is monitored• Lack of action can be tracked on documents and items
• In-page feedback mechanisms were very attractive
Why?• Sitrion analytics are built-in and
will require little to no customization
• Brightcove analytics should be able to be integrated via their API and brought into CardioLog
Why?• Other tools…
• Did not have as extensive a feature set to support the client’s requirements• Were less SharePoint-specific• Did not allow for business users to consume their reports as easily• Cost too much
How does the story end?An Independent Evaluation of Third-Party SharePoint Analytics Offerings
Chosen Tool
…and they lived happily ever after.
• Decided to address some of their near-term goals• Have reports against them based
on certain metrics• Some of the KPIs would be able to
be developed from those metrics (manually)
• In order for this to take place, an augmentation of current information being sent over the Internet to Google Analytics is required• This needs to include sending over
key user attributes as well as some classification of the pages being viewed
• Integration with SharePoint user profile data will be required to track things such as a user’s department
Warnings• Complete user information such as first and last names should not be
sent over to Google Analytics, as the data may be reused by Google in any manner• Steps should be taken so that any confidential information or trade
secrets are not accidentally injected into the Google ecosystem in the form of metatags, page titles, etc.
Wrapping UpAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
Wrapping Up
The views expressed in this presentation are not in any way endorsed by any of the product vendors, and I warrant no accuracy in the data presented.
Wrapping Up
Although I concluded that CardioLog Analytics is the leading candidate for this particular engagement, this does not by any means signify future disqualification by myself or BlueMetal in future evaluations of the other tool candidates.
Wrapping Up
In fact, another tool might be the best we could recommend to a completely different future client depending on their needs.
QuestionsAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
THANK YOUEVENT
SPONSORSWe appreciate you supporting the New York SharePoint Community!
• Diamond, Platinum, Gold, and Silver have tables scattered throughout
• Please visit them and inquire about their products & services
• To be eligible for prizes make sure to get your bingo card stamped by ALL sponsors
We Are Hiringin NYC!
• Mobile Web Software Engineer• Cloud & Services Senior Architect• Senior Java Developer• Senior Web Developer• Senior SharePoint Engineer • Devices & Mobility Associate Software
Engineer• Operations Coordinator (Part Time)Learn more:
https://www.linkedin.com/company/bluemetal-architects/careers
ReferenceAn Independent Evaluation of Third-Party SharePoint Analytics Offerings
ResourcesTool Website Tool Download/Request Tool Demo
ActualMetrics Angelfish Link
Adobe Analytics Managed Metadata Overview Link
Google Analytics
HarePoint Analytics for SharePoint Link Link
Intlock CardioLog Analytics Link
Piwik SharePoint Analytics
Webtrends SharePoint Analytics
Google Analytics for SharePoint 2013/Office 365 (CodePlex)
Resources