Community MetricsThe Novell Approach

Lee Romero

http://blog.leeromero.org

2009 October

Contents

Communities at Novell 3

Setting up the Discussion 6

Membership Metrics 10

Activity Metrics 17

Tying into Performance Mgmt 22

Some Advanced Metrics 25

Additional Metrics 28

Final Words 32

Communities at Novell

4

Community Metrics at NovellA brief overview of the program, part 1

• Novell started a CoP program in 2002/2003 time frame

• The program was primarily focused around the “solutions” and products Novell has marketed and sold

• The overall program responsibility fell to the Enterprise KM group:• KM group established the goals, methodologies and processes

• KM group also identified a set of standard infrastructure tools and supported those

‒ No new significant development / implementation was supported

‒ This was primarily a case of identifying what was in place and providing an a la carte menu for communities to work with

5

Community Metrics at NovellA brief overview of the program, part 2

• The CoP program identified levels of communities that largely guided investment and formal support for individual communities

• There were approximately a dozen “top level” communities over time

• There were also dozens of informal communities supported to a limited extent

• The a la carte menu of tools included (among other things):• Intranet web sites

• Mailing lists

• Enterprise Wiki

• Team spaces (implemented in LiveLink initially)

Setting up my discussion

7

What is a community member?It all started with a simple question

• The rest of this presentation will focus on a set of metrics we developed at Novell that were manageable, scalable and, I think, provided useful insights

• To start with, though, early on in our program, we were faced with a simple question from our community leaders:“How large is my community?”

• Which quickly turned into:“What is a community member?”

8

An AnswerA definition of community membership

• Because of our development limitations, we were faced with trying to track membership using either our existing tools or having some type of manual support

• In looking at our tools, we found that our mailing list infrastructure provided a reasonably good solution:• It already had (list) membership management functions

• People could subscribe / unsubscribe on their own

• We already knew that the lists needed to be aligned to communities

• With that insight and no other realistic option, we decided on a definition:• You are a member of a community if you are subscribed to any mailing list

associated with the community

9

An ImplementationHow we tracked membership

• Our mailing list server provided a simple XML format for members for each list• We implemented a mechanism to sync this member list data from the XML

format into a SQL database

• This, combined with some integration with other databases already available, provided a wide variety of membership reporting

• We later added on a mechanism that captured an “event” for each post to a list in the same SQL database, which led to a variety of activity based reporting

Membership Metrics

11

Basic Metrics

• With the basic structure in place, we found we could provide data to answer many questions:• How big is any given mailing list?

• How big is a community?

• How much change is there in community population over time?

• Various slicing and dicing by different demographics

• The following slides provide examples of many of these

12

Basic MetricsCommunity size

• Simple to query current size• By tracking “join” and “departure” events, it was also straightforward to

provide growth over time

• We could also measure the total overall population of the “community program”

13

Basic MetricsCommunity penetration

• Knowing the employee population size (total or by demographics), we could provide measures of “penetration” of the community program as a whole and by community against different groupings

14

Basic MetricsDemographics

• By combining community membership with HR information, we could provide break downs of membership (both overall program and individual community) on a variety of dimensions

15

Basic MetricsDemographics, continued

16

Basic MetricsOther questions

• Some other (not necessarily actionable) questions we could easily answer included:• How many communities is an employee a member of on average?

• How many communities is a community member a member of on average?

Activity Metrics

18

Activity Metrics

• Once we implemented a means to capture an “event” for each post in each mailing list, we were able to understand community activity (in this one tool) very easily

• We also were also able to answer (admittedly, simplistically) another key question we had been asked:• “How many *active* members do I have in my community?”

• We answered that with:• A member is active if they have posted at least one post during the time

period of interest.

19

Activity Metrics

• Some examples of specific metrics we were able to track

20

Activity Metrics

• We could also gain other potentially useful insights

• Identifying most active members• Potentially useful for identifying SMEs or core team members (or even

community leaders)

• Percent (and number and even individual identification) of lurkers in communities• Useful within a community to know who is there but not “active”

• Useful across communities to know when a community has a significant different rate of “lurkers” compared to other communities

21

Activity Metrics Networks within communities

• Using the activity data to connect people who corresponded, it was even possible to do data mining to get a sense of the network within a community

Performance Management

23

Performance Management

• Working with the HR group, we eventually were able to community involvement with our performance management program

• This was achieved by having community involvement embedded in the “employee self service” and “manager self service” portals

24

Performance Management

• The intent was not that a specific goal was desirable but that this provided a way to initiate conversations between a manager and an employee about involvement, encouraging:• Some thought about which communities were valuable for an employee

• Some reflection on level of involvement and activity

• Development of employees over time

Advanced Metrics

26

Advanced MetricsSome experiments into compound metrics

• Another simple question prompted some digging into other uses of the data we had:• “Why do we need to provide navigation to community sites [on the intranet],

anyway? They don’t get any traffic at all?”

• The resulting analysis attempts to draw a comparison between “web site visits” and community membership / activity• I came to call this measurement “knowledge flow”

• You can find a detailed description at:• http://blog.leeromero.org/2008/11/20/measuring-knowledge-flow-within-a-c

ommunity-of-practice/

• The eventual formula I worked out is:Kc = 2 * Pc * Ac - Pc * Ac

2

Mc

27

Advanced MetricsKnowledge Flow examples

Animations (in four dimensions!) available at:

http://blog.leeromero.org/2008/11/21/visualizing-knowledg-flow-in-a-community/

Additional Metrics

29

Additional Metrics

• On top of these metrics based on a mailing list implementation, the Novell CoP program also used a number of additional metrics

• These included:• Web site usage

• Intellectual asset production

• Specifically called out white papers as well

• “Anecdotes” from community members

• Knowledge share events

• And attendance at same

• For each of these, we provided quarter-to-quarter changes as well

30

Additional MetricsWiki metrics

• Another area we investigated but did not lock in on was use / edits of the corporate wiki

• Novell had (has) a corporate implementation of MediaWiki (in place since about 2003)

• We explored some metrics related to the Wiki for communities:• Establishing a standard Category for each community to use

• Using that Category assignment, we could:

• Track usage over time for pages in the category (it was integrated with Omniture)

• Track edits (“contributions”) over time to pages in that category

• Potentially identify anyone who edits such a page to be a community member

Final Words

32

In Summary

• Beyond all of the details here, a key takeaway is this:• With a very simple definition (of community membership) and a pretty simple

technical approach to implement that definition, Novell was able to gain a wide variety of insights

• Look at the tools you have, make some definitions (even if you know they are not perfect) and start tracking!

• Also, when you are thinking about what metrics you want to track, make sure all your metrics are actionable• How can your metrics be turned into actions to improve your communities?

33

Read more

If you would like to read more about the work behind this material, there are extensive write-ups available at:

http://blog.leeromero.org

Specifically, look under “Collaboration / Communities” on the “Posts by Topic” page