Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved. Taxonomy Strategies LLC The Business Of Organized Information

Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.

Taxonomy Strategies LLC The Business

Of OrganizedInformation

Metadata Maturity Survey Findings

Taxonomy Community of Practice Call SeriesJuly 15, 2009


Taxonomy Strategies LLCThe BusinessOf Organized

Information

Presenters

• Paul Wlodarczyk– Director, Solutions Consulting, Earley & Associates

• Ron Daniel– Principal, Taxonomy Strategies LLC

2



Information

Housekeeping

• Calls last 60 minutes

• We will take questions at the end of the call

• Lines will be muted during the call but you can email [email protected] or Skype Rebecca.M.Allen with immediate questions

• Use *6 to mute/un mute your phone

3




Metadata Maturity:Benchmarking Best Practices Survey Results, Analysis, and Conclusions

Paul WlodarczykDirector, Solutions ConsultingEarley & Associates

13 July 2009

Ron DanielPrincipalTaxonomy Strategies LLC



Information

Introduction

• Presentation is about the results of the 2009 survey of practices in Search, Metadata, Taxonomy, and Information Development.

• Audiences:– Survey participants: See the results of the survey and

how your organization compares to others.– Organization information specialists: See how to

benchmark your organization against others of your size and/or industry. Use that to justify catching up, or taking the next step, with your management.

– Information scientists: See the current state of practice and how it varies from previous surveys, or by size and/or industry of the organization.

5



Information

Background – Metadata Best Practices Survey• Goals for 2004, 2005 metadata best practices surveys:

– Help develop a Metadata Maturity Model (MMM)– Keep the MMM “tied to reality” through best practices

• MMM was a potential method to:– Identify search and metadata best practices and their

pervasiveness– Predict potential problems in consulting engagements– Help clients identify projects that are within their reach (and that

aren’t)

• In 2009 we updated the survey– Added questions on information development practices (e.g.

content reuse)

• Goals for 2009 survey:– Spot trends in best practices since 2005– Determine which practices impact Findability and Information

Quality– Help validate and refine the MMM

6



Information

Previous Metadata Maturity Model

7



Information

CMMI: The Best-Known Maturity Model

• Developed for Software Engineering, Funded by the DoD.

• ~400 Processes, in 22 Process Areas, Keyed to 5 Maturity Levels.

• Process Areas contain Specific and Generic Practices, organized by Goals and Features, and arranged into Levels

• Process Areas cover a broad range of practices beyond simple software development

• CMMI Axioms:–Individual processes at higher levels are AT RISK from supporting processes at lower levels.–A Maturity Level is not achieved until ALL the Practices in that level are in operation.



Information

Towards a Metadata Maturity Model

• Caveats:– Maturity is not a goal, it is a characterization of an

organization’s methods for achieving its core goals.– Mature processes impose expenses which must be

justified by consequent cost savings, revenue gains, or service improvements.

• Nevertheless, Maturity Models are useful as collections of best practices and stages in which to try to adopt them.

• We are looking for a model to help clients decide what to do, we are not looking to audit and certify service providers. A simpler model is appropriate.



Information

Second Metadata Maturity Model (ca. late 2005)

Practice Area Basic Intermediate Advanced Limiting

Search Capabilities • Uniform Search Box• Query Log Examination

• Index Multiple Repos.• Best Bets

• Facet Navigation UI

Metadata and taxonomy standards

• System Metadata Standards• Organization Metadata

Standard

• Multiple Repositories Comply w/ Org. Metadata Std.

• Reuse ERP Taxonomies• Taxonomy Maintenance

Process

• Taxonomy Roadmap• Highly Abstract Subject

Taxos (e.g. “Moods”)• Metadata Maintenance Doc

Tools and tool selection • Requirements, then Tools • Bakeoff Datasets • Budget for Bakeoffs • Tools, then Requirements

Staff training and hiring • Librarian or IA Expertise• Search Analyst Role

• Cross-Functional Taxonomy Creation

• Cross-functional taxonomy maintenance team

• SME Catalogers• Pre-hire Testing

Data creation and QA • CM Introduced • ROT-Elimination• Semi-auto tagging

• Quality Measures

Project management • Project Plan• X-Functional Teams

• Std. Project Methodology• Multi-Year Plan• Communication Plan• SMT Business Manager,

instead of IT Manager

• Early Termination

Executive support and ROI • External Search ROI• SMT in separate silos

• Intranet ROI Model • CEO knows Search ROI • Use it or Lose It Budgets



Information

Changes to the MMM from ‘04 to ‘05.

• Basic structure simplified from five maturity levels to four.

• Several processes moved up a level of sophistication as evidence showed they were not used at the basic level.

• More practices added, esp. around staff and training.

• The processes and their arrangement were arrived at through a mix of survey results and educated opinion.

• How have things changed in the last four years?

11



Information

Population statistics

12



Information

Participants by Organization Size

13

2009 N=2172005 N=60



Information

Participants by Job Role

14

2009 N=2172005 N=60

52%

39%

23%

39%

115%

100%

44%



Information

Participants by Industry

15

2009 N=2172005 N=60



Information

The State of Best Practices in 2009

16



Information

Search Practices, 2009 vs. 2005

17

2009 N=1292005 N=54

75%

28%

24%

30%

17%



Information

Metadata Practices, 2009 vs. 2005

18

2009 N=1272005 N=51

31%

13%

16%

19%



Information

Taxonomy Practices, 2009 vs. 2005

19

2009 N=1282005 N=52

59%

50%



Information

Comparing MMM to Reality (1)

• The MMM predicts that practices will be adopted in a certain sequence.– How accurate is that?

• Expect to see percentage not adopting increase as practice is more sophisticated.– See it in search, not in

others.– Overestimated the uptake

of query log examination.

20

Uniform Search Box 14.9

Query Log Examination 35.4

Index Multiple Repositories 28.8

“Best Bets” 39.1

Facet Navigation UI 39.1

System MD Stds. 40.4

Org. MD Std. 39.9

Multiple Repositories Comply 43.8

Reuse ERP Taxonomies NA

Taxo. Maint. Doc. 37.6

Taxonomy Roadmap 36.6

Highly Abstract Vocabularies NA

Metadata Maint. Doc. 73

Search

Standards

Percentage yet to adopt a practice



Information

Comparing MMM to Reality (2)

• How much growth in maturity is due to 1-person practices vs. group practices?– Solo practices

flagged in yellow.– Most change is due

to group practices.

21



Information

Information Development Practices, 2009

22

2009 N=1752005 N/A



Information

Analysis: IM Challenges by Company Size

23



Information

Key Question: Do mature organizations perform better?• We used the survey responses to develop a

“Metadata Maturity Quotient”– Score on individual survey responses:

• 0 = not practiced• 1 = under development• 2 = practiced in one unit• 3 = practiced in more than one unit

– Separate scores for search, metadata, taxonomy, and information development

– Overall Metadata Maturity Quotient is sum of Search, Metadata, and Taxonomy scores

– Eliminated respondents with large number of “Don’t Know / NA”

24



Information

Size Guides our Analysis

• Preliminary analysis showed that size matters• Divided respondents into groups by organization

size:– > 10,000 employees– 1,000 – 9,999 employees– 100 – 999 employees– < 100 employees

• Then into quartiles based upon Maturity Quotient:– Best in Class (top quartile)– Majority (middle two quartiles)– Laggards (bottom quartile)

25



Information

Maturity Quotient Score, by Organization Size

• Maturity increases with larger organizations because of IT investment, need

• Small organizations can get there faster with less complex deployments

• Selection bias – our small companies are more content-intense…

26



Information

Maturity Quotient Score, by Industry

27

N=128

Knowledge / content-intensive

Product / Technical

Transactional

Operational

Archival



Information

Respondents by Industry, Org Size

Small firms are more likely to be in content-intensive industries that are more mature with respect to metadata best practices.

28

Ave

rag

e I

nd

ust

ry M

eta

da

ta M

atu

rity

Number of Respondents



Information

Document Types by Organization Size

Mature firms are more document-intensive

29



Information

Metadata search, by organization size

Mature firms are more search-intensive

30



Information

Do mature organizations perform better?

• We assessed responses to “Information Management Challenges” for each group of respondents (by size and maturity level)

• Impact / importance of issues varies dramatically by organization size

• Generally, more mature organizations perform better on issues that matter, across all size groups

31



Information

“Significant Impact to the Business”

Larger companies have more intense IM needs

32



Information

“Significant Impact to the Business”

<100 Employees

100-999 Employees

1000-9,999 Employees

>10,000 Employees

Mature companies fare better than laggards regardless of organization size

33

0.0%

0.0%0.0%0.0%0.0%0.0%0.0%

0.0%0.0%0.0%0.0%0.0%



Information

Summary

• Larger companies and those in certain information-intense industries have more pain resulting from managing more unstructured content

• Generally speaking, the bigger the organization, the worse the pain

• Organizations with the most pain have adopted best practices in the areas of information development, search, metadata, and taxonomy

• Our findings show that those who adopt these practices have less pain

34



Information

Next steps

• Use the findings to refine the Metadata Maturity Model– e.g. Move query log examination out of the basic level, add

spelling correction to the basic level, add more staff practices, etc.

– Is there interest in collaborating with us on the next revision? Contact Rebecca Allen ([email protected]).

• Benchmark your organization– Take the survey and score the practices: 1 for under

development, 2 for practiced in one unit, 3 for practiced in multiple units.

– Sum the scores and compare to others in this presentation.– Is there interest in an online tool for this, or for more in-

depth diagnostic assessments based upon the MMM?

35



Information

36

Community of Practice Calls

SharePoint IA Group: http://tech.groups.yahoo.com/group/SharePointIACoP/

Taxonomy Group: http://finance.groups.yahoo.com/group/TaxoCoPSearch Group: http://tech.groups.yahoo.com/group/SearchCoP

Upcoming calls:

• August 5, 2009 – Conducting a Search Audit• September 2, 2009 – DITA• October 7, 2009 – Taxonomy Usability Testing• November 4, 2009 – Developing an Ontology• December 2, 2009 – Applications for Topic Maps• January 6, 2010 – Taxonomy Management



Information

Please fill out the survey that should be in your inbox.

Let us know what topics you are interested in and how we can improve the series.

Seth Earley

[email protected]

www.earley.com

781-820-8080

37




Questions?

Documents

Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved. Taxonomy Strategies LLC The Business Of Organized Information