Making Serious Sites SharePoint Search Mirjam van Olst Information Worker Solutions Macaw

Making Serious Sites

SharePoint Search

Mirjam van OlstInformation Worker Solutions

Macaw

Agenda

• Enterprise Search Challenges• Microsoft Search Vision• SharePoint End-user improvements• Knowledge Network• Administration Improvements• Performance and Capacity Planning• Customizing the Search UI• Develop Search Solutions


Enterprise Search Challenges

Enterprise Search Challenges

Information is stored in many places and in

many formats

People often know more than systems

Information access is controlled

Sales Executive Responding to an RFP

Business Analyst Preparing a Briefing

Finance Manager Reviewing a Budget


Microsoft Search Vision

UseUse

ShareShare

FindFind• Find information across the

desktop, intranet and Internet

• Use information via intuitive, familiar interfaces

• Share information among teams and connect people in real-time

Microsoft Search Solutions

Enable professionals to find, use, and share relevant business data quickly, easily and securely

Search As an Infrastructure Service

• Scenario spectrum– Within-site search– WSS or MOSS– Search-driven business applications - MOSS– Intranet Search – MOSS– Internet point of presence search - MOSS

Enterprise Search Architecture

...

...

X

User Requests

Load Balancer

Query servers

Web front ends

...Crawling

Content databases

Indexer

Propagation of indexes

...

External content

SharePoint Server 2007 SKUs

for Internet Sitesfor Internet Sites

for Internet Sitesfor Internet Sites

Standard EditionStandard Edition

Enterprise EditionEnterprise Edition

Microsoft® Office SharePoint® Server 2007 for Search (Standard Edition)

Microsoft® Office SharePoint® Server 2007 for Search (Enterprise Edition)

Microsoft® Office SharePoint® Server 2007 (Standard CAL)

Microsoft® Office SharePoint® Server 2007 (Enterprise CAL)

Search File shares, Web Sites, SharePoint Sites, Exchange PFs, Notes DBs OOB

Search 3rd Party Document Repositories

Indexes 200 File Types OOB (extensible)

Secure Content Access Control

Search for People and Expertise

Enhanced Search Center UI

Search Structured Data Sources

Document Limit 500,000 No Limit No Limit No Limit

Search Feature Comparison

Search Feature Comparison

WSS MOSSCan Index Local

SharePoint content

SharePoint Web, Exchange, File shares, Notes,

LOB…

Rich, relevant results X XAlerts, RSS, DYM, Dup collapsing X XScopes, Managed Properties XBest Bets, Result Removal, Query Reports XTabs XPeople Search, KN XBDC Search XAPIs Provided Query Query + Admin

Secure & Manage Your Search Deployment

Control access to information with permissions– Perform standard query time security trimming– Implement custom security trimming

Manage your search solutions– Analyze usage patterns using powerful out-of-the-box reporting– Tune relevance parameters for better results– Control deployment of desktop search with group policies

Scale your deployment– Cluster web front ends, query servers, and index databases– Improve indexing performance with a dedicated index server– Index up to 50 million items per index

Unlock Unstructured & Structured Data

Find information in unstructured repositories– Find content in web sites, file shares, SharePoint sites, public folders & Notes DBs– Use Protocol Handlers to index content in proprietary repositories– Support any file type with iFilters (over 200 supported formats out-of-the-box)

Use search as an interface for line-of-business systems– Index structured data sources using the Business Data Catalog (BDC)– Customize search results using out-of-the-box Business Data Web Parts– Make results actionable with Business Data Actions

Refine results and save important queries– Refine results using the keyword query syntax – Filter and sort result lists using relevant attributes– Save important queries and subscribe to RSS feeds or email alerts

IFilters

iFilters available for lots of different file types

http://www.citeknet.com/ for:•CAB•CHM•EXE•HLP•RAR•ZIP

https://support.cadcompany.nl/ifilter/default.aspx for:•DWG (Looking for Beta users)

http://www.adobe.com/support/downloads/detail.jsp?ftpID=2611•PDF

http://www.citeknet.com/

https://support.cadcompany.nl/ifilter/default.aspx

http://www.adobe.com/support/downloads/detail.jsp?ftpID=2611


SharePoint End-user Improvements

Relevance Improvements

• Best results on first page, every time• Enterprise Search is different from Internet Search

– Far less cross-linking of content– Lots of non-Web content: Office docs, line-of-business data,

etc.– Security is paramount

• Revamped relevance & ranking engine– Extensive collaboration with Microsoft Research, MSN,

Windows– Algorithms tuned for web and enterprise content– New search ingredients

Relevance Improvements

• New ingredients added, including– Click Distance– Anchor Text– URL Depth – Missing Metadata Creation– Automatic Language Detection

• Result is noticeably more relevant search– 100% better on all queries– 500% better on common queries

Influencing Ranking

• Best Bets• Additional Metadata • Authoritative Sites

Enhanced User Experience

• Simple yet powerful user interface– Query correction – “Did you mean?”– Hit highlighting– Duplicate Collapsing– Alerts– RSS Feeds– Security-trimmed search results– Actionable results through integration with Microsoft Office

applications, Windows Desktop Search, Live Communication Server

Enhanced Search Center

• Search Center– Web part-based query and results pages– Many options for customization

• Search tabs• OOB Search Web Parts

– Search Box– Search Core Results– Search Best Bets– Search High Confidence Results– Search Statistics– Search Paging– Search Action Links

Anatomy Of Results Page UI

SEARCH BOX

CORE SEARCH RESULTS

PAGINATIONSTATISTICS

ACTION LINKS

MATCHING KEYWORDS

AND BEST BETS

HIGH CONFIDENCE MATCH

HIDDEN SHARED OBJECT

Use of Search functionality in MOSS

• Content Query Web Part• Login Name and Display Name• My Team Sites Web Part• My Links• Colleague Tracker Web Part• Alerts

Search Center

The Search Center,the home for Enterprise Search

Business Data SearchSearch data, not just documents

• Information in LOB systems is often hard to access• MOSS 2007 can bring that data to your users

– Data is accessed through the Business Data Catalog– Exposed to many features in SharePoint

• Search can easily index the data– No need to write code– Highly customizable results– Integrated with scopes and search center

LOB Search

Find People and Expertise

Getting your job done means working withthe right peopleBuild rich personal profiles–Customize personal profile attributes–Manually update profile information–Automatically populate profiles using AD, LDAP directories, or LoB systems

Publish personal profiles using SharePoint My Sites–Create personal portals–Share files, links, and other useful information with others–Control access to information using security and privacy controls

Identify people who can help => Knowledge Network–Find people based on keyword matches with SharePoint personal profiles–Find “in-common” connections—managers, memberships, and colleagues–Group results by social distance and filter by common attributes


Vragen?


Knowledge Network

Knowledge Network

• Knowledge Network Overview• Client Profile Creation• SharePoint Server Integration• Privacy and Anonymous Brokering• Deployment and Administration• Extensibility

Knowledge Network is software for enterprise social networking that helps users collaborate more effectively by automating the discovery and sharing of

undocumented knowledge and relationships

Knowledge Network is software for enterprise social networking that helps users collaborate more effectively by automating the discovery and sharing of

undocumented knowledge and relationships

Who knows what?Who knows whom?

Better DecisionsMore Quickly

What is Knowledge Network?

Why Is Microsoft Creating KN?

• Three Core Beliefs– Most information is undocumented– It’s difficult to connect to the right person– “Weak ties” deliver significant value

• Employees are more likely to turn to colleagues for information*

• Not stored in documents• Not stored in databases• It’s Stored in peoples’ heads

Source: Delphi Consulting Group

Documented Knowledge

20%

IndividualKnowledge

80%

Most knowledge is undocumented

Core Belief #1

• Finding the right person often involves a referral by an intermediary

It’s difficult to connect to the right person

Core Belief #2

• People in our inner circle know basically the same people and the same things that we know

• People that they know have connections that we don’t have• By leveraging “weak ties” we can learn about new

opportunities, share new ideas or meet a new person or group of people

3+ DegreesTheir Extended

Circle

2 DegreesTheir Extended

Circle

1 DegreeYour Inner

Circle

You

“Weak ties” deliver significant value

Core Belief #3

The KN Solution

1. KN ClientAnalyzes email to create profile of

KeywordsColleaguesExternal Contacts

1. KN ClientAnalyzes email to create profile of


2. Publish ProfileIncremental updates

2. Publish ProfileIncremental updates

3. KN ServerAggregate profiles

Expertise informationSocial network

4. SeekerSearches for people

Who knows whatWho knows whom

5. People ResultsRanked by

Social distance to seekerExpertise and relationship relevance

• The goal is to strike the right balance between Utility, Simplicity and Privacy– Utility is how useful will this software

be to me?– Simplicity is how easy will this software be to

install, upload, maintain and use?– Privacy is how much personal information will this

software reveal and how much control do I have?

Striking the Right Balance

Privacy Concerns

The Simple Privacy Model

• Notification– Communicate steps of the profile creation and publishing process– Customers can expose privacy policy in the client profile wizard

• Control– User can choose which items to include/exclude– User can choose from 5 levels of privacy to apply to each

profile item to control who is allowed to view that information on the server

– Administrators can configure the default operation of the client, including opt-in/opt-out and the default privacy visibilities for profile items

– Administrators can determine which aspects of the product functionality to leverage including external contacts, anonymous results, and DL keywords

• Consent– KN sends no data to the server before the user has approved it

• Caveat is keywords associated with “public” Distribution Lists – aggregated into DL profiles on the server

Top Customer Questions…

• Why client-side mining?– Privacy – Nothing leaves user’s machine until they “publish”– Access to information – PSTs, future data sources– Distribute the processing

• Why not mine sources other than email?– E-mail is by far the richest and most pervasive source today– Calculating strength across different data sources adds complexity

• How long does analysis take? – Depends on a number of factors

• On the amount of email and unique contacts• On disk performance, RAM, and CPU• User activity – since analysis process runs at low priority

Client Profile Creation

Set Data Sources• Outlook folders to

analyze• Import Outlook

contacts• Import Instant

Messaging contacts

1. SyncRead each email Capture key “interaction data” in MDBRead in contacts from Outlook and IM clientsSync Colleagues from SPS profile

Select Information Run Analysis Review Profile Publish Profile

2. Contact ResolutionLookup contacts against GAL (MAPI)Internal or external?Is it a DL? DiscardCapture key properties

3. UpdateAggregate countsCheck thresholdsCalculate strength

4. Recommendation“Exclusion lists”Special rulesLimits appliedOrganization name mapping for External Contacts

KNClient.log file and MDB filesC:\Documents and Settings\<user>\Local Settings\Application Data\Microsoft\Knowledge Network

SharePoint Integration – Overview

Active Directory

Index

AD properties

DL memberships

People Profiles Index and Search

KN Client MDB

Search Center- Find People -

Extract and Publish

KeywordsColleagues

External ContactsManually enter Interests,Skills, Responsibilities, etc

SharePoint Integration – Data

SPS User Profiles

KN ProfileManagementWeb Service

KN Store

Publish KN Data to Server

Client

Server

Client

Server

KN Data

Crawl from SPS

KN Data

KN Client Data

Keywords (public)Colleagues

AD property changesSPS property changes

Keywords (non-public)External Contacts


SharePoint Integration – My Site

Keywords In Common

Profile Detail Tabs

KN MemberIndicator

Keywords Property

Colleagues External Contacts Keywords

Summary of public keywords

Desktop Search Integration

Anonymous Brokering

Seeker Initiates Search

Anonymous Results

Brokered E-mail

Reply to Help or Decline

Request Deleted

Knowledge Network

Client Deployment

System Requirements• Windows XP SP2 or Vista• Office 2007 or 2003

– Strongly recommend “Cached Exchange Mode”

• Exchange 2007 or 2003– With Active Directory

• .NET Framework 2.0

Configuration Options• Client policies (GPO)

– Update interval– Recommendation limits– Default “Show To” privacy– Opt-in/out first experience– Anonymous brokering– Retention “clean up”

• Custom privacy policy• KN Server settings

– Organization mapping– Exclusion lists

Server Deployment And Admin

Deployment• Leverages Office Server 2007

administration, provisioning, and deployment infrastructure

Administration Options• View and manage jobs• User management

– Manage KN “members”– Manage External Contacts

• Client configuration– Organization mapping– Exclusion lists

• Server statistics• E-mail templates

KN Extensibility

• Client-side– Managed API exposing data in KN DB – Retrieve user Keywords, Colleagues,

and External Contacts• Server-side

– Web service exposing a full-fledged query language for expertise/social network people search

• Can go beyond UI scenarios – e.g., multi-target queries– Web service to retrieve (read) and augment (write) profile data

• Bootstrap the profile or add custom keywords and contacts


Vragen?


Search Administration Improvements

Search Administration

• Design goals– Address SPS 2003 admin UI pain points– Unify WSS and MOSS search– Enable full programmability via the OM– Even better scalability and performance

Improved Admin & Management

• Consistent search system across WSS and SPS

• Revamped indexing management– Simpler UI– Better indexing logs

• Flexible scoping– Scopes decoupled from

content sources– Simple and compound scopes

• Schema– Map foreign schema into search

system– Streamlined UI

• Query and Result Reporting– Get visibility into user

search behavior– Logging for more

detailed custom reports• Best Bets

– Editorial results for specific queries

– Streamlined UI• Continuous propagation

– Indexed content is immediately searchable

• Updated object model and APIs– Custom tools and services

Indexing Management

• Streamlined experience and more control• One index per shared service no need

to worry about managing discrete indexes• Multiple start addresses per content source• MOSS indexes can drive the WSS search

experience– Allow upgrade from WSS to MOSS

Security

• Query-time security trimming in SPS 2003– File shares, WSS/SPS 2003, Exchange, Lotus Notes

(via mapping)• Now support pluggable authentication

for content in WSS/MOSS sites– Based on ASP.NET 2.0 model

• Minimum required crawler permission is nowjust Full Read, not administrator– Still provides the same security trimming functionality

• Ability to remove Single Items


Customization

• Search in your company is different– Different metadata might matter

• Documents: Title, Author, File location, size• Records: Patient, Doctor, Healthcare provider, SSN…

– How users meaningfully scope searches differs• “All finance documents”• “All patient records”• “All published documents”

• Customize results to “pop” metadata that matters• Customization offered at many levels

– Webparts, XSLT/CSS, full object model…– SDK available


Adding a tabCustomizing the Search results

Query Reporting

• Best way to improve searchis to understand current usage

• New out of the box usage reporting– Query volume trends, top queries, click through

rates, queries with zero results, etc. – At both site and service provider levels– Export data for extended reporting in Excel– Respond to feedback with configuration changes or

editorial results

Query Reporting

Query Reporting


Performance and Capacity Planning

Performance Improvements

• Key new features make the crawls faster so the content is fresher– More efficient SharePoint Crawling

(Change Log Crawl)– Continuous Propagation– Unified WSS and MOSS search– Security Change Only Crawl

• Maximum scale is 10s of Millions of documents per indexer

Search Performance Planning

• Key learnings– Think about freshness goals, change rate– Evaluate remote server responsiveness– Upgrade WSS 2003 sites to WSS 2007 sites to index them

faster– Balance results freshness with load on servers– In most cases, your search index is on one server

• Scale strategies might include– Dedicated WFE for crawling– Separate indexer machine– Separate SQL DB for Search– Fast disk for SQL, fast CPU for Indexer, more memory


Vragen?


Customizing the Search UI

Search UI Programmability

• Easy and powerful:– Customize using web part properties– Customize using XSLT, XML, and script

• Manipulate URL parameters

– Build web parts and apps using the query OM• keyword and SQL syntax

– Remote apps using the query web service

Customize UI With XSLT

• Web part property specifies query columns• Search web parts get results XML• Web parts apply transform to XML nodes• Search page URL parameters enable

query manipulations• Appropriate for scenarios requiring

– Change results layout, look and feel– Keyword syntax queries

• Properties must have been added as Managed Properties

• XSLT uses them for display or logic• Use the core search results web part 'Columns'

property in the tool pane:<root><root>

<SelectColumns><SelectColumns>

<Column Name=“ShirtSize”/><Column Name=“ShirtSize”/>

<Column Name=“ShirtColor"/><Column Name=“ShirtColor"/>

<Column Name=ShirtImage"/><Column Name=ShirtImage"/>

<...><...>

</SelectColumns></SelectColumns>

</root> </root>

Specify Properties To SELECT

<xsl:call-template name="DisplayString"><xsl:call-template name="DisplayString">

<xsl:with-param name="propval" select=“ShirtSize"><xsl:with-param name="propval" select=“ShirtSize">

<xsl:with-param name="proplabel“>Shirt Size:<xsl:with-param name="proplabel“>Shirt Size:

</xsl:with-param></xsl:with-param>

<xsl:call-template><xsl:call-template>

..

..

(Later…)(Later…)

..

..

<xsl:if test='string-length($propval) > 0'><xsl:if test='string-length($propval) > 0'>

<xsl:value-of select="$proplabel" /><xsl:value-of select="$proplabel" />

<xsl:text> </xsl:text><xsl:text> </xsl:text>

<xsl:value-of select:"$propval"/><xsl:value-of select:"$propval"/>

</xsl:if></xsl:if>

Displaying Custom Properties

• Pivot or drill-down on a property to offer different queries

• Construct URL, re-post to page withnew query

<xsl:template name="DisplayPivot"><xsl:template name="DisplayPivot">

<xsl:param name="value" /><xsl:param name="value" />

<xsl:param name="fieldname" /><xsl:param name="fieldname" />

<xsl:param name="label" /><xsl:param name="label" />

- <a href="http://co-113/Search/Pages/results.aspx?k={$fieldname}- <a href="http://co-113/Search/Pages/results.aspx?k={$fieldname}%3a{$value}">%3a{$value}">

<xsl:value-of select="$value"/><xsl:value-of select="$value"/>

<xsl:text> <xsl:text><xsl:text> <xsl:text>

<xsl:value-of select="$label"/></a><xsl:value-of select="$label"/></a>

</xsl:template></xsl:template>

Property Filtering

<xsl:call-template name="DisplayPivot"><xsl:call-template name="DisplayPivot">

<xsl:with-param name="value" select="fileextension"/><xsl:with-param name="value" select="fileextension"/>

<xsl:with-param name="fieldname"><xsl:with-param name="fieldname">

FileExtensionFileExtension


<xsl:with-param name="label"><xsl:with-param name="label">

FileFile


</xsl:call-template> </xsl:call-template>

Property Filtering

• Established search UI element– Properties

monty -actor:"John Cleese"monty -actor:John• same-property clauses are ORed• Equality only, prefix or exact phrase match

– And / Or / Not / Phrase"chocolate covered" -frog• No elaborate groupings

– Scopes: • scope:movies

Keyword Syntax

URL Request Parameters

• Content– results.aspx?k=fish

• Scopes– results.aspx?k=fish&s=BBC

• Sort– results.aspx?v=date – results.aspx?v=relevance

• Page– results.aspx?start=21


Develop Search Solutions

• Use query OM:– To build custom search UI web parts– To have direct access to query and results

properties– When you need to manipulate results on the server

• Same flow, keyword or SQL syntax– Set up query text and options– Execute()– Retrieve a set of IDataReader objects

Using The Query OM

• 'ResultsTableCollection': IDataReaders• Custom properties provide information on query

and query execution• One table for each requested result type

– RelevantResults– SpecialTermResults– HighConfidenceResults– DefinitionResults

Process Results

• http://<server>/_vti_bin/search.asmx• Two query methods:

– Query• Research and Reference Service-

compatible XML• Use in the Office Research pane

– QueryEx• Returns .NET DataSet• Full set of result types, features

Using The Query Web Service

Customizing With XSLT

Putting it all together


Vragen?

Macaw Beech Avenue 140 1119 PR Schiphol-Rijk Tel +31 (0)20 8 510 510 Fax +31 (0)20 8 510 511