MS Fast Search Server

FAST Search for Sharepoint

Brad FreelsTechnology [email protected]

Microsoft’s Search Vision & Strategy

People & Expertise

Connect to all your Content

Search is Everywhere

Big Bet

Enterprise

Desktop Enterprise Internet Devices

Employee ProductivityMonetization

High Value SearchInternet

Consumer Portals / Partner Portals Marketing / B2B / …

Best of Microsoft - Best of SharePoint - Best of High End Search

Interactive Visual Search - Personalization - Social Networks

eDiscoveryResearch Portals

360o customer viewsCompetitive Intelligence

…

Introducing FAST Search for SharePoint

OOB User Experience

Tailoring General Productivity

Search Platform and Architecture

Search Driven Applications

Deployment and Administration

Summary and Resources

Microsoft Enterprise Search The 2010 Wave

General productivity searchLight customization

Customized productivity search and search driven

applications

• UI Framework• Social search features and integration• SharePoint platform integration • End user and site administrator enablement

Common across the product line

• Connector Framework (BDC)• APIs and developer Experience• Admin & deployment capabilities• Operations advantages (SCOM, scripting)

SharePoint vs. FAST Search for SharePoint

User InterfaceCentral AdministrationCrawler and Connector

Similarities

SharePoint Front-end

Central Administration

UI

Connectors

• SharePoint• File Traverser

• Web • BDC• Exchange• Notes

• Query and Result Processing

• Content Processing Pipeline

• Customizability and Scalability

Differences

Query and Result

ProcessingAdvancedContent

ProcessingSearch

Core

SharePoint and FAST Search Server

Content SSA

Query SSA

Content Sources

Search Results


OOB User Experience






Explore using navigatorsthat help you understand the scope of information available,with exact counts of results

Improve relevance with usebased on how people tag and rate content in SharePoint and on click-throughs of search results

Visualize your resultsso you can quickly recognize what you’re looking for without opening separate applications

Get better answers, fasterusing a visual, interactive search experienceQuery

completion

Document thumbnail

s

Scrolling previews

Read in Office Web

Apps

Relatedsearches & people

Federated results

Sorting on any

property

Find peoplewith address book search that matches names phonetically & lets you browse organizations

Use MySite profile pagesthat recognize and track your colleagues and make it easier to share your work with others

Discover expertisebased on terms matched frommined Outlook inbox data and SharePoint MySite profiles

Connect with people and expertiseand streamline how you find and collaborate with othersFilter by title, expertise &

other attributes

Expertise matching

Phonetic name

lookup

Org browsin

gFind recent

content

Real-timepresence

Deep Refinement and SortingEnables precise control of results

Enables conversational experience across all of the results

You will never miss any content Enabling better findability and explorationDiscover non-obvious relationships across the entire result set

Exact counts shows relative weightProvides analytic view of your resultsIndicates priority and importanceThe right lever to slice and dice your content

Sort on any fieldEmpower the user to use the relevance model that best fits their needsRearrange the result set to meet specific criteriaAlphabetical, numeric, and date

Out of the Box

Sorting OptionsExact Counts

Demo


OOB User Experience






Customize search to meet your business needs

Deliver results that are contextually relevant

Search in the language of your business

Tune relevancy to improve accuracy

Create structure from unstructured content

Configure the UI to extend your application Example:

Similarity Search

Custom Query Suggestions

Visual Best BetsIdentify static content that is always relevant

Built on SharePoint KeywordsMatches keywords and synonyms that are contextually relevant to users. Include banners, videos, external websites.

Easy and quick to setupPoint and click setup for site admins. Set and forget with content expiration dates . Web Parts allow for easy page customization

Visual Notification Set Vertical Orientation

Example:Visual Best Bets

Audience-specific search experiencesUse User Context to meet the needs of diverse groups

User context

Inform-atio

n contextApplicati

on context

Social

context

Renee LoEngineeringContoso Consulting”What should I know about implementing ERP?”

Alan BrewerSales ManagerContoso Consulting”What should I know about selling ERP consulting?”

Username & Group Memberships

LocationLanguages

Business UnitDepartment

TeamTime of Day

Preferred SitesSharePoint Audiences

Interests & Current ProjectsContext of Current Task

Quickly build a contextual experienceUser based tools for creating results that are relevant to your users

Pick the right ingredients Match the proper terms and contexts to boost relevancy for targeted users to ensure your users are always finding the right content

One-way synonymsKeywords map to other termsTwo-way synonymsKeywords become equivalent to other termsBest BetsHighlights key resources that are always relevant to a keywordVisual Best BetsExtend Best Bets with pictures, video, Silverlight controlsDocument Promotion / DemotionTailor specific document relevancy

Create new user contextsSite administrators create contexts based on user profiles to deliver relevant results to the right audiences

Create new keywordsSite Administrators have powerful and simple tools to configure the search experience for groups of users

Search in the language of your businessIdentify what is important to improve the search experience

Use language that has specific meaning to your business

Users can quickly refine content using familiar termsBuild confidence that you found the correct answers the first time

Leverage corporate knowledge to make content findable

Corporate taxonomiesBusiness terminologyProduct namesAcronyms

Define custom rules to identify unique terms

Handle complex terms such as part numbers or formsSearching for ”XXX 123 abc“ finds “XXX-123-abc“ and “GG^XXX-123-abc_HH“

Profit

Taxonomy

risk

best practices

Strategy Development

customer relations

revenue

brand management

complianceSOX

supply chain

Disaster Recovery

merger

acquisition

target markets

cloud computing

mobile workforce

quality

cost savings

market share

Productivity

Social Media

IP Telephony

communications

Competition

part numbers

Global presence

direct mail

storagearchiveaudit

XML


OOB User Experience






FAST Search Extends SharePoint Superset of capabilities; common base functionality and platform

End Users

Content

Content search

People search

Common query and federation

APIs

Common connectors and

content APIs

Best High-End Content Search based on FAST ESP

Unique Social Search from SharePoint Server

2010

Common IT Pro and developer experience

FAST Search for SharePoint

SharePoint Server 2010

Advanced Content Processing

+ Profiles

FAST Search for SharePointHigh Level Architecture

Federa

tion Op

enSea

rch

Content Processo

rCrawlerIndexer

Query Processo

r

Search Center

Content

User Profiles

…

Form

at

Con

ver

sio

n

Lan

gu

ag

eD

ete

cti

on

En

tity

Extr

acti

on

Lem

ma

tizati

on

Map

per

…

Content Processing Pipeline

FAST Server(s)

SharePoint Server(s)

Other Server(s)

People

Searc

h

Index Partition

FAST Search for SharePointHigh Level Architecture

Federa

tion Op

enSea

rch

Content Processo

rCrawlerIndexer

Query Processo

r

Search Center

Content

User Profiles

…

Form

at

Con

ver

sio

n

Lan

gu

ag

eD

ete

cti

on

En

tity

Extr

acti

on

Lem

ma

tizati

on

Map

per

…

Content Processing Pipeline

FAST Server(s)


Other Server(s)

People

Searc

h

Index Partition

FAST Query SSA

FAST ConnectorSSA

Extensible Content ProcessingEnables search that has a deep understanding of your information

Transform contentusing a processing pipeline that normalizes and cleanses all of your information

Use globallywith linguistics processing for 45 languages and recognition of content in 80+ languages

Add or extend stagesthat apply sentiment analysis, translation, or other business-specific processing you need

…

Format Converter

LanguageDetector

Lemmatizer

Word Breaker

Entity Extractor

Date/TimeNormalizer

Vectorizer

Web Analyzer

Properties Mapper

How does the pipeline work?A systematic approach to interpreting your contentSequential stages perform specific tasks while ingesting content

Breaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages

Process your content to make it searchableNormalizes content so that a consistent relevancy model can be appliedIdentifies structured and unstructured metadata in your contentMaps document metadata to SharePoint Crawled Properties

Map Crawled Properties

Maps all of the metadata that was discovered by the various pipeline stages

Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.

Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.

Date and Time Normalization

Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.

Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.

Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.

Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.

Language Encoding and

Detection

Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages

Format Conversion Extracts plain text from multiple file formats, encodings, and applications

Extending Pipeline capabilitiesStraightforward way to add custom text analysis functionality

Configure Optional Processing StepsXML Properties mapperOffensive Content FilterField CollapsingVerbatim (wholeword) extractor

Use a dictionary for custom extraction

Pipeline ExtensibilityCalls external applications for custom item processing

Add Custom ProcessingPipeline Extensibility is a specially defined stage that takes a set of crawled properties, as flat text as input and maps output to another crawled property

Sandboxed executionExecutable arguments and temporary files are automatically handled with timeouts.Runs just before the Crawled Property Mapper, providing accessibility within SharePoint

Add needed metadatawith pre-built extractors that automatically tag people, locations, and company names

Extend easilyto recognize business-specific terms and concepts—tailoring search for your information

Surface in navigationrecognized properties, making search results more relevant and discoverable

Powerful Entity ExtractionEnables search-driven navigation that is relevant to your business

CONCEPT

PRODUCT

COMPANY

Tune relevancy to improve accuracyChanging content and users need require a flexible solution

Start with great relevance OOBTuned for great general productivity experienceAutomatically improves relevancy with social click-throughs and link text analysis

Create new relevance modelsMultipleRank ProfilesBlend static and dynamic ranking parameters to instantly improve search resultsCreate with simple PowerShell commandsExpose as new sorting options

Standard Sorting Options

Custom Rank Profiles

Tunable RelevanceEnables unique, business-specific search results for diverse roles

Tailor search resultsby tuning relevance ranking based on content freshness, source and 7 other attributes

Make search your ownby configuring relevance to recognize synonyms, acronyms, and other business terms

Highlight featured content using configurable best bets that visually highlight editorialized search results for targeted users

Search 2010 “Stack”The platform for Search Customization

SearchWeb Parts

SharePoint Search Index

OpenSearch /Custom Source

SharePoint Indexer

Federation OM

Web Service,

RSS

FASTSearch Index

FAST Indexer

What’s New in 2010?Primary Search Web Parts now UnsealedFederation now a key Public OM layer

All Web Parts built on federationQuery alteration, custom Runtimes, blending results from multiple sources

Web Service / RSS Enhancements

FAST Search / SharePoint Search:Shared Web Parts, RSS, Web ServiceShared Federation OMIndex and Crawling SeparateFAST unique Content Processing Pipeline

SearchWeb Parts

Federation / Query

OM

Web Service,

RSS

Content Processing

Robust query languageUse FAST Query Language (FQL) for precise query development

FQL provides a robust and expressive query language

Wildcard support - *, ?Numeric Data types (Integer, Float, Decimal, Datetime)

OperatorsDirect field access (e.g., title:othello, author:shakespeare)Numeric (COUNT, RANGE, <, <=, >, >=)Boolean (AND, OR, ANY, NOT)Rank (RANK, XRANK)Proximity (NEAR, ONEAR)Sorting (SORT, SORTFORMULA)

String (operator support for strings) Boundary (starts-with, ends-with, equals)Filter


OOB User Experience






Search Driven ApplicationsMeet all the search application needs you have across your business

“How do I support the

unique search needs of teams and work

that impact our business?”

To do so, you need a search platform that has• A deep understanding

of your information• Flexible relevance to

meet diverse needs• A customizable UX to

increase user efficiency

Sales: 360o Customer Insight

Services: Knowledge Browser

Marketing: Competitive Intelligence

Research & Development:Innovation Portal

Support:Call Center Advisor

Operations:Systems/Logistics Portal

Legal, HR, IT, Finance, ……

real estate risk

News and external opinion to monitor and send to clients

Top information from Woodgrove…new market

view report to send to clients

Drilldown to single view with all clues about a customer: portfolio, holdings, communications, annual and quarterly

customer plans, etc…

Set of Customers to explore, with rollup

Experts to help, with availability

and rating

Immediate actions on

selected items

View of information across different pivots, with

drilldown

How would you create this?Content Crawling: bring in data from lots of places

OOB connectors to SharePoint (reports, account documents), exchange public folders, shared files; BDC with customization in SPD (no code) for customer portfolio/holdings

Content processing: creating metadataNames of holdings, offerings, key concepts, companies, peopleSynonyms for key concepts (real estate ~ REIT)

OOB web parts configured for style Federation, People Search, Search actions

Custom web parts for visual navigationRoll-up configured via results collapsingCustom relevance profileSharePoint workflows for act-on-selected-items


OOB User Experience






Secure, unified access to informationIndex or federate with content, applications, and services

Access content securelywith a full-featured content crawler and both index- and query-time trimming of results

Federate queriesusing OpenSearch to quickly access existing search indexes and online information services

Extend your reachusing Business Connectivity Services to access your content sources and business applications

OpenSearch Federation

Indexing Connectors

EnterpriseContent

Business Application

s

Information Services

User Experience

Search Index

Websites and Intranet

File Shares

IBM Lotus Notes

EMC Documentum

SharePoint Server

Exchange Public Folders

AD & LDAP Profiles

Etc …

http://www.linkedin.com/home?trk=hb_logo



FAST Search for SharePointSummary of architectural elements

Custom Front-End

OpenSearch or Other Sources

SharePoint Front-end

People Search

Qu

ery

Obje

ct

Mod

el

Query and Result

Processing

Security AccessModule

SearchCore

Indexing

Federation Object Model

Query Web Service

AdvancedContent

Processing

Linguistics

Web LinkAnalysis

Connectors

• Web Crawler• JDBC

Connectors

• SharePoint• File

Traverser• Web • BDC• Exchange• Notes• Documentu

mMicrosoft System Center Operations Manager

Monitoring Services

Administration and Schema Object Model

Site Collection Level Admin UI

• Keyword Management• User Context

Management• Site

Promotion/Demotion

PowerShell

• Schema configuration• Admin configuration• Deployment

configuration

Central Administration UI

• Property mapping• Property extraction• Spell-checking

FAST Server(s)


Other Server(s)

Content

FAST Search HW – Best Practices

CPU: 2 x 2GHz+ (Quad/six core)Memory: 24-48 GBDisk:

2 x 300 GB, SAS, 10K RPM (RAID 1)

CPU: 2 x 2GHz+ (Quad/six core)Memory: 24-48 GBDisk alternatives:

1.0 TB: 8 x 300 GB, SAS, 10K RPM (RAID10)1.8 TB: 8 x 300 GB, SAS, 10K RPM (RAID 5)3.6 TB: 16 x 300 GB, SAS, 10K RPM (RAID 5+0)New: 7.2 TB: 16 x 600 GB, SAS, 10K RPM (RAID 5+0)SAN: Configured for “database performance”

Storage Server

Admin / Processing Server

FAST Search – Main ComponentsSharePoint Crawler

Capacity: ~30 mill items per crawler node, SQL server needs to be scaled for high IO

Web AnalyzerCPU/disk footprint can vary by a factor of 10 depending on the content: - number of links - length of links - internal cross link ratioAverage capacity: ~30 mill items per web analyzer nodeCan be deployed with the Indexer in normal scenarios

Indexer/search nodeTwo supported models: - Normal mode:

~15 mill items per node~25 QPS

- High Density Mode:~ 40 mill. items per node~ 7 QPS

SharePoint Crawler

Crawl DBSP CrawlPeople Crawl

Indexer/Search

FAST-FSTIDX-11Index/Search

Web Analyzer

FAST-WA-1Web Analyzer

Columns give you more indexingNeed more Doc Processors and Content Distributor roles

Rows give you more query and redundancy

More Query roles

Rows and Columns

FAST Search – Pilot/Dev Deployment

LimitationsQuery rate: < 2 QPSContent: < 1 M items

Note:This configuration should work as an absolute minimum production environment running on VMs with a SAN backend but with very limited capacity and performance. Not tested.

SP2010 Farm

All roles

FAST Search for SP 2010 Farm

All roles

FAST Search – Extra Small FarmLimitations

Query rate: < 5 QPSContent: < 8 M items

SP2010 Farm FAST Search for SharePoint 2010 Farm

Web Front EndQuery

SP CrawlPeople CrawlSQL Server

Web Front EndQuery

SP CrawlPeople CrawlSQL Server

AdminIndex (Search)

Content DistributorIndexing Dispatcher

Web Analyzer4 Docprocs+

(Index) SearchContent DistributorIndexing Dispatcher


SP2010 Farm FAST Search for SharePoint 2010 Farm

SQL 2008 Cluster

Web Front EndQuery

Web Front EndQuery

Search Admin DBCrawl DB

SharePoint

AdminContent DistributorIndexing Dispatcher

12 Docprocs+Web Analyzer

Index (Search)Content DistributorIndexing Dispatcher

12 Docprocs+Web Analyzer

QR Server

(Index) SearchQR Server

SP CrawlPeople Crawl


**

*

Note:Servers marked with * are only

needed for high availability

FAST Search – Small Deployment


FAST Search – Medium DeploymentFAST Search for SharePoint 2010 Farm

AdminContent Distributor


Index (Search)Content Distributor


Index (Search)Web Analyzer

Indexing Dispatcher12 Docprocs+



Index (Search)Web Analyzer

Indexing Dispatcher12 Docprocs+


SP2010 Farm

SQL 2008 Cluster

WFEQuery SSA

WFEQuery SSA




SharePoint DB


FAST Search – Large DeploymentSP2010 Farm

FAST Search for SharePoint 2010 Farm

SQL 2008 Cluster

Web Front EndQuery

Web Front EndQuery SP Crawl

People CrawlSP Crawl

People Crawl


SharePoint

AdminConfigServer

Content DistributorWeb Analyzer12 Docprocs+

Index (Search)Content Distributor


Index (Search)Indexing Dispatcher




Index (Search)Indexing Dispatcher



Index (Search)Web Analyzer12 Docprocs+








OOB User Experience






Tools – QR Server

Neil Richard’s Blog Enabling the QR Server Blog Post - http://tinyurl.com/3b9ren4

To access1. Go to your FAST

Search for SharePoint Server

2. Open the browser

3. Access the URL http://localhost:13280

http://www.neilrichards.net/fast/debugging-fast-for-sharepoint-part-1-the-qrserver

http://fastforsharepoint.codeplex.com/Connect to web app running FAST SSA (SP box)Use it to test FQL

FAST Search for Sharepoint Query Tool

http://fastforsharepoint.codeplex.com/

http://fastforsharepoint.codeplex.com/

Useful ResourcesFAST University TrainingMSDN & TechNetBlogs

Leonardo De Souza’s Blog http://searchunleashed.wordpresss.com

Thomas Svensen’s Bloghttp://blogs.msdn.com/b/thomsven/

Comperio Search Nuggets http://nuggets.comperiosearch.com/

Books

http://searchunleashed.wordpress.com/

http://blogs.msdn.com/b/thomsven/



http://nuggets.comperiosearch.com/

http://nuggets.comperiosearch.com/

Education

MS Fast Search Server