Technical Overview of FAST Search Server 2010 for SharePoint - SharePoint Saturday Perth

SharePoint SaturdaySponsorsGold

Silver

Bronze

Technical Overview of FAST Search Server for SharePoint 2010

Sezai KomurSharePoint Solutions Architect

CSG

SharePoint Saturday Perth 2011

What is FAST Search Server for SharePoint 2010?

• Microsoft buys FAST Search and Transfer in 2008 for $1.2 Billion US.

• Port of FAST ESP integrated with SharePoint 2010.

• New enhanced search engine integrated with SharePoint Server 2010


How is FAST Search better than SharePoint Server Search?

• Better search result quality– Better quality search engine relevancy and use of

linguistics, stemming & lemmatisation in search processing• Search Platform Extensibility• Advanced Content Processing

– Property Extraction– Document Processing Pipeline Extensibility

• Extreme Scale Search– Search Billions of documents with sub second search times

• Deep Refinement– Exact refinement number, SharePoint refiners aren't over

the entire result set


How is FAST Search better than SharePoint Server Search?

• Advanced Sorting– Sort on Managed Properties and Rank Profiles

• Tuneable Relevance with Multiple Rank Profiles• FQL Query Language• Contextual Search

– Tailor results and refinement to user profile or audience• Rich Web Indexing

– Dynamic web content and Javascript, highly customisable connector• Similar Results Detection & Results Collapsing• Thumbnails and Previews

– SharePoint 2010 Word and PowerPoint results via Office Web Apps• Visual Best Bets


FAST Search Service Applications

Two SharePoint Service Applications Communicate with FAST Servers• FAST Content Search Service Application

– Connector Configuration– Crawling

• FAST Query Search Service Application– Queries and Results from associated Web Applications– Managed Property Mapping configuration– People Search


FAST Web Services

Services that SharePoint communicates with• Content Distributors• Query Service• Administration Service• Resource Store• Log Server• See Install_Info.txt in FAST install folder and

look in IIS



SharePoint Farm

FAST Search Server 2010 for SharePoint Farm

QueryProcessing

Query SSA (Search Service

Application)

Web LinkAnalysis

Microsoft System CenterOperations Manager

Content SSA (FAST SearchConnector)- SharePoint- BDC- Exchange- Web

Site Collection Admin UI• Deployment• User Context Management• Promotion/Demotion

Indexing

QueryMatching

FAST SearchAuthorization

(FSA)

FAST IndexingConnectors

Content

Content

Monitoring

External federation sources

!!

People Search(query/crawl)

ActiveDirectory

Content

Administration

PowerShell• Schema configuration• Admin configuration• Deployment

configuration

Central Administration UI • Property mapping• Property extraction• Spell-checking

Item Processing

Web Server

Query Web Service

Federation Object Model

Custom front-end

Web Parts

FAST SearchQuery

User Profile

s


Simple Conceptual Architecture


Topology Diagram


Medium FarmSP2010 Farm

SP2010 Services Farm

FAST Search for SharePoint 2010 Farm

SQL 2008 Cluster

WFEQuery SSA

WFEQuery SSA

SP CrawlPeople Crawl

SP CrawlPeople Crawl

Crawl DBSearch Admin DB

FAST-ADM-1Admin

Content Distributor 1Indexing Dispatcher 1

Web Analyzer

FAST-FSTIDX-11Index (Search)4 Docprocs+


FAST-FSTIDX-21(Index) Search

QR Server


QR Server

FAST-ADM-2Content Distributor 2Indexing Dispatcher 2

Web Analyzer6 Docprocs+

(Enterprise Crawler)



QR Server

Crawl DB



Search Engine Basics

• Crawling– Gathering content to store in an index

• Indexing– Storing content in an index optimised for

searching• Querying

– Users execute searches against the index


Crawling


Crawling

• Connecting to sources of content to download files and data for processing

• Downloading documents or files (Items)• Working through URLs

– List or directory of items to crawl– Following links to other items

• Extracting information from files– Converting file formats to text for processing– Identifying properties or fields of information


FAST SEARCH SERVICE APPLICATIONSFAST SYSTEM DIRECTORYFAST WEB SERVICESCONNECTORS & CRAWLING

DEMO


Processing & Indexing


Item Processing• Format conversion

– IFilters – Advanced Filter Pack (Oracle Outside In) - 400+ formats

• Language and encoding detection• Lemmatizer

– linguistics normalization• Tokenizer

– word breaking• Entity extraction

– companies, locations• DateTimeNormalizer

– Date normalization• Vectorizer

– Create document vector for similarity searching• WebAnalyzer

– anchor text and link cardinality analysis• PropertiesMapper

– Map to crawled properties• PropertiesReporter

– report detected properties

Optional content pipeline

stages:• XML Properties mapper• Offensive content filter• Verbatim (whole word) extractor

(loads dictionary for custom extraction, e.g. product names)

• Field Collapsing• Entity Extraction (persons)• Document Processing Pipeline

Extension

FAST Search stores data to its Search Index after processing completes


Document Processing Pipeline…

Form

atCo

nver

sion

Lang

uage

Det

ectio

n

Entit

yEx

trac

tion

Lem

mati

zatio

n

Map

per

…


Property Extraction


Document Processing Pipeline Extensibility

• Items are processed in the Document Processing Pipeline after they are crawled and before they are stored in the index.

• Create and alter crawled property data.• You can run code and pass data to other systems

– ‘Deep’ Search of raw data– Geocoding– OCR– Audio and Video Transcription… The sky is the limit!


DOCUMENT PROCESSING PIPELINECRAWLED PROPERTIESMANAGED PROPERTIES

DEMO


Search UI


Web Parts


Refiners

• Refinement Panel Web Part• Add and edit refiners displayed by changing

filter category definition XML.• Properties specified in lower case, managed

property must have refinement enabled


Rank Profiles

• Configure Multiple Rank Profiles• Allow Selection of Rank Profile in Search UI

to change sorting• Defaulting based on user profile


SEARCH UIREFINERSDOCUMENT PREVIEWSVISUAL BEST BETS

DEMO


QUESTION AND ANSWERSP Saturday Slide Template Example


Related LinksSezai’s Bloghttp://sharepoint-sezai-moss-2007.blogspot.com/

Enterprise Search IT Professional Traininghttp://technet.microsoft.com/en-us/enterprisesearch/ff960998

Debugging and Tracing Pipeline Extensibility Stageshttps://blogs.msdn.com/b/thomsven/archive/2010/09/23/debugging-and-tracing-fast-search-pipeline-extensibility-stages.aspxhttp://techmikael.blogspot.com/2010/12/how-to-debug-and-log-fast-search.html

Shyam Nyaran’s blog – Visual Refiner Web Parthttp://www.dotnetbounce.com/archive/2011/02/06/visual-refiners-for-sharepoint-server-2010-and-fast-search.aspx

http://sharepoint-sezai-moss-2007.blogspot.com/



http://technet.microsoft.com/en-us/enterprisesearch/ff960998



https://blogs.msdn.com/b/thomsven/archive/2010/09/23/debugging-and-tracing-fast-search-pipeline-extensibility-stages.aspx



http://techmikael.blogspot.com/2010/12/how-to-debug-and-log-fast-search.html

http://techmikael.blogspot.com/2010/12/how-to-debug-and-log-fast-search.html

http://www.dotnetbounce.com/archive/2011/02/06/visual-refiners-for-sharepoint-server-2010-and-fast-search.aspx

http://www.dotnetbounce.com/archive/2011/02/06/visual-refiners-for-sharepoint-server-2010-and-fast-search.aspx

SharePoint Saturday

Thanks for listening!

Remember to submit your feedback so you can go into the raffle draw at the end of the day! And don’t forget that

you have to be at the draw to claim your prizes!

SponsorsGold

Silver

Bronze

Technology

Technical Overview of FAST Search Server 2010 for SharePoint - SharePoint Saturday Perth