44
FAST Search Server 2010 for SharePoint: a developers walkthrough MSDN Live Meeting 24/05/2010 Joris Poelmans, ECM Unit manager

MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Embed Size (px)

Citation preview

Page 1: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

FAST Search Server 2010 for SharePoint: a developers walkthrough

MSDN Live Meeting24/05/2010

Joris Poelmans, ECM Unit manager

Page 2: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

About Me

• ECM Unit Manager @ RealDolmen• SharePoint Server MVP• Blog – http://jopx.blogspot.com• Twitter – jopxtwits • Co-founder BIWUG – www.biwug.be• Co-author

– Real World SharePoint 2010 (November 2010, Wrox)– Real World SharePoint 2007 (August 2007, Wrox)– SharePoint 2007, The definitive guide (September

2007, O’Reilly)

Page 3: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Agenda

• Introduction to FAST Search Server 2010 for developers

• FAST Search Server 2010 – common developer tasks

Page 4: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

About Fast Search for SharePoint

• April 2008– Microsoft acquires Norwegian Enterprise Search Solution Fast

Search & Transfer

• Overall Vision– To provide a market leading enterprise search platform that

integrates with your existing infrastructure

• Leader in the Gartner Magic Quadrant around Information Access

4

Gar

tner

MQ

Page 5: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Microsoft Enterprise Search - Productivity

• Quick, easy, powerful search (for free with Express edition)

Search Server 2010

• OOB General Productivity Search with light customization

SharePoint Server Search

• Heavily customizable OOB General Productivity Search and easy to build High-Value Search Applications

FAST Search 2010

Comm

on across the product line

• Common UI Framework• Social search features and integration• SharePoint platform integration • End user and site administrator enablement• Common Connector Framework (BDC)• Object Model, APIs and developer XP• Admin & deployment capabilities• Operations advantages (SCOM, scripting)

Page 6: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMODEMO OOTB FAST SEARCH

Page 7: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Visual, conversational search

7

Thumbnails

Sorting on any property

Similarity Search

Visual Best Bets

Scrolling PowerPointPreviews

Deep refinement with counts

on any property

Promote / Demote Results

Extractors to generate metadata

Page 9: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Contextual Relevancy TuningAlan BrewerFinancial Services Sales”What’s the latest product news for productivity”

Kevin KennedyHealthcare Sales”What’s the latest product news for productivity”

Page 10: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Conversational

User Experience

Precise Control

Content Processing

Platform

Bene

fitFe

atur

es

Provide the best, most productive, user experience

Maximize productivity through control

Make the most of all your content

Tackle the most demanding search challenges

Visual Cues• Thumbnails & Previews• Visual Best Bets

Exploration• Deep refiners • Similarity Search• Multilevel Sorting• …on any property

Context-Based Results• Promote/Demote• User Context Relevance Control• Multiple profiles• Word/Term control• Custom/Extensibility

Property extraction • Prebuilt • Custom/Extensibility

Content coverage• 400+ document types• 82 languages• Advanced Linguistics• Offensive content filter

Extreme scale • in 3 dimensions• at best footprint

Query Expressiveness (FQL)• Relevance and sorting• Symbol & proximity search

FAST Search DifferentiatorsThe features above and beyond SharePoint Search

Page 11: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

FAST Extends Capabilities of SharePoint Server• This is the out of the box SharePoint

implementation– FAST builds on top of this

SharePoint Server

End Users

Content + Profiles

People search

Content search

Page 12: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

FAST Extends Capabilities of SharePoint Server

Unique Social Search from SharePoint Server

Common developer experience and IT Pro Foundation

People search

SharePoint Server

FAST Search for SharePoint

Content search

Page 13: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

FAST Search for SharePointSummary of architectural elements

Query and Result

Processing

SharePoint Front-end

Content Processing

And Linguistics

Connectors:- SharePoint- BDC- Exchange

Search

IndexingFAST Search Authorization

Connectors:- Web Crawler- JDBC- Lotus Notes

Content

Content

Content

Qu

ery

Ob

ject

Mod

el

Monitoring Services

Administration and Schema Object Model

Federation Object Model

Query Web Service

OpenSearch or other Sources

Custom front-end

People Search

Page 14: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Content ProcessingA new level of search quality

End Users ContentFe

dera

tion

OpenSearch Source

Content Processor CrawlerIndexerQuery

Processor

Search Center Profiles

MetadataRelevanceControl

UserContext

IndexingConnectivity

Index Partition

Form

atCo

nver

sion

Lang

uage

Det

ectio

n

Entit

yEx

trac

tion

Lem

mati

zatio

n

Map

per

Content Processing Pipeline

Page 15: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

The Content PipelineProcessing & refinement

Content Processing CapabilitiesLeverage existing metadata, use property extraction to add structure

FormatConversio

n

LanguageDetection

EntityExtraction

Configurable

Stages

Mapper

REDMOND, Wash., and OSLO, Norway — Jan. 8, 2008

Microsoft Corp. (Nasdaq “MSFT”) today announced that it will make an offer to acquire Fast Search & Transfer ASA (OSE: “FAST”), a leading provider of enterprise search solutions, through a cash tender offer for 19.00 Norwegian kroner (NOK) per share. This offer represents a 42 percent premium to the closing share price on

Jan. 4, 2008 (the last trading day prior to this announcement), and values the fully diluted equity of FAST at 6.6 billion NOK (or approximately $1.2 billion U.S. ). FAST’s board of directors has unanimously recommended that its shareholders accept the offer.

Location

Amount

Date

Company

• Strong linguistic coverage• 82 languages detected• 45 languages with advanced linguistics

features• Spell checking & synonyms improve search

experience• Unique offensive content filtering

• Configurable content pipeline• Transform and process content automatically• Index over 400 document types

•Property Extraction creates metadata• Prebuilt extractors: Person, Location,

Company, E-mail, Date, Time• Generic extractors for extension

Page 16: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMOCONTENT PROCESSING PIPELINE CONFIGURATION

Page 17: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

The Power of FQL – FAST Query Language• Used to perform exact search and narrow the

scope of search to a field• Check out – FQL Syntax reference on MSDN –

– http://msdn.microsoft.com/en-us/library/ff394606.aspx

• Two modes:– Fast Query Language (FQL)– FQL Simple Query Language

• Standard web parts use the FQL simple query language

• You cannot use FQL directly from OOB search box in SharePoint

Page 18: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

The Power of FQL -Full FQL Operators

Type Description Operators

String Enables you to specify query operations on a string of terms. This is the most common operator to use on text terms. Has a mode parameter which defaults to “phrase” but can also sue boolean or proximity operators

STRING

Boolean Enables you to combine terms and sub-expressions in a query

AND,OR,ANY,ANDNOT,NOT,COUNT

Proximity Enables you to specify the proximity of the query terms in a matching sequence of text

NEAR,ONEAR,PHRASE,STARTS-WITH,ENDS-WITH,EQUALS

Numeric Enables to specify numeric conditions in the query

RANGE,INT,FLOAT,DATETIME

Relevance Enables you to impact the relevance evaluation of a query

RANK,XRANK,FILTER

Page 19: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Using XRANK and FQL

Boost all documents that contain the name

Boost all documents that contain the name

DianneTibbott

MaryBaker

Or, in the FAST Query Language: Single (documents containing either Mary, Diane or both gets a boost of 5000): Xrank(string(”performance”), or(person:string(”diane tibbott”), person:string(”mary baker”)), boost=5000)) Accumulative (documents with both Mary and Diane gets a boost of 10000): Xrank(Xrank(string(”performance”), person:string(”diane tibbott”), boost=5000), person:string(”mary baker”), boost=5000)

Page 20: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Using Location Awareness and FQLGeo-search using sortformula

Formula sorting allows complete control of distance calculations (standard or ‘warped’ geometries)

Content Processing Search/Index Query

Sort results by distance from

end-user location

(50,100)

Index

External data

ABC

XYZ

123

Geo data

Geo specific processing

Geo

Pos_1 Pos_2

Lat Lon LonLat

Result

query=hotel&sortby=+[formula:sqrt(pow(50-latitude,2)+pow(100-longitude,2))]

Distance(from 50,100)

Page 21: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

The Power of FQL - Toolset

• FAST Search for SharePoint MOSS 2010 Query Tool - http://fastforsharepoint.codeplex.com/

• FS4SP2010 Query logger - http://fs4splogger.codeplex.com/

Page 22: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMODEMO FAST HELPER TOOLS FROM CODEPLEX

Page 23: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Agenda

• Introduction to FAST Search Server 2010 for developers

• FAST Search Server 2010 – common developer tasks

Page 24: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Top Customization Scenarios• Modify the OOB End User

Experience

Add new Refinement category Show results from federated location Modify the look and feel of OOB end

user experience Enable sorting by custom metadata Add visual Best Bet for upcoming

sales event Configure different ranking for HR

vs. Engineering department

• Create new visual elements

Show Location refinement on Chart/Maps

Show tags in tag cloud Enable export results to Spread

Sheet Summarize Financial Information

from customers in Graphs

Page 25: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Top Customization Scenarios (Ctd.)

• Query & Indexing Shims

• Create new Search Driven Applications

Show email results from personal mailbox on Exchange Server through the EWS

Index content from custom repositories like Documentum

Create content processing plug-ins to create new metadata

Create new customer search experience Indexing content Define content for search Design search experience

Create new Audio/Video/Image search experience

Page 26: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Top Customization Scenarios (Ctd.)

Expand query terms based on synonyms defined in Term Store

Augment customer results with project information

Show popular customers/people inline with search results

Show people results from other sources

• Query & Result Pipeline Plug-ins

• Create a new Search Verticals

create a new customer page that shows: Customer Contact Details Customer Project Details Customer Contacts Internal Experts Customer related documents

Page 27: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Common development scenarios – 3 scenarios explained

• Modify the OOB End User experience– Configure FAST Search webparts using

additional search refiners

• Create new visual elements– Create new Extend Search UI with extra

webparts leveraging the SharePoint object model

• Query & Indexing Shims– Create content processing plugins to

create new metadata

Page 28: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Common development scenarios – 3 scenarios explained

• Modify the OOB End User experience– Configure FAST Search webparts using

additional search refiners

• Create new visual elements– Create new Extend Search UI with extra

webparts leveraging the SharePoint object model

• Query & Indexing Shims– Create content processing plugins to

create new metadata

Page 29: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Modify the OOTB end user experienceAdding refiners– Refiners will use Managed Properties– Refinement webpart uses specific XML

syntax see http://msdn.microsoft.com/en-us/library/ff625183.aspx

– Important fields• Type: TaxonomyFilterGenerator,

ManagedPropertyFilterGenerator,RankingModelFilterGenerator,CustomFilterGenerator

• MappedProperty: defines the name of the managed property in the service application which will be used as search refiner.

Page 30: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMOUSING LOCATION AND PERSON SEARCH REFINERS

Page 31: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Common development scenarios – 3 scenarios explained

• Modify the OOB End User experience– Configure FAST Search webparts using

additional search refiners

• Create new visual elements– Create new Extend Search UI with extra

webparts leveraging the SharePoint object model

• Query & Indexing Shims– Create content processing plugins to

create new metadata

Page 32: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Extend Search UI using SharedQueryManager class

• All search web parts use SharedQueryManager class

• One shared instance of the SharedQueryManager per search page

• At least one search results web part needs to be on the page

• Use SharedQueryManager to access other classes of the Federation Object Model

SharedQueryManager.GetInstance(this.Page).QueryManager

Page 33: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMOBUILD A SEARCH RESULT DRIVEN TAG CLOUD WEBPART USING FAST DOCUMENT VECTORS

Page 34: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Common development scenarios – 3 scenarios explained

• Modify the OOB End User experience– Configure FAST Search webparts using

additional search refiners

• Create new visual elements– Create new Extend Search UI with extra

webparts leveraging the SharePoint object model

• Query & Indexing Shims– Create content processing plugins to

create new metadata

Page 35: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Content Pipeline Stages

• Format ConversioniFilters, OutSideIn

• Language and encoding detection• Lemmatizer

Linguistics normalization

• TokenizerWord breaking

• Entity ExtractionCompanies, locations

• DateTimeNormalizer Date normalization

• Vectorizer Create document vector for

similarity searching

• WebAnalyzerAnchor text and link cardinality

analysis

• PropertiesMapperMap to crawled properties

• PropertiesReporter Report detected properties

Default Optional (sleeper stages)

The different plug-ins can either be configured from UI or from config files

Extensibility options to add custom processing modules / 3rd party SW

XML Properties mapperOffensive Content FilterVerbatim (wholeword) extractor

Loads dictionary for custom extraction, e.g product names

Field CollapsingEntity Extraction

Persons

Form

atCo

nver

sion

Lang

uage

Det

ectio

n

Entit

yEx

trac

tion

Lem

mati

zatio

n

Map

per

Page 36: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Creating content processing plugins to create new metadata

• Extending the content processing pipeline– Configure entity extraction (companies,

people, location)• Standard property extractors available in FAST

for Location, Company and Person• Person property extractor – default not enabled

– modify optionalprocessing.xml

–Whole word extraction– Add custom property extractors using code

Page 37: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Pipeline Extensibility API• Motivation

– Straightforward way to add text analysis functionality– Flexibility and supportability

• Example uses– Sentiment analysis– Translation– Auto-Classification

• Mechanism– Just before Mapper– “any” binary– Runs in sandbox with timeout

Extensibility

MapperStandard processin

g

Page 38: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

DEMOWHOLE WORD EXTRACTIONSENTIMENT ANALYSIS USING CUSTOM PROPERTY EXTRACTOR

Page 39: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Summary

• FAST provides building blocks for search applications such as:– Research portals for

R&D teams– Call center portals with

360° view on customer– Employee self service

using social search and conversational search

– ...

Page 40: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

RESOURCES

Page 41: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Recommended Resources

• FAST Search Top Blogs:– On the Search -

http://blogs.msdn.com/b/arnts/– Microsoft Enterprise Search Blog -

http://blogs.msdn.com/b/enterprisesearch/– Search Unleashed -

http://searchunleashed.wordpress.com– Tech and Me (Mikael Svenson) -

http://techmikael.blogspot.com/– Knut Brandrud -

http://blogs.msdn.com/b/knutbran/

Page 43: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

THANK YOU

Page 44: MSDN Live Meeting - Introduction to FAST Search Server for SharePoint Server 2010

Stay up to date with MSDN Belux• Register for our newsletters and stay up to date:

http://www.msdn-newsletters.be– Technical updates– Event announcements and registration– Top downloads

• Follow our bloghttp://blogs.msdn.com/belux

• Join us on Facebookhttp://www.facebook.com/msdnbehttp://www.facebook.com/msdnbelux

• LinkedIn: http://linkd.in/msdnbelux/ • Twitter: @msdnbelux

Download MSDN/TechNet Desktop Gadget

http://bit.ly/msdntngadget