Upload
joris-poelmans
View
2.844
Download
0
Tags:
Embed Size (px)
Citation preview
FAST Search Server 2010 for SharePoint: a developers walkthrough
MSDN Live Meeting24/05/2010
Joris Poelmans, ECM Unit manager
About Me
• ECM Unit Manager @ RealDolmen• SharePoint Server MVP• Blog – http://jopx.blogspot.com• Twitter – jopxtwits • Co-founder BIWUG – www.biwug.be• Co-author
– Real World SharePoint 2010 (November 2010, Wrox)– Real World SharePoint 2007 (August 2007, Wrox)– SharePoint 2007, The definitive guide (September
2007, O’Reilly)
Agenda
• Introduction to FAST Search Server 2010 for developers
• FAST Search Server 2010 – common developer tasks
About Fast Search for SharePoint
• April 2008– Microsoft acquires Norwegian Enterprise Search Solution Fast
Search & Transfer
• Overall Vision– To provide a market leading enterprise search platform that
integrates with your existing infrastructure
• Leader in the Gartner Magic Quadrant around Information Access
4
Gar
tner
MQ
Microsoft Enterprise Search - Productivity
• Quick, easy, powerful search (for free with Express edition)
Search Server 2010
• OOB General Productivity Search with light customization
SharePoint Server Search
• Heavily customizable OOB General Productivity Search and easy to build High-Value Search Applications
FAST Search 2010
Comm
on across the product line
• Common UI Framework• Social search features and integration• SharePoint platform integration • End user and site administrator enablement• Common Connector Framework (BDC)• Object Model, APIs and developer XP• Admin & deployment capabilities• Operations advantages (SCOM, scripting)
DEMODEMO OOTB FAST SEARCH
Visual, conversational search
7
Thumbnails
Sorting on any property
Similarity Search
Visual Best Bets
Scrolling PowerPointPreviews
Deep refinement with counts
on any property
Promote / Demote Results
Extractors to generate metadata
Did you know?
• Thumbnail and preview feature possible to implement for SharePoint Server as well – check out http://techmikael.blogspot.com/2011/05/document-thumbnails-and-powerpoint.html#more
Contextual Relevancy TuningAlan BrewerFinancial Services Sales”What’s the latest product news for productivity”
Kevin KennedyHealthcare Sales”What’s the latest product news for productivity”
Conversational
User Experience
Precise Control
Content Processing
Platform
Bene
fitFe
atur
es
Provide the best, most productive, user experience
Maximize productivity through control
Make the most of all your content
Tackle the most demanding search challenges
Visual Cues• Thumbnails & Previews• Visual Best Bets
Exploration• Deep refiners • Similarity Search• Multilevel Sorting• …on any property
Context-Based Results• Promote/Demote• User Context Relevance Control• Multiple profiles• Word/Term control• Custom/Extensibility
Property extraction • Prebuilt • Custom/Extensibility
Content coverage• 400+ document types• 82 languages• Advanced Linguistics• Offensive content filter
Extreme scale • in 3 dimensions• at best footprint
Query Expressiveness (FQL)• Relevance and sorting• Symbol & proximity search
FAST Search DifferentiatorsThe features above and beyond SharePoint Search
FAST Extends Capabilities of SharePoint Server• This is the out of the box SharePoint
implementation– FAST builds on top of this
SharePoint Server
End Users
Content + Profiles
People search
Content search
FAST Extends Capabilities of SharePoint Server
Unique Social Search from SharePoint Server
Common developer experience and IT Pro Foundation
People search
SharePoint Server
FAST Search for SharePoint
Content search
FAST Search for SharePointSummary of architectural elements
Query and Result
Processing
SharePoint Front-end
Content Processing
And Linguistics
Connectors:- SharePoint- BDC- Exchange
Search
IndexingFAST Search Authorization
Connectors:- Web Crawler- JDBC- Lotus Notes
Content
Content
Content
Qu
ery
Ob
ject
Mod
el
Monitoring Services
Administration and Schema Object Model
Federation Object Model
Query Web Service
OpenSearch or other Sources
Custom front-end
People Search
Content ProcessingA new level of search quality
End Users ContentFe
dera
tion
OpenSearch Source
Content Processor CrawlerIndexerQuery
Processor
Search Center Profiles
MetadataRelevanceControl
UserContext
IndexingConnectivity
Index Partition
…
Form
atCo
nver
sion
Lang
uage
Det
ectio
n
Entit
yEx
trac
tion
Lem
mati
zatio
n
Map
per
…
Content Processing Pipeline
The Content PipelineProcessing & refinement
Content Processing CapabilitiesLeverage existing metadata, use property extraction to add structure
…
FormatConversio
n
LanguageDetection
EntityExtraction
Configurable
Stages
Mapper
REDMOND, Wash., and OSLO, Norway — Jan. 8, 2008
Microsoft Corp. (Nasdaq “MSFT”) today announced that it will make an offer to acquire Fast Search & Transfer ASA (OSE: “FAST”), a leading provider of enterprise search solutions, through a cash tender offer for 19.00 Norwegian kroner (NOK) per share. This offer represents a 42 percent premium to the closing share price on
Jan. 4, 2008 (the last trading day prior to this announcement), and values the fully diluted equity of FAST at 6.6 billion NOK (or approximately $1.2 billion U.S. ). FAST’s board of directors has unanimously recommended that its shareholders accept the offer.
Location
Amount
Date
Company
• Strong linguistic coverage• 82 languages detected• 45 languages with advanced linguistics
features• Spell checking & synonyms improve search
experience• Unique offensive content filtering
• Configurable content pipeline• Transform and process content automatically• Index over 400 document types
•Property Extraction creates metadata• Prebuilt extractors: Person, Location,
Company, E-mail, Date, Time• Generic extractors for extension
DEMOCONTENT PROCESSING PIPELINE CONFIGURATION
The Power of FQL – FAST Query Language• Used to perform exact search and narrow the
scope of search to a field• Check out – FQL Syntax reference on MSDN –
– http://msdn.microsoft.com/en-us/library/ff394606.aspx
• Two modes:– Fast Query Language (FQL)– FQL Simple Query Language
• Standard web parts use the FQL simple query language
• You cannot use FQL directly from OOB search box in SharePoint
The Power of FQL -Full FQL Operators
Type Description Operators
String Enables you to specify query operations on a string of terms. This is the most common operator to use on text terms. Has a mode parameter which defaults to “phrase” but can also sue boolean or proximity operators
STRING
Boolean Enables you to combine terms and sub-expressions in a query
AND,OR,ANY,ANDNOT,NOT,COUNT
Proximity Enables you to specify the proximity of the query terms in a matching sequence of text
NEAR,ONEAR,PHRASE,STARTS-WITH,ENDS-WITH,EQUALS
Numeric Enables to specify numeric conditions in the query
RANGE,INT,FLOAT,DATETIME
Relevance Enables you to impact the relevance evaluation of a query
RANK,XRANK,FILTER
Using XRANK and FQL
Boost all documents that contain the name
Boost all documents that contain the name
DianneTibbott
MaryBaker
Or, in the FAST Query Language: Single (documents containing either Mary, Diane or both gets a boost of 5000): Xrank(string(”performance”), or(person:string(”diane tibbott”), person:string(”mary baker”)), boost=5000)) Accumulative (documents with both Mary and Diane gets a boost of 10000): Xrank(Xrank(string(”performance”), person:string(”diane tibbott”), boost=5000), person:string(”mary baker”), boost=5000)
Using Location Awareness and FQLGeo-search using sortformula
Formula sorting allows complete control of distance calculations (standard or ‘warped’ geometries)
Content Processing Search/Index Query
Sort results by distance from
end-user location
(50,100)
Index
External data
ABC
XYZ
123
Geo data
Geo specific processing
Geo
Pos_1 Pos_2
Lat Lon LonLat
Result
query=hotel&sortby=+[formula:sqrt(pow(50-latitude,2)+pow(100-longitude,2))]
Distance(from 50,100)
The Power of FQL - Toolset
• FAST Search for SharePoint MOSS 2010 Query Tool - http://fastforsharepoint.codeplex.com/
• FS4SP2010 Query logger - http://fs4splogger.codeplex.com/
DEMODEMO FAST HELPER TOOLS FROM CODEPLEX
Agenda
• Introduction to FAST Search Server 2010 for developers
• FAST Search Server 2010 – common developer tasks
Top Customization Scenarios• Modify the OOB End User
Experience
Add new Refinement category Show results from federated location Modify the look and feel of OOB end
user experience Enable sorting by custom metadata Add visual Best Bet for upcoming
sales event Configure different ranking for HR
vs. Engineering department
• Create new visual elements
Show Location refinement on Chart/Maps
Show tags in tag cloud Enable export results to Spread
Sheet Summarize Financial Information
from customers in Graphs
Top Customization Scenarios (Ctd.)
• Query & Indexing Shims
• Create new Search Driven Applications
Show email results from personal mailbox on Exchange Server through the EWS
Index content from custom repositories like Documentum
Create content processing plug-ins to create new metadata
Create new customer search experience Indexing content Define content for search Design search experience
Create new Audio/Video/Image search experience
Top Customization Scenarios (Ctd.)
Expand query terms based on synonyms defined in Term Store
Augment customer results with project information
Show popular customers/people inline with search results
Show people results from other sources
• Query & Result Pipeline Plug-ins
• Create a new Search Verticals
create a new customer page that shows: Customer Contact Details Customer Project Details Customer Contacts Internal Experts Customer related documents
Common development scenarios – 3 scenarios explained
• Modify the OOB End User experience– Configure FAST Search webparts using
additional search refiners
• Create new visual elements– Create new Extend Search UI with extra
webparts leveraging the SharePoint object model
• Query & Indexing Shims– Create content processing plugins to
create new metadata
Common development scenarios – 3 scenarios explained
• Modify the OOB End User experience– Configure FAST Search webparts using
additional search refiners
• Create new visual elements– Create new Extend Search UI with extra
webparts leveraging the SharePoint object model
• Query & Indexing Shims– Create content processing plugins to
create new metadata
Modify the OOTB end user experienceAdding refiners– Refiners will use Managed Properties– Refinement webpart uses specific XML
syntax see http://msdn.microsoft.com/en-us/library/ff625183.aspx
– Important fields• Type: TaxonomyFilterGenerator,
ManagedPropertyFilterGenerator,RankingModelFilterGenerator,CustomFilterGenerator
• MappedProperty: defines the name of the managed property in the service application which will be used as search refiner.
DEMOUSING LOCATION AND PERSON SEARCH REFINERS
Common development scenarios – 3 scenarios explained
• Modify the OOB End User experience– Configure FAST Search webparts using
additional search refiners
• Create new visual elements– Create new Extend Search UI with extra
webparts leveraging the SharePoint object model
• Query & Indexing Shims– Create content processing plugins to
create new metadata
Extend Search UI using SharedQueryManager class
• All search web parts use SharedQueryManager class
• One shared instance of the SharedQueryManager per search page
• At least one search results web part needs to be on the page
• Use SharedQueryManager to access other classes of the Federation Object Model
SharedQueryManager.GetInstance(this.Page).QueryManager
DEMOBUILD A SEARCH RESULT DRIVEN TAG CLOUD WEBPART USING FAST DOCUMENT VECTORS
Common development scenarios – 3 scenarios explained
• Modify the OOB End User experience– Configure FAST Search webparts using
additional search refiners
• Create new visual elements– Create new Extend Search UI with extra
webparts leveraging the SharePoint object model
• Query & Indexing Shims– Create content processing plugins to
create new metadata
Content Pipeline Stages
• Format ConversioniFilters, OutSideIn
• Language and encoding detection• Lemmatizer
Linguistics normalization
• TokenizerWord breaking
• Entity ExtractionCompanies, locations
• DateTimeNormalizer Date normalization
• Vectorizer Create document vector for
similarity searching
• WebAnalyzerAnchor text and link cardinality
analysis
• PropertiesMapperMap to crawled properties
• PropertiesReporter Report detected properties
Default Optional (sleeper stages)
The different plug-ins can either be configured from UI or from config files
Extensibility options to add custom processing modules / 3rd party SW
XML Properties mapperOffensive Content FilterVerbatim (wholeword) extractor
Loads dictionary for custom extraction, e.g product names
Field CollapsingEntity Extraction
Persons
…
Form
atCo
nver
sion
Lang
uage
Det
ectio
n
Entit
yEx
trac
tion
Lem
mati
zatio
n
Map
per
…
Creating content processing plugins to create new metadata
• Extending the content processing pipeline– Configure entity extraction (companies,
people, location)• Standard property extractors available in FAST
for Location, Company and Person• Person property extractor – default not enabled
– modify optionalprocessing.xml
–Whole word extraction– Add custom property extractors using code
Pipeline Extensibility API• Motivation
– Straightforward way to add text analysis functionality– Flexibility and supportability
• Example uses– Sentiment analysis– Translation– Auto-Classification
• Mechanism– Just before Mapper– “any” binary– Runs in sandbox with timeout
…
Extensibility
MapperStandard processin
g
DEMOWHOLE WORD EXTRACTIONSENTIMENT ANALYSIS USING CUSTOM PROPERTY EXTRACTOR
Summary
• FAST provides building blocks for search applications such as:– Research portals for
R&D teams– Call center portals with
360° view on customer– Employee self service
using social search and conversational search
– ...
RESOURCES
Recommended Resources
• FAST Search Top Blogs:– On the Search -
http://blogs.msdn.com/b/arnts/– Microsoft Enterprise Search Blog -
http://blogs.msdn.com/b/enterprisesearch/– Search Unleashed -
http://searchunleashed.wordpress.com– Tech and Me (Mikael Svenson) -
http://techmikael.blogspot.com/– Knut Brandrud -
http://blogs.msdn.com/b/knutbran/
Recommended Resources (Ctd)• SharePoint Server 2010 Virtual Labs
(incl. FAST) - http://msdn.microsoft.com/en-us/cc707678.aspx
• SPS 2010 IW Trial Virtual Image (incl. FAST) - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=751fa0d1-356c-4002-9c60-d539896c66ce&displaylang=en
• TechNet Survival Guide – FAST Search Server 2010 for SharePoint http://social.technet.microsoft.com/wiki/contents/articles/2149.aspx
THANK YOU
Stay up to date with MSDN Belux• Register for our newsletters and stay up to date:
http://www.msdn-newsletters.be– Technical updates– Event announcements and registration– Top downloads
• Follow our bloghttp://blogs.msdn.com/belux
• Join us on Facebookhttp://www.facebook.com/msdnbehttp://www.facebook.com/msdnbelux
• LinkedIn: http://linkd.in/msdnbelux/ • Twitter: @msdnbelux
Download MSDN/TechNet Desktop Gadget
http://bit.ly/msdntngadget