Web data extraction v2 - uni-leipzig.de · 2009-04-01 · tool support Robomaker, Dapper, Lixto, Wintask. 23 Web Data Extraction ... Internet browser Web Content Extractor Low Poor

1

HTMLHTML--awareaware toolstools forforWeb data Web data extractionextraction

Student: Xavier Azagra

Supervisor: Andreas Thor

ThesisThesis presentationpresentation

2

Table of contentsTable of contents

IntroductionIntroductionData Extraction ProcessData Extraction ProcessData Extraction ToolsData Extraction ToolsRealized testsRealized testsFuture WorkFuture Work

Web Data ExtractionWeb Data Extraction

3

IntroductionIntroduction

Web Data Web Data ExtractionExtraction

We are going to center our effort in HTML data extractionThe predominant markup language for web pagesKind of semi-structured dataInformation following a nested structureSupport from W3C (World Wide Web Consortium)

4


Internet Internet growthgrowth

IntroductionIntroduction

1400 Million of Internet users

168 Million sites

May 2008 Web Server Survey - www.netcraft.com

Wikipedia – The free encyclopedia

5


IntroductionIntroductionPurposesPurposes ofof Web data Web data extractionextraction

Users Applications

Query

Integration

Extraction

Web data source

Get information from the Web to be used in other areas or by applications

Information retrieval ( e.g. Feeds, Web search engines…)

Let the user to access particular data from the Web

Economical issues ( e.g. stock market, shopping comparison…)

6


Internet was designed as a source of data for a human use. Problems appear when we want to extract data from HTML

Data not presented in HTML format:Password protected sitesCookiesSessions ID’sJavascriptDynamic content

Deep resources:Unlinked contentContextual webLimited access content

MainMain problemsproblems

Data Data extractionextraction processprocess

7


Data Data extractionextraction processprocessTypesTypes ofof contentcontent

Free text Structuredtext

Semi-structuredtext

Natural language texts

Patterns involving syntactic relations between words or semantic classes

of words

Textual information following a predefined

strict format

Use of the format description

Between unstructured collections of textual documents and fully structured tuples of

typed data

Extraction patterns are often based on tokens

and delimiters

8


WaysWays toto performperform data data extractionextraction


Manual API Wrapper

Manual Semiautomatic Automatic

PreciseTreat elements

individually

Specific Web SitesLimited specifications

Set of methodsIndependent of source

Ad hoc codeNot trivial

Error-prone

Support toolGUI support

Less Error-prone

Machine-learningtechniques

Supervised learning

9


When speaking about HTML-aware tools, before performing the extraction process, these tools turn the document into a parsing tree

HTML HTML structurestructure forfor data data extractionextraction


Each node represents a tagOuter tags are leavesExpressions to navigate through all the hierarchy

Maximum precision is found on the content of a leave

10

Presentation of the data without following a structureLogic, simple and organized content help to realize correct extractionsUnorganized content affects the HTML tree structure

Bad constructed HTML source documentsBad placed tagsRepeated tagsNo closed tags

Nested data elementsElements that are nesting data and then element by element could

contain differences


HTML HTML problemsproblems toto extractextract data (I)data (I)


11


HTML HTML problemsproblems toto extractextract data (II)data (II)


Problems choosing the correct Web page source exampleContent structure could change depending on some factorsExample: Result page of Web Search Engines

Problems using scripts or dynamic contentHidden or changing informationSyntax different to HTMLJavascript, PHP, AJAX or Flash

12


Data Data extractionextraction toolstoolsTaxonomyTaxonomy (I)(I)

13


Data Data extractionextraction toolstoolsTaxonomyTaxonomy (II)(II)

Languages forwrapper

development

Ontology-based NLP-based

Assist wrapper constructionAlternatives to general purpose

languages

Wrapperinduction Modeling-based HTML-aware

Rely on inherent structural features of

HTML documents

Based on syntactic and semantic constraints

Rules derived from a givenset of training examples

Try to locate in Web pagesportions of data that implicitly

conform to a structure

Extraction relyingdirectly on the data

14


Data Data extractionextraction toolstoolsFlowFlow ofof datadata

http://

INPUT

URL

Data File

Data extractionprocess

OUTPUT

Wrapper

XML, HTML,RSS/ATOMTEXT

Modules,CSV, email,JSON, XSL,Google Maps,Flash…

15


Data Data extractionextraction toolstoolsStructureStructure

10 HTML-aware tools

Categorization of this tools using several criterias

Test-bench scenarios

16


Data Data extractionextraction toolstoolsUsedUsed HTMLHTML--awareaware toolstools

DapperRobomakerRoadrunnerXWRAPLixto

WebharvestGoldseekerWinTaskAutomation AnywhereWeb Content Extractor

Commercial and non commercial toolsShell and GUI support toolsScreen scrapping and non screen scrapping toolsLinux and Windows tools…

17



10 HTML-aware tools



18


Data Data extractionextraction toolstoolsGUIGUI

No GUI- Shell commands- Configuration files and coding- Input files- Roadrunner

Integrated browser- Direct Interaction between the tool and the navigation browser- Visualize information of the Web elements- Lixto, Robomaker, Web Content Extractor

Web browser- Loads Javascript and Dynamic content- Separation between the tool and the window browser- Automation anywhere, Wintask

19


Data Data extractionextraction toolstoolsResilienceResilience

Capacity of continuing to work properly in the ocurrence ofchanges in the pages for which they are targeted

Common changes to:the data the structure

Add, erase or modify elementsthe visual designintroduce new technologies (AJAX, PHP, Javascript…)

The resilience grad varies depending the used tool

20


Data Data extractionextraction toolstoolsAdaptivenessAdaptiveness

Grade of a wrapper for built pages of a specific Web source on a given application domain to work properly withpages from another source in the same application domain

From all of the taxonomy of web data extraction tools onlythe Ontology-based tools feature fully resilience andadaptativeness properties

21


Data extraction toolsData extraction toolsScripting and expressionsScripting and expressions

The atomicity of the HTML parsing tree is found in a leave (outer tag)

Necessity to extract information in a more precise way

Self-scripting syntax

Regular expressions Patterns

Remove specialcharacters

Others

WintaskWeb ContentExtractorGoldseeker

Lixto Robomaker

Date formatting Text replacing

Robomaker LixtoRobomaker

Robomaker

22


Data extraction toolsData extraction toolsInput variablesInput variables

In some cases we need input variables to realize searches through Internet:

EbayWeb search enginesYoutubeAmazon…

We want to extract data from the resulting pages, we need tool support

Robomaker, Dapper, Lixto, Wintask

23


Input/Output formatsInput/Output formats

Data extraction toolsData extraction tools

RSS/Atom Feed, REST Web

Service, Web ClipHTMLRobomaker

XMLHTMLLixto

XMLHTMLXWRAP

XML, HTMLHTMLRoadRunner

XML, RSS, HTML, Modules,

Atom Feed, CSV,JSON,XSL,

YAML, email

HTMLDapper

Output FormatsInput Formats

TextHTML anddocumentsGoldSeeker

File, Excel, DB, SQL script File, MySQL script File, HTML, XML, HTTP

submit

HTMLWeb

Content Extractor

File, Excel, DB, EXE

HTML anddocuments

AutomationAnywhere

File, Excel, DBHTML anddocumentsWinTask

XMLHTMLWebHarvest

Output Formats

Input Formats

24


General features (I)General features (I)


FreeExecution timeResilienceComplexityInterface

No, requires licenseVery GoodGoodMedium

Program GUI, Internet browser

Lixto

YesGoodGoodMediumInternet browserXWRAP

YES, GNU GPL LicenseGoodPoorMediumLinux ShellRoadRunner

YesVery GoodVery goodMediumProgram GUI,

Internet browser

Robomaker

YesVery GoodGoodLowInternet browserDapper

25


General features (II)General features (II)


FreeExecution timeResilienceComplexityInterface

NoPoorPoorLowProgram GUI,

Internet browser

Web Content Extractor

NoGoodPoorLowProgram GUI,

Internet browser

Automation

Anywhere

NoGoodPoorMediumProgram GUI,

internet browser

Wintask

Yes, GNU LGPL

LicensePoorGoodMediumInternet

browserGoldseeker

YesGoodGoodHighProgram GUIWebHarvest

26


Advanced characteristicsAdvanced characteristics


GoodNoYesNoYesDapperGoodYesYesYesYesRobomakerPoorNoNoNoNoRoadrunnerPoorNoYesNoNoXWRAPGoodYesYesYesYesLixto

Javascriptor Dynamic

content

More than one page

Non static content pages

Scripts usage

Input variables

GoodNo No NoNoWeb Content Extractor

GoodYesNo NoNoAutomation

Anywhere

GoodYesNoYesBy scriptWintaskPoorNo YesYesNoGoldseekerPoorNoYesNoNoWebHarvest

27



10 HTML-aware tools



28


Realized testsRealized testsMethodologyMethodology

Created/SelectedWeb page

Selected Tool

Selected data

Toolresult

Correctresult

Compare TestResult

29


Realized testsRealized testsWeb Web searchsearch enginesengines (I)(I)

One of the most usedresources of the Web

Use of input variables anddynamic result pages

Yahoo! Search uses a livesearch input form

30


aaaWeb Content ExtractorrrrAutomation AnywhererrrWinTaskaaaLixtoaraRobomakeraaaDapper

MS Live SearchYahoo! SearchGoogle Search

Realized testsRealized testsWeb Web searchsearch enginesengines (II)(II)

31


aWeb Content

Extractor

rAutomationAnywhere

rWinTaskaLixtoaRobomaker

a / rDapper

Ebaysearch

Realized testsRealized testsEbayEbay

The most important auction shop of Internet

Use of input variables and dynamic result pages

Fields containing variable content

32


rWeb Content

Extractor

rAutomationAnywhere

rWinTaskrLixtorRobomakerrDapper

Pageflakes

Realized testsRealized testsDynamic content Web pagesDynamic content Web pages

AJAX based start page

Use of Dynamic content and personalized user modules

33

For each test:Realize a modification to the source pageUpload to a test serverExecute the tool and see if problems appear


Realized testsRealized testsResilience tests (I)Resilience tests (I)

1- Obtain a result page of Amazon.com2- Download the source page and related files3- Upload to a test server4- Configure tools to extract 4 fields: title, book format, new price and valuation

34


Deleting contentModifying CSS style tagsDuplicating extracted dataChanging order of extracted data

Realized testsRealized testsResilience tests (II)Resilience tests (II)

Deleting content Example:

rWeb

Content Extractor

aLixto

aRobomaker

aDapper

Erase td[0]

35

Designed a published books Web Page

We are going to extract data from the Last Published edition column with different precision each time:

All the information of the rowDate of the last publicationYear of the last publication2 last digits of the year of the last publication


Realized testsRealized testsPrecision tests (I)Precision tests (I)

36

Web Data Extraction

Done three different modifications to the source page with different characteristics to:

Extract data from formatted textExtract data using styled text (class attribute)Extract data from CSV formatted text

Realized testsPrecision tests (II)Precision tests (II)

37

Web Data Extraction

Realized tests

aaaaWeb

Content Extractor

rrrrAutomation Anywhere

rrrrWinTaskaaaaLixtoaaaaRobomakerrrrrDapper

2 last digits of the year of the last publication

Year of the last

publication

Date of the last

publication

All the information of

the last published

edition

Precision tests (III)Precision tests (III)Example: Extracting data from CSV source

38

Web Data Extraction

Future work

Given a Web source which features the tool accomplish. Useful to find the most suitable tool

Testing with non visual GUI tools

Realize a detailed document that contains all the realized work

…

39

Web Data Extraction

Thanks for your attention!

Documents

Web data extraction v2 - uni-leipzig.de · 2009-04-01 · tool support Robomaker, Dapper, Lixto, Wintask. 23 Web Data Extraction ... Internet browser Web Content Extractor Low Poor