ALEXANDRIA -

Analysing and Exploring Web Archives

Elena Demidova

L3S Research Center, Hannover

RESAW Seminar, London, December 2014

This presentation contains contributions from Helge Holzmann, Avishek

Anand, Mohammad Alrifai, Thomas Risse and Wolfgang Nejdl.

http://alexandria-project.eu

1

The (incomplete) World of Web Archives

Web Archives are an important part of our culture

2 but they are underused.

Recently an increasing interest can be observed

• Historians

Web Archives are an important part of our culture

2 but they are underused.

Recently an increasing interest can be observed

• Historians

2

Web Archive initiative

• Historians

• Social Sciences

• Journalism

• Law

• Historians

• Social Sciences

• Journalism

• Law

Web Archives

2 are interesting temporal collections

• good documentation of entities, topics, events, societies, etc.

• direct access to official communications, news publications and the public view

2 for many disciplines

• Large scale validation of hypothesis• Large scale validation of hypothesis

• Derivation and development of new hypothesis and theories

2 but rise many challenges

• huge amounts of unstructured data

• incomplete (e.g. many dead links)

• incoherent (e.g. linked pages are crawled at different times)

• challenging syntax (e.g. erroneous HTML) and semantic (e.g. endless and senseless sentences)

• restricted usage (outside of Alexandria)

3

What are the User Needs?

On a high level

• Searching

• Browsing

• Analysis of “something”

• Visualization• Visualization

But a “one size fits all” approach is not possible

Every research discipline has different needs

We need to learn what they are by

• Developing basic technologies for

temporal access and analytics

• Providing initial tools to foster the discussion

4

Top-100 most controversial Wikipedia

articles in English and German

The Alexandria Project

Motivation

Optimal access to Web archives requires new models and algorithms for retrieval, exploration, and analytics, taking into account

• the temporal dimension of Web archives

• structured semantic information available on the Web

• social media and network information

Objectives

• Evolution-Aware Entity-Based Enrichment and Indexing

• Aggregating Social Networks and Streams

• Temporal Retrieval and Ranking

• Collaborative Exploration and Analytics

Project runtime: 03/2014 – 02/2019

5

The Alexandria Project

Entity

Resolut

ion &

Evolutio

n

Time-AwareEntity Graph

t4t3t2t1

tnow

t2t3t4t

t1

6

WebWebWebWeb

Web

Social Networks & Streams

Linked Open Data Cloud

REvo

luti

Web Archive& Indext4

t3t2t1

tnow

t4tnow

Time- and Entity-Based Retrieval

1

2

3

4

6

7Aggregation

&Time-AwareIndexing

Entity

Linking 5

Improvement

Enrichment

complex query

Collaborative Exploration & Analytics

The Research Environment

Datasets

• German Web Crawl of the Internet Archive

• German Academic Web snapshots from L3S

• Twitter collection from L3S

• Wikipedia• Wikipedia

• Various Linked Data sources like DBpedia, Freebase, YAGO

Technical Infrastructure

• Apache Hadoop implementing Map Reduce for parallel processing

of large data sets.

• Apache Hbase - the Hadoop database, a distributed, scalable, big

data store.

• Apache Pig - a platform for analysing large data sets7

Web Archives as Data Set

Enables studying of long-term changes and evolutions

8

1997

2000

2006

2009

2014

Today: First Alexandria Studies

• Quantitative analysis of the German Internet Archive Dataset

• WikiTimes: Temporal event-centric database based on Wikipedia

• Entity name evolution on Wikipedia• Entity name evolution on Wikipedia

9

The Dawn of Today’s Popular Domains

A Study on 18 Years of the Archived German WebA Study on 18 Years of the Archived German Web

10

Helge Holzmann, Wolfgang Nejdl, Avishek Anand. In press.

How does the Web change and evolve?

• Is the Web really growing old and if so, how can we

characterize it?

• How has the size of web pages changed over time?

• Do websites from different categories (like business,

universities and technology) have different growth rates?

11

The German Internet Archive Dataset

18 years (1996 – 2013) of web data from the German (.de) domain

courtesy The Internet Archive

Approx. 80TB data (w/o duplicates) + CDX Index

11,174, 079 domains spanning > 2 years

The Popular German Web

Subset: 100 most popular German (.de) domains from 17 categories of

Alexa rankings

12

Domain Emergence

How many of the today‘s popular domains were known in a certain year?

13

,%

URL Age Evolution

Average age of the Web is increasing almost linear

Mostly caused by the long-living pages (e.g. in Universities domain)

14

URL Age Distribution (Normalized)

Normalized by the number of URLs in a year

Almost 70% of URLs are younger than a year at any time

15

Evolution of the Web's URL Volume

Growth = # Born - # Died

# Died URLs is rather constant, # Born URLs is growing

16

URL Size Evolution of Alive Sites

Linear growth of size over the years (content and/or markup)

A major growth in size is contributed by new born URLs

17

Web Dynamics Conclusions

• Extensive longitudinal study on 18 years of the popular German Web

• Analyzed how popular domains of today have grown

• In terms of age / volume / size

• Popular educational domains have been around for very long

• Shopping and game websites mainly emerged during last decade

• The Web is actually getting older

• at least the old part of it

• Domains grow exponentially

• doubling their volume every two years

• Tomorrow’s newborn URLs will be bigger than today

• resource planning and allocation, e.g., for Web archives

18

WikiTimes

A Knowledge Base of News Events with Daily SummariesA Knowledge Base of News Events with Daily Summaries

19

Giang Binh Tran and Mohammad Alrifai. 2014. Indexing and analyzing wikipedia's

current events portal, the daily news summaries by the crowd. In Proceedings of

the WWW Companion '14.

Wikipedia Current

Events Portal

� date

� daily updates

• category

• links to a story • links to a story

• links to news

• locations

20

News Stories on Wikipedia

21

WikiTimes System

http://wikitimes.l3s.de

http://wikitimes.l3s.de/rdf/

22

http://wikitimes.l3s.de

http://wikitimes.l3s.de/rdf/

WikiTimes Timelines

23

Next steps: creating event-centric collections from the Web

• WikiTimes contains about 900 stories from past 15 years and is

growing.

• We intend to create event-centric collections using WikiTimes

story timelines and a focused Web crawler

• Interesting scenarios from research perspective(s)?• Interesting scenarios from research perspective(s)?

24

Insights into Entity Name Evolution

on Wikipedia

25

Helge Holzmann, Thomas Risse. Web Information Systems

Engineering – WISE 2014.

� Entities evolve over time

� Names

� e.g., Leningrad to Saint Petersburg; Jorge Mario Bergoglio to Pope Francis

� Relations

� e.g., Brad Pitt – Jennifer Aniston to Angelina Jolie

� Roles

� e.g., Obama - from Senator to President; President - from Bush to Obama

Introduction: Entity Evolution

� e.g., Obama - from Senator to President; President - from Bush to Obama

26

� Challenges for Information Retrieval in Web Archives

� String Matching

� Names in queries do not match the name in texts anymore

� Entity Search – Finding entities that have evolved over time

� Evolution rarely covered in knowledge bases (DBpedia, YAGO, Freebase,

etc.)

t1914 1924 1991 today

LeningradPetrogradSt. Petersburg St. PetersburgSt. Petersburg

� According to Wikipedia guidelines:

� In case of a name change, new article, redirect from former name

� Reason is missing, only explanation in revision history (sometimes)

� Former names should be mentioned at the beginning of an article

� E.g., Thamesdown → Swindon (1997)

� “On 1 April 1997 it was made administratively independent of Wiltshire County

Handling of Name Changes on Wikipedia

� “On 1 April 1997 it was made administratively independent of Wiltshire County

Council, with its council becoming a new unitary authority. It adopted the name

Swindon on 24 April 1997. The former Thamesdown name and logo are still

used by the main local bus company of Swindon, called Thamesdown

Transport Limited.”

27

Research Questions:

•Do excerpts of limited length exist in articles that are dedicated to name changes?

•How many sentences do excerpts span that cover former name, new name and date of

change?

Handling of Name Changes on Wikipedia

28

� List pages provide semi-structured information

� e.g., List of city name changes: “Edo → Tokyo (1868)”

� Served as a starting point for our analysis

� Dataset

� 19 semi-structured seed lists (9 redundant) ���� 10 remaining (only geo names):

� Geographical renaming

� List of city name changes

� List of administrative division name changes

� 7 lists dedicated to renamings of cities in certain countries

� 1,926 distinct entities

Analysis

� 1,926 distinct entities

� 2,852 name changes

� 2,782 articles (of 1,898 entities)

� 766 entities with names resolvable to multiple articles

� 28 entities could not be resolved

29

� Most changes: 11 (Plovdiv, Bulgaria)

� Kendros (Kendrisos/Kendrisia) → Odryssa → Eumolpia → Philipopolis →

Trimontium → Ulpia → Flavia → Julia → Paldin/Ploudin → Poulpoudeva →

Filibe → Plovdiv

� In average: 1.48 changes, 2.39 different names

1. From evolution lists to excerpts

Results

• 696 entities remaining with 918 name changes annotated with dates

30

� 572 complete names changes are mentioned in articles

(preceding, succeeding name and date)

� 62.3% of the 918 considered name changes

2. Analyzing excerpts

• Sentence distance

Results

• More than 85% of the 572 considered changes are completely mentioned

in excerpts with 10 sentences or less

• More than two-thirds of the found changes have a sentence

distance of less than 3 (excerpts spanning 3 sentences or less)

→ There are (short) passages in Wikipedia articles dedicated to describing

an entity’s evolution!

31

� So far results only for geographic entities

� Manually parsed “List of renamed products” (unstructured)

� Same analysis

Generalization

� Similar results

� 80% of name changes reported in articles (vs. 62.3%)

32

� 80% of name changes reported in articles (vs. 62.3%)

� 91.7% span 10 sentences or less (vs. 85.3%)

� Again more than two-thirds of the changes have a sentence distance < 3

� 79.7% vs. 66.7%

� Assumption: Similar text patterns

� Enables automatic classification / detection of evolutions

� Demo @ Digital Libraries 2014

� http://evobase.l3s.de/DL2014_demo

Generalization: Evolution Base

33

http://evobase.l3s.de/DL2014_demo/Accenture

Conclusion

We are making the first steps to, e.g.:

• Analyse the content of the archives

• Develop the algorithms to get better entity- and event-centric

access access

• Create knowledge bases and collections to support these

algorithms

A long way to go2

Questions?

Thank you

for your attention!

http://alexandria-project.eu

34