HOBBITProject Overview
Axel Ngonga
Horizon 2020GA No 688227
01/12/2016–30/11/2018
ESWC 2016Crete, GreeceJune 1, 2016
Axel Ngonga (InfAI) Project Overview June 1, 2016 1 / 13
A Lot of Data
1
1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataAxel Ngonga (InfAI) Project Overview June 1, 2016 2 / 13
A Lot of Tools
2
2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngAxel Ngonga (InfAI) Project Overview June 1, 2016 3 / 13
Core Questions
Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?
Axel Ngonga (InfAI) Project Overview June 1, 2016 4 / 13
Many Questions
Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?
Axel Ngonga (InfAI) Project Overview June 1, 2016 5 / 13
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users
Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× faster
Archiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users
Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysis
Open-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users
Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users
Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13
GERBIL
Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1
Axel Ngonga (InfAI) Project Overview June 1, 2016 7 / 13
HOBBIT
Rationale
A community-driven benchmarking framework for the community
Focus on Big Linked DataCover all steps of the Linked Data lifecycle
Used by a growing number of companiesMature and maturing technologies
Open benchmarks based on industrial dataand use cases
Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13
HOBBIT
Rationale
A community-driven benchmarking framework for the community
Focus on Big Linked DataCover all steps of the Linked Data lifecycle
Used by a growing number of companiesMature and maturing technologies
Open benchmarks based on industrial dataand use cases
Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13
Aims
1 Gather real requirementsPerformance indicatorsPerformance thresholds
2 Develop benchmarks based on real data3 Provide universal benchmarking platform
Standardized hardwareComparable results
4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association
Axel Ngonga (InfAI) Project Overview June 1, 2016 9 / 13
Overview
Data Collection
Industrydata
Measure Collection
Benchmark Creation
Benchmark 1
KPIsTasks
KPIsTasksKPIsTasks
KPIsTasks
KPIsTasks
KPIsTasks
Benchmark 2
Benchmark n
HOBBITPlatform
Solution 1
Solution k
Solution 2
Challenges
Reports
Participants/Community
Axel Ngonga (InfAI) Project Overview June 1, 2016 10 / 13
Architecture
Controller
Data Generator
Task Generator
Data Generator
Data Generator
Task Generator
Task Generator
FrontendSystem Adapter
System
data flowcreates component
Store
SPARQL Endpoint
Analysis
BenchmarkEvaluator Module
Eval. Store
Message BusNode Observer
Logging
Axel Ngonga (InfAI) Project Overview June 1, 2016 11 / 13
We Offer Benchmarks
Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity
Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction
Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised
Storage and CurationTriple storesVersioningIncl. updates
Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks
Axel Ngonga (InfAI) Project Overview June 1, 2016 12 / 13
Features of the HOBBIT platform
Addresses all steps of the LinkedData LifecycleBenchmarks derived from industryuse casesReal data under the bechmarksScalable size of benchmarksOpen-source implementationOnline instance on server clusterUses established deploymenttechnologies
Axel Ngonga (InfAI) Project Overview June 1, 2016 13 / 13
Join HOBBIT
Participate in the surveyJoin the HOBBIT communityJoin the split sessionsProvide KPIsProvide datasetsJoin the platform development
Axel Ngonga (InfAI) Project Overview June 1, 2016 14 / 13
Thank You
http://project-hobbit.eu/get-involved/
http://goo.gl/forms/1iRIoG4Xpb
https://twitter.com/hobbit_project
Axel Ngonga (InfAI) Project Overview June 1, 2016 15 / 13