RDF Analytics... SPARQL and Beyond

Digital Enterprise Research Institute www.deri.ie

Enabling networked knowledge

© Copyright 2011 Digital Enterprise Research Institute. All rights reserved.



@fsheer

Fadi Maali

RDF Analytics… SPARQL and Beyond…

[email protected]



Why analytics (1/2)



Why analytics (2/2)



Appetite Whetting (1/3)

Google accurately detects Flu trend ahead of the U.S. Center for Disease Control.

http://www.google.org/flutrends/about/how.html



http://www.dailymail.co.uk/sciencetech/article-2120416/Twitter-predicts-stock-prices-accurately-investment-tactic-say-scientists.html





http://www.nature.com/srep/2011/111215/srep00196/full/srep00196.html

Flavor pyramids for North American and East Asian cuisines



Data Science and RDF

Ø  Can we do “data science” using RDF data?

§  Do we have the data?

§  Do we have the tools?

Ø  Why should we use RDF?



RDF Characteristics

§  Graph data model

§  Clearly defined semantics

§  Support Web-scale distributed publication



Available RDF Data

§  Freebase has 1.2 billion triples (Google) §  The LOD Cloud has more than 31 billion triples §  Embedded RDF data: schema.org, Drupal…

http://lod-cloud.net/



Available RDF Tools

In this presentation we focus on the standard SPARQL: q  W3C Recommendation

q  Supports Querying, transforming and updating RDF data

q  Large number of available implementations

q  Define a communication protocol

q  427 public SPARQL endpoints registered on the DataHub* * http://sw.deri.org/~aidanh/docs/epmonitorISWC.pdf



RDF Data… a graph



SELECT ?name WHERE{ ?p :name ?name . }ORDER BY ?name

SPARQL… Simple queries



SELECT ?gender (COUNT(*) AS ?count) WHERE{ ?p :gender ?gender } GROUP BY ?gender

SPARQL… BI queries



SELECT ?gender (COUNT(*) AS ?count) WHERE{ ?p :gender ?gender } GROUP BY ?gender




SELECT ?name (COUNT(?n) AS ?neighbours) WHERE{ ?p :knows ?n . ?p :name> ?name . } GROUP BY ?p ?name ORDER BY desc(?neighbours)




SELECT ?name (COUNT(?n) AS ?neighbours) WHERE{ ?p :knows ?n . ?p :name> ?name . } GROUP BY ?p ?name ORDER BY desc(?neighbours)





Ø  How influential a person is within a social network Ø  How a road is within an urban network Ø  How central an employee in an enterprise



SPARQL… Graph measure

Can we use SPARQL to compute shortest paths in the graph? Short answer: NO! Long answer: Let’s try!



SELECT ?v1 ?v2 (MIN(?l) AS ?shortestPath) WHERE{ { ?v1 :knows ?v2 BIND (1 AS ?l) } UNION { ?v1 :knows{2} ?v2 BIND (2 AS ?l) } UNION { ?v1 :knows{3} ?v2 BIND (3 AS ?l) } FILTER (?v1 != ?v2) } GROUP BY ?v1 ?v2

SPARQL… graph measure







Ø  finding directions between physical locations

Ø  finding the most direct way to contact a person

Ø  finding the min-delay communication path



SPARQL… clustering

Can we do clustering using SPARQL? YES! Peer-pressure algorithm implemented using (almost only) SPARQL*

* http://yarcdata.com/blog/?p=318



DROP GRAPH <urn:ga/g/xjz1> ; CREATE GRAPH <urn:ga/g/xjz1>; INSERT {GRAPH <urn:ga/g/xjz1> {?s :cluster ?clus3}} WHERE { SELECT ?s (SAMPLE(?clus) AS ?clus3) { { SELECT ?s (MAX(?clusCt) AS ?maxClusCt) { SELECT ?s ?clus (COUNT(?clus) AS ?clusCt) WHERE { ?s :knows ?o . GRAPH <urn:ga/g/xjz0> { ?o :cluster ?clus } } GROUP BY ?s ?clus } GROUP BY ?s } { SELECT ?s ?clus (COUNT(?clus) AS ?clusCt) WHERE { ?s :knows ?o . GRAPH <urn:ga/g/xjz0> { ?o :cluster ?clus } } GROUP BY ?s ?clus } FILTER (?clusCt = ?maxClusCt) } GROUP BY ?s }




DROP GRAPH <urn:ga/g/xjz1> ; CREATE GRAPH <urn:ga/g/xjz1>; INSERT {GRAPH <urn:ga/g/xjz1> {?s :cluster ?clus3}} WHERE { SELECT ?s (SAMPLE(?clus) AS ?clus3) { { SELECT ?s (MAX(?clusCt) AS ?maxClusCt) { SELECT ?s ?clus (COUNT(?clus) AS ?clusCt) WHERE { ?s :knows ?o . GRAPH <urn:ga/g/xjz0> { ?o :cluster ?clus } } GROUP BY ?s ?clus } GROUP BY ?s } { SELECT ?s ?clus (COUNT(?clus) AS ?clusCt) WHERE { ?s :knows ?o . GRAPH <urn:ga/g/xjz0> { ?o :cluster ?clus } } GROUP BY ?s ?clus } FILTER (?clusCt = ?maxClusCt) } GROUP BY ?s }




SPARQL Expressivity

Ø  BI-like operations (rollup and drilldown)

Ø  Graph Measures

Ø  Iterative algorithms (Clustering)



SPARQL Scalability…

One approach is to use a scale-out architecture… think MapReduce or Hadoop q  Translate SPARQL into MapReduce

q  Process RDF data directly in MapReduce



All examples used in this presentation and equivalent of some of them using Pig Latin is available at: https://github.com/fadmaa/rdf-analytics

Conclusion

Ø  Can we do “data science” using RDF data?

§  Do we have the data? YES

§  Do we have the tools? Almost

v  Is SPARQL expressive enough? Almost v  Does it scale? Yes… in principle, No in practice v  Is it usable/easy? Not really

Education

RDF Analytics... SPARQL and Beyond