INARC

Charu C. Aggarwal (I2 Contributions)

Scalable Graph Querying and Indexing

Task I2.2

Charu C. Aggarwal

IBMCollaborators (across all tasks): Jiawei Han (UIUC), Guojun Qi

(UIUC), Thomas Huang (UIUC), Tarek Abdelzaher (UIUC), Xifeng Yan (UCSB), Arjit Khan (UCSB), Nan Li (UCSB), Amotz Bar-Noy (CUNY), Simon Shamoun (CUNY)

Other Tasks: I1.2 (funded), I3 (collaborator)

INARC

Overview of Contributions

Project I1 Contributions:

Methods for Sensor Selection in Dynamic Information Networks (Collaboration with A. Bar-Noy (CUNY) and S. Shamoun (CUNY))=>Submitted to DCOSS 2011

Data fusion of heterogeneous data with the use of network links: specific application to text and visual data

QoI robust inference with the use of heterogeneous data fusion

Joint work with G. Qi (UIUC) and T. Huang (UIUC)

Papers accepted in CVPR, WWW 2011; one submitted to KDD 2011

Collaboration with Tarek Abdelzaher (UIUC) on data selection for regression and fact-finders for fusion=> Submitted to DCOSS 2011, Fusion 2011

Project I2 Contributions: Large Scale Indexing (Focus of this talk):

Methods for Indexing Massive Disk-Resident Graphs (Aggarwal (IBM), Zhao (UIUC), and Han (UIUC))=> Submit to PVLDB 2012

Methods for Indexing Dynamic Network Streams (Aggarwal (IBM), Khan (UCSB), Yan (UCSB))=> Submit to PVLDB 2012

Methods for label-based query index (Joint work with Li (UCSB))=>SDM 2011

INARC

Overview of Contributions

Project I3 Contributions:

Unfunded Collaborator for project I3=> Actively collaborated with Jiawei Han on mining of networks with heterogeneous links and incomplete attributes (Task I3.1)

Designed methods for clustering heterogeneous information networks with heterogeneous links and incomplete attributes (joint work with Yizhou Sun (UIUC) and Jiawei Han (UIUC)) => Submitted to KDD 2011

Designed methods for link inference in the noise and heterogeneous information network scenario (joint work with Barbier (UIUC), Gupta (UIUC), Sun (UIUC) and Han (UIUC))=> Submitted to ASONAM 2011

INARC

Scalable Indexing and Querying Massive Graphs (I2.2)

Indexing Methods for Large Scale Static and Dynamic Networks

Methods for Indexing Massive Disk-Resident Graphs (Aggarwal (IBM), Zhao (UIUC), and Han (UIUC))

Application to Shortest Path Queries

Can be extended to connectivity and other structural queries

Methods for Indexing Dynamic Network Streams (Aggarwal (IBM), Khan (UCSB), Yan (UCSB))

Applications such as social and information networks show continuous edge-base activity

Results in edge streams

Methods for frequency-based and structural indexing of graph streams

INARC

Label-based Querying of Massive Dynamic Graphs

Many networks have labels associated with some of the nodes which may need to be learned or queried

Eg. Which node belongs to a specific topic?

Challenges:

Network may be dynamic with edges and nodes which continuously evolve over time

The nodes in the network may contain content, which needs to be used in the classification process

The classification-queries need to be resolved in real-time.

Designed methods for constructing dynamic structural index, which can be continuously updated and used for label-based queries (Aggarwal (IBM) and Li (UCSB)): Accepted to SIAM Conference on Data Mining, 2011.

INARC

Indexing Graphs: The compression approach

Core idea is to coarsen the graph into a smaller number of nodes

Method for coarsening depends upon application:

In static (disk-resident) methods, we use dense pattern mining methods in order to coarsen graph into a smaller number of nodes

In dynamic (stream-based) methods, we use a hash-based probabilistic sketch in order to coarsen graph into a smaller number of nodes.

Use probabilistic query-processing on the coarsened graph in order to provide approximate responses to queries

INARC

Static Indexing of Disk-Resident Graphs

Design methods for indexing massive disk-resident networks

Typical social and information networks may be massive and may need to be stored on disk

For example, for a network containing 10^7 nodes, we may have of the order of 10^(13) edges

Network cannot be held in main memory, and in some cases not even on disk

First present methods where network cannot be held in main memory, but is stored on disk

Design methods for shortest path indexing methods: For a given pair of nodes s and t, determine shortest distance between them

Very fast for memory resident graphs=> Very slow for disk-resident case

INARC

Static Indexing of Disk-Resident Graphs

Core Approach for static indexing: Determine dense sets of nodes in the network using a pattern mining approach

Compress sets of nodes into supernodes

Node compression will result in self-loops: Eliminate self-loops

Perform query-processing on compressed graph

Note that passing through super-node is equivalent to passing through a set of edges: associate node-penalty with super-node

Observation: Vast majority of edges lie in dense regions which are compressed into self-loops

Massive reduction in size of network: allows main-memory storage for query-processing

INARC

Setting Node Penalties (Observations)

When we run the query-processing technique on the coarsened graph as a surrogate, passing through a super-node is modeled as entering through a random node in the compressed subgraph and exiting another random node

Use probabilistic node penalty associated with such nodes corresponding to pairwise shortest-path distances in the compressed portion

Modified Query Processing: Determine shortest-path distance between s’ and t’ in compressed graph with node-penalties

Note that node-penalties are random variables and the shortest-path distance could be average or worst-case.

Important: Dense region compression leads to histogram with small number of buckets (most distances are 1 or 2 within region).

INARC

Setting Node Penalties

INARC

Query Processing

Start at node s’ and progressively maintain the random variables corresponding to the shortest average (or worst-case) distances to the nodes.

Maintain probabilistic label with each node containing the probabilistic histogram with shortest-path distance

Grow set S of nodes with probabilistic shortest path distances

In each iteration, we grow set S by one node after updating the node labels of its nearest neighbors (requires probabilistic addition of distances of shortest path to edge weight and node penalty).

Terminate on reaching sink node t’

Provides either approximation using compressed graph or access compressed fragments from disk to get exact distances

INARC

Adding shortest path distances to node penalty

INARC

Other details of approach

Other algorithmic components of approach

A dense pattern mining algorithm which is designed in order to determine the dense regions of the network for compression purposes

Design a sampling-based approach which uses a limited number of passes over the disk-resident data set

A method for reconstructing the shortest path from the compressed shortest path distance

Retrieve fragments from disk-resident data set

Use shortest path distance within each fragment to reconstruct the distance

INARC

Experimental Results (DBLP Data Set)

INARC

Dynamic Indexing of Massive Graph Streams (Joint work with A. Khan (UCSB) and X. Yan (UCSB))

The previous methods were designed for static indexing of massive graphs

In many applications, we need indexes for the case in which the graphs may continuously be updated over time (edge activity)

Edge-based activity in networks result in a stream of edges

Activity in a social network can be modeled as an edge-stream

Assumption is much more rigorous in previous case

Not enough space to hold even the edges on disk and their arrival times

Not enough space to hold even the total number of possible distinct edges on disk

Hard to perform structural analysis when the network is so dynamic and cannot even be stored at a given point in time

INARC

Dynamic Indexing of Massive Graph Streams

Use a 3-dimensional hash-based sketch structure in order to create the dynamic index

Borrows the concept of a sketch structure from data stream analysis and generalizes it to the case of structural data

Queries which can be handled:

Determine the frequency of a given edge in the graph

Determine the aggregate frequency of edges in a subgraph

Determine the minimum frequency of edges in a subgraph

Determine all edges with frequency above a given threshold

Determine the connected components with frequency above a given threshold

INARC

The sketch based approach: Core Idea

Model: Assume that edges are received continuously over time

Use a 3-dimensional matrix of size h X h X w

The parameter h indicates the range of the hashing

The parameter w indicates the number of hash functions

For an edge (i, j) the node i is hashed into the first dimension using k-th hash function g^k() with range [0, h-1]

The node j is hashed into the second dimension using k-th hash function g^k() in the range [0, h-1]

There are w slices of size h X h corresponding to different hash functions g^k() where k lies in the range [1, w]

Multiple hash functions provide robustness

The entire frequency behavior of edges is mapped into the structure of size h X h X w

INARC

The sketch based approach: Core Idea

Model: Assume that edges are received continuously over time

Use a 3-dimensional matrix of size h X h X w

The parameter h indicates the range of the hashing

The parameter w indicates the number of hash functions

For an edge (i, j) the node i is hashed into the first dimension using k-th hash function g^k() with range [0, h-1]

The node j is hashed into the second dimension using k-th hash function g^k() in the range [0, h-1]

There are w slices of size h X h corresponding to different hash functions g^k() where k lies in the range [1, w]

Multiple hash functions provide robustness

The entire frequency behavior of edges is mapped into the structure of size h X h X w

INARC

The sketch based approach: Stream Incremental Update

The update process is straightforward and requires an incremental step involving the application of w different hash functions for each incoming edge (i, j)

Start off by setting each cell in the h X h x w synopsis matrix V to 0

Assume incoming edge (i, j) and frequency r(i, j)For each incoming edge (i, j) and k-th hash

function g^k(), update the cell V(g^k(i), g^k(j), k) by r(i, j)

The update process is applied for each of the w different hash functions

INARC

Subgraph Frequency-based Queries

Query: Determine frequency of edges in subgraph with node set S

For each edge (i, j) in subgraph S, determine the value of V(g^k(i), g^k(j), k) for different hash functions k

Determine the minimum value of V(g^k(i), g^k(j), k) over the different hash functions k

Sum up value over different edges Key Result: Derived mathematical error bound

=> With probability at least 1 – (|S|*|S|/(h*h*f))^w the error is at most L.f, where L is the total frequency of edges received=> Details of proof available in paper.

INARC

Inverse Frequency-based Queries

Query: Determine all edges with frequency above a given threshold

Key Augmentation to Data Structure: Each hash cell contains an inverted list of all the nodes which map to it

For a given frequency threshold determine all hash cells (for each hash function) above the threshold

Determine the intersection of all the inverted lists pointed to by determined hash cells=> using multiple hash functions to reduce collision error.

Report only edges which are present in the intersection of the hash functions => error reduces logarithmically with number of hash functions

INARC

Connected-subgraph Queries

Query: Determine all connected subgraphs containing only edges with frequency above a given threshold

Determine edges using the approach discussed in previous query

Determine connected components with the use of the found edges

Numerous other structural queries can be handled with the use of the compressed graph.Error bounds hold for a subset of

queries

INARC

Experimental Results (last.fm Data Set)

INARC

Label-based querying of Massive Graphs (Joint work With N. Li (UCSB))

We assume that we have a massive graph with nodes interconnected by edges, and nodes which contain content.

The nodes and edges of the graph are continuously received over time.

Need to respond to label-based queries dynamically using both the structure and content

Need to use the content in the classification process

Core Idea:

Construct semi-bipartite augmentation of the network which uses the content in the form of synthetic word nodes

Create a fast inverted index which can respond to label-based queries.

INARC

Augmented Bipartite Network

INARC

Key Results

Method is designed for dynamic classification queries (no static pre-processing) A dynamic classification index

Maintain the semi-bipartite augmentation dynamically

Maintain inverted representation with the word nodes=>Incorporation of content with structure

Perform random walk starting at query node and report majority label as the appropriate class

Theoretical Result: Sampling a certain number of walks reduces the error estimation exponentially=> Proof available in paper => Use of Hoeffding bound

Paper accepted in SDM, 2011.

INARC

Experimental Results

INARC

Military Relevance

Military information networks are extremely large and require dynamic and time-varying methods for querying and indexing

Require real-time responses to structural queries for data which is large and may be added to rapidly => network stream scenario

Methods are also applicable to resource-constrained environments even for very large data sets

Classification query provides methods to identify nodes of relevance in a given information network query => which is the node most relevant to a given topic? => Where is the information I want?

INARC

The Path Ahead: Future Work

Extend the kinds of queries which can be handled by different kinds of methods:Structural connectivity queriesShortest paths in dynamic graph streamsExtend first method to dynamic case

Extend methods to uncertain graphsChallenge of uncertainty: Expected path length

methods do not work!Even a small probability of disconnection of

source and sink leads to expected path length of infinity

Design probabilistic model which uses the shortest path with threshold probability

INARC

Publications (Accepted)

C. C. Aggarwal (IBM), N. Li (UCSB). On Dynamic Node-Classification in Content-based Networks, Accepted to the SDM Conference, 2011.

G. Qi (UIUC), C. C. Aggarwal (IBM), T. Huang. (UIUC) Towards Semantic Knowledge Propagation between Text and Web Images, Accepted to the WWW Conference, 2011.

G. Qi (UIUC), C. C. Aggarwal (IBM), T. Huang (UIUC). Towards Cross-Category Learning of Visual Concepts. Accepted to the CVPR Conference, 2011.

C. C. Aggarwal (IBM), A. Khan (UCSB), X. Yan (UCSB). On Flow Authority Discovery in Social Networks, Accepted to the SDM Conference, 2011.

C. C. Aggarwal, Y. Xie, P. Yu. On Community Discovery in Locally Heterogeneous Networks, Accepted to the SDM Conference, 2011.

C. C Aggarwal, Y. Zhao, P. Yu. On Wavelet Decomposition of Uncertain Streams, CIKM Conference, 2010.

M. Masud, L. Khan, B. Thuraisingham, C. Aggarwal (IBM), J Gao (UIUC), J. Han (UIUC), On Novel Class Detection in Concept Drifting Data Streams, ICDM Conference, 2010

INARC

Publications (Submitted)

G. Qi (UIUC), C. C. Aggarwal (IBM), T. Huang (UIUC). Community Detection with Edge Content, Submitted to the SIGIR Conference, 2011.

Y. Sun (UIUC), C. C. Aggarwal (IBM), J. Han (UIUC), On Community Discovery in Heterogeneous Networks with Incomplete Attributes, Submitted to the KDD Conference, 2011.

G. Qi (UIUC), C, C, Aggarwal (IBM), T. Huang (UIUC), Transfer Learning with Distance Functions between Text and Images, Submitted to the KDD Conference, 2011.

C. C. Aggarwal (IBM), A. Bar-Noy (CUNY), S. Shamoun (CUNY). On Sensor Selection in Linked Information Networks, Submitted to the DCOSS Conference, 2011.

INARC

Publications (Submitted)

Y. Sun (UIUC), C. C. Aggarwal (IBM), J. Han (UIUC), On Link Inference in Bibliographic Networks, Submitted to the ASONAM Conference, 2011.

M. Gupta (UIUC), C, C, Aggarwal (IBM), J. Han (UIUC), On Evolutionary Clustering and Analysis of Heterogeneous Information Networks, Submitted to the ASONAM Conference, 2011.

C. C. Aggarwal (IBM), A. Bar-Noy (CUNY), S. Shamoun (CUNY). On Sensor Selection in Linked Information Networks, Submitted to the DCOSS Conference, 2011.

C. C. Aggarwal (IBM), P. Zhao (UIUC), J. Han (UIUC). On Shortest Path Indexing with Disk-Resident Graphs, Submitted to the PVLDB Conference, 2012.

C. C. Aggarwal (IBM), A. Khan (UCSB), X. Yan (UCSB). On Query Processing of Massive Graph Streams, Submitted to the PVLDB Conference, 2012.

INARC

Publications (Submitted)

T. Abdelzaher (UIUC), D. Wang (UIUC), H. Ahmadi (UIUC), J. Pasternack (UIUC), M. Gupta (UIUC), J. Han (UIUC), O. Fetemieh (UIUC), H. Le (UIUC) C. C. Aggarwal (IBM), On Bayesian Information of Fact-Finding in Information networks, Submitted to the Fusion Conference, 2011.

D. Wang (UIUC), H. Ahmadi (UIUC), T. Abdelzaher (UIUC), H. Chenji. R. Stoleru, C, C, Aggarwal (IBM). Data Models for Optimizing Quality-of-Information in Cost-Sensitive Data Fusion, Submitted to the DCOSS Conference, 2011.